Stop staging the 9.2 GB ChEMBL DB into every task workdir

GET_FINAL_METABOLITES_STATIC took chembl_db as a `path` input, so Nextflow
copied the ~9.2 GB SQLite DB into each task's work directory. With ~29
concurrent tasks that is hundreds of GB of I/O, against a process that also
declared only 1 GB of memory. Every task failed with exit 1 and retried ten
times, producing 746 errors and an empty 1b_final_metabolites/ output.

Pass the DB as a `val` path instead so tasks read it in place from the
dreamdock-data PVC, and give the process parameterised growing memory.
This commit is contained in:
Olamide Isreal
2026-07-28 10:05:17 +01:00
parent 9e75f44f1a
commit 73e2eea4be
2 changed files with 21 additions and 5 deletions

13
main.nf
View File

@@ -29,6 +29,14 @@ params.bt_max_retries = 10
params.bt_fail_action = 'ignore' // 'terminate' or 'ignore'
params.bt_max_forks = 0 // 0 = unlimited, set to N to limit concurrency
// GET_FINAL_METABOLITES_STATIC — queries the ~9.2 GB ChEMBL SQLite DB on the PVC.
// The DB is opened read-only/immutable and is NOT staged into the task workdir, so the
// memory below covers rdkit + the biotransformer CSV, not the database itself.
params.chembl_initial_memory = 4 // GB - starting memory
params.chembl_growth_memory = 4 // GB - additional memory per retry attempt
params.chembl_max_retries = 3
params.chembl_fail_action = 'ignore' // 'terminate' or 'ignore' after retries exhausted
//CONPLEX
params.keep_enst = 'false' //'true' //'false' //to keep individual protein data created by conplex step
params.conplex_initial_memory = 5 // GB - starting memory for conplex
@@ -121,7 +129,10 @@ workflow {
default:
println("Invalid mode specified: ${params.mode}")
}
chembl_ch = Channel.fromPath(params.chembl_db).collect()
// Pass the ChEMBL DB as a plain path string rather than a staged file: at ~9.2 GB,
// staging it per task copied the DB into every work directory. The process reads it
// read-only from the PVC mount instead.
chembl_ch = Channel.value(params.chembl_db)
// GET_FINAL_METABOLITES(HUMAN_TRANSFORMER.out) // TODO: LOCALIZE
GET_FINAL_METABOLITES_STATIC(HUMAN_TRANSFORMER.out, chembl_ch)
//CONPLEX