Stop staging the 9.2 GB ChEMBL DB into every task workdir

GET_FINAL_METABOLITES_STATIC took chembl_db as a `path` input, so Nextflow
copied the ~9.2 GB SQLite DB into each task's work directory. With ~29
concurrent tasks that is hundreds of GB of I/O, against a process that also
declared only 1 GB of memory. Every task failed with exit 1 and retried ten
times, producing 746 errors and an empty 1b_final_metabolites/ output.

Pass the DB as a `val` path instead so tasks read it in place from the
dreamdock-data PVC, and give the process parameterised growing memory.
This commit is contained in:
Olamide Isreal
2026-07-28 10:05:17 +01:00
parent 9e75f44f1a
commit 73e2eea4be
2 changed files with 21 additions and 5 deletions

View File

@@ -274,18 +274,23 @@ process GET_FINAL_METABOLITES {
}
process GET_FINAL_METABOLITES_STATIC {
memory 1.GB
// The ChEMBL DB is ~9.2 GB. It is passed as a `val` path (not a staged `path`
// input) so Nextflow does not copy it into every task work directory — with
// concurrent tasks that multiplied into hundreds of GB of I/O. The DB lives on
// the dreamdock-data PVC and is opened read-only/immutable, so sharing one copy
// across tasks is safe.
memory { params.chembl_initial_memory.toInteger().GB + (task.attempt - 1) * params.chembl_growth_memory.toInteger().GB }
container "${params.container_chembl}"
containerOptions "${params.containerOptions}"
publishDir "${params.outdir}/${params.project_name}/1b_final_metabolites", mode: 'copy'
errorStrategy { task.attempt <= 10 ? 'retry' : 'ignore' }
maxRetries 10
errorStrategy { task.attempt <= params.chembl_max_retries.toInteger() ? 'retry' : params.chembl_fail_action }
maxRetries { params.chembl_max_retries.toInteger() }
input:
path smiles_csv
path chembl_db // Add ChEMBL SQLite database as input
val chembl_db // Path to the ChEMBL SQLite DB on the PVC (not staged into the workdir)
output:
path "${smiles_csv.simpleName}.txt"