omic/digital-trial
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Digital Trials pipeline configured for WES

Browse Source 
Source-only snapshot of the cluster branch for WES execution. Large
reference files (HPA/MANE/ensemble FASTA, model weights, ~597 MB) are
omitted: they are baked into the container images at build time and
mounted from the dreamdock-data PVC at runtime, and exceed the Gitea
request size limit.

Pipeline entry point is main.nf, which orchestrates the biotransformer,
conplex and tissue modules as a single workflow. Ligand inputs are read
from the eureka workspace; protein_zarr and chembl_db come from the
dreamdock-data PVC.
This commit is contained in:
Olamide Isreal
2026-07-27 21:59:52 +01:00
commit 9e75f44f1a
86 changed files with 10142 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
.nextflow.log.3 Normal file
View File

@@ -0,0 +1,117 @@
Mar-09 15:03:54.815 [main] DEBUG nextflow.cli.Launcher - $> nextflow run test.nf -params-file /data/dreamdock-data/caline-run6/digtrial/caline-run6.json
Mar-09 15:03:55.224 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.3
Mar-09 15:03:55.255 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/omic/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,nf-azure@1.16.0,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.0,nf-k8s@1.0.0,nf-tower@1.11.3,nf-wave@1.12.1
Mar-09 15:03:55.294 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Mar-09 15:03:55.295 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Mar-09 15:03:55.299 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Mar-09 15:03:55.314 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Mar-09 15:03:55.342 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugra/digital_trials/nextflow.config
Mar-09 15:03:55.346 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugra/digital_trials/nextflow.config
Mar-09 15:03:55.386 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/omic/.nextflow/secrets/store.json
Mar-09 15:03:55.391 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6ef81f31] - activable => nextflow.secret.LocalSecretsProvider@6ef81f31
Mar-09 15:03:55.422 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Mar-09 15:03:56.211 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Mar-09 15:03:56.234 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [dreamy_panini] DSL2 - revision: 0e14afbd94
Mar-09 15:03:56.236 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Mar-09 15:03:56.238 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-tower@1.11.3]
Mar-09 15:03:56.240 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.11.3
Mar-09 15:03:56.252 [main] INFO org.pf4j.AbstractPluginManager - Plugin 'nf-tower@1.11.3' resolved
Mar-09 15:03:56.252 [main] INFO org.pf4j.AbstractPluginManager - Start plugin 'nf-tower@1.11.3'
Mar-09 15:03:56.269 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-tower@1.11.3
Mar-09 15:03:56.340 [main] DEBUG nextflow.Session - Session UUID: c52cdca5-7bcc-4d96-8a19-824595cbfc30
Mar-09 15:03:56.341 [main] DEBUG nextflow.Session - Run name: dreamy_panini
Mar-09 15:03:56.341 [main] DEBUG nextflow.Session - Executor pool size: 80
Mar-09 15:03:56.354 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Mar-09 15:03:56.362 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Mar-09 15:03:56.395 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.3 build 5949
Created: 02-06-2025 20:56 UTC
System: Linux 6.11.0-26-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 2939340@k8s-node23 [127.0.1.1]
CPUs: 80 - Mem: 251.6 GB (33.6 GB) - Swap: 0 (0)
Mar-09 15:03:56.430 [main] DEBUG nextflow.Session - Work-dir: /data/bugra/digital_trials/work [btrfs]
Mar-09 15:03:56.430 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugra/digital_trials/bin
Mar-09 15:03:56.457 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Mar-09 15:03:56.477 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Mar-09 15:03:56.483 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Mar-09 15:03:56.486 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Mar-09 15:03:56.520 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Mar-09 15:03:56.533 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 81; maxThreads: 1000
Mar-09 15:03:56.691 [main] DEBUG nextflow.Session - Session start
Mar-09 15:03:57.165 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Mar-09 15:03:58.082 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.083 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.093 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Mar-09 15:03:58.108 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=80; memory=251.6 GB; capacity=80; pollInterval=100ms; dumpInterval=5m
Mar-09 15:03:58.111 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Mar-09 15:03:58.146 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'HUMAN_TRANSFORMER': maxForks=0; fair=false; array=0
Mar-09 15:03:58.247 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.247 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.249 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'GET_FINAL_METABOLITES_STATIC': maxForks=0; fair=false; array=0
Mar-09 15:03:58.264 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:gpu_process` matches labels `gpu_process` for process with name PREPROCESS_PROTEIN
Mar-09 15:03:58.269 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.270 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.271 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PREPROCESS_PROTEIN': maxForks=0; fair=false; array=0
Mar-09 15:03:58.337 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.337 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.338 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'CONPLEX_ALL': maxForks=0; fair=false; array=0
Mar-09 15:03:58.371 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.371 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.372 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'NETWORK_ENRICHMENT': maxForks=0; fair=false; array=0
Mar-09 15:03:58.380 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.380 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.381 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'TISSUE_DISTRIBUTION': maxForks=0; fair=false; array=0
Mar-09 15:03:58.393 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Mar-09 15:03:58.394 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-09 15:03:58.395 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'BIO_METRICS': maxForks=0; fair=false; array=0
Mar-09 15:03:58.398 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: HUMAN_TRANSFORMER, METABOLITES_BY_MASS, GET_FINAL_METABOLITES_STATIC, CONPLEX_ALL, NETWORK_ENRICHMENT, GET_FINAL_METABOLITES, CONPLEX, BIO_METRICS, SUPER_TRANSFORMER, TISSUE_DISTRIBUTION, MERGE_DRUG, MERGE_INTERACTIONS, PREPROCESS_PROTEIN, ORDERED_SEQUENCE
Mar-09 15:03:58.401 [main] DEBUG nextflow.Session - Igniting dataflow network (11)
Mar-09 15:03:58.408 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > HUMAN_TRANSFORMER
Mar-09 15:03:58.410 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > GET_FINAL_METABOLITES_STATIC
Mar-09 15:03:58.411 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PREPROCESS_PROTEIN
Mar-09 15:03:58.412 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > CONPLEX_ALL
Mar-09 15:03:58.412 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NETWORK_ENRICHMENT
Mar-09 15:03:58.412 [PathVisitor-3] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /mnt/dreamdock-data/digital_trials/input/blank/; pattern: *.fasta; options: [:]
Mar-09 15:03:58.412 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /data/dreamdock-data/caline-run6/digtrial/input/; pattern: *.csv; options: [:]
Mar-09 15:03:58.413 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > TISSUE_DISTRIBUTION
Mar-09 15:03:58.414 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > BIO_METRICS
Mar-09 15:03:58.415 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_e39e54e7aa543fd0: /data/bugra/digital_trials/main_biotransformer.nf
Script_d345b7a65a8f3255: /data/bugra/digital_trials/main_conplex.nf
Script_ec2c89c0dbf76553: /data/bugra/digital_trials/test.nf
Script_ec01f2a38db15aa5: /data/bugra/digital_trials/main_tissue.nf
Mar-09 15:03:58.415 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Mar-09 15:03:58.415 [main] DEBUG nextflow.Session - Session await
Mar-09 15:03:58.625 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:03:58.629 [Task submitter] INFO nextflow.Session - [9f/8b91ca] Submitted process > HUMAN_TRANSFORMER (1)
Mar-09 15:04:06.589 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: HUMAN_TRANSFORMER (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/9f/8b91caff0c52ded09b143e6e91cbc0]
Mar-09 15:04:06.591 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Mar-09 15:04:06.623 [TaskFinalizer-1] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Mar-09 15:04:06.635 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:04:06.636 [Task submitter] INFO nextflow.Session - [08/710d4c] Submitted process > GET_FINAL_METABOLITES_STATIC (1)
Mar-09 15:04:11.034 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: GET_FINAL_METABOLITES_STATIC (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/08/710d4cc4e2be03a44b2df2383e7d9b]
Mar-09 15:04:11.122 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:04:11.125 [Task submitter] INFO nextflow.Session - [53/ea8e99] Submitted process > CONPLEX_ALL (1)
Mar-09 15:06:33.396 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: CONPLEX_ALL (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/53/ea8e99cffaf7ca1272d091bfac2929]
Mar-09 15:06:33.433 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:06:33.434 [Task submitter] INFO nextflow.Session - [fb/95c05c] Submitted process > TISSUE_DISTRIBUTION (1)
Mar-09 15:06:33.442 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:06:33.442 [Task submitter] INFO nextflow.Session - [9a/9129e3] Submitted process > BIO_METRICS (1)
Mar-09 15:06:33.448 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-09 15:06:33.449 [Task submitter] INFO nextflow.Session - [16/77a5f6] Submitted process > NETWORK_ENRICHMENT (1)
Mar-09 15:06:43.140 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 6; name: NETWORK_ENRICHMENT (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/16/77a5f693ac1bd2bc124ddc7ec802e3]
Mar-09 15:06:44.636 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: TISSUE_DISTRIBUTION (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/fb/95c05c31fa620d6901fbfc0de32259]
Mar-09 15:06:56.374 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: BIO_METRICS (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/9a/9129e302133c74df23101091230822]
Mar-09 15:06:56.382 [main] DEBUG nextflow.Session - Session await > all processes finished
Mar-09 15:06:56.476 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Mar-09 15:06:56.477 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-09 15:06:56.486 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Mar-09 15:06:56.487 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Mar-09 15:06:56.501 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=6; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=3m 18s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=3; peakCpus=3; peakMemory=11 GB; ]
Mar-09 15:06:56.581 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Mar-09 15:06:56.671 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-tower@1.11.3'
Mar-09 15:06:56.671 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-tower
Mar-09 15:06:56.673 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Mar-09 15:06:56.674 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye