omic/digital-trial
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Digital Trials pipeline configured for WES

Browse Source 
Source-only snapshot of the cluster branch for WES execution. Large
reference files (HPA/MANE/ensemble FASTA, model weights, ~597 MB) are
omitted: they are baked into the container images at build time and
mounted from the dreamdock-data PVC at runtime, and exceed the Gitea
request size limit.

Pipeline entry point is main.nf, which orchestrates the biotransformer,
conplex and tissue modules as a single workflow. Ligand inputs are read
from the eureka workspace; protein_zarr and chembl_db come from the
dreamdock-data PVC.
This commit is contained in:
Olamide Isreal
2026-07-27 21:59:52 +01:00
commit 9e75f44f1a
86 changed files with 10142 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
.nextflow.log.4 Normal file
View File

@@ -0,0 +1,117 @@
Feb-27 17:43:13.583 [main] DEBUG nextflow.cli.Launcher - $> nextflow run test.nf -params-file /data/dreamdock-data/caline-run6/digtrial/caline-run6.json
Feb-27 17:43:13.956 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.3
Feb-27 17:43:13.997 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/omic/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,nf-azure@1.16.0,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.0,nf-k8s@1.0.0,nf-tower@1.11.3,nf-wave@1.12.1
Feb-27 17:43:14.037 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-27 17:43:14.038 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-27 17:43:14.042 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Feb-27 17:43:14.057 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Feb-27 17:43:14.085 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugra/digital_trials/nextflow.config
Feb-27 17:43:14.088 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugra/digital_trials/nextflow.config
Feb-27 17:43:14.130 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/omic/.nextflow/secrets/store.json
Feb-27 17:43:14.136 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6ef81f31] - activable => nextflow.secret.LocalSecretsProvider@6ef81f31
Feb-27 17:43:14.166 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Feb-27 17:43:14.913 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Feb-27 17:43:14.935 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [wise_snyder] DSL2 - revision: 5cb3581ad6
Feb-27 17:43:14.938 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Feb-27 17:43:14.939 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-tower@1.11.3]
Feb-27 17:43:14.942 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.11.3
Feb-27 17:43:14.955 [main] INFO org.pf4j.AbstractPluginManager - Plugin 'nf-tower@1.11.3' resolved
Feb-27 17:43:14.955 [main] INFO org.pf4j.AbstractPluginManager - Start plugin 'nf-tower@1.11.3'
Feb-27 17:43:14.972 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-tower@1.11.3
Feb-27 17:43:15.045 [main] DEBUG nextflow.Session - Session UUID: ab7a7317-9f8e-49f9-8ba6-8349b24a36ef
Feb-27 17:43:15.046 [main] DEBUG nextflow.Session - Run name: wise_snyder
Feb-27 17:43:15.047 [main] DEBUG nextflow.Session - Executor pool size: 80
Feb-27 17:43:15.059 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Feb-27 17:43:15.067 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Feb-27 17:43:15.104 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.3 build 5949
Created: 02-06-2025 20:56 UTC
System: Linux 6.11.0-26-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 3937567@k8s-node23 [127.0.1.1]
CPUs: 80 - Mem: 251.6 GB (5.3 GB) - Swap: 0 (0)
Feb-27 17:43:15.139 [main] DEBUG nextflow.Session - Work-dir: /data/bugra/digital_trials/work [btrfs]
Feb-27 17:43:15.140 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugra/digital_trials/bin
Feb-27 17:43:15.158 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Feb-27 17:43:15.176 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Feb-27 17:43:15.182 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Feb-27 17:43:15.185 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Feb-27 17:43:15.217 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Feb-27 17:43:15.229 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 81; maxThreads: 1000
Feb-27 17:43:15.409 [main] DEBUG nextflow.Session - Session start
Feb-27 17:43:15.879 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Feb-27 17:43:16.759 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:16.760 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:16.767 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Feb-27 17:43:16.777 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=80; memory=251.6 GB; capacity=80; pollInterval=100ms; dumpInterval=5m
Feb-27 17:43:16.780 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Feb-27 17:43:16.815 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'HUMAN_TRANSFORMER': maxForks=0; fair=false; array=0
Feb-27 17:43:16.900 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:16.900 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:16.902 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'GET_FINAL_METABOLITES_STATIC': maxForks=0; fair=false; array=0
Feb-27 17:43:16.915 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:gpu_process` matches labels `gpu_process` for process with name PREPROCESS_PROTEIN
Feb-27 17:43:16.920 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:16.920 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:16.922 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PREPROCESS_PROTEIN': maxForks=0; fair=false; array=0
Feb-27 17:43:16.984 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:16.984 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:16.985 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'CONPLEX_ALL': maxForks=0; fair=false; array=0
Feb-27 17:43:17.016 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:17.016 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:17.017 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'NETWORK_ENRICHMENT': maxForks=0; fair=false; array=0
Feb-27 17:43:17.025 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:17.025 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:17.026 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'TISSUE_DISTRIBUTION': maxForks=0; fair=false; array=0
Feb-27 17:43:17.039 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Feb-27 17:43:17.039 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Feb-27 17:43:17.040 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'BIO_METRICS': maxForks=0; fair=false; array=0
Feb-27 17:43:17.043 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: HUMAN_TRANSFORMER, METABOLITES_BY_MASS, GET_FINAL_METABOLITES_STATIC, CONPLEX_ALL, NETWORK_ENRICHMENT, GET_FINAL_METABOLITES, CONPLEX, BIO_METRICS, SUPER_TRANSFORMER, TISSUE_DISTRIBUTION, MERGE_DRUG, MERGE_INTERACTIONS, PREPROCESS_PROTEIN, ORDERED_SEQUENCE
Feb-27 17:43:17.046 [main] DEBUG nextflow.Session - Igniting dataflow network (11)
Feb-27 17:43:17.053 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > HUMAN_TRANSFORMER
Feb-27 17:43:17.069 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > GET_FINAL_METABOLITES_STATIC
Feb-27 17:43:17.071 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PREPROCESS_PROTEIN
Feb-27 17:43:17.071 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /data/dreamdock-data/caline-run6/digtrial/input/; pattern: *.csv; options: [:]
Feb-27 17:43:17.071 [PathVisitor-3] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /mnt/dreamdock-data/digital_trials/input/blank/; pattern: *.fasta; options: [:]
Feb-27 17:43:17.071 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > CONPLEX_ALL
Feb-27 17:43:17.072 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NETWORK_ENRICHMENT
Feb-27 17:43:17.073 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > TISSUE_DISTRIBUTION
Feb-27 17:43:17.073 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > BIO_METRICS
Feb-27 17:43:17.074 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_c432c676a3e7c0cb: /data/bugra/digital_trials/test.nf
Script_e39e54e7aa543fd0: /data/bugra/digital_trials/main_biotransformer.nf
Script_d345b7a65a8f3255: /data/bugra/digital_trials/main_conplex.nf
Script_ec01f2a38db15aa5: /data/bugra/digital_trials/main_tissue.nf
Feb-27 17:43:17.075 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Feb-27 17:43:17.075 [main] DEBUG nextflow.Session - Session await
Feb-27 17:43:17.299 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:43:17.303 [Task submitter] INFO nextflow.Session - [b8/48fb21] Submitted process > HUMAN_TRANSFORMER (1)
Feb-27 17:43:24.518 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: HUMAN_TRANSFORMER (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/b8/48fb213f9b113a32b1649ada113509]
Feb-27 17:43:24.519 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Feb-27 17:43:24.547 [TaskFinalizer-1] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Feb-27 17:43:24.559 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:43:24.560 [Task submitter] INFO nextflow.Session - [f5/a871d4] Submitted process > GET_FINAL_METABOLITES_STATIC (1)
Feb-27 17:43:28.371 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: GET_FINAL_METABOLITES_STATIC (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/f5/a871d46063710b8620594ede8bc8f2]
Feb-27 17:43:28.423 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:43:28.424 [Task submitter] INFO nextflow.Session - [54/3e3c99] Submitted process > CONPLEX_ALL (1)
Feb-27 17:45:36.007 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: CONPLEX_ALL (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/54/3e3c99cd7077434aa2732f31dd7b5c]
Feb-27 17:45:36.075 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:45:36.075 [Task submitter] INFO nextflow.Session - [24/f1f52e] Submitted process > TISSUE_DISTRIBUTION (1)
Feb-27 17:45:36.081 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:45:36.082 [Task submitter] INFO nextflow.Session - [aa/de8363] Submitted process > BIO_METRICS (1)
Feb-27 17:45:36.087 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Feb-27 17:45:36.088 [Task submitter] INFO nextflow.Session - [f3/73824b] Submitted process > NETWORK_ENRICHMENT (1)
Feb-27 17:45:44.940 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 6; name: NETWORK_ENRICHMENT (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/f3/73824bf939c5be1858680c6af0e5e6]
Feb-27 17:45:47.754 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: TISSUE_DISTRIBUTION (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/24/f1f52e39bf1496db70950a7c5a04c8]
Feb-27 17:45:58.442 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: BIO_METRICS (1); status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/digital_trials/work/aa/de8363f3884d595a7a7df0291508ef]
Feb-27 17:45:58.448 [main] DEBUG nextflow.Session - Session await > all processes finished
Feb-27 17:45:58.542 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Feb-27 17:45:58.542 [main] DEBUG nextflow.Session - Session await > all barriers passed
Feb-27 17:45:58.548 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Feb-27 17:45:58.549 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Feb-27 17:45:58.557 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=6; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=3m 1s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=3; peakCpus=3; peakMemory=11 GB; ]
Feb-27 17:45:58.693 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Feb-27 17:45:58.772 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-tower@1.11.3'
Feb-27 17:45:58.772 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-tower
Feb-27 17:45:58.774 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Feb-27 17:45:58.775 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye