omic/digital-trial
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Digital Trials pipeline configured for WES

Browse Source 
Source-only snapshot of the cluster branch for WES execution. Large
reference files (HPA/MANE/ensemble FASTA, model weights, ~597 MB) are
omitted: they are baked into the container images at build time and
mounted from the dreamdock-data PVC at runtime, and exceed the Gitea
request size limit.

Pipeline entry point is main.nf, which orchestrates the biotransformer,
conplex and tissue modules as a single workflow. Ligand inputs are read
from the eureka workspace; protein_zarr and chembl_db come from the
dreamdock-data PVC.
This commit is contained in:
Olamide Isreal
2026-07-27 21:59:52 +01:00
commit 9e75f44f1a
86 changed files with 10142 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

94
nf_metabol_screen_adaptive/.nextflow.log Normal file
View File

@@ -0,0 +1,94 @@
Jul-21 03:07:10.282 [main] DEBUG nextflow.cli.Launcher - $> nextflow run test.nf
Jul-21 03:07:10.349 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.2
Jul-21 03:07:10.366 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,nf-azure@1.16.0,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.0,nf-k8s@1.0.0,nf-tower@1.11.2,nf-wave@1.12.1
Jul-21 03:07:10.387 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-21 03:07:10.388 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-21 03:07:10.390 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-21 03:07:10.399 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-21 03:07:10.415 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/nextflow.config
Jul-21 03:07:10.417 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/nextflow.config
Jul-21 03:07:10.442 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jul-21 03:07:10.445 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@759d81f3] - activable => nextflow.secret.LocalSecretsProvider@759d81f3
Jul-21 03:07:10.450 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jul-21 03:07:10.819 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-21 03:07:10.830 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [confident_euler] DSL2 - revision: b00b655f5b
Jul-21 03:07:10.831 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-21 03:07:10.831 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-21 03:07:10.871 [main] DEBUG nextflow.Session - Session UUID: 5e2c3d52-6405-43b3-a203-f96c06488e9b
Jul-21 03:07:10.871 [main] DEBUG nextflow.Session - Run name: confident_euler
Jul-21 03:07:10.871 [main] DEBUG nextflow.Session - Executor pool size: 20
Jul-21 03:07:10.878 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-21 03:07:10.883 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=60; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Jul-21 03:07:10.901 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.2 build 5947
Created: 13-05-2025 21:58 UTC (14:58 PDT)
System: Linux 6.8.0-52-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu122.04
Encoding: UTF-8 (UTF-8)
Process: 3902257@alien [127.0.1.1]
CPUs: 20 - Mem: 62.6 GB (836.3 MB) - Swap: 2 GB (96.4 MB)
Jul-21 03:07:10.924 [main] DEBUG nextflow.Session - Work-dir: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/work [ext2/ext3]
Jul-21 03:07:10.925 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/bin
Jul-21 03:07:10.935 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-21 03:07:10.946 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-21 03:07:10.967 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Jul-21 03:07:10.993 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-21 03:07:11.002 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 21; maxThreads: 1000
Jul-21 03:07:11.051 [main] DEBUG nextflow.Session - Session start
Jul-21 03:07:11.055 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/execution_trace.txt
Jul-21 03:07:11.178 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-21 03:07:11.314 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-21 03:07:11.315 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-21 03:07:11.320 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-21 03:07:11.324 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=20; memory=62.6 GB; capacity=20; pollInterval=100ms; dumpInterval=5m
Jul-21 03:07:11.327 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-21 03:07:11.348 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'METABOLITE_SCREEN': maxForks=0; fair=false; array=0
Jul-21 03:07:11.396 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-21 03:07:11.397 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-21 03:07:11.398 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-21 03:07:11.399 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_9d98cafae0c3b8e5: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/test.nf
Script_c8d2b32ed0aeeac9: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/main.nf
Jul-21 03:07:11.399 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-21 03:07:11.399 [main] DEBUG nextflow.Session - Session await
Jul-21 03:07:11.552 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-21 03:07:11.554 [Task submitter] INFO nextflow.Session - [f9/21c381] Submitted process > METABOLITE_SCREEN
Jul-21 03:07:11.590 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 127; error: -; workDir: /dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/work/f9/21c381bcd7594501602fb392f344c9]
Jul-21 03:07:11.591 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=60; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Jul-21 03:07:11.600 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=METABOLITE_SCREEN; work-dir=/dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/work/f9/21c381bcd7594501602fb392f344c9
error [nextflow.exception.ProcessFailedException]: Process `METABOLITE_SCREEN` terminated with an error exit status (127)
Jul-21 03:07:11.615 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'METABOLITE_SCREEN'
Caused by:
Process `METABOLITE_SCREEN` terminated with an error exit status (127)
Command executed:
set -e
python /app/screen.py -i metabolites-rev.tsv -z protein_seq.zarr
Command exit status:
127
Command output:
(empty)
Command error:
.command.sh: line 3: python: command not found
Work dir:
/dbs/metabolite-screen-alt/nf_metabol_screen_adaptive/work/f9/21c381bcd7594501602fb392f344c9
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Jul-21 03:07:11.621 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-21 03:07:11.628 [TaskFinalizer-1] DEBUG nextflow.Session - Session aborted -- Cause: Process `METABOLITE_SCREEN` terminated with an error exit status (127)
Jul-21 03:07:11.643 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-21 03:07:11.643 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-21 03:07:11.648 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=18ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-21 03:07:11.648 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-21 03:07:11.649 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-21 03:07:12.149 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-21 03:07:12.189 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-21 03:07:12.197 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

67
nf_metabol_screen_adaptive/.nextflow.log.1 Normal file
View File

@@ -0,0 +1,67 @@
Jul-15 07:43:54.338 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf
Jul-15 07:43:54.407 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 24.10.3
Jul-15 07:43:54.418 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: nf-amazon@2.9.2,nf-azure@1.10.2,nf-cloudcache@0.4.2,nf-codecommit@0.2.2,nf-console@1.1.4,nf-google@1.15.3,nf-tower@1.9.3,nf-wave@1.7.4
Jul-15 07:43:54.443 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-15 07:43:54.443 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-15 07:43:54.445 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-15 07:43:54.452 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-15 07:43:54.463 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/nextflow.config
Jul-15 07:43:54.464 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/nextflow.config
Jul-15 07:43:54.479 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jul-15 07:43:54.481 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@17ca8b92] - activable => nextflow.secret.LocalSecretsProvider@17ca8b92
Jul-15 07:43:54.484 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-15 07:43:54.738 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-15 07:43:54.752 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-15 07:43:54.763 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [boring_faggin] DSL2 - revision: b00b655f5b
Jul-15 07:43:54.764 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-15 07:43:54.764 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-15 07:43:54.796 [main] DEBUG nextflow.Session - Session UUID: 063177a9-1ffb-407e-a41d-e6a1184c5dd1
Jul-15 07:43:54.796 [main] DEBUG nextflow.Session - Run name: boring_faggin
Jul-15 07:43:54.797 [main] DEBUG nextflow.Session - Executor pool size: 32
Jul-15 07:43:54.803 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-15 07:43:54.807 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-15 07:43:54.825 [main] DEBUG nextflow.cli.CmdRun -
Version: 24.10.3 build 5933
Created: 16-12-2024 15:34 UTC (07:34 PDT)
System: Linux 6.8.0-58-generic
Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 3458331@bigbertha [127.0.1.1]
CPUs: 32 - Mem: 61.9 GB (4.3 GB) - Swap: 8 GB (128 KB)
Jul-15 07:43:54.842 [main] DEBUG nextflow.Session - Work-dir: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/work [ext2/ext3]
Jul-15 07:43:54.842 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/bin
Jul-15 07:43:54.848 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-15 07:43:54.853 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-15 07:43:54.872 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-15 07:43:54.877 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 33; maxThreads: 1000
Jul-15 07:43:55.002 [main] DEBUG nextflow.Session - Session start
Jul-15 07:43:55.003 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/execution_trace.txt
Jul-15 07:43:55.083 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-15 07:43:55.196 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-15 07:43:55.196 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-15 07:43:55.199 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-15 07:43:55.204 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=32; memory=61.9 GB; capacity=32; pollInterval=100ms; dumpInterval=5m
Jul-15 07:43:55.205 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-15 07:43:55.246 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-15 07:43:55.247 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-15 07:43:55.247 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-15 07:43:55.247 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_9d98cafae0c3b8e5: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/test.nf
Script_c8d2b32ed0aeeac9: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/./main.nf
Jul-15 07:43:55.247 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-15 07:43:55.247 [main] DEBUG nextflow.Session - Session await
Jul-15 07:43:55.334 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-15 07:43:55.336 [Task submitter] INFO nextflow.Session - [5b/95e3bb] Submitted process > METABOLITE_SCREEN
Jul-15 07:44:05.790 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 0; error: -; workDir: /data/bugrasWorkspace3/metabolite-screen-alt/nf_metabol_screen_adaptive/work/5b/95e3bb5e9a086140ca514eab04dd78]
Jul-15 07:44:05.791 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-15 07:44:06.031 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-15 07:44:06.090 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-15 07:44:06.090 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-15 07:44:06.091 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jul-15 07:44:06.094 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=2.8s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-15 07:44:06.094 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-15 07:44:06.095 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-15 07:44:06.349 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-15 07:44:06.431 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-15 07:44:06.436 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-15 07:44:06.436 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

80
nf_metabol_screen_adaptive/.nextflow.log.2 Normal file
View File

@@ -0,0 +1,80 @@
Jul-11 04:23:14.724 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf -bg -with-tower 'http://nucleus.omic.ai:8000/api'
Jul-11 04:23:14.779 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 24.10.3
Jul-11 04:23:14.789 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: nf-amazon@2.9.2,nf-azure@1.10.2,nf-cloudcache@0.4.2,nf-codecommit@0.2.2,nf-console@1.1.4,nf-google@1.15.3,nf-tower@1.9.3,nf-wave@1.7.4
Jul-11 04:23:14.808 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-11 04:23:14.808 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-11 04:23:14.810 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-11 04:23:14.820 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-11 04:23:14.831 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-11 04:23:14.832 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-11 04:23:14.849 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jul-11 04:23:14.851 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@26a94fa5] - activable => nextflow.secret.LocalSecretsProvider@26a94fa5
Jul-11 04:23:14.853 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-11 04:23:15.119 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-11 04:23:15.133 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-11 04:23:15.145 [main] INFO nextflow.cli.CmdRun - Launching `test.nf` [admiring_magritte] DSL2 - revision: 5f66a2b283
Jul-11 04:23:15.145 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-11 04:23:15.146 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-tower@1.9.3]
Jul-11 04:23:15.146 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.9.3
Jul-11 04:23:15.150 [main] INFO org.pf4j.AbstractPluginManager - Plugin 'nf-tower@1.9.3' resolved
Jul-11 04:23:15.150 [main] INFO org.pf4j.AbstractPluginManager - Start plugin 'nf-tower@1.9.3'
Jul-11 04:23:15.165 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-tower@1.9.3
Jul-11 04:23:15.195 [main] DEBUG nextflow.Session - Session UUID: 8abf60bb-5459-4d36-9886-d8f0064971ac
Jul-11 04:23:15.195 [main] DEBUG nextflow.Session - Run name: admiring_magritte
Jul-11 04:23:15.195 [main] DEBUG nextflow.Session - Executor pool size: 32
Jul-11 04:23:15.200 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-11 04:23:15.203 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-11 04:23:15.205 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-11 04:23:15.205 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-11 04:23:15.206 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-11 04:23:15.243 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-11 04:23:15.269 [main] DEBUG nextflow.cli.CmdRun -
Version: 24.10.3 build 5933
Created: 16-12-2024 15:34 UTC (07:34 PDT)
System: Linux 6.8.0-58-generic
Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 2298940@bigbertha [127.0.1.1]
CPUs: 32 - Mem: 61.9 GB (882.4 MB) - Swap: 8 GB (972 KB)
Jul-11 04:23:15.288 [main] DEBUG nextflow.Session - Work-dir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work [ext2/ext3]
Jul-11 04:23:15.288 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/bin
Jul-11 04:23:15.295 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-11 04:23:15.301 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-11 04:23:15.309 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Jul-11 04:23:15.438 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-11 04:23:15.444 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 33; maxThreads: 1000
Jul-11 04:23:15.651 [main] DEBUG nextflow.Session - Session start
Jul-11 04:23:15.653 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/execution_trace.txt
Jul-11 04:23:15.656 [main] DEBUG io.seqera.tower.plugin.TowerClient - Creating Seqera Platform observer -- endpoint=http://nucleus.omic.ai:8000/api; requestInterval=1s; aliveInterval=1m; maxRetries=5; backOffBase=3; backOffDelay=250
Jul-11 04:23:15.893 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-11 04:23:15.972 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-11 04:23:15.973 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-11 04:23:15.975 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-11 04:23:15.979 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=32; memory=61.9 GB; capacity=32; pollInterval=100ms; dumpInterval=5m
Jul-11 04:23:15.980 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-11 04:23:16.020 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-11 04:23:16.140 [main] INFO io.seqera.tower.plugin.TowerClient - Monitor the execution with Seqera Platform using this URL: http://localhost:8000/watch/J8JA6yEb
Jul-11 04:23:16.141 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-11 04:23:16.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-11 04:23:16.141 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_c8d2b32ed0aeeac9: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/./main.nf
Script_85e5efa6aea507c6: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/test.nf
Jul-11 04:23:16.141 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-11 04:23:16.141 [main] DEBUG nextflow.Session - Session await
Jul-11 04:23:16.243 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-11 04:23:16.245 [Task submitter] INFO nextflow.Session - [cf/a7b064] Submitted process > METABOLITE_SCREEN
Jul-11 04:23:34.062 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 0; error: -; workDir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work/cf/a7b0643787b68bed3189f63698ed53]
Jul-11 04:23:34.063 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-11 04:23:34.299 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-11 04:23:34.362 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-11 04:23:34.362 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-11 04:23:34.364 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jul-11 04:23:34.366 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=7.7s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-11 04:23:34.366 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-11 04:23:34.367 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-11 04:23:34.614 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-11 04:23:34.743 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-11 04:23:34.748 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin 'nf-tower@1.9.3'
Jul-11 04:23:34.748 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-tower
Jul-11 04:23:34.749 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-11 04:23:34.749 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

67
nf_metabol_screen_adaptive/.nextflow.log.3 Normal file
View File

@@ -0,0 +1,67 @@
Jul-10 05:08:04.417 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf
Jul-10 05:08:04.464 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 24.10.3
Jul-10 05:08:04.474 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: nf-amazon@2.9.2,nf-azure@1.10.2,nf-cloudcache@0.4.2,nf-codecommit@0.2.2,nf-console@1.1.4,nf-google@1.15.3,nf-tower@1.9.3,nf-wave@1.7.4
Jul-10 05:08:04.515 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-10 05:08:04.516 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-10 05:08:04.517 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-10 05:08:04.523 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-10 05:08:04.533 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-10 05:08:04.534 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-10 05:08:04.549 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jul-10 05:08:04.551 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@17ca8b92] - activable => nextflow.secret.LocalSecretsProvider@17ca8b92
Jul-10 05:08:04.556 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-10 05:08:04.812 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-10 05:08:04.827 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-10 05:08:04.834 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [drunk_swartz] DSL2 - revision: 8594c93ec6
Jul-10 05:08:04.834 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-10 05:08:04.835 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-10 05:08:04.859 [main] DEBUG nextflow.Session - Session UUID: 925b41d5-7b1c-4ed9-b34c-88420509e2e8
Jul-10 05:08:04.859 [main] DEBUG nextflow.Session - Run name: drunk_swartz
Jul-10 05:08:04.859 [main] DEBUG nextflow.Session - Executor pool size: 32
Jul-10 05:08:04.863 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-10 05:08:04.866 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-10 05:08:04.878 [main] DEBUG nextflow.cli.CmdRun -
Version: 24.10.3 build 5933
Created: 16-12-2024 15:34 UTC (07:34 PDT)
System: Linux 6.8.0-58-generic
Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 324567@bigbertha [127.0.1.1]
CPUs: 32 - Mem: 61.9 GB (4 GB) - Swap: 8 GB (5.6 MB)
Jul-10 05:08:04.889 [main] DEBUG nextflow.Session - Work-dir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work [ext2/ext3]
Jul-10 05:08:04.889 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/bin
Jul-10 05:08:04.894 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-10 05:08:04.899 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-10 05:08:04.921 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-10 05:08:04.926 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 33; maxThreads: 1000
Jul-10 05:08:05.053 [main] DEBUG nextflow.Session - Session start
Jul-10 05:08:05.054 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/execution_trace.txt
Jul-10 05:08:05.123 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-10 05:08:05.213 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-10 05:08:05.213 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-10 05:08:05.216 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-10 05:08:05.220 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=32; memory=61.9 GB; capacity=32; pollInterval=100ms; dumpInterval=5m
Jul-10 05:08:05.221 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-10 05:08:05.260 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-10 05:08:05.261 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-10 05:08:05.261 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-10 05:08:05.261 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_c8d2b32ed0aeeac9: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/./main.nf
Script_041bcf7aceb7d897: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/test.nf
Jul-10 05:08:05.261 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-10 05:08:05.261 [main] DEBUG nextflow.Session - Session await
Jul-10 05:08:05.359 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-10 05:08:05.360 [Task submitter] INFO nextflow.Session - [0a/8843c6] Submitted process > METABOLITE_SCREEN
Jul-10 05:08:37.851 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 0; error: -; workDir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work/0a/8843c652f011dbd021cd836c526d44]
Jul-10 05:08:37.852 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-10 05:08:38.827 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-10 05:08:38.852 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-10 05:08:38.852 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-10 05:08:38.853 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jul-10 05:08:38.857 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=13.6s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-10 05:08:38.857 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-10 05:08:38.858 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-10 05:08:39.681 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-10 05:08:39.893 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-10 05:08:39.897 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-10 05:08:39.897 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

67
nf_metabol_screen_adaptive/.nextflow.log.4 Normal file
View File

@@ -0,0 +1,67 @@
Jul-10 05:06:57.796 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf
Jul-10 05:06:57.858 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 24.10.3
Jul-10 05:06:57.870 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/root/.nextflow/plugins; core-plugins: nf-amazon@2.9.2,nf-azure@1.10.2,nf-cloudcache@0.4.2,nf-codecommit@0.2.2,nf-console@1.1.4,nf-google@1.15.3,nf-tower@1.9.3,nf-wave@1.7.4
Jul-10 05:06:57.889 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-10 05:06:57.890 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-10 05:06:57.891 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-10 05:06:57.899 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-10 05:06:57.909 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-10 05:06:57.911 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/nextflow.config
Jul-10 05:06:57.927 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /root/.nextflow/secrets/store.json
Jul-10 05:06:57.929 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@17ca8b92] - activable => nextflow.secret.LocalSecretsProvider@17ca8b92
Jul-10 05:06:57.931 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-10 05:06:58.207 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-10 05:06:58.222 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-10 05:06:58.232 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [nice_lagrange] DSL2 - revision: 8594c93ec6
Jul-10 05:06:58.233 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-10 05:06:58.233 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-10 05:06:58.266 [main] DEBUG nextflow.Session - Session UUID: 63ec0e24-1334-45e4-a396-bb1b7734bb1c
Jul-10 05:06:58.266 [main] DEBUG nextflow.Session - Run name: nice_lagrange
Jul-10 05:06:58.266 [main] DEBUG nextflow.Session - Executor pool size: 32
Jul-10 05:06:58.271 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-10 05:06:58.274 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-10 05:06:58.292 [main] DEBUG nextflow.cli.CmdRun -
Version: 24.10.3 build 5933
Created: 16-12-2024 15:34 UTC (07:34 PDT)
System: Linux 6.8.0-58-generic
Runtime: Groovy 4.0.23 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 318707@bigbertha [127.0.1.1]
CPUs: 32 - Mem: 61.9 GB (5.9 GB) - Swap: 8 GB (692 KB)
Jul-10 05:06:58.308 [main] DEBUG nextflow.Session - Work-dir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work [ext2/ext3]
Jul-10 05:06:58.309 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/bin
Jul-10 05:06:58.314 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-10 05:06:58.319 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-10 05:06:58.338 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-10 05:06:58.344 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 33; maxThreads: 1000
Jul-10 05:06:58.461 [main] DEBUG nextflow.Session - Session start
Jul-10 05:06:58.463 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/execution_trace.txt
Jul-10 05:06:58.544 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-10 05:06:58.620 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-10 05:06:58.620 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-10 05:06:58.623 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-10 05:06:58.627 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=32; memory=61.9 GB; capacity=32; pollInterval=100ms; dumpInterval=5m
Jul-10 05:06:58.628 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-10 05:06:58.667 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-10 05:06:58.668 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-10 05:06:58.668 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-10 05:06:58.668 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_c8d2b32ed0aeeac9: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/./main.nf
Script_041bcf7aceb7d897: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/test.nf
Jul-10 05:06:58.668 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-10 05:06:58.668 [main] DEBUG nextflow.Session - Session await
Jul-10 05:06:58.759 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-10 05:06:58.761 [Task submitter] INFO nextflow.Session - [b1/b3e41f] Submitted process > METABOLITE_SCREEN
Jul-10 05:07:34.617 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 0; error: -; workDir: /data/bugrasWorkspace3/metabolite-screen/nf_metabol_screen_adaptive/work/b1/b3e41f08bc233c93e7ac82702bb856]
Jul-10 05:07:34.617 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=96; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-10 05:07:35.660 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-10 05:07:35.716 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-10 05:07:35.716 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-10 05:07:35.718 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jul-10 05:07:35.721 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=15.9s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-10 05:07:35.721 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-10 05:07:35.722 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-10 05:07:35.973 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-10 05:07:36.099 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-10 05:07:36.104 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-10 05:07:36.104 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

69
nf_metabol_screen_adaptive/.nextflow.log.5 Normal file
View File

@@ -0,0 +1,69 @@
Jul-03 12:08:35.777 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf
Jul-03 12:08:35.914 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.3
Jul-03 12:08:35.944 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/omic/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,nf-azure@1.16.0,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.0,nf-k8s@1.0.0,nf-tower@1.11.3,nf-wave@1.12.1
Jul-03 12:08:35.978 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-03 12:08:35.979 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-03 12:08:35.983 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-03 12:08:35.996 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-03 12:08:36.024 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugra/metabolite-screen/nf_metabol_screen/nextflow.config
Jul-03 12:08:36.028 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugra/metabolite-screen/nf_metabol_screen/nextflow.config
Jul-03 12:08:36.073 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/omic/.nextflow/secrets/store.json
Jul-03 12:08:36.078 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@29314cc9] - activable => nextflow.secret.LocalSecretsProvider@29314cc9
Jul-03 12:08:36.086 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-03 12:08:36.792 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-03 12:08:36.833 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-03 12:08:36.850 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [insane_woese] DSL2 - revision: 063b5cd9a9
Jul-03 12:08:36.852 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-03 12:08:36.852 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-03 12:08:36.922 [main] DEBUG nextflow.Session - Session UUID: a62b6efd-60bb-4b97-a6e1-a68592f7d53f
Jul-03 12:08:36.923 [main] DEBUG nextflow.Session - Run name: insane_woese
Jul-03 12:08:36.923 [main] DEBUG nextflow.Session - Executor pool size: 80
Jul-03 12:08:36.935 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-03 12:08:36.943 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Jul-03 12:08:36.975 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.3 build 5949
Created: 02-06-2025 20:56 UTC
System: Linux 6.11.0-26-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 3786813@k8s-node23 [127.0.1.1]
CPUs: 80 - Mem: 251.6 GB (220.6 GB) - Swap: 0 (0)
Jul-03 12:08:37.006 [main] DEBUG nextflow.Session - Work-dir: /data/bugra/metabolite-screen/nf_metabol_screen/work [btrfs]
Jul-03 12:08:37.007 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugra/metabolite-screen/nf_metabol_screen/bin
Jul-03 12:08:37.022 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-03 12:08:37.034 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-03 12:08:37.060 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Jul-03 12:08:37.087 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-03 12:08:37.098 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 81; maxThreads: 1000
Jul-03 12:08:37.228 [main] DEBUG nextflow.Session - Session start
Jul-03 12:08:37.233 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugra/metabolite-screen/nf_metabol_screen/execution_trace.txt
Jul-03 12:08:37.394 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-03 12:08:37.590 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jul-03 12:08:37.591 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jul-03 12:08:37.598 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jul-03 12:08:37.609 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=80; memory=251.6 GB; capacity=80; pollInterval=100ms; dumpInterval=5m
Jul-03 12:08:37.612 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jul-03 12:08:37.646 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'METABOLITE_SCREEN': maxForks=0; fair=false; array=0
Jul-03 12:08:37.724 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: METABOLITE_SCREEN
Jul-03 12:08:37.726 [main] DEBUG nextflow.Session - Igniting dataflow network (1)
Jul-03 12:08:37.727 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > METABOLITE_SCREEN
Jul-03 12:08:37.728 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_181c5ac3efe09ab5: /data/bugra/metabolite-screen/nf_metabol_screen/main.nf
Script_3b80613f4955047b: /data/bugra/metabolite-screen/nf_metabol_screen/test.nf
Jul-03 12:08:37.728 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jul-03 12:08:37.728 [main] DEBUG nextflow.Session - Session await
Jul-03 12:08:38.010 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jul-03 12:08:38.016 [Task submitter] INFO nextflow.Session - [3d/10f68f] Submitted process > METABOLITE_SCREEN
Jul-03 12:08:44.784 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: METABOLITE_SCREEN; status: COMPLETED; exit: 0; error: -; workDir: /data/bugra/metabolite-screen/nf_metabol_screen/work/3d/10f68ffcfb09a7b35f3b90f43efbd9]
Jul-03 12:08:44.786 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Jul-03 12:08:45.543 [main] DEBUG nextflow.Session - Session await > all processes finished
Jul-03 12:08:45.581 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Jul-03 12:08:45.582 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jul-03 12:08:45.586 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Jul-03 12:08:45.596 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=3.7s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ]
Jul-03 12:08:45.596 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jul-03 12:08:45.598 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jul-03 12:08:46.431 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jul-03 12:08:46.525 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jul-03 12:08:46.538 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jul-03 12:08:46.538 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

78
nf_metabol_screen_adaptive/.nextflow.log.6 Normal file
View File

@@ -0,0 +1,78 @@
Jul-03 12:08:15.992 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -profile docker test.nf
Jul-03 12:08:16.116 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.3
Jul-03 12:08:16.165 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/omic/.nextflow/plugins; core-plugins: nf-amazon@2.15.0,nf-azure@1.16.0,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.0,nf-k8s@1.0.0,nf-tower@1.11.3,nf-wave@1.12.1
Jul-03 12:08:16.218 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-03 12:08:16.219 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-03 12:08:16.223 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Jul-03 12:08:16.239 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-03 12:08:16.269 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /data/bugra/metabolite-screen/nf_metabol_screen/nextflow.config
Jul-03 12:08:16.274 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /data/bugra/metabolite-screen/nf_metabol_screen/nextflow.config
Jul-03 12:08:16.319 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /home/omic/.nextflow/secrets/store.json
Jul-03 12:08:16.323 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@1ddd3478] - activable => nextflow.secret.LocalSecretsProvider@1ddd3478
Jul-03 12:08:16.339 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `docker`
Jul-03 12:08:17.003 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [k8s, docker]
Jul-03 12:08:17.044 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-03 12:08:17.064 [main] DEBUG nextflow.cli.CmdRun - Launching `test.nf` [insane_snyder] DSL2 - revision: 49b8a74e0c
Jul-03 12:08:17.066 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-03 12:08:17.066 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-03 12:08:17.150 [main] DEBUG nextflow.Session - Session UUID: 8cc53326-d69c-4269-9519-4eb256ccbbd2
Jul-03 12:08:17.151 [main] DEBUG nextflow.Session - Run name: insane_snyder
Jul-03 12:08:17.152 [main] DEBUG nextflow.Session - Executor pool size: 80
Jul-03 12:08:17.163 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-03 12:08:17.180 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=240; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Jul-03 12:08:17.322 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.04.3 build 5949
Created: 02-06-2025 20:56 UTC
System: Linux 6.11.0-26-generic
Runtime: Groovy 4.0.26 on OpenJDK 64-Bit Server VM 21.0.7+6-Ubuntu-0ubuntu124.04
Encoding: UTF-8 (UTF-8)
Process: 3786432@k8s-node23 [127.0.1.1]
CPUs: 80 - Mem: 251.6 GB (220.7 GB) - Swap: 0 (0)
Jul-03 12:08:17.386 [main] DEBUG nextflow.Session - Work-dir: /data/bugra/metabolite-screen/nf_metabol_screen/work [btrfs]
Jul-03 12:08:17.386 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /data/bugra/metabolite-screen/nf_metabol_screen/bin
Jul-03 12:08:17.401 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-03 12:08:17.413 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-03 12:08:17.440 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Jul-03 12:08:17.469 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-03 12:08:17.483 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 81; maxThreads: 1000
Jul-03 12:08:17.707 [main] DEBUG nextflow.Session - Session start
Jul-03 12:08:17.717 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /data/bugra/metabolite-screen/nf_metabol_screen/execution_trace.txt
Jul-03 12:08:17.950 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-03 12:08:18.079 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_e80a6c5924994093: /data/bugra/metabolite-screen/nf_metabol_screen/test.nf
Script_181c5ac3efe09ab5: /data/bugra/metabolite-screen/nf_metabol_screen/main.nf
Jul-03 12:08:18.081 [main] DEBUG nextflow.Session - Session aborted -- Cause: No such property: chunks for class: nextflow.script.WorkflowBinding
Jul-03 12:08:18.096 [main] ERROR nextflow.cli.Launcher - @unknown
groovy.lang.MissingPropertyException: No such property: chunks for class: nextflow.script.WorkflowBinding
at groovy.lang.Binding.getVariable(Binding.java:61)
at nextflow.script.WorkflowBinding.getVariable(WorkflowBinding.groovy:143)
at groovy.lang.Binding.getProperty(Binding.java:117)
at nextflow.script.WorkflowBinding.getProperty(WorkflowBinding.groovy:132)
at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:167)
at groovy.lang.Closure.getPropertyTryThese(Closure.java:325)
at groovy.lang.Closure.getPropertyDelegateFirst(Closure.java:315)
at groovy.lang.Closure.getProperty(Closure.java:301)
at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
at Script_e80a6c5924994093$_runScript_closure1$_closure2.doCall(Script_e80a6c5924994093:13)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
at groovy.lang.Closure.call(Closure.java:433)
at groovy.lang.Closure.call(Closure.java:412)
at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:205)
at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:189)
at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:51)
at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
at nextflow.script.BaseScript.run0(BaseScript.groovy:182)
at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
at nextflow.script.BaseScript.run(BaseScript.groovy:193)
at nextflow.script.parser.v1.ScriptLoaderV1.runScript(ScriptLoaderV1.groovy:246)
at nextflow.script.parser.v1.ScriptLoaderV1.runScript(ScriptLoaderV1.groovy)
at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:246)
at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:139)
at nextflow.cli.CmdRun.run(CmdRun.groovy:379)
at nextflow.cli.Launcher.run(Launcher.groovy:513)
at nextflow.cli.Launcher.main(Launcher.groovy:673)

1
nf_metabol_screen_adaptive/.nextflow.pid Normal file
View File

@@ -0,0 +1 @@
2298940

20
nf_metabol_screen_adaptive/Dockerfile Normal file
View File

@@ -0,0 +1,20 @@
FROM mambaorg/micromamba:2.1.1
USER root
COPY ./app /app
ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN micromamba install -n base -y -c conda-forge --channel-priority flexible \
python=3.12 \
numpy \
"pandas<3" \
scikit-learn \
parallelbar \
zarr=3.0.7 \
onnxruntime \
procps-ng && \
pip install \
/app/bitexpand-0.1.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl \
/app/mol_fingerprint-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl

BIN
nf_metabol_screen_adaptive/app/bitexpand-0.1.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Normal file
View File

Binary file not shown.

134
nf_metabol_screen_adaptive/app/convert.py Normal file
View File

@@ -0,0 +1,134 @@
#!/usr/bin/env python3
import argparse
import pandas as pd
import sys
from pathlib import Path
def create_smiles_file(drug_csv_path, metabolite_file_path, output_path="smiles.smi"):
"""
Create smiles.smi file by combining drug CSV and metabolite file.
Args:
drug_csv_path: Path to CSV file with 'SMILES' header
metabolite_file_path: Path to tab-separated file with SMILES and metabolite IDs
output_path: Output file path (default: smiles.smi)
"""
# Convert to Path objects for better handling
drug_csv_path = Path(drug_csv_path)
metabolite_file_path = Path(metabolite_file_path)
output_path = Path(output_path)
# Check if input files exist
if not drug_csv_path.exists():
raise FileNotFoundError(f"Drug CSV file not found: {drug_csv_path}")
if not metabolite_file_path.exists():
raise FileNotFoundError(f"Metabolite file not found: {metabolite_file_path}")
# Read drug CSV
try:
drug_df = pd.read_csv(drug_csv_path)
except Exception as e:
raise Exception(f"Error reading drug CSV file: {e}")
# Validate drug CSV has SMILES column
if 'SMILES' not in drug_df.columns:
raise ValueError("Drug CSV file must have 'SMILES' column")
# Check if metabolite file contains "NO_METABOLITES"
try:
with open(metabolite_file_path, 'r') as f:
first_line = f.readline().strip()
except Exception as e:
raise Exception(f"Error reading metabolite file: {e}")
# Handle no metabolites case
if first_line == "NO_METABOLITES":
print("No metabolites found, creating output file with drugs only")
# Create drug dataframe with drug_N format
drug_output = drug_df.copy()
drug_output['ID'] = [f'drug_{i}' for i in range(len(drug_df))]
drug_output = drug_output[['SMILES', 'ID']]
# Write only drugs to output file
try:
drug_output.to_csv(output_path, sep='\t', header=False, index=False)
except Exception as e:
raise Exception(f"Error writing output file: {e}")
print(f"Successfully created {output_path}")
print(f" - {len(drug_df)} drugs")
print(f" - 0 metabolites")
return
# Read metabolite file normally (tab-separated, no header)
try:
metabolite_df = pd.read_csv(metabolite_file_path, sep='\t', header=None, names=['SMILES', 'ID'])
except Exception as e:
raise Exception(f"Error reading metabolite file: {e}")
# Validate metabolite file has at least 2 columns
if metabolite_df.shape[1] < 2:
raise ValueError("Metabolite file must have at least 2 columns (SMILES and ID)")
# Create drug dataframe with drug_N format
drug_output = drug_df.copy()
drug_output['ID'] = [f'drug_{i}' for i in range(len(drug_df))]
drug_output = drug_output[['SMILES', 'ID']]
# Use metabolite dataframe as-is (already has SMILES and ID columns)
metabolite_output = metabolite_df[['SMILES', 'ID']]
# Combine drug and metabolite dataframes
combined_df = pd.concat([drug_output, metabolite_output], ignore_index=True)
# Write to output file (tab-separated, no header)
try:
combined_df.to_csv(output_path, sep='\t', header=False, index=False)
except Exception as e:
raise Exception(f"Error writing output file: {e}")
print(f"Successfully created {output_path}")
print(f" - {len(drug_df)} drugs")
print(f" - {len(metabolite_df)} metabolites")
def main():
parser = argparse.ArgumentParser(
description="Create smiles.smi file by combining drug CSV and metabolite file",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
python3 create_smiles.py drug.csv metabolites.txt
python3 create_smiles.py drug.csv metabolites.txt -o compounds.smi
"""
)
parser.add_argument(
'drug_csv',
help='Path to CSV file with SMILES column containing drug compounds'
)
parser.add_argument(
'metabolite_file',
help='Path to tab-separated file with SMILES and metabolite IDs'
)
parser.add_argument(
'-o', '--output',
default='smiles.smi',
help='Output file path (default: smiles.smi)'
)
args = parser.parse_args()
try:
create_smiles_file(args.drug_csv, args.metabolite_file, args.output)
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()

225
nf_metabol_screen_adaptive/app/get_round_2.py Executable file
View File

@@ -0,0 +1,225 @@
import argparse
import pandas as pd
import numpy as np
from collections import defaultdict
from itertools import chain
from pathlib import Path
import zarr
from zarr.storage import LocalStore
# Load reference data
seq_exist = pd.read_csv('/app/MANE_all_transcipts.csv')
def get_round_2(threshold, workdir, round, drug_csv: Path):
workdir = Path(workdir)
fasta_name = "patient_0"
drug_name = drug_csv.stem
name_out = f"{drug_name}_{fasta_name}"
# Load all zarr score stores produced by screen.py.
results_zarr = list(workdir.glob('*_results.zarr'))
if not results_zarr:
raise FileNotFoundError("No results Zarr stores found")
print("Loading Zarr results...")
# Lazy zarr handles per file; we read the (proteins, compounds) score matrix
# in protein chunks rather than loading it fully — keeps peak memory bounded
# regardless of compound count.
zarr_groups = []
compound_ids_per = []
protein_ids_per = []
for z_path in results_zarr:
store = LocalStore(z_path, read_only=True)
group = zarr.open(store, mode="r")
zarr_groups.append(group)
# numpy 2.4+ returns StringDType for these arrays; convert via tolist()
# because direct .astype(str) raises "cannot cast StringDType to StrDType".
compound_ids_per.append(np.array(group["compound_ids"][:].tolist(), dtype=str))
protein_ids_per.append(np.array(group["protein_ids"][:].tolist(), dtype=str))
# All zarrs share the same compound axis (drug + same metabolites).
all_compound_ids = compound_ids_per[0]
for cids in compound_ids_per[1:]:
if cids.shape != all_compound_ids.shape or not np.array_equal(cids, all_compound_ids):
raise ValueError("All *_results.zarr stores must share the same compound_ids axis")
all_protein_ids = np.concatenate(protein_ids_per) if len(protein_ids_per) > 1 else protein_ids_per[0]
n_proteins = sum(g["scores"].shape[0] for g in zarr_groups)
n_compounds = all_compound_ids.shape[0]
# ---- Compound-axis indices (tiny) ----
drug_mask_cmpd = np.array([s.startswith("drug") for s in all_compound_ids], dtype=bool)
if not drug_mask_cmpd.any():
raise ValueError("No 'drug_*' compounds present in compound_ids")
drug_cols_idx = np.where(drug_mask_cmpd)[0]
drug0_search = np.where(all_compound_ids == 'drug_0')[0]
if drug0_search.size == 0:
raise ValueError("'drug_0' not found in compound_ids")
drug0_idx = int(drug0_search[0])
# ---- Stream the score matrix in protein chunks, writing significant_interactions
# incrementally and accumulating per-protein summary stats. Avoids both the
# full score matrix load and the giant in-memory above-threshold DataFrame. ----
# Per chunk peak ≈ CHUNK_PROTEINS * n_compounds * 4 bytes; for 1000×38k ≈ 150 MB.
CHUNK_PROTEINS = 1000
drug_position = np.zeros(n_proteins, dtype=np.int64)
drug_0_col_full = np.empty(n_proteins, dtype=np.float32)
prot_count = defaultdict(int) # protein_id -> count of above-threshold compounds
prot_sum = defaultdict(float) # protein_id -> sum of above-threshold scores
# Open significant_interactions.tsv before the loop (round 2 only); write rows per chunk.
sig_path = workdir / f'{name_out}_significant_interactions.tsv'
sig_writer = None
smile_dict = None
if round == 2:
test_smiles = pd.read_csv(workdir / 'smiles.smi', sep='\t', header=None)
smile_dict = dict(zip(test_smiles[1], test_smiles[0]))
sig_writer = open(sig_path, 'w')
sig_writer.write("drug/metabolite\tsmile\ttranscipt\tconplex_score\n")
cursor = 0
for group in zarr_groups:
scores_arr = group["scores"]
n_this = scores_arr.shape[0]
for start in range(0, n_this, CHUNK_PROTEINS):
end = min(start + CHUNK_PROTEINS, n_this)
chunk = np.asarray(scores_arr[start:end]) # (chunk_size, n_compounds)
global_start = cursor + start
global_end = cursor + end
chunk_size = end - start
# drug score columns + max-per-protein for this chunk
chunk_max_drug = chunk[:, drug_cols_idx].max(axis=1)
# drug_position[i] = # compounds with score strictly > max_drug_score_i.
# Matches production's stable-sort + first-drug-position convention.
drug_position[global_start:global_end] = (chunk > chunk_max_drug[:, None]).sum(axis=1)
# drug_0 column
drug_0_col_full[global_start:global_end] = chunk[:, drug0_idx].astype(np.float32, copy=False)
# Above-threshold rows for this chunk
pi_local, ci = np.where(chunk > threshold)
if pi_local.size:
scores_vec = chunk[pi_local, ci].astype(np.float32, copy=False)
# Per-protein aggregates accumulated via bincount on local indices.
counts_local = np.bincount(pi_local, minlength=chunk_size)
sums_local = np.bincount(pi_local, weights=scores_vec.astype(np.float64), minlength=chunk_size)
for pi in np.flatnonzero(counts_local):
pid = all_protein_ids[global_start + pi]
prot_count[pid] += int(counts_local[pi])
prot_sum[pid] += float(sums_local[pi])
# Write significant_interactions rows for this chunk directly to the file.
if sig_writer is not None:
compound_ids_vec = all_compound_ids[ci]
protein_ids_vec = all_protein_ids[global_start + pi_local]
smiles_vec = np.array(
[smile_dict.get(c, "UNKNOWN") for c in compound_ids_vec]
)
pd.DataFrame({
"drug/metabolite": compound_ids_vec,
"smile": smiles_vec,
"transcipt": protein_ids_vec,
"conplex_score": scores_vec,
}).to_csv(sig_writer, sep='\t', index=False, header=False)
cursor += n_this
if sig_writer is not None:
sig_writer.close()
drug_0_score = pd.DataFrame({
"Drug": "drug_0",
"Transcript": all_protein_ids,
"Score": drug_0_col_full,
})
if round == 2:
drug_0_score.to_csv(workdir / f'{name_out}_drug_scores.tsv', sep='\t', index=False)
# ---- Per-protein summary (only proteins with any above-threshold compound) ----
drug_pos_map = dict(zip(all_protein_ids, drug_position))
if round == 1 and not prot_count:
# Equivalent to old "transcipts_summary.empty" early-out for round 1.
pd.DataFrame().to_csv(workdir / 'round_1.csv', index=False)
with open(workdir / 'round_1.fasta', 'w') as f:
f.write("all_data_is_filtered_out\n")
return 'STOP NO TRANSCIPTS ABOVE THRESHOLD'
if prot_count:
proteins_with_above = list(prot_count.keys())
# mean dtype kept as float32 to match the precision of the previous
# groupby-on-float32 path (CSV output is bit-identical that way).
means = np.array(
[prot_sum[p] / prot_count[p] for p in proteins_with_above],
dtype=np.float32,
)
summary = pd.DataFrame({
'transcipt_name': proteins_with_above,
'number_of_iteracting_compounds': [prot_count[p] for p in proteins_with_above],
'mean_binding_above_threshold': means,
})
else:
summary = pd.DataFrame(columns=[
'transcipt_name', 'number_of_iteracting_compounds', 'mean_binding_above_threshold'
])
summary['drug_position'] = summary['transcipt_name'].map(drug_pos_map)
path_out = workdir / f'round_{round}.csv'
if len(summary) == 0:
with path_out.open("w") as f:
f.write("NO TRANSCIPTS ABOVE THRESHOLD")
return 'STOP NO TRANSCIPTS ABOVE THRESHOLD'
summary['if_drug_above_threshold'] = (
summary['number_of_iteracting_compounds'] > summary['drug_position']
)
# MANE symbol lookup; strip mutated-protein "_2" suffix.
clean_transcript_names = [str(t).split('_')[0] for t in summary['transcipt_name']]
seq_lookup = seq_exist.set_index('transcipt')['symbol']
summary['protein_name'] = [
seq_lookup.get(t, "") for t in clean_transcript_names
]
summary.to_csv(path_out, index=False)
# ---- Round 1 also writes the round-2 FASTA ----
if round == 1:
name_2_filtered = list(summary['protein_name'])
transcipts_2 = list(chain.from_iterable(
list(seq_exist[seq_exist['symbol'] == sym]['transcipt'])
for sym in name_2_filtered
))
transcipts_2 = list(np.unique(transcipts_2))
existing = set(summary['transcipt_name'])
transcipts_2 = [t for t in transcipts_2 if t not in existing]
fasta_new = []
for t in transcipts_2:
row = seq_exist[seq_exist['transcipt'] == t]
if not row.empty:
fasta_new.append(f">{t}")
fasta_new.append(row['seq'].iloc[0])
with open(workdir / 'round_2.fasta', 'w') as f:
for line in fasta_new:
f.write(f"{line}\n")
print(f"Round {round} processing complete.")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Process and screen fragments.")
parser.add_argument("--threshold", required=True, type=float,
help="Threshold for 1. round of conplex scores")
parser.add_argument("--workdir", required=True, type=Path, help="workdir")
parser.add_argument("--round", required=True, type=int, help="Round 1 or 2")
parser.add_argument("--drug-csv", required=True, type=Path,
help="csv file with drug smiles")
args = parser.parse_args()
get_round_2(args.threshold, args.workdir, args.round, args.drug_csv)

143
nf_metabol_screen_adaptive/app/old_get_round_20.py Normal file
View File

@@ -0,0 +1,143 @@
import argparse
import pandas as pd
import numpy as np
from itertools import chain
from pathlib import PosixPath as Path
seq_exist = pd.read_csv('/app/MANE_all_transcipts.csv')
def get_round_2(threshold, workdir, round, drug_csv : Path):
workdir = Path(workdir)
# Get output names
# Get patient fasta name
# fasta_files = list(workdir.glob('*_variants_transcript_id_mutations.fasta'))
# if not fasta_files:
# raise FileNotFoundError("No variants transcript mutations fasta file found")
# fasta_name = fasta_files[0].stem.replace('_variants_transcript_id_mutations', '')
fasta_name = "patient_0"
# Get test drug
drug_name = drug_csv.stem
name_out = f"{drug_name}_{fasta_name}"
# Get all work dir complex files
results_files = list(workdir.glob('*_results.tsv'))
if not results_files:
raise FileNotFoundError("No results TSV files found")
round_1_score_list = [str(f) for f in results_files]
# Read 1st round
transcipts_1 = [
pd.read_csv(i, sep='\t', header=None).sort_values([2], ascending=False)
for i in round_1_score_list
]
# Get position of drug interaction vs all metabolites
drug_pos = [
[
n
for n, j in enumerate(list(i[0]))
if j[:4] == 'drug'
]
for i in transcipts_1
]
# Drug score
drug_0_score = pd.concat([i[i[0] =='drug_0'] for i in transcipts_1]).rename({0:'Drug', 1:'Transcript', 2:'Score'}, axis=1)
# Filter all below threshold
transcipts_1 = [i[i[2] > threshold] for i in transcipts_1]
# Save transcript_complex above threshold to one file
if round == 2:
inter_import = pd.concat(transcipts_1, ignore_index=True).rename({0:'drug/metabolite',1:'transcipt',2:'conplex_score'},axis=1)
# Add drug
test_smiles = pd.read_csv(workdir / 'smiles.smi', sep='\t', header=None)
smi_ = [test_smiles[test_smiles[1] == i].iloc[0][0] for i in list(inter_import['drug/metabolite'])]
inter_import['smile'] = smi_
inter_import[['drug/metabolite','smile','transcipt','conplex_score']].to_csv(
workdir / f'{name_out}_significant_interactions.tsv', sep='\t', index=False
)
drug_0_score.to_csv(workdir / f'{name_out}_drug_scores.tsv', sep='\t', index=False)
transcipts_1 = pd.DataFrame([
(Path(j).stem.replace('_results', ''), i.shape[0], i[2].mean(), k[0])
for i, j, k in zip(transcipts_1, round_1_score_list, drug_pos)
if i.shape[0] != 0
])
# Check if any interaction is above threshold for first round
if round == 1:
if transcipts_1.shape == (0,0):
transcipts_1.to_csv(workdir / 'round_1.csv', index=False)
with open(workdir / 'round_1.fasta', 'w') as f:
f.write("all_data_is_filtered_out\n")
return 'STOP NO TRANSCIPTS ABOVE THRESHOLD'
transcipts_1 = transcipts_1.rename({
0:'transcipt_name',
1:'number_of_iteracting_compounds',
2:'mean_binding_above_threshold',
3:'drug_position'
}, axis='columns')
path_out = workdir / f'round_{round}.csv'
if len(transcipts_1) == 0:
with path_out.open("w") as f:
f.write("NO TRANSCIPTS ABOVE THRESHOLD")
return 'STOP NO TRANSCIPTS ABOVE THRESHOLD'
transcipts_1['if_drug_above_threshold'] = transcipts_1.iloc[:,1] > transcipts_1.iloc[:,3]
# If protein is mutated it has _2 in name, removes it
transcipt_names = [i.split('_')[0] for i in list(transcipts_1['transcipt_name'])]
temp = [
seq_exist[seq_exist['transcipt'] == i]
for i in transcipt_names
]
transcipts_1['protein_name'] =[
i.iloc[0]['symbol'] if len(i) > 0 else ""
for i in temp
]
# Save data on first round
transcipts_1.to_csv(path_out, index=False)
# Get fasta for second round
if round == 1:
# Get all transcripts of proteins above threshold
name_2_filttered = list(transcipts_1['protein_name'])
transcipts_2 = [list(seq_exist[seq_exist['symbol'] == i]['transcipt']) for i in name_2_filttered]
transcipts_2 = list(chain(*transcipts_2))
transcipts_2 = list(np.unique((transcipts_2)))
# Filter out transcripts already ran through complex
transcipts_2 = list(np.array(transcipts_2)[
[i not in list(transcipts_1['transcipt_name']) for i in transcipts_2]
])
fasta_new = [
['>'+i, seq_exist[seq_exist['transcipt'] == i]['seq'].iloc[0]]
for i in transcipts_2
]
fasta_new = list(chain(*fasta_new))
# Write fasta to run
with open(workdir / 'round_2.fasta', 'w') as f:
for line in fasta_new:
f.write(f"{line}\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Process and screen fragments.")
parser.add_argument("--threshold", required=True, type=float, help="Threshold for 1. round.of conplex scores")
parser.add_argument("--workdir", required=True, type=Path, help="workdir")
parser.add_argument("--round", required=True, type=int, help="Round 1 or 2")
parser.add_argument("--drug-csv", required=True, type=Path, help="csv file with drug smiles")
args = parser.parse_args()
get_round_2(args.threshold, args.workdir, args.round, args.drug_csv)

127
nf_metabol_screen_adaptive/app/old_screen.py Normal file
View File

@@ -0,0 +1,127 @@
# Custom Rust modules
import mol_fingerprint as mf
import bitexpand
import pandas as pd
import numpy as np
from pathlib import Path
from argparse import ArgumentParser
import zarr
from zarr.storage import LocalStore
from sklearn.preprocessing import normalize
from parallelbar import progress_map
import onnxruntime as ort
print('CUDA available:', 'CUDAExecutionProvider' in ort.get_available_providers())
def parse_args():
parser = ArgumentParser()
parser.add_argument("-i","--input-tsv", type=Path, required=True, help="Input TSV file with molecules to screen. It must not have any labels and has only two columns, 1st molecule_label, 2nd molecule_smiles")
parser.add_argument("-z","--zarr-protein", type=Path, required=True, help="Input Zarr DB that holds the protein ids and their normalized projection vectors")
return parser
parser = parse_args()
args = parser.parse_args()
# path_fp_basedir : Path = args.input_dir
# my_id :int = args.chunk_id
path_metabolites : Path = args.input_tsv
if not path_metabolites.is_file():
raise ValueError(f"{path_metabolites} IS MISSING OR NOT A FILE")
df_metabolite = pd.read_csv(
path_metabolites,
sep="\t",
header=None,
names=["smiles" ,"compound_id"]
)
path_zarr = args.zarr_protein
store_zarr = LocalStore(path_zarr , read_only=True)
group_zarr = zarr.open(store_zarr, mode="r")
protein_vecs = (group_zarr["normalized"][:])
path_onnx = Path("/app/drug_projector.onnx")
def load_model(model_path):
"""Load ONNX model with GPU support"""
model_path = Path(model_path)
# Set up providers for GPU execution
# providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
providers = ['CPUExecutionProvider']
# Create inference session
session = ort.InferenceSession(str(model_path), providers=providers)
# Verify GPU is being used
print(f"Available providers: {session.get_providers()}")
return session
session = load_model(path_onnx)
def run_inference(input_vector):
"""Run inference on the model"""
# Get input/output names
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
# Ensure input is the correct shape and type
# if input_vector.shape != (2048,):
# raise ValueError(f"Input shape should be (2048,), got {input_vector.shape}")
# Add batch dimension if needed
input_data = input_vector
# Run inference
outputs = session.run([output_name], {input_name: input_data})
# Return the output vector (remove batch dimension)
return outputs[0]
def normalize_vectors_sklearn(vectors):
"""Normalize using sklearn - often fastest for large arrays"""
return normalize(vectors, norm='l2', axis=1)
def project_drugs(smiles : list[str]) -> np.ndarray :
fps = bitexpand.expand_bits(np.array([
np.frombuffer(fp,dtype=np.uint8)
for fp in mf.generate_fingerprints(smiles)
]))
return normalize_vectors_sklearn(run_inference(fps))
pvecs = project_drugs(df_metabolite["smiles"].values.tolist())
my_results = np.dot(protein_vecs,pvecs.T)
pids = group_zarr["ids"][:]
path_outdir = Path("./")
def write_protein(i : int):
df_out = df_metabolite[["compound_id"]].copy()
df_out["protein_id"] = pids[i]
df_out["score"] = my_results[i]
# df_out["protein_sequence"] = group_zarr["sequences"][i]
path_out = path_outdir / f"{pids[i]}_results.tsv"
df_out.to_csv(
path_out,
sep="\t",
index=False,
header=False
)
tasks = list(range(my_results.shape[0]))
progress_map(
write_protein, tasks
)
print("PROCESS COMPLETED SUCCESSFULLY")

144
nf_metabol_screen_adaptive/app/screen.py Normal file
View File

@@ -0,0 +1,144 @@
# Custom Rust modules
import mol_fingerprint as mf
import bitexpand
import pandas as pd
import numpy as np
from pathlib import Path
from argparse import ArgumentParser
import zarr
from zarr.storage import LocalStore
from zarr.codecs import ZstdCodec
from sklearn.preprocessing import normalize
from parallelbar import progress_map
import onnxruntime as ort
print('CUDA available:', 'CUDAExecutionProvider' in ort.get_available_providers())
def parse_args():
parser = ArgumentParser()
parser.add_argument("-i","--input-tsv", type=Path, required=True, help="Input TSV file with molecules to screen. It must not have any labels and has only two columns, 1st molecule_label, 2nd molecule_smiles")
parser.add_argument("-z","--zarr-protein", type=Path, required=True, help="Input Zarr DB that holds the protein ids and their normalized projection vectors")
return parser
parser = parse_args()
args = parser.parse_args()
path_metabolites : Path = args.input_tsv
if not path_metabolites.is_file():
raise ValueError(f"{path_metabolites} IS MISSING OR NOT A FILE")
df_metabolite = pd.read_csv(
path_metabolites,
sep="\t",
header=None,
names=["smiles" ,"compound_id"]
)
path_zarr = args.zarr_protein
store_zarr = LocalStore(path_zarr, read_only=True)
group_zarr = zarr.open(store_zarr, mode="r")
protein_vecs = (group_zarr["normalized"][:])
path_onnx = Path("/app/drug_projector.onnx")
def load_model(model_path):
"""Load ONNX model with GPU support"""
model_path = Path(model_path)
# Set up providers for GPU execution
# providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
providers = ['CPUExecutionProvider']
# Create inference session
session = ort.InferenceSession(str(model_path), providers=providers)
# Verify GPU is being used
print(f"Available providers: {session.get_providers()}")
return session
session = load_model(path_onnx)
def run_inference(input_vector):
"""Run inference on the model"""
# Get input/output names
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
# Run inference
outputs = session.run([output_name], {input_name: input_vector})
return outputs[0]
def normalize_vectors_sklearn(vectors):
"""Normalize using sklearn - often fastest for large arrays"""
return normalize(vectors, norm='l2', axis=1)
def project_drugs(smiles : list[str]) -> np.ndarray :
fps = bitexpand.expand_bits(np.array([
np.frombuffer(fp,dtype=np.uint8)
for fp in mf.generate_fingerprints(smiles)
]))
return normalize_vectors_sklearn(run_inference(fps))
pvecs = project_drugs(df_metabolite["smiles"].values.tolist())
my_results = np.dot(protein_vecs, pvecs.T)
pids = group_zarr["ids"][:]
# Create output zarr store
path_outdir = Path("./")
zarr_name = path_zarr.stem + "_results.zarr"
path_output_zarr = path_outdir / zarr_name
# Create zarr store
store_output = LocalStore(path_output_zarr, read_only=False)
group_output = zarr.open(store_output, mode="w")
out_compress =[
ZstdCodec( level=13 )
]
# Store the data
# Similarity matrix: (n_proteins, n_compounds)
z1 = group_output.create_array(
"scores",
dtype="float32",
shape=my_results.shape,
chunks=(min(1000, my_results.shape[0]), my_results.shape[1]),
compressors=out_compress
)
z1[:] = my_results
# Compound IDs
z2 = group_output.create_array(
"compound_ids",
dtype="str",
shape=(len(df_metabolite),),
compressors=out_compress
)
z2[:] = df_metabolite["compound_id"].values
# Protein IDs
z3 = group_output.create_array(
"protein_ids",
dtype="str",
shape=(len(pids),),
compressors=out_compress
)
z3[:] = pids
# Store metadata
group_output.attrs['n_proteins'] = my_results.shape[0]
group_output.attrs['n_compounds'] = my_results.shape[1]
group_output.attrs['source_zarr'] = str(path_zarr)
print(f"Results saved to {path_output_zarr}")
print(f"Shape: {my_results.shape[0]} proteins x {my_results.shape[1]} compounds")
print("PROCESS COMPLETED SUCCESSFULLY")

14
nf_metabol_screen_adaptive/docker-compose.yml Normal file
View File

@@ -0,0 +1,14 @@
version: '3.8'
services:
metabolite-screen:
build:
context: .
dockerfile: Dockerfile
image: harbor.cluster.omic.ai/omic/metabolite-screen:adaptive-1.1.1
command: bash
stdin_open: true
tty: true
volumes:
- /mnt/dreamdock-data/digital_trials/workdir/52/6cdef3797446aec540843481dae1ca:/workdir
- /mnt/dreamdock-data/digital_trials:/mnt/dreamdock-data/digital_trials

223
nf_metabol_screen_adaptive/execution_timeline.html Normal file
View File

File diff suppressed because one or more lines are too long

2
nf_metabol_screen_adaptive/execution_trace.txt Normal file
View File

@@ -0,0 +1,2 @@
task_id hash native_id name status exit submit duration realtime %cpu peak_rss peak_vmem rchar wchar
1 f9/21c381 3902375 METABOLITE_SCREEN FAILED 127 2025-07-21 03:07:11.554 35ms 18ms - - - - -

24
nf_metabol_screen_adaptive/main.nf Normal file
View File

@@ -0,0 +1,24 @@
#!/usr/bin/env nextflow
process METABOLITE_SCREEN {
memory '10 GB'
container "harbor.cluster.omic.ai/omic/metabolite-screen:adaptive"
// containerOptions "--rm --gpus all"
// errorStrategy 'ignore'
input:
path input_tsv // TSV file with the smiles
path input_zarr // ZARR DB holding all the protein vectors and their names
output:
path "*_result.tsv"
script:
"""
set -e
python /app/screen.py -i ${input_tsv} -z ${input_zarr}
"""
}

113
nf_metabol_screen_adaptive/nextflow.config Normal file
View File

@@ -0,0 +1,113 @@
// Nextflow configuration file
// Execution report
report {
enabled = true
file = "execution_report.html"
overwrite = true}
// Resource usage tracking
trace {
enabled = true
file = "execution_trace.txt"
overwrite = true
}
// Error reporting
timeline {
enabled = true
file = "execution_timeline.html"
overwrite = true
}
// // Max resources
// executor {
// $local {
// memory = '80 GB'
// cpus = 2
// }
// }
// Default parameters (overridden by test.nf)
params {
// Set defaults here
// input = '/mnt/ZINC-22/conplex/total.zarr'
nshards = 12
outdir = "$launchDir"
}
profiles {
k8s {
process.executor = 'k8s'
process.namespace = 'bioinformatics'
process.debug = true
workDir = "/workdir/work"
k8s {
serviceAccount = 'nextflow-sa'
namespace = 'bioinformatics'
storageClaimName = 'conplex-data'
storageMountPath = '/workdir'
pullPolicy = 'IfNotPresent'
cleanup = true // delete pods after Ctrl+C or finished?
// RUN AS DIFFERENT USERS
// securityContext = [fsGroup: 1000]
// securityContext = [
// runAsUser: 1000,
// fsGroup: 1000,
// runAsNonRoot: true
// ]
}
// Use container image
// process.container = 'harbor.cluster.omic.ai/omic/faiss-indexer:latest'
// Pod-level customization
process.pod = [
[env: 'NXF_DEBUG', value: '0'],
[label: 'omic-app', value: 'conplex'],
[imagePullSecret: 'gitlab-registry-secret'],
[volumeClaim: 'fingerprint-data', mountPath: '/fingerprint-data']
//[nodeSelector: [gpu: 'yes', 'kubernetes.io/hostname': 'k8s-node25']],
// [privileged: true],
]
// process {
// withName: 'trainIndex' {
// container = 'harbor.cluster.omic.ai/omic/conplex-faiss-train:latest'
// accelerator = 1
// memory = '75 GB'
// cpus = 1
// // GPU-specific pod settings
// pod = [
// [nodeSelector: [gpu: 'yes']]
// ]
// }
// // Specific settings for addToIndex process (CPU only)
// withName: 'addToIndex' {
// container = 'harbor.cluster.omic.ai/omic/conplex-faiss-add:latest'
// memory = '75 GB'
// cpus = 1
// // Explicitly ensure no GPU requirements
// // No accelerator property set, and no GPU nodeSelector
// }
// }
}
docker {
// Docker/Singularity configuration
docker {
enabled = true
runOptions = '--rm'
}
}
}

17
nf_metabol_screen_adaptive/test.nf Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env nextflow
// params.input_tsv = '../sample/in-molecule/test_known_drugs.tsv' //metabolites.tsv'
params.input_tsv = '../sample/in-molecule/metabolites-rev.tsv' //metabolites.tsv'
params.input_zarr = '../data/protein_seq.zarr' //mane_all.zarr'
// Include the buildFaissIndex workflow from main.nf
include { METABOLITE_SCREEN } from './main.nf'
workflow {
input_tsv = file(params.input_tsv)
input_zarr = file(params.input_zarr)
// Call the combined workflow with both inputs
protein_scores = METABOLITE_SCREEN(input_tsv, input_zarr)
}