Fix NodeJS retrieval when Docker is unavailable#1054
Merged
Conversation
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
640c201 to
bec7ab6
Compare
kinow
reviewed
May 5, 2026
Contributor
There was a problem hiding this comment.
I accidentally used a micromamba env with nodejs first, and thought it worked. I couldn't find any .sif files like I did with cwltool which was odd.
So I was cleaning up the files and reviewing what I did when I saw that and uninstalled NodeJS. I tried this again, and the .sif file for NodeJS was actually downloaded, but one test failed.
========================================================================================== FAILURES ==========================================================================================
_____________________________________________________________________________ cwl test: step_input_default_value _____________________________________________________________________________
[gw6] linux -- Python 3.14.4 /gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/bin/python
CWL test execution failed.
Returned non-zero but it should be zero
Test: job:
file:///gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
output:
count_output: 16
tool:
file:///gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl
label: step_input_default_value
id: 49
doc: Test default value on step input parameter
tags:
- inline_javascript
- workflow
line: '493'
------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------
2026-05-05 09:09:38.658 INFO Processing workflow 14c86613-091a-4050-94e9-8faaf8bea15a
2026-05-05 09:09:38.658 INFO Building workflow execution plan
2026-05-05 09:09:38.679 INFO COMPLETED building of workflow execution plan
2026-05-05 09:09:38.679 INFO EXECUTING workflow 14c86613-091a-4050-94e9-8faaf8bea15a
2026-05-05 09:09:38.737 INFO COPYING from /gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /scratch/tmp/streamflow/477fa312-1387-4adf-918c-ac5741864669/657250c5-4351-4ad2-a901-70e6e9a65618/whale.txt on local file-system
2026-05-05 09:09:38.738 INFO COMPLETED copy from /gpfs/home/bsc/<USER>/cwl/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /scratch/tmp/streamflow/477fa312-1387-4adf-918c-ac5741864669/657250c5-4351-4ad2-a901-70e6e9a65618/whale.txt on local file-system
2026-05-05 09:09:38.740 INFO EXECUTING step /step1 (job /step1/0) locally into directory /scratch/tmp/streamflow/81c928e1-95a2-4000-b6ca-61f912dfa918:
wc -l
2026-05-05 09:09:38.760 INFO COMPLETED Step /step1
2026-05-05 09:09:38.763 INFO Evaluating expression for step /step2 (job /step2/0)
FATAL: Image file already exists: "node_alpine.sif" - will not overwrite
2026-05-05 09:09:38.793 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
Traceback (most recent call last):
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/core/recovery.py", line 42, in wrapper
await func(*args, **kwargs)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/workflow/step.py", line 736, in _execute_command
command_output := await command_task
^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/command.py", line 1304, in execute
result = utils.eval_expression(
expression=self.expression,
...<2 lines>...
expression_lib=self.expression_lib,
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/utils.py", line 742, in eval_expression
cwl_utils.expression.interpolate(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
expression,
^^^^^^^^^^^
...<9 lines>...
container_engine=_get_container_engine(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/expression.py", line 227, in interpolate
e = evaluator(
js_engine, scan[w[0] + 1 : w[1]], rootvars, jslib, fullJS, **kwargs
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/expression.py", line 173, in evaluator
return cast(CWLOutputType, js_engine.eval(ex, jslib, **kwargs))
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 473, in eval
returncode, stdout, stderr = self.exec_js_process(
~~~~~~~~~~~~~~~~~~~~^
fn,
^^^
...<3 lines>...
container_engine=container_engine,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 201, in exec_js_process
new_proc = self.new_js_proc(
js_engine_code,
force_docker_pull=force_docker_pull,
container_engine=container_engine,
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/cwl_utils/sandboxjs.py", line 442, in new_js_proc
raise JavascriptException(
...<5 lines>...
)
cwl_utils.errors.JavascriptException: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
2026-05-05 09:09:38.811 WARNING Job /step2/0 failure can not be recovered. Failure manager is not enabled.
2026-05-05 09:09:38.812 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, singularity run node:alpine
2026-05-05 09:09:38.814 INFO FAILED Step /step2
2026-05-05 09:09:38.816 ERROR FAILED Workflow execution
Traceback (most recent call last):
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/runner.py", line 112, in main
asyncio.run(_async_main(parsed_args))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/runners.py", line 204, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/runners.py", line 127, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/gpfs/home/bsc/<USER>/.local/share/uv/python/cpython-3.14.4-linux-x86_64-gnu/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/runner.py", line 93, in _async_main
await streamflow.cwl.main.main(
workflow_config=workflow_config, context=context, args=args
)
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/cwl/main.py", line 93, in main
output_tokens = await executor.run()
^^^^^^^^^^^^^^^^^^^^
File "/gpfs/home/bsc/<USER>/cwl/streamflow/streamflow/workflow/executor.py", line 175, in run
raise WorkflowExecutionException("FAILED Workflow execution")
streamflow.core.exception.WorkflowExecutionException: FAILED Workflow execution
Test failed unexpectedly: common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
Test default value on step input parameter
------------------------------------------------------------------------------------- Captured log call --------------------------------------------------------------------------------------
WARNING cwltest:plugin.py:82 Test failed unexpectedly: common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/count-lines9-wf.cwl common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/empty.json
WARNING cwltest:plugin.py:83 Test default value on step input parameter
----------------------------------------------------------- generated xml file: /gpfs/home/bsc/<USER>/cwl/streamflow/junit.xml ------------------------------------------------------------
================================================================================== short test summary info ===================================================================================
SKIPPED [1] common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/conformance_test_v1.0.cwltest.yaml: Test 'docker_entrypoint' is in the exclude list.
========================================================================= 1 failed, 195 passed, 1 skipped in 16.56s ==========================================================================Retrying that, it worked, and all tests passed, without the need to install or load NodeJS modules.
Thanks @GlassOfWhiskey
Member
Author
|
@kinow I found the time to investigate the problem. This is due to the fact that |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1054 +/- ##
==========================================
- Coverage 64.16% 64.16% -0.01%
==========================================
Files 93 93
Lines 17553 17561 +8
Branches 2158 2160 +2
==========================================
+ Hits 11263 11268 +5
- Misses 6009 6011 +2
- Partials 281 282 +1 ☔ View full report in Codecov by Harness. |
LanderOtto
approved these changes
May 18, 2026
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
bec7ab6 to
30203cd
Compare
kinow
reviewed
May 30, 2026
kinow
left a comment
kinow
left a comment
Contributor
There was a problem hiding this comment.
Hmmm, I'm really not able to test this on MN5. It keeps insisting that NodeJS is not found, but it doesn't try to use Singularity. See my comment in the cwl-utils PR (which is related to this too): common-workflow-language/cwl-utils#424 (review)
Member
Author
Unfortunately I don't have access to Mare Nostrum to run tests. |
Contributor
|
@GlassOfWhiskey it might be due to an issue @adamnovak found cwl-utils -- DataBiosphere/toil#5529 (comment) |
kinow
reviewed
Jun 7, 2026
Contributor
There was a problem hiding this comment.
@GlassOfWhiskey it works!
Previously, I tried testing this with the testing suite, but it was hard to isolate and replicate the test with & without the patch (because of how cwl-conformance-tests.shcreates the venvs/etc.).
So I first removed the rm -rf at the end of cwl-conformance-tests.sh script, ran the conformance tests, and got one of the tests that produced the NodeJS exception.
version: v1.0
workflows:
workflow_name:
type: cwl
config:
file: "/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/io-any-wf-1.cwl"
settings: "/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/io-any-record.json"
docker:
- step: /
deployment:
type: singularity
config: {}
Running this with 0.2.0rc2, I consistently get the NodeJS exception. But switching to this branch, and pip install -e . again:
(python-3.13) (venv) [<USER>@glogin4 streamflow]$ streamflow run --debug test.yml
2026-06-07 15:49:40.345 INFO Processing workflow 6e4724b7-cc3e-45ee-bbc5-c34f5f37dd74
2026-06-07 15:49:40.345 INFO Building workflow execution plan
2026-06-07 15:49:40.352 DEBUG Translating Workflow /
2026-06-07 15:49:40.352 DEBUG Translating WorkflowStep /step1
2026-06-07 15:49:40.355 DEBUG Translating CommandLineTool /step1
2026-06-07 15:49:40.361 INFO COMPLETED building of workflow execution plan
2026-06-07 15:49:40.361 INFO EXECUTING workflow 6e4724b7-cc3e-45ee-bbc5-c34f5f37dd74
2026-06-07 15:49:40.364 DEBUG Step /bar-injector received inputs ['0']
2026-06-07 15:49:40.364 DEBUG Retrieving available locations for job /bar-injector/0 on __LOCAL__.
2026-06-07 15:49:40.364 DEBUG Available locations for job /bar-injector/0 on __LOCAL__ are ['__LOCAL__'].
2026-06-07 15:49:40.364 DEBUG Job /bar-injector/0 allocated locally
2026-06-07 15:49:40.365 DEBUG COMPLETED Step __deploy__/__LOCAL__
2026-06-07 15:49:40.365 DEBUG Job /bar-injector/0 changed status to RUNNING
2026-06-07 15:49:40.365 DEBUG COMPLETED Step /bar-injector/__schedule__
2026-06-07 15:49:40.365 DEBUG Job /bar-injector/0 changed status to COMPLETED
2026-06-07 15:49:40.366 DEBUG Step /bar-token-transformer received inputs ['0']
2026-06-07 15:49:40.366 DEBUG Step /bar-injector received termination token with Status COMPLETED
2026-06-07 15:49:40.366 DEBUG Step /step1/bar-token-transformer received inputs ['0']
2026-06-07 15:49:40.366 DEBUG Step /bar-token-transformer received termination token with Status COMPLETED
2026-06-07 15:49:40.366 DEBUG COMPLETED Step /bar-injector
2026-06-07 15:49:40.366 DEBUG COMPLETED Step /bar-token-transformer
2026-06-07 15:49:40.366 DEBUG Step /step1/__transfer__/bar received inputs ['0']
2026-06-07 15:49:40.366 DEBUG Step /step1/__schedule__ received inputs ['0']
2026-06-07 15:49:40.366 DEBUG Step /step1/bar-token-transformer received termination token with Status COMPLETED
2026-06-07 15:49:40.367 DEBUG Retrieving available locations for job /step1/0 on __LOCAL__.
2026-06-07 15:49:40.367 DEBUG Available locations for job /step1/0 on __LOCAL__ are ['__LOCAL__'].
2026-06-07 15:49:40.367 DEBUG Job /step1/0 allocated locally
2026-06-07 15:49:40.367 DEBUG COMPLETED Step /step1/bar-token-transformer
2026-06-07 15:49:40.368 DEBUG Step /step1/__schedule__ received termination token with Status COMPLETED
2026-06-07 15:49:40.368 DEBUG COMPLETED Step /step1/__schedule__
2026-06-07 15:49:40.368 DEBUG Step /step1 received inputs ['0']
2026-06-07 15:49:40.368 DEBUG Step /step1/__transfer__/bar received termination token with Status COMPLETED
2026-06-07 15:49:40.368 DEBUG COMPLETED Step /step1/__transfer__/bar
2026-06-07 15:49:40.369 DEBUG Job /step1/0 started
2026-06-07 15:49:40.369 DEBUG Job /step1/0 changed status to RUNNING
2026-06-07 15:49:40.369 DEBUG Job /step1/0 inputs: {
"bar": {
"cow": 5,
"moo": 1
}
}
2026-06-07 15:49:40.369 INFO EXECUTING step /step1 (job /step1/0) locally into directory /scratch/tmp/streamflow/9c1948b6-d778-43d6-a64a-5683a9fed031:
true
2026-06-07 15:49:40.369 DEBUG Step /step1 received termination token with Status COMPLETED
2026-06-07 15:49:40.369 DEBUG EXECUTING command cd /scratch/tmp/streamflow/9c1948b6-d778-43d6-a64a-5683a9fed031 && export HOME="/scratch/tmp/streamflow/9c1948b6-d778-43d6-a64a-5683a9fed031" && export TMPDIR="/scratch/tmp/streamflow/5c71d144-ea0f-4119-a86f-88b8edb9d3e8" && true 2>&1 on __LOCAL__/__LOCAL__ for job /step1/0
2026-06-07 15:49:41.128 DEBUG Job /step1/0 changed status to COMPLETED
2026-06-07 15:49:41.128 DEBUG Step /t1-collector-transformer received inputs ['0']
2026-06-07 15:49:41.129 DEBUG COMPLETED Job /step1/0 terminated
2026-06-07 15:49:41.129 INFO COMPLETED Step /step1
2026-06-07 15:49:41.129 DEBUG Step /t1-collector received inputs ['0']
2026-06-07 15:49:41.129 DEBUG Step /t1-collector/__schedule__ received inputs ['0']
2026-06-07 15:49:41.129 DEBUG Step /t1-collector-transformer received termination token with Status COMPLETED
2026-06-07 15:49:41.129 DEBUG Retrieving available locations for job /t1-collector/0 on __LOCAL__.
2026-06-07 15:49:41.130 DEBUG Available locations for job /t1-collector/0 on __LOCAL__ are ['__LOCAL__'].
2026-06-07 15:49:41.130 DEBUG Job /t1-collector/0 allocated locally
2026-06-07 15:49:41.130 DEBUG COMPLETED Step /t1-collector-transformer
2026-06-07 15:49:41.131 DEBUG Step /t1-collector/__schedule__ received termination token with Status COMPLETED
2026-06-07 15:49:41.131 DEBUG COMPLETED Step /t1-collector/__schedule__
2026-06-07 15:49:41.131 DEBUG Step /t1-collector received termination token with Status COMPLETED
2026-06-07 15:49:41.132 DEBUG COMPLETED Step /t1-collector
2026-06-07 15:49:41.132 INFO COMPLETED workflow execution
{
"t1": {
"cow": 5,
"moo": 1
}
}
(python-3.13) (venv) [<USER>@glogin4 streamflow]$ echo $?
0
Switching back to git checkout 0.2.0rc2 , then pip install -e . and re-running the test.yml file, I can confirm the error comes back. So this PR really fixes it 🙏 🎉 🚀
$ streamflow run --debug test.yml 2>&1 | grep -B2 -A2 NodeJS
2026-06-07 15:53:28.428 DEBUG Step /step1 received termination token with Status COMPLETED
2026-06-07 15:53:28.428 DEBUG EXECUTING command cd /scratch/tmp/streamflow/d9a50748-ba11-491f-8c05-8a0f5044d50c && export HOME="/scratch/tmp/streamflow/d9a50748-ba11-491f-8c05-8a0f5044d50c" && export TMPDIR="/scratch/tmp/streamflow/3def5f88-d123-42c3-ab54-2e8ae1235007" && true 2>&1 on __LOCAL__/__LOCAL__ for job /step1/0
2026-06-07 15:53:28.538 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:alpine
Traceback (most recent call last):
File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/core/recovery.py", line 42, in wrapper
--
...<5 lines>...
)
cwl_utils.errors.JavascriptException: NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:alpine
2026-06-07 15:53:28.541 WARNING Job /step1/0 failure can not be recovered. Failure manager is not enabled.
2026-06-07 15:53:28.541 ERROR NodeJSEngine requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:alpine
2026-06-07 15:53:28.541 DEBUG Job /step1/0 changed status to FAILED
2026-06-07 15:53:28.541 DEBUG FAILED Job /step1/0 terminatedSorry for the mess testing this :) This PR is indeed fixing the problem 🙏 🙇
Contributor
|
Running conformance tests manually in the BSC MN5 login node with Internet, using CWL v1.0 with StreamFlow 0.2.0rc2 resulted in 75 failures. Running that again after switching to this branch: I will spend some minutes looking at these 26 errors to see if I can isolate what's wrong, and maybe test other PRs here or in cwl-utils, or see if I can create issues (EDIT: #1086). |
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
30203cd to
b78b055
Compare
Fix #1053 by adding `_get_container_engine()` to detect the available container runtime by probing `docker`, `singularity`, `podman`, and `udocker` in order, caching the result with `@cached(FIFOCache(1))` from `cachebox`. The detected engine is passed as `container_engine` to `cwl_utils.expression.interpolate()` in `eval_expression()`, fixing CWL JavaScript expression evaluation on systems without Docker.
GlassOfWhiskey
force-pushed
the
node-with-singularity
branch
from
b78b055 to
973963b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #1053 by adding
_get_container_engine()to detect the available container runtime by probingdocker,singularity,podman, andudockerin order, caching the result with@cached(FIFOCache(1))fromcachebox. The detected engine is passed ascontainer_enginetocwl_utils.expression.interpolate()ineval_expression(), fixing CWL JavaScript expression evaluation on systems without Docker.