Slurm batch system: RuntimeError: Unknown jobID, could not be converted

[CGuichard]

When using Toil with --batchSystem=slurm some workflows triggered the following error: "RuntimeError: Unknown jobID, could not be converted"

Logs:

Traceback (most recent call last):
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 329, in run
    while self._runStep():
          ^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 316, in _runStep
    if self.checkOnJobs():
       ^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 256, in checkOnJobs
    statuses = self.boss.with_retries(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 655, in with_retries
    return operation(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/slurm.py", line 423, in coalesce_job_exit_codes
    status_dict = self._get_job_details(job_id_list)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/slurm.py", line 466, in _get_job_details
    status_dict.update(self._getJobDetailsFromScontrol(job_id_list))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/slurm.py", line 810, in _getJobDetailsFromScontrol
    job[key] += " " + bits[0]  # type: ignore[has-type]  # we depend on the previous iteration to populate key
        ^^^
UnboundLocalError: cannot access local variable 'key' where it is not associated with a value

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/venv/lib/python3.12/site-packages/toil/common.py", line 1308, in start
    return self._runMainLoop(rootJobDescription)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/common.py", line 1913, in _runMainLoop
    ).run()
      ^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/leader.py", line 299, in run
    self.innerLoop()
  File "/tmp/venv/lib/python3.12/site-packages/toil/leader.py", line 987, in innerLoop
    update: UpdatedBatchJobInfo = self.batchSystem.getUpdatedBatchJob(maxWait=2)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 583, in getUpdatedBatchJob
    raise self.GridEngineThreadException(
toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThreadException: Unexpected GridEngineThread failure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/venv/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/cwl/cwltoil.py", line 4572, in main
    outobj = toil.start(wf1)
             ^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/common.py", line 1310, in start
    self._shutdownBatchSystem()
  File "/tmp/venv/lib/python3.12/site-packages/toil/common.py", line 1919, in _shutdownBatchSystem
    self._batchSystem.shutdown()
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 621, in shutdown
    self.background_thread.killJobs()
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 222, in killJobs
    batchJobID = self.getBatchSystemID(jobID)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/venv/lib/python3.12/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 116, in getBatchSystemID
    raise RuntimeError("Unknown jobID, could not be converted")
RuntimeError: Unknown jobID, could not be converted

It seems the problem is raised in _getJobDetailsFromScontrol when parsing scontrol stdout. Unfortunately I could not obtain the stdout that triggered this error.

I implemented a fix for this method here: csgroup-oss@4ffb655. It's simple python logic to fix UnboundLocalError by ensuring the variable cannot be unbound, I don't know if it will fix my problem but it should prevent the crash for this reason at least.

Is this fix interesting for Toil, should I open a PR?

Environment

• toil (toil[cwl]): 9.5.0
• python: 3.12
• slurm: 23.11.8-BullSequana.1.0

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1853

[adamnovak]

We could definitely take a PR for this. I don't think ending up with an UnboundLocalError is the right way to handle unintelligible output from Slurm.

But we do probably want to still produce some kind of error if we can't understand what Slurm is saying (and also report the offending Slurm output).

Probably we should be retrying the Slurm command in this case, and then failing the workflow if it exists but consistently gives back something we don't understand.