FOREIGN KEY constraint failed after a restart from another leader

[OlivierRo]

Toil version : 9.1.0
Python version : 3.10.18
Context : One leader launches workflow from container A. Container A is stopped. Another leader from container B then restarts previous workflow using "--restart" option.

Everything works fine except at the end :

**[2026-06-25T07:49:31+0000] [MainThread] [I] [toil.lib.history] Workflow 832e46b2-8f1f-4aa1-9bc7-21c8a39cb145 stopped. Success: True
Workflow 832e46b2-8f1f-4aa1-9bc7-21c8a39cb145 stopped. Success: True

Traceback (most recent call last):
  File "/usr/local/bin/toil-cwl-runner", line 6, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/toil/cwl/cwltoil.py", line 4475, in main
    with Toil(options, workflow_name=workflow_name, trs_spec=trs_spec) as toil:
  File "/usr/local/lib/python3.10/site-packages/toil/common.py", line 1061, in __exit__
    HistoryManager.record_workflow_attempt(
  File "/usr/local/lib/python3.10/site-packages/toil/lib/retry.py", line 299, in call
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/toil/lib/history.py", line 536, in record_workflow_attempt
    cur.execute(
sqlite3.IntegrityError: FOREIGN KEY constraint failed**

"Normal" launch command :
exec /usr/local/bin/toil-cwl-runner --jobStore=file:<jobStoreDir> --logLevel=DEBUG --logFile=<logFile> --writeLogs=<logDir> --writeLogsFromAllJobs=true --batchSystem=mesos --mesosMaster=<mesosMaster>:5050 --mesosRole=PRIORITY_050 --clean=never --cleanWorkDir=always --preserve-entire-environment --outdir=<outDir> --realTimeLogging=true --disableCaching --linkImports --disableHotDeployment --moveExports --maxLogFileSize=10eb --retryCount=0 <cwlPath> <argsJsonPath>

"Replay" launch command :
exec /usr/local/bin/toil-cwl-runner --jobStore=file:<jobStoreDir> --logLevel=DEBUG --logFile=<logFile> --writeLogs=<logDir> --writeLogsFromAllJobs=true --batchSystem=mesos --mesosMaster=<mesosMaster>:5050 --mesosRole=PRIORITY_050 --clean=never --cleanWorkDir=always --preserve-entire-environment --outdir=<outDir> --realTimeLogging=true **--restart** --disableCaching --linkImports --disableHotDeployment --moveExports --maxLogFileSize=10eb --retryCount=0 <cwlPath> <argsJsonPath>

If I restart container A to restart previous workflow instead of container B, everything finishes OK.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1847

[adamnovak]

Ah OK the history system doesn't contemplate that it could see some of the restarts of the workflow but not the initial setup.

You should be able to reproduce this with just one container if you delete the history database from ~/.toil before the restart.

You should be able to work around this by doing export TOIL_HISTORY=false (or otherwise setting that environment variable) so we don't try and use the history database.

We'll need to fix this by writing a workflow entry to the history database when we restart a workflow we didn't know was started.

[OlivierRo]

Workaround works great, thank you.