Reframe 4.9 added support for expected failures. This would be useful to documents tests which broke after an update, but decided to keep the update image regardless.
However running with reframe 4.10 yields the error:
[ OK ] ( 35/180) ModulePathTestCirrusEX %path=/opt/cray/pe/lmod/modulefiles/compiler/gnu/10.0 /16e67a2c @cirrus-ex:login+Default
[ OK ] ( 36/180) ModulePathTestCirrusEX %path=/opt/cray/pe/lmod/modulefiles/compiler/aocc/4.1 /2a1eca75 @cirrus-ex:login+Default
[ OK ] ( 37/180) ModulePathTestCirrusEX %path=/opt/cray/pe/lmod/modulefiles/compiler/intel/2023.2 /05472861 @cirrus-ex:login+Default
[ OK ] ( 38/180) ModulePathTestCirrusEX %path=/opt/cray/pe/lmod/modulefiles/mix_compilers /e82cd9f4 @cirrus-ex:login+Default
[ OK ] ( 39/180) ModulePathTestCirrusEX %path=/opt/cray/pe/lmod/modulefiles/perftools/25.03.0 /b330a6fc @cirrus-ex:login+Default
[ PASSED ] Ran 39/180 test case(s) from 91 check(s) (0 failure(s), 0 expected failure(s), 0 skipped, 0 aborted)
[==========] Finished on Thu Jul 16 16:26:36 2026+0100
============================== PERFORMANCE REPORT ==============================
┍━━━━━━━━┯━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━┑
│ name │ sysenv │ job_nodelist │ pvar │ punit │ pval │ presult │
┝━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━━━━━━┿━━━━━━━━┿━━━━━━━━━┿━━━━━━━━┿━━━━━━━━━━━┥
┕━━━━━━━━┷━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━┙
ERROR: run session stopped: key error: <reframe.core.schedulers.slurm._SlurmJob object at 0x7f955756dd90>
ERROR: Traceback (most recent call last):
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/frontend/cli.py", line 1780, in main
runner.runall(testcases, restored_cases)
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/core/logging.py", line 1138, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/frontend/executors/__init__.py", line 731, in runall
self._runall(testcases)
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/frontend/executors/__init__.py", line 824, in _runall
self._policy.exit()
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/frontend/executors/policies.py", line 430, in exit
self._poll_tasks()
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/frontend/executors/policies.py", line 480, in _poll_tasks
sched.poll(*jobs)
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/core/schedulers/slurm.py", line 561, in poll
self._cancel_if_blocked(jobs)
File "/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/environments/benchmarks/opt/linux-rhel9-zen5/none-none/reframe-4.10.1-ps53qgxgr25spcg4wdr6dqbbgz73qmpq/lib/python3.11/site-packages/reframe/core/schedulers/slurm.py", line 606, in _cancel_if_blocked
pending_reasons[pending_job].setdefault([])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: <reframe.core.schedulers.slurm._SlurmJob object at 0x7f955756dd90>
Log file(s) saved in '/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/run-reframe/reframe.out', '/mnt/lustre/e1000/home/z19/z19/lparisi/nfs-testing/run-reframe/reframe.log'
Reframe 4.9 added support for expected failures. This would be useful to documents tests which broke after an update, but decided to keep the update image regardless.
However running with reframe 4.10 yields the error: