Intermittent `KeyError` when polling Slurm job arrays

[pranav-sivaraman]

Description

ReFrame intermittently raises a KeyError shortly after starting independent job arrays on the same Slurm partition. It is most likely to reproduce when running the same check with multiple programming environments.

Steps to reproduce

Save the following files and run:

reframe --nocolor -c checks.py -C config.yaml -r

The failure is timing-dependent, so the command may need to be run multiple times.

checks.py

import reframe as rfm
import reframe.utility.sanity as sn


@rfm.simple_test
class SlurmJobArray(rfm.RunOnlyRegressionTest):
    """Reproducer for two job arrays polled on the same partition."""

    valid_systems = ["generic-cluster:compute"]
    valid_prog_environs = ["env-a", "env-b"]

    num_array_tasks = 32
    num_tasks = 1
    num_tasks_per_node = 1

    @run_after("setup")
    def set_job_array(self):
        self.job.options += [f"--array=0-{self.num_array_tasks - 1}"]

    # Stagger elements so both environment-specific arrays overlap in polling.
    executable = "bash"
    executable_opts = [
        "-c",
        "'sleep $((SLURM_ARRAY_TASK_ID % 4 + 1)); echo Hello world'",
    ]

    @sanity_function
    def validate(self):
        count = sn.count(sn.findall(r"Hello world", self.stdout))
        return sn.assert_eq(
            count,
            self.num_array_tasks,
            msg="expected {1} Hello world lines, but found {0}",
        )

config.yaml

systems:
- name: generic-cluster
  descr: Generic Slurm reproducer
  hostnames: ['.*']
  partitions:
  - name: compute
    descr: Generic compute partition
    scheduler: slurm
    launcher: srun
    environs: [env-a, env-b]
    max_jobs: 10

environments:
- name: env-a
- name: env-b

Error output

[ReFrame Setup]
  version:           4.10.3
  command:           '/ssd0/data/users/sivaramp/.local/bin/reframe --nocolor -c checks.py -C config.yaml -r'
  launched by:       sivaramp@generic-login1
  working directory: '/ssd0/data/users/sivaramp'
  settings files:    '<builtin>', '/ssd0/data/users/sivaramp/config.yaml'
  selected system:   'generic-cluster'
  check search path: '/ssd0/data/users/sivaramp/checks.py'
  stage directory:   '/ssd0/data/users/sivaramp/stage'
  output directory:  '/ssd0/data/users/sivaramp/output'
  log files:         '/tmp/rfm-nqw1n9ov.log'
  results database:  [off] '/ssd0/data/users/sivaramp/.reframe/reports/results.db'

[==========] Running 1 check(s)
[==========] Started on Tue Sep  1 17:57:22 2026+0000

[----------] start processing checks
[ RUN      ] SlurmJobArray /4c932770 @generic-cluster:compute+env-a
[ RUN      ] SlurmJobArray /4c932770 @generic-cluster:compute+env-b
[  PASSED  ] Ran 0/2 test case(s) from 1 check(s) (0 failure(s), 0 expected failure(s), 0 skipped, 0 aborted)
[==========] Finished on Tue Sep  1 17:57:24 2026+0000
ERROR: run session stopped: key error: '44153'
ERROR: Traceback (most recent call last):
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/cli.py", line 1780, in main
    runner.runall(testcases, restored_cases)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/logging.py", line 1139, in _fn
    return fn(*args, **kwargs)
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/__init__.py", line 731, in runall
    self._runall(testcases)
    ~~~~~~~~~~~~^^^^^^^^^^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/__init__.py", line 824, in _runall
    self._policy.exit()
    ~~~~~~~~~~~~~~~~~^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/policies.py", line 430, in exit
    self._poll_tasks()
    ~~~~~~~~~~~~~~~~^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/policies.py", line 480, in _poll_tasks
    sched.poll(*jobs)
    ~~~~~~~~~~^^^^^^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/schedulers/slurm.py", line 561, in poll
    self._cancel_if_blocked(jobs)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/schedulers/slurm.py", line 605, in _cancel_if_blocked
    pending_job = pending_jobs[jobid]
                  ~~~~~~~~~~~~^^^^^^^
KeyError: '44153'

Log file(s) saved in '/tmp/rfm-nqw1n9ov.log'