ReFrame hangs/waits on stale REQUEUED Slurm job?

[pramodk]

@vkarak - let me know if this makes sense, I quickly gathered below from an external cluster:

Description

ReFrame can hang / wait indefinitely when a Slurm job fails during node startup/prolog handling and Slurm accounting records the top-level job as REQUEUED, even though the job is no longer active.

In this case, the job hit a prolog/startup issue on a GPU node. The job then entered CG while epilog handling was still active, and I cancelled it manually. After cancellation:

• squeue no longer showed the job
• scontrol show job reported the job id as invalid
• sacct continued to report the top-level job as REQUEUED
• ReFrame continued polling ?

Environment

• ReFrame version: master branch f1744f0
• Scheduler: Slurm
• Launcher: srun
• Test type: RunOnlyRegressionTest
• Test command: simple nvidia-smi query

Logs

$ /path/to/reframe -C sites/site-config.py -c checks/nvidia_smi.py --system cluster:gpu_preflight -r

[2026-07-17T12:45:56.983732] info: reframe: [ReFrame Setup]
[2026-07-17T12:45:56.983968] info: reframe:   version:           4.10.1
[2026-07-17T12:45:56.984149] info: reframe:   command:           '/path/to/reframe -C sites/site-config.py -c checks/nvidia_smi.py
--system cluster:gpu_preflight -r'
[2026-07-17T12:45:56.984840] info: reframe:   selected system:   'cluster'
[2026-07-17T12:45:56.985176] info: reframe:   stage directory:   '/path/to/reframe/prefix/stage'
[2026-07-17T12:45:56.985339] info: reframe:   output directory:  '/path/to/reframe/prefix/output'

[2026-07-17T12:45:57.023300] debug: reframe:   > Loaded 2 test(s)
[2026-07-17T12:45:57.024122] debug: reframe:   Generated test case: nvidia_smi_slurm @cluster:gpu_preflight+builtin
[2026-07-17T12:45:57.024586] verbose: reframe: Generated 1 test case(s)
[2026-07-17T12:45:57.028355] info: reframe: [ RUN      ] nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin

[2026-07-17T12:45:57.053647] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: [CMD] 'srun --version'
[2026-07-17T12:45:57.068742] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Setting up <class 'type'> job for
test 'nvidia_smi_slurm' (scheduler: 'slurm', launcher: 'srun')
[2026-07-17T12:45:57.091882] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Generating the run script
[2026-07-17T12:45:57.108022] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: [CMD] 'sbatch rfm_job.sh'
[2026-07-17T12:45:57.193867] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Spawned run job (id=48)
[2026-07-17T12:45:57.196217] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'

ReFrame then continued polling job 48:

  [2026-07-17T12:52:45.627454] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'
  [2026-07-17T12:52:45.664555] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Entering stage: run
  [2026-07-17T12:52:45.668806] debug2: reframe: Bumped 0 test(s)

  [2026-07-17T12:53:27.277247] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'
  [2026-07-17T12:53:27.313000] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Entering stage: run
  [2026-07-17T12:53:27.316935] debug2: reframe: Bumped 0 test(s)

  [2026-07-17T12:53:47.245694] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'
  [2026-07-17T12:53:47.279160] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Entering stage: run

...
...

  [2026-07-17T16:15:07.244951] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'
  [2026-07-17T16:15:07.271352] debug2: reframe: Current tests: 1
  [2026-07-17T16:15:07.273198] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Entering stage: run
  [2026-07-17T16:15:07.276642] debug2: reframe: Bumped 0 test(s)

  [2026-07-17T16:15:17.242564] debug: reframe: [CMD] 'sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,end,nodelist'
  [2026-07-17T16:15:17.273976] debug2: reframe: Current tests: 1
  [2026-07-17T16:15:17.283561] debug: nvidia_smi_slurm /13e5e118 @cluster:gpu_preflight+builtin: Entering stage: run
  [2026-07-17T16:15:17.289189] debug2: reframe: Bumped 0 test(s)

The job was originally submitted at 2026-07-17T12:45:57, so ReFrame was still polling job 48 roughly 3.5 hours later.

Slurm State After Cancellation

  $ squeue -a -j 48 -o "%.18i %.12T %.30R %.20N"
  slurm_load_jobs error: Invalid job id specified

  $ scontrol show job 48
  slurm_load_jobs error: Invalid job id specified

  $ sacct -S 2026-07-17 -P -j 48 -o jobid,state,exitcode,reason,nodelist,start,end,elapsed
  JobID|State|ExitCode|Reason|NodeList|Start|End|Elapsed
  48|REQUEUED|0:54|None|gpu-node-7|2026-07-17T12:45:57|2026-07-17T12:46:01|00:00:04
  48.batch|CANCELLED|||gpu-node-7|2026-07-17T12:45:57|2026-07-17T12:50:17|00:04:20
  48.extern|CANCELLED|||gpu-node-7|2026-07-17T12:45:57|2026-07-17T12:50:17|00:04:20

Generated Slurm Script

Anonymized node names:

  #!/bin/bash
  #SBATCH --job-name="rfm_nvidia_smi_slurm"
  #SBATCH --ntasks=1
  #SBATCH --ntasks-per-node=1
  #SBATCH --output=rfm_job.out
  #SBATCH --error=rfm_job.err
  #SBATCH --time=24:0:0
  #SBATCH --partition=gpu
  #SBATCH --reservation=preflight
  #SBATCH --nodelist=gpu-node-[...]
  #SBATCH --gpus-per-task=4

  srun nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader,nounits

[vkarak]

ReFrame behaves as expected here, because it treats the REQUEUED state as pending:

reframe/reframe/core/schedulers/slurm.py

Line 53 in 780c2c4

'REQUEUED',

I am not sure why Slurm reports the job as requeued and then it is never requeued.