Skip to content

slurm: keep slurm_monitor alive on pre-release Slurm versions#171

Open
sdmcclain wants to merge 3 commits into
facebookresearch:mainfrom
sdmcclain:fix-slurm-monitor-prerelease-version
Open

slurm: keep slurm_monitor alive on pre-release Slurm versions#171
sdmcclain wants to merge 3 commits into
facebookresearch:mainfrom
sdmcclain:fix-slurm-monitor-prerelease-version

Conversation

@sdmcclain

Copy link
Copy Markdown
Contributor

Summary

gcm slurm_monitor crash-loops to a supervisord FATAL — taking down the Slurm controller's telemetry/accounting — on any cluster whose Slurm build reports a SchedMD pre-release version. sinfo -V returns e.g. slurm 26.05.2-0pre1; the version was parsed by int()-splitting each dotted component, so int("2-0pre1") raised ValueError: invalid literal for int() with base 10: '2-0pre1', which was then re-raised as a click.UsageError (a misleading Usage: ... Try --help banner) and exited non-zero on every start.

Two changes:

  • monitoring/slurm/sinfo.py: get_slurm_version() had its own fragile int()-split of sinfo -V. Delegate to clusterscope.slurm_version() (already a dependency), which parses SchedMD pre-release and zero-padded versions robustly (companion clusterscope change) — one parser instead of two.
  • monitoring/sink/utils.py: drop the except ValueError -> click.UsageError branch in write_to_sink_with_retries, so a runtime ValueError (a data/parse error) falls through to the existing generic except Exception -> click.ClickException handler instead of being mislabeled a CLI usage error.

Test Plan

get_slurm_version() is now return clusterscope.slurm_version(). That function parses every SchedMD form that previously crashed (with sinfo -V output mocked):

$ python -c "
from unittest.mock import patch
import clusterscope.lib as lib
for s in ['26.05.2-0pre1', '24.11.2-0pre1', '24.05.0']:
    with patch.object(lib, 'get_unified_info') as m:
        m.return_value.get_slurm_version.return_value = s
        print(f'{s:16} -> {lib.slurm_version()}')
"
26.05.2-0pre1    -> (26, 5, 2)
24.11.2-0pre1    -> (24, 11, 2)
24.05.0          -> (24, 5, 0)

sink/utils.py is an error-path change, verified by inspection of the try/except: the OutOfRetries and generic Exception -> ClickException branches are unchanged; only the intermediate ValueError -> UsageError reclassification is removed, so a data/parse error no longer surfaces as a bogus CLI usage error.

## Summary

`gcm slurm_monitor` crash-loops to a supervisord FATAL — taking down the Slurm controller's telemetry/accounting — on any cluster whose Slurm build reports a SchedMD pre-release version. `sinfo -V` returns e.g. `slurm 26.05.2-0pre1`; the version was parsed by `int()`-splitting each dotted component, so `int("2-0pre1")` raised `ValueError: invalid literal for int() with base 10: '2-0pre1'`, which was then re-raised as a `click.UsageError` (a misleading `Usage: ... Try --help` banner) and exited non-zero on every start.

Two changes:

- `monitoring/slurm/sinfo.py`: `get_slurm_version()` had its own fragile `int()`-split of `sinfo -V`. Delegate to `clusterscope.slurm_version()` (already a dependency), which parses SchedMD pre-release and zero-padded versions robustly (companion clusterscope change) — one parser instead of two.
- `monitoring/sink/utils.py`: drop the `except ValueError -> click.UsageError` branch in `write_to_sink_with_retries`, so a runtime `ValueError` (a data/parse error) falls through to the existing generic `except Exception -> click.ClickException` handler instead of being mislabeled a CLI usage error.

## Test Plan

`get_slurm_version()` is now `return clusterscope.slurm_version()`. That function parses every SchedMD form that previously crashed (with `sinfo -V` output mocked):

```
$ python -c "
from unittest.mock import patch
import clusterscope.lib as lib
for s in ['26.05.2-0pre1', '24.11.2-0pre1', '24.05.0']:
    with patch.object(lib, 'get_unified_info') as m:
        m.return_value.get_slurm_version.return_value = s
        print(f'{s:16} -> {lib.slurm_version()}')
"
26.05.2-0pre1    -> (26, 5, 2)
24.11.2-0pre1    -> (24, 11, 2)
24.05.0          -> (24, 5, 0)
```

`sink/utils.py` is an error-path change, verified by inspection of the `try/except`: the `OutOfRetries` and generic `Exception -> ClickException` branches are unchanged; only the intermediate `ValueError -> UsageError` reclassification is removed, so a data/parse error no longer surfaces as a bogus CLI usage error.
@github-actions

Copy link
Copy Markdown

CI Commands

The following CI workflows run automatically on every push and pull request:

Workflow What it runs
GPU Cluster Monitoring Python CI lint, tests, typecheck, format, deb build, pyoxidizer builds
Go packages CI shelper tests, format, lint

The following commands can be used by maintainers to trigger additional tests that require access to secrets:

Command Description Requires approval?
/metaci tests Runs Meta internal integration tests (pytest) Yes — a maintainer must trigger the command and approve the deployment request
/metaci integration tests Same as above (alias) Yes

Note: Only repository maintainers (OWNER association) can trigger /metaci commands. After commenting the command, a maintainer must also navigate to the Actions tab and approve the deployment to the graph-api-access environment before the jobs will run. See the approval guidelines for what to approve or reject.

# int()-split of `sinfo -V` here.
import clusterscope

return clusterscope.slurm_version()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you'll need to

publish clusterscope

upgrade gcm's clusterscope

Comment thread gcm/monitoring/slurm/sinfo.py Outdated
sdmcclain and others added 2 commits July 13, 2026 15:17
Reformat the `print_tb` assert to satisfy the pinned `ufmt` (ufmt==2.5.0 / black); fixes the failing `nox` ufmt check.
Co-authored-by: lucca bertoncini <32229669+luccabb@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants