slurm: keep slurm_monitor alive on pre-release Slurm versions#171
Open
sdmcclain wants to merge 3 commits into
Open
slurm: keep slurm_monitor alive on pre-release Slurm versions#171sdmcclain wants to merge 3 commits into
sdmcclain wants to merge 3 commits into
Conversation
## Summary
`gcm slurm_monitor` crash-loops to a supervisord FATAL — taking down the Slurm controller's telemetry/accounting — on any cluster whose Slurm build reports a SchedMD pre-release version. `sinfo -V` returns e.g. `slurm 26.05.2-0pre1`; the version was parsed by `int()`-splitting each dotted component, so `int("2-0pre1")` raised `ValueError: invalid literal for int() with base 10: '2-0pre1'`, which was then re-raised as a `click.UsageError` (a misleading `Usage: ... Try --help` banner) and exited non-zero on every start.
Two changes:
- `monitoring/slurm/sinfo.py`: `get_slurm_version()` had its own fragile `int()`-split of `sinfo -V`. Delegate to `clusterscope.slurm_version()` (already a dependency), which parses SchedMD pre-release and zero-padded versions robustly (companion clusterscope change) — one parser instead of two.
- `monitoring/sink/utils.py`: drop the `except ValueError -> click.UsageError` branch in `write_to_sink_with_retries`, so a runtime `ValueError` (a data/parse error) falls through to the existing generic `except Exception -> click.ClickException` handler instead of being mislabeled a CLI usage error.
## Test Plan
`get_slurm_version()` is now `return clusterscope.slurm_version()`. That function parses every SchedMD form that previously crashed (with `sinfo -V` output mocked):
```
$ python -c "
from unittest.mock import patch
import clusterscope.lib as lib
for s in ['26.05.2-0pre1', '24.11.2-0pre1', '24.05.0']:
with patch.object(lib, 'get_unified_info') as m:
m.return_value.get_slurm_version.return_value = s
print(f'{s:16} -> {lib.slurm_version()}')
"
26.05.2-0pre1 -> (26, 5, 2)
24.11.2-0pre1 -> (24, 11, 2)
24.05.0 -> (24, 5, 0)
```
`sink/utils.py` is an error-path change, verified by inspection of the `try/except`: the `OutOfRetries` and generic `Exception -> ClickException` branches are unchanged; only the intermediate `ValueError -> UsageError` reclassification is removed, so a data/parse error no longer surfaces as a bogus CLI usage error.
CI CommandsThe following CI workflows run automatically on every push and pull request:
The following commands can be used by maintainers to trigger additional tests that require access to secrets:
|
luccabb
approved these changes
Jul 13, 2026
| # int()-split of `sinfo -V` here. | ||
| import clusterscope | ||
|
|
||
| return clusterscope.slurm_version() |
Contributor
There was a problem hiding this comment.
Reformat the `print_tb` assert to satisfy the pinned `ufmt` (ufmt==2.5.0 / black); fixes the failing `nox` ufmt check.
Co-authored-by: lucca bertoncini <32229669+luccabb@users.noreply.github.com>
tooji
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gcm slurm_monitorcrash-loops to a supervisord FATAL — taking down the Slurm controller's telemetry/accounting — on any cluster whose Slurm build reports a SchedMD pre-release version.sinfo -Vreturns e.g.slurm 26.05.2-0pre1; the version was parsed byint()-splitting each dotted component, soint("2-0pre1")raisedValueError: invalid literal for int() with base 10: '2-0pre1', which was then re-raised as aclick.UsageError(a misleadingUsage: ... Try --helpbanner) and exited non-zero on every start.Two changes:
monitoring/slurm/sinfo.py:get_slurm_version()had its own fragileint()-split ofsinfo -V. Delegate toclusterscope.slurm_version()(already a dependency), which parses SchedMD pre-release and zero-padded versions robustly (companion clusterscope change) — one parser instead of two.monitoring/sink/utils.py: drop theexcept ValueError -> click.UsageErrorbranch inwrite_to_sink_with_retries, so a runtimeValueError(a data/parse error) falls through to the existing genericexcept Exception -> click.ClickExceptionhandler instead of being mislabeled a CLI usage error.Test Plan
get_slurm_version()is nowreturn clusterscope.slurm_version(). That function parses every SchedMD form that previously crashed (withsinfo -Voutput mocked):sink/utils.pyis an error-path change, verified by inspection of thetry/except: theOutOfRetriesand genericException -> ClickExceptionbranches are unchanged; only the intermediateValueError -> UsageErrorreclassification is removed, so a data/parse error no longer surfaces as a bogus CLI usage error.