Skip to content

Add scontrol_topology GCM monitor for periodic Slurm topology collection#167

Open
mitthu wants to merge 1 commit into
facebookresearch:mainfrom
mitthu:export-D110108419
Open

Add scontrol_topology GCM monitor for periodic Slurm topology collection#167
mitthu wants to merge 1 commit into
facebookresearch:mainfrom
mitthu:export-D110108419

Conversation

@mitthu

@mitthu mitthu commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary:
Adds a new GCM monitor that runs scontrol show topo every 60 seconds and publishes the results as logs for the fair_slurm_topology Scuba table. This supports two use cases from T275974051: SLA compliance monitoring for partial racks and topology-aware job debugging for GB300.

The monitor handles both block topology format (BlockName/BlockIndex/BlockSize, used on CW clusters) and switch/tree topology format (SwitchName/Level/LinkSpeed/Switches). Node lists in Slurm hostlist notation (e.g., g3-129-[057,059,063]) are expanded into individual node names using the existing nodelist() parser, with a 10000-entry truncation safety limit.

Differential Revision: D110108419

…ction

Summary:
Adds a new GCM monitor that runs `scontrol show topo` every 60 seconds and publishes the results as logs for the `fair_slurm_topology` Scuba table. This supports two use cases from T275974051: SLA compliance monitoring for partial racks and topology-aware job debugging for GB300.

The monitor handles both block topology format (`BlockName`/`BlockIndex`/`BlockSize`, used on CW clusters) and switch/tree topology format (`SwitchName`/`Level`/`LinkSpeed`/`Switches`). Node lists in Slurm hostlist notation (e.g., `g3-129-[057,059,063]`) are expanded into individual node names using the existing `nodelist()` parser, with a 10000-entry truncation safety limit.

Differential Revision: D110108419
@mitthu mitthu requested review from calebho and giongto35 as code owners June 29, 2026 22:33
@meta-cla meta-cla Bot added the cla signed label Jun 29, 2026
@meta-codesync

meta-codesync Bot commented Jun 29, 2026

Copy link
Copy Markdown

@mitthu has exported this pull request. If you are a Meta employee, you can view the originating Diff in D110108419.

@github-actions

Copy link
Copy Markdown

CI Commands

The following CI workflows run automatically on every push and pull request:

Workflow What it runs
GPU Cluster Monitoring Python CI lint, tests, typecheck, format, deb build, pyoxidizer builds
Go packages CI shelper tests, format, lint

The following commands can be used by maintainers to trigger additional tests that require access to secrets:

Command Description Requires approval?
/metaci tests Runs Meta internal integration tests (pytest) Yes — a maintainer must trigger the command and approve the deployment request
/metaci integration tests Same as above (alias) Yes

Note: Only repository maintainers (OWNER association) can trigger /metaci commands. After commenting the command, a maintainer must also navigate to the Actions tab and approve the deployment to the graph-api-access environment before the jobs will run. See the approval guidelines for what to approve or reject.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants