feat(controller): surface scheduling stalls as WorkloadSchedulingBlocked - #309
feat(controller): surface scheduling stalls as WorkloadSchedulingBlocked#309ntheanh201 wants to merge 5 commits into
Conversation
Documents a proposed Job-tier condition (WorkloadSchedulingBlocked) for workload pods that cannot schedule — motivated by a real 7h Pending incident on an on-prem H200 cluster where the certification showed only InProgress/WorkloadRunning. ADR-gated per repo convention; awaiting approval before implementation. Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
Implements ADR-075. When a workload's pods cannot be placed (the scheduler explicitly rejected them, PodScheduled=False/Unschedulable), the Job now reports InProgress with reason WorkloadSchedulingBlocked and a condition message relaying the scheduler's own FailedScheduling diagnosis — instead of silently reading WorkloadRunning while the pod sits Pending for hours (observed: 7h stall with zero operator-visible signal). - Clock-neutral: the detector runs before the clock-start write, so WorkloadStartTime stays unset while blocked and timeoutPerJob/stall budgets never charge a workload that has not started (extends the issue NVIDIA#213 pending-time contract to the unschedulable case). - Detector owns both set and clear of the persisted schedulingBlockedSince marker; the caller only holds clock-neutral while the marker is set, so a transient detector miss cannot reset the grace clock. - Grace window defaults to 5m, overridable via spec.schedulingStallGraceSeconds; requires explicit PodScheduled=False (pods the scheduler has not processed yet are not evidence). - New Event involvedObject.name field index for keyed FailedScheduling lookups; the Job condition message relays the newest event text. - Integration case job-scheduling-blocked pins the full contract (condition + relayed message + epoch-normalized schedulingBlockedSince + absent workloadStartTime); unit tests cover grace window, mixed pods, recovery, and the no-condition pod case. Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
Adds the schedulingBlockedSince status field (ADR-075) to the Job API reference and enumerates the InProgress/Failed condition reasons, which were previously undocumented. Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
1debc1b to
42a688b
Compare
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…5 amendment) Live testing on a 3-node on-prem cluster exposed a guard race: the WorkloadStartTime != nil early-return in checkSchedulingBlocked prevented detection in the common case where the clock started on reconcile 1 (workload created) and the scheduler rejected the pod on reconcile 2. Amended decision: detection runs on every reconcile of a non-terminal workload. When blocked persists past the grace window, the caller clears WorkloadStartTime (pausing the clock) in the same write that sets WorkloadSchedulingBlocked; the first-observe logic restores it on recovery. Timeout accounting stays honest in both directions. Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
42a688b to
fa8ab59
Compare
The Job controller's scheduling-stall detector (ADR-075) lists Events via the involvedObject.name field index, and the manager watches Events for the same purpose. The manager-role only granted create+patch on events, causing 'Failed to watch *v1.Event: events is forbidden' spam on every startup and preventing the Event index from syncing — which made the WorkloadSchedulingBlocked condition message fall back to the generic 'Workload pods are unschedulable' instead of relaying the scheduler's FailedScheduling diagnosis. Verified live on a 3-node on-prem cluster: before this change, 'kubectl auth can-i list events' returned no; after applying the rule with the same content, the watch error cleared and the condition message carried the scheduler's diagnosis. Signed-off-by: The Anh Nguyen <ntheanh201@gmail.com>
asivanadi0
left a comment
asivanadi0
left a comment
There was a problem hiding this comment.
Draft feedback — treating this as reviewable, not an approve/request-changes gate.
The clock-pause amendment (detect after WorkloadStartTime is set, clear it once blocked past grace) matches the incident you describe, and the unit matrix around grace / mixed pods / event relay is in good shape.
Same ADR numbering collision as on #288: both PRs land an ADR-075. Please renumber one before either merges.
Two concrete gaps below (Reason filter vs SchedulingGated, and the --wait work the ADR still claims is in scope).
| for _, cond := range pod.Status.Conditions { | ||
| if cond.Type == corev1.PodScheduled && | ||
| cond.Status == corev1.ConditionFalse { | ||
| schedulingRejected = true |
There was a problem hiding this comment.
Draft feedback: Decision §3 and the API/docs all require PodScheduled=False with reason Unschedulable, but this only checks Type/Status. Pods waiting on a scheduling gate (Reason=SchedulingGated — common with ResourceClaims / DRA on GPU nodes) also carry PodScheduled=False and no nodeName, so after schedulingStallGraceSeconds they would surface as WorkloadSchedulingBlocked even though the scheduler has not rejected them.
On an HGX cert fleet that uses claim templates, that looks like a real false positive: the Job would flip to WorkloadSchedulingBlocked and pause the timeout clock while the gate is still doing intentional work.
Suggest gating on cond.Reason == corev1.PodReasonUnschedulable (and adding a SchedulingGated fixture that must stay non-blocked past grace). SchedulerError is a separate question — worth an explicit call either way.
| prints Job/Workflow condition-reason transitions for InProgress workloads | ||
| (e.g. `[watch] communication/nccl-all-reduce: InProgress — WorkloadSchedulingBlocked (2m)`) | ||
| so a stalled category is distinguishable from a progressing one without | ||
| leaving the stream. |
There was a problem hiding this comment.
Draft feedback: Implementation still lists a pkg/certification/certification.go --wait enhancement as in-scope (reason transitions like InProgress — WorkloadSchedulingBlocked (2m)), and Consequences claim operators see the reason in the --wait stream. This PR does not touch certification.go.
Either land that waiter change here, or demote it to an explicit follow-up and soften the Consequences / Decision §7 wording so the ADR matches what actually ships. As written, a reader of the ADR will expect the watch stream to distinguish a scheduling stall from a progressing category, and the current code will not.
| } | ||
| if job.Status.SchedulingBlockedSince != nil { | ||
| // Within the grace window: requeue, keep the clock as-is. | ||
| return ctrl.Result{RequeueAfter: r.getWorkloadRequeueInterval()}, nil |
There was a problem hiding this comment.
Draft feedback / clarifying question on the clock-pause contract during grace.
When WorkloadStartTime is already set and pods then become unschedulable, this branch requeues for the whole grace window with the clock as-is. timeoutPerJob therefore keeps burning until grace elapses and the blocked path clears WorkloadStartTime. The API comment and troubleshooting docs say blocked time does not count against timeoutPerJob — that is only true after the condition surfaces, not for the grace interval itself.
Is burning up to schedulingStallGraceSeconds of timeout budget on a newly-unschedulable-but-previously-running workload intentional (noise filter), or should the pause start when SchedulingBlockedSince is first recorded? If intentional, a one-line note on the field docs / ADR would save the next reader from treating the stronger claim as literal.
2 participants
Summary
Adds a first-class
WorkloadSchedulingBlockedcondition on the Job when a workload's pods cannot be placed — replacing the current behavior where unschedulable pods silently read asWorkloadRunningwhiletimeoutPerJobburns against hardware that never ran.Motivated by a real incident on an on-prem 3-node HGX H200 cluster: a per-node loopback job was hard-pinned to one node whose GPUs were held by tenants; the pod sat
Pendingfor 7 hours (FailedScheduling×114) with no operator-visible signal.What this adds
WorkloadSchedulingBlocked— set when the workload's pods carryPodScheduled=False/Unschedulablepast a grace window (default 5 min, tunable viaspec.schedulingStallGraceSeconds)WorkloadStartTimeis cleared when blocked persists (pausingtimeoutPerJoband the stall clock) and restored on the next genuinely-running observation — extends the issue [Bug]: Suspended TrainJob is treated as Running, causing spurious node failures on Kueue-managed clusters Labels: bug , needs-triage · Component: Controller / Reconcilers #213 pending-time contract to the unschedulable caseFailedSchedulingevent text) via a new EventinvolvedObject.namefield indexschedulingBlockedSincestatus field persists the blocked-episode start so controller restarts don't reset the grace clockCloses #310
Design
Full design in ADR-075 (Status: Proposed).
Testing
job-scheduling-blockedpinning the full contractWorkloadSchedulingBlockedwith the relayed scheduler diagnosis within the grace window,workloadStartTimecorrectly pausedType of Change
Component(s) Affected
Testing
make manifests generaterun (CRDs regenerated for new fields)Checklist
git commit -s)