fix(foreman): make the CustomResourceState config produce usable metrics - #1650
Merged
Defilan merged 2 commits intoAug 24, 2026
Merged
Conversation
foreman.crs.enabled has never worked. Enabling it yields no Foreman metrics at
all, which is why every panel in the dashboard shipped alongside it renders
empty and its three alert rules can never fire. Four defects, each verified
against a live kube-state-metrics:
1. The ConfigMap held a bare `resources:` mapping.
--custom-resource-state-config-file expects a whole document. Given the bare
form kube-state-metrics parses an empty config, logs NOTHING, and serves no
custom metrics: indistinguishable from a healthy install with no CRs. Now
wrapped in kind: CustomResourceStateMetrics / spec:.
2. Every gauge wrote `valueFrom: {path: [...]}`. The schema is []string, so
once (1) was fixed kube-state-metrics crashlooped with "cannot unmarshal
!!map into []string".
3. gauge.valueFrom is RELATIVE to gauge.path, and path is required. With an
absolute valueFrom and no path the config parses and emits nothing,
silently. Measured: absolute form 0 series, path form 10.
4. Enum status fields used type: Gauge with a `labelsMapping` block.
labelsMapping is not a kube-state-metrics field and was ignored, and a Gauge
value must be a number, boolean or RFC3339 string, so a Gauge over
status.phase ("Ready") emits nothing. These are StateSets.
Because (4) moves label values from a mapped lowercase form to the CRs' actual
enum values, the dashboard's queries move with it: phase="ready" becomes
phase="Ready", verdict="incomplete" becomes "INCOMPLETE".
The same pass adds the labels the fleet view needs to be worth rendering. It
could previously list nodes but not say what any of them was, or what it was
doing:
- every resource now labels its series with the object name, so a stuck task
can be named rather than only counted
- foreman_fleetnode_info gains current_task, version and agent_kind
- foreman_fleetnode_role_info is new, one series per role, because spec.roles
is a list and joining it into one label would make "which nodes can review?"
a substring match
- foreman_agentictask_info gains node (status.assignedNode) and kind, which is
the join a fleet view is built on
Drops foreman_agentictask_tasks_failed: with phase a real StateSet it is
exactly phase{phase="Failed"}, nothing queried it, and it cost 630 series here.
Verified on a live cluster: all 14 metric families emit, and every query in the
shipped dashboard returns data.
Refs defilantech#1643
Signed-off-by: Christopher Maher <chris@mahercode.io>
The machine an agent pod runs on, as a metric label. The property itself ships in the defilantech#1640 implementation; this exposes it to fleet dashboards. Empty for off-cluster agents. Refs defilantech#1640, defilantech#1643 Signed-off-by: Christopher Maher <chris@mahercode.io>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
7 tasks
Merged
doonga
pushed a commit
to greyrock-labs/home-ops
that referenced
this pull request
Aug 24, 2026
…mkube (0.9.19 ➔ 0.9.20) (#402) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts-mirror/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.19` → `0.9.20` | --- ### Release Notes <details> <summary>defilantech/LLMKube (ghcr.io/home-operations/charts-mirror/llmkube)</summary> ### [`v0.9.20`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0920-2026-08-24) [Compare Source](defilantech/LLMKube@v0.9.19...v0.9.20) ##### Features - **foreman:** opt-in archival of task audit records and transcripts ([#​1655](defilantech/LLMKube#1655)) ([c577133](defilantech/LLMKube@c577133)) - **foreman:** report the Kubernetes node on FleetNode.status, not as identity ([#​1649](defilantech/LLMKube#1649)) ([7563e8a](defilantech/LLMKube@7563e8a)) ##### Bug Fixes - **controller:** carry Model tolerations onto the prefetch Job ([#​1622](defilantech/LLMKube#1622)) ([d0db838](defilantech/LLMKube@d0db838)) - **controller:** clear the controller's own schedulingStatus once a service is Ready ([#​1633](defilantech/LLMKube#1633)) ([315f34d](defilantech/LLMKube@315f34d)) - Empty payload.repo silently defeats the upstream-base fetch: task branches cut from a stale fork HEAD ([#​1625](defilantech/LLMKube#1625)) ([#​1626](defilantech/LLMKube#1626)) ([27a015a](defilantech/LLMKube@27a015a)) - **foreman:** carry the in-pod result extras on every Job-mode branch ([#​1657](defilantech/LLMKube#1657)) ([d1ce51c](defilantech/LLMKube@d1ce51c)) - **foreman:** free the agent's in-process slot while a Job-mode task runs ([#​1635](defilantech/LLMKube#1635)) ([0d64fbb](defilantech/LLMKube@0d64fbb)) - **foreman:** gate the make-invoked CI checks, and pin them against the workflows ([#​1642](defilantech/LLMKube#1642)) ([4e7ab7b](defilantech/LLMKube@4e7ab7b)) - **foreman:** make the CustomResourceState config produce usable metrics ([#​1650](defilantech/LLMKube#1650)) ([1620443](defilantech/LLMKube@1620443)) ##### Documentation - add ROCm host-retune runbook for >64GB on Strix Halo ([#​1387](defilantech/LLMKube#1387)) ([fff5e59](defilantech/LLMKube@fff5e59)) - fix guide instructions that fail against the shipped API ([#​1629](defilantech/LLMKube#1629)) ([fba0bf4](defilantech/LLMKube@fba0bf4)) - **proposals:** foreman run, an unattended orchestration loop ([#​1652](defilantech/LLMKube#1652)) ([df57f49](defilantech/LLMKube@df57f49)) - reframe the multi-GPU guide from an Issue [#​2](defilantech/LLMKube#2) validation plan into a deployment guide ([#​1631](defilantech/LLMKube#1631)) ([907ac8d](defilantech/LLMKube@907ac8d)) - **runbook:** correct the metal-agent memory-pressure runbook against pkg/agent ([#​1630](defilantech/LLMKube#1630)) ([f665674](defilantech/LLMKube@f665674)) - serving one model across two DGX Sparks via llama.cpp RPC ([#​1620](defilantech/LLMKube#1620)) ([7eaf04b](defilantech/LLMKube@7eaf04b)) </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC40MC4wIiwidXBkYXRlZEluVmVyIjoiNDQuNDAuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/402
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make
foreman.crs.enabledproduce metrics — it has never emitted a single one —and add the labels a fleet view actually needs.
Why
Fixes #1643. Refs #1602, #1640.
Four defects, each reproduced against a live kube-state-metrics, detailed in
#1643: the ConfigMap was not a complete
CustomResourceStateMetricsdocument(KSM silently parses an empty config),
gauge.valueFromused an object formthat crashloops KSM,
valueFromis relative to a requiredpaththe configomitted (parses fine, emits nothing), and enum fields used
type: Gaugewith afabricated
labelsMappingkey instead ofStateSet.How
kind: CustomResourceStateMetrics/spec:.path+ relativevalueFromfor the numeric gauges;StateSetwith the CRs'real enum values for phase/verdict.
(
phase="ready"→phase="Ready").foreman_agentictask_tasks_failed: with phase a real StateSet it isexactly
phase{phase="Failed"}, nothing queried it, and it cost 630 series.every resource stamps
name; FleetNode info gainscurrent_task,version,agent_kind, andkubernetes_node(the [BUG] FLEET_NODE_NAME is set from spec.nodeName but never read, so FleetNode identity is the pod, not the node #1640 property, shipped in feat(foreman): report the Kubernetes node on FleetNode.status, not as identity #1649);role_infois new, one series per role (spec.rolesis a list — a joinedlabel would make "which nodes can review?" a substring match); AgenticTask
info gains
nodeandkind, the join a fleet table is built on.Deployment notes recorded in #1643: kube-prometheus-stack's explicit
--resourceslist excludes custom resources, and KSM needs RBAC for theforeman API group — both outside what this chart can ship.
Verification
Deployed end-to-end on a live cluster: all 14 metric families emit
(
foreman_fleetnode_phase40 series,role_info23, heartbeat 10, …) and everyquery in the shipped dashboard returns data.
helm unittest: 47/47, includingregression tests for the document wrapper, the
valueFromshape, and theStateSet conversion.
Checklist
make testpasses locallymake lintpasses locallygit commit -s) per DCOAssisted-by: Claude Code (diagnosed the four defects against a live
kube-state-metrics with a three-way probe, implemented the fixes, and verified
every metric family and dashboard query live before submitting).