Skip to content

fix(foreman): make the CustomResourceState config produce usable metrics - #1650

Merged
Defilan merged 2 commits into
defilantech:mainfrom
Defilan:feat/crs-fleet-role-and-current-task
Aug 24, 2026
Merged

fix(foreman): make the CustomResourceState config produce usable metrics#1650
Defilan merged 2 commits into
defilantech:mainfrom
Defilan:feat/crs-fleet-role-and-current-task

Conversation

@Defilan

@Defilan Defilan commented Aug 23, 2026

Copy link
Copy Markdown
Member

What

Make foreman.crs.enabled produce metrics — it has never emitted a single one —
and add the labels a fleet view actually needs.

Why

Fixes #1643. Refs #1602, #1640.

Four defects, each reproduced against a live kube-state-metrics, detailed in
#1643: the ConfigMap was not a complete CustomResourceStateMetrics document
(KSM silently parses an empty config), gauge.valueFrom used an object form
that crashloops KSM, valueFrom is relative to a required path the config
omitted (parses fine, emits nothing), and enum fields used type: Gauge with a
fabricated labelsMapping key instead of StateSet.

How

  • Wrap the config as kind: CustomResourceStateMetrics / spec:.
  • path + relative valueFrom for the numeric gauges; StateSet with the CRs'
    real enum values for phase/verdict.
  • The shipped dashboard's queries move with the enum casing
    (phase="ready"phase="Ready").
  • Drops foreman_agentictask_tasks_failed: with phase a real StateSet it is
    exactly phase{phase="Failed"}, nothing queried it, and it cost 630 series.
  • New labels so the fleet view can say what a node is and what it is doing:
    every resource stamps name; FleetNode info gains current_task, version,
    agent_kind, and kubernetes_node (the [BUG] FLEET_NODE_NAME is set from spec.nodeName but never read, so FleetNode identity is the pod, not the node #1640 property, shipped in feat(foreman): report the Kubernetes node on FleetNode.status, not as identity #1649);
    role_info is new, one series per role (spec.roles is a list — a joined
    label would make "which nodes can review?" a substring match); AgenticTask
    info gains node and kind, the join a fleet table is built on.

Deployment notes recorded in #1643: kube-prometheus-stack's explicit
--resources list excludes custom resources, and KSM needs RBAC for the
foreman API group — both outside what this chart can ship.

Verification

Deployed end-to-end on a live cluster: all 14 metric families emit
(foreman_fleetnode_phase 40 series, role_info 23, heartbeat 10, …) and every
query in the shipped dashboard returns data. helm unittest: 47/47, including
regression tests for the document wrapper, the valueFrom shape, and the
StateSet conversion.

Checklist

  • Tests added/updated
  • make test passes locally
  • make lint passes locally
  • Commit messages follow conventional commits
  • All commits are signed off (git commit -s) per DCO
  • AI assistance (if any) is disclosed above, per CONTRIBUTING.md
  • Documentation updated (if user-facing change) — values.yaml comments cover the surface

Assisted-by: Claude Code (diagnosed the four defects against a live
kube-state-metrics with a three-way probe, implemented the fixes, and verified
every metric family and dashboard query live before submitting).

foreman.crs.enabled has never worked. Enabling it yields no Foreman metrics at
all, which is why every panel in the dashboard shipped alongside it renders
empty and its three alert rules can never fire. Four defects, each verified
against a live kube-state-metrics:

1. The ConfigMap held a bare `resources:` mapping.
   --custom-resource-state-config-file expects a whole document. Given the bare
   form kube-state-metrics parses an empty config, logs NOTHING, and serves no
   custom metrics: indistinguishable from a healthy install with no CRs. Now
   wrapped in kind: CustomResourceStateMetrics / spec:.

2. Every gauge wrote `valueFrom: {path: [...]}`. The schema is []string, so
   once (1) was fixed kube-state-metrics crashlooped with "cannot unmarshal
   !!map into []string".

3. gauge.valueFrom is RELATIVE to gauge.path, and path is required. With an
   absolute valueFrom and no path the config parses and emits nothing,
   silently. Measured: absolute form 0 series, path form 10.

4. Enum status fields used type: Gauge with a `labelsMapping` block.
   labelsMapping is not a kube-state-metrics field and was ignored, and a Gauge
   value must be a number, boolean or RFC3339 string, so a Gauge over
   status.phase ("Ready") emits nothing. These are StateSets.

Because (4) moves label values from a mapped lowercase form to the CRs' actual
enum values, the dashboard's queries move with it: phase="ready" becomes
phase="Ready", verdict="incomplete" becomes "INCOMPLETE".

The same pass adds the labels the fleet view needs to be worth rendering. It
could previously list nodes but not say what any of them was, or what it was
doing:

- every resource now labels its series with the object name, so a stuck task
  can be named rather than only counted
- foreman_fleetnode_info gains current_task, version and agent_kind
- foreman_fleetnode_role_info is new, one series per role, because spec.roles
  is a list and joining it into one label would make "which nodes can review?"
  a substring match
- foreman_agentictask_info gains node (status.assignedNode) and kind, which is
  the join a fleet view is built on

Drops foreman_agentictask_tasks_failed: with phase a real StateSet it is
exactly phase{phase="Failed"}, nothing queried it, and it cost 630 series here.

Verified on a live cluster: all 14 metric families emit, and every query in the
shipped dashboard returns data.

Refs defilantech#1643

Signed-off-by: Christopher Maher <chris@mahercode.io>
The machine an agent pod runs on, as a metric label. The property itself
ships in the defilantech#1640 implementation; this exposes it to fleet dashboards.
Empty for off-cluster agents.

Refs defilantech#1640, defilantech#1643

Signed-off-by: Christopher Maher <chris@mahercode.io>
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Defilan
Defilan merged commit 1620443 into defilantech:main Aug 24, 2026
25 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 24, 2026
doonga pushed a commit to greyrock-labs/home-ops that referenced this pull request Aug 24, 2026
…mkube (0.9.19 ➔ 0.9.20) (#402)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/home-operations/charts-mirror/llmkube](https://github.com/defilantech/LLMKube) | patch | `0.9.19` → `0.9.20` |

---

### Release Notes

<details>
<summary>defilantech/LLMKube (ghcr.io/home-operations/charts-mirror/llmkube)</summary>

### [`v0.9.20`](https://github.com/defilantech/LLMKube/blob/HEAD/CHANGELOG.md#0920-2026-08-24)

[Compare Source](defilantech/LLMKube@v0.9.19...v0.9.20)

##### Features

- **foreman:** opt-in archival of task audit records and transcripts ([#&#8203;1655](defilantech/LLMKube#1655)) ([c577133](defilantech/LLMKube@c577133))
- **foreman:** report the Kubernetes node on FleetNode.status, not as identity ([#&#8203;1649](defilantech/LLMKube#1649)) ([7563e8a](defilantech/LLMKube@7563e8a))

##### Bug Fixes

- **controller:** carry Model tolerations onto the prefetch Job ([#&#8203;1622](defilantech/LLMKube#1622)) ([d0db838](defilantech/LLMKube@d0db838))
- **controller:** clear the controller's own schedulingStatus once a service is Ready ([#&#8203;1633](defilantech/LLMKube#1633)) ([315f34d](defilantech/LLMKube@315f34d))
- Empty payload.repo silently defeats the upstream-base fetch: task branches cut from a stale fork HEAD ([#&#8203;1625](defilantech/LLMKube#1625)) ([#&#8203;1626](defilantech/LLMKube#1626)) ([27a015a](defilantech/LLMKube@27a015a))
- **foreman:** carry the in-pod result extras on every Job-mode branch ([#&#8203;1657](defilantech/LLMKube#1657)) ([d1ce51c](defilantech/LLMKube@d1ce51c))
- **foreman:** free the agent's in-process slot while a Job-mode task runs ([#&#8203;1635](defilantech/LLMKube#1635)) ([0d64fbb](defilantech/LLMKube@0d64fbb))
- **foreman:** gate the make-invoked CI checks, and pin them against the workflows ([#&#8203;1642](defilantech/LLMKube#1642)) ([4e7ab7b](defilantech/LLMKube@4e7ab7b))
- **foreman:** make the CustomResourceState config produce usable metrics ([#&#8203;1650](defilantech/LLMKube#1650)) ([1620443](defilantech/LLMKube@1620443))

##### Documentation

- add ROCm host-retune runbook for >64GB on Strix Halo ([#&#8203;1387](defilantech/LLMKube#1387)) ([fff5e59](defilantech/LLMKube@fff5e59))
- fix guide instructions that fail against the shipped API ([#&#8203;1629](defilantech/LLMKube#1629)) ([fba0bf4](defilantech/LLMKube@fba0bf4))
- **proposals:** foreman run, an unattended orchestration loop ([#&#8203;1652](defilantech/LLMKube#1652)) ([df57f49](defilantech/LLMKube@df57f49))
- reframe the multi-GPU guide from an Issue [#&#8203;2](defilantech/LLMKube#2) validation plan into a deployment guide ([#&#8203;1631](defilantech/LLMKube#1631)) ([907ac8d](defilantech/LLMKube@907ac8d))
- **runbook:** correct the metal-agent memory-pressure runbook against pkg/agent ([#&#8203;1630](defilantech/LLMKube#1630)) ([f665674](defilantech/LLMKube@f665674))
- serving one model across two DGX Sparks via llama.cpp RPC ([#&#8203;1620](defilantech/LLMKube#1620)) ([7eaf04b](defilantech/LLMKube@7eaf04b))

</details>

---

### Configuration

📅 **Schedule**: (in timezone America/New_York)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC40MC4wIiwidXBkYXRlZEluVmVyIjoiNDQuNDAuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==-->

Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/402
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] foreman.crs.enabled produces no metrics at all: four defects in the CustomResourceState config

1 participant