EVE-k: gate storage readiness correctly and report it truthfully

[eriknordmark]

Description

On an EVE-k node, cluster storage could be blocked forever by a daemonset that is
none of Longhorn's, be declared ready before the node could actually serve a volume,
and report success either way. Three related fixes, plus the reporting that makes the
outcome visible.

Nothing but Longhorn's own daemonsets gates storage. checkLonghornReady swept
every daemonset in longhorn-system and required each to be Ready on this node. EVE's
collect-info leaves a SupportBundle agent daemonset behind there which never becomes
Ready, so a single leaked object blocked every volume on the node for as long as it
existed. Only the three expected daemonsets gate readiness now.

A running instance-manager is required before storage counts as ready. Longhorn
runs a volume's engine and replica processes inside the instance-manager pod, so the
node cannot serve any volume until one is running — and because that pod is owned by an
InstanceManager CR rather than a DaemonSet, the daemonset sweep never observed it.
Both the shell and pillar readiness paths now wait for it.

Both ready budgets absorb it. That pod pulls a ~440 MB image: measured at 8m20s and
8m41s on single-disk topologies, and over 20 minutes on two-disk ZFS, which alone
exhausted the old 20-minute budget and left nothing for the node and kubevirt checks
ahead of it. kubeapi's readiness poll is widened to 45 minutes, but only when Longhorn
is in the predicate.

Two waits run back to back before volumemgr proceeds, and only the first was widened
originally. The second — the one that blocks until Longhorn and CDI can actually serve
a volume — kept an unexplained 20-minute cap of its own, so the same contended link
could still expire it and leave every volume to defer and retry. It is now a named
bound at 30 minutes: by the time it starts the node and Longhorn are up, so what
remains is mostly CDI, whose images are what the slow half of a first boot spends its
time on.

And volumemgr says what happened. VolumeMgrStatus.Initialized was hardcoded true,
so a node whose storage never came up was indistinguishable from a healthy one. It now
reports the truth along with UnmetCondition, the gate still outstanding, and diag
surfaces it. The status is published from its own task started before the readiness
waits, so a node stuck on those waits says which gate it is on instead of publishing
nothing for tens of minutes — the waits can run the better part of an hour, and that was
exactly the window with no reporting.

PR dependencies

None. This PR supersedes #6259, whose two commits are included here: the
instance-manager gate and the widened budget. #6259 is closed in favour of this one.

They were merged because they cannot land independently: #6259 makes
checkLonghornReady end in the instance-manager gate, whose real implementation builds
a Longhorn client from the on-device kubeconfig, while this PR's two positive
checkLonghornReady tests drive that function with a fake clientset. Whichever landed
second would have turned those tests red, and CI on the first could not have warned.
Carrying both here lets the stub that resolves it (kubeapi: stub the instance-manager gate in the daemonset tests, folded into the gate commit) ship with the change that
needs it.

How to test and validate this PR

go build -tags k ./..., go vet -tags k ./kubeapi/... ./cmd/volumemgr/... ./cmd/diag/...
and go test -tags k ./kubeapi/... ./cmd/volumemgr/... ./cmd/diag/... under pkg/pillar.
The k tag is required: this code is behind //go:build k, and a build without it
compiles none of it while still reporting success. Every commit in the series is green on
its own.

New unit tests: the stray-daemonset case and the three daemonset-failure cases; the
instance-manager state mapping and its list-error path; the readiness budget selection;
the reported status for ready / unmet / still-waiting; and the diag warning's presence,
absence and pre-publication silence. evetest/tests/diag covers the summary end to end.

On an EVE-k device the observable effects are:

• cat /run/volumemgr/VolumeMgrStatus/volumemgr.json reports Initialized: false with
  an UnmetCondition naming the outstanding gate while storage is not usable, and does
  so during the wait rather than only after it resolves.
• diag prints WARNING: cluster storage not ready: <condition> in that state.
• With a leaked longhorn-support-bundle-agent daemonset present, volumes are created
  normally instead of stalling in CREATING_VOLUME.

Result in the kvm→k conversion matrix

The stray-daemonset fix, the truthful reporting and the readiness-publish task were
carried in integration image 0.0.0-newgo-allprs2-f40d7876 through the 7-leg
EVE-kvm→EVE-k conversion matrix: 7/7, every leg faster than on the two previous
green images of that line. That image also carries #5971, so it exercises the pillar
readiness paths against the Go kube-init bring-up. It is a no-regression result rather
than a targeted test of these fixes; the targeted evidence is the unit and evetest
coverage above.

Changelog notes

On EVE-k, a leaked daemonset in the longhorn-system namespace no longer prevents
volumes from being created, storage is not declared ready until the node can actually
serve a volume, and a node whose cluster storage did not come up now reports that
instead of reporting success.

PR Backports

• 17.0-stable: yes, the stray-daemonset fix in particular — a leaked SupportBundle
  daemonset blocks all volumes on affected nodes today.
• 16.0-stable / 14.5-stable / 13.4-stable: to be decided by the maintainers.

Checklist

• I've provided a proper description
• I've added the proper documentation
• I've tested my PR on amd64 device
• I've tested my PR on arm64 device
• I've written the test verification instructions
• I've set the proper labels to this PR
• PR's title follows the template
• I've checked the boxes above, or I've provided a good reason why I didn't

Reasons for the unchecked boxes: pkg/pillar/docs/volumemgr.md gains the readiness
fields, but there is no new operator knob to document beyond that. Exercised on an amd64
EVE-k node via the conversion matrix above; arm64 untested. Labels left as they are.

Relationship to #6257

This PR does not carry the orphaned-PVC reclaim (gcPVCs in
cmd/volumemgr/initialvolumestatus.go). That work is #6257's — it is the
counterpart to the stale-generation purge sweep introduced there, and it is
kept in one place so the two cannot drift apart.

The reclaim was briefly carried here and has been removed, because #6240 and
#6257 are siblings off master and both touch volumemgr: carrying it in both
means any branch integrating the two gets it twice, under different SHAs, and
has to dedup by hand. Integration branches should take it from #6257 —
#6271 is based on #6257 and delivers it transitively.

No code dependency in either direction: this PR builds and tests standalone
on master.

[codecov]

Codecov Report

❌ Patch coverage is 32.74854% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 24.72%. Comparing base (886b5d9) to head (a9f4e85).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/pillar/cmd/volumemgr/volumemgrstatus.go	27.94%	49 Missing ⚠️
pkg/pillar/cmd/diag/diag.go	10.71%	25 Missing ⚠️
pkg/pillar/cmd/volumemgr/volumemgr.go	0.00%	15 Missing ⚠️
pkg/pillar/kubeapi/longhorninstancemanager.go	50.00%	14 Missing ⚠️
pkg/pillar/cmd/volumemgr/initialvolumestatus.go	57.69%	9 Missing and 2 partials ⚠️
pkg/pillar/kubeapi/kubeapi.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6240      +/-   ##
==========================================
+ Coverage   24.27%   24.72%   +0.45%     
==========================================
  Files         512      524      +12     
  Lines       93831    95656    +1825     
==========================================
+ Hits        22777    23655     +878     
- Misses      69251    69998     +747     
- Partials     1803     2003     +200     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrewd-zededa]

Can we subscribe to longhorn readiness in the future and republish VolumeMgrStatus if there are future longhorn control plane issues?

[andrewd-zededa]

I don't see anyone using this field yet, is there a consumer coming in another PR?

[eriknordmark]

No consumer — and Initialized right above it has never had one either; nodeagent is the only subscriber and reads just RemainingSpace. The motivation is making the state available to tests and diagnosis: the status lands in /run/volumemgr/VolumeMgrStatus/volumemgr.json, so a test can ssh in and assert on it, and collect-info picks it up in the bundle.

Adding diag as a consumer was easy, so I've done it here — it prints a warning naming the outstanding gate when cluster storage didn't become usable.

Nothing reports this to the controller today: ZInfoClusterNode carries only the node_status enum and pillar doesn't populate it, so that's an eve-api change and a separate PR. Let's discuss what else we'd want in the API in this area. One thing to design around: these two fields are a one-shot startup outcome, not a live condition — volumemgr decides once after its wait and republishes the same value.