feat(core): rework HelmClusterAddonRepository status semantics

[drey]

Summary

HelmClusterAddonRepository status is reworked so that Ready means one thing for both repository kinds, the synchronization schedule lives in status fields instead of a condition's timestamp, and failures to read a repository are retried with an exponential backoff instead of a fixed 5-minute poll. The whole status is now derived by a pure function, Evaluate(Inputs) Decision, which the reconciler feeds and applies in a single patch. Reconciling and Stalled are added following the kstatus convention, so the resource is legible to kubectl wait, CD pipelines and dashboards. The controller image had no unit tests before this branch; it now has 60+, including a cross-check against the real sigs.k8s.io/cli-utils kstatus implementation.

Why

Four concrete problems, all visible in the code being replaced:

Ready confirmed nothing for OCI repositories. For a helm repository it mirrored the internal HelmRepository; for OCI, which has no such object at the repository level, EnsureRepositorySecrets returned a synthetic success with an empty artifact:

return OCIRepoResult{
    Artifact: &meta.Artifact{},
    Status: status.Status{ Status: metav1.ConditionTrue, Reason: helmv1alpha1.ReasonSuccess, ... },
}

Writing a secret was the only thing that had actually happened.

The schedule was stored inside a condition. isRepoSyncRequired and requeueAtSyncInterval both read Synced.lastTransitionTime, which forced a two-phase state machine inside one reconcile pass — the first phase existed only to advance that timestamp, as its own comment conceded:

// EnsureAddonCharts is a two-phase state machine: the first pass only marks
// the Synced condition Reconciling, the second pass performs the actual chart
// fetch. Run the second pass in the same reconcile ... otherwise predicates
// that ignore status-only changes would stall the scheduled sync.

No backoff and no observability of the schedule. ChartsSyncInterval was a hard-coded 5 minutes; an unreachable repository was polled at that rate forever, and neither the last successful sync nor the next scheduled one appeared anywhere in the status.

Not legible to kstatus. For a custom resource, kstatus decides on metadata.deletionTimestamp, status.observedGeneration versus metadata.generation, and the Reconciling/Stalled conditions. Without the latter two, a broken repository read as Current.

Key changes

Status API — api/v1alpha1/{conditions,helm_cluster_addon_repository}.go, crds/

• New condition types Reconciling and Stalled, plus reasons AuxiliaryResourcesFailed, CatalogUpdateFailed, AwaitingInitialSync, ProgressingWithRetry, RetriesExceeded, AuthenticationFailed, SourceNotFound, SourceRejectedRequest, InvalidRepositoryURL, UnsupportedRepositoryType.
• New status fields lastSuccessfulSyncTime, nextSyncTime, consecutiveFetchFailures; regenerated CRD, deepcopy and the Russian description mirror.
• Print columns: Last Sync and Age by default, Next Sync and the Ready message under -o wide.

Pure status evaluation — internal/reconcile/helmclusteraddonrepository/evaluate.go (new)

• Evaluate(Inputs) Decision returns the entire desired status, a RequeueAfter and an error that is only ever a cluster-write failure. No client, no clock, no randomness inside: Now and Jitter are inputs, so every expected timestamp in the tests is exact.
• Ready is an ordered switch: secrets failure → internal repository not ready → stalled → a successful fetch this pass → inherited evidence → otherwise Unknown. The "inherited evidence" arm is a deliberate latch: a transient read failure does not flip Ready to False, because installed addons keep working and only the catalog goes stale.
• Inapplicable Reconciling/Stalled are removed from the array (apimeta.RemoveStatusCondition), not set to False — matching upstream Flux and keeping the UI clean.

Scheduling and backoff — same file

• SyncInterval = 5m, MaxSyncBackoff = 1h, MaxFetchFailures = 5, SyncBackoffJitter = 0.1. Delay is min(SyncInterval * 2^(n-1), MaxSyncBackoff): 5m → 10m → 20m → 40m → 1h, then Stalled/RetriesExceeded. The base equals the normal cadence on purpose, so a broken repository is never polled more often than a healthy one.
• ShouldAttempt decides whether a pass attempts a sync at all; nextSyncTime survives restarts and leader changes.

Terminal versus transient read failures — internal/client/repository/{errors.go,helm.go,oci.go}

• New TerminalError with AsTerminal/TerminalFromStatusCode: 401/403 → AuthenticationFailed, 404 → SourceNotFound, other 4xx → SourceRejectedRequest; 5xx, DNS and connection failures stay retriable. OCI maps *transport.Error through the same table and rejects a url with no image name as InvalidRepositoryURL.
• An exhausted backoff now reports the last real cause instead of a bare timed out waiting for the condition.
• Bug fix: a single chart version that is not valid semver no longer fails the whole catalog read — it is skipped and logged, matching what the OCI client already did.

Reconcile flow and service reshaping — internal/reconcile/.../reconciler.go, internal/services/*

• The reconciler collects inputs and applies one patch via the new status.Manager.PatchStatus. The two-phase machine is gone.
• BaseRepoService.EnsureSecrets replaces two divergent call sites; HelmRepoService.EnsureInternalHelmRepository returns (InternalRepositoryState, error) — an unhealthy object is information, an API failure is the caller's problem; RepoSyncService.Sync returns SyncOutcome{Fetch, Catalog} and takes an injected client factory so the catalog write and pruning are testable.
• Deleted: HelmRepoResult, RepoSyncResult, EnsureAddonCharts, isRepoSyncRequired, isRepoSyncInProgress, EnsureRepositorySecrets, ChartsSyncInterval, requeueAtSyncInterval, and the never-called status.Manager.InitializeConditions.
• Deletion fix: the CRD's url regex (^(https?|oci)://.+$) admits strings url.Parse rejects, so a repository can acquire an unparsable url after its internal objects exist. The cleanup switch now has a default: branch, otherwise deleting such a repository orphaned the internal HelmRepository and both secrets.

Tests — from zero to 60+ in the controller image

• Table-driven coverage of every Evaluate branch, asserting the three invariants on every case; scheduling arithmetic pinned exactly; fake-client tests for secrets, internal state, catalog write and pruning, and the reconcile flow end to end.
• kstatus_test.go renders the produced status as unstructured and asks the real cli-utils for its verdict, so the contract is checked against the library rather than against our reading of it. Adds sigs.k8s.io/cli-utils (test-only usage).
• E2E: a UntilConditionAbsent helper, assertions that a healthy repository carries neither abnormal-true condition, and a stall-then-recover scenario.

Documentation — docs/README.md, docs/README.ru.md, CRD descriptions

• A diagnostics section in both languages: what each condition means and a table mapping Ready/Synced combinations to what an operator should do.

Incidental chore — internal/webhook/helmclusteraddon/webhook.go

• fmt.Errorf("...helmclusteraddon/%s owns chart claim: %w", err) had %s consuming err and %w with no argument, so the message was malformed and the cause unwrapped. It failed go vet, and go vet runs inside go test, so it blocked the whole module's test gate. Fixed in its own commit.

Review focus / risks

• Ready changed meaning with no schema signal. It used to report only the internal source object's readiness; it now reports whether the repository is usable as a whole. Anything alerting on Ready, and the Deckhouse UI, should be reviewed. Deliberately shipped as feat(core): rather than feat(core)!: — cliff.toml sets breaking_always_bump_major = true, and a jump from v0.1.0 to v1.0.0 was judged disproportionate for a change that breaks no schema and no existing manifest.
• Verify the latch is the behaviour you want. Ready=True alongside Synced=False is legal and means "installed addons keep working, the catalog is stale". It is bounded — it escalates to Ready=False/Stalled=RetriesExceeded once the backoff caps, roughly 75 minutes of failures. Known residual gap: if the internal object is unhealthy and the catalog write fails in the same pass, neither Ready nor Synced records evidence and Ready drops to Unknown until the next attempt. Closing it fully would need a dedicated lastSuccessfulFetchTime field, which was deliberately not added.
• Condition removal through the status patch. Reconciling/Stalled are removed, not set to False, which relies on client.MergeFrom producing a JSON merge patch that replaces the whole array. TestReconcileRemovesStalledOnRecovery covers it against the fake client — worth confirming once against a real API server, since a silent failure here would leave a repository reporting Failed to kstatus forever.
• Upgrade path for existing objects. On rollout every repository reconciles immediately and nextSyncTime == nil forces an attempt, so the exposure is seconds. Two things to expect anyway: an inherited Ready=True satisfies the new "evidence" definition before any read has happened under the new code, and objects whose status.observedGeneration lagged (the old status manager took the minimum across conditions) may flap True → Unknown → True once. A repository already dead at upgrade time keeps Ready=True for about 80 minutes before RetriesExceeded.
• E2E has never executed. No cluster was available while developing this; the suite compiles and passes go vet, and its first real run will be in CI. One CI-specific trap is already handled: the deliberate-stall scenario makes the controller log errors, and the e2e log watcher accumulates errors suite-wide and asserts them empty at teardown, so tests/e2e/default_config.yaml excludes that repository's name. Watch the helm lifecycle spec's duration on the first run.
• Backoff constants are hard-coded. No spec.interval; the cadence is observable through nextSyncTime but not configurable. Adding a field later is additive.
• Two repo tools are broken independently of this branch, so CI is the only real check for them: the pinned bin/golangci-lint-v2.8.0 is built with go1.25.5 and refuses its config against a newer local toolchain, and task validation:doc-changes does not compile on main (undefined: RunNoCyrillicValidation). Linting was verified with golangci-lint v2.12.2 and the repo's own config — --new-from-rev reports zero new issues in both modules.