Skip to content

fix(core): propagate force reconcile to internal sources - #70

Merged
drey merged 1 commit into
mainfrom
fix/oci-repo-force-reconcile
Sep 4, 2026
Merged

fix(core): propagate force reconcile to internal sources#70
drey merged 1 commit into
mainfrom
fix/oci-repo-force-reconcile

Conversation

@drey

@drey drey commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

The reconcile.helm.deckhouse.io/force annotation now reaches the source objects instead of only the HelmRelease. On a HelmClusterAddon it is stamped on that addon's internal OCIRepository or HelmChart, whichever its repository type produces. On an oci:// HelmClusterAddonRepository it is pushed onto the internal OCIRepository of every addon that references the repository. A forced reconciliation therefore re-pulls the chart artifact rather than only re-running the release, which is what docs/EXAMPLE.md already describes as "a full reconciliation cycle".

Why

Force on a HelmClusterAddon moved the release, not the source. applyHelmReleaseSpec stamped reconcile.werf.io/requestedAt and forceAt from addon.ForceReconcileRequired(), but applyOCIRepositorySpec read the annotation off the repository (repo.ForceReconcileRequired()) and applyHelmChartSpec set no reconcile annotations at all. Forcing an addon re-ran the release against the artifact the source already had; a tag overwritten in the registry was picked up only at the next InternalRepositoryInterval.

The repository branch inside applyOCIRepositorySpec was a race, not a mechanism. It could only fire if an addon happened to reconcile for an unrelated reason while the repository's force annotation was still present — the addon controller watches HelmClusterAddonRepository with predicate.GenerationChangedPredicate{}, so an annotation change enqueues no addons, and the repository reconciler consumes the annotation right after its own sync attempt. It is replaced here by an explicit push and removed.

The asymmetry is specific to OCI. A helm:// repository owns an internal HelmRepository that already receives the request (helm_repo_service.go), re-indexes, and its HelmCharts follow their source on their own. An oci:// repository owns no internal source object at all — the artifact is pulled by a per-addon OCIRepository — so nothing carried the request.

Key changes

Force propagation from an addoninternal/services/oci_repo_service.go, internal/services/chart_service.go

  • applyOCIRepositorySpec now stamps on addon.ForceReconcileRequired(); the repo.ForceReconcileRequired() branch is gone.
  • applyHelmChartSpec stamps the same annotations, so the helm:// path behaves like the OCI one. It previously set none.
  • Ordering already worked: the addon reconciler ensures its source before reconcileForceAnnotation consumes the annotation.

Force propagation from an oci:// repositoryinternal/services/oci_repo_service.go, internal/reconcile/helmclusteraddonrepository/reconciler.go

  • New OCIRepoService.ForceReconcileInternalRepositories(ctx, repoName): lists addons through the existing index.AddonRepository field index and patches the annotations onto each addon's internal OCIRepository. An addon with no source yet is skipped, and so is one whose source disappears between the get and the patch — a force request must not be blocked by an addon that never built one.
  • Called from finish() when the pass actually attempted a synchronization, the repository is OCI, and the force annotation is still present. It runs after the status patch and before the annotation is consumed, so a failure returns an error, leaves the request in place, and retries.
  • finish() takes the repository type as a new parameter.
  • Addons are not woken by the annotation itself — that stays deliberately unwatched. They wake only if an OCIRepository actually produces a new artifact revision, through the existing ResourceVersionChangedPredicate watch.

Shared annotation helperinternal/services/base.go, internal/services/release_service.go

  • setReconcileRequestAnnotations moved out of release_service.go, generalized from *helmv2.HelmRelease to metav1.Object, and is now the single implementation behind all four call sites (three copies before).

Testsinternal/services/{oci_repo_service,chart_service}_test.go, internal/reconcile/helmclusteraddonrepository/reconciler_test.go

  • New chart_service_test.go; the OCI service test harness gains the AddonRepository index, as does the repository reconciler harness.
  • Each new behavior has its complement: an unannotated addon must leave the source clean, and a scheduled (unforced) repository sync must not stamp its addons' sources — a fresh timestamp on every pass would make the source controller re-pull continuously.

Review focus / risks

  • forceAt is probably inert on source objects. OCIRepositoryStatus and HelmChartStatus embed only meta.ReconcileRequestStatus (lastHandledReconcileAt), while meta.ShouldHandleForceRequest requires GetLastHandledForceRequestStatus() / lastHandledForceAt, which only HelmRelease has. The annotation that does the work is requestedAt: it wakes nelm-source-controller immediately, and the re-pull follows from the digest under the tag having changed. Both are stamped, matching what the code already did for HelmRelease and for the old repository branch. This could not be confirmed against the controller — only nelm-source-controller/api is vendored, not its implementation. Worth a second opinion from someone who knows that fork.
  • A forced oci:// repository now fans out. One Get + Patch per addon of the repository, sequentially, and then as many artifact pulls as there are addons. On a large repository behind a rate-limited registry this is a burst that did not exist before. Confirm the blast radius is acceptable, or say if it needs throttling.
  • The lookup depends on the field index. ForceReconcileInternalRepositories uses client.MatchingFields{index.AddonRepository: ...}; the index is registered in cmd/operator-helm-controller/main.go:80. A label selector on the OCIRepository objects was rejected on purpose: the internal OCIRepository carries only the addon name, and a newly added repository label would appear on existing objects only once their addon reconciles for some other reason — the addon has no periodic requeue on the happy path, so the gap after an upgrade would be open-ended.
  • Placement inside finish(). Verify the guard attempted && repoType == InternalOCIRepository && repo.ForceReconcileRequired() is the right condition — in particular that propagating on an attempt whose fetch failed is desired. It is deliberate and consistent with the annotation being consumed on attempted alone, but it means a forced repository whose registry is down still nudges the addon sources.
  • No e2e coverage. All new tests are unit tests over the fake client, which does not model nelm-source-controller's reaction to the annotations. The end-to-end claim — force an addon, get a re-pull of an overwritten tag — has not been exercised in a cluster.

The force annotation reached only the HelmRelease. applyOCIRepositorySpec
read it off the repository instead of the addon, and applyHelmChartSpec set
no reconcile annotations at all, so forcing an addon re-ran the release
against the artifact the source already had.

The repository branch it replaces was a race: the addon controller watches
HelmClusterAddonRepository with GenerationChangedPredicate, so an annotation
change enqueues no addons, and the repository reconciler consumes the
annotation right after its own attempt.

An oci:// repository owns no internal source object - the artifact is pulled
by a per-addon OCIRepository - so a force on it is now pushed onto those,
from finish(), before the annotation is consumed. The helm:// path needs no
equivalent: there the internal HelmRepository carries the request and its
HelmCharts follow the re-indexed source on their own.

setReconcileRequestAnnotations moves to base.go and takes a metav1.Object,
replacing three copies.

Signed-off-by: Ilya Drey <ilya.drey@flant.com>
@drey
drey merged commit 29e0ff4 into main Sep 4, 2026
5 of 6 checks passed
@drey
drey deleted the fix/oci-repo-force-reconcile branch September 4, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant