fix(core): propagate force reconcile to internal sources - #70
Merged
Conversation
The force annotation reached only the HelmRelease. applyOCIRepositorySpec read it off the repository instead of the addon, and applyHelmChartSpec set no reconcile annotations at all, so forcing an addon re-ran the release against the artifact the source already had. The repository branch it replaces was a race: the addon controller watches HelmClusterAddonRepository with GenerationChangedPredicate, so an annotation change enqueues no addons, and the repository reconciler consumes the annotation right after its own attempt. An oci:// repository owns no internal source object - the artifact is pulled by a per-addon OCIRepository - so a force on it is now pushed onto those, from finish(), before the annotation is consumed. The helm:// path needs no equivalent: there the internal HelmRepository carries the request and its HelmCharts follow the re-indexed source on their own. setReconcileRequestAnnotations moves to base.go and takes a metav1.Object, replacing three copies. Signed-off-by: Ilya Drey <ilya.drey@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
reconcile.helm.deckhouse.io/forceannotation now reaches the source objects instead of only theHelmRelease. On aHelmClusterAddonit is stamped on that addon's internalOCIRepositoryorHelmChart, whichever its repository type produces. On anoci://HelmClusterAddonRepositoryit is pushed onto the internalOCIRepositoryof every addon that references the repository. A forced reconciliation therefore re-pulls the chart artifact rather than only re-running the release, which is whatdocs/EXAMPLE.mdalready describes as "a full reconciliation cycle".Why
Force on a
HelmClusterAddonmoved the release, not the source.applyHelmReleaseSpecstampedreconcile.werf.io/requestedAtandforceAtfromaddon.ForceReconcileRequired(), butapplyOCIRepositorySpecread the annotation off the repository (repo.ForceReconcileRequired()) andapplyHelmChartSpecset no reconcile annotations at all. Forcing an addon re-ran the release against the artifact the source already had; a tag overwritten in the registry was picked up only at the nextInternalRepositoryInterval.The repository branch inside
applyOCIRepositorySpecwas a race, not a mechanism. It could only fire if an addon happened to reconcile for an unrelated reason while the repository's force annotation was still present — the addon controller watchesHelmClusterAddonRepositorywithpredicate.GenerationChangedPredicate{}, so an annotation change enqueues no addons, and the repository reconciler consumes the annotation right after its own sync attempt. It is replaced here by an explicit push and removed.The asymmetry is specific to OCI. A
helm://repository owns an internalHelmRepositorythat already receives the request (helm_repo_service.go), re-indexes, and itsHelmCharts follow their source on their own. Anoci://repository owns no internal source object at all — the artifact is pulled by a per-addonOCIRepository— so nothing carried the request.Key changes
Force propagation from an addon —
internal/services/oci_repo_service.go,internal/services/chart_service.goapplyOCIRepositorySpecnow stamps onaddon.ForceReconcileRequired(); therepo.ForceReconcileRequired()branch is gone.applyHelmChartSpecstamps the same annotations, so thehelm://path behaves like the OCI one. It previously set none.reconcileForceAnnotationconsumes the annotation.Force propagation from an
oci://repository —internal/services/oci_repo_service.go,internal/reconcile/helmclusteraddonrepository/reconciler.goOCIRepoService.ForceReconcileInternalRepositories(ctx, repoName): lists addons through the existingindex.AddonRepositoryfield index and patches the annotations onto each addon's internalOCIRepository. An addon with no source yet is skipped, and so is one whose source disappears between the get and the patch — a force request must not be blocked by an addon that never built one.finish()when the pass actually attempted a synchronization, the repository is OCI, and the force annotation is still present. It runs after the status patch and before the annotation is consumed, so a failure returns an error, leaves the request in place, and retries.finish()takes the repository type as a new parameter.OCIRepositoryactually produces a new artifact revision, through the existingResourceVersionChangedPredicatewatch.Shared annotation helper —
internal/services/base.go,internal/services/release_service.gosetReconcileRequestAnnotationsmoved out ofrelease_service.go, generalized from*helmv2.HelmReleasetometav1.Object, and is now the single implementation behind all four call sites (three copies before).Tests —
internal/services/{oci_repo_service,chart_service}_test.go,internal/reconcile/helmclusteraddonrepository/reconciler_test.gochart_service_test.go; the OCI service test harness gains theAddonRepositoryindex, as does the repository reconciler harness.Review focus / risks
forceAtis probably inert on source objects.OCIRepositoryStatusandHelmChartStatusembed onlymeta.ReconcileRequestStatus(lastHandledReconcileAt), whilemeta.ShouldHandleForceRequestrequiresGetLastHandledForceRequestStatus()/lastHandledForceAt, which onlyHelmReleasehas. The annotation that does the work isrequestedAt: it wakes nelm-source-controller immediately, and the re-pull follows from the digest under the tag having changed. Both are stamped, matching what the code already did forHelmReleaseand for the old repository branch. This could not be confirmed against the controller — onlynelm-source-controller/apiis vendored, not its implementation. Worth a second opinion from someone who knows that fork.oci://repository now fans out. OneGet+Patchper addon of the repository, sequentially, and then as many artifact pulls as there are addons. On a large repository behind a rate-limited registry this is a burst that did not exist before. Confirm the blast radius is acceptable, or say if it needs throttling.ForceReconcileInternalRepositoriesusesclient.MatchingFields{index.AddonRepository: ...}; the index is registered incmd/operator-helm-controller/main.go:80. A label selector on theOCIRepositoryobjects was rejected on purpose: the internalOCIRepositorycarries only the addon name, and a newly added repository label would appear on existing objects only once their addon reconciles for some other reason — the addon has no periodic requeue on the happy path, so the gap after an upgrade would be open-ended.finish(). Verify the guardattempted && repoType == InternalOCIRepository && repo.ForceReconcileRequired()is the right condition — in particular that propagating on an attempt whose fetch failed is desired. It is deliberate and consistent with the annotation being consumed onattemptedalone, but it means a forced repository whose registry is down still nudges the addon sources.