add source/destination parity Celery task - #312
Open
maperu wants to merge 3 commits into
Open
Conversation
New `check_source_destination_parity` task emits gauges for the count of destinations without a same-label source and sources without a same-label destination, plus per-label counters so a Datadog monitor can alert on the aggregate and surface which specific labels are unpaired. Same-label parity is the contract used by `sync_update_destination` in lemur/sources/service.py: source-imported certs are auto-attached to a destination only when the labels match exactly. Drift (renamed source, destination added without its discovery counterpart) breaks that auto-attach silently. This task makes the drift observable. The task isn't wired into the Celery beat schedule here -- scheduling goes in k8s-resources/k8s/lemur/chart/config/lemur.conf.py, and only on commercial/gov (dev/sandbox intentionally has source/destination split as part of the lemur-dev isolation work).
Splits pure-observability Celery tasks (read DB, emit metrics, no state change) into their own module. lemur/common/celery.py was mixing business-logic tasks (cert rotation, source sync) with monitoring tasks; this PR starts the separation by putting the new check_source_destination_parity task in lemur/common/monitoring.py. Existing monitoring-only tasks (report_celery_last_success_metrics, report_*_task signal handlers) stay in celery.py for now -- they're referenced by the Celery beat schedule in k8s-resources by full module path, so moving them is a coordinated cross-repo change worth doing as a separate cleanup PR.
maperu
marked this pull request as ready for review
June 17, 2026 18:05
Clarify purpose of monitoring tasks in comments.
henrymwang
approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
check_source_destination_paritytolemur/common/celery.py.The task does a strict 1:1 label match between sources and destinations and emits:
lemur.parity.destinations_missing_source(gauge): count of destination labels with no matching sourcelemur.parity.sources_missing_destination(gauge): count of source labels with no matching destinationlemur.parity.unmatched_destination(counter per label)lemur.parity.unmatched_source(counter per label)The per-label counters give the Datadog monitor enough breadcrumbs to surface which specific labels are unpaired when an alert fires.
Why this matters:
sync_update_destinationinlemur/sources/service.pyauto-attaches a destination to source-imported certs only when labels match exactly. A renamed source, or a destination added without its discovery counterpart, silently breaks that auto-attach loop. This task makes the drift observable.Beat scheduling is out of scope here — that lives in
k8s-resources/k8s/lemur/chart/config/lemur.conf.pyand will be wired up only for commercial/gov (dev/sandbox intentionally has source/destination split after the lemur-dev isolation work).Part of the P0 "Lemur monitoring gaps" cleanup item.