Problem
is_enrolled (sei_omnigent/omni/adapters/alertmanager.py) returns False → NoOp for a non-matching enrollment label, which is correct for a legitimately-unenrolled group but indistinguishable from a structural skew where the receiver's expected label key and the producer's emitted label key diverge. In that state the incident responder is silently dead-but-green: every group falls through to the human, zero auto-investigations, no error, no log, no metric.
Surfaced in the de-walle xreview (PR #245): the walle→sei-omnigent label rename created exactly this skew shape. It was closed structurally (producer + consumer renamed in one atomic deploy), but the general observability gap remains.
Proposed approach
Add a fail-loud signal when the receiver processes webhooks but is_enrolled is False for all groups over a window — e.g. an ERROR log + a metric (enrolled_groups == 0 while webhooks_received > 0), turning the silent no-op into an observable signal. A debug-level "group not enrolled (key=…)" line is the lighter first step.
Un-defer trigger (from the xreview systems lens)
Prioritize this the moment EITHER holds:
- the enrollment label key is edited independently in one repo (code vs the #1212 AM matcher) — the lockstep that currently closes the skew structurally no longer holds; or
- the receiver and its AM config move to separate Flux Kustomizations with independent reconcile — time-separability returns and the silent skew can recur on any future label change.
References
Problem
is_enrolled(sei_omnigent/omni/adapters/alertmanager.py) returnsFalse→NoOpfor a non-matching enrollment label, which is correct for a legitimately-unenrolled group but indistinguishable from a structural skew where the receiver's expected label key and the producer's emitted label key diverge. In that state the incident responder is silently dead-but-green: every group falls through to the human, zero auto-investigations, no error, no log, no metric.Surfaced in the de-walle xreview (PR #245): the
walle→sei-omnigentlabel rename created exactly this skew shape. It was closed structurally (producer + consumer renamed in one atomic deploy), but the general observability gap remains.Proposed approach
Add a fail-loud signal when the receiver processes webhooks but
is_enrolledisFalsefor all groups over a window — e.g. an ERROR log + a metric (enrolled_groups == 0 while webhooks_received > 0), turning the silent no-op into an observable signal. A debug-level "group not enrolled (key=…)" line is the lighter first step.Un-defer trigger (from the xreview systems lens)
Prioritize this the moment EITHER holds:
References
bdchatham-designs designs/sei-agentic-mesh/xreview/plt-de-walle-code.md