feat: lock an incident after a configurable max window#6621
Open
aditya-786 wants to merge 1 commit into
Open
Conversation
A correlation rule keeps extending the same incident while matching alerts keep arriving within the timeframe, so a long-running condition can live in a single incident forever. This adds an optional per-rule max_incident_window (seconds): once an incident is older than the window, measured from its creation time, the next matching alert opens a fresh incident instead. The field is nullable and defaults off, so existing rules are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #5175
A correlation rule keeps extending the same incident for as long as matching alerts keep arriving within the rule timeframe. For a condition that trickles alerts indefinitely, that means a single incident can stay open forever and grow without bound, with no way to cap how long one incident represents.
This adds an optional per-rule max incident window (in seconds). Once an incident is older than the window, measured from its creation time, the next matching alert opens a fresh incident instead of extending the old one. The field is nullable and defaults to off, so existing rules behave exactly as before.
How it works
The decision already exists:
get_incident_for_grouping_ruleinkeep/api/core/db.pycomputes anis_incident_expiredflag, and the rules engine already creates a fresh incident (linked viasame_incident_in_the_past_id) whenever that flag is set. This change adds one more way for an incident to be considered expired.compute_incident_expiredinkeep/api/utils/incident_expiry.py, which reproduces the existing checks (resolved/merged/deleted status, or last alert older than the timeframe) and adds the new one (max_windowset andnow - creation_timepast the window). It depends only ondatetime, so it is unit-tested in isolation.get_incident_for_grouping_rulenow calls that helper. Whenmax_incident_windowisNone, the result is identical to today.Rulegains a nullablemax_incident_windowcolumn (with an additive Alembic migration), threaded throughcreate_rule/update_ruleand the rules API (maxIncidentWindow). The window is anchored on the incident creation time; resolution logic is untouched.Tests
tests/test_incident_max_window.pyunit-testscompute_incident_expired(runs underpytest --noconftest, no DB):max_window=Noneit expires only on terminal status or a last alert older than the timeframeFail-before / pass-after verified locally (neutering the new clause fails the lock test), and the existing
tests/test_rules_engine.pycorrelation/grouping suite passes unchanged against the modifiedget_incident_for_grouping_rule.Notes
Anchored on incident creation time (not first-alert), which is the simplest and most predictable semantics; happy to switch the anchor if maintainers prefer. Correlation rules are not part of the env/file provisioning path, so no provisioning changes are needed. UI wiring for the rule builder can follow; the backend enforces the window regardless of how the rule is created.