Repository navigation
fix: merge high correlation alerts across correlation methods - #1870
Merged
fabclmnt merged 3 commits intoOct 30, 2026
Conversation
When several correlation methods have warn_high_correlations enabled, check_correlation_alerts overwrote a column's correlated fields with the result of the last method instead of merging them (the set().update() call was a no-op). A column could then lose fields flagged by an earlier method, making the alerts asymmetric (y listed x, but x did not list y).
fabclmnt
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To be able to merge a pull request, there are a few checks:
[x] the
blackstyle should be enforced (make lint)[ ] the docs should be regenerated (make docs)
[ ] the pull request should pass the tests (make test)
[ ] the examples should be generated (make examples)
Problem
When more than one correlation method has
warn_high_correlationsenabled (e.g.pearsonplus the defaultauto),check_correlation_alertsis meant to consolidate the correlated fields per column. However,set(fields).update(...)is a no-op, and the next line overwrites the entry. The last method wins for every column, so fields flagged by an earlier method are lost and the alerts become asymmetric.Example, with pearson enabled and one outlier shared by
xandy:y, xz and y~z;The alerts were
x: [z],y: [x, z],z: [x]. Soyis reported as correlated withx, butxis not reported as correlated withy.Solution
Merge each method's correlated fields into the column's list, keeping the order and without duplicates.
Tests
Added
test_correlation_alerts_are_merged_across_methodsintests/unit/test_correlations.py; it fails before the change and passes after. Rantests/unit/test_correlations.py,tests/unit/test_describe.py,tests/unit/test_output.pyandtests/issues/test_issue824.py.