Context
From technical review (TECH_REVIEW_AND_STRESS_TEST_PLAN.md): a single coarse blocking key (e.g. state) often yields one huge block for core-audience data, which can cause high memory or OOM with fuzzy matching (similarity matrix per block). Users need to further partition without pre-creating a composite column.
Goal
Support multiple blocking keys so that blocks are defined by the combination of values across those columns (e.g. blocking_key=["state", "zip_code"] subdivides within state by zip). Same semantics as today for a single key when blocking_key is one string; extend to accept a list of column names.
Tasks
Reference
TECH_REVIEW_AND_STRESS_TEST_PLAN.md §1.4 (Memory & block size), §1.5 (Suggestions 1, 4), §2.5 Phase C (C2). Current implementation: single blocking_key in matcher/matcher.py and matcher/deduplicator.py.
Context
From technical review (TECH_REVIEW_AND_STRESS_TEST_PLAN.md): a single coarse blocking key (e.g. state) often yields one huge block for core-audience data, which can cause high memory or OOM with fuzzy matching (similarity matrix per block). Users need to further partition without pre-creating a composite column.
Goal
Support multiple blocking keys so that blocks are defined by the combination of values across those columns (e.g.
blocking_key=["state", "zip_code"]subdivides within state by zip). Same semantics as today for a single key whenblocking_keyis one string; extend to accept a list of column names.Tasks
blocking_keyasUnion[str, list[str]]onMatcher.match(),Matcher.match_fuzzy(), andDeduplicatorequivalents. Single string = current behavior (one key). List = block by the tuple of values for those columns (e.g. (state, zip_code))._paired_blocks_by_key(or equivalent) so that whenblocking_keyis a list, blocks are defined by rows that match on all listed columns. Null handling: consistent with current (e.g. null in any key column → one block for the null tuple, or document).blocking_key="state"vs no blocking when blocks align; same results withblocking_key=["state", "zip_code"]vsblocking_key="state"when zip subdivides (no cross-state matches). Deduplicator. Edge: all-null in one of the keys.blocking_keyas str or list of str; recommend multiple keys when one key gives huge blocks (e.g. state + zip).Reference
TECH_REVIEW_AND_STRESS_TEST_PLAN.md §1.4 (Memory & block size), §1.5 (Suggestions 1, 4), §2.5 Phase C (C2). Current implementation: single
blocking_keyinmatcher/matcher.pyandmatcher/deduplicator.py.