Skip to content

Implement multiple blocking keys #6

Description

@nickteff

Context

From technical review (TECH_REVIEW_AND_STRESS_TEST_PLAN.md): a single coarse blocking key (e.g. state) often yields one huge block for core-audience data, which can cause high memory or OOM with fuzzy matching (similarity matrix per block). Users need to further partition without pre-creating a composite column.

Goal

Support multiple blocking keys so that blocks are defined by the combination of values across those columns (e.g. blocking_key=["state", "zip_code"] subdivides within state by zip). Same semantics as today for a single key when blocking_key is one string; extend to accept a list of column names.

Tasks

  • API: Accept blocking_key as Union[str, list[str]] on Matcher.match(), Matcher.match_fuzzy(), and Deduplicator equivalents. Single string = current behavior (one key). List = block by the tuple of values for those columns (e.g. (state, zip_code)).
  • Block construction: Extend _paired_blocks_by_key (or equivalent) so that when blocking_key is a list, blocks are defined by rows that match on all listed columns. Null handling: consistent with current (e.g. null in any key column → one block for the null tuple, or document).
  • Validation: Blocking key(s) must exist in both left and right; clear error if not.
  • Tests: Same results with blocking_key="state" vs no blocking when blocks align; same results with blocking_key=["state", "zip_code"] vs blocking_key="state" when zip subdivides (no cross-state matches). Deduplicator. Edge: all-null in one of the keys.
  • Docs: README and Matcher docstring: document blocking_key as str or list of str; recommend multiple keys when one key gives huge blocks (e.g. state + zip).

Reference

TECH_REVIEW_AND_STRESS_TEST_PLAN.md §1.4 (Memory & block size), §1.5 (Suggestions 1, 4), §2.5 Phase C (C2). Current implementation: single blocking_key in matcher/matcher.py and matcher/deduplicator.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions