Prune parquet row groups using fully dictionary-encoded columns - #23851
Draft
DarkWanderer wants to merge 8 commits into
Draft
DarkWanderer wants to merge 8 commits into
DarkWanderer wants to merge 8 commits into
Conversation
Adds an opt-in `dictionary_filter_on_read` config option (default false) that uses a fully dictionary-encoded BYTE_ARRAY (Utf8/Binary) column chunk's dictionary as an exact row-group membership index. Unlike the existing bloom-filter pruning stage, this is exact rather than probabilistic, so it can prune both `IN`/`=` (value absent) and `NOT IN`/`!=` (value is the chunk's only value) predicates. A column chunk is only used if its page encoding stats prove every data page came from the dictionary (`PLAIN_DICTIONARY` or `RLE_DICTIONARY` only) -- chunks that fell back to `PLAIN` encoding partway through are ignored, since a writer's dictionary fallback means the dictionary is no longer the complete value set. This depends on a new arrow-rs API to decode a dictionary page independent of the row-by-row array reader (arrow-rs#9010), not yet released; see the following commit for the temporary patch.
…anch Switches the [patch.crates-io] block from a local filesystem path to https://github.com/DarkWanderer/arrow-rs/tree/get-dictionary, so the build is reproducible for anyone (including CI) rather than only on one machine. Still temporary -- to be removed once arrow-rs releases a version with the dictionary decode API and the dependency is bumped normally.
github-actions
Bot
added
documentation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
row_groups_pruned_dictionary was sorted after the page-index metrics in EXPLAIN output; group it with the other row-group-level pruning metrics (statistics, bloom filter) instead, and update the golden sqllogictest output accordingly. Also update the temporary Cargo.toml patch comment to point at arrow-rs PR apache#10420, which supersedes the now-closed apache#9011. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23851 +/- ##
========================================
Coverage 82.49% 82.50%
========================================
Files 1140 1141 +1
Lines 438529 439018 +489
Branches 438529 439018 +489
========================================
+ Hits 361783 362200 +417
- Misses 54874 54933 +59
- Partials 21872 21885 +13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Thanks @DarkWanderer -- do you have any benchmarks / examples showing this in action / showing how it is faster? The idea in theory sounds reasonable but it would be good to verify it in practice |
Contributor
Author
|
Thanks for feedback @alamb Here is a benchmark on synthetic dataset mimicking the shape of observability traces data. The specific property that highlights the performance gain is - TraceIDs are pseudo-random (so stats-based pruning doesn't help), but are clustered in time. I imagine it would be applicable to any "needle in haystack" scan where the search key is clustered along sorting key. Wall-clock timeCriterion,
Dictionary pruning is ~comparable to bloom filter in wall clock, but importantly, doesn't require extra writes; also, it can be used together with bloom filtering, compounding the win. More importantly, it allows better performance for "in" and "not in" scenarios which degrade bloom performance or cannot be handled at all respectively. Bytes readThis measures the efficiency gain from more precise page pruning. This win is not fully highlighted by using local SSD - but in S3-backed object-store context, I expect that reduction to have a much more pronounced effect.
Grain of salt: this dataset mimics a very specific domain I was optimizing for, I am not sure how well the wins translate to general use cases. |
Contributor
|
Thank you for this @DarkWanderer I am sorry I haven't had a chance to review this -- it sounds great. But I am totally swamped at the moment (and for the forseeable future 😢 ) |
etseidl
added a commit
to apache/arrow-rs
that referenced
this pull request
Sep 25, 2026
# Which issue does this PR close? - Closes #9010. - Supersedes #9011 (closed as stale) # Rationale for this change This change provides low-level API necessary for enabling dictionary-based pruning in DataFusion: apache/datafusion#23851 # What changes are included in this PR? - `parquet::file::metadata::dictionary::decode_dictionary_page`: decodes a BYTE_ARRAY dictionary page (Thrift header parse, decompress, PLAIN decode) into a `Utf8`/`Binary` Arrow array. - `ParquetMetaDataReader::read_column_dictionary` (sync) and `read_column_dictionary_async` (async) to fetch and decode a given row group/column's dictionary page from a `ParquetMetaData`, returning `Ok(None)` if the chunk has no dictionary page. - `ParquetRecordBatchStreamBuilder::get_row_group_column_dictionary` convenience method mirroring `get_row_group_column_bloom_filter`. # Are these changes tested? Yes: a round-trip unit test for a dictionary-encoded string column, a non-BYTE_ARRAY rejection test, and sync + async reader tests that decode a real dictionary page written through `ArrowWriter`. # Are there any user-facing changes? Yes, three new public APIs (see above). No changes to existing API. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>
# Conflicts: # Cargo.lock # Cargo.toml # datafusion/datasource-parquet/src/metrics.rs # datafusion/datasource-parquet/src/opener/mod.rs # datafusion/datasource-parquet/src/row_group_filter.rs # datafusion/physical-expr-common/src/metrics/value.rs # datafusion/proto/src/logical_plan/file_formats.rs # datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt # datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt # datafusion/sqllogictest/test_files/explain_analyze.slt # datafusion/sqllogictest/test_files/limit_pruning.slt # datafusion/sqllogictest/test_files/push_down_filter_parquet.slt
apache/arrow-rs#10420 is merged to arrow-rs main as b9b1d5005; the [patch.crates-io] block (repointed during the upstream/main merge) now tracks that branch instead of the old DarkWanderer/arrow-rs fork. Cargo.lock is re-pinned to the merged commit for every patched arrow-*/parquet crate. Review changed the final API before merging, so adapt to it: - ParquetRecordBatchStreamBuilder::get_row_group_column_dictionary was renamed to get_column_chunk_dictionary. - Dictionary pages now always decode as Binary, never Utf8, so DictionaryStatistics::insert drops its Utf8/StringArray arm. Also: - Fix a tag collision in ParquetOptions' proto message: our dictionary_filter_on_read (38) collided with upstream's newly added max_in_list_size (also 38). Move ours to the next free tag, 39, and regenerate proto-common/proto-models. - Adapt dictionary_filter.rs's test helper to reader.rs's ParquetFileReader, which upstream refactored to build its ParquetObjectReader internally rather than taking one as a field. - Migrate two more PruningPredicate::try_new call sites (deprecated since 55.0.0) to PruningPredicateBuilder, needed for a clean `-D warnings` clippy run. - Fix a FixedSizeListArray::value_offset deprecation (arrow-rs 60.0.0) in unnest.rs by switching to value_offset_at. - Add the row_groups_pruned_dictionary / bytes_processed metrics to the .slt golden files upstream's merge didn't touch (DataSourceExec lines outside the merge's conflict hunks). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…_pruning.slt Blank lines 8 and 10 of the header were missing the leading `#`, which HawkEye's license header check flagged as a conflict. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Dictionary pages are essentially free, exact indexes for
IN/NOT INqueries. Bloom filters can prove a target value is absent from a row group (useful for pruning=/IN), but they are probabilistic (non-zero false-positive rate) and cannot prove aNOT IN/!=exclusion, since that would require confirming every distinct value in the row group is excluded -- information a bloom filter doesn't retain. A fully dictionary-encoded column chunk does retain the complete set of distinct values, so it can prune exactly in both directions:IN/=when the dictionary is disjoint from the target set, andNOT IN/!=when the dictionary is a subset of the excluded set. This approach stacks with bloom filters: dictionary pruning subsumes bloom-filter pruning where it applies, and is exact (no false positives) where bloom filters are only probabilistic.What changes are included in this PR?
datafusion.execution.parquet.dictionary_filter_on_readconfig option (default
false).DictionaryStatistics(dictionary_filter.rs): an exactPruningStatisticsimpl backed by decoded dictionary values, plusis_fully_dictionary_encoded, which gates use of a chunk'sdictionary on its page encoding stats being exactly
PLAIN_DICTIONARY/RLE_DICTIONARY(a writer that fell back toPLAINpartway through no longer has a complete dictionary).RowGroupAccessPlanFilter::prune_by_dictionaryand aLoadDictionaries/PruneWithDictionariesstage in the Parquetopener's state machine, mirroring the existing bloom-filter stage.
row_groups_pruned_dictionarymetric,.sltcoverage(
parquet_dictionary_pruning.slt, plus the metric added to existingEXPLAIN ANALYZEgolden files), and a criterion benchmark comparingstatistics-only vs. bloom-filter vs. dictionary pruning.
apache/arrow-rs#10420 (apache/arrow-rs#9010), the arrow-rs API this
depends on for decoding a Parquet dictionary page, is now merged to
arrow-rs
main. The[patch.crates-io]block inCargo.tomltracksapache/arrow-rsmainin the meantime and will be removed once arelease containing it ships and the arrow/parquet version pins are
bumped to it.
Are these changes tested?
Yes: unit tests in
dictionary_filter.rsmirroringbloom_filter.rs(absent value,
INlist absent, present value,!=sole value,NOT INlist, and a PLAIN-fallback chunk correctly not pruned), new.sltcoverage showingrow_groups_pruned_dictionaryinEXPLAIN ANALYZE, and a criterion benchmark.Are there any user-facing changes?
Yes: a new opt-in config option
dictionary_filter_on_read(defaultfalse, no behavior change unless enabled) and a newrow_groups_pruned_dictionarymetric inEXPLAIN ANALYZE.🤖 Generated with Claude Code