Feat:[Draft][22715] add dictionaries as a supported group column type by Rich-T-kid · Pull Request #23187 · apache/datafusion

Rich-T-kid · 2026-06-25T13:52:52Z

Which issue does this PR close?

works towards closing feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY #22682.
replacement for Optimize Dictionary groupings #21765
- Optimize Dictionary groupings #21765 (review)

Rationale for this change

This PR introduces a specialized GroupColumn implementation for dictionary-typed columns inside GroupValuesColumn, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback.

The Implementation is only about 175+ lines of code. the remaining LOC is adding extensive test at the GroupColumn trait level as well as testing the GroupValuesColumn GroupValues trait and how it inter-opts with multi-dictionary group by's.

What changes are included in this PR?

Adds a DictionaryGroupValueBuilder struct implementing the GroupColumn trait for Dictionary-typed group-by columns, supporting a configurable subset of value types
Extends the type-check gate in GroupValuesColumn::try_new (the matches! block) to accept Dictionary(_, value_type) where value_type is already supported.
Adds schema-level support so emitted dictionary group key columns round-trip through the output schema correctly
removes casting thats done for each dictionary array in emit

Are these changes tested?

yes. a majority of this PR is test

Are there any user-facing changes?

no. this is a pure perf boost for users.

…ore inner append

Rich-T-kid · 2026-07-01T04:28:17Z

@kumarUjjawal could you run the dictionary benchmarks on this PR? Thx

Rich-T-kid · 2026-07-01T05:08:03Z

            }
        }
+        DataType::Dictionary(key_dt, value_dt) => {
+            let new_field = Field::new("", *value_dt.clone(), true);


Since this field is never read again it may be fine to ignore the name field.

should be weary of similar issues to #21765 (comment)

Rich-T-kid · 2026-07-01T13:33:07Z

@kumarUjjawal wanted to bump this 😄

geoffreyclaude · 2026-07-01T15:06:25Z

run benchmark dictionary_group_values

adriangbot · 2026-07-01T15:09:30Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856707692-778-smmmp 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (eb41915) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-01T15:21:31Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                             HEAD                                   rich-T-kid_dictionary-groupValuesColumn-impl
-----                                                                             ----                                   --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00                       1.15    869.8±5.08µs 71.9 MElem/sec    1.00  756.8±139.44µs 82.6 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00                         1.22    779.9±5.80µs 80.1 MElem/sec    1.00   636.7±43.49µs 98.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00                        1.17    809.2±5.84µs 77.2 MElem/sec    1.00  692.8±108.32µs 90.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00                      2.45      6.3±0.01ms 10.0 MElem/sec    1.00      2.6±0.04ms 24.5 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00                         1.24    800.2±6.35µs 78.1 MElem/sec    1.00    645.5±6.09µs 96.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00                        1.42    160.1±0.87µs 48.8 MElem/sec    1.00   112.7±15.66µs 69.3 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00                          1.28    104.0±1.39µs 75.1 MElem/sec    1.00     81.5±0.45µs 95.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00                         1.30    121.2±1.05µs 64.5 MElem/sec    1.00     92.9±9.63µs 84.1 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00                          1.17    109.7±1.26µs 71.2 MElem/sec    1.00    94.1±13.82µs 83.0 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00                        2.66    678.0±2.09µs 11.5 MElem/sec    1.00   255.3±12.37µs 30.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10     1.04      4.4±0.02ms 57.4 MElem/sec    1.00      4.2±0.01ms 59.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10       1.02      4.1±0.02ms 60.5 MElem/sec    1.00      4.1±0.01ms 61.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10      1.01      4.2±0.02ms 59.2 MElem/sec    1.00      4.2±0.01ms 60.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10    1.54     16.5±0.05ms 15.2 MElem/sec    1.00     10.7±0.06ms 23.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10       1.01      4.2±0.02ms 59.3 MElem/sec    1.00      4.2±0.01ms 60.2 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10      1.13    618.7±3.27µs 50.5 MElem/sec    1.00    549.9±6.40µs 56.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10        1.00    512.1±3.06µs 61.0 MElem/sec    1.00    511.2±3.10µs 61.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10       1.05    550.0±3.63µs 56.8 MElem/sec    1.00    522.5±2.60µs 59.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10        1.02    526.6±2.85µs 59.3 MElem/sec    1.00    515.0±2.45µs 60.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10      1.74   1479.0±6.20µs 21.1 MElem/sec    1.00    847.9±5.26µs 36.9 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	380.1s
Peak memory	592.6 MiB
Avg memory	94.8 MiB
CPU user	227.9s
CPU sys	14.8s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	330.1s
Peak memory	464.6 MiB
Avg memory	67.9 MiB
CPU user	225.8s
CPU sys	19.2s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-01T18:55:54Z

@codex review

kumarUjjawal · 2026-07-02T03:17:29Z

@kumarUjjawal wanted to bump this 😄

@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week.

Rich-T-kid · 2026-07-02T04:31:26Z

@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week.

@kumarUjjawal Sorry to hear that. I hope you feel better! no rush on the review!

Rich-T-kid

Revision #2

github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 25, 2026

Rich-T-kid added 5 commits June 29, 2026 18:43

introduce dictionarys as a supported group column type

e6b6dce

add schema support for dictionarys

8ce3f90

introduce high level GroupValuesColumn test

4dcc627

introduce groupColumn trait test

c3b71dd

git issues

1ffc7f0

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 2b60132 to 1ffc7f0 Compare June 30, 2026 20:23

Rich-T-kid added 4 commits June 30, 2026 16:30

introduce edge case/ regression section

3f7ff57

inital impl

d13e1a7

working implementation of dictionary for groupValuesCOlumns

14a4d40

benchmarks show perf boost over groupvaluerows, TODO:dedupe items bef…

52f620a

…ore inner append

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 3f7ff57 to e6b6dce Compare July 1, 2026 04:25

Rich-T-kid commented Jul 1, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs

Rich-T-kid commented Jul 1, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs Outdated

Rich-T-kid added 2 commits July 1, 2026 09:44

fix clippy errors & inline final builder

02ff545

add cache for arc ptr

26abc2c

trim down test

446e8d6

Rich-T-kid mentioned this pull request Jul 1, 2026

Test: add more aggregation focused dictionary sql logic test #23280

Open

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from eb41915 to 770abfe Compare July 1, 2026 15:30

trim test LOC again

243a557

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 770abfe to 243a557 Compare July 1, 2026 15:32

Rich-T-kid changed the title ~~[Draft][22715] introduce dictionarys as a supported group column type~~ Feat:[Draft][22715] add dictionaries as a supported group column type Jul 1, 2026

Rich-T-kid commented Jul 2, 2026

View reviewed changes

trim PR

7af7080

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 4ee52da to 7af7080 Compare July 2, 2026 19:18

Uh oh!

Conversation

Rich-T-kid commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

Rich-T-kid Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geoffreyclaude commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

kumarUjjawal commented Jul 2, 2026

Uh oh!

Rich-T-kid commented Jul 2, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Rich-T-kid commented Jun 25, 2026 •

edited

Loading

Rich-T-kid commented Jul 1, 2026 •

edited

Loading