feat: DH-22441: allow Dictionary encoding for Barrage and Arrow clients

[lbooker42]

Enables Apache Arrow Dictionary Encoding support in Deephaven's Barrage / Arrow Flight wire paths, adding both writer-side emission of DictionaryBatch messages (plus dict-index columns in RecordBatch) and reader-side consumption of DictionaryBatch messages to decode dictionary-encoded columns. Dictionary encoding is a transport-only optimization for low-cardinality columns: each unique value is sent once and rows are replaced with compact integer indices (Int8/Int16/Int32/Int64), selected via the new DICTIONARY_ENCODED_* variants of ColumnEncoding — auto-detected from structural guarantees or specified explicitly through the BARRAGE_SCHEMA_ATTRIBUTE. Both snapshot and subscription paths are supported; subscription dictionaries grow via delta DictionaryBatch messages and are reset (flushed and re-accumulated) when they outgrow the table or viewport, keeping state bounded on both sides of the wire.

Changes

• Add dictionary-encoded column support end-to-end: schema generation (DictionaryEncoding metadata in BarrageUtil.schemaFromTable), writer emission (DictionaryChunkWriter producing DictionaryBatch deltas plus index buffers, with per-stream LocalDictionaryWriterState and cross-stream SharedDictionaryWriterState/SharedWriterDictionary tracked by DictionaryWriterRegistry), and reader decoding (DictionaryChunkReader expanding indices against a per-stream DictionaryReaderRegistry; DictionaryReaderValues stores decoded values in fixed power-of-two pages — configurable via the DictionaryReaderValues.pageCapacity property — for O(1), allocation-free lookup).
• Extend Barrage/Arrow readers to process DictionaryBatch IPC messages and wire dictionary registries into chunk reader factories: BarrageMessageReaderImpl (Java client and server subscription path), ArrowToTableConverter (DoPut/IPC ingestion), and WebBarrageMessageReader plus WebChunkReaderFactory (GWT/JS client, including a DictionaryWriterRegistry super-source for GWT compatibility). Writer plumbing threads a persistent DictionaryWriterRegistry through BarrageMessageWriter, BarrageMessageProducer, and ArrowFlightUtil so subscription streams accumulate delta dictionaries across cycles.
• Add unboxed per-type *DictionaryWriterValueMap classes (fastutil map plus list) and null-aware *ToIntegerCastNullAware chunk-hashing kernels, both generated via the replication framework (ReplicateBarrageUtils, ReplicateHashing).
• Add coverage across unit / integration tests — DictionaryReaderRegistryTest, DictionaryWriterRegistryTest, DictionaryWriterStateTest, dictionary cases in BarrageColumnRoundTripTest, BarrageMessageRoundTripTest, and BarrageChunkFactoryTest — plus native-client encoding tests for Go, C++, and R and a GWT integration test (DictionaryEncodedTestGwt).
• Update Barrage schema documentation with a dictionary-encoding example and guidance on index-width selection, cardinality limits (128 for Int8, 32,768 for Int16), and the delta/reset behavior.

[github-actions]

⚠️ Failed to deploy docs previews for 5fd263c

Please check the logs for more details.

[Copilot]

Pull request overview

Enables Apache Arrow Dictionary Encoding support in Deephaven’s Barrage / Arrow Flight wire paths, adding both writer-side emission of DictionaryBatch messages (plus dict-index columns in RecordBatch) and reader-side consumption of DictionaryBatch messages to decode dictionary-encoded columns.

Changes:

• Add dictionary-encoded column support end-to-end: schema generation (DictionaryEncoding), writer emission (DictionaryBatch + index buffers), and reader decoding (dictionary registry + index expansion).
• Extend Barrage/Arrow readers to process DictionaryBatch IPC messages and wire dictionary registries into chunk readers.
• Add coverage across unit / integration tests and update Barrage schema documentation with a dictionary-encoding example and guidance.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/jetty/src/test/java/io/deephaven/server/jetty/BarrageChunkFactoryTest.java	Adds Arrow Flight interop test verifying dict-encoded download behavior.
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/util/BarrageUtilTest.java	Adds schema-level tests for dictionary encodings and id assignment.
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/chunk/DictionaryWriterStateTest.java	Unit tests for per-stream dictionary state (delta + stable indices).
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/chunk/DictionaryWriterRegistryTest.java	Tests writer registry reuse and delta tracking.
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/chunk/DictionaryReaderRegistryTest.java	Tests reader registry update semantics (replace/delta/ids/nulls).
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/chunk/DictionaryChunkWriterTest.java	Tests boxing semantics (notably NaN canonicalization) for dictionary keys.
extensions/barrage/src/test/java/io/deephaven/extensions/barrage/chunk/BarrageColumnRoundTripTest.java	Adds dictionary encoding round-trip + overflow tests.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/util/BarrageUtil.java	Adds dict encoding inference from schema + dict-id assignment in schema creation; removes prior hard rejection of dictionary fields.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/util/BarrageMessageReaderImpl.java	Adds DictionaryBatch handling and per-stream dictionary registry wiring for decoding.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/util/ArrowToTableConverter.java	Adds DictionaryBatch IPC handling for Arrow-to-table conversion.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/ColumnEncoding.java	Expands dictionary encodings to explicit index-width variants + helper predicate.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DictionaryWriterState.java	Implements per-dictionary-id state tracking (value→index + deltas).
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DictionaryWriterRegistry.java	Tracks per-stream dictionary states and associated value writers.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DictionaryReaderRegistry.java	Stores decoded per-id dictionary values for expansion during reads.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DictionaryChunkWriter.java	Implements dict index serialization and delta-values chunk construction.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DictionaryChunkReader.java	Implements dict index expansion into Deephaven chunks using registry contents.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DefaultChunkWriterFactory.java	Creates DictionaryChunkWriter when Field has DictionaryEncoding.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/chunk/DefaultChunkReaderFactory.java	Creates DictionaryChunkReader when Field has DictionaryEncoding and wires dict registries/readers.
extensions/barrage/src/main/java/io/deephaven/extensions/barrage/BarrageMessageWriterImpl.java	Emits DictionaryBatch messages before each RecordBatch and wires writer registry/state.
docs/groovy/how-to-guides/data-import-export/barrage-schema.md	Documents how to apply dictionary encoding via BARRAGE_SCHEMA_ATTRIBUTE with cautions.

[Copilot]

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 64 out of 64 changed files in this pull request and generated 2 comments.

[margaretkennedy]

Documentation LGTM

[kosak]

I didn't much like the C++ changes that Claude wrote. The GetScalar() and AppendScalar() calls are very expensive (e.g. GetScalar() does a heap allocation for every time). Also I didn't like the change that Claude made that made every call to ToArrowTable() do a roundtrip through ToClientTable() in order to hide RunEndEncoded and Dictionary from the caller. Who are we to say what Arrow types the caller is prepared to handle?

It is more in line with "policy up, implementation down" to allow the caller to specify how much processing ToArrowTable should do, rather than unilaterally deciding to hide that from the caller.

I have prepared these changes (actually Claude wrote all of the first part... maybe the Claude I used is smarter than the Claude you used?) in lbooker42#6

I propose that if you like those changes, you can accept that PR into this branch; and then we can continue with merging this branch into main

[niloc132]

To confirm, unlike RLE this is opt-in, so we don't need to worry about supporting it out of the box for new releases?