feat(format): read and write sparse structural pages

docs/src/format/file/encoding.md

@@ -327,6 +327,144 @@ The protobuf for the full zip layout describes the compression of the data buffe
 size of the control words and how many bits we have per value (for fixed-width data) or how many bits we
 have per offset (for variable-width data).
 
+### Sparse Page Layout
+
+Sparse pages require Lance 2.3. They represent flat or nested Arrow structure directly as slot-domain mappings instead
+of dense repetition and definition events. Writers emit this layout only in files declared as 2.3. The layout is
+identified only by `PageLayout`; field metadata does not identify the layout of an existing page.
+
+A domain is a layer-local integer coordinate space `[0, num_slots)`, and a slot is one element in that space. The
+outer-most domain contains the page's top-level rows. Each layer maps its parent domain to the next layer's parent
+domain, and the terminal child domain contains the leaf value slots stored in value chunks.
+
+Structural layers are ordered from outer-most to inner-most:
+
+- validity maps a nullable item or struct slot to valid or null
+- list maps non-empty parent slots to variable-size child ranges
+- fixed-size-list maps each parent slot to a child range of a fixed dimension
+
+The layer list may be empty for a flat, non-nullable leaf page. In that case the scheduling domain and
+`num_visible_items` must be equal. The explicit writer currently emits its normalized all-valid layer even when a flat
+page could use this shorter wire representation.
+
+A list slot that is valid and absent from `non_empty_positions` is an empty list. Maps use the same structural contract
+as lists. The terminal child-domain size equals `SparseLayout.num_visible_items`.
+
+`SparseLayout.num_visible_items` is the number of leaf value slots encoded in value chunks. Null leaf slots count
+because they still occupy positions in Arrow's leaf value buffer; a nullable primitive with 100 slots, including 30
+nulls, has 100 visible items. `SparseLayout.num_items` is the number of entries in the equivalent dense repetition and
+definition stream. It equals `num_visible_items` plus one structural placeholder for every list slot without children.
+The first layer's `num_slots` is the logical top-level row count used for projection.
+
+Position sets have four semantic representations: `empty`, `all`, one non-empty `range`, or an `explicit`
+delta-compressed `u64` buffer. Count sets are `empty`, one positive `constant` value, or an `explicit` compressed `u64`
+buffer. Every layer has a `SparseValiditySet` whose meaning is explicit:
+
+- `SPARSE_VALIDITY_NULL_POSITIONS`: stored positions are null and all other positions are valid
+- `SPARSE_VALIDITY_VALID_POSITIONS`: stored positions are valid and all other positions are null
+
+The unspecified validity meaning is invalid. Both polarities are part of the wire contract and have identical Arrow
+semantics after normalization.
+
+#### Writer Selection
+
+The Lance 2.3 writer selects this layout when a field sets `lance-encoding:structural-encoding=sparse`. The same request
+is an input error for earlier file versions. This metadata controls writer selection only: readers always use
+`PageLayout` to determine the layout of an encoded page and must not use field metadata for that decision.
+
+The default Lance 2.3 writer policy remains unchanged, as do explicit `miniblock` and `fullzip` requests. Writers
+normalize Arrow validity and list structure once, then use that semantic structure for either dense
+repetition/definition serialization or sparse position/count serialization. All-valid layers use null positions plus
+`empty`; all-null layers use valid positions plus `empty`. Other layers choose the validity polarity with the lower
+semantic encoded cost, with ties using null positions.
+
+Pages without a value payload keep the existing canonical `ConstantLayout`: structural-only types such as an empty
+struct, and leaf pages whose visible values are all null, do not emit `SparseLayout`. An explicitly sparse page with
+at least one non-null visible value does emit `SparseLayout`, even when all non-null values are equal. This boundary
+avoids introducing a second structural-only representation without evidence that it improves the existing constant
+encoding.
+
+#### Buffers and Selective Reads
+
+A sparse page contains the following physical buffers:
+
+| Buffer | Contents |
+| ------ | -------- |
+| 0 | Value chunk metadata, one 8-byte entry per chunk |
+| 1 | Mini-block compressed value chunks without repetition or definition levels |
+| 2+ | One buffer for each explicit position or count set, in structural-layer field order |
+
+Each value chunk metadata entry stores `(chunk_size / 8) - 1` as little-endian `u32`, followed by its visible value
+count as little-endian `u32`. Chunk sizes must be positive multiples of 8 and fit this representation. The sum of
+chunk sizes must equal buffer 1 exactly and the sum of chunk value counts must equal `num_visible_items`. A value chunk
+contains at most 32,768 visible values. `num_buffers` describes the number of value buffers inside every chunk and
+excludes the structural buffers.
+
+General-compressed sparse buffers use the existing length-prefixed LZ4 or Zstd representation and must not contain
+another general-compression wrapper. SparseLayout does not impose additional size or descriptor-complexity limits on
+otherwise representable buffers.
+
+Readers normalize structural metadata once, project requested top-level ranges through each layer, and read only value
+chunks that intersect the resulting leaf ranges. When no leaf range remains, readers rebuild offsets and validity from
+the structural plan without reading buffer 1.
+
+#### Caching and Point Reads
+
+Reader initialization loads buffer 0 and every explicit structural buffer, then validates and normalizes them into a
+cached page plan. The cached state contains parsed value-chunk descriptors and prefix offsets, decoded semantic
+position/count sets, validity, and the ordered structural layers. It does not contain value payload bytes from buffer
+1. The plan is cached per field and page and reused by later scans, range reads, and takes.
+
+After that plan is cached, reading one primitive leaf value reads only the value chunk that contains it. A cold read
+first loads the structural metadata and then the intersecting value chunk. Reading one top-level list or
+fixed-size-list value may intersect multiple leaf chunks and reads each intersecting chunk. A selection whose projected
+structure contains no leaf slots reads no value chunk.
+
+#### Validation
+
+Readers must reject malformed sparse metadata instead of inferring or repairing it. Required checks include:
+
+- physical buffer count, chunk-count bounds, and every checked offset/size range
+- first-layer row domain, adjacent parent/child domain chaining, and terminal visible-value domain
+- semantic set cardinality, explicit position ordering and bounds, and validity meaning
+- exact `num_items`
+- list non-empty positions being valid, count cardinality, positive counts, and child-count sum
+- fixed-size-list dimension and checked child-domain multiplication
+- value chunk byte/value sums, size representation and alignment, general-compression headers, descriptor buffer
+  count, and complete chunk consumption
+
+```protobuf
+%%% proto.message.SparseLayout %%%
+```
+
+```protobuf
+%%% proto.message.SparseStructuralLayer %%%
+```
+
+```protobuf
+%%% proto.message.SparseValidityLayer %%%
+```
+
+```protobuf
+%%% proto.message.SparseListLayer %%%
+```
+
+```protobuf
+%%% proto.message.SparseFixedSizeListLayer %%%
+```
+
+```protobuf
+%%% proto.message.SparseValiditySet %%%
+```
+
+```protobuf
+%%% proto.message.SparsePositionSet %%%
+```
+
+```protobuf
+%%% proto.message.SparseCountSet %%%
+```
+
 ### Constant Page Layout
 
 This layout is used when all (visible) values in the page are the same scalar value.
@@ -549,7 +687,7 @@ options. However, they can also be set in the field metadata in the schema.
 | `lance-encoding:dict-values-compression-level` | Integers (scheme dependent) | Varies by scheme | Compression level for dictionary values general compression                             |
 | `lance-encoding:general`             | `off`, `on`                          | `off`            | Whether to apply general compression.                                                   |
 | `lance-encoding:packed`              | Any string                           | Not set          | Whether to apply packed struct encoding (see above).                                    |
-| `lance-encoding:structural-encoding` | `miniblock`, `fullzip`               | Not set          | Force a particular structural encoding to be applied (only useful for testing purposes) |
+| `lance-encoding:structural-encoding` | `miniblock`, `fullzip`, `sparse`     | Not set          | Select a structural encoding; `sparse` requires Lance 2.3.                               |
 
 ### Configuration Details
 

docs/src/format/file/versioning.md

@@ -5,10 +5,10 @@ major number is changed when the file format itself is modified while the minor
 strategy is modified. Newer versions will typically have better performance and compression but may not be readable
 by older versions of Lance.
 
-In addition, the `next` alias points to an unstable format version and should not be used for production use cases.
-Breaking changes could be made to unstable encodings and that would mean that files written with these encodings are
-no longer readable by any newer versions of Lance. The `next` version should only be used for experimentation and
-benchmarking upcoming features.
+Any version explicitly labeled unstable, including the current 2.3 format and the `next` alias, should not be used for
+production use cases. Unstable formats have no compatibility guarantee: breaking encoding changes may make files
+written by one Lance build unreadable by later builds. They should only be used for experimentation and benchmarking
+upcoming features.
 
 The `stable` and `next` aliases are resolved by the specific Lance release you are using. During a format rollout
 (for example, 2.3), prefer explicit version pinning for deterministic behavior across environments.
@@ -21,7 +21,7 @@ The following values are supported:
 | 2.0            | 0.16.0                | Any                   | Rework of the Lance file format that removed row groups and introduced null support for lists, fixed size lists, and primitives |
 | 2.1            | 0.38.1                | Any                   | Enhances integer and string compression, adds support for nulls in struct fields, and improves random access performance with nested fields. |
 | 2.2            | None                  | Any                   | Adds support for newer nested type/encoding capabilities (including map support) and 2.2-era storage features. |
-| 2.3 (unstable) | None                  | Any                   | Adds experimental encodings for upcoming features. |
+| 2.3 (unstable) | None                  | Unspecified           | Adds sparse structural pages and other experimental encodings. |
 | legacy         | N/A                   | N/A                   | Alias for 0.1 |
 | stable         | N/A                   | N/A                   | Alias for the default version for new datasets in the Lance release you are running. |
 | next           | N/A                   | N/A                   | Alias for the latest unstable version in the Lance release you are running.|

protos/encodings_v2_1.proto

@@ -147,6 +147,128 @@ message FullZipLayout {
   repeated RepDefLayer layers = 8;
 }
 
+// A layout used for sparse flat or nested pages where Arrow structure is represented directly
+// in layer-local slot domains instead of as dense repetition / definition events.
+//
+// Structural layers are ordered from outer-most to inner-most. Values remain mini-block
+// compressed and are split into independently readable chunks.
+message SparseLayout {
+  // Description of the compression of values.
+  CompressiveEncoding value_compression = 1;
+  // Number of value buffers in each mini-block chunk. This does not include structural buffers.
+  uint64 num_buffers = 2;
+  // Number of entries in the equivalent dense repetition / definition stream. This equals
+  // num_visible_items plus one structural placeholder for every list slot without children.
+  // Null leaf slots count as visible items because they still occupy positions in Arrow's
+  // leaf value buffer. For example, a nullable primitive with 100 slots, 30 of them null,
+  // has num_items = num_visible_items = 100.
+  uint64 num_items = 3;
+  // Number of leaf value slots encoded in the value chunks, including null leaf slots.
+  uint64 num_visible_items = 4;
+  // If true, chunk-local value buffer sizes use u32. Otherwise they use u16.
+  bool has_large_chunk = 5;
+  // Structural layers ordered from outer-most to inner-most. This may be empty for a flat,
+  // non-nullable leaf page whose scheduling domain equals num_visible_items.
+  repeated SparseStructuralLayer structural_layers = 6;
+}
+
+// A domain is a layer-local integer coordinate space [0, num_slots). A slot is one
+// element in that space. The outer-most domain is the page's top-level rows; each
+// layer's child domain is the next layer's parent domain, and the terminal child
+// domain contains num_visible_items leaf value slots.
+message SparseStructuralLayer {
+  // Exactly one layer kind is required.
+  oneof layer {
+    SparseValidityLayer validity = 1;
+    SparseListLayer list = 2;
+    SparseFixedSizeListLayer fixed_size_list = 3;
+  }
+}
+
+message SparseValidityLayer {
+  // Number of nullable item or struct slots in this layer's parent and child domain.
+  uint64 num_slots = 1;
+  // Validity for the slots in this layer.
+  SparseValiditySet validity = 2;
+}
+
+message SparseListLayer {
+  // Number of list, large-list, or map slots in this layer's parent domain.
+  uint64 num_slots = 1;
+  // Number of slots in this layer's child domain.
+  uint64 num_child_slots = 2;
+  // Non-empty parent slots. Valid parent slots absent from this set are empty lists.
+  SparsePositionSet non_empty_positions = 3;
+  // Positive child counts corresponding one-for-one with non_empty_positions.
+  SparseCountSet counts = 4;
+  // Validity for the parent slots in this layer.
+  SparseValiditySet validity = 5;
+}
+
+message SparseFixedSizeListLayer {
+  // Number of fixed-size-list slots in this layer's parent domain.
+  uint64 num_slots = 1;
+  // Number of children per parent slot. The child domain has num_slots * dimension slots.
+  uint64 dimension = 2;
+  // Validity for the parent slots in this layer.
+  SparseValiditySet validity = 3;
+}
+
+message SparseValiditySet {
+  enum Meaning {
+    SPARSE_VALIDITY_UNSPECIFIED = 0;
+    // Stored positions are null; all other positions are valid.
+    SPARSE_VALIDITY_NULL_POSITIONS = 1;
+    // Stored positions are valid; all other positions are null.
+    SPARSE_VALIDITY_VALID_POSITIONS = 2;
+  }
+
+  Meaning meaning = 1;
+  SparsePositionSet positions = 2;
+}
+
+message SparsePositionEmpty {}
+
+message SparsePositionAll {}
+
+message SparsePositionRange {
+  uint64 start = 1;
+  uint64 length = 2;
+}
+
+message SparsePositionSet {
+  oneof positions {
+    // Delta-compressed u64 positions. Cardinality is num_positions.
+    CompressiveEncoding explicit = 1;
+    // One contiguous, non-empty range.
+    SparsePositionRange range = 2;
+    // Every position in the domain.
+    SparsePositionAll all = 3;
+    // No positions in the domain.
+    SparsePositionEmpty empty = 4;
+  }
+  // Semantic cardinality of this set.
+  uint64 num_positions = 5;
+}
+
+message SparseCountEmpty {}
+
+message SparseCountConstant {
+  // Child count shared by every non-empty list slot.
+  uint64 value = 1;
+}
+
+message SparseCountSet {
+  oneof counts {
+    // Compressed u64 child counts. Cardinality comes from the containing position set.
+    CompressiveEncoding explicit = 1;
+    // One positive child count shared by every non-empty list slot.
+    SparseCountConstant constant = 2;
+    // No counts; valid only when there are no non-empty list slots.
+    SparseCountEmpty empty = 3;
+  }
+}
+
 // A layout used for pages where all (visible) values are the same scalar value.
 //
 // This generalizes the prior AllNullLayout semantics for file_version >= 2.2.
@@ -206,6 +328,8 @@ message PageLayout {
     // A layout where large binary data is encoded externally
     // and only the descriptions are put in the page
     BlobLayout blob_layout = 4;
+    // A sparse structural layout. This variant requires file version 2.3 or later.
+    SparseLayout sparse_layout = 5;
   }
 }
 

protos/file2.proto

@@ -207,4 +207,4 @@ message ColumnMetadata {
 //
 // This file format is extremely minimal.  It is a building block for
 // creating more useful readers and writers and not terribly useful by itself.
-// Other protobuf files will describe how this can be extended.
+// Other protobuf files will describe how this can be extended.

rust/lance-encoding/src/constants.rs

@@ -55,6 +55,8 @@ pub const STRUCTURAL_ENCODING_META_KEY: &str = "lance-encoding:structural-encodi
 pub const STRUCTURAL_ENCODING_MINIBLOCK: &str = "miniblock";
 /// Value for fullzip structural encoding
 pub const STRUCTURAL_ENCODING_FULLZIP: &str = "fullzip";
+/// Value for sparse structural encoding
+pub const STRUCTURAL_ENCODING_SPARSE: &str = "sparse";
 
 // Byte stream split metadata keys
 /// Metadata key for byte stream split encoding configuration

rust/lance-encoding/src/decoder.rs

@@ -2717,8 +2717,9 @@ pub trait DecodeArrayTask: Send {
 
 impl DecodeArrayTask for Box<dyn StructuralDecodeArrayTask> {
     fn decode(self: Box<Self>) -> Result<(ArrayRef, u64)> {
-        StructuralDecodeArrayTask::decode(*self)
-            .map(|decoded_array| (decoded_array.array, decoded_array.data_size))
+        let decoded_array = StructuralDecodeArrayTask::decode(*self)?;
+        decoded_array.repdef.ensure_exhausted()?;
+        Ok((decoded_array.array, decoded_array.data_size))
     }
 }
 
@@ -2740,10 +2741,7 @@ impl NextDecodeTask {
     // suggesting the user try a smaller batch size.
     #[instrument(name = "task_to_batch", level = "debug", skip_all)]
     fn into_batch(self, emitted_batch_size_warning: Arc<Once>) -> Result<(RecordBatch, u64)> {
-        let (struct_arr, data_size) = self
-            .task
-            .decode()
-            .map_err(|e| Error::internal(format!("Error decoding batch: {}", e)))?;
+        let (struct_arr, data_size) = self.task.decode()?;
         let batch = RecordBatch::from(struct_arr.as_struct());
         if data_size > BATCH_SIZE_BYTES_WARNING {
             emitted_batch_size_warning.call_once(|| {
@@ -2983,6 +2981,25 @@ mod tests {
         }
     }
 
+    struct InvalidInputDecodeTask;
+
+    impl DecodeArrayTask for InvalidInputDecodeTask {
+        fn decode(self: Box<Self>) -> Result<(ArrayRef, u64)> {
+            Err(Error::invalid_input_source("malformed sparse page".into()))
+        }
+    }
+
+    #[test]
+    fn next_decode_task_preserves_invalid_input_errors() {
+        let err = NextDecodeTask {
+            task: Box::new(InvalidInputDecodeTask),
+            num_rows: 0,
+        }
+        .into_batch(Arc::new(Once::new()))
+        .unwrap_err();
+        assert!(matches!(err, Error::InvalidInput { .. }));
+    }
+
     #[test]
     fn test_read_zero_dimension_fsl_errors_instead_of_panicking() {
         // Simulates reading a column whose stored schema declares a