`compact_files()` fails on any dataset containing null structs with non-nullable children — scanner null propagation breaks the read→write roundtrip

[mediamana]

Dataset.optimize.compact_files() fails with Invalid user input: The field 'x' contained null values even though the field is marked non-null in the schema on any dataset that contains at least one null struct whose child fields are declared non-nullable — even though the data on disk is perfectly valid.

The root cause is a contradiction between Lance's read path and its write validation:

1. On write, data following the "validity-first" convention is accepted: parent struct null (validity carried by the struct), children physically filled with 0.0 and declared not null. This is exactly how the GeoArrow spec mandates point columns (x/y children MUST be non-nullable, nullability is carried by the outer struct).
2. On read, the scanner applies Arrow null propagation: children of a null struct come back as nulls, the physical fillers are not surfaced.
3. Compaction is a read→write roundtrip: the writer then validates the scanner's own output against the schema, sees nulls in a non-nullable child, and rejects it.

Net effect: Lance refuses to rewrite data it just read itself. A single row with a null struct makes the whole dataset permanently un-compactable (and the same applies to any other internal rewrite path with the same roundtrip).

Reproduction (20 lines)

import pyarrow as pa
import lance

# Valid "validity-first" data: parent struct null, children physically 0.0, declared non-null
# (this is the GeoArrow point layout: x/y children are "not null" per spec)
x = pa.array([1.0, 0.0, 3.0, 0.0, 5.0], type=pa.float64())
y = pa.array([1.0, 0.0, 3.0, 0.0, 5.0], type=pa.float64())
mask = pa.array([False, True, False, True, False])  # True = parent struct is null
fields = [pa.field("x", pa.float64(), nullable=False),
          pa.field("y", pa.float64(), nullable=False)]
pos = pa.StructArray.from_arrays([x, y], fields=fields, mask=mask)
tbl = pa.table({"id": pa.array([1, 2, 3, 4, 5], type=pa.int64()), "position": pos})

path = "repro.lance"
lance.write_dataset(tbl.slice(0, 2), path, mode="create")   # write is ACCEPTED
lance.write_dataset(tbl.slice(2, 3), path, mode="append")

ds = lance.dataset(path)
back = ds.to_table()["position"].combine_chunks()
print("parent nulls:", back.null_count, "| child x nulls:", back.field(0).null_count)
# -> parent nulls: 2 | child x nulls: 2   (fillers lost, nulls propagated to child on read)

ds.optimize.compact_files()   # -> OSError: Invalid user input: The field `x` contained
                              #    null values even though the field is marked non-null

Expected behavior

compact_files() should succeed on any dataset that Lance itself accepted at write time. Either:

• the compaction path should preserve/restore the physical child values under null parents (keep the validity-first layout), or
• the writer validation should accept child nulls that are fully masked by the parent struct's validity bitmap.

Actual behavior

OSError: Invalid user input: The field 'x' contained null values even though the field is marked non-null in the schema, rust/lance-file/src/writer.rs:397

Environment

• Reproduced on pylance 6.0.0, 7.0.0 and 8.0.0 (latest) — including with a dataset created by 8.0.0 itself (pyarrow 25.0.0)
• Python 3.12/3.13, Windows and Linux x86_64, local filesystem
• Dataset format v2.x

Impact

This is particularly relevant to the ongoing geospatial work (#4482): the GeoArrow spec requires point columns to be struct<x: double not null, y: double not null> with nullability on the struct. Any real-world geo dataset has rows without a geometry → every GeoArrow-style dataset with nullable geometries becomes un-compactable. In our production fleet-telemetry deployment (geoarrow.point column, ~10-75% of rows without GPS), compaction has been failing on 100% of datasets, silently accumulating 1,000-1,500 small data files per daily dataset.

Workaround we use

Full rewrite: read the table, restore the validity-first layout (fill child nulls with 0.0, keep the parent validity bitmap), write_dataset(mode="overwrite"), recreate indices. The rewritten dataset compacts natively again — until new null-struct rows arrive.

Related

• Potential panic when writing invalid batch (nullability violated) #1936 (write-side validation of unmasked nulls — the fix there is what now rejects the scanner's own output)
• StructArray take doesn't make fields nullable apache/arrow-rs#6727 (StructArray operations don't reconcile child nullability)
• bug(python): null objects are stored/retrieved as non-null with default values lancedb/lancedb#3105 (mirror symptom: null structs read back as non-null objects with default values)