feature: track exact file-version composition refactor

[Xuanwo]

Goal

Refactor Lance file-version handling so V1, V2.0, V2.1, V2.2, V2.3, and future formats are parallel exact versions whose behavior is composed explicitly.

Every supported stable version must remain readable and writable. This refactor must not change selector strings, default versions, persisted protobufs, manifest values, footer bytes, DataFile metadata, or existing dataset behavior.

PR #7979 remains the frozen, non-mergeable reference for the final layout. The stack below implements that layout as reviewable, independently compiling steps.

Target contract

• lance-file owns both the caller-facing LanceFileVersion selector and the exact ConcreteFileVersion identity.
• Neither type implements Ord or PartialOrd. Stable and Next resolve once at an API or recovery boundary and are never persisted.
• Persisted manifest strings, DataFile numbers, standard footer numbers, and embedded footer numbers use distinct exact codecs.
• lance-file::versions::{v1,v2_0,v2_1,v2_2,v2_3} explicitly composes each exact reader and writer from reusable mechanisms.
• lance::dataset::versions owns dataset write, read, schema, rewrite, commit, and physical-index policy.
• lance-encoding contains version-free mechanisms. Shared algorithms receive neither version type and do not reconstruct version knowledge through capability booleans.
• Sharing is allowed for mechanisms and genuinely identical execution. Version composition does not inherit from another version or from a shared default.
• One dataset manifest references one exact data-file version, and all files in that manifest must agree with it.
• Low-level Rust source compatibility is not a constraint for this stack. Workspace callers and bindings move together.

Compatibility that must remain unchanged

• The current release mappings remain stable -> v2.1 and next -> v2.3.
• V2.0 DataFile metadata remains (2, 0). The standard writer continues to emit footer (0, 3), while self-described and mini-Lance output continues to emit (2, 0); readers continue to accept both.
• An old manifest with no data_storage_format and no fragments continues to recover through release policy: the deprecated V2 writer flag uses the current Stable mapping, otherwise it resolves to V1.
• Appending to an existing dataset continues to use the manifest version and ignore a caller-provided conflicting selector. Changing that conflict into an error is a separate behavior change.
• The default TwoFileShuffler continues to write its temporary data and offsets files as V2.1 instead of adopting the currently ignored format_version argument.
• Distributed auxiliary-file merge continues to reject empty input and uses the first shard's exact version for output. The unreachable V2.0 fallback is not a compatibility contract, and this stack does not add mixed-version shard rejection.
• V2.3 is unstable. Only current-writer round-trip coverage is required; no compatibility code is added for earlier unstable revisions.

Pull request stack

Merge in order. Each PR is based on the preceding branch, compiles independently, and removes the old path when ownership moves. After a predecessor merges, retarget the next PR to main and merge current main into its branch if needed.

• feat: introduce exact file format identity and persisted codecs #7879 — Part 1: exact identity and persisted codecs
• test: lock exact file-version wire contracts #8019 — Part 2: checked-in exact wire fixtures
• refactor: make v1 the canonical legacy file owner #8020 — Part 3: V1 as the canonical legacy owner
• refactor: make encoding mechanisms version-free #8021 — Part 4: version-free encoding mechanisms
• refactor: add exact current-format writers #8022 — Part 5: exact current-format writer implementations
• refactor: activate exact current-format writers #8023 — Part 6: activate exact writer dispatch
• refactor!: compose exact current-format readers #8024 — Part 7: exact current-format reader composition
• refactor: centralize exact dataset write policy #8025 — Part 8: exact dataset write policy
• refactor: move file-version selectors to lance-file #8026 — Part 9: selector ownership in lance-file
• refactor: centralize dataset version policies #8027 — Part 10: remaining dataset version policies
• refactor: make index file versions exact #8028 — Part 11: exact index-file consumers
• ci: enforce exact file-version boundaries #8029 — Part 12: CI enforcement of version boundaries

Completion gates

• Every supported stable exact version has a checked-in read fixture and a current-writer round trip.
• V1, V2.0, V2.1, and V2.2 pass read, create, append, overwrite, rewrite/compaction, schema evolution, and physical index-output coverage; V2.3 passes current round-trip coverage.
• Tests assert manifest strings, DataFile numbers, standard footer numbers, and self-described/mini-Lance footer numbers.
• Production code contains no version ordering and no version checks outside selector resolution, persisted decoding/recovery, and declared file/dataset dispatch boundaries.
• lance-encoding imports neither LanceFileVersion nor ConcreteFileVersion.
• Rust workspace, Python storage-version tests, and Java JNI tests pass with unchanged public binding parameters and returned version strings.
• ci/check_file_version_boundaries.py --self-test and the full boundary scan pass.
• Repository formatting and full Rust, Python, and Java lint gates pass on the cumulative stack.

This issue is complete when all twelve PRs are merged and the architecture and compatibility gates pass without a stable wire-format or existing-behavior change.