feat: propagate file_format_version to CommitBuilder.storageFormat() - #730
Draft
ivscheianu wants to merge 1 commit into
Draft
feat: propagate file_format_version to CommitBuilder.storageFormat()#730ivscheianu wants to merge 1 commit into
ivscheianu wants to merge 1 commit into
Conversation
When file_format_version is set in Spark write options, lance-spark correctly encodes fragment data files in the requested format. However, at commit time the manifest's data_storage_format was never updated because CommitBuilder.storageFormat() was never called. Forward writeOptions.getFileFormatVersion() to commitBuilder.storageFormat() in all code paths that construct a CommitBuilder: - LanceBatchWrite.commit() (batch write Append/Overwrite) - StagedCommit.commitNewTable() and commitExistingTable() (staged catalog operations CREATE/REPLACE/CREATE_OR_REPLACE) - SparkPositionDeltaWrite (row-level UPDATE/DELETE/MERGE, Spark 3.4+3.5) - AddColumnsBackfillBatchWrite (add-columns backfill) - UpdateColumnsBackfillBatchWrite (column rewrite backfill) Add fileFormatVersion field to StagedCommitOptions so staged commit paths receive the value from the catalog's CreateTableSpec resolution. When fileFormatVersion is null (user didn't set the option), nothing changes — behavior is identical to before.
ivscheianu
marked this pull request as draft
July 29, 2026 04:46
hamersaw
self-requested a review
July 29, 2026 14:16
hamersaw
added a commit
to lance-format/lance
that referenced
this pull request
Aug 3, 2026
…ersion::from_str (#8063) Fixes #8066 The JNI `parse_storage_format` (used by `CommitBuilder.storageFormat()`) had a hand-rolled match that only accepted prefixed aliases (`"v2_1"`, `"v2.1"`) while `extract_write_params` uses `LanceFileVersion::from_str` which accepts the canonical numeric forms (`"2.1"`, `"2.2"`). This surfaced when [lance-spark#730](lance-format/lance-spark#730) started propagating `file_format_version` to `CommitBuilder.storageFormat()`, breaking some tests. **Fix:** - Replace the custom match with `name.parse::<LanceFileVersion>()`. - Extend `FromStr` to also accept the prefixed aliases so no previously valid input is rejected. - Update `CommitBuilder.storageFormat()` Javadoc. - Add tests for canonical forms, prefixed aliases, case-insensitivity, and invalid input. --------- Co-authored-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
file_format_versionis set in Spark write options, lance-spark correctly encodes fragment data files in the requested format. However, at commit timeCommitBuilder.storageFormat()was never called, so the manifest'sdata_storage_formatwas never updated. This causes lance-core'scheck_storage_versionto reject the commit.Closes #729
Depends on: lance-format/lance#8063 (lance-core must accept numeric format strings like "2.1", "2.2" in
CommitBuilder.storageFormat())Changes
Forward
writeOptions.getFileFormatVersion()tocommitBuilder.storageFormat()in all code paths that construct aCommitBuilder:LanceBatchWrite.commit()— batch write (Append/Overwrite)StagedCommit.commitNewTable()andcommitExistingTable()— staged catalog operationsSparkPositionDeltaWrite(Spark 3.4 + 3.5) — row-level UPDATE/DELETE/MERGEAddColumnsBackfillBatchWrite— add-columns backfillUpdateColumnsBackfillBatchWrite— column rewrite backfillAdd
fileFormatVersionfield toStagedCommitOptionsso staged commit paths receive the value from the catalog'sCreateTableSpecresolution.Backward Compatibility
When
fileFormatVersionis null (user didn't set the option),storageFormatis not called onCommitBuilder. Behavior is identical to before.Test Plan
All 19 existing unit tests pass.
StagedCommitOptionsTestupdated to verifygetFileFormatVersion()accessor.CI Note
The full test suite requires lance-core with the
parse_storage_formatfix from lance#8063, which extends the JNI format string parser to accept numeric strings ("2.1", "2.2") in addition to the existing prefixed variants ("v2.1", "v2_1"). Without that change, any test path that exercisesCommitBuilder.storageFormat()with a numeric string will fail with "Unknown storage format".