Skip to content

feat(catalog/rest): add client for atomic multi-table transaction commits#2786

Draft
steveis wants to merge 3 commits into
apache:mainfrom
steveis:feat-multi-table-commit
Draft

feat(catalog/rest): add client for atomic multi-table transaction commits#2786
steveis wants to merge 3 commits into
apache:mainfrom
steveis:feat-multi-table-commit

Conversation

@steveis

@steveis steveis commented Jul 8, 2026

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Client support for the REST spec's POST /v1/{prefix}/transactions/commit (atomic multi-table commits):

  • Transaction::prepare_commit() — runs the transaction's actions against its base table and returns the resulting TableCommit without sending it, as the per-table building block. Unlike Transaction::commit, it does not refresh or retry; conflict handling across a transaction group belongs to the caller (documented).
  • CommitTransactionRequest wire type (table_changes: Vec<CommitTableRequest>) — CommitTableRequest already documented that identifier must be present for this request.
  • RestCatalog::commit_transaction(Vec<TableCommit>) — submits the group; HTTP 409 maps to a retryable CatalogCommitConflicts error, other statuses mirror update_table's handling. An empty commit list is a no-op that does not contact the server.

Kept on RestCatalog rather than the Catalog trait since multi-table atomicity is a catalog capability, not a universal guarantee — happy to discuss trait-level capability exposure as a follow-up.

Use case: a Postgres→Iceberg CDC producer committing per-cadence row deltas to many tables plus a changelog table, atomically. Validated end-to-end against Lakekeeper (which implements the endpoint): two tables, one commit, changes visible on both or neither.

Are these changes tested?

Unit tests: prepare_commit produces the expected updates/requirements without a catalog and errors on an empty transaction; mockito-based REST tests for the happy path (body carries table-changes for both tables), 409 → retryable conflict, and the empty no-op. cargo test -p iceberg --lib (1385) and cargo test -p iceberg-catalog-rest --lib (47) pass.

AI assistance disclosure

Per the Guidelines for AI-assisted Contributions: this PR was drafted with AI assistance (Anthropic's Claude, via Claude Code), then reviewed, tested, and validated by the author. cargo fmt, clippy, the full unit suites, and public-api snapshots were run locally; the endpoint was validated end-to-end against Lakekeeper.

Areas reviewers may want to focus on (our known uncertainties):

  • prepare_commit deliberately does not refresh the base table or retry; we'd like feedback on whether that contract (conflict handling belongs to the caller) matches where the transaction API is heading.
  • Status-code handling mirrors update_table; 404 is mapped to TableNotFound, though for this endpoint a 404 could also mean the catalog does not implement transactions — happy to adjust the message or kind.

steveis and others added 3 commits July 8, 2026 06:41
…mits

Implements the client side of the REST spec's
POST /v1/{prefix}/transactions/commit:

- Transaction::prepare_commit() runs a transaction's actions and returns
  the resulting TableCommit without sending it to a catalog, as the
  per-table building block of a multi-table commit.
- RestCatalog::commit_transaction(Vec<TableCommit>) submits the group in
  one CommitTransactionRequest; the catalog applies all or nothing.

CommitTableRequest already documented that identifier must be present
for CommitTransactionRequest; this supplies the missing caller. No
client-side retry: on conflict every prepared table may be stale, so
re-preparing the group is the caller's decision (409 errors are marked
retryable).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atomicity; explicit 503

Review feedback: successes return no refreshed metadata (200/204 empty
body), so callers must reload affected tables before further
transactions; 409 = did-not-apply, marked retryable; 500/502/503/504 =
commit-state-unknown, deliberately NOT retryable since resubmitting can
double-apply. 503 now has an explicit state-unknown arm instead of
falling through to the generic error path.
@steveis

steveis commented Jul 9, 2026

Copy link
Copy Markdown
Author

Pushed 9596abc addressing review feedback from Andrei Tserakhau (built the equivalent in iceberg-go):

  • Documented the 200/204 contract: success returns no refreshed metadata, so callers must reload affected tables before building further transactions.
  • Documented the error-mapping atomicity surface and added an explicit 503 arm: 409 = did-not-apply, marked retryable; 500/502/503/504 = commit-state-unknown, deliberately NOT retryable (blind resubmission can double-apply).

Design notes for reviewers: the base Catalog trait is not widened — commit_transaction is inherent on RestCatalog only. prepare_commit consumes the Transaction; unlike Go, Rust's ownership makes consume-on-prepare the safer default (a prepared transaction can't also be commit()ed), while still supporting the build-N-then-batch pattern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST catalog: client support for atomic multi-table commits (POST /v1/{prefix}/transactions/commit)

1 participant