Skip to content

fix(spec): tolerate DuckDB manifest entry schemas#2791

Open
fm100 wants to merge 1 commit into
apache:mainfrom
fm100:tolerate-malformed-manifest
Open

fix(spec): tolerate DuckDB manifest entry schemas#2791
fm100 wants to merge 1 commit into
apache:mainfrom
fm100:tolerate-malformed-manifest

Conversation

@fm100

@fm100 fm100 commented Jul 9, 2026

Copy link
Copy Markdown

Which issue does this PR close?

  • Closes #.

What changes are included in this PR?

This PR makes manifest metadata parsing more tolerant of manifests written by affected versions of the DuckDB Iceberg extension.

According to the Iceberg spec, the Avro manifest metadata schema key should contain the table schema at the time the manifest was written. A DuckDB Iceberg extension bug wrote the manifest entry schema into that key instead of the table schema. DuckDB fixed this in duckdb/duckdb-iceberg#1020, but that fix has not been released yet.

Even after a fixed DuckDB Iceberg extension release is available, tables created by the buggy versions can still contain manifests with the bad metadata. Reopening those tables with a newer writer does not automatically rewrite existing manifest files, so readers need to tolerate the malformed metadata to read those existing tables.

For unpartitioned manifests, Iceberg Rust can still read the manifest entries without the table schema. This PR falls back to an empty schema when:

  • the manifest metadata schema value cannot be parsed as an Iceberg table schema
  • the manifest partition spec is empty

The fallback remains limited to unpartitioned manifests because partitioned manifests need the table schema to bind partition source fields.

Are these changes tested?

Yes. This PR adds a unit test covering manifest metadata with a DuckDB-style manifest entry schema for an unpartitioned table.

Tested with:

cargo test -p iceberg test_manifest_metadata_with_manifest_entry_schema_for_unpartitioned_table

@kevinjqliu

kevinjqliu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

In general, I think its a good idea for the parsing logic to be more tolerable, it allows us to fix forward malformed metadata (like this exact case). And we should still enforce spec conformance somewhere in the stack 😄

I'll take a look at the PR when i get some time

@kevinjqliu kevinjqliu self-requested a review July 9, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants