fix(unixfs): customizable dag-pb order via unixfs-v1-2026 profile (IPIP-550) - #1212
Open
lidel wants to merge 7 commits into
Open
fix(unixfs): customizable dag-pb order via unixfs-v1-2026 profile (IPIP-550)#1212lidel wants to merge 7 commits into
lidel wants to merge 7 commits into
Conversation
Opt-in Data-first PBNode field ordering behind the new io.UnixFS_v1_2026 profile, per IPIP-550. Default output is unchanged: all existing profiles keep the legacy Links-first order and their CIDs. - ipld/merkledag: PBNodeFieldOrder global and a Data-first encoder used when a profile opts in (candidate for upstreaming to go-codec-dagpb) - ipld/unixfs/io: PBNodeFieldOrder profile parameter and UnixFS_v1_2026, wired through ApplyGlobals - tests assert byte-exact fixtures from the IPIP table Refs ipfs/specs#550
Temporary pin so the PBNode field ordering tests from ipfs/gateway-conformance#304 run against boxo gateway backends. Switch back to a tagged release once one ships.
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #1212 +/- ##
==========================================
- Coverage 64.34% 64.31% -0.03%
==========================================
Files 269 270 +1
Lines 27258 27270 +12
==========================================
Hits 17539 17539
- Misses 8004 8016 +12
Partials 1715 1715
... and 6 files with indirect coverage changes 🚀 New features to boost your workflow:
|
v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again.
lidel
marked this pull request as ready for review
August 27, 2026 20:06
lidel
marked this pull request as draft
August 30, 2026 11:31
Derive PBNodeDataFirst bytes from dagpb.AppendEncode by moving the trailing Data field to the front, instead of a second hand-written encoder. One encoder owns link sorting and field presence, so a decoded block with unsorted links now re-encodes sorted under both orders, and byte parity with links-first holds by construction. Unknown PBNodeFieldOrder values return an error instead of silently encoding links-first. - fieldorder.go: moveDataFirst; godoc spells out the process-wide nature of the setting, the set-once-at-startup constraint, and cites the DAG-PB strictness section - coding.go: switch on the order after AppendEncode, error on unknown - fieldorder_test.go: unknown-order test; property test over random nodes against a protowire-based links-first oracle, covering unsorted links, nil and empty Data, multi-byte length prefixes, and CIDv0, identity, and CIDv1 link hashes
Build a three-chunk file and a sharded directory from fixed inputs under UnixFS_v1_2025 and UnixFS_v1_2026 and pin the root CIDs; the file CIDs match `ipfs add --chunker=size-1000` output under each profile. Every dag-pb block in both DAGs must lead with the profile's first field, and the two DAGs must decode to the same nodes. - profile.go: UnixFS_v1_2026 written as a full literal so the test asserts each parameter instead of reconstructing the copy - profile_test.go: per-field asserts for all three profiles; ApplyGlobals subtest checks all six globals; saveAndRestoreGlobals also restores chunk.DefaultBlockSize and helpers.DefaultLinksPerBlock so applied profiles no longer leak into later tests
Every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included; single-chunk raw-leaf files keep theirs. Existing links-first directories are re-encoded when reopened through the directory API (MFS directories on their next access), a sharded root first and each child shard as it is loaded. The godocs also state why the setting is a process-wide global and that it must be applied once at startup. - profile.go: field, profile and ApplyGlobals godoc - doc.go: UnixFS_v1_2026 in the profile list, Global Settings section - CHANGELOG.md: scope, MFS re-encode, ✨ marker, PR link
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 30, 2026
Pins the ipfs/boxo#1212 branch tip: data-first bytes derive from dagpb.AppendEncode so link sorting is inherited, unknown field order values return an error, and pinned end-to-end CIDs cover the profile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ipfs/specs#550 proposes an opt-in
Data-before-LinksPBNode field order so streaming readers get HAMT parameters (hashType,fanout) before links. Boxo already reads both orders via go-codec-dagpb1, but has no way to produceData-first blocks or select the proposedunixfs-v1-2026profile.Fix
PBNodeFieldOrderparameter onUnixFSProfileand aUnixFS_v1_2026profile:UnixFS_v1_2025plusdata-firstorderingData-first bytes aredagpb.AppendEncodeoutput with the trailingDatafield moved to the front, so link sorting and field presence stay with the reference encoder; unknown order values return an errorImportant
Opt-in; no defaults change: every existing profile keeps
Links-first bytes and CIDs. The order is process-wide, applied once at startup. Opting in re-encodes existing links-first directories (new CIDs) on their next access and covers non-UnixFSProtoNodeencodes too; both are documented onmerkledag.DefaultPBNodeFieldOrder.Testing
ipfs addoutput per profile;-race/-shufflecleanFootnotes
DAG-PB spec, Protobuf Strictness, rule 2: decoders should accept either
PBNodefield order, "as IPFS data exists in both forms". ↩