Sunset x-ipfs-path header and unsupported characters

[achingbrain]

HTTP header values can only include characters from a limited set, notably not the full unicode smorgasbord.

The HTTP gateway spec doesn't say what to do when the x-ipfs-path header would end up containing these characters.

Should they be percent-escaped? Or should the header be omitted entirely?

[lidel]

I'd lean towards percent-escaping, but interop is tricky.
It may break existing users who do not percent-decode, but still better than skipping I think?

[achingbrain]

Percent encoding would be the least lossy way of doing it, though it may disrupt clients. That said if you try to set a Headers value with an invalid character it'll through so clients are already disrupted.

Percent encoding can also inflate header value lengths - what happens as the header value gets longer? There's no header value limit in the HTTP spec but Apache limits header values to 8Kb, Tomcat limits the whole header to 8Kb. More modern servers allow more.

[lidel]

I've been thinking about it, looked at how this behaves today and I think we should percent-encode the whole value.

IIUC current behavior is already broken is silent ways for subser of non-ASCII paths 🙈 . boxo/Kubo put the decoded path in the header, so non-ASCII goes out as raw UTF-8 bytes, and JS fetch reads them back as garbled latin-1. Go's net/http also replaces control characters like %0A with a space, so data is lost. Nobody gets correct values for non-ASCII paths today, which means changing the format breaks less than it seems.

Proposal: the header carries the path in the same form as it appears in a URL.

• gateways SHOULD percent-encode the value per RFC 3986 (UTF-8 bytes; a literal % becomes %25)
• clients SHOULD percent-decode it before comparing or displaying
• gateways MAY omit the header if the encoded value is too long for common header size limits (encoding grows non-ASCII up to 3x, and 8K limits are still common)

Why full encoding, not partial escaping or a new format:

• plain ASCII paths do not change at all, so the common case keeps working
• it is lossless: one decode step gets the exact path back, with no guessing about which parts are encoded
• it is future-proof: any filename UnixFS allows (unicode, %, control characters) survives the round trip
• it matches the rest of the spec, which already uses RFC 3986 percent-encoding for filename*¹
• consumers I checked already cope: service-worker-gateway percent-decodes the value, and caches treat it as an opaque key

so, if we go this route, next steps:

• spec PR updating the X-Ipfs-Path (or new header like Ipfs-Uri, see https://specs.ipfs.tech/ipfs-uri/) section of the path gateway spec (with small IPIP that is copy-paste of the above "why")
• boxo: emit the escaped path (handler.go echoes the decoded contentPath.String() today)
• @helia/verified-fetch: fix: escape invalid characters in x-ipfs-path helia-verified-fetch#375, using the same encoding so there is one format on the wire
• gateway-conformance: add a fixture with a non-ASCII filename and assert the encoded value

clients can ship before specs and server impl. i guess?

Footnotes

1. Content-Disposition in path-gateway: filename* wraps UTF-8 in percent-encoding (RFC 3986, section 2.1). ↩

[achingbrain]

I think I would prefer an x-ipfs-url header and deprecate x-ipfs-path since the percent encoding is intrinsic to the format and works without us having to invent anything.

Clients can convert that to an IPFS path if they need to.

[lidel]

Agree, x-ipfs-path missing and new header is likely less break across all things.

• Given spec: ipfs:// and ipns:// URI schemes #544 and https://datatracker.ietf.org/doc/html/rfc6648 should we name it ipfs-uri and put percent-encoded ipfs:// or ipns:// URI there?
• Or.. keep value being a path, and just name it without x- prefix, ipfs-path ?

[achingbrain]

Yes, ipfs-uri header (no x-) with properly encoded URI sounds like the best path (no pun intended) forward.