feat(kad): provider record spillover

[gmelodie]

Summary

Adds kad-dht/provider-record-spillover.md, a working-draft specification for an opt-in extension to the Kademlia DHT ADD_PROVIDER flow that addresses provider record hotspots on popular keys.

Problem

Under the base kad-dht protocol, the k closest peers to a key unconditionally accumulate every ADD_PROVIDER record for it.

For popular keys, this creates an unbounded load concentration with no mechanism for a peer to decline new provider records or for advertisers to spill over to other peers. This has been a known scalability issue, including:

• libp2p/go-libp2p-kad-dht#316
• ipfs/kubo#5613

This proposal implements the “rejection + sloppy hashing / spillover” approach originally proposed in libp2p/specs#163, drawing from the Coral DHT paper:

Freedman & Mazières, 2003

What This Spec Defines

Server-side — provider record limits

A node MAY enforce maxProvidersPerKey.

Once at capacity, it MUST reject ADD_PROVIDER requests from new providers for that key, while always accepting re-advertisements from existing providers so records can continue to be refreshed.

Wire protocol

A new optional providerStatus field (field 11) is added to the ADD_PROVIDER response message with two possible values:

• accepted (0)
• rejected (1)

For backward compatibility:

• An absent providerStatus field MUST be treated as accepted.
• providerStatus is response-only.
• Advertisers MUST NOT set it in requests.
• Receivers MUST ignore it if present in a request.

Client-side — spillover algorithm

The advertiser performs the normal iterative FIND_NODE lookup, then processes the sorted candidate list in chunks of size α, from closest to farthest.

After each chunk, it counts peers that:

• accepted the provider record, or
• did not respond

toward the replication target k.

If k has not yet been reached, the advertiser continues to the next chunk (a spillover round).

The process stops when either:

• k successful placements are reached, or
• all candidates are exhausted.

Non-responding peers count as accepted to preserve compatibility with nodes that predate this extension.

Compatibility

This extension is fully backward compatible.

Nodes that do not implement the extension:

• never write providerStatus
• silently ignore providerStatus if received

No changes are made to:

• GET_PROVIDERS
• FIND_NODE
• any other message type

Related

• DHT provider records: adopt sloppiness, rejection and backtracking from Coral libp2p/specs#163
• Limit the size of provider sets libp2p/go-libp2p-kad-dht#316
• Kademlia provider record spillover libp2p/specs#719

[guillaumemichel]

Hi @gmelodie, thanks for working on this! Provider record spillover is a good idea and I believe the network would benefit from it — hotspots on popular keys are a long-standing issue (libp2p/go-libp2p-kad-dht#316). Some high-level feedback before a detailed review:

1. Not backward compatible in its current shape. Today ADD_PROVIDER is effectively fire-and-forget in the deployed network: go-libp2p-kad-dht servers send no response and clients don't read one. So the spec builds on a response message that doesn't exist in practice, and if servers started enforcing maxProvidersPerKey now, all current clients would silently lose placements without ever learning they were rejected. We need a phased rollout — ship clients that read providerStatus and implement spillover first, and only enable server-side rejection once client adoption is sufficient. Adoption takes a long time (6+ months at best), so the spec should include explicit deployment guidance. Related: since support can't be detected, every ADD_PROVIDER to a legacy server would cost a timeout during the transition — worth addressing (protocol version bump?).

2. Spec gaps: the exact response message for accept/reject isn't defined (the current spec defines success as echoing the request), there's no recommended maxProvidersPerKey, the read path isn't discussed (clients that query the closest peers directly, e.g. the accelerated DHT client, never see spilled-over records), and the suggested oldest-timeReceived eviction would let an attacker flush honest providers by rotating peer IDs — a censorship vector that doesn't exist today.

3. Process: an addition to the protocol needs an IPIP — the change should be an src/ipips/ipip-XXXX.md with the spec change under src/routing/, using this repo's conventions rather than the libp2p spec template.

4. Ownership: this is significant implementation work plus a multi-stage network transition to manage. With IPFS development at Shipyard winding down, someone would need to step up to drive the implementation and the rollout — is that something you'd be interested in owning?

[richard-ramos]

Could we define the complete ADD_PROVIDER response?
It is not clear if the server should echo the request with providerStatus added, or send a new message containing only type and providerStatus.

It would also help to define which fields must be present and what response is sent for an invalid request.

[richard-ramos]

limit-but not prevent-replication

this aint guaranteed, because a malicious node can return accepted and not store the record. In this case Silence counts as accepted. If k peers do this, then the advertiser reaches the target and does not spill over.

It probably makes sense to indicate that the protocol cannot prove that storing the provider happened, and the responses are acknowledgments and not proof of record being stored.

[richard-ramos]

I think we gotta diferentiate between missig a field in the response or not receiving any response at all:

A missing field probably means that the server side is using the old response format. However it is possible for the server to not send a response too while keeping the stream open (see here and here), meaning that if a client is waiting for a timeout, it would timeout.

Also there's the possibility of the stream closing, resetting, dial failings or writes failing.

The spec needs to define which of these outcomes count as accepted. i.e. transport failures for sure should not count as accepted

[richard-ramos]

I agree with this being wire-level backward compatible, but seems to me that we need to expand the next section a bit.

i.e.
Currently other implementations like go-libp2p-kad-dht do not return nor read ADD_PROVIDER response

This basically means that:

• go-libp2p wont detect rejection from a server that contains Field 11
• a client that supports this must wait for a timeout when contacting go-libp2p.

This should probably be mentioned.

I.e. these are the possible combinations that happen:

• Old client / Old server = Provider is published as fire and forget, record is stored
• Old client / New server = Client sends the provider and assumes succes, but server may have discarded the record
• New client / Old server = Client waits for providerStatus but old server never replies, so every request translates into a timeout
• New client / Old server = Rejection and the spillover work as intended

[richard-ramos]

Or as mentioned in one of the comments, perhaps it's worth of a new kademlia version, in case the consensus determines that this differs too much from base kademlia implementation

[richard-ramos]

There seems to be a contradiction between this and line 73

the node MUST reject the request

Does this optional eviction policy overrides line 73 that the node MUST reject the new provider?

If a node evicts an existing provider when a new one arrives, should it store the new record and return accepted or should it evict the old recod and still return rejected? (this would remove a record without adding a replacement).

Please define the precedence between a record being rejected and being evicted.

[richard-ramos]

Worht mentioning that this could let an attacker evict honest providers if they rotate their peer ID?

[richard-ramos]

This only happens when α >= k, right?