feat: add experimental on-demand pinning by ihlec · Pull Request #11252 · ipfs/kubo

ihlec · 2026-03-26T11:06:56Z

Summary

Automatically pin content when DHT provider counts fall below a configurable replication target, and unpin once
replication has been above target for a grace period.

Helps keeping critical data around, without wasting storage on overly replicated CIDs.

The feature is described in this draft: ipfs/specs#532
The feature is gated behind Experimental.OnDemandPinningEnabled.

CLI commands for ipfs pin ondemand:

add -- register CIDs for on-demand pinning
rm -- deregister and unpin
ls -- list registered CIDs

Design

Pin partitioning: the checker needs to distinguish its pins from user
pins to avoid accidental deletion. This implementation uses boxo's pin name
field ("on-demand") for that.
Storage budget: skips pinning when repo usage exceeds
StorageMax * StorageGCWatermark.
Idle timeout: DAG fetches timeout after 2 minutes
without receiving new blocks (allows large downloads while skipping dead records).
Provide after pin: the checker publishes a DHT provider record after pinning.
Sybil limitation: provider counts come from DHT queries, which are
susceptible to Sybil manipulation. Documented as a known limitation.

Configuration Parameters (TBD)

Option	Default	Description
`OnDemandPinning.ReplicationTarget`	`5`	Minimum providers in DHT (excluding self)
`OnDemandPinning.CheckInterval`	`"10m"`	How often the checker runs
`OnDemandPinning.UnpinGracePeriod`	`"24h"`	How long above target before unpinning

Tests

Unit tests for the checker and store in ondemandpin/.

Visualization of Feature

(out of scope for this CLI PR, but helpful to get the idea)

ihlec · 2026-03-31T11:59:19Z

Please focus the review on:

1. Core logic — `ondemandpin/`

ondemandpin/store.go (143 lines) — Datastore with CRUD to manage on-demand pinning CIDs.
ondemandpin/checker.go (282 lines) — Frequently queries the DHT for provider counts and pins/unpins accordingly.

2. Kubo integration — wiring and CLI

core/node/ondemandpin.go (168 lines) — translates Kubo's pinning system into a simple interface for checker.go.
core/commands/pin/ondemandpin.go (280 lines) — Standard cmds boilerplate for on-demand pinning.

*The rest of the files are configuration, glue code, and documentation.

ihlec · 2026-04-01T13:12:49Z

@guillaumemichel Please re-run the checks. I addressed the failing ones for overfull lines , typos, and changelog.

lidel · 2026-04-10T23:53:51Z

Triage note: we run out of time to review this in 0.41 iteration, but we've added it to 0.42. Thank you for your contribution and patience. 🙏

gammazero · 2026-05-12T14:58:33Z

Triage: need careful review and testing.

lidel · 2026-05-25T13:43:05Z

Triage: short on review bandwidth for v0.42, so we'd like to land this in v0.43 instead. I've rebased on latest master and moved the changelog entry to docs/changelogs/v0.43.md. Nothing for you to do; we'll continue review against v0.43 once the time comes.

ihlec · 2026-07-10T13:14:58Z

Hi @lidel , just checking in on this. Please let me know if there’s any additional documentation, context, or testing steps I can provide to help ease the review.

lidel

Thanks for waiting @ihlec. This deserved a real review a lot sooner than it got one.

Being straight with you: maintainer time on Kubo is a small shared budget this quarter, and this is a lot of code. A new package, new config, new commands, and a loop that pins and unpins on its own. That is not something I can skim, and a feature that deletes pins by itself really shouldn't be skimmed. I kept not finding the hours, and the release ran out first.

So I'm moving it to draft and off v0.43. That is not a rejection. I believe the idea is good and could be added to future Kubo, but pin ownership, the pin and unpin loop, and the name all need a decision before more code goes in. The inline comments say where I would take each one to be less brittle and respect user config. Feel free to push back on any of it, this is a first-pass set of comments, possible I misunderstood something.

Add a background checker that automatically pins content when DHT provider counts fall below a configurable replication target and unpins once enough providers exist again after a grace period. Gated behind Experimental.OnDemandPinningEnabled. New CLI commands: ipfs pin ondemand {add,rm,ls} Safety measures: - storage budget check (respects StorageMax/GCWatermark) - idle timeout on recursive DAG fetches (2 min without progress) - pin partitioning via pin name to distinguish on-demand pins from persitent pins.

…adding v0.42 changelog entry

On-demand pinning is not landing in v0.43. Review is open and the design needs changes first, so park the highlight in vFUTURE.md and restore v0.43.md to the upstream skeleton.

…structive

…nown replica number

…ilures (also parallel now)

checkAll looked up CIDs one by one and blocked the sweep on Pin, so a slow fetch delayed every other CID. Successful checks also cleared NextCheckAt, so stable CIDs were DHT-queried again every CheckInterval. Run due checks concurrently, set NextCheckAt from the outcome so deadband/grace CIDs wait longer, and stop FindProviders once the decision is known. Enqueue no longer drops when busy. Co-authored-by: Cursor <cursoragent@cursor.com>

ihlec · 2026-08-04T07:01:11Z

Hi @lidel, I pushed the CI fixes.

Resolves fakeClock error in unit test (format mismatch)

ihlec requested a review from a team as a code owner March 26, 2026 11:06

ihlec mentioned this pull request Mar 26, 2026

IPIP: On-Demand Pinning Based on DHT Provider Counts ipfs/specs#532

Draft

ihlec force-pushed the master branch 2 times, most recently from 48a25c8 to 8802724 Compare March 31, 2026 11:29

guillaumemichel mentioned this pull request Mar 31, 2026

Release 0.42 #11227

Closed

40 tasks

ihlec commented Apr 1, 2026

View reviewed changes

Comment thread core/commands/pin/ondemandpin.go Outdated

lidel added the need/maintainers-input Needs input from the current maintainer(s) label Apr 10, 2026

gammazero added exp/expert Having worked on the specific codebase is important status/blocked Unable to be worked further until needs are met labels May 12, 2026

lidel mentioned this pull request May 25, 2026

Release 0.43 #11298

Open

59 tasks

lidel requested changes Jul 14, 2026

View reviewed changes

lidel marked this pull request as draft July 14, 2026 13:15

ihlec and others added 12 commits July 22, 2026 08:58

fix: addressed failing checks by cleaning up comments, spelling, and …

1015c18

…adding v0.42 changelog entry

docs: move changelog highlight to vFUTURE

a0a150f

On-demand pinning is not landing in v0.43. Review is open and the design needs changes first, so park the highlight in vFUTURE.md and restore v0.43.md to the upstream skeleton.

fix(pin): reserve "kubo:on-demand" pin names, make ondemand rm non-de…

4815e8c

…structive

fix(ondemandpin): do not overwrite pins created during provider lookup

b653636

fix(ondemandpin): derive unpin grace period from DHT record validity

e66ba68

fix(ondemandpin): validate OnDemandPinning config at daemon start

e022855

fix(ondemandpin): key ownership on kubo:on-demand pin name only

d8d5a54

fix(ondemandpin): refuse Null routing and treat failed lookups as unk…

212f15e

…nown replica number

fix(ondemandpin): add min/max replication deadband and grace jitter

bc6fb3f

fix(ondemandpin): exponential check backoff for failing CIDs

fa13068

fix(ondemandpin): do not apply checkTimeout to Pin fetches

c3b6891

ihlec added 5 commits July 22, 2026 09:07

fix(ondemandpin): do not resurrect records removed during a check

974103c

fix(ondemandpin): more details for ls and DryRun mode

34a679a

fix(ondemandpin): make ls --live explicit about routing and lookup fa…

3066da3

…ilures (also parallel now)

docs(ondemandpin): clarify naming and ipfs-cluster differentiation

6d89f24

fix(ondemandpin): announce pins via DHTProvider.StartProviding

34f707b

ihlec force-pushed the master branch from a3ba36e to 34f707b Compare July 22, 2026 07:12

ihlec marked this pull request as ready for review July 22, 2026 07:17

Merge branch 'master' into master

674d6d1

ihlec commented Jul 30, 2026

View reviewed changes

Comment thread core/commands/pin/ondemandpin.go Outdated

ihlec commented Jul 30, 2026

View reviewed changes

Comment thread ondemandpin/store.go

ihlec and others added 3 commits August 4, 2026 08:57

fix(ondemandpin): apply go fix for loop var and time JSON tags

8298ce3

Merge branch 'master' into master

d0857a8

lidel marked this pull request as draft August 4, 2026 14:25

ihlec commented Aug 4, 2026

View reviewed changes

Comment thread ondemandpin/checker_test.go Outdated

fix(ondemandpin): make fakeClock return UTC

7771e6b

Resolves fakeClock error in unit test (format mismatch)

ihlec marked this pull request as ready for review August 4, 2026 14:52

Uh oh!

Uh oh!

Conversation

ihlec commented Mar 26, 2026

Summary

Design

Configuration Parameters (TBD)

Tests

Visualization of Feature

Uh oh!

ihlec commented Mar 31, 2026

1. Core logic — ondemandpin/

2. Kubo integration — wiring and CLI

Uh oh!

ihlec commented Apr 1, 2026

Uh oh!

Uh oh!

lidel commented Apr 10, 2026

Uh oh!

gammazero commented May 12, 2026

Uh oh!

lidel commented May 25, 2026

Uh oh!

ihlec commented Jul 10, 2026

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ihlec commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

1. Core logic — `ondemandpin/`