Skip to content

A record its route never asked for no longer leaves a checkpoint hole; copy reservations stay per pass - #1470

Merged
kriszyp merged 18 commits into
mainfrom
fix/checkpoint-hole-coverage-gates
Nov 21, 2026
Merged

kriszyp merged 18 commits into
mainfrom
fix/checkpoint-hole-coverage-gates

Conversation

@kriszyp

@kriszyp kriszyp commented Nov 19, 2026 •

Copy link
Copy Markdown
Member
The daughter sets unasked crates aside; her brother flags a cracked crate the route owed.

⊙ Problem

Checkpoint bookkeeping kept origins out of checkpoint resume longer than needed. A record the receive route filters by design (an excluded or non-replicating table, a local-only record) became a checkpoint hole (harper-pro#1213). A copy pass's reservation also carried other passes' bounds, and settling could write a new one (harper-pro#1219).

🤔 Keep a sharding-restricted route's floor relayable (harper-pro#1214 not applied)

harper-pro#1214 asked to store relayable only with full coverage. I kept main's behavior: a restricted route relays a floor with no generation, and that floor clears a downstream route's level-2 qualifiers. Suppressing it kept the downstream node checkpoint-grade for an origin it no longer fully received.

💡 Solution

⚖️ Alternatives

  • Match each dropped record to its request — a frame names no request.
  • Keep copy fills behind more receiver checks — a copy does not say which tables or start the sender walked, so nothing proves the lost record came back. A sender attestation is harper-pro#1464.

🤔 Owner decision: per-socket request scope, not "every filtered row is a hole"

The capped framing review preferred a hole for every table-filtered row, exempting only local-only rows. That keeps a selective receiver's origin non-checkpoint-grade until a rebuild whenever an older peer forwards an excluded table: the harper-pro#1213 case. Unfixed: the reservation cost below, a brittle closing-socket fixture; pre-existing majors are in harper-pro#1462 and harper-pro#1465.

🤔 Each copy pass reserves for itself instead of borrowing another pass's reservation

A borrowed bound disappears when its owner settles. Cost: when another pass reserved further ahead, this pass commits its own reservations, about one extra awaited commit each time a row's distance past the copy start doubles beyond 60 s. Not benchmarked.

🔧 Changes

✅ Verification

  • 15 new or rewritten fake-socket cases drive the real replicateOverWS handler. 12 fail on main's build; 3 pin behavior main shares (an owed drop's hole stands, a redeclared table still holes, a restricted route stays relayable). The closing-socket case also fails with only its guard removed.

  • npm run test:unit: 2848 passing, 0 failing; lint:required and typecheck pass.

  • CI's cluster shards passed subscriptionCheckpointResume, shardingBootstrap and checkpointHoleRebuild at 1f0d081 (not run locally).

  • Docs: none; no documented surface covers checkpoint holes.

  • PR QA: Does a real cluster with route-excluded and non-replicating tables keep every origin checkpoint-grade, while an owed loss still voids it? — queued @ 1f0d081, qa-pr-harper-pro-1470-1f0d081

Performance: an applied live record does no new work; a dropped one adds a scope lookup. A copied row's bound checks read one map instead of two under this pass's committed bound, up to three past it, plus the reservation commits above.

Regression tests

🤖 Generated by Claude Opus 5.5 (Anthropic); posted via @kriszyp.

🤖 Generated with Claude Code

Refs harper-pro#1213, harper-pro#1214, harper-pro#1219

Release target: main — P3 checkpoint bookkeeping follow-ups, no backport requested

Related PRs: #1467 overlaps (a widened route's copy must keep standing holes), #717 overlaps (blob receive fault path beside the filtered-record blob discard), 5 others independent
Complexity: complicated

Deferred review findings, all filed

Deferred findings

Bundle — this PR covers 3 refs.

Closes #1213
Closes #1214
Closes #1219

Origin — the dispatch brief this PR was written from

Checkpoint hole and coverage gates: no origin-wide hole for filtered tables, no relayable floor on restricted routes, tighter hole/reservation lifetimes

Base: harper-pro main after #1092 merges.

  1. A dropped record of a filtered table marks an origin-wide route hole that nothing clears #1213: in recordReplicationHole and its callers, skip hole marking for drops that happen by design, and make sure restoreCompletedCopyPass no longer re-queues them.
  2. A sharding-restricted route stores and relays a peer floor as relayable #1214: compute relayable with the restricted-origins gate that routeFullCoverage uses.
  3. Copy-bound bookkeeping keeps holes and reservations longer than needed in rare races #1219: work through each race listed in the issue. Each gets a unit regression that shows the over-retention on main and its release with the fix.

Acceptance

  • Unit regressions for each case, failing on main.
  • Cluster: subscriptionCheckpointResume.test.mjs and shardingBootstrap.test.mjs pass.
  • A projection with a filtered table leaves its origin checkpoint-grade.

Dispatch: task proposal-kriszyp-hp-route-filter-coverage-gates-e19793 · queued by session:proposal · ran by claude/opus/xhigh · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain(codex); declined=muse; rounds=13; full=4 @ 1f0d081

Review-Attention: deep ~35m (critical: replicationConnection.ts; decisions: do-less-alternative, request-evidence, perf-cost, release-target-unclear, owner-decision-round-13) @ 1f0d081

kriszyp and others added 18 commits November 19, 2026 11:47
… floor; copy reservations hold only their own pass

- A record its route never owed (a table the receive route excludes, a table
  that does not replicate here, a local-only record) writes only the
  record-lock poison: no route hole, no projection hole row, no pass hole
  count and no residency hold (harper-pro#1213).
- A sharding-restricted route stores a received floor as not relayable, and
  the relay withholds a stored floor while the origin's link is restricted
  (harper-pro#1214).
- A copy pass's reservation row carries only the bounds that pass reserved,
  and settlement starts no reservation write and aborts once the copy
  watermark moves (harper-pro#1219).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…s that skipped a filtered table

Review round 1 of the checkpoint-hole change:
- an unowed drop of a sharded table's record still holds the cursor, as before,
  so an owned-slot copy cannot certify over it;
- a copy pass that dropped a filtered table's record fills no hole, since it did
  not deliver that table (it previously counted as a hole);
- the relay reads the sharding agreement once per floor capture and checks each
  origin's published certificate from its status view, so emission does no
  catalog read and cannot throw;
- settlement checks the pass it began with against the copy watermark and
  reports failure to its caller once replaced;
- a forwarded local-only record's announced blobs are discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
A projection pass that delivered no rows of a route-excluded table cannot
vouch for a hole in it, whether the sender skipped the table or forwarded it
and the receiver dropped it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…nership

Review round 2:
- only a full-coverage copy fills a hole: a projection's route certifies no
  coverage and this node cannot see its sender's table scope, so a projection
  that omitted a table (an explicit table list, a sender-side exclusion) could
  release a hole it did not repair;
- the relay reads the sharding agreement only when it has origins to relay;
- both unowed-drop callers return once the connection is superseded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
… left no table out

Review round 3: route coverage that was merely unset (a level-1 peer, or a
route never classified) counted as full, so a copy that walked fewer tables
could release a standing hole. A copy now fills holes only when the route was
proven full coverage at its COPY_START and its subscription request excluded
no table; projections skip the hole snapshot they can no longer use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…proven on it

Review round 4: a copy cannot say which subscription request it answers, and
the route's coverage flag lives on the connection object across reconnects.
A copy now fills holes only when every request on its socket asked for every
table with full coverage proven on that same socket, so an earlier partial
request or a level-1 reconnect after a level-2 socket cannot lend it authority.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Framing recheck after review round 5: a completed copy cannot show that its
walk carried a lost record. COPY_START and COPY_COMPLETE name neither the
request a copy answers nor the scope and start its walk used, and sender-side
table filters or a retention-forced copy that still filters by its old start
leave the record out of a walk that reports nothing missing. Every receiver-side
gate added in earlier rounds had another such gap, all present on main.

Holes are now released only by rebuilding or replacing the node, the
documented recovery. This removes the COPY_START hole snapshot, the fill gate
state, the foreign-peer release loop, and settleCopyBounds' filled/filledFrom
parameters. Re-enabling copy fills needs a sender attestation (harper-pro#1464).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…le fill wording

Review round 6: the revision test that guarded a pass's merged origins and
their HOLE_COPY_BOUND fence comes back without the removed fill; settlement
returns false when no store resolves instead of recording a no-op as settled;
comments and item 33 no longer describe copy fills.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…wnstream

Review round 7: core treats a genless certificate as a level-2 downgrade, so
the floor a restricted relay keeps forwarding at an unchanged floor is what
clears a downstream route's stale level-2 qualifiers. Withholding it left
those qualifiers in place. The receive-side gate (#1214) stays; the bound
store check in settlement now requires a writable store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…ownstream demotion

Review round 8: storing a sharding-restricted route's new floors as not
relayable (the change proposed in harper-pro#1214) stops the relay of the
genless floor that clears a downstream route's level-2 qualifiers, so a
downstream node kept checkpoint-grade coverage for an origin its relay no
longer fully received. Main's behaviour fails closed: the relayed floor is
genless (a restricted route stores no level-2 coverage), consistent with the
relay's log, and never trusted by admission. The test now pins that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…its table out

Review round 10: a table redeclared replicate: false (or newly route-excluded)
after a subscription request still has that request's in-flight records owed;
the receiver's live filter must not turn them into unowed drops. A dropped
record is now unowed only when every request this socket sent for the route's
node excluded its table; otherwise it is a hole as before. The local-only
branch no longer discards announced blobs, which a legacy transfer can share
with an owed record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…tables

Framing recheck after round 10: an explicit subscription list leaves out every
table it does not name, including one created after the request, so the
socket keeps a mode and one set (the union of explicit lists, or the
intersection of exclusion lists) rather than materializing local table names.
A later request that excludes a table never widens what the socket may treat
as unowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…r the table filter

Review round 11: the sender may serve any entry of a subscription request, so
a table counts as left out only if every entry left it out. A forwarded
local-only record is unowed even when its table's filter matched first. Once
a request has asked for every table, the scope is not recomputed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
An unsent request must not widen which tables count as never owed.

Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check

Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ws drops a send on a closing socket without throwing while already received
frames still arrive, so a widening request sent then must not mark a table
owed. An unrestricted request now takes a shared terminal scope instead of
allocating one.

Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriszyp
kriszyp marked this pull request as ready for review November 21, 2026 00:21
@kriszyp
kriszyp merged commit 2807734 into main Nov 21, 2026
50 checks passed
@kriszyp
kriszyp deleted the fix/checkpoint-hole-coverage-gates branch November 21, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant