Repository navigation
A record its route never asked for no longer leaves a checkpoint hole; copy reservations stay per pass - #1470
Merged
Merged
Conversation
… floor; copy reservations hold only their own pass - A record its route never owed (a table the receive route excludes, a table that does not replicate here, a local-only record) writes only the record-lock poison: no route hole, no projection hole row, no pass hole count and no residency hold (harper-pro#1213). - A sharding-restricted route stores a received floor as not relayable, and the relay withholds a stored floor while the origin's link is restricted (harper-pro#1214). - A copy pass's reservation row carries only the bounds that pass reserved, and settlement starts no reservation write and aborts once the copy watermark moves (harper-pro#1219). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…s that skipped a filtered table Review round 1 of the checkpoint-hole change: - an unowed drop of a sharded table's record still holds the cursor, as before, so an owned-slot copy cannot certify over it; - a copy pass that dropped a filtered table's record fills no hole, since it did not deliver that table (it previously counted as a hole); - the relay reads the sharding agreement once per floor capture and checks each origin's published certificate from its status view, so emission does no catalog read and cannot throw; - settlement checks the pass it began with against the copy watermark and reports failure to its caller once replaced; - a forwarded local-only record's announced blobs are discarded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
A projection pass that delivered no rows of a route-excluded table cannot vouch for a hole in it, whether the sender skipped the table or forwarded it and the receiver dropped it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…nership Review round 2: - only a full-coverage copy fills a hole: a projection's route certifies no coverage and this node cannot see its sender's table scope, so a projection that omitted a table (an explicit table list, a sender-side exclusion) could release a hole it did not repair; - the relay reads the sharding agreement only when it has origins to relay; - both unowed-drop callers return once the connection is superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
… left no table out Review round 3: route coverage that was merely unset (a level-1 peer, or a route never classified) counted as full, so a copy that walked fewer tables could release a standing hole. A copy now fills holes only when the route was proven full coverage at its COPY_START and its subscription request excluded no table; projections skip the hole snapshot they can no longer use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…proven on it Review round 4: a copy cannot say which subscription request it answers, and the route's coverage flag lives on the connection object across reconnects. A copy now fills holes only when every request on its socket asked for every table with full coverage proven on that same socket, so an earlier partial request or a level-1 reconnect after a level-2 socket cannot lend it authority. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Framing recheck after review round 5: a completed copy cannot show that its walk carried a lost record. COPY_START and COPY_COMPLETE name neither the request a copy answers nor the scope and start its walk used, and sender-side table filters or a retention-forced copy that still filters by its old start leave the record out of a walk that reports nothing missing. Every receiver-side gate added in earlier rounds had another such gap, all present on main. Holes are now released only by rebuilding or replacing the node, the documented recovery. This removes the COPY_START hole snapshot, the fill gate state, the foreign-peer release loop, and settleCopyBounds' filled/filledFrom parameters. Re-enabling copy fills needs a sender attestation (harper-pro#1464). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…le fill wording Review round 6: the revision test that guarded a pass's merged origins and their HOLE_COPY_BOUND fence comes back without the removed fill; settlement returns false when no store resolves instead of recording a no-op as settled; comments and item 33 no longer describe copy fills. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…wnstream Review round 7: core treats a genless certificate as a level-2 downgrade, so the floor a restricted relay keeps forwarding at an unchanged floor is what clears a downstream route's stale level-2 qualifiers. Withholding it left those qualifiers in place. The receive-side gate (#1214) stays; the bound store check in settlement now requires a writable store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…ownstream demotion Review round 8: storing a sharding-restricted route's new floors as not relayable (the change proposed in harper-pro#1214) stops the relay of the genless floor that clears a downstream route's level-2 qualifiers, so a downstream node kept checkpoint-grade coverage for an origin its relay no longer fully received. Main's behaviour fails closed: the relayed floor is genless (a restricted route stores no level-2 coverage), consistent with the relay's log, and never trusted by admission. The test now pins that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…its table out Review round 10: a table redeclared replicate: false (or newly route-excluded) after a subscription request still has that request's in-flight records owed; the receiver's live filter must not turn them into unowed drops. A dropped record is now unowed only when every request this socket sent for the route's node excluded its table; otherwise it is a hole as before. The local-only branch no longer discards announced blobs, which a legacy transfer can share with an owed record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…tables Framing recheck after round 10: an explicit subscription list leaves out every table it does not name, including one created after the request, so the socket keeps a mode and one set (the union of explicit lists, or the intersection of exclusion lists) rather than materializing local table names. A later request that excludes a table never widens what the socket may treat as unowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
…r the table filter Review round 11: the sender may serve any entry of a subscription request, so a table counts as left out only if every entry left it out. A forwarded local-only record is unowed even when its table's filter matched first. Once a request has asked for every table, the scope is not recomputed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793
An unsent request must not widen which tables count as never owed. Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ws drops a send on a closing socket without throwing while already received frames still arrive, so a widening request sent then must not mark a table owed. An unrestricted request now takes a shared terminal scope instead of allocating one. Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-route-filter-coverage-gates-e19793 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
Checkpoint bookkeeping kept origins out of checkpoint resume longer than needed. A record the receive route filters by design (an excluded or non-replicating table, a local-only record) became a checkpoint hole (harper-pro#1213). A copy pass's reservation also carried other passes' bounds, and settling could write a new one (harper-pro#1219).
🤔 Keep a sharding-restricted route's floor relayable (harper-pro#1214 not applied)
harper-pro#1214 asked to store
relayableonly with full coverage. I kept main's behavior: a restricted route relays a floor with no generation, and that floor clears a downstream route's level-2 qualifiers. Suppressing it kept the downstream node checkpoint-grade for an origin it no longer fully received.💡 Solution
⚖️ Alternatives
🤔 Owner decision: per-socket request scope, not "every filtered row is a hole"
The capped framing review preferred a hole for every table-filtered row, exempting only local-only rows. That keeps a selective receiver's origin non-checkpoint-grade until a rebuild whenever an older peer forwards an excluded table: the harper-pro#1213 case. Unfixed: the reservation cost below, a brittle closing-socket fixture; pre-existing majors are in harper-pro#1462 and harper-pro#1465.
🤔 Each copy pass reserves for itself instead of borrowing another pass's reservation
A borrowed bound disappears when its owner settles. Cost: when another pass reserved further ahead, this pass commits its own reservations, about one extra awaited commit each time a row's distance past the copy start doubles beyond 60 s. Not benchmarked.
🔧 Changes
replication/replicationConnection.tsrecordUnowedDrophandles a record the route never owed: poison only, the sharded-table hold kept.recordReplicationHoleis now for owed records only.socketRequestScopeandfoldSentRequestScoperecord which tables every sent request left out, folded only afterws.sendon an open socket; the every-table-requested state is one shared object,EVERY_TABLE_REQUESTED.recordUnowedDroporrecordReplicationHolefrom that scope and the local-only flag; the standalone local-only drop usesrecordUnowedDrop.passReservedBoundsholds the pass's own reservation, cleared atCOPY_START. The copy-row path,reserveCopyStartBounds,copyBoundsCommittedandpersistCopyBoundsuse it;maxCopiedVersionByOriginstays the certification gate (mergePersistedCopyBoundsdoc).settlePassCopyBoundschecks the pass is still current and waits for an in-flight reservation write.settleCopyBoundsloses its hole-fill parameters; theCOPY_STARThole snapshot and the cross-peer release are removed.receivedOriginFloorsdocuments why a restricted route keepsrelayable.replication/DESIGN.md: item 25 and item 33 state the owed-only hole rule and the request scope, item 27 the restricted-route relay, and 33's row-ownership note that holes clear only on rebuild and a reservation carries only its pass's bounds.integrationTests/cluster/checkpointHoleRebuild.test.mjs: header names where admission refusal is now covered.✅ Verification
15 new or rewritten fake-socket cases drive the real
replicateOverWShandler. 12 fail on main's build; 3 pin behavior main shares (an owed drop's hole stands, a redeclared table still holes, a restricted route stays relayable). The closing-socket case also fails with only its guard removed.npm run test:unit: 2848 passing, 0 failing;lint:requiredandtypecheckpass.CI's cluster shards passed
subscriptionCheckpointResume,shardingBootstrapandcheckpointHoleRebuildat 1f0d081 (not run locally).Docs: none; no documented surface covers checkpoint holes.
PR QA: Does a real cluster with route-excluded and non-replicating tables keep every origin checkpoint-grade, while an owed loss still voids it? — queued @ 1f0d081, qa-pr-harper-pro-1470-1f0d081
Performance: an applied live record does no new work; a dropped one adds a scope lookup. A copied row's bound checks read one map instead of two under this pass's committed bound, up to three past it, plus the reservation commits above.
Regression tests
unitTests/replication/residencyControls.test.mjsunitTests/replication/originFloorSeqRow.test.mjs: a hole still refuses admission after a later pass settles; a hole revision keeps every origin the pass lost.🤖 Generated by Claude Opus 5.5 (Anthropic); posted via @kriszyp.
🤖 Generated with Claude Code
Refs harper-pro#1213, harper-pro#1214, harper-pro#1219
Release target: main — P3 checkpoint bookkeeping follow-ups, no backport requested
Related PRs: #1467 overlaps (a widened route's copy must keep standing holes), #717 overlaps (blob receive fault path beside the filtered-record blob discard), 5 others independent
Complexity: complicated
Deferred review findings, all filed
Deferred findings
ae208e47dd1d5f1fmajor concurrency —replication/replicationConnection.ts— A stale qualified floor can survive a restriction during a relay scan — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462f84c9a7f32887312major data-integrity —replication/replicationConnection.ts— Concurrent projection losses can erase a durable origin fence — filed Concurrent projection drops can lose an origin from a checkpoint hole row's next revision #14659a2d0c51ddd65274major concurrency —replication/replicationConnection.ts— Retired socket can enqueue a late floor or sequence update — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462270f5ef2e107b7c0major data-integrity —replication/replicationConnection.ts— Persisted downstream relay qualification can survive upstream restriction — filed A relayed origin floor stays in use downstream, or stays relayable, after the relay's link stops carrying every table #1461124fae7424e44762major data-integrity —replication/replicationConnection.ts— A table-only exclusion leaves a pinned bare floor relayable — filed A relayed origin floor stays in use downstream, or stays relayable, after the relay's link stops carrying every table #1461edbdd849c235a214minor performance —replication/replicationConnection.ts— Live-only and retrieval sockets allocate a copy reservation map — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462a25dceb4ed4f4237minor reliability —replication/recordLockPoison.ts— Intentional drop produces a replication-hole warning — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #146234a8196d8073f873minor concurrency —replication/replicationConnection.ts— Copy settlement callbacks and bookkeeping can outlive their attempt — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462d6eebe3e102a08d4minor correctness —replication/replicationConnection.ts— A discarded refresh still advances request markers and rejects valid earlier replies — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462ec4aa58eacde07c4minor correctness —replication/replicationConnection.ts— Replacement copy retains the abandoned attempt's exact peaks — filed Checkpoint copy-bound, relay and request-marker follow-ups: attempt-scoped settlement and peaks, relay agreement, blob discard, warning wording #1462Bundle — this PR covers 3 refs.
Closes #1213
Closes #1214
Closes #1219
Origin — the dispatch brief this PR was written from
Checkpoint hole and coverage gates: no origin-wide hole for filtered tables, no relayable floor on restricted routes, tighter hole/reservation lifetimes
Base: harper-pro main after #1092 merges.
recordReplicationHoleand its callers, skip hole marking for drops that happen by design, and make surerestoreCompletedCopyPassno longer re-queues them.relayablewith the restricted-origins gate thatrouteFullCoverageuses.Acceptance
subscriptionCheckpointResume.test.mjsandshardingBootstrap.test.mjspass.Dispatch: task
proposal-kriszyp-hp-route-filter-coverage-gates-e19793· queued by session:proposal · ran by claude/opus/xhigh · worker kzyp-xps-1Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain(codex); declined=muse; rounds=13; full=4 @ 1f0d081
Review-Attention: deep ~35m (critical: replicationConnection.ts; decisions: do-less-alternative, request-evidence, perf-cost, release-target-unclear, owner-decision-round-13) @ 1f0d081