Repository navigation
Conversation
… facts keyed by generation and connection One persisted per-route scope record replaces the coverage-only widening checks: [routeScope, route] holds the narrowest coverage and table list a level-2 route accepted progress under since its last completed copy. A narrowing is written synchronously before the request that narrows (or at once when learned between requests), a full route that widens in coverage or tables forces a full copy, and only a completed pass widens the record, before its copy cursor is removed (harper-pro#1142, #1212). - PendingRouteState verifies marks against core's generation rule, lets a later settled partialHistory supersede an earlier batch, and retries an unpersisted certifiedFrom only under the epoch that earned it; genless in-memory facts bind to the first generation that carries them; the hole callback reads the generation before its await and holds the frame when superseded; PendingOriginFloors.restore skips revoked or re-generated floors (harper-pro#1145). - Copy passes on links below level 2 keep no route marks or barriers (harper-pro#1143). - Restoration waits for the copy to finalize, and an origin missing from the route row or a genless fullCoverage no longer holds the cursor (harper-pro#1146). - One scan builds both copy-bound views (harper-pro#1149). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…; tighten route-mark restores - A pass carries the route-scope revision its socket's request left, not one read at COPY_START; every live narrowing bumps the revision, and removal promotes only the pass that finished, when it restored a full, non-widened copy. - A cursor from before scopes were recorded is judged against the record; a copy the route needs whole strips every resume entry from the request. - A failed batch no longer replays a partialHistory a later settle replaced, a certification on a revoked or holed row is not retried, and a commit after a new request writes no certification captured before it. - A superseded socket retires only its own provisional holes; genless facts bind with core's generation rule; the live-coverage listener stops after closing its socket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…ed batch still owns - A copy-cursor removal acts only for the finalized pass that armed it, through the barrier and retries. - A failed batch rolls back each settled version only while it is still the one that batch set. - A request narrowing what an earlier request on its socket covered bumps the scope revision, so an in-flight copy cannot promote over it; a failed-open exclusion recompute records no coverage narrowing at request build. - An unpersisted certification is not retried on a row whose coverage or generation moved. - Provisional holes are shared across a route's sockets, so a superseded call never retires a hole a replacement confirmed. - Fresh copies reset the local resume flags; request build binds facts by markGeneration; the exclusion listener stops on a closed or superseded socket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…e only the clear guards it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
The route-scope record type is no longer exported, and promotion states that an absent record is the next request's to seed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…he row refuses - Cursor removal and its retries act only while the arming pass is current, finalized and still the copy watermark's pass, rechecked after the settle and remove awaits. - A failed batch's retry drops each certification whose row moved its coverage or generation, or holds a voiding fact. - Each hole-recording call retires only its own provisional token, so it cannot consume another call's hold after a rotation. - A request built while the exclusion recompute has failed open again records the fallback as a narrowing: it cannot be told from a real narrowing that coincided. - The #1142 case asserts B's own entry carries no partial mark, so only the scope record can drive its copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…ve narrowings out of restoration - Every read-and-write of [routeScope, route] holds a cross-thread tryLock on it, so a worker finishing a copy at a failover handoff cannot overwrite a narrowing the replacement worker recorded; a held lock closes the socket or retries the removal. - A failed batch's retry keeps a clear coalesced with a refused certification, less the certification. - A narrowing learned on the socket before or during a copy leaves the pass and its cursor partial. - A provisional hole is the writing call's to confirm: request build and floor receipt no longer re-assert it. - A forced fresh copy fills the scope of the request that started it; a closing socket sends no subscription update. - The cluster test keeps every node that started for teardown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
… what was last asked for Adopted from the framing recheck: COPY_START names no request, so a copy's scope was inferred from socket state and a later or overlapping request could make it promote a scope it never walked. - A request whose direct-route scope differs from the socket's earlier one is recorded, then reconnects, so every copy runs on a socket with one scope. - The scope record keeps the last submitted scope; rev moves on a narrowing or any change in it, from any worker, so a replacement worker asking for the proven scope still fences an older widening copy. Unchanged requests write nothing. - The scope is recorded while the local engine can certify, whatever the peer does, so a narrowing during a level-1 interval is known when the peer certifies again. - Provisional holes live only in the shared provisional map: they suppress floors, and no route mark is written from one until its call confirms it. - Full coverage is not restored once the socket learned a narrowing; request and exclusion listeners stop once a close handshake begins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…arrowing; flag a scope repair - A later request on a socket whose revision moved (another worker asked for something else) reconnects, so a buffered copy cannot take a revision earned after it. - A live coverage narrowing retires queued and attached positive marks (the owner epoch), so a full-coverage restoration queued before it cannot land after. - A malformed scope record reads as the narrowest scope and its repair request resumes no saved cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…nder its own generation - A malformed scope record's repair writes a revision no earlier cursor or pass can hold, so an interrupted repair cannot resume an old walk or promote an old pass. - A certification clear re-asserts a confirmed hole only under the generation it was lost in. - Trimmed comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…quest scope Main (harper-pro#1470) added a socketRequestScope in the same closure: the tables every request sent on the socket left out. This branch's variable of that name, the one route scope a level-2 socket carries, becomes socketRouteScope. No behaviour change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
harper-pro#1470's tests sent a second, different-scope request on a level-2 socket. This branch never sends one: a scope change reconnects, and a closing socket builds no request. The narrowing test now checks that the socket is retired with the narrowing recorded and the cursor held before the in-flight record, instead of a hole. The closing-socket test keeps its assertions and stops waiting for a second request frame. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
…carries more than one scope A level-2 socket now reconnects on a scope change, so the fold from harper-pro#1470 runs across requests only on a link below level 2. Two cases there: a table a later request leaves out stays owed (its dropped record is a hole), and a table a later request asks for becomes owed once that request is sent. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
kriszyp
force-pushed
the
fix/checkpoint-route-state-lifecycle
branch
from
November 21, 2026 01:19
0069970 to
7703c75
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⊙ Problem
A level-2 checkpoint route resumes from its certified floor. The floor proves a prefix only for the requested coverage and tables. If a restart first revealed a narrower scope (#1142) or a table entered scope (#1212), the next resume could skip rows the node never received. This PR also fixes four related lifecycle faults: #1145, #1143, #1146 and #1149. #1147 was already fixed by #1092.
💡 Solution
Each direct level-2 route keeps a durable scope record: the narrowest scope it accepted progress under since its last completed copy, and what it last asked for. A widening on a full route forces a full copy. Only a completed full-coverage copy widens the record, and only if no request on any worker changed the record since that copy's request.
⚖️ Alternatives
Framing-Verdict: option-set-too-narrow at planning, resolved on the facts below. Then better-alternative-exists at two rechecks: the first was adopted, the second reached the cap.
🤔 Seed an absent scope record from the request, or fail closed at upgrade?
Seeded: #1092 shipped in v5.4.0, so failing closed would fully copy every level-2 route at the upgrade. The gap left is #1456; a one-line change reverses it.
🤔 Copy a table widening on a route that is not full?
No. The route never certifies, so item 26's gap remains until coverage widens and forces a full copy. We ruled out per-table backfill because the sender takes one start position per connection.
🤔 Bind a copy to its request by reconnecting, or by a request token?
Reconnect.
COPY_STARTnames no request. Reconnect before sending a request if its scope changed or another worker changed the revision. A token inCOPY_STARTchanges the wire protocol. Older senders would still need to reconnect.🤔 When a table leaves scope on a level-2 socket, reconnect or mark a hole?
Reconnect. #1470 sends the narrowed request and marks a hole for that table's in-flight record. Here the socket is retired, the cursor stays before the record, and the narrowing is recorded, so putting the table back forces a full copy. #1470's fold still decides below level 2.
🤔 Ship with what review left open at the cap?
Your call. Two framing rechecks did not pass. Codex still prefers an attempt ID to guard cross-worker cursor writes. Main also leaves that deletion unguarded. Open majors follow.
Open at the cap
start_timecutoff: a widening forces a copy from 0. Main does this for coverage widening. This PR adds table widening as a trigger.clearCopyBoundedCertificationis the same on main. It fails closed: the route refuses that origin's checkpoints until a full copy. The guard is left to A copy-bound certification clear can re-assert a route hole under a newer origin generation #1466 because the clear has six callers, and two pass neither a row nor a generation; a guard that misreads those drops a real hole.🔧 Changes
replication/routeScope.ts(new):scopeWidens,intersectScope,sameScoperecordRequestScopekeeps the proven and last requested scopes. It bumpsrevwhen scope narrows or the request changes.withRouteScopeLock, a cross-threadtryLockreplication/replicationConnection.ts:copyPassScopeandpromoteRouteScope, with the copy cursor carrying its scoperecordLiveCoveragerecords a live narrowing and retires queued positive marksPendingRouteState: generation-matched verification; settles rolled back only while still owned; a certification refused by its row dropped from retries; an owner epochPendingOriginFloors: restore skips floors dropped or re-generated since the takeCopyPassState.marksRoutegates the copy's route and restoration marks on level 2; the removal barrier runs only for marks or unverified batches to read back.finalizedgates restorationcopyPassRouteRestoreddecides cursor removalscanCopyBoundsbuilds the merged and committed views in one scan.replication/replicator.ts,replication/subscriptionManager.ts: an exclusion broadcast whose recompute failed carriesfailedOpen, and the live listener records nothing for it.replication/copyCursorWatermark.ts: the cursor type carriesscope/scopeRev.replication/DESIGN.mddocuments the route-scope rule, when finalization and level-2 checks apply, and how holes and floors are restored.routeScope.test.mjs,routeStateLifecycle.test.mjs,originFloors.test.mjs,routeScopeWidening.test.mjs, thefixture-route-scopecomponent (config, schema withFlippeddeclaredreplicate: false, theRouteStateprobe), andfixture-origin-log.Performance: nothing per applied record, and none of it is benchmarked.
✅ Verification
7703c75). The new rollback, refused-certification and coalesced-clear tests fail on the pre-fix source.7703c75: every check passed, including the cluster shards that runrouteScopeWidening.test.mjs,subscriptionCheckpointResume,removedOriginResumeCursorand thedeclarativeResidencysharding suites. That the new suite fails on main is traced, not run.Verification details
routeScope.test.mjsandrouteStateLifecycle.test.mjsare new;originFloors.test.mjsis updated.residencyControls.test.mjs: A record its route never asked for no longer leaves a checkpoint hole; copy reservations stay per pass #1470's narrowing test now asserts the level-2 reconnect, the held cursor and the recorded narrowing. Two new tests keep A record its route never asked for no longer leaves a checkpoint hole; copy reservations stay per pass #1470's request-scope fold covered on a level-1 link, in both directions; each fails when the fold is broken.distfrom the pre-fix source before testing for failure.7703c75, rebased onto main:routeScopeWidening,checkpointHoleRebuild,subscriptionCheckpointResumeandremovedOriginResumeCursorpass.0069970another session held a*:9925wildcard listener, which starves the integration loopback pool, so no cluster suite ran locally then. CI also runssubscriptionCheckpointResume,removedOriginResumeCursoranddeclarativeResidency.🤖 Generated by Claude Opus 5.5 with Claude Code; posted via @kriszyp.
Release target: v5.4 — the linked issues carry no milestone; 5.4.x releases are cut from main, and nothing here asks for an older line
Related PRs: #1092 overlaps (this branch reworks its widening and route-state code), #717 overlaps (both change who owns a hole recordReplicationHole marks), #1301 overlaps (its failover routes reach the same route rows and scope key), #1470 overlaps (its request-scope fold and this branch's reconnect both decide what a socket owes after a scope change), #1475 overlaps (it changes the sender's resume start, which a forced widening copy sets to 0), 4 others independent
Complexity: complicated
Deferred findings (10, each filed)
Deferred findings
0291f94493b73fb1major data-integrity —replication/routeScope.ts— Absent scope can trust pre-upgrade narrowed history — filed A route narrowed before upgrading to the route-scope build can widen without a copy #14563d3a7d40d3185d13major data-integrity —replication/replicationConnection.ts— Checkpoint coverage and hole release ignore the request table filter — filed Checkpoint coverage and hole release ignore the request's table list #1457219a7dc926df9bc0major data-integrity —replication/replicationConnection.ts— Copy-bound clearing restamps a confirmed hole onto a newer generation — filed A copy-bound certification clear can re-assert a route hole under a newer origin generation #14663afd6972edef3906minor reliability —replication/routeScope.ts— Requests can record failed-open coverage as partial — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #1459426dd8a6cba864f1minor performance —replication/replicationConnection.ts— One-scan copy-bound loader allocates per row — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #145910bbb64b68dbd671minor data-integrity —replication/replicationConnection.ts— Superseded partial-history skip also skips clearCertifiedFirst verification — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #1459880a7b7572b57b4bnit documentation —replication/DESIGN.md— Non-full routes do not backfill newly included tables automatically — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #145922b9c5f78b7a3450nit correctness —replication/replicationConnection.ts— copyPassRouteRestored treats an empty nodes row as restored — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #1459eeec7f2c11c49aebnit correctness —replication/replicationConnection.ts— Request scope uses nodeId while promotion uses the route row — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #1459cc8bbc891a675d50nit maintainability —replication/replicationConnection.ts— Scope promotion skips an absent record — filed Checkpoint route state: deferred review follow-ups (copy-bound scan cost, superseded clear check, empty route row) #1459🤔 defer this PR's major data-integrity finding
0291f94493b73fb1(Absent scope can trust pre-upgrade narrowed history) to #1456 instead of fixing it here?Bundle — this PR covers 6 refs.
Closes #1142
Closes #1212
Closes #1143
Closes #1145
Closes #1146
Closes #1149
Origin — the dispatch brief this PR was written from
Checkpoint route-state lifecycle: widening, generation scope, copy-pass completion
Base: harper-pro main after #1092 merges.
Order:
exclusionOriginsunchanged). Persist what the next request needs to know that its table scope or coverage widened, and mark the affected origins partial so the stored floor is not used for the new scope.Each fix gets a regression that fails on main. Note in the PR that #1147 is already done by #1092.
Evidence
Issues harper-pro#1142, #1143, #1145, #1146, filed 2026-10-27 as Review-Deferrals from harper-pro#1092's cross-model review after its scope was narrowed.
Acceptance
replicateon; the next resume copies that table's entries below the old floor (fails on main).subscriptionCheckpointResume.test.mjs,removedOriginResumeCursor,declarativeResidencyand the replication unit tests pass.Dispatch: task
proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d· queued by session:proposal · ran by claude/opus/xhigh · worker kzyp-xps-1Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain,domain(codex); declined=muse; rounds=9; full=1 @ 7703c75
Review-Attention: deep ~105m (critical: copyCursorWatermark.ts, replicationConnection.ts +3; decisions: do-less-alternative, requirement-shift, upgrade-bootstrap, perf-cost, deferred-0291f944, owner-decision-round-9) @ 7703c75