Skip to content

A widened checkpoint route copies the history it never received - #1467

Open
kriszyp wants to merge 14 commits into
mainfrom
fix/checkpoint-route-state-lifecycle
Open

kriszyp wants to merge 14 commits into
mainfrom
fix/checkpoint-route-state-lifecycle

Conversation

@kriszyp

@kriszyp kriszyp commented Nov 19, 2026 •

Copy link
Copy Markdown
Member
Kris's son unties a rope barrier and sends two dogs to fetch old datefruit crates while his sister chalks a route slate with one grove struck out.

⊙ Problem

A level-2 checkpoint route resumes from its certified floor. The floor proves a prefix only for the requested coverage and tables. If a restart first revealed a narrower scope (#1142) or a table entered scope (#1212), the next resume could skip rows the node never received. This PR also fixes four related lifecycle faults: #1145, #1143, #1146 and #1149. #1147 was already fixed by #1092.

💡 Solution

Each direct level-2 route keeps a durable scope record: the narrowest scope it accepted progress under since its last completed copy, and what it last asked for. A widening on a full route forces a full copy. Only a completed full-coverage copy widens the record, and only if no request on any worker changed the record since that copy's request.

⚖️ Alternatives

Framing-Verdict: option-set-too-narrow at planning, resolved on the facts below. Then better-alternative-exists at two rechecks: the first was adopted, the second reached the cap.

🤔 Seed an absent scope record from the request, or fail closed at upgrade?

Seeded: #1092 shipped in v5.4.0, so failing closed would fully copy every level-2 route at the upgrade. The gap left is #1456; a one-line change reverses it.

🤔 Copy a table widening on a route that is not full?

No. The route never certifies, so item 26's gap remains until coverage widens and forces a full copy. We ruled out per-table backfill because the sender takes one start position per connection.

🤔 Bind a copy to its request by reconnecting, or by a request token?

Reconnect. COPY_START names no request. Reconnect before sending a request if its scope changed or another worker changed the revision. A token in COPY_START changes the wire protocol. Older senders would still need to reconnect.

🤔 When a table leaves scope on a level-2 socket, reconnect or mark a hole?

Reconnect. #1470 sends the narrowed request and marks a hole for that table's in-flight record. Here the socket is retired, the cursor stays before the record, and the narrowing is recorded, so putting the table back forces a full copy. #1470's fold still decides below level 2.

🤔 Ship with what review left open at the cap?

Your call. Two framing rechecks did not pass. Codex still prefers an attempt ID to guard cross-worker cursor writes. Main also leaves that deletion unguarded. Open majors follow.

Open at the cap
  • Blocker, malformed scope record: a repair writes a clock-based revision, but live narrowing and promotion do not yet treat a malformed row as a fence, and two repairs at one clock value share a revision. Reaching it needs a corrupted record.
  • Major, start_time cutoff: a widening forces a copy from 0. Main does this for coverage widening. This PR adds table widening as a trigger.
  • Major, performance: counted under 🔧 Changes, not measured.
  • Major, pre-existing: a copy-bound clear can carry a hole into a newer generation, #1466. clearCopyBoundedCertification is the same on main. It fails closed: the route refuses that origin's checkpoints until a full copy. The guard is left to A copy-bound certification clear can re-assert a route hole under a newer origin generation #1466 because the clear has six callers, and two pass neither a row nor a generation; a guard that misreads those drops a real hole.
  • Minor: a request built while the exclusion recompute has failed open records full coverage as lost, which costs one extra full copy once coverage returns. It is in #1459 with the other deferrals.

🔧 Changes

request build ── recordRequestScope (cross-thread lock) ──► widened? ── mark partial, force full copy
      │ scope or revision changed on this socket ──► reconnect first
COPY_START ──► pass.scope / pass.scopeRev: the socket's one request (a resume keeps its cursor's)
copy finalized ──► restore partialHistory:false ──► barrier read-back ──► settle copy bounds
             ──► promoteRouteScope (lock, revision unchanged) ──► remove copy cursor

Performance: nothing per applied record, and none of it is benchmarked.

  • Per request: one locked scope read; a write only to seed, repair or record a change.
  • Commits with floors add one map of origin generations.
  • Settling route marks adds one map over main, plus rollback records. Batches with certifications now get a read-back.
  • Staging a copy cursor uses one spread to add two fields.
  • Per copy-bound load: one range scan validating each row. Loads are skipped while the origin-state epoch has not moved.

✅ Verification

  • Replication unit suite passes (2162 at 7703c75). The new rollback, refused-certification and coalesced-clear tests fail on the pre-fix source.
  • CI at 7703c75: every check passed, including the cluster shards that run routeScopeWidening.test.mjs, subscriptionCheckpointResume, removedOriginResumeCursor and the declarativeResidency sharding suites. That the new suite fails on main is traced, not run.
  • PR QA: Does a widened level-2 route copy the missed table once, then resume normally on later reconnects? — not finished in 90m @ 0069970 (status not readable by the rebase run), qa-pr-harper-pro-1467-0069970; the rebase changes no widening code.
  • Docs: no API, configuration or default changes. The receiver-side limits are tracked for the docs in #1459.
Verification details

🤖 Generated by Claude Opus 5.5 with Claude Code; posted via @kriszyp.

Release target: v5.4 — the linked issues carry no milestone; 5.4.x releases are cut from main, and nothing here asks for an older line
Related PRs: #1092 overlaps (this branch reworks its widening and route-state code), #717 overlaps (both change who owns a hole recordReplicationHole marks), #1301 overlaps (its failover routes reach the same route rows and scope key), #1470 overlaps (its request-scope fold and this branch's reconnect both decide what a socket owes after a scope change), #1475 overlaps (it changes the sender's resume start, which a forced widening copy sets to 0), 4 others independent
Complexity: complicated

Deferred findings (10, each filed)

Deferred findings

🤔 defer this PR's major data-integrity finding 0291f94493b73fb1 (Absent scope can trust pre-upgrade narrowed history) to #1456 instead of fixing it here?

Bundle — this PR covers 6 refs.

Closes #1142
Closes #1212
Closes #1143
Closes #1145
Closes #1146
Closes #1149

Origin — the dispatch brief this PR was written from

Checkpoint route-state lifecycle: widening, generation scope, copy-pass completion

Base: harper-pro main after #1092 merges.

Order:

  1. One widening detector covering both Widening after a restart is missed when only a relayed origin recorded partial coverage #1142 (a relayed origin's partial coverage lost on restart) and A table-scope widening resumes floor-only and skips the new table's entries below the floor #1212 (a table-scope change with exclusionOrigins unchanged). Persist what the next request needs to know that its table scope or coverage widened, and mark the affected origins partial so the stored floor is not used for the new scope.
  2. Scope route-state facts to the generation and connection that produced them #1145: key route-state facts by generation and connection.
  3. Checkpoint route marks and barriers run on base copies over level-0/1 links #1143: skip route marks and barriers on level-0/1 links.
  4. Copy-pass completion: restore whole history only after finalization; unstick cursor removal #1146: copy-pass completion ordering and cursor removal.
  5. Read copy-bound rows once per load #1149: read copy-bound rows once per load.

Each fix gets a regression that fails on main. Note in the PR that #1147 is already done by #1092.

Evidence

Issues harper-pro#1142, #1143, #1145, #1146, filed 2026-10-27 as Review-Deferrals from harper-pro#1092's cross-model review after its scope was narrowed.

Acceptance

Dispatch: task proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d · queued by session:proposal · ran by claude/opus/xhigh · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain,domain(codex); declined=muse; rounds=9; full=1 @ 7703c75

Review-Attention: deep ~105m (critical: copyCursorWatermark.ts, replicationConnection.ts +3; decisions: do-less-alternative, requirement-shift, upgrade-bootstrap, perf-cost, deferred-0291f944, owner-decision-round-9) @ 7703c75

kriszyp and others added 14 commits November 20, 2026 17:48
… facts keyed by generation and connection

One persisted per-route scope record replaces the coverage-only widening
checks: [routeScope, route] holds the narrowest coverage and table list a
level-2 route accepted progress under since its last completed copy. A
narrowing is written synchronously before the request that narrows (or at
once when learned between requests), a full route that widens in coverage
or tables forces a full copy, and only a completed pass widens the record,
before its copy cursor is removed (harper-pro#1142, #1212).

- PendingRouteState verifies marks against core's generation rule, lets a
  later settled partialHistory supersede an earlier batch, and retries an
  unpersisted certifiedFrom only under the epoch that earned it; genless
  in-memory facts bind to the first generation that carries them; the hole
  callback reads the generation before its await and holds the frame when
  superseded; PendingOriginFloors.restore skips revoked or re-generated
  floors (harper-pro#1145).
- Copy passes on links below level 2 keep no route marks or barriers
  (harper-pro#1143).
- Restoration waits for the copy to finalize, and an origin missing from
  the route row or a genless fullCoverage no longer holds the cursor
  (harper-pro#1146).
- One scan builds both copy-bound views (harper-pro#1149).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…; tighten route-mark restores

- A pass carries the route-scope revision its socket's request left, not
  one read at COPY_START; every live narrowing bumps the revision, and
  removal promotes only the pass that finished, when it restored a full,
  non-widened copy.
- A cursor from before scopes were recorded is judged against the record;
  a copy the route needs whole strips every resume entry from the request.
- A failed batch no longer replays a partialHistory a later settle
  replaced, a certification on a revoked or holed row is not retried, and
  a commit after a new request writes no certification captured before it.
- A superseded socket retires only its own provisional holes; genless
  facts bind with core's generation rule; the live-coverage listener stops
  after closing its socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…ed batch still owns

- A copy-cursor removal acts only for the finalized pass that armed it, through the barrier and retries.
- A failed batch rolls back each settled version only while it is still the one that batch set.
- A request narrowing what an earlier request on its socket covered bumps the scope revision, so an in-flight copy cannot promote over it; a failed-open exclusion recompute records no coverage narrowing at request build.
- An unpersisted certification is not retried on a row whose coverage or generation moved.
- Provisional holes are shared across a route's sockets, so a superseded call never retires a hole a replacement confirmed.
- Fresh copies reset the local resume flags; request build binds facts by markGeneration; the exclusion listener stops on a closed or superseded socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…e only the clear guards it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
The route-scope record type is no longer exported, and promotion states that an absent record is the next request's to seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…he row refuses

- Cursor removal and its retries act only while the arming pass is current, finalized and still the copy watermark's pass, rechecked after the settle and remove awaits.
- A failed batch's retry drops each certification whose row moved its coverage or generation, or holds a voiding fact.
- Each hole-recording call retires only its own provisional token, so it cannot consume another call's hold after a rotation.
- A request built while the exclusion recompute has failed open again records the fallback as a narrowing: it cannot be told from a real narrowing that coincided.
- The #1142 case asserts B's own entry carries no partial mark, so only the scope record can drive its copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…ve narrowings out of restoration

- Every read-and-write of [routeScope, route] holds a cross-thread tryLock on it, so a worker finishing a copy at a failover handoff cannot overwrite a narrowing the replacement worker recorded; a held lock closes the socket or retries the removal.
- A failed batch's retry keeps a clear coalesced with a refused certification, less the certification.
- A narrowing learned on the socket before or during a copy leaves the pass and its cursor partial.
- A provisional hole is the writing call's to confirm: request build and floor receipt no longer re-assert it.
- A forced fresh copy fills the scope of the request that started it; a closing socket sends no subscription update.
- The cluster test keeps every node that started for teardown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
… what was last asked for

Adopted from the framing recheck: COPY_START names no request, so a copy's scope was inferred from socket state and a
later or overlapping request could make it promote a scope it never walked.

- A request whose direct-route scope differs from the socket's earlier one is recorded, then reconnects, so every copy runs on a socket with one scope.
- The scope record keeps the last submitted scope; rev moves on a narrowing or any change in it, from any worker, so a replacement worker asking for the proven scope still fences an older widening copy. Unchanged requests write nothing.
- The scope is recorded while the local engine can certify, whatever the peer does, so a narrowing during a level-1 interval is known when the peer certifies again.
- Provisional holes live only in the shared provisional map: they suppress floors, and no route mark is written from one until its call confirms it.
- Full coverage is not restored once the socket learned a narrowing; request and exclusion listeners stop once a close handshake begins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…arrowing; flag a scope repair

- A later request on a socket whose revision moved (another worker asked for something else) reconnects, so a buffered copy cannot take a revision earned after it.
- A live coverage narrowing retires queued and attached positive marks (the owner epoch), so a full-coverage restoration queued before it cannot land after.
- A malformed scope record reads as the narrowest scope and its repair request resumes no saved cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…nder its own generation

- A malformed scope record's repair writes a revision no earlier cursor or pass can hold, so an interrupted repair cannot resume an old walk or promote an old pass.
- A certification clear re-asserts a confirmed hole only under the generation it was lost in.
- Trimmed comments that restated their code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dispatch-Task: proposal-kriszyp-hp-checkpoint-route-state-lifecycle-8ff21d
…quest scope

Main (harper-pro#1470) added a socketRequestScope in the same closure: the
tables every request sent on the socket left out. This branch's variable of
that name, the one route scope a level-2 socket carries, becomes
socketRouteScope. No behaviour change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
harper-pro#1470's tests sent a second, different-scope request on a level-2
socket. This branch never sends one: a scope change reconnects, and a closing
socket builds no request. The narrowing test now checks that the socket is
retired with the narrowing recorded and the cursor held before the in-flight
record, instead of a hole. The closing-socket test keeps its assertions and
stops waiting for a second request frame.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
…carries more than one scope

A level-2 socket now reconnects on a scope change, so the fold from
harper-pro#1470 runs across requests only on a link below level 2. Two cases
there: a table a later request leaves out stays owed (its dropped record is a
hole), and a table a later request asks for becomes owed once that request is
sent.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Dispatch-Task: rebase-merge-kriszyp-harper-pro-1467-00699702a452
@kriszyp
kriszyp force-pushed the fix/checkpoint-route-state-lifecycle branch from 0069970 to 7703c75 Compare November 21, 2026 01:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment