Skip to content

Resume a sender's own log past the last applied entry when the receiver holds no floor - #1475

Open
ldt1996 wants to merge 17 commits into
mainfrom
fix/no-floor-resume-append-order
Open

ldt1996 wants to merge 17 commits into
mainfrom
fix/no-floor-resume-append-order

Conversation

@ldt1996

@ldt1996 ldt1996 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1454.

⊙ Problem

When a peer reconnected before it held a delivery checkpoint (floor) for a sender, the sender resumed its own log from the peer's highest received key. A write that reserved a lower key and committed while the link was down was never sent, yet confirmation markers and later floors counted it as delivered.

💡 Solution

The peer now names the last entry of the sender's log it applied (resumeAfter). The sender resends everything appended after that entry, in log order and whatever the keys, then returns to its normal read once a pass proves a floor. Older peers ignore the field and keep today's behavior.

Performance: Steady-state links: a few branch checks per pass, no allocation. A no-floor catch-up: one anchor probe per connection, one comparison per own entry, one floor read per pass. The handoff resends nothing. A benchmark is being run separately; its numbers will be added.

⚖️ Alternatives

Approaches weighed
  • Base copy whenever no floor exists: provable, but every link would full-copy on the upgrade that introduces floors, and again while a revocation is pending.
  • Withhold markers and checkpoints on an unproven link: the link never gets a floor, so remove_node can't drain it and confirmations time out.
  • Fold local into the aggregate range with an exact start: exactStart applies to every log in startByLog, so relayed logs would be scanned from 0.

🔧 Changes

  • replication/replicationConnection.ts
    • Receiver: with no valid floor (none yet, or a revocation pending), sends resumeAfter: originLogKey (ownLogResumeAnchor). Frames arrive in the sender's append order from a proven start, so everything appended before that entry is held.
    • Sender: reads its own log past that entry with exactStart + resumeAfterExactStart (ownLogPastAnchorRange), in slices of about 1,000 entries cut at transaction ends, before the unchanged loop over the other logs, so relayed origins can't starve.
      • A cut pass sends no marker and certifies no floor.
      • After a drained pass whose floor is above every own key already sent, the own log returns to the main range just below that floor, resending nothing.
    • Missing anchor: forces a base copy unless the purge bound and tracking start show nothing at or past it was purged; otherwise reads from the oldest retained entry.
    • Read failure: stops this node's writes on that connection. A corrupt frame is reported and recovered as on the main range, which delivers the readable prefix; any other failure closes before the pending frame is flushed. A key-filtered fallback could move the peer's next anchor past a skipped entry.
    • An add_node start_time cutoff travels beside the anchor (startCutoff) and stays the own-log key filter.
    • A torn own-log tail and a relayed mid-log break are handled independently, so the torn tail still reconnects.
    • A log-order base copy sends the sender's own anchor in COPY_START.
    • Test hook HARPER_TEST_WITHHOLD_ORIGIN_FLOORS.
  • replication/DESIGN.md: the no-floor resume rule.
  • unitTests/replication/ownLogResume.test.mjs: helpers, slice predicates, and a real RocksDB log appended at keys 10, 30, 20, where reading from 30 skips 20 and reading past 30 returns it.
  • integrationTests/cluster/noFloorLateAppendResume.test.mjs: the issue's validation with floors withheld. The late write must arrive, and a later confirmed write must not return before it. A second case holds a write keyed before an update_node start_time cutoff; it must not arrive.
  • docs/reference/replication/overview.md: the confirmation Limits bullet now names only the cases that still resume by key.

✅ Verification

  • Unit tests: 69 passing in the touched suites. The full unitTests/replication run's failures also fail on main or are flaky timing tests (repairDeleteEchoRuns, updateNodeShard, nodeUpdateWatcher).
  • tsc --noEmit, oxlint and prettier clean.
  • Local cluster runs: both no-floor cases, txnlogTearReplication, txnlogReadableTearReplication, recordLockConfirmation, lateAppendResume, addNodeStartTime, removeNodeDrain and dropTableOfflinePeer pass. The cutoff case fails with the filter forced to 0.
Open points
  • No test reaches the cut-slice, handoff, read-failure, from-start, forced-copy or torn-tail-beside-mid-log branches. Worth adding: a write after the handoff must arrive, and a backlog over 1,000 own entries.
  • Under steady local writes the floor may never pass every key sent, so a no-floor link keeps the sliced read for its life instead of resending.
  • A corrupt own-log frame delivers the torn transaction's readable prefix, as the main range does and the tear suites pin; changing that belongs to both ranges together.
  • The own-log iterator is released on failure and retirement, not on teardown, like the existing range.
  • The review's "no restart test" self-check is a false positive: the integration test kills and restarts the peer.
  • Still uncovered: a sender whose own log was empty at copy time and that has sent no own frame since gives no anchor, so that link keeps the key rule.
  • The proof assumes earlier links started from a proven position; pre-upgrade raw-key resumes aren't covered, no worse than today.

🤖 Generated with Claude Code

Release target: main — fixes the floor-certified resume path, which ships only from main
Related PRs: #1458 overlaps (same DESIGN.md paragraph; its Limits bullet names this gap), #1467 overlaps (its widened-route copy must clear the own-log anchor), #1335 overlaps (changes the copy boundary the own anchor relies on), #1301 overlaps (borrowed origin cursors must not feed the anchor), 3 others independent

Deferred findings

  • 0b40ef8693540f82 nit data-integrity — replication/replicationConnection.ts — torn own prefix delivered as a complete transaction — NOT FILED
  • 56d7c683c2b2a146 nit data-integrity — replication/replicationConnection.ts — torn own prefix shares a frame with a relayed record of equal txnLogKey — NOT FILED

Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; blocked=cursor-composer(not-installed); declined=cursor-grok,muse; rounds=9; full=1 @ 8eafe31

Review-Attention: deep ~65m (critical: replicationConnection.ts; decisions: doc-enumerates-key-resume-cases, handoff-only-past-sent-keys, seed-max-with-anchor, do-less-alternative, perf-cost, release-target-unclear; raised: degraded review) @ 8eafe31

Lavinia and others added 5 commits November 20, 2026 02:01
…er holds no floor

A receiver with no valid floor for a certifying origin resumed that origin's
own log at its received key maximum, and the sender's key-filtered read
dropped a write that reserved a lower key and committed while the link was
down; markers and later floors then certified past it (harper-pro#1454).

The receiver now names the highest key it applied from the origin
(resumeAfter), and the sender reads its own log past that entry in append
order with no key filter. A base copy anchored at an own-log entry sends that
anchor as the origin cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l back past a skipped entry

Review of the no-floor resume (harper-pro#1454):
- The own log is read in slices cut at transaction boundaries and interleaved
  with the other logs, so a busy local log cannot starve relayed origins. A
  pass whose slice was cut sends no marker and certifies no floor.
- After a drained pass, every own key below the floor read before it is
  delivered, so the own log returns to the other logs' range just below that
  floor and the per-record overhead ends.
- A failed own-log read stops sending this node's writes on that connection
  and reconnects, instead of a key-filtered fallback that could move the
  peer's next anchor past an entry it skipped.
- An anchor the log no longer holds forces a base copy when it may have been
  purged; otherwise the log is read from its oldest retained entry.
- The integration test waits until the peer's saved cursor reaches 'newer'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…another log flushes its frame

Review round 2 of the no-floor resume (harper-pro#1454):
- A failed own-log read mid-transaction no longer moves on to the other logs,
  whose first record would flush the partial frame and let the peer anchor on
  it. The failure is handled right after the slice and the cut frame is dropped.
- The own-log slice is a plain loop before the unchanged main loop, so links
  without a no-floor resume pay nothing and the catch-up pays no generator.
- An anchor the log cannot resume past forces a copy unless the purge bound and
  tracking start show nothing at or past it was purged.
- The own-log iterator is released on failure as well as on retirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…may fail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ldt1996
ldt1996 marked this pull request as ready for review November 20, 2026 04:25
@ldt1996
ldt1996 requested a review from kriszyp as a code owner November 20, 2026 04:25
ldt1996 and others added 2 commits November 20, 2026 06:36
…e own-log read stops at one

A torn frame stops the own-log range like any range, but the failure path
dropped the pending frame, which is the last complete transaction before the
break, and bypassed the main range's corrupt-frame report and recovery. The
txnlog tear suites then found the peer one row short and no break report.

The frame is now dropped only when the last own entry left its transaction
open, and a corrupt-frame stop goes through the existing handling: reported
once and held at a mid-log break, reconnected otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge does

The tear suites insert one transaction and tear a frame inside it; the main
range delivers the entries before the break, and these tests pin that. The
own-log failure path dropped them as a cut transaction. A corrupt frame now
keeps them, and any other failure closes before the pending frame is flushed,
so nothing needs dropping. Verified locally: txnlogTearReplication,
txnlogReadableTearReplication and noFloorLateAppendResume pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple things to look at and merge to resolve, but I think this looks good.

stopMarkers();
ownLogRange = undefined;
subscribedNodeIds[getThisNodeId(auditStore)] = false;
if (range.corruptFrameStop?.breaks > 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] A corrupt own-log read can commit an incomplete transaction on the peer

Consider a two-record transaction on subscribed tables. The slice encodes the first record with endTxn === false, then the iterator stops at a corrupt second record. This branch leaves the partial frame queued. The next relayed transaction can flush it at replication/replicationConnection.ts:10255-10268, and the end-of-pass path flushes it at replication/replicationConnection.ts:11607-11614 even without relay traffic. The receiver applies that frame as a transaction and can persist its key as origin progress at replication/replicationConnection.ts:12658-12660. Withholding markers and floors still leaves half the transaction visible, and the saved key can become the next resume anchor. Preserve queued frames for completed transactions, including when a later filtered entry leaves one queued, and discard a queued frame when corruption cuts short the transaction it encodes. Check both cases with the same two-record fixture and verify the incomplete transaction changes neither peer records nor its durable anchor.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 62f0c0d, 1428bea and 7b60da3, on both reads rather than only the own-log one, so the outcome doesn't depend on whether the peer held a floor.

  • The send loop's ranges set trackCorruptTransactions, so core records the key of a transaction whose last returned entry didn't end it.
  • Both flush points (end_txn and the key switch) drop the queued frame instead of sending it when its key is recorded as cut and some origin in the frame has had no closing entry read (currentTransaction.open). A closing entry counts whether it was sent or filtered out, so a complete transaction with a filtered tail stays queued, and one origin's end can't clear another's cut prefix in a shared-key frame.
  • A copy boundary that fails while tailing is checked before it's replaced, and the own-origin recovery scan before its flush.

The tear suites already use your fixture's shape: one multi-record transaction torn inside. They now check that B keeps only the transactions before the torn one, that A logs Withholding transaction …, and that once the log is repaired the whole torn transaction arrives. B's records, cursor and anchor never see the prefix. The partial-append case no longer applies the torn entry with a wrong value.

Limits, now in DESIGN.md: after a repair, a key-rule link (an older peer, or a relayed origin with no floor) passes the withheld transaction, and blob chunks already streamed for it stay on the peer until blob cleanup.

const ownLogSent = excludedNodes
? !excludedNodes.includes(getThisNodeName())
: logName === 'local';
if (ownLogAnchor !== undefined && !ownLogRange && ownLogSent) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Append-order resume can send records excluded by add_node.start_time

The receiver folds the operator’s cutoff into startTime at replication/replicationConnection.ts:13884-13898. The sender then replaces that lower bound with zero. If a write reserved its key before the cutoff and appended after the peer’s saved anchor while it was down, the new path delivers it. The existing contract excludes keys at or before the cutoff, as documented in integrationTests/cluster/addNodeStartTime.test.mjs:2-8. Carry the cutoff independently of the resume cursor and enforce it when replaying the suffix or restarting from the oldest retained entry. A discriminating cluster check reserves two late writes on opposite sides of the cutoff and commits both after the anchor. Only the post-cutoff write should arrive.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5cb06a6, with the two-sided check added in 8804d12.

The receiver sends its cutoff as startCutoff beside resumeAfter. The sender keeps it as the own-log key filter on the append-order read, on the read from the oldest retained entry, and after the handoff (ownLogFilterStart).

In noFloorLateAppendResume.test.mjs, one late write is reserved before the cutoff and another after it. Both commit after the peer's anchor. The pre-cutoff one never arrives and the post-cutoff one does.

});
}
}
const ownLogSliced = ownLogRange !== undefined && ownLogRange !== auditLogIterable;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] The performance account omits the work added during no-floor catch-up and handoff

Supply the **Performance:** statement for the receiver’s exact-anchor probe, the per-entry slice check, and the handoff at replication/replicationConnection.ts:11559-11570. The handoff creates an aggregate local reader below the sampled floor and can reread and resend entries the append-order pass just delivered. The tail above a floor can be large when a held transaction pins it. Measure catch-up time, CPU, and bytes resent with that tail and an active relayed origin.

Comment thread replication/replicationConnection.ts Outdated
// certification check below still sees its failure.
if (range?.corruptFrameStop?.breaks > 0 && auditStore.reusableIterable) {
if (range.corruptFrameStop.midLogBreak) {
const stoppedRange = range?.corruptFrameStop?.breaks > 0 ? range : ownLogCorruptRange;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] An aggregate log’s mid-log break masks recovery of a torn local log tail

The slice saves a torn local range for reconnect handling at replication/replicationConnection.ts:10657-10660. If a relayed log in the aggregate range also has a mid-log break, the selection always picks the aggregate range. That branch clears sendLogRecoveryPending at replication/replicationConnection.ts:11656-11658, so even after the local log advances, the local sender stays disabled until an unrelated reconnect. Handle the two stop states independently. A focused check can stop the local range at a torn tail, stop a relayed range mid-log, then advance the local log and verify that the connection retries from the peer’s cursor while the damaged relay remains held.

ldt1996 and others added 5 commits November 20, 2026 07:35
…at it sent, and recover a torn own tail beside a relayed break

Review of #1475:
- The receiver sends its add_node start_time cutoff beside resumeAfter, and the
  sender keeps it as the own-log key filter on the append-order read, the read
  from the oldest entry and after the handoff (ownLogFilterStart). A late write
  keyed before the cutoff no longer arrives; a new integration case pins it.
- The own log returns to the key-ranged read only once the floor read before a
  drained pass is above every own key the slice sent, so that read resends none
  of them, however long a held transaction keeps the floor low.
- A torn own-log tail and a relayed mid-log break are handled independently: the
  torn tail still reconnects, and each mid-log break is reported once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ppend-order

# Conflicts:
#	replication/DESIGN.md
…o earlier versions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pped-range scan on a clean pass

The peer holds every own key up to its anchor, so the handoff waits for a
floor above it rather than 0, and resends none of them. The corrupt-frame
check allocates only when a range actually stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in the confirmation Limits bullet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kriszyp

kriszyp commented Nov 20, 2026

Copy link
Copy Markdown
Member

The QA run on 8eafe31ed6 (core 694cca02a0) found a late-append loss left by the forced base-copy path, and it silently drops acknowledged rows.

When the sender forces a bounded base copy (currentSequenceId = 0), either from the purged-anchor branch (replicationConnection.ts ~10952-10962) or from retention decision === 'copy' (~10939-10949), the own-origin subscription keeps the receiver's startTime (its anchor or key maximum). The only reset before the copy is if (otherGeneration && currentSequenceId === 0) (~10975-10979), so a forced copy that is not otherGeneration filters out own rows with version <= ownLogAnchor. The comment at ~10973-10974 says that filter "holds only for a position taken from this copy". On the copy path ownLogAnchor is cleared (~10981), so nothing lowers it again. A row whose key was reserved before the anchor but appended after the receiver's last apply is dropped permanently, and a write with replicatedConfirmation: 1 still returns HTTP 200. A source read on this head confirms the mechanism.

Measured: after purging local/1.txnlog past the anchor, the origin logged "forcing a bounded base-copy resync" and finished the copy, but the peer held 3 of 4 rows. The row late-purged (1795164044631.1453, below the anchor 1795164044651.6023) never arrived. The retention arm with update_node { start_time: cutoff } dropped late-after-cutoff-ret the same way, where the expected result delivers it and still excludes before-cutoff-ret. A cold reproduction went from 5 of 7 failing to 7 of 7 passing with a 7-line counterfactual: reset own startTime to 0 (or to just the operator cutoff) when the pass turns into a copy after the own subscription was seeded. The trigger needs a peer that was down, a late lower-keyed own write, and a forced copy (purged anchor or retention), so it is narrow, but the loss is silent.

This shares the cutoff plumbing with the open review thread "Append-order resume can send records excluded by add_node.start_time" (~11478). That thread is the opposite direction (over-delivery there, under-delivery here), so one fix may cover both. It is a residual arm of the same class as A reconnect before the first delivery checkpoint can skip an older-keyed write for good, and markers still confirm it, which this PR fixes.

The other seams probed passed: post-COPY_START anchors, a 3-node relay chain with 1,150 sliced rows, a multi-record atomic anchor with a 40 KB blob, and sharded same-shard and cross-shard cases. The test file is ready if useful: qa5542-no-floor-ownlog-edges.test.mjs.

— Claude Opus 5.5, on behalf of Kris

ldt1996 and others added 5 commits November 20, 2026 19:13
…its prefix

Review of #1475: a corrupt frame inside a source transaction left the entries
read before it queued, and the next transaction or the end-of-pass flush sent
them, so the peer committed part of the transaction and its cursor (and, with
no floor, its resumeAfter anchor) passed the rest.

The send loop's ranges now set trackCorruptTransactions, so core records the
key of a transaction whose last returned entry did not end it, and both points
that flush a queued transaction drop it when that key is recorded on a range
being read (cutByCorruptFrame). Transactions before it are still sent, and a
complete one stays queued even when its last entries were filtered out. This
applies to the key-ranged read as well as the append-order own-log read, so a
peer gets the same result whichever path serves it.

The tear suites pinned the old prefix delivery. They now expect B to keep only
the transactions before the torn one, A to log the withheld transaction, and,
after the log is repaired, the whole torn transaction to arrive, since B's
cursor never passed it. The partial-append case no longer applies the torn
entry with the wrong value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…opy fallback and recovery scan

Review round on 62f0c0d:
- A key alone does not name a transaction across logs: another origin's whole
  transaction can share a cut one's key. The queued transaction now records
  whether an entry closing it was read (sent or filtered out), and only one
  that never closed is withheld. On a healthy link that check ends the test
  before any range is consulted.
- A copy boundary that fails while tailing is replaced before the next pass;
  its cut transaction is now dropped while the range still holds the evidence.
- The own-origin recovery scan sets trackCorruptTransactions and checks its
  queued transaction before the closing flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dropTableOfflinePeer scenario 3 failed in CI on this branch (and 1 in 2 runs
locally): after a pre-stamp node upgrades, its peer holds no floor for it, so
the append-order read replays the node's old-build entries as soon as the link
opens. That sent the structure of a table the node recreated on the old build
before the node stamped it as newer than the drop; the peer refused the
unstamped generation, latched its decoder refused for the connection, and
skipped the node's next write to it. The sender only resent a structure when
its shape changed, so the stamp never reached the peer on that link.

The sender now also resends when the table's createdTime differs from the one
it last sent. Stamping sets it on the same table object the connection sends
from, so the next record carries the stamp and the peer accepts it. Locally:
the race recurred (refusal 455 ms before the stamp) and the scenario passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rives

Review of #1475 asked for a discriminating check on both sides of an add_node
start_time cutoff. The suite already showed a late write reserved before the
cutoff staying out; a second case reserves one after it, commits it after the
peer's anchor, and requires it to arrive, so the cutoff filters only what it
should.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…narrow the repair claim

Review round 2:
- A frame can carry several origins' transactions under one key, so a single
  ended flag let one origin's closing entry clear another's cut prefix. The
  queued frame now keeps the origins with no closing entry read (sent or
  filtered out); it is withheld when its key is cut and any origin is open.
- A stamp written by another thread reached only the catalog and that
  thread's class, so a sender here kept comparing undefined to undefined. A
  stamp is now broadcast (replication-table-stamped) and each thread sets it on
  its class, so the structure resend sees it.
- ownLogAppendAnchor is not honored for a peer that refuses copies, where the
  anchor probe is skipped and a missing anchor would reconnect forever.
- DESIGN.md: the cut transaction arrives whole after repair only on links that
  resume past the anchor or below the floor, not on key-rule links; blob chunks
  streamed for a withheld transaction stay on the peer until blob cleanup; the
  per-entry cost on a healthy link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A reconnect before the first delivery checkpoint can skip an older-keyed write for good, and markers still confirm it

2 participants