Repository navigation
Conversation
…er holds no floor A receiver with no valid floor for a certifying origin resumed that origin's own log at its received key maximum, and the sender's key-filtered read dropped a write that reserved a lower key and committed while the link was down; markers and later floors then certified past it (harper-pro#1454). The receiver now names the highest key it applied from the origin (resumeAfter), and the sender reads its own log past that entry in append order with no key filter. A base copy anchored at an own-log entry sends that anchor as the origin cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l back past a skipped entry Review of the no-floor resume (harper-pro#1454): - The own log is read in slices cut at transaction boundaries and interleaved with the other logs, so a busy local log cannot starve relayed origins. A pass whose slice was cut sends no marker and certifies no floor. - After a drained pass, every own key below the floor read before it is delivered, so the own log returns to the other logs' range just below that floor and the per-record overhead ends. - A failed own-log read stops sending this node's writes on that connection and reconnects, instead of a key-filtered fallback that could move the peer's next anchor past an entry it skipped. - An anchor the log no longer holds forces a base copy when it may have been purged; otherwise the log is read from its oldest retained entry. - The integration test waits until the peer's saved cursor reaches 'newer'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…another log flushes its frame Review round 2 of the no-floor resume (harper-pro#1454): - A failed own-log read mid-transaction no longer moves on to the other logs, whose first record would flush the partial frame and let the peer anchor on it. The failure is handled right after the slice and the cut frame is dropped. - The own-log slice is a plain loop before the unchanged main loop, so links without a no-floor resume pay nothing and the catch-up pays no generator. - An anchor the log cannot resume past forces a copy unless the purge bound and tracking start show nothing at or past it was purged. - The own-log iterator is released on failure as well as on retirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…may fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e own-log read stops at one A torn frame stops the own-log range like any range, but the failure path dropped the pending frame, which is the last complete transaction before the break, and bypassed the main range's corrupt-frame report and recovery. The txnlog tear suites then found the peer one row short and no break report. The frame is now dropped only when the last own entry left its transaction open, and a corrupt-frame stop goes through the existing handling: reported once and held at a mid-log break, reconnected otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge does The tear suites insert one transaction and tear a frame inside it; the main range delivers the entries before the break, and these tests pin that. The own-log failure path dropped them as a cut transaction. A corrupt frame now keeps them, and any other failure closes before the pending frame is flushed, so nothing needs dropping. Verified locally: txnlogTearReplication, txnlogReadableTearReplication and noFloorLateAppendResume pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kriszyp
left a comment
There was a problem hiding this comment.
A couple things to look at and merge to resolve, but I think this looks good.
| stopMarkers(); | ||
| ownLogRange = undefined; | ||
| subscribedNodeIds[getThisNodeId(auditStore)] = false; | ||
| if (range.corruptFrameStop?.breaks > 0) { |
There was a problem hiding this comment.
[major] A corrupt own-log read can commit an incomplete transaction on the peer
Consider a two-record transaction on subscribed tables. The slice encodes the first record with endTxn === false, then the iterator stops at a corrupt second record. This branch leaves the partial frame queued. The next relayed transaction can flush it at replication/replicationConnection.ts:10255-10268, and the end-of-pass path flushes it at replication/replicationConnection.ts:11607-11614 even without relay traffic. The receiver applies that frame as a transaction and can persist its key as origin progress at replication/replicationConnection.ts:12658-12660. Withholding markers and floors still leaves half the transaction visible, and the saved key can become the next resume anchor. Preserve queued frames for completed transactions, including when a later filtered entry leaves one queued, and discard a queued frame when corruption cuts short the transaction it encodes. Check both cases with the same two-record fixture and verify the incomplete transaction changes neither peer records nor its durable anchor.
There was a problem hiding this comment.
Fixed in 62f0c0d, 1428bea and 7b60da3, on both reads rather than only the own-log one, so the outcome doesn't depend on whether the peer held a floor.
- The send loop's ranges set
trackCorruptTransactions, so core records the key of a transaction whose last returned entry didn't end it. - Both flush points (
end_txnand the key switch) drop the queued frame instead of sending it when its key is recorded as cut and some origin in the frame has had no closing entry read (currentTransaction.open). A closing entry counts whether it was sent or filtered out, so a complete transaction with a filtered tail stays queued, and one origin's end can't clear another's cut prefix in a shared-key frame. - A copy boundary that fails while tailing is checked before it's replaced, and the own-origin recovery scan before its flush.
The tear suites already use your fixture's shape: one multi-record transaction torn inside. They now check that B keeps only the transactions before the torn one, that A logs Withholding transaction …, and that once the log is repaired the whole torn transaction arrives. B's records, cursor and anchor never see the prefix. The partial-append case no longer applies the torn entry with a wrong value.
Limits, now in DESIGN.md: after a repair, a key-rule link (an older peer, or a relayed origin with no floor) passes the withheld transaction, and blob chunks already streamed for it stay on the peer until blob cleanup.
| const ownLogSent = excludedNodes | ||
| ? !excludedNodes.includes(getThisNodeName()) | ||
| : logName === 'local'; | ||
| if (ownLogAnchor !== undefined && !ownLogRange && ownLogSent) { |
There was a problem hiding this comment.
[major] Append-order resume can send records excluded by add_node.start_time
The receiver folds the operator’s cutoff into startTime at replication/replicationConnection.ts:13884-13898. The sender then replaces that lower bound with zero. If a write reserved its key before the cutoff and appended after the peer’s saved anchor while it was down, the new path delivers it. The existing contract excludes keys at or before the cutoff, as documented in integrationTests/cluster/addNodeStartTime.test.mjs:2-8. Carry the cutoff independently of the resume cursor and enforce it when replaying the suffix or restarting from the oldest retained entry. A discriminating cluster check reserves two late writes on opposite sides of the cutoff and commits both after the anchor. Only the post-cutoff write should arrive.
There was a problem hiding this comment.
Fixed in 5cb06a6, with the two-sided check added in 8804d12.
The receiver sends its cutoff as startCutoff beside resumeAfter. The sender keeps it as the own-log key filter on the append-order read, on the read from the oldest retained entry, and after the handoff (ownLogFilterStart).
In noFloorLateAppendResume.test.mjs, one late write is reserved before the cutoff and another after it. Both commit after the peer's anchor. The pre-cutoff one never arrives and the post-cutoff one does.
| }); | ||
| } | ||
| } | ||
| const ownLogSliced = ownLogRange !== undefined && ownLogRange !== auditLogIterable; |
There was a problem hiding this comment.
[major] The performance account omits the work added during no-floor catch-up and handoff
Supply the **Performance:** statement for the receiver’s exact-anchor probe, the per-entry slice check, and the handoff at replication/replicationConnection.ts:11559-11570. The handoff creates an aggregate local reader below the sampled floor and can reread and resend entries the append-order pass just delivered. The tail above a floor can be large when a held transaction pins it. Measure catch-up time, CPU, and bytes resent with that tail and an active relayed origin.
| // certification check below still sees its failure. | ||
| if (range?.corruptFrameStop?.breaks > 0 && auditStore.reusableIterable) { | ||
| if (range.corruptFrameStop.midLogBreak) { | ||
| const stoppedRange = range?.corruptFrameStop?.breaks > 0 ? range : ownLogCorruptRange; |
There was a problem hiding this comment.
[minor] An aggregate log’s mid-log break masks recovery of a torn local log tail
The slice saves a torn local range for reconnect handling at replication/replicationConnection.ts:10657-10660. If a relayed log in the aggregate range also has a mid-log break, the selection always picks the aggregate range. That branch clears sendLogRecoveryPending at replication/replicationConnection.ts:11656-11658, so even after the local log advances, the local sender stays disabled until an unrelated reconnect. Handle the two stop states independently. A focused check can stop the local range at a torn tail, stop a relayed range mid-log, then advance the local log and verify that the connection retries from the peer’s cursor while the damaged relay remains held.
…at it sent, and recover a torn own tail beside a relayed break Review of #1475: - The receiver sends its add_node start_time cutoff beside resumeAfter, and the sender keeps it as the own-log key filter on the append-order read, the read from the oldest entry and after the handoff (ownLogFilterStart). A late write keyed before the cutoff no longer arrives; a new integration case pins it. - The own log returns to the key-ranged read only once the floor read before a drained pass is above every own key the slice sent, so that read resends none of them, however long a held transaction keeps the floor low. - A torn own-log tail and a relayed mid-log break are handled independently: the torn tail still reconnects, and each mid-log break is reported once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ppend-order # Conflicts: # replication/DESIGN.md
…o earlier versions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pped-range scan on a clean pass The peer holds every own key up to its anchor, so the handoff waits for a floor above it rather than 0, and resends none of them. The corrupt-frame check allocates only when a range actually stopped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in the confirmation Limits bullet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The QA run on When the sender forces a bounded base copy ( Measured: after purging This shares the cutoff plumbing with the open review thread "Append-order resume can send records excluded by The other seams probed passed: post- — Claude Opus 5.5, on behalf of Kris |
…its prefix Review of #1475: a corrupt frame inside a source transaction left the entries read before it queued, and the next transaction or the end-of-pass flush sent them, so the peer committed part of the transaction and its cursor (and, with no floor, its resumeAfter anchor) passed the rest. The send loop's ranges now set trackCorruptTransactions, so core records the key of a transaction whose last returned entry did not end it, and both points that flush a queued transaction drop it when that key is recorded on a range being read (cutByCorruptFrame). Transactions before it are still sent, and a complete one stays queued even when its last entries were filtered out. This applies to the key-ranged read as well as the append-order own-log read, so a peer gets the same result whichever path serves it. The tear suites pinned the old prefix delivery. They now expect B to keep only the transactions before the torn one, A to log the withheld transaction, and, after the log is repaired, the whole torn transaction to arrive, since B's cursor never passed it. The partial-append case no longer applies the torn entry with the wrong value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…opy fallback and recovery scan Review round on 62f0c0d: - A key alone does not name a transaction across logs: another origin's whole transaction can share a cut one's key. The queued transaction now records whether an entry closing it was read (sent or filtered out), and only one that never closed is withheld. On a healthy link that check ends the test before any range is consulted. - A copy boundary that fails while tailing is replaced before the next pass; its cut transaction is now dropped while the range still holds the evidence. - The own-origin recovery scan sets trackCorruptTransactions and checks its queued transaction before the closing flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dropTableOfflinePeer scenario 3 failed in CI on this branch (and 1 in 2 runs locally): after a pre-stamp node upgrades, its peer holds no floor for it, so the append-order read replays the node's old-build entries as soon as the link opens. That sent the structure of a table the node recreated on the old build before the node stamped it as newer than the drop; the peer refused the unstamped generation, latched its decoder refused for the connection, and skipped the node's next write to it. The sender only resent a structure when its shape changed, so the stamp never reached the peer on that link. The sender now also resends when the table's createdTime differs from the one it last sent. Stamping sets it on the same table object the connection sends from, so the next record carries the stamp and the peer accepts it. Locally: the race recurred (refusal 455 ms before the stamp) and the scenario passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rives Review of #1475 asked for a discriminating check on both sides of an add_node start_time cutoff. The suite already showed a late write reserved before the cutoff staying out; a second case reserves one after it, commits it after the peer's anchor, and requires it to arrive, so the cutoff filters only what it should. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…narrow the repair claim Review round 2: - A frame can carry several origins' transactions under one key, so a single ended flag let one origin's closing entry clear another's cut prefix. The queued frame now keeps the origins with no closing entry read (sent or filtered out); it is withheld when its key is cut and any origin is open. - A stamp written by another thread reached only the catalog and that thread's class, so a sender here kept comparing undefined to undefined. A stamp is now broadcast (replication-table-stamped) and each thread sets it on its class, so the structure resend sees it. - ownLogAppendAnchor is not honored for a peer that refuses copies, where the anchor probe is skipped and a missing anchor would reconnect forever. - DESIGN.md: the cut transaction arrives whole after repair only on links that resume past the anchor or below the floor, not on key-rule links; blob chunks streamed for a withheld transaction stay on the peer until blob cleanup; the per-entry cost on a healthy link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #1454.
⊙ Problem
When a peer reconnected before it held a delivery checkpoint (floor) for a sender, the sender resumed its own log from the peer's highest received key. A write that reserved a lower key and committed while the link was down was never sent, yet confirmation markers and later floors counted it as delivered.
💡 Solution
The peer now names the last entry of the sender's log it applied (
resumeAfter). The sender resends everything appended after that entry, in log order and whatever the keys, then returns to its normal read once a pass proves a floor. Older peers ignore the field and keep today's behavior.Performance: Steady-state links: a few branch checks per pass, no allocation. A no-floor catch-up: one anchor probe per connection, one comparison per own entry, one floor read per pass. The handoff resends nothing. A benchmark is being run separately; its numbers will be added.
⚖️ Alternatives
Approaches weighed
remove_nodecan't drain it and confirmations time out.localinto the aggregate range with an exact start:exactStartapplies to every log instartByLog, so relayed logs would be scanned from 0.🔧 Changes
replication/replicationConnection.tsresumeAfter: originLogKey(ownLogResumeAnchor). Frames arrive in the sender's append order from a proven start, so everything appended before that entry is held.exactStart+resumeAfterExactStart(ownLogPastAnchorRange), in slices of about 1,000 entries cut at transaction ends, before the unchanged loop over the other logs, so relayed origins can't starve.add_nodestart_timecutoff travels beside the anchor (startCutoff) and stays the own-log key filter.COPY_START.HARPER_TEST_WITHHOLD_ORIGIN_FLOORS.replication/DESIGN.md: the no-floor resume rule.unitTests/replication/ownLogResume.test.mjs: helpers, slice predicates, and a real RocksDB log appended at keys 10, 30, 20, where reading from 30 skips 20 and reading past 30 returns it.integrationTests/cluster/noFloorLateAppendResume.test.mjs: the issue's validation with floors withheld. The late write must arrive, and a later confirmed write must not return before it. A second case holds a write keyed before anupdate_nodestart_timecutoff; it must not arrive.docs/reference/replication/overview.md: the confirmation Limits bullet now names only the cases that still resume by key.✅ Verification
unitTests/replicationrun's failures also fail onmainor are flaky timing tests (repairDeleteEchoRuns,updateNodeShard,nodeUpdateWatcher).tsc --noEmit, oxlint and prettier clean.txnlogTearReplication,txnlogReadableTearReplication,recordLockConfirmation,lateAppendResume,addNodeStartTime,removeNodeDrainanddropTableOfflinePeerpass. The cutoff case fails with the filter forced to 0.Open points
🤖 Generated with Claude Code
Release target: main — fixes the floor-certified resume path, which ships only from main
Related PRs: #1458 overlaps (same DESIGN.md paragraph; its Limits bullet names this gap), #1467 overlaps (its widened-route copy must clear the own-log anchor), #1335 overlaps (changes the copy boundary the own anchor relies on), #1301 overlaps (borrowed origin cursors must not feed the anchor), 3 others independent
Deferred findings
0b40ef8693540f82nit data-integrity —replication/replicationConnection.ts— torn own prefix delivered as a complete transaction — NOT FILED56d7c683c2b2a146nit data-integrity —replication/replicationConnection.ts— torn own prefix shares a frame with a relayed record of equal txnLogKey — NOT FILEDReview-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; blocked=cursor-composer(not-installed); declined=cursor-grok,muse; rounds=9; full=1 @ 8eafe31
Review-Attention: deep ~65m (critical: replicationConnection.ts; decisions: doc-enumerates-key-resume-cases, handoff-only-past-sent-keys, seed-max-with-anchor, do-less-alternative, perf-cost, release-target-unclear; raised: degraded review) @ 8eafe31