fix(sidecar): crash-recovery reconcile searches for the wrong on-chain tag - #641
Open
harrymove-ctrl wants to merge 1 commit into
Open
fix(sidecar): crash-recovery reconcile searches for the wrong on-chain tag#641harrymove-ctrl wants to merge 1 commit into
harrymove-ctrl wants to merge 1 commit into
Conversation
…oncile lookup The whole point of PR #562's idempotency work is: if the sidecar crashes right after minting a paid Walrus blob but before its journal write lands, a retry must find and adopt that orphaned blob instead of minting a second paid one. The lookup that's supposed to find it was searching for the wrong tag. The durable-upload register step (walrus-upload-journal.ts — the only path every real /api/remember call goes through today) tagged the minted blob's on-chain metadata with memwal_migration_job, an unrelated, dead-code constant borrowed from the V1->V2 migration feature (its only reader, findOwnedBlobObjects, has zero callers anywhere in the codebase). The reconcile scan that runs on a lost journal, scanOwnerForJobBlob (walrus-query.ts), searches for memwal_job_id instead. The two could never match, so a crash at exactly the wrong moment would silently mint twice — the precise failure PR #562 exists to prevent. memwal_job_id was already the correct key, used correctly, by the legacy (non-durable) upload path — but that path is dead code today, since every current caller sets remember_job_id: Some(...), which always routes into the durable path instead. Fixes it by introducing a single shared MEMWAL_JOB_TAG_KEY constant in util.ts and having both write sites (durable + legacy) and the read site import it instead of each hardcoding the string literal, so the two sides structurally cannot drift apart again without someone deliberately un-importing the shared constant. New regression test pins the constant's value and greps all three files to confirm none hardcodes a competing 'memwal_job_id' literal. 193/193 sidecar scripts tests pass (7/7 in the directly affected files). tsc --noEmit: no new errors (one pre-existing, unrelated error in mcp/__tests__/integration.test.ts, confirmed present on unmodified dev too).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary — High severity, found while writing manual-test guides for PR #562
PR #562's whole point: if the sidecar crashes right after minting a paid Walrus blob but before its journal write lands locally, a retry must find and adopt that orphaned blob instead of minting a second paid one. The reconcile lookup that's supposed to find it was searching for the wrong tag — so it never worked for any real write.
Root cause
walrus-upload-journal.ts— the only path every real/api/remembercall goes through today) tagged the minted blob's on-chain metadata withmemwal_migration_job, an unrelated, dead-code constant borrowed from the V1→V2 migration feature (its only reader,findOwnedBlobObjects, has zero callers anywhere in the codebase).scanOwnerForJobBlob(walrus-query.ts), searches formemwal_job_idinstead.memwal_job_idwas already the correct key, used correctly, by the legacy (non-durable) upload path (walrus-upload.ts) — but that path is dead code today, since every current caller setsremember_job_id: Some(...), which always routes into the durable path instead.sidecar-find-blob-matching.test.ts,sidecar-find-blob-by-job.test.ts) mock the metadata fetch and never exercise the real register step's actual attributes.Fix
Introduces a single shared
MEMWAL_JOB_TAG_KEYconstant inutil.ts. Both write sites (durable + legacy) and the read site now import it instead of each hardcoding the string literal — so the two sides structurally cannot drift apart again without someone deliberately un-importing the shared constant.Secondary finding, not fixed here (flagging separately)
The reconcile scan currently fires on literally every first attempt of every write, not only genuine crash retries as its own comment claims —
consume_preparation_claimflips statuspending→runningwithblob_idstillNULLbeforeupload_resume_dispositionreads the row, which treats that as "a prior attempt was in-flight." Combined with the bug above, every write has been paying for an exhaustive on-chain metadata scan of the account's entire memory history for zero actual protection — not incorrect, but real, avoidable latency that grows with a user's memory count. Worth a follow-up (services/server/src/routes/remember.rs:803-809vs.jobs.rs:1440-1451,1503-1504).Validation
sidecar-find-blob-by-job.test.ts) pinsMEMWAL_JOB_TAG_KEY's value and greps all three touched files to confirm none hardcodes a competing"memwal_job_id"literal.npm testinservices/server/scripts: 193/193 pass (7/7 in the directly affected test files).tsc --noEmit: no new errors (one pre-existing, unrelated error inmcp/__tests__/integration.test.ts, confirmed present on unmodifieddevtoo).findOwnedBlobObjects(the only reader ofmemwal_migration_job) has zero callers anywhere in the codebase before touching the write side, so this change doesn't affect any other consumer.Note on testing this live
This bug is only reachable via a genuine process crash mid-upload — not safely reproducible on shared staging. The manual-test guide for PR #562 (already delivered separately) explicitly does not attempt to trigger this specific path; it only covers the separately-correct concurrent-request idempotency guard. Confirming this fix live needs a backend engineer deliberately crashing the sidecar mid-register in a controlled environment.