Skip to content

fix(sidecar): crash-recovery reconcile searches for the wrong on-chain tag - #641

Open
harrymove-ctrl wants to merge 1 commit into
devfrom
fix/reconcile-blob-tag-mismatch
Open

fix(sidecar): crash-recovery reconcile searches for the wrong on-chain tag#641
harrymove-ctrl wants to merge 1 commit into
devfrom
fix/reconcile-blob-tag-mismatch

Conversation

@harrymove-ctrl

Copy link
Copy Markdown
Collaborator

Summary — High severity, found while writing manual-test guides for PR #562

PR #562's whole point: if the sidecar crashes right after minting a paid Walrus blob but before its journal write lands locally, a retry must find and adopt that orphaned blob instead of minting a second paid one. The reconcile lookup that's supposed to find it was searching for the wrong tag — so it never worked for any real write.

Root cause

  • The durable-upload register step (walrus-upload-journal.ts — the only path every real /api/remember call goes through today) tagged the minted blob's on-chain metadata with memwal_migration_job, an unrelated, dead-code constant borrowed from the V1→V2 migration feature (its only reader, findOwnedBlobObjects, has zero callers anywhere in the codebase).
  • The reconcile scan that runs on a lost journal, scanOwnerForJobBlob (walrus-query.ts), searches for memwal_job_id instead.
  • The two could never match. A crash at exactly the wrong moment would silently mint a second paid blob on retry — the precise failure PR Fix: make remember writes idempotent — no duplicate paid Walrus blob #562 exists to prevent.
  • memwal_job_id was already the correct key, used correctly, by the legacy (non-durable) upload path (walrus-upload.ts) — but that path is dead code today, since every current caller sets remember_job_id: Some(...), which always routes into the durable path instead.
  • No existing test caught this: both existing tests (sidecar-find-blob-matching.test.ts, sidecar-find-blob-by-job.test.ts) mock the metadata fetch and never exercise the real register step's actual attributes.

Fix

Introduces a single shared MEMWAL_JOB_TAG_KEY constant in util.ts. Both write sites (durable + legacy) and the read site now import it instead of each hardcoding the string literal — so the two sides structurally cannot drift apart again without someone deliberately un-importing the shared constant.

Secondary finding, not fixed here (flagging separately)

The reconcile scan currently fires on literally every first attempt of every write, not only genuine crash retries as its own comment claims — consume_preparation_claim flips status pendingrunning with blob_id still NULL before upload_resume_disposition reads the row, which treats that as "a prior attempt was in-flight." Combined with the bug above, every write has been paying for an exhaustive on-chain metadata scan of the account's entire memory history for zero actual protection — not incorrect, but real, avoidable latency that grows with a user's memory count. Worth a follow-up (services/server/src/routes/remember.rs:803-809 vs. jobs.rs:1440-1451,1503-1504).

Validation

  • New regression test (sidecar-find-blob-by-job.test.ts) pins MEMWAL_JOB_TAG_KEY's value and greps all three touched files to confirm none hardcodes a competing "memwal_job_id" literal.
  • npm test in services/server/scripts: 193/193 pass (7/7 in the directly affected test files).
  • tsc --noEmit: no new errors (one pre-existing, unrelated error in mcp/__tests__/integration.test.ts, confirmed present on unmodified dev too).
  • Confirmed findOwnedBlobObjects (the only reader of memwal_migration_job) has zero callers anywhere in the codebase before touching the write side, so this change doesn't affect any other consumer.

Note on testing this live

This bug is only reachable via a genuine process crash mid-upload — not safely reproducible on shared staging. The manual-test guide for PR #562 (already delivered separately) explicitly does not attempt to trigger this specific path; it only covers the separately-correct concurrent-request idempotency guard. Confirming this fix live needs a backend engineer deliberately crashing the sidecar mid-register in a controlled environment.

…oncile lookup

The whole point of PR #562's idempotency work is: if the sidecar crashes
right after minting a paid Walrus blob but before its journal write lands,
a retry must find and adopt that orphaned blob instead of minting a second
paid one. The lookup that's supposed to find it was searching for the wrong
tag.

The durable-upload register step (walrus-upload-journal.ts — the only path
every real /api/remember call goes through today) tagged the minted blob's
on-chain metadata with memwal_migration_job, an unrelated, dead-code
constant borrowed from the V1->V2 migration feature (its only reader,
findOwnedBlobObjects, has zero callers anywhere in the codebase). The
reconcile scan that runs on a lost journal, scanOwnerForJobBlob
(walrus-query.ts), searches for memwal_job_id instead. The two could never
match, so a crash at exactly the wrong moment would silently mint twice —
the precise failure PR #562 exists to prevent.

memwal_job_id was already the correct key, used correctly, by the legacy
(non-durable) upload path — but that path is dead code today, since every
current caller sets remember_job_id: Some(...), which always routes into
the durable path instead.

Fixes it by introducing a single shared MEMWAL_JOB_TAG_KEY constant in
util.ts and having both write sites (durable + legacy) and the read site
import it instead of each hardcoding the string literal, so the two sides
structurally cannot drift apart again without someone deliberately
un-importing the shared constant.

New regression test pins the constant's value and greps all three files to confirm none hardcodes a competing 'memwal_job_id' literal. 193/193 sidecar scripts tests pass (7/7 in the directly affected files). tsc --noEmit: no new errors (one pre-existing, unrelated error in mcp/__tests__/integration.test.ts, confirmed present on unmodified dev too).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants