Fix: answer MCP initialize locally, connect relayer in background (#415)

[hungtranphamminh]

Summary

Why

On a credentialed cold start (the common returning-user case), memwal-mcp
awaited a full relayer round-trip — TLS + GET /version + SSE handshake + the
endpoint event — before answering the MCP initialize handshake, because
the stdio↔SSE bridge forwarded initialize to the relayer rather than answering
it locally. When the cold-start path was slow (npx fetch + relayer TLS), this
exceeded the MCP client's 30s connection timeout, so the client SIGTERM'd the
process and no memory tools loaded for that session. Reported in #415
(WALM-321) with three timeout occurrences on the mainnet relayer.

What

• Answer initialize locally and instantly, and serve a static cold-start
  tools/list, before the relayer session exists — wiring stdin before the
  connect.
• Run the relayer connect in a background retry loop; buffer tools/call
  until connected, then flush in arrival order and emit
  notifications/tools/list_changed so the client re-lists the authoritative
  upstream tool set (spliced with the locally-served memwal_login/memwal_logout).
• Bound the connect with a new MEMWAL_MCP_CONNECT_TIMEOUT_MS (default 10s),
  shared across the compatibility check and the SSE connect, so a hung relayer
  degrades to a tool-call error instead of a failed startup.
• initialize is still forwarded upstream for capability negotiation, with its
  reply suppressed (one-shot, so a reused id still gets its real reply).

Solution

Mirrors the existing "auth-required" mode, which already answers initialize
locally when credentials are missing — this applies the same pattern to the
credentialed bridge path. The connect is decoupled from the handshake; the
existing reconnect + in-flight replay machinery owns delivery of any buffered
request after a stale-session 404 (so nothing is double-posted, even when a
concurrent reconnect races the flush). No new architectural surface; behaviour
is unchanged once the stream is up.

Preserved invariants: reconnect + in-flight replay, the login/logout tools/list
splice, default-namespace injection, the no-creds-wipe-on-401 semantics, and the
auth-required → bridge hot-handoff.

Types of Changes

• Bug fix (non-breaking)
• Test

Benchmark impact

• No benchmark-affecting change (MCP client DX; no recall / answer / extraction path touched)

Testing

• Tested locally
• Added/updated integration tests
• All new and existing tests pass (packages/mcp: 15/15, tsc clean)

New end-to-end tests (spawn the built binary against a mock relayer):

• initialize answered locally well before a slow relayer's SSE endpoint; static
  tools/list served instantly; buffered tools/call served after connect;
  upstream initialize reply suppressed; cold list == post-connect spliced set.
• Hung relayer → bounded connect → tool-call error envelope; process stays alive
  (no SIGTERM); buffered call closed out on shutdown.
• Flush-time 404 → reconnect de-duplicates (no double memory write / double
  reply), including a request arriving during the reconnect backoff and a
  concurrent server-pump-eof reconnect.
• Reusing the initialize id for a later request still returns that request's
  real reply (result and error).

Checklist

• Follows the project's commit + code conventions
• No ticket IDs / competitor names / local-doc paths in code comments
• Docs updated if needed (CHANGELOG entry added; MEMWAL_MCP_CONNECT_TIMEOUT_MS documented there)
• Respects the privacy floor (no server-readable plaintext index)
• All new and existing tests pass

Related

• GitHub: memwal-mcp blocks the MCP initialize handshake on a relayer connection during cold start, tripping the client's 30s connection timeout #415
• Linear: WALM-321

[ducnmm]

@hungtranphamminh this is the right fix for #415 / WALM-321. Answering initialize + a static tools/list locally, wiring stdin before the SSE connect, and bounding the relayer with MEMWAL_MCP_CONNECT_TIMEOUT_MS (never 0) is the correct shape. The flush/reconnect path is careful: in-flight owns id-bearing requests so a flush-time 404 cannot double-post, suppressUpstreamReplies is a count that resets on replay, and reused initialize ids still get a real reply. Tests cover the actual failure mode.

Approve.

Nits, not blocking:

1. Rebase dev. Branch is ~41 commits behind. MCP files look isolated, but please rebase before merge.

2. Cold-start TOOL_DEFINITIONS must stay locked to the sidecar. You added memwal_remember_bulk and memwal_health and documented the mirror. OAuth-scoped sessions (memwal:read only) will still advertise write tools until tools/list_changed. Acceptable for the handshake, but a comment next to LOCAL_TOOLS_LIST that scoped clients over-advertise until refresh would help the next editor.

3. Truncated JSDoc above failRequest — the previous failPendingForward comment is left open and then overwritten. Cosmetic.

4. serverInfo.version is still "0.0.1" while CHANGELOG is 0.0.7. Pre-existing in auth-required too; not introduced here.

CI is green. Ready to merge after the rebase.

[ducnmm]

@hungtranphamminh second-pass review after a closer look at the new background connect vs the existing reconnect() / adoptCredentials protocol. The handshake shape is still right; I am reversing the earlier approve because the new path is a second session publisher.

Request changes

1. connectInBackground ignores credentialGeneration and can flush after a login reconnect.

sse = await openSseStream(creds.relayerUrl, creds);
signalFirstConnect();
await flushPendingForward();

reconnect() already discards a handshake that finished after a key rotation (openingGeneration !== credentialGeneration). This loop does not.

Same-account memwal_login during the first connect:

1. Cold start puts initialize + tools/call in both pendingForward and inFlight.
2. adoptCredentials (same accountId) does not purge. It bumps credentialGeneration and reconnect("login-credentials-updated") replays all of inFlight on the new session.
3. The in-flight openSseStream from connectInBackground then completes, overwrites sse, and flushPendingForward() POSTs pendingForward again.

That is a double durable write (remember / remember_bulk) plus a second upstream initialize with only one suppress arm (second reply for the same id). If the stale handshake wins, later POSTs use the new bearer against the old session — postMessage only reconnects on 404, not 401.

Account-change is only half-fixed: pendingForward is purged, but connectInBackground can still publish account A’s session after B’s reconnect. The new live-login test asserts -32001 for the purged id and never sends a follow-up call, so it cannot see a clobbered session.

reconnect() already has the discard protocol. Smallest fix: do not have a second publisher — initial connect should be reconnect("initial", true) (or equivalent). If you keep two loops, on generation mismatch abort the candidate, do not flush, and do not overwrite a session reconnect() already published.

Please add a test: same-account memwal_login while the first SSE handshake is held → each buffered tools/call delivered once, initialize replied once, subsequent recall authenticated as the new bearer.

2. Account-change purge vs initialize suppression.

On account change, purge does suppressUpstreamReplies.delete(id) and returns for initialize without adding it to closedOutIds. If initialize was already POSTed and the reply is already queued on the old stream, iter.next() can still yield it after abort. With the arm cleared, the pump writes a second initialize reply.

Leave the one-shot arm so the late reply is consumed, or drop late initialize-shaped replies. Do not clear the arm and also skip closedOutIds.

What is still good

• Local initialize + static tools/list before SSE is the right #415 / WALM-321 fix.
• Flush-404 / concurrent server-pump-eof de-dup is careful for that pair; suppressUpstreamReplies as a count is correct once the first upstream initialize reply has been consumed.
• resolveConnectTimeoutMs() never treats 0 as unbounded.
• Auth-required listChanged: true is required for the post-login refresh.

Nits (after 1–2)

• Rebase dev (~40 commits behind; real conflicts in auth-required.ts + CHANGELOG 0.0.8). Keep sidecar title/annotations and remember_bulk/health on the cold list.
• NAMESPACE_TOOLS still omits memwal_remember_bulk.
• Truncated JSDoc above failRequest.
• serverInfo.version still "0.0.1" (pre-existing).
• A hung relayer does not fail the tool call until stdin closes; between retries the client’s own timeout fires. The PR body overstates “degrades to a tool-call error”.

[ducnmm]

@hungtranphamminh / follow-up from Henry: the two blockers from the second-pass review are closed on 0d9bdd40 + this commit.

• First connect still uses openSseStream + flushPendingForward (flush-404 / concurrent reconnect tests stay green).
• A handshake that finishes after credentialGeneration moved, or after reconnect() already published sse, is aborted. If login already owns reconnect, we wait it out and flush only leftover notifications.
• Same-account login now drops id-bearing pendingForward so a mid-flush rotation cannot double-post.
• Account-change purge keeps the initialize suppress arm and does not put that id in closedOutIds (id reuse still works).
• New test: same-account login during a delayed first handshake — each buffered tools/call POSTed once on the new bearer. Account-change test also asserts initialize answered once.

packages/mcp tests: 18/18.

Approve. Please still glance at CI after the rebase onto current dev.