fix(cli): don't lose the queued message when a remote launch fails#1058
Open
junmo-kim wants to merge 5 commits into
Open
fix(cli): don't lose the queued message when a remote launch fails#1058junmo-kim wants to merge 5 commits into
junmo-kim wants to merge 5 commits into
Conversation
claudeRemoteLauncher's respawn loop retried claudeRemote() immediately on every throw with no backoff or limit. A deterministic launch failure (bad auth, invalid model/args, spawn failure) respawned in a tight loop instead of giving up, hammering the same failure forever. Track whether onReady() fired at least once per attempt to tell an immediate/deterministic failure apart from a failure after real progress, back off between immediate-failure retries, and after 3 consecutive immediate failures drop the message that keeps triggering them and reset the streak, instead of respawning forever. The session keeps running so a later, unrelated message still gets its own budget. The streak reset on a non-throwing attempt is itself gated on having reached onReady, not applied unconditionally -- otherwise a message that keeps getting parked and re-picked-up on alternating attempts (e.g. an isolated command hitting the same deterministic failure) would reset the streak every other attempt and the cap would never fire.
…very MessageQueue2.collectBatch() acks a message (fires onBatchConsumed, which the hub uses to mark it consumed) at dequeue time, before the message ever reaches the SDK. If claudeRemote() then throws before onReady -- e.g. the process dies right after picking up the message -- the catch block only logged and retried, so the message vanished: the hub already thinks it was delivered, but the CLI never acted on it. Track the message returned from nextMessage() (whether freshly dequeued or held in `pending` across a mode change) as in-flight until the next onReady confirms it was handled, and restore it to the front of the queue (preserving isolation via unshiftIsolated when needed) if the attempt throws and will be retried. Restoring happens even if the throw races with a user-initiated switch/exit, so a message is not silently dropped by that unrelated shutdown either. When the immediate-failure cap from the previous commit is reached, the in-flight message is dropped instead of restored: unshifting it back would just feed it into another immediate failure on the very next attempt, storming again. This mirrors cursorLegacyRemoteLauncher's existing drop-and-reset policy on its own consecutive-failure cap.
There was a problem hiding this comment.
Findings
- [Major] Restored messages lose their hub identity —
inFlightMessageonly stores the combined text/mode/isolation state, butMessageQueue2.collectBatch()has already emittedmessages-consumedfor the originallocalIds before this point. Requeueing viaunshift()/unshiftIsolated()without those ids means the retried prompt is no longer tied to the hub row, and if the retry later hits the cap the original web message remains marked consumed even though the launcher is explicitly dropping it. Evidencecli/src/claude/claudeRemoteLauncher.ts:423, contextcli/src/utils/MessageQueue2.ts:374andcli/src/agent/sessionBase.ts:76.
Suggested fix:type InFlightMessage = { items: Array<{ message: string; mode: EnhancedMode; isolate: boolean; localId?: string }>; }; const restoreInFlightMessage = () => { if (!inFlightMessage) return; for (const item of [...inFlightMessage.items].reverse()) { if (item.isolate) { session.queue.unshiftIsolated(item.message, item.mode, item.localId); } else { session.queue.unshift(item.message, item.mode, item.localId); } } inFlightMessage = null; };
- [Major] Successful local-only turns do not reset the immediate-failure streak — the new reset only runs when
onReady()fires, butclaudeRemote()can consume and complete a message without ever callingonReady()for/clear(claudeRemote.ts:102) or any other successful path before the SDK result loop. After two startup failures, a successful/clearleavesimmediateFailureCountat 2, so the next unrelated launch failure is treated as the third failure and drops that new message after one try. Evidencecli/src/claude/claudeRemoteLauncher.ts:485.
Suggested fix:let deliveredMessageThisAttempt = false; // Set this in both branches that return a real message to claudeRemote(). inFlightMessage = { message: msg.message, mode: msg.mode, isolate: msg.isolate }; deliveredMessageThisAttempt = true; // After await claudeRemote(...) completes normally: if (reachedReadyThisAttempt || deliveredMessageThisAttempt) { inFlightMessage = null; immediateFailureCount = 0; }
Questions
- None.
Summary
- Review mode: initial
- The retry cap addresses the respawn loop, but the current implementation still breaks queued-message identity on restore and can carry failure count across a successful local-only command. Both affect user prompts added by this diff's retry/drop behavior.
Testing
- Not run:
bunis not installed in this runner (/bin/bash: bun: command not found).
HAPI Bot
MessageQueue2.collectBatch() already collects each queue item's localId (it fires onBatchConsumed with the full list to ack them), but only exposed the joined `message` string to callers, discarding the per-item localIds and their original boundaries in the process. When claudeRemoteLauncher restores a dequeued-but-undelivered batch after a launch failure, it re-added the joined string as a single new queue item with no localId, orphaning the retried prompt from the hub row(s) it originated from (and from cancel-by-localId). Expose the pre-join `items` breakdown (message + localId per item) alongside the existing joined `message` field on collectBatch()/waitForMessagesAndGetAsString() -- purely additive, so the other callers of waitForMessagesAndGetAsString() (grok, kimi, opencode, cursor, codex, runAgentSession) are unaffected. On restore, unshift each original item individually in reverse order, so the localId and relative order of a multi-message batch are both preserved instead of just the first item's.
…uccess claudeRemote.ts's /clear handling delivers the queued message to the SDK, then calls onSessionReset()/onCompletionEvent() and returns successfully without ever calling onReady(). The success-path streak reset only cleared on reachedReadyThisAttempt, so a successful /clear between two unrelated immediate launch failures did not reset the streak: an unrelated message's very next failure could hit the 3-in-a-row cap after just 1 failure, and the resulting banner would misreport "3 times in a row". Track whether nextMessage() actually handed a message to the SDK this attempt (deliveredMessageThisAttempt), separately from whether the attempt reached onReady, and reset the streak on either signal. The livelock-prone case this guards against (a message parked into `pending` and the attempt returning without ever delivering anything) leaves both flags false, so it still does not reset the streak.
There was a problem hiding this comment.
Findings
- No issues found.
Questions
- None.
Summary
- Review mode: follow-up after new commits
- Prior bot findings around localId-preserving restore and successful no-
onReadyreset appear addressed in the latest diff. Residual risk: I did not execute the PR tests in this review environment.
Testing
- Not run (automation; PR content not executed).
HAPI Bot
Upstream tiann#1056 added cli/src/claude/claudeRemoteLauncher.test.ts covering resume-anchor / compact-outcome behaviour, colliding (add/add) with the test file this branch introduced for launch-failure recovery. Keep upstream's file verbatim and move this branch's suite to a sibling file, claudeRemoteLauncher.launchFailure.test.ts, so the two independent module-mock setups stay in separate vitest file scopes. The launcher source auto-merged cleanly: tiann#1056 relocated consumeOneTimeFlags() into the onSessionFound/onSessionReset callbacks, orthogonal to this branch's respawn-storm guard and message-restore logic.
There was a problem hiding this comment.
Findings
- No issues found.
Questions
- None.
Summary
- Review mode: follow-up after new commits
- Reviewed the full current PR diff (
cli/src/claude/claudeRemoteLauncher.ts,cli/src/utils/MessageQueue2.ts, andcli/src/claude/claudeRemoteLauncher.launchFailure.test.ts) plus the prior bot review context. No blocker/major/minor findings met the confidence threshold. Residual risk: I did not execute the PR tests because this review treats PR code as untrusted.
Testing
- Not run (automation; PR content not executed).
HAPI Bot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MessageQueue2.collectBatch()acks a message at dequeue time — it firesonBatchConsumed, which the hub uses to mark the message consumed — before the message ever reaches the SDK. InclaudeRemoteLauncher, ifclaudeRemote()throws beforeonReady(spawn failure, invalid model/args, auth failure, resume-anchor rejection, or the process dying right after picking the message up), the catch block logs and retries. The message is gone: the hub believes it was delivered, but the CLI never acted on it.The retry side has its own gap. The respawn loop calls
claudeRemote()again immediately on every throw, with no backoff and no limit. When the failure is deterministic — bad auth, an invalid model, a missing binary — this respawns in a tight loop against the same failure indefinitely.The two interact, which is why this PR fixes both. Restoring the message without a retry limit would feed it straight back into the same deterministic failure on the next attempt, turning a silent loss into an unbounded storm.
Solution
Two commits, each self-contained.
1. Cap and back off consecutive launch failures. Track whether
onReady()fired at least once during an attempt, which distinguishes an immediate (deterministic) failure from one that occurs after real progress. Back off between immediate-failure retries, and after 3 consecutive immediate failures stop respawning: report it, drop the message that keeps triggering the failure, and reset the streak. The session stays alive, so a later unrelated message gets its own budget. The streak reset is gated on having reachedonReadyrather than applied on any non-throwing attempt — an attempt that merely parks a message intopendingand returns without progress must not clear the streak, or the cap can never fire. An attempt that did deliver a message and completed normally does reset it, including local-only commands like/clearthat finish without ever reachingonReady.2. Restore the in-flight message. Track the message returned from
nextMessage()as in-flight until the nextonReadyconfirms it was handled, and unshift it back to the front of the queue (viaunshiftIsolatedwhen it is an isolated message) if the attempt throws and will be retried.collectBatch()joins same-mode messages into one string, so the batch is restored item by item with each message's ownlocalIdpreserved, keeping the retried prompt tied to its hub row. On the cap path the message is dropped rather than restored, mirroringcursorLegacyRemoteLauncher's existing drop-and-reset policy on its own consecutive-failure cap.MAX_IMMEDIATE_RESPAWN_FAILURES(3) and the backoff (1s, overridable viaCLAUDE_REMOTE_RESPAWN_BACKOFF_MS) follow the same shape ascursorLegacyRemoteLauncher'sCURSOR_LEGACY_TRANSIENT_BACKOFF_MS. No new abstraction is introduced — the guard lives in the existing loop.Tests
cli/src/claude/claudeRemoteLauncher.test.ts, mockingclaudeRemote()at the subprocess boundary the waycursorLegacyRemoteLauncher.test.tsmocksspawn:onReadythrow is restored to the queue for the next attemptonReadyand then throws resets the streak, so a later failure run gets a fresh budget/compact,/clear) hitting a deterministic failure terminates at the cap rather than alternating between park and respawn foreverlocalIdinstead of collapsing into one id-less string/clearthat succeeds withoutonReadyclears the streak, so a later unrelated failure gets the full budget