Skip to content

fix(cli): don't lose the queued message when a remote launch fails#1058

Open
junmo-kim wants to merge 5 commits into
tiann:mainfrom
junmo-kim:fix/remote-launch-message-loss
Open

fix(cli): don't lose the queued message when a remote launch fails#1058
junmo-kim wants to merge 5 commits into
tiann:mainfrom
junmo-kim:fix/remote-launch-message-loss

Conversation

@junmo-kim

@junmo-kim junmo-kim commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

MessageQueue2.collectBatch() acks a message at dequeue time — it fires onBatchConsumed, which the hub uses to mark the message consumed — before the message ever reaches the SDK. In claudeRemoteLauncher, if claudeRemote() throws before onReady (spawn failure, invalid model/args, auth failure, resume-anchor rejection, or the process dying right after picking the message up), the catch block logs and retries. The message is gone: the hub believes it was delivered, but the CLI never acted on it.

The retry side has its own gap. The respawn loop calls claudeRemote() again immediately on every throw, with no backoff and no limit. When the failure is deterministic — bad auth, an invalid model, a missing binary — this respawns in a tight loop against the same failure indefinitely.

The two interact, which is why this PR fixes both. Restoring the message without a retry limit would feed it straight back into the same deterministic failure on the next attempt, turning a silent loss into an unbounded storm.

Solution

Two commits, each self-contained.

1. Cap and back off consecutive launch failures. Track whether onReady() fired at least once during an attempt, which distinguishes an immediate (deterministic) failure from one that occurs after real progress. Back off between immediate-failure retries, and after 3 consecutive immediate failures stop respawning: report it, drop the message that keeps triggering the failure, and reset the streak. The session stays alive, so a later unrelated message gets its own budget. The streak reset is gated on having reached onReady rather than applied on any non-throwing attempt — an attempt that merely parks a message into pending and returns without progress must not clear the streak, or the cap can never fire. An attempt that did deliver a message and completed normally does reset it, including local-only commands like /clear that finish without ever reaching onReady.

2. Restore the in-flight message. Track the message returned from nextMessage() as in-flight until the next onReady confirms it was handled, and unshift it back to the front of the queue (via unshiftIsolated when it is an isolated message) if the attempt throws and will be retried. collectBatch() joins same-mode messages into one string, so the batch is restored item by item with each message's own localId preserved, keeping the retried prompt tied to its hub row. On the cap path the message is dropped rather than restored, mirroring cursorLegacyRemoteLauncher's existing drop-and-reset policy on its own consecutive-failure cap.

MAX_IMMEDIATE_RESPAWN_FAILURES (3) and the backoff (1s, overridable via CLAUDE_REMOTE_RESPAWN_BACKOFF_MS) follow the same shape as cursorLegacyRemoteLauncher's CURSOR_LEGACY_TRANSIENT_BACKOFF_MS. No new abstraction is introduced — the guard lives in the existing loop.

Tests

cli/src/claude/claudeRemoteLauncher.test.ts, mocking claudeRemote() at the subprocess boundary the way cursorLegacyRemoteLauncher.test.ts mocks spawn:

  • a message dequeued and acked before a pre-onReady throw is restored to the queue for the next attempt
  • a deterministic failure stops respawning at exactly the cap, with the drop reported to the user, instead of spinning
  • backoff elapses between immediate-failure retries
  • an attempt that reaches onReady and then throws resets the streak, so a later failure run gets a fresh budget
  • an isolated message (/compact, /clear) hitting a deterministic failure terminates at the cap rather than alternating between park and respawn forever
  • a restored batch keeps each message's localId instead of collapsing into one id-less string
  • a /clear that succeeds without onReady clears the streak, so a later unrelated failure gets the full budget

claudeRemoteLauncher's respawn loop retried claudeRemote() immediately
on every throw with no backoff or limit. A deterministic launch
failure (bad auth, invalid model/args, spawn failure) respawned in a
tight loop instead of giving up, hammering the same failure forever.

Track whether onReady() fired at least once per attempt to tell an
immediate/deterministic failure apart from a failure after real
progress, back off between immediate-failure retries, and after 3
consecutive immediate failures drop the message that keeps triggering
them and reset the streak, instead of respawning forever. The session
keeps running so a later, unrelated message still gets its own budget.
The streak reset on a non-throwing attempt is itself gated on having
reached onReady, not applied unconditionally -- otherwise a message
that keeps getting parked and re-picked-up on alternating attempts
(e.g. an isolated command hitting the same deterministic failure)
would reset the streak every other attempt and the cap would never
fire.
…very

MessageQueue2.collectBatch() acks a message (fires onBatchConsumed,
which the hub uses to mark it consumed) at dequeue time, before the
message ever reaches the SDK. If claudeRemote() then throws before
onReady -- e.g. the process dies right after picking up the message --
the catch block only logged and retried, so the message vanished: the
hub already thinks it was delivered, but the CLI never acted on it.

Track the message returned from nextMessage() (whether freshly
dequeued or held in `pending` across a mode change) as in-flight until
the next onReady confirms it was handled, and restore it to the front
of the queue (preserving isolation via unshiftIsolated when needed) if
the attempt throws and will be retried. Restoring happens even if the
throw races with a user-initiated switch/exit, so a message is not
silently dropped by that unrelated shutdown either.

When the immediate-failure cap from the previous commit is reached,
the in-flight message is dropped instead of restored: unshifting it
back would just feed it into another immediate failure on the very
next attempt, storming again. This mirrors
cursorLegacyRemoteLauncher's existing drop-and-reset policy on its own
consecutive-failure cap.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Restored messages lose their hub identity — inFlightMessage only stores the combined text/mode/isolation state, but MessageQueue2.collectBatch() has already emitted messages-consumed for the original localIds before this point. Requeueing via unshift()/unshiftIsolated() without those ids means the retried prompt is no longer tied to the hub row, and if the retry later hits the cap the original web message remains marked consumed even though the launcher is explicitly dropping it. Evidence cli/src/claude/claudeRemoteLauncher.ts:423, context cli/src/utils/MessageQueue2.ts:374 and cli/src/agent/sessionBase.ts:76.
    Suggested fix:
    type InFlightMessage = {
        items: Array<{ message: string; mode: EnhancedMode; isolate: boolean; localId?: string }>;
    };
    
    const restoreInFlightMessage = () => {
        if (!inFlightMessage) return;
        for (const item of [...inFlightMessage.items].reverse()) {
            if (item.isolate) {
                session.queue.unshiftIsolated(item.message, item.mode, item.localId);
            } else {
                session.queue.unshift(item.message, item.mode, item.localId);
            }
        }
        inFlightMessage = null;
    };
  • [Major] Successful local-only turns do not reset the immediate-failure streak — the new reset only runs when onReady() fires, but claudeRemote() can consume and complete a message without ever calling onReady() for /clear (claudeRemote.ts:102) or any other successful path before the SDK result loop. After two startup failures, a successful /clear leaves immediateFailureCount at 2, so the next unrelated launch failure is treated as the third failure and drops that new message after one try. Evidence cli/src/claude/claudeRemoteLauncher.ts:485.
    Suggested fix:
    let deliveredMessageThisAttempt = false;
    
    // Set this in both branches that return a real message to claudeRemote().
    inFlightMessage = { message: msg.message, mode: msg.mode, isolate: msg.isolate };
    deliveredMessageThisAttempt = true;
    
    // After await claudeRemote(...) completes normally:
    if (reachedReadyThisAttempt || deliveredMessageThisAttempt) {
        inFlightMessage = null;
        immediateFailureCount = 0;
    }

Questions

  • None.

Summary

  • Review mode: initial
  • The retry cap addresses the respawn loop, but the current implementation still breaks queued-message identity on restore and can carry failure count across a successful local-only command. Both affect user prompts added by this diff's retry/drop behavior.

Testing

  • Not run: bun is not installed in this runner (/bin/bash: bun: command not found).

HAPI Bot

Comment thread cli/src/claude/claudeRemoteLauncher.ts Outdated
Comment thread cli/src/claude/claudeRemoteLauncher.ts Outdated
MessageQueue2.collectBatch() already collects each queue item's
localId (it fires onBatchConsumed with the full list to ack them), but
only exposed the joined `message` string to callers, discarding the
per-item localIds and their original boundaries in the process.

When claudeRemoteLauncher restores a dequeued-but-undelivered batch
after a launch failure, it re-added the joined string as a single new
queue item with no localId, orphaning the retried prompt from the hub
row(s) it originated from (and from cancel-by-localId).

Expose the pre-join `items` breakdown (message + localId per item)
alongside the existing joined `message` field on
collectBatch()/waitForMessagesAndGetAsString() -- purely additive, so
the other callers of waitForMessagesAndGetAsString() (grok, kimi,
opencode, cursor, codex, runAgentSession) are unaffected. On restore,
unshift each original item individually in reverse order, so the
localId and relative order of a multi-message batch are both
preserved instead of just the first item's.
…uccess

claudeRemote.ts's /clear handling delivers the queued message to the
SDK, then calls onSessionReset()/onCompletionEvent() and returns
successfully without ever calling onReady(). The success-path streak
reset only cleared on reachedReadyThisAttempt, so a successful /clear
between two unrelated immediate launch failures did not reset the
streak: an unrelated message's very next failure could hit the
3-in-a-row cap after just 1 failure, and the resulting banner would
misreport "3 times in a row".

Track whether nextMessage() actually handed a message to the SDK this
attempt (deliveredMessageThisAttempt), separately from whether the
attempt reached onReady, and reset the streak on either signal. The
livelock-prone case this guards against (a message parked into
`pending` and the attempt returning without ever delivering anything)
leaves both flags false, so it still does not reset the streak.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • No issues found.

Questions

  • None.

Summary

  • Review mode: follow-up after new commits
  • Prior bot findings around localId-preserving restore and successful no-onReady reset appear addressed in the latest diff. Residual risk: I did not execute the PR tests in this review environment.

Testing

  • Not run (automation; PR content not executed).

HAPI Bot

Upstream tiann#1056 added cli/src/claude/claudeRemoteLauncher.test.ts covering
resume-anchor / compact-outcome behaviour, colliding (add/add) with the
test file this branch introduced for launch-failure recovery.

Keep upstream's file verbatim and move this branch's suite to a sibling
file, claudeRemoteLauncher.launchFailure.test.ts, so the two independent
module-mock setups stay in separate vitest file scopes. The launcher
source auto-merged cleanly: tiann#1056 relocated consumeOneTimeFlags() into the
onSessionFound/onSessionReset callbacks, orthogonal to this branch's
respawn-storm guard and message-restore logic.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • No issues found.

Questions

  • None.

Summary

  • Review mode: follow-up after new commits
  • Reviewed the full current PR diff (cli/src/claude/claudeRemoteLauncher.ts, cli/src/utils/MessageQueue2.ts, and cli/src/claude/claudeRemoteLauncher.launchFailure.test.ts) plus the prior bot review context. No blocker/major/minor findings met the confidence threshold. Residual risk: I did not execute the PR tests because this review treats PR code as untrusted.

Testing

  • Not run (automation; PR content not executed).

HAPI Bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant