Session mutation transport failures cause zombie sessions, duplicate responses, and unmanageable session list

[Zenine]

Summary

Three symptoms traced to the same root cause: session_turn_mutation_transport_unavailable — the local-to-cloud session mutation sync layer fails silently and never recovers, producing zombie sessions, duplicate responses, and an unmanageable session list.

Related: #272 (stale transcript on open) — likely the same transport failure from the reader's side.

Environment

• Happier CLI: 0.2.10-dev.83 (dev channel)
• Platform: Linux (aarch64, Ubuntu 24.04)
• Connection: stable (Codex sessions on the same machine have zero issues)

Symptoms

1. Zombie sessions — alive process, dead UI

Sessions stay "alive" (process running, claude child process present) but become completely unresponsive in the UI. Their logs freeze at a single line:

[API] Authoritative session mutation remains queued and is blocking later mutations
  mutationKind: "session_turn"
  deliveryStatus: "retryable"
  reason: "session_turn_mutation_transport_unavailable"

At time of investigation, 6 out of 7 active dev-channel sessions were in this state, with log ages ranging from 7 to 39 hours. These sessions cannot be stopped or archived from the UI — they just sit in the list indefinitely.

2. Duplicate responses

The same assistant response renders multiple times in the UI. Mechanism from logs:

• A completed turn's mutation fails to deliver (server returns HTTP 404)
• Client marks it retryable and retries indefinitely
• Worst case observed: 11,970 attempts over 4.2 days for a single mutation
• When transport briefly recovers, the queued mutations flush in bulk
• UI receives and renders multiple copies of the same turn

Evidence from daemon log:

mutationId: "daemon-observed-exit:b686a332..."
attempts: 11970
ageMs: 362319353  (4.2 days)
deliveryDiagnostic.http.status: 404

3. Dead session records never cleaned up

When the daemon restarts (e.g. --takeover on self-update), it sends SIGTERM to child sessions. Some sessions are not reattached by the new daemon:

[DAEMON RUN] Startup reattach alive marker scan finished {aliveMarkerCount: 2}
[DAEMON RUN] Reattached 2 sessions from disk markers

But if 4 sessions were tracked before the restart, the other 2 become orphans: process dead, no exit log written, but their daemon-sessions.dev/pid-*.json record persists forever. The UI shows them as sessions that can't be stopped or archived.

4. UI regression — sorting and swipe gestures removed

Previously available features that helped work around the above issues are now gone:

• Sorting (could push unresponsive sessions to the bottom)
• Swipe-left to close/archive (quick way to dismiss dead sessions)

Without these, the only way to clean up is manually deleting files from ~/.happier/tmp/daemon-sessions.dev/.

Root cause analysis

The session_turn_mutation_transport_unavailable error indicates the HTTP transport to api.happier.dev returns 404 for session mutations. This affects 71% of all session logs (44 out of 62) on this machine, suggesting a systematic issue rather than transient network failure.

The daemon log also shows:

[WARN] Connected-service quota policy remains disabled because startup hydration failed
  error: "Failed to list connected service auth groups (404)"

This 404 from the /v3/connect/claude-subscription/groups endpoint may be related — if the server doesn't recognize the session/subscription, mutations would also fail.

Key architectural difference vs Codex: Codex has no "every turn must sync to a remote state machine" layer. Its sessions are purely local (process + .jsonl), so none of these failure modes exist.

Steps to reproduce

1. Start multiple sessions via Happier dev channel
2. Use them normally for a few hours
3. Observe that older sessions' logs freeze at the mutation_transport_unavailable message
4. Try to interact with those sessions in the UI — no response
5. Try to stop or archive them — no effect
6. Open a session after transport reconnects — same response may appear multiple times

Expected behavior

• Transport failures should have a bounded retry with backoff, not infinite retry
• After N failures, the mutation should be dropped or the session marked as degraded (local-only mode)
• Dead sessions (process gone) should be reaped automatically, not left as permanent UI ghosts
• Sorting and swipe-to-dismiss should be available for manual cleanup

Suggested fixes

1. Bounded retry + circuit breaker for session mutations — stop after N attempts, fall back to local-only
2. Periodic liveness scan in the daemon — check if tracked session PIDs are still alive, remove dead records
3. Idempotent mutation delivery — deduplicate on the server side to prevent duplicate rendering even if retried
4. Restore sort and swipe gestures in the session list UI

[Zenine]

Additional finding: regression introduced in dev channel, preview was clean

Checked all local logs across both channels:

• Preview era (0.2.2-preview, before ~Aug 21): zero occurrences of session_turn_mutation_transport_unavailable across all session and daemon logs, including a 26MB session log spanning weeks of heavy use.
• Dev era (0.2.10-dev.80 → 0.2.10-dev.83, Aug 21 onwards): 32 session logs hit this error, starting from the very first dev-channel session on Aug 21.

The preview version did have server-side errors (push token fetch returning 500), but those were isolated to notifications and never blocked session mutations or caused zombie sessions.

This suggests the session_turn_mutation_transport_unavailable failure mode is a regression introduced somewhere between 0.2.2-preview and 0.2.10-dev.80 — possibly a change in the session mutation transport layer, a new server contract (released_server_v0_2_1), or stricter mutation ordering that doesn't degrade gracefully.

[happier-bot]

Thanks @Zenine — the version comparison and transport logs were decisive here.

We confirmed that the dev CLI was connecting to the exact released v0.2.1 server contract, which supports the legacy transcript transport but does not expose the session turn-mutation route. The CLI was nevertheless treating those turn mutations as immediately deliverable authoritative work. Their repeated failures could keep the durable outbox active and block compatible transcript delivery.

The source fix is in 5f4390868f. Against that known released contract, turn mutations now remain durable but parked: they consume no retry attempts, request no reconnects, do not block the supported transcript path, and replay when a capable server is observed.

We did not find evidence that retrying the turn mutation itself creates duplicate assistant rows—the turn and transcript write paths use stable idempotency identities. We’re therefore keeping the duplicate-response, stale-marker, and session-list observations separate rather than applying speculative fixes.

Once this correction reaches a dev build, please retry the affected flow. If duplicates or stale markers remain, please provide a fresh diagnostic ID plus the affected session ID and approximate time so we can trace those paths independently.

cc: @leeroybrun