Connected Services: a Codex session that adopts a pool generation while idle keeps using the previous account and parks on the next turn

[karolzlot]

Summary

When a Connected Services pool rotates because another session hit a usage limit, an idle Codex session adopts the new member in its metadata and credential file but keeps serving on the previous account, so its next turn fails and the pool selector parks it with no_eligible_member while the new member is healthy.

What happened (current behavior)

The idle session's binding advanced to the new pool member and its CODEX_HOME/auth.json was rewritten to that member's account, but its running Codex app-server received no account/login/start, so the next turn was answered by the previous account and rejected with usage_limit_exceeded, and the failure was then reported under the new member's profile id.

Expected behavior

A session that adopts a new auth generation serves its next turn on the newly selected member, and a usage-limit failure is attributed to the account that actually served the request.

Reproduction steps

1. On a self-hosted relay, put at least two Codex profiles from different ChatGPT accounts into one Connected Services pool, with distinguishable plans so the serving account is identifiable from the provider rate_limits payload.
2. Start two Codex sessions on that pool in appServer backend mode. Call them A and B.
3. Run session A until a turn completes, then leave it idle.
4. Run session B until the provider reports a usage limit, so the daemon rotates the group to the next member and advances the group generation.
5. Confirm session A now reports the new member in its metadata and that its CODEX_HOME/auth.json holds the new member's account.
6. Send a message to session A. The turn fails with a usage limit whose reset time and plan type belong to the previous account, and the daemon logs the switch attempt as "resultStatus":"no_eligible_member" with the new member excluded as current_active.

Severity

medium

Frequency

once

Happier version

0.2.11-dev.1

Platform

Linux x64 (Debian 13 container), Codex CLI 0.147.0, appServer backend mode

Server version

0.2.11-dev.1

Deployment type

self-hosted

What changed recently?

No response

Diagnostics ID

No response

Additional context

Observed timeline, working control, supporting evidence and remaining uncertainty

T0 is the moment the affected session's group binding advanced to the new member. The session had ended its previous turn about 7 minutes before T0 and was idle. Its Codex app-server process started about 55 minutes before T0 and was never restarted.

Observed:

• T0: the affected session's group binding advances to the new member and the group generation increments. The rotation was triggered by the other session's usage limit, not by this one.
• T0+14s: its CODEX_HOME/auth.json is rewritten. The credential's own account id and its chatgpt_plan_type claim both belong to the new member, so the file on disk is correct.
• Around the same second, Codex's app-server request trace for that process shows only account/read and account/rateLimits/read. No account/login/start is received before, during or after the swap.
• T0+2m06s: the next turn is rejected with usage_limit_exceeded. The provider payload carries the previous account's plan_type and a 5 hour window reset. The new member has no 5 hour window at all and its weekly window resets four days later, so the response could not have come from it.
• T0+2m08s: the runner reports the failure with the new member's profile id. The selector excludes that member as current_active on fresh quota evidence showing it not exhausted, marks every other member quota_exhausted, and returns no_eligible_member. The recovery record is armed as waiting with a wake time about 3h48m later, taken from an unrelated member's reset.

Working control, same pool, same generation, same host, same CLI version and same Codex binary: the session that hit the limit itself adopted the new member inside its existing app-server process without a restart, and its next turn 6 seconds later carried the new member's plan type and weekly window. The credential apply mechanism therefore works; the divergence is specific to a session that adopts a generation it did not trigger.

Supporting evidence for the same boundary: at T0+2s, 12 seconds before the credential file was written and 2 minutes before the turn that the previous account actually served, the runtime identity probe for the affected session already reported the new member's provider account id with "probeStatus":"verified" and "proofKind":"runtime_exact". The affected Connected Services corridor in the observed build is identical to public 0.2.11-dev.1 at 8cf3c40, where that probe returns the identity cached from the applied credential (source: 'applied_credential') and only refreshes it from the live account when the cached value does not match the refresh selection, so an identity that matches intent but not the live session is returned unverified against the provider.

Remaining uncertainty: whether the credential apply on the idle session reported success or failure is not recoverable, because the RPC result payload is not logged on either side and the daemon records no switch attempt for a session that only observes a new generation. Codex's own log store had already pruned the control session's window, so the presence of account/login/start in the working path was not verified in the same artifact.

Workaround: sending another message after the failure runs on the new member, because the failed turn resets the Codex session first. Until then the session stays parked.

Related: #276 and #277 also end with a session that never resumes, but both stop at the pool policy phase with auto_switch_disabled and a recovery record that carries no wake time. Here the policy allows the switch, the record does carry a wake time, and the session parks because the failure is attributed to a member that did not serve the request.

[happier-bot]

Thanks for the idle-versus-triggering-session comparison, especially the distinguishable provider quota windows. The reported behavior is not correct: an adopted pool generation must be applied to the live app-server before the next turn uses it, and a failure must remain attributed to the account that served that turn.

The root cause is not yet established. In the canonical source path inspected, account/login/start is awaited before the applied_credential identity is recorded and before credential persistence. That does not explain the reported trace with no login request, so changing the identity cache or restarting the app-server would be a speculative fix rather than a demonstrated correction.

The existing concurrency test also does not settle this: it starts an active prompt first, so it does not prove the idle-recipient case. The deciding check is to hold the live-login response at the app-server boundary during idle fan-out, submit the next prompt, and verify both request ordering and the failure's captured account identity. The recipient apply result and loaded runner version are needed to reconcile that with the historical trace.

No speculative auth-switch change was committed for this issue. The recovery-presentation correction linked in #348 and #349 does not resolve this one. Keeping it open for the idle fan-out investigation; no provider restart, unconditional extra identity polling, or transcript replay was added.

cc: @leeroybrun

[karolzlot]

Still reproducible on Happier dev at af1b427 with Codex 0.153.4. A minimal reproduction with stock Codex CLI 0.147.0 and 0.153.4: complete a turn on A over WebSocket, leave the thread idle, await account/login/start for B and confirm B with account/read, then start another turn. The request reuses the socket authenticated as A. HTTP uses B correctly. If the preceding turn instead ends with a provider usage-limit error, Codex discards the socket and the next turn opens a new one on B, reproducing the idle-versus-failing-session difference.

The attribution error also reproduces at Happier's runtime boundary with the real Codex process and a local mock provider: generation apply succeeds for B, but A's usage-limit error on the old socket is classified under B's profile and generation. The full daemon fan-out and final pool-selection result were not exercised. Codex's connection reuse check does not compare the connection's authenticated account with the newly applied auth.

Correction to the earlier diagnostic claim: absence of account/login/start from the retained SQLite log does not establish that it was never received. This reproduction fails despite a successful login response and an account read confirming B.

[happier-bot]

The new stock-Codex reproduction establishes the immediate cause with much higher confidence: this is an upstream Codex app-server transport-cache defect exposed by a successful Happier hot-auth change.

Codex's session-scoped ModelClientState retains a cached WebSocket session across account/login/start. The login operation updates the auth manager, but WebSocket reuse does not compare the connection's authenticated account with the newly selected account. That explains the observed split: HTTP uses B, an idle session's retained WebSocket still uses A, and a preceding provider error appears to recover because Codex discards that socket and creates a new one.

Happier currently has no safe way to invalidate only that cache without restarting Codex. The socket and internal force_http_fallback operation are private to Codex, and the app-server API exposes no cache-invalidation RPC. account/logout is not an acceptable substitute: it does not provide the required socket contract and may revoke credentials. Restarting app-server is also rejected because it can break thread/runtime continuity and is unnecessary once Codex invalidates the connection at the correct boundary.

The appropriate upstream correction is for Codex to invalidate/drain the cached response WebSocket at a safe turn boundary when authentication changes, or key reuse by the authenticated identity/generation. An app-server RPC exposing that safe operation would also give hosts an explicit boundary. Happier should continue hot-materializing and applying credentials; it should not add a parallel HTTP provider, transcript replay, or restart workaround.

The Happier-side attribution/recovery fencing remains important, but it cannot make a socket authenticated as A start serving B. Keeping this issue open while the upstream capability is reported/tracked is appropriate.

Thanks for the A/B probe and the provider-error control case; they resolved the ambiguity left by the original retained-log evidence.

cc: @leeroybrun

[karolzlot]

Added the stock-Codex app-server reproduction and HTTP/provider-error controls to the related upstream report: openai/codex#32173.

[karolzlot]

A fix for the stale WebSocket reuse is available in Codex 0.155.0-alpha.10. I didn't test it yet.

openai/codex#44489, merged September 10, resets cached WebSocket and incremental response state when auth ownership changes. It includes regression tests for account switches within and between turns. I verified that the alpha release contains the fix; the current stable 0.154.0 does not.