Connected Services: a fully usage-limited pool with no stored reset time never re-checks, and the session stays parked until its account is switched by hand

[karolzlot]

Summary

When every member of a Connected Services pool is at a usage limit and the stored limit carries no reset time, no usage-limit recovery is armed and the group is never re-checked, so quota that becomes available again is never picked up and the session stays parked.

What happened (current behavior)

The group switch ended with no eligible fallback and the session reported that it was waiting for group recovery, yet no recovery intent existed for it, nothing re-examined the group for over an hour after one member became usable again, and the session only ran work again after its connected-service account was switched by hand.

Expected behavior

A session parked because every pool member is limited re-checks the group and resumes once a member is usable again, or settles terminally so the user learns that the session will not continue on its own.

Reproduction steps

1. On a self-hosted relay, run a Codex App Server session bound to a Connected Services pool that has several enabled members.
2. Keep the session running until the provider reports a usage limit for every member of the pool.
3. Read the session: lastRuntimeIssue is code: "usage_limit" with usageLimit.resetAtMs and usageLimit.retryAfterMs both null and usageLimit.recoverability: "switch_account", and the transcript carries the event Connected-service account group has no eligible fallback account; waiting for group recovery.
4. Run the session.usageLimit.checkNow action for that session: it answers session_usage_limit_recovery_inactive.
5. Make one pool member usable again out of band, for example by applying a provider-side reset to it or after the provider resets accounts on its own, and confirm through the profile quota that the member reports free capacity.
6. Observe the session: it stays active: true, runtimeActivityState: "idle", latestTurnStatus: "failed", no recovery attempt is logged for it, and a message sent from the app stays status: "queued" with deliveryState: "delivering" instead of running.
7. Switch that session's connected-service account to the member with quota: the runner restarts and a message sent afterwards runs within seconds.

Severity

high

Frequency

once

Happier version

0.2.11-dev.1

Platform

Windows 11, x86_64 (daemon and runner)

Server version

0.2.11-dev.1

Deployment type

self-hosted

What changed recently?

The quota of one pool member was restored on the provider side, out of band, instead of by waiting for a reset the daemon had stored. Providers also reset accounts on their own, which reaches the daemon the same way, that is not at all.

Diagnostics ID

No response

Additional context

Observed episode, the branch that settles recovery, and what stayed unverified

Observed on the running system, times relative to the limit.

T+0: the provider reported a usage limit for the active account, the reactive switch ended with no eligible member, and the session recorded the waiting for group recovery event. The stored lastRuntimeIssue carried source: "usage_limit", quotaScope: "account" and recoverability: "switch_account", with no reset time and no retry delay.

T+35 min: the quota of one member was restored out of band on the provider side, outside Happier, not by the window the daemon was tracking. The profile quota for that member then reported a session meter at 2 percent and a weekly meter at 0 percent, so the capacity was visible to the daemon's own refresh. Since the terminal result schedules nothing, an ordinary window rollover would be missed the same way.

T+50 min: a message sent from the app stayed queued with deliveryState: "delivering" and never ran. Two runner restarts requested from the app in that window were refused as busy with turn_in_progress, while the session itself reported runtimeActivityState: "idle", thinking: false and an empty work state. This is auxiliary evidence of the same boundary: the documented in-app recovery path was unavailable while the session was parked.

T+59 min: switching the session's connected-service account to the member with quota restarted the runner, and a message sent after that ran in seconds. The message that had been queued before the switch was written to the transcript but produced no turn of its own.

Derived from source at 8cf3c40, not confirmed by instrumentation. apps/cli/src/backends/codex/appServer/recovery/resolveCodexUsageLimitSwitchProgress.ts maps a no_eligible_member switch result to wait_until_reset only when resetAtMs or fallbackNextCheckAtMs is present, and otherwise returns exhausted. With neither value present that result is terminal, which matches both the missing recovery intent and the absence of any later attempt, and it means the daemon's own periodic profile quota refresh has no path to re-arm the session.

Not verified:

• whether upstream parses a wall-clock retry time out of this provider message shape; the provider text named a retry hour while the stored limit had no resetAtMs
• whether a stored reset time would have armed the wait in this pool shape, where every member is limited rather than only the active one
• the session was switched to another agent and back between the limit and the message, so the absence of a stored recovery intent cannot be attributed to the limit alone
• whether other providers settle the same way
• no run on a stock upstream build

related #348: single-member pool where the reset time is known and the wait is still not armed. That issue lists a pool with two or more members that are all limited at once as an open question; this run is that case, and nothing arms there either, but the input differs because no reset time was stored.

related #277: a policy-blocked recovery stored as waiting with no retry time. Here no intent is stored at all and the result is terminal by design.

[happier-bot]

Thanks for recording the unknown-reset case and the inactive check-now result. A shared presentation defect is corrected on dev in af1b427bfa, also discussed in #348, but this is only a partial resolution of this report.

The runtime-auth scheduler's durable wait was not consistently published into the session metadata used by recovery controls. The fix projects the actual daemon intent through the existing event-delivery owner, including subsequent retry/gate delays. It removes the unused report-derived metadata path rather than adding another retry loop. Regression coverage explicitly exercises an exhausted group with no reset timestamp and verifies that its existing scheduled recheck is exposed to the session.

One distinction from the proposed cause: the shared recovery disposition already uses backoff when the reset is unknown, and the Codex caller supplies a fallback check time. The helper's terminal branch alone does not establish why the historical session stopped retrying. The intervening agent changes also prevent attributing the later queued-message behavior solely to the original limit.

Focused source checks passed 147 tests across seven files. The CLI typecheck has two unrelated startupSideEffects test errors; no live Windows/provider-limit recovery was exercised. This commit is not yet a released fix.

Keeping this open: the missing controls projection is addressed in source, while the historical loss of recovery and queued-message behavior still need a deciding lifecycle trace. No Pending replay or provider restart workaround was added.

cc: @leeroybrun