Flaky: CodexModelsPerformanceTests workspace-scale budget too tight for CI runners

[gmkbenjamin]

Summary

CodexModelsPerformanceTests."workspace scale snapshot build stays within end to end budget" fails intermittently on swift-test-macos because its 1.5s wall-clock budget leaves too little headroom over what CI runners actually spend.

Observed on PR #2468, whose diff does not touch the measured code path:

✘ Test "workspace scale snapshot build stays within end to end budget" recorded an issue at
  CodexModelsPerformanceTests.swift:69:9: Expectation failed:
  (median → 1.535083428 seconds) < (.seconds(1.5) → 1.5 seconds)

Over budget by 35ms — 2.3%.

Evidence that it's runner variance

Same test, same measured code path, three runs within ~50 minutes of each other:

Run	Branch	Total test time	Result
30742339094	main	4.747s	✅
30745800653	fix/claude-startup-auto-cli	2.938s	✅
30745802243	feature/upcoming-resets-schedule	5.968s	❌

That's a 2× spread on identical work. The test runs 1 warmup + 3 measured builds and asserts the median is under 1.5s, so the main run at 4.747s total is already spending roughly 1.19s per build — only ~25% under the limit. A loaded runner closes that gap.

The same branch also passed the same shard a few hours earlier, at sha d2c20da3 (run 30738241213, all macOS shards green). The only Sources/CodexBarCore/ file touched between that green run and the red one is Providers/ProviderFetchPlan.swift — none of CodexLocalProjectUsageIndexer.swift, CodexLocalProjectUsageModels.swift, or CodexLocalProjectUsageProjection.swift changed. The measured code path is identical across the pass and the fail.

For contrast, the same test on local Apple Silicon:

✔ Test "workspace scale snapshot build stays within end to end budget" passed after 0.532 seconds

~0.13s per build — over 10× headroom. The budget is calibrated closer to dev hardware than to CI runners.

Why it doesn't self-heal

.github/workflows/ci.yml:150 sets CODEXBAR_TEST_RETRY_NON_TIMEOUT_FAILURES=0. The harness detects the failure (First-pass failed groups: 1) but only retries timeouts, so a 35ms perf blip fails the shard and the whole PR outright.

Suggested fixes

Roughly in order of preference:

1. Measure work, not wall-clock. Assert on something deterministic about the indexing (allocations, passes over the session set, or a counter of records processed) so the test catches real algorithmic regressions instead of runner weather. This is the only option that makes the test meaningfully immune to CI load.
2. Raise the budget to something like 3s. Cheap, preserves the "catches a catastrophic regression" intent, and still fails on a genuine order-of-magnitude slowdown.
3. Let this suite retry — exempt performance suites from CODEXBAR_TEST_RETRY_NON_TIMEOUT_FAILURES=0, so a single unlucky sample doesn't red-flag an unrelated PR.

Happy to send a PR for whichever direction you prefer.

Test introduced in 5c571d7 ("Test and document workspace indexing").

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 3, 2026, 6:38 PM ET / 22:38 UTC.

Summary
Current main and the shipped v0.47.0 release contain the exact runner-aware repair for this flaky test. The repository’s conservative personal-repository profile reserves the final closure action to the owner.

Reproducibility: no. Current main no longer has the 1.5-second median assertion: it uses the fastest of three runs and a 2-second budget scaled 3× on CI.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Include redacted logs or terminal output.

Maintainer decision needed

Question	Recommendation
Should the repository owner close this issue now that v0.47.0 ships the exact timing-budget repair?	Close as resolved: Confirm that the shipped best-of-three, CI-scaled timing guard resolves the reported runner-variance failure.

Why: The repository’s generic personal-repository profile does not permit automatic closure, despite strong source and release evidence.

Next step
No repair PR is needed; the owner must only confirm closure under the conservative repository policy.

Review details

Best possible solution:

Keep the shipped runner-aware regression guard and have the repository owner close this resolved report.

Do we have a high-confidence way to reproduce the issue?

No. Current main no longer has the 1.5-second median assertion: it uses the fastest of three runs and a 2-second budget scaled 3× on CI.

Is this the best way to solve the issue?

Yes. The merged repair preserves deterministic workspace-shape checks while separating algorithmic-regression detection from shared-runner scheduling noise.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• No implementation risk remains; only owner confirmation is required before closure under the repository’s conservative cleanup policy.

Codex review notes: model internal, reasoning high; reviewed against 47e1d0426236; fix evidence: release v0.47.0, commit fd7857a065b3.

Label changes

Label justifications:

• P3: The reported CI flake is already repaired and shipped; only low-urgency closure confirmation remains.
• impact:other: The report concerns CI test reliability rather than data, security, session, message, or provider behavior.

Evidence reviewed

What I checked:

• Current implementation: Current main measures the fastest of three snapshot builds, applies a 2-second budget through the CI-aware timing helper, and retains the 30-project/240-session shape assertions. (Tests/CodexBarTests/CodexModelsPerformanceTests.swift:68, 47e1d0426236)
• Fix provenance: The merged timing-budget commit explicitly replaces the hard 1.5-second median guard with best-of-three measurement and a 3× CI budget factor. (Tests/CodexBarTests/TestTimingBudget.swift:8, fd7857a065b3)
• Released source: The v0.47.0 tag contains the same fastest-of-three and runner-aware 2-second budget implementation now on main. (Tests/CodexBarTests/CodexModelsPerformanceTests.swift:68, 6a16c23313a7)
• Release provenance: The signed v0.47.0 tag and appcast publish the release after the fix commit; the affected test files have no diff between that tag and current main. (appcast.xml:6, 6a16c23313a7)

Likely related people:

• Peter Steinberger: Authored merged commit fd7857a and remains the release-era owner of the affected test files. (role: introduced runner-aware repair; confidence: high; commits: fd7857a065b3, 6a16c23313a7; files: Tests/CodexBarTests/CodexModelsPerformanceTests.swift, Tests/CodexBarTests/TestTimingBudget.swift)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[kiranmagic7]

This appears fixed on current main by #2509 (fd7857a).

The guard now measures the best of three runs, routes its 2-second base budget through TestTimingBudget with a 3× CI factor, and keeps the deterministic 30-project / 240-session shape assertions intact. All hosted checks on #2509 passed, including both macOS test shards.

I also verified fd7857a is an ancestor of current main (e17b14d) and ran:

• swift test --filter CodexModelsPerformanceTests — 1 test passed
• CI=1 swift test --skip-build --filter CodexModelsPerformanceTests — 1 test passed

No additional PR is needed; this looks ready to close.

[LeoLin990405]

This looks already resolved by #2509 (merged 2026-08-03, ~12h after this was filed).

#2509 introduced TestTimingBudget with a 3× slowdown factor on CI (isLoadedRunner ? 3 : 1) and reworked this exact test to assert the fastest of the three measured builds (the run least interrupted by co-scheduled work) against TestTimingBudget.scaled(.seconds(2)). On CI that's a 6s ceiling instead of the old 1.5s median — and TestTimingBudget's own doc comment cites "the workspace-snapshot budget … observed at ~1.1x its limit on a loaded machine," i.e. exactly this scenario.

Verified on current main (fd7857a0):

• local: ✔ passed after 0.535 seconds
• simulated CI (GITHUB_ACTIONS=true → 6s budget): ✔ passed after 0.551 seconds

The measured code path is unchanged, so the ~2.3%-over-budget blip described here can no longer trip the ceiling. Looks safe to close as fixed by #2509.