ci(e2e): report Cypress retries in CI so flaky specs are visible

[fra-shipper]

Fixes #3024.

Root cause

cypress.config.ts sets retries: 3, so a spec that fails and then passes on retry reports the job green with no trace anywhere a human will look. #3023 is a concrete example: a genuine oauth_auth failure on windows-latest-3 was absorbed by a retry, and finding it required pulling and grepping job logs by hand. The existing after:spec handler discarded both of its arguments and only called killChainlit(), so this information was never captured.

Fix

Implements Option B from the issue (self-contained, no Cypress Cloud dependency, works on forked PRs):

1. cypress/support/retryReport.ts adds collectRetriedTests, a pure function that flags a test as retried when its overall state is passed but at least one attempt has state === 'failed'. Tests that exhaust every retry and still fail are excluded, since those are already visible via the job's exit status.
2. cypress.config.ts's after:spec handler now receives (spec, results), accumulates retried tests for the shard, and writes them to cypress/reports/retries.json after every spec (so a killed job still leaves partial data).
3. .github/workflows/e2e-tests.yaml uploads that file as a per-shard artifact (cypress-retries-<os>-<containers>, same pattern as the existing screenshot upload), and a new report-retries job downloads every shard's artifact, merges them, and writes a table to $GITHUB_STEP_SUMMARY.
4. When retries are found on a same-repo PR, the job also posts (or updates) a single PR comment. Top-level permissions: read-all is unchanged; the report-retries job itself is granted pull-requests: write, and .github/workflows/ci.yaml's e2e-tests: call site now also grants contents: read / pull-requests: write on the reusable-workflow job entry — a reusable workflow can only narrow permissions inherited from its caller, never widen them, so without this the grant inside e2e-tests.yaml would have had no effect even on same-repo PRs. On forked PRs GITHUB_TOKEN is read-only, so the comment step is skipped ahead of time based on pull_request.head.repo.full_name and additionally wrapped in try/catch as a fallback; it never uses pull_request_target. The step summary is always the primary, unconditional output.

The unrelated dead read-only-banner testid assertion in cypress/e2e/thread_resume/spec.cy.ts flagged in the issue as a separate item is left untouched, as the issue itself scopes it out.

Testing

• node --test cypress/support/retryReport.test.ts -> 3/3 pass (a test passing on first attempt is ignored, a test that fails then passes is reported with its attempt count, a test that fails every attempt is ignored). Verified this is a genuine regression test: temporarily reverted collectRetriedTests to a no-op returning [] (matching the old discard-everything after:spec behavior) and reran — the retry-detection case failed with an AssertionError as expected — then restored the real implementation and confirmed 3/3 pass again.
• npx prettier --check on all touched files -> clean.
• npx eslint on cypress.config.ts, cypress/support/retryReport.ts, cypress/support/retryReport.test.ts -> clean.
• npx tsc --noEmit -p tsconfig.json -> no new errors from this change (only 4 pre-existing TS2428 WeakMap errors from Cypress's vendored lodash types, confirmed present identically on a clean checkout via git stash -u, and unrelated to this change — this tsconfig is also not wired into any CI script; pnpm type-check only recurses into the frontend/libs workspace packages).
• actionlint on the modified workflow files -> clean, exit 0.
• The modified e2e-tests.yaml was parsed with yaml.safe_load to confirm it still parses correctly and the job list is ['prepare', 'e2e-tests', 'report-retries'] as intended.
• The permissions: fix in ci.yaml (a reusable-workflow call site can only narrow, never widen, the caller's inherited permissions) was verified by re-reading the affected files against GitHub's documented permission-inheritance semantics for workflow_call, since that runtime behavior cannot be exercised locally or with actionlint (which validates syntax, not inheritance semantics).
• Husky's pre-commit hook (lint-staged, which runs format:files, lint:fix, and actionlint on staged files) ran on every commit and made no further modifications, confirming the diff was already compliant.
• Not run: pnpm test:e2e (the full Cypress suite), since it requires the full Chainlit backend and a live browser; the pure retry-detection logic is covered by the node:test regression test instead, and the artifact-upload/aggregation job will be exercised directly by this repo's own e2e-tests CI workflow on this PR.

---

Summary by cubic

Makes Cypress retries visible in CI so flaky specs no longer fail silently (fixes #3024). Previously, a spec that failed on one attempt and passed on retry reported a green job with no trace anywhere a human would look — #3023 is a real oauth_auth failure that was hidden this way.

• Records tests that failed at least one attempt but still passed and writes them to cypress/reports/retries.json per shard.
• Uploads that file as an artifact per shard; a new report-retries job merges them into a $GITHUB_STEP_SUMMARY table.
• Posts or updates a PR comment on same-repo PRs; forked PRs skip the comment since the token is read-only.
• Adds pull-requests: write at the ci.yaml call site so the grant passes through the reusable workflow (permissions can only be narrowed, never widened).

^{Written for commit bb55b2f. Summary will update on new commits.}

[cubic-dev-ai]

6 issues found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tsconfig.json">

<violation number="1" location="tsconfig.json:9">
P2: By excluding cypress/support/retryReport.test.ts from the project tsconfig, the new test file is removed from all static type-checking. It only runs under node --test, which strips types without checking them, so type errors in the test (e.g. the loosely-typed asTests casts) can land in CI undetected. Keep the test type-checked instead of excluding it: give the test its own tsconfig that enables allowImportingTsExtensions (with a noEmit check), or include it with that flag set, so it stays covered without breaking the main compile.</violation>
</file>

<file name=".github/workflows/e2e-tests.yaml">

<violation number="1" location=".github/workflows/e2e-tests.yaml:124">
P2: When a later PR run has no retried tests, this early return leaves the previous marker comment untouched. Remove or update the existing marker comment on the zero-retry path instead of returning before comment reconciliation.</violation>

<violation number="2" location=".github/workflows/e2e-tests.yaml:155">
P3: The PR comment reports `retries.length` as "test attempt(s)", but each entry represents one retried test and can contain multiple retry attempts. Say "test(s)" or compute the total retry attempts.</violation>

<violation number="3" location=".github/workflows/e2e-tests.yaml:158">
P2: When a PR has more comments than the first API page, this call may not find the marker and creates a duplicate report comment. Paginate the comments before searching so the workflow maintains one comment.</violation>
</file>

<file name="cypress/support/retryReport.test.ts">

<violation number="1" location="cypress/support/retryReport.test.ts:9">
P2: This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs `node --test cypress/support/retryReport.test.ts`, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a `pnpm test` script entry) so it actually runs on every PR.</violation>
</file>

<file name="cypress.config.ts">

<violation number="1" location="cypress.config.ts:67">
P2: When `cypress open` runs a spec, `results` is undefined for `after:spec`; this dereference throws before `killChainlit()` runs. Guard the collection when results are unavailable.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: By excluding cypress/support/retryReport.test.ts from the project tsconfig, the new test file is removed from all static type-checking. It only runs under node --test, which strips types without checking them, so type errors in the test (e.g. the loosely-typed asTests casts) can land in CI undetected. Keep the test type-checked instead of excluding it: give the test its own tsconfig that enables allowImportingTsExtensions (with a noEmit check), or include it with that flag set, so it stays covered without breaking the main compile.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tsconfig.json, line 9:

<comment>By excluding cypress/support/retryReport.test.ts from the project tsconfig, the new test file is removed from all static type-checking. It only runs under node --test, which strips types without checking them, so type errors in the test (e.g. the loosely-typed asTests casts) can land in CI undetected. Keep the test type-checked instead of excluding it: give the test its own tsconfig that enables allowImportingTsExtensions (with a noEmit check), or include it with that flag set, so it stays covered without breaking the main compile.</comment>

<file context>
@@ -5,5 +5,6 @@
   },
-  "include": ["cypress/**/*.ts", "cypress.config.ts"]
+  "include": ["cypress/**/*.ts", "cypress.config.ts"],
+  "exclude": ["cypress/support/retryReport.test.ts"]
 }
</file context>

[cubic-dev-ai]

P2: When a later PR run has no retried tests, this early return leaves the previous marker comment untouched. Remove or update the existing marker comment on the zero-retry path instead of returning before comment reconciliation.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/e2e-tests.yaml, line 124:

<comment>When a later PR run has no retried tests, this early return leaves the previous marker comment untouched. Remove or update the existing marker comment on the zero-retry path instead of returning before comment reconciliation.</comment>

<file context>
@@ -73,3 +73,112 @@ jobs:
+                .addHeading('Cypress retries', 3)
+                .addRaw('No specs needed a retry in this run.')
+                .write();
+              return;
+            }
+
</file context>

[cubic-dev-ai]

P2: When a PR has more comments than the first API page, this call may not find the marker and creates a duplicate report comment. Paginate the comments before searching so the workflow maintains one comment.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/e2e-tests.yaml, line 158:

<comment>When a PR has more comments than the first API page, this call may not find the marker and creates a duplicate report comment. Paginate the comments before searching so the workflow maintains one comment.</comment>

<file context>
@@ -73,3 +73,112 @@ jobs:
+            const body = `${marker}\n**${retries.length} Cypress test attempt(s) were retried in this run.** See the job summary for details.`;
+
+            try {
+              const { data: comments } = await github.rest.issues.listComments({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
</file context>

Suggested change

	const { data: comments } = await github.rest.issues.listComments({
	owner: context.repo.owner,
	repo: context.repo.repo,
	issue_number: pullRequest.number
	});
	const comments = await github.paginate(
	github.rest.issues.listComments,
	{
	owner: context.repo.owner,
	repo: context.repo.repo,
	issue_number: pullRequest.number
	}
	);

[cubic-dev-ai]

P2: This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs node --test cypress/support/retryReport.test.ts, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a pnpm test script entry) so it actually runs on every PR.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cypress/support/retryReport.test.ts, line 9:

<comment>This regression test is never executed: nothing in any CI workflow or npm/pnpm script runs `node --test cypress/support/retryReport.test.ts`, and tsconfig.json excludes the file. The retry-detection logic therefore has no automated coverage in CI, so a future change to collectRetriedTests can silently regress. Wire the test into CI (e.g. a workflow step or a `pnpm test` script entry) so it actually runs on every PR.</comment>

<file context>
@@ -0,0 +1,63 @@
+const asTests = (tests: unknown) =>
+  tests as CypressCommandLine.RunResult['tests'];
+
+test('ignores a test that passed on its first attempt', () => {
+  const tests = asTests([
+    {
</file context>

[cubic-dev-ai]

P2: When cypress open runs a spec, results is undefined for after:spec; this dereference throws before killChainlit() runs. Guard the collection when results are unavailable.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cypress.config.ts, line 67:

<comment>When `cypress open` runs a spec, `results` is undefined for `after:spec`; this dereference throws before `killChainlit()` runs. Guard the collection when results are unavailable.</comment>

<file context>
@@ -41,12 +56,21 @@ export default defineConfig({
 
-      on('after:spec', async () => {
+      on('after:spec', async (spec, results) => {
+        retriedTests.push(...collectRetriedTests(spec.relative, results.tests));
+        await mkdir(dirname(RETRY_REPORT_PATH), { recursive: true });
+        await writeFile(
</file context>

Suggested change

	retriedTests.push(...collectRetriedTests(spec.relative, results.tests));
	if (results)
	retriedTests.push(...collectRetriedTests(spec.relative, results.tests));

[cubic-dev-ai]

P3: The PR comment reports retries.length as "test attempt(s)", but each entry represents one retried test and can contain multiple retry attempts. Say "test(s)" or compute the total retry attempts.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/e2e-tests.yaml, line 155:

<comment>The PR comment reports `retries.length` as "test attempt(s)", but each entry represents one retried test and can contain multiple retry attempts. Say "test(s)" or compute the total retry attempts.</comment>

<file context>
@@ -73,3 +73,112 @@ jobs:
+            if (!pullRequest || isFork) return;
+
+            const marker = '<!-- cypress-retry-report -->';
+            const body = `${marker}\n**${retries.length} Cypress test attempt(s) were retried in this run.** See the job summary for details.`;
+
+            try {
</file context>

Suggested change

	const body = `${marker}\n**${retries.length} Cypress test attempt(s) were retried in this run.** See the job summary for details.`;
	const body = `${marker}\n**${retries.length} Cypress test(s) were retried in this run.** See the job summary for details.`;