Skip to content

Scheduled coverage still missing after #4848: native macOS crash-report assertions and updater download/install #4885

Description

@yomgui

Context

#4849 makes the scheduled Full regression report journeys this Linux lane cannot run as skipped: lane cannot satisfy prerequisites, and #4848 owns building the enterprise binary in that lane so the six packaged journeys execute. Even once #4848 lands, two coverage dimensions of those journeys remain unexecuted anywhere automatically, because unpacked Linux --dir coverage is not release/installer/native-macOS coverage. The second-opinion review of #4849 named both; this issue owns them.

1. Native macOS crash-report assertions (desktop-quit-path)

evals/specs/desktop-quit-path.e2e.test.ts:76–90 asserts "no crash report written" only when CRASH_REPORTS_OBSERVABLE (process.platform === "darwin", evals/worlds/desktop-quit-path.ts:31). On Linux the branch records Crash-report check skipped — needs: macOS DiagnosticReports, and only the exit code/signal is the witness. The negative half of the quit-path claim (#4787) — a clean quit leaves nothing in ~/Library/Logs/DiagnosticReports — is therefore never asserted by CI.

2. Updater download/install, not just the attempt (packaged-preactivation-updater)

evals/specs/packaged-preactivation-updater.e2e.test.ts:72–90 asserts that an activated enterprise install starts an update check (activity.checks > 0). Its own comment states why nothing more is asserted: the Den is unreachable, the network may refuse the manifest, and an unpacked Linux directory cannot self-update. So "the update was downloaded and applied, and the same profile boots afterwards" is not covered by this spec. released-enterprise-activated covers an in-place update path but only with a released baseline binary, which no lane provides (#4848).

Proposal

  • A macOS execution lane (self-hosted or hosted macOS runner, or a signed Daytona macOS host if/when available) that packages the enterprise flavor as a real installable artifact (electron-builder --mac), runs desktop-quit-path with CRASH_REPORTS_OBSERVABLE === true, and runs an updater journey against a controlled update feed (the world already exposes updaterActivity(); add a local feed serving a newer signed build) asserting download → install → relaunch → same profile boots.
  • Keep the Linux assertions as they are; do not weaken them to make the macOS lane easier.

Acceptance criteria (measurable)

  • A scheduled run on macOS lists desktop-quit-path as passed with the evidence line No crash report appears in … DiagnosticReports recorded (not the Crash-report check skipped line), for both fresh and activated installs and both stimuli (SIGTERM, Browser.close).
  • An updater journey on a packaged, installable artifact records passed with evidence that downloads > 0, the update was applied (version string changes across relaunch), and the pre-update profile still boots — not only checks > 0.
  • Both journeys declare their prerequisites in TestNeeds vocabulary (needs: { platform: "darwin" }, any feed/env var) and journey-catalog.mjs mirrors them so the parity guard in journey-ci.test.mjs passes; the Linux lane keeps reporting them as skipped with the reason, never as passed.
  • Run URLs for both are linked here.

Refs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions