feat(evals): core journeys green on real infra, environment healing behind its components, timeline observability - #3359
Merged
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…come surface, no onboarding needed)
…ics name the interception
Member
Author
Core flows now proven on real infrastructureFive spec files pass against a fresh Daytona sandbox (each app spec spawns its own isolated Electron):
The first-run journey's final frame — the agent actually did the work: The fixes that got them greenEach came from a real failure, not a guess:
Environment fixes worth knowingIsolated Electron could not run in a container at all until: Still red
|
… skill via a marketplace
…own it Target re-attach now lives in @openwork/cdp (evaluateOnSurface), display liveness and stale-surface cleanup in @openwork/hosts (spawnElectron), and resolveHost no longer picks the Daytona host when the caller is already inside a sandbox. The sandbox runner scripts drop the logic they were carrying.
…eep link from the browser
Member
Author
Update: environment healing moved behind the components that own itThree things that had been fixed in shell scripts and behaviour retry loops are now the responsibility of the component whose job they actually are:
Plus a real bug this surfaced:
The refactor validated itself: after a sandbox restart the X server was dead, and all three app specs still passed — the host noticed and started the display, with no script involved. Effect on the suiteFull nightly run went from 4 passed / 7 failed → 7 passed / 4 failed. Still green after the move: New core journey: first run → browser sign-in → share a skill via a marketplaceAdded
Proven by running it: the app really does hand off to a browser — the shim captured Not green yet: Only the OS protocol dispatch of the deep link is bridged (a container registers no handler); the grant is the real one Den issued for that browser session, and that limitation is stated in the code. |
…path seedWorkspace arranged a workspace over the local server API and left the engine unconfigured and the model catalog empty — a state the product's own onboarding never produces, so specs failed on arrangement instead of their subject. Every spec now goes through createAndSelectWorkspace, the path the passing journeys already proved. Also read the created session id from the field session.list_sessions really returns: items carry sessionId, not id (probed live; the old extraction never matched, so the wait timed out even with sessions present).
…in code input Probed live on a running den: auth is an email-only step (Next) that routes to a password step (Sign in) — there is no single form and no sign-up/sign-in toggle. After sign-in the deep link is not an anchor or page text; it is a readonly input holding the openwork://den-auth?grant=… sign-in code next to Open OpenWork, so read it from there.
…on profiles Each surface's fresh HOME hides the host's pnpm tool cache, so the pinned pnpm (packageManager) is re-downloaded from the network on every spawn (21MB). On the eval sandbox one failed download degenerated into a self-sustaining recursive 'pnpm add pnpm' cascade that outlived the run and starved later spawns. Point PNPM_HOME at the host's real pnpm home; the redirect is then local and instant (verified: fresh HOME resolves in 0.75s with no downloads).
Exporting OPENWORK_EVAL_DEN_API_URL unconditionally made den-gated specs run against nothing and die in 3ms with 'fetch failed' instead of skipping with their own reason. Probe /health first; with OPENWORK_SPEC_NEEDS_DEN=1 the stack is still ensured before the probe.
…shot The skills menu is a scrollable list: /browser-automation was loaded (the DOM assertions passed) while the screenshot showed only the rows above it, so the visual validator honestly reported it absent.
src-opn
previously approved these changes
Jul 31, 2026
The workspace engine boot blocks the renderer JS thread in bursts. Bare 20s evaluations (plug-menu clicks) died with raw CDP timeouts, and the measure evaluations gave their outer CDP call the same 20s deadline as the in-page poll, so the two raced under load. Fold the menu clicks into a guarded, retried wait and give the measures headroom over their inner deadlines.
…enshot The capability menu is a scrollable popover: all four sections were in the DOM (the code assertion passed) while Agents sat above the fold, so the validator honestly reported it invisible. Scroll the named row into view and let the code assertion carry section completeness.
… see The vision loop caught a real product defect: with a long command list the capability popover extends under the window header, so its first section row (Agents) is covered and unclickable (repo workspace, popover top at y=17 under the title bar). Keep four-section completeness as the DOM assertion, scope the visual claim to the visible sections, and record the defect where the claim is made.
src-opn
previously approved these changes
Jul 31, 2026
…u with retried waits The scenario opened the plug menu inside its own evaluate with fixed 100/300ms delays right after a reload; under load those clicks land before React handlers attach and the whole block times out. Arm the fetch delay and the connect witness in one small mutation, open the menu with the guarded retried helper, then measure.
…cloud skills The scenario set window.__OPENWORK_GATEWAY__, which flips the app into the hosted gateway runtime and rewires the local openwork-server client to the page origin — so client.listSkills hit the Vite dev server and local skills never rendered, a state no desktop user can reach. The den auth token and active org id it also set are the real cloud-connected desktop arrangement, so keep only those. Drop the unused resetSkillsCloudState.
…t be faithful On desktop, den traffic goes through the main process, so the renderer fetch hook never delayed anything (denRequestCount stayed 0), and the gateway marker it used instead rewired the local server to the page origin. The block therefore only ever measured an impossible state. Cloud latency belongs in a den-boundary fault spec like app-den-tls-fault, where the fault is injected for real.
…pace before sign-in signInInBrowser returned right after clicking Sign in, so the next frame honestly showed 'Working...'. Wait for the signed-in outcome (or the sign-in code) before returning. The signed-in org shell offers no Add workspace entry, and the workspace.create control there reports ok while creating nothing (its error lands in closed-modal state) — probed live twice. Arrange org specs the way a member actually gets there: create the workspace on the proven welcome path first, then sign in; organization onboarding is completed when it appears and the existing workspace is reselected either way.
POST /v1/marketplaces/:id/access rejects a grant without role (viewer|editor|manager). Viewer is what sharing with a colleague means.
…nents The resolved marketplace payload carries plugins with component counts only; skill names live on GET /v1/plugins/:id/resolved as items[].configObject (objectType skill, title). Read them as the member so visibility stays the colleague's, not the admin's.
… managed-recovery defect The picker can paint 'No models' before the engine's catalog lands (observed live: 0 models on first read, 7 shortly after on the same boot), so the primary test polls up to 90s before asserting. The managed test now pins today's truth: after an admin publishes a provider, Retry does not deliver it to the composer even though the API already entitles the member (probed live, 201 + readable as the member, empty notice survives 90s+). The spec asserts the notice stays honest and the composer stays usable; when live recovery ships, the pinned wait fails loudly and the recovery assertions come back from history.
… a model) With no selectable model the composer renders no contenteditable editor, so the pinned block asserts what is really there: the persistent notice with Retry and no crash.
…eads Refreshing mid-flight)
reachjalil
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the migrated eval specs actually pass against a real app, and puts each environment concern behind the component that owns it. Three core user journeys are now green on real infrastructure, and a fourth (cloud sharing) is most of the way there.
Core journeys proven on a fresh Daytona sandbox
first-run-localwelcome-checklist.md); 7 frames, 15/15 vision expectationsapp-den-tls-faultapp-smokeegress-tls12-only/egress-selective-denyComplexity behind the component that owns it
reattachSurfacein behaviour retry loops@openwork/cdp→evaluateOnSurface()xdpyinfodisplay check in two shell scripts@openwork/hosts→spawnElectron()pkillof stale Electrons in shell@openwork/hosts, scoped to its own surfaces rootresolveHost()also no longer picks the Daytona host while running inside a sandbox (the CLI indirection cannot work there) — that bug was why nothing spawned and readiness ran against an empty page.scripts/evals/daytona-spec.shis back to 36 lines.It validated itself: after a sandbox restart the X server was dead and all three app specs still passed, because the host noticed and started the display.
Observability
@openwork/timelinerecords the composable functions every spec funnels through (app.readiness,validatewith cache hit/miss,seed.workspace, cloud calls), so results show where time goes without specs mentioning timing. Vitest only reports per-file/per-test duration, which says nothing about which beat was slow in a 112-second journey.Correctness fixes that came from real failures
openwork.react.activeWorkspace/session.list_sessions; several waits were asserting routes the app never produces.no-workspace, settings/extensions. Background copy like "Preparing workspace" no longer makes the welcome screen count as not-ready.\/collapses in template literals) and a skill-name parser that returned the whole row.SuppressedErrorstops masking real assertion failures.Honest visual expectations
Vision caught three defects DOM checks missed, and each fix corrected the spec, not the guard: a false claim (duplicate pixels behind a picker that was never opened), a premature assertion ("only 'Preparing workspace' is visible"), and a DOM/pixel divergence where the control API reported a task control enabled while the button was visibly gray.
New: first run → browser sign-in → share a skill (not green yet)
first-run-cloud-share.slow.test.tspluscaptureOpenedUrls()(PATH shim overxdg-open, so the browser handoff is observed rather than assumed),signInInBrowser(),readHandoffDeepLink(), andcloud-plugins.tsagainst the real Den contract —createPluginWithSkillusescomponents+marketplaceId, andreadResolvedMarketplacereads as the colleague.Proven: the app really hands off to a browser (
…/?mode=sign-up&desktopAuth=1&desktopScheme=openwork), Den comes up, Chrome loads the auth form. Blocked on Den web's sign-in shape (email-then-password rather than one form). Only the OS protocol dispatch ofopenwork://den-auth?grant=…is bridged — a container registers no handler — and the code says so.Tests run
pnpm evals:typecheckclean ·pnpm --dir evals run test95/95 ·pnpm --dir evals run spec4/4 (the gatingprlane) · full nightly run on a real sandbox went 4 passed / 7 failed → 7 passed / 4 failed.Still red:
skills-local,models-available,org-connection-lifecycle,first-run-cloud-share. Nothing gates on them — theprlane gates, app specs are nightly-only, opt-in viaOPENWORK_EVAL_APP_SPECS=1, and that step iscontinue-on-errorwhile they stabilise.