fix(runner): rebuild PR-pinned demos when the PR gets a new commit (DEV-3338)

[demtario]

Context

A demo built against an open PR (ht_version = the PR number) kept serving the PR's first build after the PR got new commits, after update_demo, and even for a brand-new demo with the same files. The cause is the build cache: the key was v2:<framework>:<PR number>:<filesHash> and builds are cached forever, but a PR number names whatever commit the PR has now. This blocks the common flow where a PR's demo link should show the PR's current behavior after review changes.

What this PR does:

• PR builds are cached per commit and install that commit. The commit comes from the x-commit-key header of a HEAD https://pkg.pr.new/handsontable@<pr>. The cache key becomes …:<pr>@<sha>:…, and the copy of package.json sent to the build container pins every pkg.pr.new/<pkg>@<pr> dependency to @<sha>. The stored source keeps the PR number; pinning the sha there would freeze the demo, because the version resolver ranks the package.json pin above the stored ref.
• Opening a PR demo refreshes it. On a document view of /d/<id>/ or /embed/<id>/, the worker compares the PR's current commit with the one the artifact was built from (new ht_built_sha column). If the PR has moved on, one view claims a rebuild through the existing BuildJob alarm, and every viewer (including one that lost the claim race) gets a self-refreshing "Updating to the latest commit (abc1234)" wait page (503) until the new build lands. Released versions and nightlies are untouched and never contact pkg.pr.new. A first build still running keeps its existing "still building" page.
• The claim is decided in D1, not from the KV-cached row. It is a single conditional UPDATE, so two viewers cannot start two builds. A commit that was already attempted is never claimed again (new ht_attempt_sha column), so a commit that fails to build costs one container, not one per view. A refresh stuck past the 30-minute stale window can be reclaimed, matching PATCH. The PR version itself is also read from D1, so a cached row cannot refresh a demo its owner already moved off the PR.
• Commit lookups: builds ask pkg.pr.new fresh. Views use a 60s KV copy (at most one HEAD per PR per minute) and confirm with a fresh HEAD before claiming, so a colo still holding an older commit cannot rebuild a demo backwards. If pkg.pr.new is unreachable on a view, nothing changes and the demo is served as before; a build that cannot learn the commit skips the cache rather than reuse the bare-number key that held the stale builds. A refresh check that throws also serves the current build, never a 500.
• Every build path records the commit it starts as attempted: MCP create (pending row), MCP PATCH (detached), and the view-triggered refresh. A view-triggered build goes through budgetGate as an anonymous caller.
• Side effect: update_demo with unchanged files now picks up a PR's new commit, with no hot-mcp change.
• Migration 0010_pr_build_sha.sql adds ht_built_sha and ht_attempt_sha. CI applies migrations before the API deploy (master.yml).

Out of scope, still tracked on DEV-3338:

• A commit sha as ht_version is still refused by the second validation (item 2).
• The Tier-1 live preview in the parcel environment (/edit, /share, and the javascript/typescript/react/react-js "Open live demo" links in the handsontable PR comment; vue runs on vue-cli and works) fails on pkg.pr.new builds. Measured locally, the cause is different from the ticket's: the identical ?./?? line works from registry 18.1.1 but fails when handsontable comes from a URL dependency, so widening HOT_BLOCKING_SYNTAX would not fix it (about 330 files would need downleveling).

What to expect after deploy:

• Every existing PR demo rebuilds once, on its first view (one container each).
• If a refresh build fails, viewers get the previous build again, with no notice.
• The wait page's 503 on d/:id / embed/:id is already excluded from the api-5xx-rate alert.

Types of changes

• New example
• Update to an existing example
• README / documentation change
• Demo runner (runner/) change
• CI / tooling change

How was this verified?

• New spec runner/pipeline/pr-demo-refresh.test.mjs (22 tests) covers:
  • the moved-commit view (wait page, exactly one scheduled job with the sha, row claimed, a second view schedules nothing);
  • a demo already on the PR's commit;
  • a failed commit not retried;
  • an owner rebuild not shown as a PR refresh;
  • stale-building reclaim;
  • a stale KV commit confirmed before claiming;
  • a cached row naming a PR the demo was moved off;
  • release versions never contacting pkg.pr.new;
  • budget denied, and pkg.pr.new unreachable;
  • the cache key and the container package.json pinned to the sha;
  • the BuildJob alarm recording ht_built_sha without contacting pkg.pr.new;
  • MCP create and PATCH recording the attempt;
  • an unknown commit skipping the cache, the claim loser waiting, a first build keeping its own page, a failed first build rebuilt for the next commit;
  • an unschedulable claim undone exactly (including updated_at), and a throwing refresh check serving the current build.
• Every guard was mutation-checked. I removed each one (D1 version check, attempt guard, stale reclaim, fresh confirm, fresh build lookup, PATCH and pending attempt) and confirmed the matching test fails. Against master's share.ts / index.ts / snapshot-jobs.ts, the behavioral tests fail and the guard tests pass.
• Claim SQL against real SQLite. I ran the conditional UPDATE with all migrations (0001–0010) applied in node:sqlite. First claim: 1 row. A second claim while building: 0. The same commit after a failure: 0. A new commit: 1.
• Full pipeline suite: pnpm test in runner/ passes 2717/2717. I traced the run to confirm no test makes a real request to pkg.pr.new; the one existing spec that started to is now stubbed.
• Typecheck: tsc --noEmit on workers/api passes, and so does pnpm -r run typecheck after pnpm build.
• pkg.pr.new behavior checked with curl -I: a HEAD returns x-commit-key with the full sha and access-control-allow-origin: *. @13766, @f66dd0d and @<full sha> return the same etag. The sha URL resolves for handsontable, @handsontable/react-wrapper, @handsontable/vue3 and @handsontable/angular-wrapper.
• Not verified in workerd: the real D1 meta.changes, the outbound HEAD from a Worker, and the DO path. Post-deploy check:
  1. Open https://demos.handsontable.com/d/1g4g311j4o/ (PR 13766). Expect one wait page, then the f66dd0d build.
  2. Then run wrangler d1 execute handsontable-demos --remote --command "SELECT build_status, ht_built_sha, ht_attempt_sha FROM demos WHERE id='1g4g311j4o'".

Checklist

• New/renamed example: added to runner/config/frameworks.json (see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.com (n/a, no example changed)
• New example: added a row to the tables in README.md (n/a)
• Ran pnpm build (and pnpm dev) in the affected example/server-example locally (runner packages built; pnpm dev not run for this change)

Related issue(s):

1. DEV-3338 Demo runner: PR-number builds go stale, a commit cannot be pinned, and /edit fails on handsontable ES builds

---

Note

Medium Risk
Changes the public demo view and build-cache paths, triggers container rebuilds from anonymous views, and depends on pkg.pr.new HEAD; mitigated by atomic D1 claims, graceful degradation, and broad pipeline tests.

Overview
Fixes DEV-3338: demos pinned to a PR number (ht_version = PR id) no longer stay on the first commit forever because the build cache keyed only on the PR number.

Build path: PR builds resolve the current commit via HEAD to pkg.pr.new (x-commit-key), use cache keys like 13766@<sha>, pin package.json deps to that sha in the container, and persist ht_built_sha / ht_attempt_sha (migration 0010). Unknown commits skip cache read/write so stale bare-number entries are not reused. MCP create/rebuild and snapshot jobs pass prSha through the same logic.

View path: Document loads on /d/ and /embed/ call refreshPrDemo, which compares live PR commit to ht_built_sha, atomically claims one rebuild in D1 (no double-schedule, no retry of failed attempts), schedules BuildJob, and returns a 503 “Updating to the latest commit” page—or keeps serving the last good build on budget/pkg.pr.new/DB failures.

Tests: New pr-demo-refresh.test.mjs plus harness stubs for pkg.pr.new HEAD and PR-refresh UPDATE semantics.

^{Reviewed by Cursor Bugbot for commit 09cede9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bca52eb. Configure here.}