Skip to content

feat: integrate visual regression testing via Storybook test-runner + MCP screenshot tool#1640

Merged
jumpinjackie merged 5 commits intomasterfrom
copilot/integrate-visual-regression-testing
Apr 30, 2026
Merged

feat: integrate visual regression testing via Storybook test-runner + MCP screenshot tool#1640
jumpinjackie merged 5 commits intomasterfrom
copilot/integrate-visual-regression-testing

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 29, 2026

Establishes a self-hostable visual regression baseline ahead of the Blueprint.js replacement, so any component-level visual regressions are caught automatically. Also adds an MCP-callable screenshot tool so an agent can "see" a story in real time.

Visual regression testing (@storybook/test-runner + jest-image-snapshot)

  • .storybook/test-runner.jspostVisit hook captures #storybook-root per story and compares against stored PNGs via jest-image-snapshot. Waits for DOM attachment + short networkidle window (2 s, soft timeout) before snapping.
  • visual-snapshots/ — Baseline PNGs committed for all stable stories; these are the comparison targets for every visual regression run.
  • Failure threshold: 1% pixel difference (tolerates sub-pixel anti-aliasing variance).
// .storybook/test-runner.js
module.exports = {
  setup() { expect.extend({ toMatchImageSnapshot }); },
  async postVisit(page, context) {
    const root = page.locator("#storybook-root");
    await root.waitFor({ state: "attached", timeout: 10000 });
    await page.waitForLoadState("networkidle", { timeout: 2000 }).catch(() => {});
    expect(await root.screenshot({ animations: "disabled" })).toMatchImageSnapshot({
      customSnapshotsDir: "visual-snapshots",
      customSnapshotIdentifier: context.id,
      failureThreshold: 0.01, failureThresholdType: "percent",
    });
  },
};

Story exclusions via no-visual-regression tag

Stories that require a live MapGuide backend, WebGL, or external assets are tagged with tags: ['no-visual-regression'] and automatically skipped via --excludeTags no-visual-regression:

  • compact-viewer.stories.tsx — requires a real map tile server
  • map.stories.tsx — requires WebGL / live MapGuide
  • mapguide.stories.tsx — requires a live MapGuide backend

The _ImageIconSprite story was also fixed to use correct sprite class names (zoom-in, zoom-out-fixed, etc.) so it produces a stable baseline.

npm scripts

Script Purpose
yarn test:visual Compare against baselines (Storybook must be running on port 4000; excludes no-visual-regression stories)
yarn test:visual:ci Spin up Storybook + run tests in one command
yarn test:visual:update Regenerate all baselines (-u; excludes no-visual-regression stories)

MCP / agent screenshot tool (tools/capture-story.ts)

Dual-mode TypeScript script:

  • CLI: yarn capture:story --story "Common Elements/Button Variants" — navigates the Storybook iframe, waits for #storybook-root, saves PNG to /tmp/, prints path.
  • MCP server: yarn mcp:storybook — stdio JSON-RPC server exposing a capture_story tool that returns the screenshot as an inline base64 PNG image. Register in your MCP client config:
{
  "storybook-screenshot": {
    "command": "yarn",
    "args": ["mcp:storybook"],
    "cwd": "/path/to/mapguide-react-layout"
  }
}

Full MCP integration documentation (CLI flags, story path format, client registration snippets for Claude Desktop and VS Code/Copilot, tool parameter reference, and troubleshooting) is in docs_dev/content/VISUAL_REGRESSION.md.

CI (playwright.config.ts + .github/workflows/main.yml)

The visual-regression CI job has been commented out in .github/workflows/main.yml. CI produces inconsistent rendering results across environments, making automated baseline comparisons unreliable. Visual regression tests should be run locally as needed using yarn test:visual or yarn test:visual:update. The job definition is preserved in the workflow file (commented) so it can be re-enabled in the future.

Copilot AI and others added 2 commits April 29, 2026 14:07
… MCP screenshot tool

Agent-Logs-Url: https://github.com/jumpinjackie/mapguide-react-layout/sessions/8badf0ba-c860-4488-9c63-889363ddd622

Co-authored-by: jumpinjackie <563860+jumpinjackie@users.noreply.github.com>
…stories to exclude them from visual regression testing and fix icon sprite story

Co-authored-by: Copilot <copilot@github.com>
@jumpinjackie
Copy link
Copy Markdown
Owner

@copilot Update PR description in light of my changes added on top

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 29, 2026

@copilot Update PR description in light of my changes added on top

Updated the PR description to reflect your follow-up commit (a2d23c7). Here's a summary of what I added:

Your changes on top:

  • Tagged compact-viewer, map, and mapguide story files with tags: ['no-visual-regression'] to opt out of visual tests (these require a live MapGuide/WebGL backend and were the source of the 17 failing tests)
  • Added --excludeTags no-visual-regression to both test:visual and test:visual:update scripts so those stories are automatically skipped
  • Fixed the _ImageIconSprite story by using correct sprite class names (zoom-in, zoom-out-fixed, etc. instead of icon-zoom-in, etc.)
  • Removed the now-excluded compact-viewer baselines and re-baselined affected snapshots

The PR description now reflects the full combined state of both commits.

@jumpinjackie
Copy link
Copy Markdown
Owner

@copilot Add a new .md or update an exising relevant one to cover how to use the MCP integration

…guide; add to sidebar

Agent-Logs-Url: https://github.com/jumpinjackie/mapguide-react-layout/sessions/44c5c457-b6aa-43d8-b682-e603a07a4e18

Co-authored-by: jumpinjackie <563860+jumpinjackie@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 29, 2026

@copilot Add a new .md or update an exising relevant one to cover how to use the MCP integration

Updated docs_dev/content/VISUAL_REGRESSION.md (commit b4d216d) with a full MCP integration guide covering:

  • Story exclusion — how to tag stories with no-visual-regression to skip them, with the currently excluded files listed
  • CLI mode — all flags (--story, --url, --out) with a story-path-to-ID conversion table
  • MCP server mode — step-by-step registration snippets for both Claude Desktop (mcpServers config) and VS Code / GitHub Copilot (.vscode/mcp.json)
  • capture_story tool reference — parameter table and response format
  • Example agent interaction — shows the request/response flow
  • Troubleshooting table — covers the four most common failure modes

Also added VISUAL_REGRESSION.md to docs_dev/content/_sidebar.md under the Guide section so it appears in the docs navigation.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 29, 2026

Bundle Size Report ✅ PASS

Metric Size Gzip Brotli
Total 12.10 MB 2.58 MB 1.84 MB
Diff +7.90 KB 🔴 ↑ +1.54 KB 🔴 ↑ -377.00 B 🟢 ↓
Change +0.06% +0.06% -0.02%

Top Changes

File Before After Diff
~ vendor-debug.js 1.04 MB 1.04 MB -310.00 B 🟢 ↓
~ vendor.js 435.45 KB 435.39 KB -67.00 B 🟢 ↓

Generated by build-size-diff Commit: 50d3cf1

@jumpinjackie jumpinjackie marked this pull request as ready for review April 29, 2026 16:17
@coveralls
Copy link
Copy Markdown

coveralls commented Apr 29, 2026

Coverage Status

coverage: 60.427% (-0.02%) from 60.446% — copilot/integrate-visual-regression-testing into master

…configurations and updated test scripts

Co-authored-by: Copilot <copilot@github.com>
@jumpinjackie
Copy link
Copy Markdown
Owner

@copilot Disable (by commenting, not removing) the visual regression test step/job for CI. I cannot fathom why CI produces a different set of results and I don't want to waste anymore credits finding out why. I will manually visual regression test locally as needed.

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 30, 2026

@copilot Disable (by commenting, not removing) the visual regression test step/job for CI. I cannot fathom why CI produces a different set of results and I don't want to waste anymore credits finding ...

Done in commit 7a24cd7. The entire visual-regression job in .github/workflows/main.yml is now commented out with a note pointing to the local commands (yarn test:visual / yarn test:visual:update) for when you want to run them manually.

@jumpinjackie jumpinjackie merged commit ebb68e9 into master Apr 30, 2026
4 of 5 checks passed
@jumpinjackie jumpinjackie deleted the copilot/integrate-visual-regression-testing branch April 30, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants