Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .agents/skills/verify-toolkit/map/mcp-apps.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ the host theme, and calls the tool again through the host. The goals are in
- `view-theme` The view follows the host's light or dark theme. Proof: `drive`.
- `range-roundtrip` The range picker calls `render_sentry_chart` through the host with the new period and redraws without remounting. Proof: `drive`.
- `open-link` "Open in Sentry" asks the host to open the result's Sentry URL with `ui/open-link`. Proof: `drive`.
- `grouped` A `groupBy` result draws one series per top group, with a legend. Proof: `drive`.
- `grouped` A `groupBy` result draws one series per top group, with a legend that shows each group's short label and event count, and no peak, since one group's bucket would read as the chart's peak. Proof: `drive`.
- `top-by-volume` A grouped chart of a non-additive aggregate, such as p95 by release, plots the groups with the most events, not the sparse groups with the highest values. Proof: `drive`.
- `null-gaps` A bucket with no data is null in the result, not 0, and the view draws it as a gap; a bucket with no data on either side is a dot, and its tooltip row reads "no data". Proof: `drive`.
- `compact-result` The result for 5 series of 168 buckets (7 days at 1h) stays under 16 KiB, and its text content is the same JSON. Proof: `drive`.
- `lifecycle` The view follows a `host-context-changed` theme change, shows a cancelled state on `ui/notifications/tool-cancelled`, and answers `ui/resource-teardown`. Proof: `drive`.
- `default-csp` The view loads and runs under the spec's default CSP with no console errors, so it needs no `csp` domains. Proof: `drive`.

Expand All @@ -31,12 +34,13 @@ Preconditions:
- Launch from `../SKILL.md` has run on this checkout, and `$VERIFY_RUN` is set.
- A Chromium build for playwright-core is installed. If the script cannot launch it, run `pnpm --filter @sentry/mcp-core exec playwright-core install chromium-headless-shell`.

- **Every `drive` claim: act as the host.** Run `node .agents/skills/verify-toolkit/scripts/mcp-apps-host.mjs --out "$VERIFY_RUN/mcp-apps"`. It starts the mock server, calls the tool, reads the view, and mounts it in headless Chromium inside a host page that uses `AppBridge`. Every line it prints, also in `checks.txt`, is `PASS`. The screenshots `chart-light.png`, `chart-dark.png`, `chart-range-7d.png`, and `chart-grouped-dark.png` show each state. `host-log.json` holds the view's `tools/call` and `ui/open-link` requests to the host.
- **Every `drive` claim: act as the host.** Run `node .agents/skills/verify-toolkit/scripts/mcp-apps-host.mjs --out "$VERIFY_RUN/mcp-apps"`. It starts the mock server, calls the tool, reads the view, and mounts it in headless Chromium inside a host page that uses `AppBridge`. Every line it prints, also in `checks.txt`, is `PASS`. Its grouped call is `p95(span.duration)` by `release` over 7 days at 1h. The screenshots `chart-light.png`, `chart-dark.png`, `chart-range-7d.png`, and `chart-grouped-dark.png` show each state. `host-log.json` holds the view's `tools/call` and `ui/open-link` requests to the host.

## Gotchas

- The mock server answers from built packages. Rerun `pnpm run build` after changing mcp-core or the mocks, or the script drives stale code.
- The view is generated. After changing `packages/mcp-core/chart-app/`, run `pnpm --filter @sentry/mcp-core generate-chart-app`, then build.
- For a grouped request, the mock returns releases in the order that the request's `sort` gives. Three releases are busy; five have 1 to 57 events, every one at 30s, so a sort by p95 picks those five and leaves most of their buckets null.
- The host page sets the default CSP with a meta tag in `srcdoc`. Real hosts serve the view from a sandbox origin with CSP headers.

## Gaps
Expand Down
64 changes: 59 additions & 5 deletions .agents/skills/verify-toolkit/scripts/mcp-apps-host.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@
// stdio server, call the tool, read its `ui://` view, and mount the view in a
// headless browser through the ext-apps AppBridge, the way Claude or VS Code
// does. Screenshots the chart in light and dark, after the range picker calls
// the tool again through the host, and split by an attribute. Writes evidence
// and one PASS or FAIL line per check to checks.txt in --out. See ../SKILL.md.
// the tool again through the host, and split by an attribute. The grouped
// chart is p95 by release over 7 days at 1h, where the mock has busy releases
// and sparse ones whose p95 is higher, as in getsentry/toolkit#2451. Writes
// evidence and one PASS or FAIL line per check to checks.txt in --out. See
// ../SKILL.md.
import { appendFileSync, mkdirSync, writeFileSync } from "node:fs";
import { createRequire } from "node:module";
import { dirname, join, resolve } from "node:path";
Expand Down Expand Up @@ -54,7 +57,11 @@ const GROUPED_CHART = {
yAxis: "p95(span.duration)",
groupBy: "release",
period: "7d",
interval: "1h",
};
// Largest structured result for GROUPED_CHART: 5 series of 168 buckets. The
// compact result is about 12 KB; one object per bucket was about 200 KB.
const GROUPED_BUDGET_BYTES = 16 * 1024;
// The spec's default view CSP: no network, no eval.
const DEFAULT_CSP =
"default-src 'none'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; media-src 'self' data:; connect-src 'none'";
Expand Down Expand Up @@ -263,7 +270,7 @@ try {
rangeCall?.params?.name === TOOL &&
rangeCall.params.arguments.period === "7d" &&
rangeResult?.result?.structuredContent?.timeRange?.statsPeriod === "7d",
`"${before}" -> view called ${TOOL} period=7d through the host, got ${rangeResult?.result?.structuredContent?.series?.[0]?.buckets?.length} buckets -> ${range}`,
`"${before}" -> view called ${TOOL} period=7d through the host, got ${rangeResult?.result?.structuredContent?.times?.length} buckets -> ${range}`,
);

await frame.getByRole("button", { name: "Open in Sentry" }).click();
Expand All @@ -279,13 +286,60 @@ try {
`ui/open-link ${openLink?.params?.url}`,
);

const groupedChart = groupedResult.structuredContent;
const groupedBytes = JSON.stringify(groupedChart).length;
check(
"compact-result",
groupedBytes < GROUPED_BUDGET_BYTES &&
groupedResult.content[0]?.text?.length === groupedBytes,
`${groupedChart.series.length} series x ${groupedChart.times.length} buckets: structuredContent ${groupedBytes} bytes, text ${groupedResult.content[0]?.text?.length} bytes (budget ${GROUPED_BUDGET_BYTES})`,
);
const counts = groupedChart.series.map((s) => s.count);
check(
"top-by-volume",
counts.length === 5 &&
counts.slice(0, 3).every((count) => count > 1000) &&
counts.every((count, i) => i === 0 || count <= counts[i - 1]),
`p95 by release picks the releases with the most events: ${groupedChart.series.map((s) => `${s.label.slice(0, 7)}=${s.count}`).join(", ")}`,
);

await mount("dark", GROUPED_CHART, groupedResult);
const legend = await frame.locator(".legend").innerText();
const groupedStats = await frame.locator(".stats").innerText();
const grouped = await shot("chart-grouped-dark.png");
check(
"grouped",
legend.split("\n").filter(Boolean).length === 3,
`top groups "${legend.replace(/\n/g, " | ")}" -> ${grouped}`,
legend.split("\n").filter(Boolean).length === 5 &&
groupedChart.peak === null &&
!groupedStats.includes("Peak"),
`top groups "${legend.replace(/\n/g, " | ")}", stats "${groupedStats.replace(/\n/g, " ")}", peak ${groupedChart.peak} -> ${grouped}`,
);

// The sparsest charted release has data in 4 of 168 buckets. Its empty
// buckets are null in the result and draw as gaps: 4 dots, no line to 0.
const sparse = groupedChart.series.at(-1);
const sparseColor = "#2ba1db";
const dots = await frame
.locator(`svg circle[fill="${sparseColor}"][r="2.5"]`)
.count();
const lines = await frame
.locator(`svg path[stroke="${sparseColor}"]`)
.count();
const groupedBox = await frame.locator("svg").boundingBox();
// The left edge of the plot, where the sparse release has no data.
await page.mouse.move(
groupedBox.x + 50,
groupedBox.y + groupedBox.height / 2,
);
const gapTooltip = await frame.locator(".tooltip").innerText();
const withData = sparse.values.filter((v) => v !== null).length;
check(
"null-gaps",
!sparse.values.includes(0) &&
dots === withData &&
lines === 0 &&
gapTooltip.includes("no data"),
`${sparse.label.slice(0, 7)} has ${withData} of ${sparse.values.length} buckets with data and no 0s; drawn as ${dots} dots and ${lines} lines; tooltip at the left edge "${gapTooltip.replace(/\s+/g, " ")}"`,
);

// The rest of the lifecycle: a theme change, a cancelled call, teardown.
Expand Down
Binary file modified docs/evidence/mcp-apps-chart/chart-grouped-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 6 additions & 3 deletions docs/evidence/mcp-apps-chart/checks.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
PASS mcp-apps#tool-meta: render_sentry_chart _meta.ui.resourceUri=ui://sentry/chart/v1.html, openai/outputTemplate=ui://sentry/chart/v1.html
PASS mcp-apps#view-resource: resources/read ui://sentry/chart/v1.html: text/html;profile=mcp-app, 16 KiB HTML
PASS mcp-apps#tool-meta: render_sentry_chart _meta.ui.resourceUri=ui://sentry/chart/v3.html, openai/outputTemplate=ui://sentry/chart/v3.html
PASS mcp-apps#view-resource: resources/read ui://sentry/chart/v3.html: text/html;profile=mcp-app, 17 KiB HTML
PASS mcp-apps#view-renders: light chart with hover tooltip "Sep 10, 16:00 UTC count() 91" -> docs/evidence/mcp-apps-chart/chart-light.png
PASS mcp-apps#view-theme: host theme light=rgb(255, 255, 255) dark=rgb(30, 26, 37) -> docs/evidence/mcp-apps-chart/chart-dark.png
PASS mcp-apps#range-roundtrip: "errors · level:error · Last 24 hours" -> view called render_sentry_chart period=7d through the host, got 56 buckets -> docs/evidence/mcp-apps-chart/chart-range-7d.png
PASS mcp-apps#open-link: ui/open-link https://sentry-mcp-evals.sentry.io/explore/discover/homepage/?dataset=errors&queryDataset=error-events&query=level%3Aerror&project=-1&field=count%28%29&sort=-count%28%29&statsPeriod=7d&mode=aggregate&yAxis=count%28%29
PASS mcp-apps#grouped: top groups "frontend@2.4.1 | frontend@2.4.0 | frontend@2.3.9" -> docs/evidence/mcp-apps-chart/chart-grouped-dark.png
PASS mcp-apps#compact-result: 5 series x 168 buckets: structuredContent 12304 bytes, text 12304 bytes (budget 16384)
PASS mcp-apps#top-by-volume: p95 by release picks the releases with the most events: 3f9c2e1=20160, 8b41d07=10080, c05e9a3=3360, 1d7f4b2=57, e2a86c9=23
PASS mcp-apps#grouped: top groups "3f9c2e1 · 20.2K events | 8b41d07 · 10.1K events | c05e9a3 · 3.4K events | 1d7f4b2 · 57 events | e2a86c9 · 23 events", stats "Interval 1h", peak null -> docs/evidence/mcp-apps-chart/chart-grouped-dark.png
PASS mcp-apps#null-gaps: e2a86c9 has 4 of 168 buckets with data and no 0s; drawn as 4 dots and 0 lines; tooltip at the left edge "Sep 4, 01:00 UTC 3f9c2e1 6.44s 8b41d07 7.59s c05e9a3 8.04s 1d7f4b2 no data e2a86c9 no data"
PASS mcp-apps#lifecycle: host-context-changed -> light, tool-cancelled -> "The chart request was cancelled.", ui/resource-teardown -> {}
PASS mcp-apps#default-csp: 0 console errors under the default CSP
21 changes: 20 additions & 1 deletion packages/mcp-core/chart-app/chart.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { scaleFraction, valueScale } from "./chart";
import { scaleFraction, shortLabel, valueScale } from "./chart";

function plotted(values: number[]) {
const scale = valueScale(values);
Expand Down Expand Up @@ -46,3 +46,22 @@ describe("valueScale", () => {
expect(valueScale([0, 0])).toEqual({ bottom: 0, top: 1, step: 1 });
});
});

describe("shortLabel", () => {
it.each([
["a commit SHA", "3f9c2e1a3f9c2e1a3f9c2e1a3f9c2e1a3f9c2e1a", "3f9c2e1"],
[
"a SHA after a package",
"frontend@3f9c2e1a3f9c2e1a3f9c2e1a3f9c2e1a3f9c2e1a",
"frontend@3f9c2e1",
],
["a short version", "frontend@2.4.1", "frontend@2.4.1"],
[
"a long value",
"/api/0/organizations/{organization_id_or_slug}/events/",
"/api/0/organizations/{organizat…",
],
])("shortens %s", (_, label, expected) => {
expect(shortLabel(label)).toBe(expected);
});
});
135 changes: 81 additions & 54 deletions packages/mcp-core/chart-app/chart.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
/**
* Draws a render_sentry_chart result as SVG: one area (or line, when there
* are several series) per series, a y grid, time labels, a hatched band over
* buckets that are still filling, and a hover tooltip. Values come from an
* untrusted tool result, so text only ever goes in through textContent.
* buckets that are still filling, and a hover tooltip. A bucket with no data
* is a gap, not 0, and a bucket with data on neither side is a dot. Values
* come from an untrusted tool result, so text only ever goes in through
* textContent.
*/
import type { ChartResult } from "../src/tools/support/chart/schema";

const SVG_NS = "http://www.w3.org/2000/svg";
/** A plotted bucket: x, y, and its index into `times`. */
type Point = readonly [x: number, y: number, index: number];
const HEIGHT = 220;
const MARGIN = { top: 8, right: 8, bottom: 22, left: 48 };
export const SERIES_COLORS = [
Expand All @@ -32,6 +36,16 @@ const compact = new Intl.NumberFormat(undefined, {
maximumFractionDigits: 1,
});

export function formatCount(count: number): string {
return compact.format(count);
}

/** A legend-sized label: a 40-character commit SHA becomes its short form. */
export function shortLabel(label: string): string {
const short = label.replace(/\b([0-9a-f]{7})[0-9a-f]{33}\b/gi, "$1");
return short.length > 32 ? `${short.slice(0, 31)}…` : short;
}

export function formatValue(value: number, chart: ChartResult): string {
if (chart.valueType === "duration") {
const ms = value * (DURATION_MS[chart.valueUnit ?? "millisecond"] ?? 1);
Expand Down Expand Up @@ -137,12 +151,12 @@ export function drawChart(
const width = Math.max(container.clientWidth, 280);
const plotWidth = width - MARGIN.left - MARGIN.right;
const plotHeight = HEIGHT - MARGIN.top - MARGIN.bottom;
const times = chart.series[0]?.buckets.map((b) => Date.parse(b.time)) ?? [];
const times = chart.times.map((time) => Date.parse(time));
const first = times[0] ?? 0;
const last = times.at(-1) ?? first;
const span = Math.max(last - first, 1);
const scale = valueScale(
chart.series.flatMap((s) => s.buckets.map((b) => b.value)),
chart.series.flatMap((s) => s.values.filter((v) => v !== null)),
);
const { bottom, top, step } = scale;
const x = (time: number) =>
Expand Down Expand Up @@ -210,8 +224,8 @@ export function drawChart(
}

// Buckets still filling: a hatched band from the first one to the end.
const firstFilling = chart.series[0]?.buckets.findIndex((b) => b.incomplete);
if (firstFilling !== undefined && firstFilling >= 0) {
const firstFilling = chart.incomplete[0];
if (firstFilling !== undefined) {
const fillingStart = x(times[Math.max(firstFilling - 1, 0)] ?? last);
svg(
"rect",
Expand All @@ -238,48 +252,57 @@ export function drawChart(
);

const single = chart.series.length === 1;
const path = (pts: readonly Point[]) =>
pts.map(([px, py], i) => `${i ? "L" : "M"}${px},${py}`).join("");
chart.series.forEach((series, index) => {
const color = SERIES_COLORS[index % SERIES_COLORS.length] ?? "#7553ff";
const points = series.buckets.map(
(b) => [x(Date.parse(b.time)), y(b.value)] as const,
);
const split = series.buckets.findIndex((b) => b.incomplete);
const settled = split < 0 ? points : points.slice(0, Math.max(split, 1));
const filling = split < 0 ? [] : points.slice(Math.max(split - 1, 0));
const path = (pts: readonly (readonly [number, number])[]) =>
pts.map(([px, py], i) => `${i ? "L" : "M"}${px},${py}`).join("");
if (single && points.length > 1) {
svg(
"path",
{
d: `${path(points)}L${points.at(-1)?.[0]},${y(0)}L${points[0]?.[0]},${y(0)}Z`,
fill: color,
"fill-opacity": 0.16,
},
root,
);
}
svg(
"path",
{ d: path(settled), fill: "none", stroke: color, "stroke-width": 2 },
root,
);
if (points.length === 1) {
const [px, py] = points[0] ?? [0, 0];
svg("circle", { cx: px, cy: py, r: 4, fill: color }, root);
// Runs of buckets with data. A null bucket ends a run, so it draws as a
// gap instead of a drop to 0.
const runs: Point[][] = [];
let run: Point[] | null = null;
for (const [i, time] of times.entries()) {
const value = series.values[i];
if (value === null || value === undefined) {
run = null;
continue;
}
if (!run) runs.push((run = []));
run.push([x(time), y(value), i]);
}
if (filling.length > 1) {
svg(
"path",
{
d: path(filling),
fill: "none",
stroke: color,
"stroke-width": 2,
"stroke-dasharray": "4 3",
},
root,
);
for (const pts of runs) {
const [px, py] = pts[0] ?? [0, 0];
if (pts.length === 1) {
svg("circle", { cx: px, cy: py, r: 2.5, fill: color }, root);
continue;
}
if (single) {
svg(
"path",
{
d: `${path(pts)}L${pts.at(-1)?.[0]},${y(0)}L${px},${y(0)}Z`,
fill: color,
"fill-opacity": 0.16,
},
root,
);
}
const settled =
firstFilling === undefined
? pts
: pts.filter(([, , i]) => i < firstFilling);
const filling =
firstFilling === undefined
? []
: pts.filter(([, , i]) => i >= firstFilling - 1);
const line = { fill: "none", stroke: color, "stroke-width": 2 };
if (settled.length > 1) svg("path", { d: path(settled), ...line }, root);
if (filling.length > 1) {
svg(
"path",
{ d: path(filling), ...line, "stroke-dasharray": "4 3" },
root,
);
}
}
});

Expand Down Expand Up @@ -325,26 +348,30 @@ export function drawChart(
crosshair.setAttribute("visibility", "visible");
const timeRow = document.createElement("div");
timeRow.className = "time";
const incomplete = chart.series[0]?.buckets[index]?.incomplete;
const incomplete = chart.incomplete.includes(index);
timeRow.textContent = `${formatTooltipTime(time, format)}${incomplete ? " (still filling)" : ""}`;
const rows = chart.series.map((series, s) => {
const bucket = series.buckets[index];
const value = series.values[index] ?? null;
const marker = markers[s];
if (bucket && marker) {
if (marker) {
marker.setAttribute("cx", String(px));
marker.setAttribute("cy", String(y(bucket.value)));
marker.setAttribute("visibility", "visible");
marker.setAttribute("cy", String(y(value ?? 0)));
marker.setAttribute(
"visibility",
value === null ? "hidden" : "visible",
);
}
const row = document.createElement("div");
row.className = "row";
const swatch = document.createElement("span");
swatch.className = "swatch";
swatch.style.background = SERIES_COLORS[s % SERIES_COLORS.length] ?? "";
const label = document.createElement("span");
label.textContent = series.label;
const value = document.createElement("span");
value.textContent = bucket ? formatValue(bucket.value, chart) : "–";
row.append(swatch, label, value);
label.textContent = shortLabel(series.label);
const shown = document.createElement("span");
shown.textContent =
value === null ? "no data" : formatValue(value, chart);
row.append(swatch, label, shown);
return row;
});
tooltip.replaceChildren(timeRow, ...rows);
Expand Down
Loading
Loading