Skip to content

Report exec_command and search_docs failures as failures - #137

Merged
masnwilliams merged 1 commit into
mainfrom
hypeship/report-tool-failures
Aug 4, 2026
Merged

Report exec_command and search_docs failures as failures#137
masnwilliams merged 1 commit into
mainfrom
hypeship/report-tool-failures

Conversation

@masnwilliams

@masnwilliams masnwilliams commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two tools return their error text in a normal content block with no isError, so every failure they hit is recorded as a success. Same defect as the one fixed for execute_playwright_code, in the two remaining places it exists.

Found by cross-checking MCP analytics against the Kernel API's own traces during rollout monitoring: in one hour the API returned 4 × 400 on /browsers/{id}/process/exec while exec_command reported zero failures. The tool has looked spotless in every check for a week.

// before
} catch (error) {
  return { content: [{ type: "text", text: `Error executing command: ${error}` }] };
}

Changes

  • exec_command now goes through throwToolError, the same funnel every other Kernel API tool uses. It flags the result as an error and classifies the API status, so these failures land as KernelApiError400 / 404 / 429 rather than being invisible.
  • search_docs returns errorResponse(...) instead, for both the unconfigured branch and the catch. It calls Mintlify, not the Kernel API, so a Kernel status classification would be wrong; it just needs to report failure as failure.

I swept every other tool: all remaining catch blocks already route through throwToolError or errorResponse.

Client-visible change

exec_command's message format changes from Error executing command: <err> to Error in exec_command (exec): <err>, matching every other tool. Content is otherwise the same and the result stays a text block. search_docs messages are byte-identical.

Effect on the numbers

The reported error rate will rise slightly as these failures stop being counted as successes. That is the fix working, not a regression — the same shift seen when playwright started reporting honestly.

Tests

bun test covers search_docs' misconfiguration path end to end: a real MCP client over an in-memory transport asserting isError and the exact text. Verified it fails against the pre-fix code. exec_command's path needs a live Kernel API call, so it is verified against the preview deployment rather than in the suite.


Note

Low Risk
Localized MCP response handling and observability fixes; no auth or data-model changes, with a minor message format change for exec_command errors.

Overview
MCP tools were returning failure text as normal success responses (isError unset), which skewed analytics—same issue previously fixed for execute_playwright_code.

exec_command now routes Kernel API failures through throwToolError, so clients get isError and errors are classified by HTTP status (e.g. KernelApiError400). The user-visible message format becomes Error in exec_command (exec): …, aligned with other Kernel tools.

search_docs uses errorResponse for missing Mintlify env vars and for search failures, since those are not Kernel API errors.

Adds an in-memory MCP test that search_docs reports missing configuration with isError: true and unchanged error text.

Reviewed by Cursor Bugbot for commit 5b39e59. Bugbot is set up for automated code reviews on this repo. Configure here.

Both tools returned their error text in a normal content block with no isError,
so every failure was recorded as a success. exec_command now goes through
throwToolError like every other Kernel API tool, which also classifies the
status; search_docs calls Mintlify rather than the Kernel API, so it returns an
isError result without a Kernel classification.
@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mcp Ready Ready Preview Aug 4, 2026 3:40pm

@dcruzeneil2 dcruzeneil2 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This fixes the misleading success reporting and follows the existing error-handling conventions without changing successful tool behavior. The current test coverage is sufficient for this focused change. Approved!

@masnwilliams
masnwilliams merged commit af256a6 into main Aug 4, 2026
10 checks passed
@masnwilliams
masnwilliams deleted the hypeship/report-tool-failures branch August 4, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants