Report exec_command and search_docs failures as failures - #137
Merged
Conversation
Both tools returned their error text in a normal content block with no isError, so every failure was recorded as a success. exec_command now goes through throwToolError like every other Kernel API tool, which also classifies the status; search_docs calls Mintlify rather than the Kernel API, so it returns an isError result without a Kernel classification.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dcruzeneil2
approved these changes
Aug 4, 2026
dcruzeneil2
left a comment
There was a problem hiding this comment.
Looks good to me. This fixes the misleading success reporting and follows the existing error-handling conventions without changing successful tool behavior. The current test coverage is sufficient for this focused change. Approved!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two tools return their error text in a normal content block with no
isError, so every failure they hit is recorded as a success. Same defect as the one fixed forexecute_playwright_code, in the two remaining places it exists.Found by cross-checking MCP analytics against the Kernel API's own traces during rollout monitoring: in one hour the API returned 4 × 400 on
/browsers/{id}/process/execwhileexec_commandreported zero failures. The tool has looked spotless in every check for a week.Changes
exec_commandnow goes throughthrowToolError, the same funnel every other Kernel API tool uses. It flags the result as an error and classifies the API status, so these failures land asKernelApiError400/404/429rather than being invisible.search_docsreturnserrorResponse(...)instead, for both the unconfigured branch and the catch. It calls Mintlify, not the Kernel API, so a Kernel status classification would be wrong; it just needs to report failure as failure.I swept every other tool: all remaining catch blocks already route through
throwToolErrororerrorResponse.Client-visible change
exec_command's message format changes fromError executing command: <err>toError in exec_command (exec): <err>, matching every other tool. Content is otherwise the same and the result stays a text block.search_docsmessages are byte-identical.Effect on the numbers
The reported error rate will rise slightly as these failures stop being counted as successes. That is the fix working, not a regression — the same shift seen when playwright started reporting honestly.
Tests
bun testcoverssearch_docs' misconfiguration path end to end: a real MCP client over an in-memory transport assertingisErrorand the exact text. Verified it fails against the pre-fix code.exec_command's path needs a live Kernel API call, so it is verified against the preview deployment rather than in the suite.Note
Low Risk
Localized MCP response handling and observability fixes; no auth or data-model changes, with a minor message format change for
exec_commanderrors.Overview
MCP tools were returning failure text as normal success responses (
isErrorunset), which skewed analytics—same issue previously fixed forexecute_playwright_code.exec_commandnow routes Kernel API failures throughthrowToolError, so clients getisErrorand errors are classified by HTTP status (e.g.KernelApiError400). The user-visible message format becomesError in exec_command (exec): …, aligned with other Kernel tools.search_docsuseserrorResponsefor missing Mintlify env vars and for search failures, since those are not Kernel API errors.Adds an in-memory MCP test that
search_docsreports missing configuration withisError: trueand unchanged error text.Reviewed by Cursor Bugbot for commit 5b39e59. Bugbot is set up for automated code reviews on this repo. Configure here.