fix(a11y): guard each overwriteCommand in the accessibility wrap loop [SDK-7452] - #212
Conversation
AccessibilityModule.onBeforeExecute wrapped every entry of the server-sent commandsToWrap list in one unguarded loop. The list can name a command the active driver never registered: appium sessions omit web-only commands, and the list also carries Selenium-shaped entries (startA11yScanning, stopA11yScanning, performScan with class HttpCommandExecutor, library org.openqa.selenium) meant for other SDKs. WebdriverIO's overwriteCommand throws on an unknown name, so the first such entry aborted the whole loop, left every command after it unwrapped, and surfaced as "Error in onBeforeExecute: overwriteCommand: no command to be overwritten: startA11yScanning". Guard each overwriteCommand call individually so an unknown name is skipped (debug-logged) and the rest of the list still wraps. Ports the guard already shipped on the v9 line in 8539e5f to v8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…9 line The v8 guard body is byte-identical to the one on v9; the comment was not. Drop the v8-specific narrative for v9's rationale, minus the isAppAccessibility history that never applied to v8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
changeset-from-pr.yml generates .changeset/pr-<number>.md from the PR's Release section on this branch, so a hand-written file only duplicates the CHANGELOG entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
🔴 Blocking findings — fix required
Full reasoning, evidence and suggested fixes are in the SDK PR Review Agent's report from your local run. Change map (generated deterministically from the diff)graph LR
subgraph nnode_agent["node-agent"]
npackages_browserstack_service_src_cli_modules_accessibilityModule_ts["accessibilityModule.ts<br/>~27 lines"]
npackages_browserstack_service_tests_cli_modules_accessibilityModule_test_ts["accessibilityModule.test.ts<br/>~27 lines"]
npackages_browserstack_service_src_util_ts["util.ts<br/>~23 lines"]
npackages_browserstack_service_src_accessibility_handler_ts["accessibility-handler.ts<br/>~19 lines"]
npackages_browserstack_service_tests_accessibility_handler_test_ts["accessibility-handler.test.ts<br/>~19 lines"]
n_changeset_pr_212_md["pr-212.md<br/>~6 lines"]
end
↻ This verdict comment is the review anchor — it's updated in place on each run (the gate posts its status separately). — SDK PR Review Agent |
Review findings (relay)The automated review comment above posted as 1. Critical — the per-command guard lands only on the CLI flow
The same wrap loop exists a second time in the non-CLI (service/direct) flow. If that flow receives the same server-sent list, the first Selenium-shaped entry ( What could be established: the counterpart call site exists, is unguarded on that line, and iterates the identical source list. What could not be established: whether Suggested fix — either mirror the guard at 2. Critical — the new test mutates the shared
The test assigns three entries to the module-scope singleton A failing run therefore leaks a populated Suggested fix: const originalCommandsToWrap = accessibilityScripts.commandsToWrap
try {
accessibilityScripts.commandsToWrap = [...] as any
// ...arrange, act, assert
} finally {
accessibilityScripts.commandsToWrap = originalCommandsToWrap
}or move the save/restore into What's good: the regression test is a real one — asserting Coverage: 4 of 4 diff regions judged, no gaps. |
getAppA11yResultsSummary and getA11yResultsSummary caught their failure with a bare
`catch`, logged a fixed "No accessibility summary was found." and returned {}. The
sibling getters (getAppA11yResults, getA11yResults) bind the error and debug-log it;
these two discarded it.
The message is the same whether the results API errored, returned nothing, or the
30s poll in getAppA11yResultResponse timed out, so an empty summary on a real run
cannot be told apart from an API failure. Bind the error and debug-log it, matching
the results getters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit put the cause behind BStackLogger.debug, which does not print at the default log level -- on O11Y build t4x7uep76ef2im4wwjv0rcm6jk6aohbkrdqhni6h the run carried the change and still emitted a bare "No accessibility summary was found." with no DEBUG line anywhere. The message is already an error, so the cause belongs on that line. An empty summary now reports whether the results API errored, returned no payload, or the 30s poll in getAppA11yResultResponse timed out, without needing logLevel: debug. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pollApi throws a plain object, not an Error:
throw { data: {}, headers: {}, message: ... }
so `${error}` produced a literal "[object Object]" — which is what O11Y build
eo2oxrhccxnhwxf5f3gcg4i8zqhx2ddsgagusp93 logged, still saying nothing about the cause.
The useful field is `message`, carrying the server's message from the response body.
Render `error?.message` with a util.inspect fallback, in all three getters. The results
getter also stops claiming "No accessibility summary was found" when it failed to fetch
results, and drops its debug duplicate.
Worth noting for whoever reads the next failure: of pollApi's paths only a non-404 error
response throws. A poll that exhausts the upper time limit, a 404 with a missing
next_poll_time header, and a request with no response all RETURN { data: {} } instead. So
this error line means a hard HTTP error from the results API, not a timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A local App Automate run of the android start-a11y cell produced:
ERROR @wdio/browserstack-service: No accessibility summary was found.
Error: { data: {}, headers: {}, message: undefined }
message is undefined because the results API answered with a JSON body that has no
`message` field — the ternary only covers an EMPTY body, so a JSON error body of any
other shape yields undefined and the reader learns nothing. The status code and the
body itself were never carried at all.
Carry statusCode and body on the rejection, and fall back to `HTTP <code>: <body>` when
the body has no message, so an empty summary always names the HTTP failure behind it.
JSON.parse is also guarded: a non-JSON error body previously threw inside the catch,
replacing the real failure with a SyntaxError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A local run showed the results API answering HTTP 422 "test_run_uuid is invalid", but the log did not say which uuid was sent, so the failure could not be attributed to the SDK or to the service. Both App A11y getters now report testRunUuid and sessionId alongside the error. Neither is a secret; the a11y JWT is still never logged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AccessibilityHandler.before() iterates the SAME server-sent accessibilityScripts.commandsToWrap list as the CLI module and called browser.overwriteCommand with no per-command boundary, so the Selenium-shaped entries (startA11yScanning/stopA11yScanning/performScan, class HttpCommandExecutor) produce the SDK-7452 symptom on that flow as well. It is reachable for App Automate — the handler takes isAppAutomate and branches on isAppAccessibilityAutomationSession — and before() has no try/catch of its own, so a throw rejects the whole hook rather than being logged as it is on the CLI side. v9 already guards this loop; this ports that guard, including its debug message. v9's prevImpl/orig binding is a separate change needing a different commandWrapper signature and is deliberately not ported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regression test assigned to the module-scope accessibilityScripts singleton and reset it as the last statement of the test body, so any earlier throw — a failed expect, or onBeforeExecute rejecting — left a populated commandsToWrap behind for every later test in the file, turning one real failure into a cascade. afterEach only calls vi.resetAllMocks(), which does not restore a plain property written onto the mocked module. Capture the original and restore it in a finally block, rather than resetting to a hardcoded [] that would be wrong if the shared mock ever gains a non-empty default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review findings addressedBoth findings were valid. Fixed in 1. The per-command guard lands only on the CLI flow — confirmed, and the two questions the review left open both resolve against the code:
The deciding detail: v9 already guards this loop ( Deliberately not ported from v9: its On the suggested shared 2. The new test mutates the shared
Full suite after both: build clean, 46/46 files, 1048/1048 tests, 🤖 Generated with Claude Code |
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
|
RUN_TESTS |
|
🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge. |
1 similar comment
|
🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge. |
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
| const browser = this._browser as WebdriverIO.Browser | ||
| browser.overwriteCommand(command.name, this.commandWrapper.bind(this, command), command.class === 'Element') | ||
| try { | ||
| browser.overwriteCommand(command.name, this.commandWrapper.bind(this, command), command.class === 'Element') |
There was a problem hiding this comment.
Warning — the service-flow guard ships without the regression test its CLI-flow twin got.
The guard itself is correct, to be clear: browser.overwriteCommand is synchronous and throws when the driver has no matching base command, so this try/catch confines a single bad accessibilityScripts.commandsToWrap entry to its own iteration, the forEach continues wrapping the rest, PerformanceTester.end(...) below is now reached, and the failure still surfaces via BStackLogger.debug with both the command name and the error.
What's missing is the evidence. This is the mirror of the change to src/cli/modules/accessibilityModule.ts in the same PR, and that side did get a regression test — "skips a command the driver did not register without aborting the wrap loop". The service flow has its own established suite at packages/browserstack-service/tests/accessibility-handler.test.ts, which this PR doesn't touch.
That asymmetry matters because of how this failure presents: the loop aborting on the first Selenium-shaped entry exits 0 and shows up only as silently unwrapped commands and zero accessibility scans. So the untested half is precisely the one that can regress unnoticed while the CLI half stays green in CI.
Suggested fix — add the symmetric case to tests/accessibility-handler.test.ts: stub accessibilityScripts.commandsToWrap with three entries, mock browser.overwriteCommand to throw on the first call, invoke the before path, then assert that (a) overwriteCommand was still called for the remaining two entries and (b) the method returns normally.
Capture and restore the original commandsToWrap in beforeEach/afterEach rather than inside the test body — the CLI test learned that one the hard way, so a failing assertion here can't leak the mutated singleton into later tests in the file.
DEF-12: the guard added to AccessibilityHandler.before() changed behaviour in a file the
repo tests densely (accessibility-handler.test.ts, 26 cases) with no test of its own.
Drives the real payload shape — an HttpCommandExecutor-class startA11yScanning between two
Element commands — and asserts all three wraps are attempted and the trailing command still
wraps. Verified to fail without the guard ("expected spy to be called 3 times, but got 2")
and pass with it.
The existing suite never reached this loop: the shared browser mock has no
overwriteCommand, so `'overwriteCommand' in browser` was false in every other case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
|
🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge. |
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
|
RUN_TESTS |
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
|
🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge. |
|
🟢 SDK PR Review gate is green — the SDK PR Review Agent has run on the current head commit (verdict: This gate confirms a review ran on the latest commit. The verdict itself is advisory — read the findings and use your judgement; it does not block merge. A native GitHub reviewer approval is still separately required by branch protection before this PR can merge. |
|
RUN_TESTS |
What is this about?
AccessibilityModule.onBeforeExecutewrapped every entry of the server-sentaccessibility.options.commandsToWraplist in a single unguarded loop.The list can name a command the active driver never registered: an appium session omits
web-only commands, and the binary also sends Selenium-shaped entries at the tail of the list
(
startA11yScanning,stopA11yScanning,performScan, classHttpCommandExecutor,library
org.openqa.selenium) that are meant for another SDK.@wdio/utilsmonad.jsthrows for a name absent from the driver's command registry, so the first such entry escaped
the
forEach, aborted the rest ofonBeforeExecute, and surfaced as:Every entry after the first unknown name was silently left unwrapped.
Each
overwriteCommandcall is now guarded individually — an unknown name is skipped atdebug level and the rest of the list still wraps. The guard body is byte-identical to the one
already shipped on the v9 line in
8539e5f; this back-ports it to v8, so the two branchesconverge rather than diverge.
A per-command guard is used rather than a filter on
class/library, because it also coversthe appium case (a driver legitimately not registering a web-only command), which a filter on
the Selenium-shaped entries would not.
Verification
8.48.0Error in onBeforeExecute: overwriteCommand...TypeError: browser.stopA11yScanning is not a functionrun-sample-test-stop-a11y)FAILED in iosPASSED in iosbuild_status_stats3tlcu82axqlqqzfkwwxm16e4h2irkgct9wwgdub96or8l79ooffi9veslxggg7qggxoqmabqh3i3mgprUnit-level: reverting only
accessibilityModule.tstoorigin/v8makes the new test fail withthe exact production error string and
overwriteCommandcalled 2 of 3 times; with the guard itis called 3 of 3. Full suite on node 18.20.4:
npm run buildclean, 46/46 files, 1048/1048 tests.Related Jira task/s
https://browserstack.atlassian.net/browse/SDK-7452
Release (mandatory for every PR — required for the
ready-for-reviewlabel)Version bump: (required — tick exactly one)
Release notes type: (optional)
Release notes (customer-facing): (optional but encouraged)
Error in onBeforeExecuteand left the remaining commands unwrapped for auto-scanning. Applies to both the CLI and the non-CLI flow.Release notes (internal): (required — engineer-facing; what actually changed / why)
AccessibilityModule.onBeforeExecutenow guards eachbrowser.overwriteCommandcall individually, so acommandsToWrapentry the driver never registered is skipped (debug-logged) instead of throwing out of theforEachand aborting the rest of the hook. Back-port of8539e5ffrom the v9 line; the guard body is byte-identical there.AccessibilityHandler.before()(the non-CLI flow) iterates the same server-sentcommandsToWraplist and was equally unguarded — and, unlike the CLI path, has notry/catchof its own, so a throw rejected the whole hook. v9 already guards this loop; that guard is ported here, with a regression test (the shared browser mock had nooverwriteCommand, so no existing case reached this loop).getAppA11yResults/getAppA11yResultsSummary/getA11yResultsSummarycaught with a barecatchand logged a fixed string, discarding the cause. They now render the caught value (pollApirejects with a plain object, so${error}produced[object Object]) and name thetestRunUuid/sessionIdqueried.pollApinow carriesstatusCodeandbodyon its rejection and falls back toHTTP <code>: <body>when the error body has nomessagekey, and guardsJSON.parseso a non-JSON error body no longer replaces the real failure with aSyntaxError.Checklist
PR Validations
Run Tests: Comment RUN_TESTS to trigger sanity tests.
🤖 Generated with Claude Code