Skip to content

Commit 8aa803a

Browse files
authored
Adopt Anthropic computer and browser toolsets (#92)
* Adopt Anthropic GA computer and browser tools * Preserve active tab when closing background tabs * Harden Anthropic toolset state handling * Accept Anthropic bearer token credentials * Keep bearer tokens out of API key helpers * Test Anthropic bearer token isolation * Document Anthropic browser JavaScript option --------- Co-authored-by: rgarcia <72655+rgarcia@users.noreply.github.com>
1 parent 2a2009c commit 8aa803a

32 files changed

Lines changed: 951 additions & 603 deletions

docs/architecture.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -178,14 +178,15 @@ Catalog compilation composes provider behavior rather than replacing the whole
178178
catalog:
179179

180180
- Ordinary function tools stay ordinary.
181-
- Anthropic native browser/computer declarations replace only their own
182-
placeholders and merge required beta headers with caller headers.
181+
- Anthropic's browser/computer client-toolset declarations replace only their
182+
own placeholders; their member calls and results retain `toolset_name`
183+
through pi's function-tool transcript.
183184
- OpenAI streams through pi's builtin Responses transport and its automatic
184185
prompt caching by default; a Browser Loop-owned adapter handles OpenAI's native
185186
computer tool and tool-search namespace round-trips.
186-
- Anthropic's native browser tool falls back to an equivalent function-tool
187-
declaration when the active credential cannot access `browser_20260701`;
188-
the selected tool identity, name, schema, and executor remain unchanged.
187+
- Anthropic's `computer_toolset_20260801` and `browser_toolset_20260801` may be
188+
selected together; calls remain separated by their computer and viewport
189+
coordinate frames.
189190
- Google's current predefined browser toolset serializes one `computer_use`
190191
declaration plus exact exclusions through the Browser Loop-owned Interactions API
191192
adapter. Excluded calls fail with a named catalog error instead of reaching

packages/browser-loop/CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
## Unreleased
44

5+
- Update Anthropic's computer and browser integrations to use the
6+
`computer_toolset_20260801` and `browser_toolset_20260801` client toolsets.
7+
Member calls/results preserve `toolset_name`, sequential batches stop at the
8+
first failure, and both toolsets may be selected together.
59
- Rename the product and unpublished package to Browser Loop: the repository is
610
prepared for `kernel/browser-loop`, the package is `@onkernel/browser-loop`,
711
and release tags use `browser-loop/v*`. The public `loop` namespace, `Loop*`

packages/browser-loop/README.md

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -318,8 +318,8 @@ function tools.
318318
```ts
319319
const tools = [
320320
loop.providers.anthropic.tools.computer({
321-
version: "20260701",
322-
enableZoom: true,
321+
version: "20260801",
322+
zoom: true,
323323
}),
324324
loop.tools.browser.snapshot(),
325325
];
@@ -330,9 +330,9 @@ Available groups:
330330
```ts
331331
loop.providers.openai.tools.computer();
332332

333-
loop.providers.anthropic.source;
334-
loop.providers.anthropic.tools.computer({ version: "20260701" });
335-
loop.providers.anthropic.tools.browser({ version: "20260701" });
333+
loop.providers.anthropic.sources;
334+
loop.providers.anthropic.tools.computer({ version: "20260801" });
335+
loop.providers.anthropic.tools.browser({ version: "20260801" });
336336

337337
loop.providers.google.source;
338338
loop.providers.google.toolsets.browser({ exclude: ["right_click"] });
@@ -341,6 +341,16 @@ loop.providers.google.toolsets.browser({ exclude: ["right_click"] });
341341
loop.toolsets.browser();
342342
```
343343

344+
Anthropic's browser toolset leaves `javascript_exec` disabled unless it is
345+
enabled explicitly:
346+
347+
```ts
348+
loop.providers.anthropic.tools.browser({
349+
version: "20260801",
350+
javascript: true,
351+
});
352+
```
353+
344354
The Google browser set exposes the current predefined action names and uses
345355
normalized coordinates in `[0, 999]`. Its native `computer_use` declaration
346356
excludes every unselected browser action. If Google emits an excluded name
@@ -357,11 +367,12 @@ than on the wire. [Models and native surfaces](docs/supported-models.md) lists
357367
which models carry which surface.
358368

359369
Provider-native caller-visible names are fixed by protocol. Version/tool/model
360-
mismatches fail during catalog compilation. If an Anthropic credential cannot
361-
access `browser_20260701`, Browser Loop retries with an equivalent `browser` function
362-
tool and remembers that choice for the credential and process. Every
363-
`loop.providers.*` tool surface exposes its first-party `source` (or versioned
364-
`sources`), and every returned provider spec carries the applicable URL.
370+
mismatches fail during catalog compilation. Anthropic's
371+
`computer_toolset_20260801` and `browser_toolset_20260801` declarations may be
372+
selected together; Browser Loop preserves member `toolset_name` fields and
373+
sequential batch semantics across pi's transcript.
374+
Every `loop.providers.*` tool surface exposes its first-party `source` or
375+
`sources`, and every returned provider spec carries the applicable URL.
365376

366377
## Catalog compilation
367378

@@ -434,8 +445,8 @@ require different transports fails to compile.
434445
`openai-computer-use` api instead, which a Browser Loop adapter handles; that same
435446
adapter also covers tool-search namespace round-trips regardless of api,
436447
since pi's builtin transport does not replay them.
437-
- **Anthropic**: exact native declarations, beta-header composition, and
438-
adaptive model preparation. No api fork — every Anthropic model streams
448+
- **Anthropic**: client-toolset declarations, member/result transcript
449+
adaptation, and adaptive model preparation. Every Anthropic model streams
439450
through pi's builtin transport.
440451
- **Google**: a model selected without Google's native browser toolset streams
441452
through pi's builtin transport. Selecting

packages/browser-loop/docs/supported-models.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,7 @@ models and need their own entry. Each cites first-party documentation.
2222

2323
| provider | models | surfaces |
2424
| --- | --- | --- |
25-
| `anthropic` | `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser |
26-
| `anthropic` | `claude-fable-5` family | computer |
25+
| `anthropic` | `claude-fable-5`, `claude-mythos-5`, `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser |
2726
| `openai` | `gpt-5.6-sol`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5` | computer |
2827
| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-2.5-computer-use-preview-10-2025` | browser |
2928

@@ -93,9 +92,9 @@ npx tsx packages/browser-loop/scripts/native-action-probe.ts --provider openai -
9392

9493
Update that provider's adapter under `src/pi/providers/` to execute the actions the
9594
probe returns, then add or adjust the `COMPUTER_USE_NATIVE_SURFACES` entry, citing the
96-
provider's documentation. Anthropic's computer tool version and its
97-
`computer-use-*` beta header are chosen by pi-ai per model, so a new dated
98-
version there usually means bumping pi-ai rather than editing this package.
95+
provider's documentation. Anthropic's client toolsets are adapted in
96+
`src/pi/providers/anthropic/toolsets.ts`; a new dated toolset version requires
97+
updating its declarations and member transcript adapter together.
9998

10099
**A model rejects a tool Browser Loop sends.** Add a `LOOP_MODEL_QUIRKS` entry with the
101100
observed error as its `reason`, scoped as narrowly as the evidence supports: a

packages/browser-loop/examples/anthropic-native-smoke.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ const CONFIGS: Record<string, readonly LoopAgentTool[]> = {
1616
computer: loop.toolsets.computer(),
1717
browser: loop.toolsets.browser(),
1818
mixed: loop.toolsets.mixed(),
19-
"native-computer": [loop.providers.anthropic.tools.computer({ version: "20260701", enableZoom: true })],
20-
"native-browser": [loop.providers.anthropic.tools.browser({ version: "20260701" })],
19+
"native-computer": [loop.providers.anthropic.tools.computer({ version: "20260801", zoom: true })],
20+
"native-browser": [loop.providers.anthropic.tools.browser({ version: "20260801" })],
2121
};
2222

2323
const PROMPT = [

packages/browser-loop/examples/shared/tools.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ export function toolsForModel(model: LoopModelRef): LoopAgentTool[] {
2424
// Claude 5 can use Anthropic's native browser tool; older models use portable
2525
// Loop tools plus the explicit semantic action-plan surface.
2626
return loop.providers.anthropic.supports.browser(modelId)
27-
? [loop.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]
27+
? [loop.providers.anthropic.tools.browser({ version: "20260801", javascript: true })]
2828
: structuredBrowserTools();
2929
case "google":
3030
// Current Gemini computer-use models expect Google's predefined browser actions.

packages/browser-loop/src/core/actions/browser.ts

Lines changed: 68 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,15 +23,20 @@ export const BROWSER_ACTION_TYPES = [
2323
"browser_find",
2424
"browser_click",
2525
"browser_hover",
26+
"browser_mouse_down",
27+
"browser_mouse_up",
2628
"browser_drag",
2729
"browser_fill",
2830
"browser_scroll_to",
2931
"browser_scroll",
3032
"browser_type",
3133
"browser_key",
34+
"browser_hold_key",
3235
"browser_navigate",
3336
"browser_list_tabs",
3437
"browser_new_tab",
38+
"browser_switch_tab",
39+
"browser_close_tab",
3540
"browser_screenshot",
3641
"browser_evaluate",
3742
] as const;
@@ -138,6 +143,20 @@ export interface BrowserActionHover {
138143
tab_id?: string;
139144
}
140145

146+
export interface BrowserActionMouseDown {
147+
type: "browser_mouse_down";
148+
x: number;
149+
y: number;
150+
tab_id?: string;
151+
}
152+
153+
export interface BrowserActionMouseUp {
154+
type: "browser_mouse_up";
155+
x: number;
156+
y: number;
157+
tab_id?: string;
158+
}
159+
141160
export interface BrowserActionDrag {
142161
type: "browser_drag";
143162
from: { x: number; y: number };
@@ -180,9 +199,16 @@ export interface BrowserActionKey {
180199
tab_id?: string;
181200
}
182201

202+
export interface BrowserActionHoldKey {
203+
type: "browser_hold_key";
204+
text: string;
205+
duration: number;
206+
tab_id?: string;
207+
}
208+
183209
export interface BrowserActionNavigate {
184210
type: "browser_navigate";
185-
/** A URL, or the sentinels "back" / "forward" for history navigation. */
211+
/** A URL, or the sentinels "back", "forward", or "reload". */
186212
url: string;
187213
tab_id?: string;
188214
}
@@ -195,6 +221,16 @@ export interface BrowserActionNewTab {
195221
type: "browser_new_tab";
196222
}
197223

224+
export interface BrowserActionSwitchTab {
225+
type: "browser_switch_tab";
226+
tab_id: string;
227+
}
228+
229+
export interface BrowserActionCloseTab {
230+
type: "browser_close_tab";
231+
tab_id: string;
232+
}
233+
198234
export interface BrowserActionScreenshot {
199235
type: "browser_screenshot";
200236
/** Optional crop region, [x0, y0, x1, y1] in viewport pixels. */
@@ -216,15 +252,20 @@ export type BrowserAction =
216252
| BrowserActionFind
217253
| BrowserActionClick
218254
| BrowserActionHover
255+
| BrowserActionMouseDown
256+
| BrowserActionMouseUp
219257
| BrowserActionDrag
220258
| BrowserActionFill
221259
| BrowserActionScrollTo
222260
| BrowserActionScroll
223261
| BrowserActionTypeText
224262
| BrowserActionKey
263+
| BrowserActionHoldKey
225264
| BrowserActionNavigate
226265
| BrowserActionListTabs
227266
| BrowserActionNewTab
267+
| BrowserActionSwitchTab
268+
| BrowserActionCloseTab
228269
| BrowserActionScreenshot
229270
| BrowserActionEvaluate;
230271

@@ -403,6 +444,14 @@ export function createBrowserActionSchemaByType(options: BrowserActionSchemaOpti
403444
},
404445
{ additionalProperties: false },
405446
),
447+
browser_mouse_down: Type.Object(
448+
{ type: Type.Literal("browser_mouse_down"), x: Type.Number(), y: Type.Number(), tab_id: TabId() },
449+
{ additionalProperties: false },
450+
),
451+
browser_mouse_up: Type.Object(
452+
{ type: Type.Literal("browser_mouse_up"), x: Type.Number(), y: Type.Number(), tab_id: TabId() },
453+
{ additionalProperties: false },
454+
),
406455
browser_drag: Type.Object(
407456
{
408457
type: Type.Literal("browser_drag"),
@@ -459,16 +508,33 @@ export function createBrowserActionSchemaByType(options: BrowserActionSchemaOpti
459508
},
460509
{ additionalProperties: false },
461510
),
511+
browser_hold_key: Type.Object(
512+
{
513+
type: Type.Literal("browser_hold_key"),
514+
text: Type.String({ description: "Key or chord to hold." }),
515+
duration: Type.Number({ minimum: 0, maximum: 30 }),
516+
tab_id: TabId(),
517+
},
518+
{ additionalProperties: false },
519+
),
462520
browser_navigate: Type.Object(
463521
{
464522
type: Type.Literal("browser_navigate"),
465-
url: Type.String({ description: "URL to navigate to, or \"back\" / \"forward\" for history navigation." }),
523+
url: Type.String({ description: "URL to navigate to, or \"back\", \"forward\", or \"reload\"." }),
466524
tab_id: TabId(),
467525
},
468526
{ additionalProperties: false },
469527
),
470528
browser_list_tabs: Type.Object({ type: Type.Literal("browser_list_tabs") }, { additionalProperties: false }),
471529
browser_new_tab: Type.Object({ type: Type.Literal("browser_new_tab") }, { additionalProperties: false }),
530+
browser_switch_tab: Type.Object(
531+
{ type: Type.Literal("browser_switch_tab"), tab_id: Type.String() },
532+
{ additionalProperties: false },
533+
),
534+
browser_close_tab: Type.Object(
535+
{ type: Type.Literal("browser_close_tab"), tab_id: Type.String() },
536+
{ additionalProperties: false },
537+
),
472538
browser_screenshot: Type.Object(
473539
{
474540
type: Type.Literal("browser_screenshot"),

packages/browser-loop/src/core/anthropic-native.ts

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ function asNativeInput(args: unknown): NativeInput {
1414

1515
const MAX_KEY_REPEAT = 100;
1616

17-
/** Map one `computer_20260701` tool input onto canonical computer-plane actions. */
17+
/** Map one Anthropic computer toolset member onto canonical computer-plane actions. */
1818
export function mapNativeComputerInput(input: NativeInput): ComputerUseAction[] {
1919
switch (input.action) {
2020
case "screenshot":
@@ -52,19 +52,19 @@ export function mapNativeComputerInput(input: NativeInput): ComputerUseAction[]
5252
}
5353
case "hold_key":
5454
// Canonical keypress duration is milliseconds; the native tool speaks seconds.
55-
return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input) * 1000 }];
55+
return [{ type: "keypress", keys: [text(input)], duration: durationSeconds(input, 300) * 1000 }];
5656
case "wait":
57-
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
57+
return [{ type: "wait", ms: durationSeconds(input, 300) * 1000 }];
5858
case "cursor_position":
5959
return [{ type: "cursor_position" }];
6060
case "zoom":
6161
return [{ type: "zoom", region: region(input.region) }];
6262
default:
63-
throw new Error(`unsupported computer_20260701 action "${input.action}"`);
63+
throw new Error(`unsupported computer toolset member "${input.action}"`);
6464
}
6565
}
6666

67-
/** Map one `browser_20260701` tool input onto canonical browser-plane actions. */
67+
/** Map one Anthropic browser toolset member onto canonical browser-plane actions. */
6868
export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
6969
const tab = tabId(input);
7070
switch (input.action) {
@@ -100,12 +100,20 @@ export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
100100
return [{ type: "browser_click", ...pageTarget(input.target), ...modifiers(input.modifiers), ...tab }];
101101
case "right_click":
102102
return [{ type: "browser_click", ...pageTarget(input.target), button: "right", ...modifiers(input.modifiers), ...tab }];
103+
case "middle_click":
104+
return [{ type: "browser_click", ...pageTarget(input.target), button: "middle", ...modifiers(input.modifiers), ...tab }];
103105
case "double_click":
104106
return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 2, ...modifiers(input.modifiers), ...tab }];
105107
case "triple_click":
106108
return [{ type: "browser_click", ...pageTarget(input.target), num_clicks: 3, ...modifiers(input.modifiers), ...tab }];
107109
case "hover":
108110
return [{ type: "browser_hover", ...pageTarget(input.target), ...tab }];
111+
case "mouse_move":
112+
return [{ type: "browser_hover", ...coordinateTarget(input.target, "target"), ...tab }];
113+
case "left_mouse_down":
114+
return [{ type: "browser_mouse_down", ...coordinateTarget(input.target, "target"), ...tab }];
115+
case "left_mouse_up":
116+
return [{ type: "browser_mouse_up", ...coordinateTarget(input.target, "target"), ...tab }];
109117
case "left_click_drag":
110118
return [{ type: "browser_drag", from: coordinateTarget(input.from, "from"), to: coordinateTarget(input.target, "target"), ...tab }];
111119
case "scroll":
@@ -124,12 +132,18 @@ export function mapNativeBrowserInput(input: NativeInput): ComputerUseAction[] {
124132
const repeat = clampRepeat(input.repeat);
125133
return Array.from({ length: repeat }, () => ({ type: "browser_key" as const, text: text(input), ...tab }));
126134
}
135+
case "hold_key":
136+
return [{ type: "browser_hold_key", text: text(input), duration: durationSeconds(input, 30), ...tab }];
127137
case "wait":
128-
return [{ type: "wait", ms: durationSeconds(input) * 1000 }];
138+
return [{ type: "wait", ms: durationSeconds(input, 30) * 1000 }];
139+
case "switch_tab":
140+
return [{ type: "browser_switch_tab", tab_id: requireString(input.tab_id, "tab_id") }];
141+
case "close_tab":
142+
return [{ type: "browser_close_tab", tab_id: requireString(input.tab_id, "tab_id") }];
129143
case "javascript_exec":
130144
return [{ type: "browser_evaluate", code: text(input), ...tab }];
131145
default:
132-
throw new Error(`unsupported browser_20260701 action "${input.action}"`);
146+
throw new Error(`unsupported browser toolset member "${input.action}"`);
133147
}
134148
}
135149

@@ -191,10 +205,10 @@ function requireString(value: unknown, field: string): string {
191205
return value;
192206
}
193207

194-
function durationSeconds(input: NativeInput): number {
208+
function durationSeconds(input: NativeInput, maximum: number): number {
195209
const value = input.duration;
196210
if (typeof value !== "number" || !Number.isFinite(value) || value < 0) throw new Error("invalid duration: expected a non-negative number");
197-
return Math.min(value, 100);
211+
return Math.min(value, maximum);
198212
}
199213

200214
function clampRepeat(value: unknown): number {

0 commit comments

Comments
 (0)