Owl Detection Report & Benchmark Generator

Automated comparison of Owl Browser vs vanilla Playwright and Puppeteer on CreepJS — the industry-standard fingerprint detection tool. Run per release to verify detection and performance. Also includes a performance benchmark mode.

Results are displayed at owlbrowser.net/benchmark.

What this generates

Detection Report (default)

Screenshots and parsed fingerprint data showing:

• Playwright & Puppeteer have identical fingerprint hashes (canvas, WebGL, audio, fonts) — they leak the real device
• Owl Browser has completely different hashes per OS profile — genuine C++ source-level spoofing
• Headless detection: Playwright 100%, Puppeteer 100%, Owl Browser 0%
• GPU: Playwright/Puppeteer expose SwiftShader (dead giveaway), Owl shows real GPU profiles

Performance Benchmark (--benchmark)

Times cold start, navigation, screenshot, and full cycle for all three browsers:

• 10 sequential iterations per browser
• Statistics: min, max, avg, median, p95
• Raw timing data included for reproducibility
• Same machine, same container, same network — fair comparison

Agentic Usage Benchmark (--agentic)

Puts two real agent CLIs — Claude Code and Codex — on real web tasks through three MCP browser servers — Owl Browser MCP, Playwright MCP, and Chrome DevTools MCP — all driving the same locally hosted model, and measures what each combination costs and whether it got the right answer. Tool calls and payload sizes are counted at the MCP transport, not taken from the agents' own reporting. See Agentic benchmark below.

Quick Start

1. Pull the image

docker pull ghcr.io/olib-ai/owl-detection-report:latest

2. Create your .env file

cp .env.example .env
# Edit with your values

OWL_BROWSER_URL=http://your-owl-instance:8080
OWL_BROWSER_TOKEN=your-token

# Optional — S3 upload
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret
S3_BUCKET=your-bucket
S3_PREFIX=detection-reports

3. Run detection report

docker run --rm \
  --network host \
  --env-file .env \
  -v $(pwd)/output:/output \
  ghcr.io/olib-ai/owl-detection-report:latest

4. Run benchmark (sequential + concurrency)

docker run --rm \
  --network host \
  --env-file .env \
  -v $(pwd)/output:/output \
  ghcr.io/olib-ai/owl-detection-report:latest --benchmark

5. Run concurrency benchmark only

Keeps existing sequential data, only updates concurrency results:

docker run --rm \
  --network host \
  --env-file .env \
  -v $(pwd)/output:/output \
  ghcr.io/olib-ai/owl-detection-report:latest --concurrency

6. Run agentic benchmark only

docker run --rm \
  --network host \
  --env-file .env \
  -e AGENTIC_BASE_URL=http://<lm-studio-host>:1234 \
  -e AGENTIC_AUTH_TOKEN=lmstudio \
  -e AGENTIC_MODEL=qwen/qwen3.6-27b \
  -v $(pwd)/output:/output \
  ghcr.io/olib-ai/owl-detection-report:latest --agentic

--agentic runs only the agentic benchmark and writes agentic.json. Nothing else runs. It needs an LLM endpoint (AGENTIC_BASE_URL) and both agent CLIs; see Agentic benchmark.

7. When to run

Run the detection report and benchmark per Owl Browser release — results only change when the browser updates. No need for daily cron.

When using S3 upload, the -v volume mount is optional since files go directly to S3.

Build from source

git clone https://github.com/Olib-AI/owl-detection-report.git
cd owl-detection-report
docker build -t owl-detection-report .

# Detection report
docker run --rm --network host --env-file .env -v $(pwd)/output:/output owl-detection-report

# Benchmark
docker run --rm --network host --env-file .env -v $(pwd)/output:/output owl-detection-report --benchmark

Environment Variables

Variable	Required	Default	Description
OWL_BROWSER_URL	Yes	—	Owl Browser REST API endpoint (e.g. http://localhost:8080)
OWL_BROWSER_TOKEN	Yes	—	Owl Browser API token
OUTPUT_DIR	No	/output	Output directory inside the container
AWS_ACCESS_KEY_ID	No	—	AWS credentials for S3 upload
AWS_SECRET_ACCESS_KEY	No	—	AWS credentials for S3 upload
S3_BUCKET	No	—	S3 bucket name
S3_PREFIX	No	detection-reports/	S3 key prefix
AWS_REGION	No	us-east-1	AWS region
CLOUDFRONT_DISTRIBUTION_ID	No	—	CloudFront distribution for cache invalidation

Agentic benchmark only (--agentic)

Variable	Required	Default	Description
AGENTIC_BASE_URL	Yes	—	OpenAI/Anthropic-compatible endpoint serving the pinned local model (e.g. LM Studio at http://<host>:1234)
AGENTIC_AUTH_TOKEN	No	—	Token for that endpoint (LM Studio accepts any value)
AGENTIC_MODEL	No	qwen/qwen3.6-27b	Model id — identical for every agent and every stack
AGENTIC_AGENTS	No	claude-code-cli,codex-cli	Which agent CLIs to run
AGENTIC_LLM_FLATTEN	No	namespace	How the Codex shim encodes flattened tools (namespace, qualified, dotted, plain)
AGENTIC_RUNS	No	5	Runs per (agent, stack, task) cell
AGENTIC_TASK_TIMEOUT	No	300	Seconds before a single agent run is killed
AGENTIC_STACKS	No	all	Comma-separated stack ids to run (owl,playwright,chrome-devtools)
AGENTIC_TASKS	No	all	Comma-separated task ids to run — for debugging, not for publishing
AGENTIC_FIXTURE_PORT	No	8899	Port the harness serves its fixture pages on
AGENTIC_FIXTURE_BASE_URL	No	auto	URL every stack uses to reach the fixture pages. Set this when the Owl instance cannot reach the harness's auto-detected LAN address
AGENTIC_FIXTURE_HOST	No	auto	Just the host part, if auto-detection picks the wrong interface
OWL_MCP_PROFILE	No	agent	Owl MCP tool surface. Over HTTP this is set on the Owl instance, not by the harness
AGENTIC_OWL_TRANSPORT	No	http	http (POST $OWL_BROWSER_URL/mcp) or stdio (npx @olib-ai/owl-browser-mcp)
OWL_MCP_URL	No	$OWL_BROWSER_URL/mcp	Override the Owl MCP endpoint
AGENTIC_PLAYWRIGHT_MCP_SPEC	No	@playwright/mcp@0.0.79	npm spec for Playwright MCP — pinned, change deliberately
AGENTIC_CHROME_DEVTOOLS_MCP_SPEC	No	chrome-devtools-mcp@1.6.0	npm spec for Chrome DevTools MCP
AGENTIC_OWL_MCP_SPEC	No	@olib-ai/owl-browser-mcp@2.1.4	npm spec for the Owl stdio MCP server (stdio transport only)

Output

Detection report

/output/
  report.json           # Parsed metrics + screenshot paths
  screenshots/
    playwright.webp     # Vanilla Playwright baseline
    puppeteer.webp      # Vanilla Puppeteer baseline
    owl-windows.webp    # Owl Browser with Windows profile
    owl-macos.webp      # Owl Browser with macOS profile
    owl-linux.webp      # Owl Browser with Linux profile

Benchmark

/output/
  benchmark.json        # Timing data for all three browsers

Agentic benchmark

/output/
  agentic.json          # Per-(agent, stack, task) accuracy and cost across the matrix

All files are overwritten on each run — no historical data stored.

Benchmark Methodology

All three browsers are benchmarked sequentially in the same Docker container on the same machine:

1. Cold start — Playwright/Puppeteer: launch a new browser process + create page. Owl Browser: create a new context within the running engine.
2. Navigation — Navigate to https://example.com and wait for networkidle.
3. Screenshot — Capture a viewport screenshot.
4. Full cycle — Create → navigate → screenshot → close.

Each step is timed individually. 10 iterations per browser. Results include min, max, avg, median, p95, and raw timing arrays so anyone can verify.

The architectural difference: Playwright and Puppeteer launch a new OS process for each browser instance. Owl Browser creates lightweight contexts within an already-running engine — no process spawn overhead.

Agentic benchmark

--agentic answers a different question from the timing benchmark: when a real LLM agent uses each MCP browser server to do real work, what does it cost and does it get the right answer?

The matrix is 2 agent CLIs x 3 MCP servers x 6 tasks x 5 runs, every cell driving the same locally hosted weights.

What is held constant

Dimension	Value
Model	One local model (default qwen/qwen3.6-27b) served over an OpenAI/Anthropic-compatible endpoint
Prompt	One template, in src/agentic.py, copied into agentic.json. Never tuned per stack or per agent
Tasks	The same six, in the same order
Runs	AGENTIC_RUNS (default 5) per cell
Machine	All arms on the same host, interleaved in time

Pinning the model locally is a credibility feature: anyone can reload the same weights, quantization and context length and re-run. A hosted model cannot offer that. It also means USD cost is meaningless — the agent CLIs apply hosted pricing to a local model — so cost is reported as null and the raw CLI figure is kept only as cost_usd_reported in the run details.

Agents

id	Agent	How it is driven
claude-code-cli	Claude Code	ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN, -p print mode, --output-format stream-json
codex-cli	Codex CLI	Isolated CODEX_HOME with a custom model_provider pointing at <base>/v1, plus a namespace-flattening shim

Codex specifics that were established by testing, not by reading docs: wire_api must be "responses" ("chat" is rejected by codex-cli 0.146.0); the provider id lmstudio is reserved and cannot be redefined, so a custom id is used; codex exec blocks on stdin even when the prompt is an argument, so stdin is closed; and an inline bearer_token is rejected, so the MCP token is passed via bearer_token_env_var.

default_tools_approval_mode is set to approve for every stack. The default, auto, defers to each tool's own annotations, so a server that ships read-only hints has its calls auto-approved while one that does not gets an approval prompt a non-interactive run cannot answer — the call comes back as user cancelled MCP tool call and never reaches the server. That is a harness artifact, not a product difference, and it was caught by the transport-vs-client divergence metric rather than by inspection.

The Codex namespace shim. Codex sends MCP tools to the model wrapped in an OpenAI-Responses container:

{"type": "namespace", "name": "mcp__owl", "tools": [{"type": "function", ...}]}

The local model ignores namespace-typed tools, so the model reports having no browser tools and starts guessing. A controlled A/B isolated it to the wrapper: namespaced produced no tool call, the identical tool flattened produced one. So src/agentic_proxy_llm.py sits in front of the model endpoint, flattens those tools, and re-annotates the returned function_call with the namespace field Codex's own router needs — the qualified-name and dotted-name forms are both rejected by that router, which is why the annotation is the one that works. Same tools, same schemas, same servers; only the encoding changes. Override with AGENTIC_LLM_FLATTEN (namespace default, or qualified / dotted / plain to reproduce the failures).

Antigravity (agy) is deliberately not an arm. It cannot be pointed at the pinned local weights (agy models returns nothing; --model has no base-url or provider option) and has no documented non-interactive auth, so a third party could not reproduce a run. agentic.json records this in not_benchmarkable rather than silently omitting it.

Stacks under test

id	Stack	Transport	Launch
owl	Owl Browser MCP	HTTP	POST $OWL_BROWSER_URL/mcp with a bearer token
playwright	Playwright MCP	stdio	npx -y @playwright/mcp@0.0.79 --headless --isolated --browser chromium
chrome-devtools	Chrome DevTools MCP	stdio	npx -y chrome-devtools-mcp@1.6.0 --headless --isolated

Versions are pinned, never @latest. Each stack is probed before the run; an unavailable one is recorded with "skipped": true and a reason, and the rest still run. Stacks live in one table in src/agentic_stacks.py; adding a fourth is one entry.

Measurement: at the transport, not at the agent

Every stack sits behind a counting proxy, so all arms are measured by the same code path instead of by each agent CLI's own bookkeeping (which differs between CLIs and silently drops call types in at least one of them):

• stdio stacks are wrapped by src/mcp_proxy_stdio.js — spawns the real server, pipes stdin/stdout byte for byte, passes stderr through untouched, parses a copy, counts JSON-RPC batch arrays element by element, and propagates signals and exit codes.
• the HTTP stack goes through src/agentic_proxy_http.py — a streaming reverse proxy that never buffers, because streamable HTTP may answer with SSE.

Both emit the same JSONL transport log, scored by src/agentic_transport.py.

Metrics

Per (agent, stack, task, run), aggregated to median, mean, min, max and stdev:

The headline is whether the agent finished the job correctly. Stacks are ranked on correct_rate; everything else is context and tie-breakers. A stack that is cheap and fast but wrong is the worst outcome, not a good one.

Metric	Meaning
completed	The agent produced a final answer of any kind
correct	That answer matched ground truth — this is "finished the job"
wrong_answer	It produced a confident answer that was incorrect. The trust metric
gave_up_honestly	It said it could not find or do the thing. A good outcome, never counted as a wrong answer
outcome	One of correct, wrong, gave_up, timeout, blocked, harness_error
tool_calls.transport	Calls the server actually executed, from the proxy. Headline
tool_calls.client	Calls the agent attempted, from the CLI stream. Secondary
call_divergence_runs	Runs where the two disagree — how silent permission denial is detected
client_meta_calls	The agent's own MCP discovery calls (resources/*). Excluded from the browser-tool count so they cannot fake a divergence
time_to_first_action_ms	Run start to the first tools/call reaching the server. Separates "slow to start" from "slow to work"
agent_turns	Model turns
text_tokens (alias payload_tokens)	The text the servers returned, through one fixed tokenizer (tiktoken/cl100k_base, named in harness.tokenizer). Excludes images
image_tokens	Estimated token cost of the images the servers returned (see below)
payload_total_tokens / image_token_share	Text + image, and the fraction of the payload that is pixels
payload_bytes	Raw bytes. Reported for completeness, never headlined — a biased proxy
client_tokens	Model-server token usage split into input, cached, output
duration_ms	Wall clock for the whole agent run

completed and correct are never collapsed. A stack can be at 100% completed and 40% correct; that gap is the story.

Efficiency is reported over correct runs only, and separately over all runs. A failed run is usually short and cheap, so averaging failures in makes a bad stack look efficient. Each result row carries correct_runs and all_runs blocks, the summary carries *_median_correct_runs and *_median_all_runs, and efficiency_basis states which the comparison uses (correct_runs_only). With zero correct runs the correct-runs figures are null, not zero.

Every aggregate reports median, mean, min, max and stdev — never a bare mean.

If tiktoken is unavailable, payload_tokens is reported as null rather than estimated with a different tokenizer.

No composite score is computed. Showing the columns and letting the reader weigh them avoids picking weights that happen to favour us. Losses are published unchanged.

Cost is not reported. The model is local, so there is nothing to price, and the figure the agent CLIs emit is a hosted pricing table applied to local token counts (Claude Code reported USD 0.39 for a local run). It is retained per run as cost_usd_reported for auditing only; cost_note in the JSON says so.

Auditing the grading

Every run publishes its raw answer text, the expected value, and the grading_rule that produced the verdict, so correctness calls can be checked rather than trusted. Rules are mechanical — exact value, whole-word regex, or "the first standalone integer must be 6" — never a fuzzy "looks about right". Each task also reports any_stack_correct, so a task nobody solved reads as a hard task rather than as a differentiator.

Images are counted, and they count against everyone

A benchmark that counts only text flatters whichever stack leans hardest on screenshots. Images are therefore first-class, and the accounting is symmetric: if an Owl run calls a screenshot tool, its image tokens count exactly the same.

Field	Meaning
image_blocks	Image content blocks the server returned
image_bytes	Decoded bytes of those images
image_dimensions	WxH per image, parsed out of the PNG/JPEG/WebP header — never a declared field
image_tokens	Estimated from those dimensions
image_tokens_basis	estimated, measured or none — never mixed silently. measured is reserved for a model server that reports image tokens separately; the local server does not, so real runs report estimated
image_dimensions_measured	False if a header could not be parsed and a viewport-sized fallback was used
image_sources	The tools that returned images
image_blocks_forced / image_blocks_chosen	Images from a non-screenshot tool (the stack's normal output gave the model no choice) vs. images from a screenshot tool (the model asked)

The formula is recorded in harness.image_token_formula so anyone can check the arithmetic or recompute with a different one: ceil(width * height / 750). Token usage reported by a local model server folds image tokens into the input count without breaking them out, so the estimate is computed from measured pixels rather than guessed from a usage field — and labelled as an estimate everywhere it appears.

Vision required: a capability metric, not a cost line

Reported next to correctness, not buried in the efficiency column. A stack that can be driven correctly with zero images can be driven by a text-only model; one that cannot, cannot — at any price. That is a categorical difference, and it is exactly the segment small, fast, local, open text-only models serve.

agentic.json carries a top-level vision object (by_stack and by_agent_stack), the same block on every result row (results[].vision) and every task (tasks[].vision), plus text_only per run:

Field	Meaning
vision_required	True when no correct run finished without an image. null if there were no correct runs
vision_mode	none, forced, chosen or mixed
text_only_runs / text_only_correct_runs / correct_runs / total_runs	The counts behind it
text_only_correct_rate	Correct runs that used zero images
images_per_correct_run	Average image blocks per correct run
image_blocks_forced / image_blocks_chosen / image_sources	Structural vs. model-chosen, and which tools produced them

The claim is deliberately narrow and is repeated inside the JSON: on these tasks, with this model, stack X consumed N images per correct run and stack Y consumed zero. It is never stated as "stack X requires vision" in general — Playwright MCP has an accessibility-snapshot mode and Chrome DevTools MCP has text affordances, so a stack that could have run text-only while the model reached for pixels anyway is a different, weaker finding (chosen, a usability observation) than a stack whose normal tool output returns an image (forced, a structural one). If a competitor completes the suite text-only, that is published unchanged.

The prompt is byte-identical, and that is asserted

The prompt is not tuned for Owl. It lives in exactly one place (PROMPT_TEMPLATE in src/agentic.py), is built once per task, and is reused verbatim by every agent and every stack. Task text states the goal only — it names no tool, no workflow and no vocabulary belonging to any one server.

This is enforced mechanically, not just intended:

Field	Meaning
results[].run_details[].prompt_sha256	Hash of the exact prompt bytes that arm was sent
tasks[].prompt_sha256 / tasks[].prompt_text	The canonical hash and the full text, for inspection
harness.prompt_identity_verified	False if any arm's hash differed
harness.prompt_mismatches	Which arms differed
harness.run_config / config_sha256	Model, timeout, turn limit, decoding settings, retry policy — recorded and hashed per run

A hash mismatch logs an error and makes the run exit non-zero: an unverified fairness claim is worth nothing.

The one legitimate difference is that each stack advertises its own tools and its own server instructions — that is the product under test, and it is why the baseline block matters. It is disclosed rather than hidden: stacks[].tools_catalog and stacks[].server_instructions dump each server's full advertised surface, captured from the wire, so the difference is auditable rather than asserted.

No arm is helped past a failure: no per-stack retries, no extra turn for one arm, no per-stack grading leniency.

The baseline section

agentic.json carries a baseline entry per (agent, stack): one no-op prompt that forbids tool use, measuring what merely connecting the server costs. This is dominated by the tool schemas the model has to carry, so a larger advertised tool surface taxes every task before any work is done. tool_count is therefore reported as a fairness factor, not a neutral fact.

Measured with the configurations above: Owl advertises 19 tools, Playwright MCP 24, Chrome DevTools MCP 29 — but tool count and schema size are not the same thing, so the baseline reports the tokenized tools/list payload alongside the count.

Tasks

Six tasks with verifiable ground truth, each a different shape:

id	Shape	Target	What it exercises
cheapest-mystery-book	list-extraction	live	Compare prices across a paginated category listing
catalogue-last-page	deep-pagination	live	Reach page 50 of 50 and read the last row
book-upc-table	table-read	live	Pull one exact value out of an HTML table
travel-stock-drilldown	detail-drilldown	live	Open a detail page from a listing and read a number only visible there
quote-form-submit	form-fill	fixture	Fill six fields, submit, report the server's confirmation code
restricted-figure	negative-control	fixture	The correct answer is "not available" — catches agents that hallucinate

Live targets are books.toscrape.com, a purpose-built scraping sandbox whose catalogue is frozen; those tasks carry live_site: true so they can be reported separately if the site ever changes. The rest are served by the harness from src/agentic_fixtures/. The form's confirmation code is only rendered after a submission where all six fields match exactly, so it cannot be guessed.

Fixture pages are served on 0.0.0.0:$AGENTIC_FIXTURE_PORT and advertised at the harness's detected LAN address, because the Owl instance usually runs in a different container. If Owl cannot reach that address, set AGENTIC_FIXTURE_BASE_URL. When the fixture site is unreachable its two tasks are dropped for every arm and listed in harness.skipped_tasks — never dropped for one arm only.

Each task records solved_by_any_stack and solved_by, so a task no arm could do reads as a hard task rather than as a bad stack.

Fairness and known deviations

• One prompt template for everything; it is written into agentic.json so anyone can check.
• Claude Code runs with every built-in tool disabled (--tools ""), so the MCP server is its only capability.
• Codex cannot disable its built-in shell. It runs under the read-only sandbox, which has no network access, so the MCP server is still its only route to the web — but the shell tool stays visible to the model. This is recorded in harness.notes.
• Runs are interleaved (run -> task -> agent -> stack) so no arm gets a systematically better slice of the network.
• An arm that cannot do a task is recorded as failing it, never omitted.

The transport is not symmetric, and it is not corrected for. Owl is reached over HTTP (POST $OWL_BROWSER_URL/mcp), so every one of its tool calls pays a network round trip. The other two are stdio servers npx-spawned on the same machine as the agent, so theirs is a pipe write. That difference favours the stdio stacks on every wall-clock number here, and no adjustment is applied — an adjustment would be a guess, and a benchmark that quietly corrects its own numbers in its own favour is worth nothing.

It is also the architectural trade being measured rather than an accident. A stdio MCP server is one browser bound to one agent process, spawned and torn down with it. Owl is a server: one instance serves many agents at once, holds an isolated context per agent, and outlives all of them. For a single agent on a laptop the stdio shape is better. For a fleet that shape does not exist. Read the wall-clock column with that in mind — token and tool-call counts are unaffected by transport.

Running it

export OWL_BROWSER_URL=http://localhost          # Owl instance
export OWL_BROWSER_TOKEN=your-token
export AGENTIC_BASE_URL=http://<lm-studio-host>:1234
export AGENTIC_AUTH_TOKEN=lmstudio
export AGENTIC_MODEL=qwen/qwen3.6-27b
export OUTPUT_DIR=./output

python -m src.main --agentic

Smoke test one cell first — the full matrix is agents x stacks x 6 tasks x AGENTIC_RUNS agent invocations plus one baseline per (agent, stack), and a local 27B model is not fast:

AGENTIC_RUNS=1 AGENTIC_AGENTS=claude-code-cli AGENTIC_STACKS=owl \
  AGENTIC_TASKS=book-upc-table python -m src.main --agentic

samples/agentic.sample.json is a real (not fabricated) reduced run for front-end development: "sample": true, one run per cell, a subset of tasks. It must never be published as a release measurement.

VPS Deployment

One-time setup

# Login to GitHub Container Registry
echo "YOUR_GITHUB_PAT" | docker login ghcr.io -u YOUR_GITHUB_USERNAME --password-stdin

# Create env file
sudo nano /etc/owl-report.env
# Paste your production config (OWL_BROWSER_URL, token, AWS creds)

# Test detection report
docker pull ghcr.io/olib-ai/owl-detection-report:latest
docker run --rm --network host --env-file /etc/owl-report.env ghcr.io/olib-ai/owl-detection-report:latest

# Test benchmark
docker run --rm --network host --env-file /etc/owl-report.env ghcr.io/olib-ai/owl-detection-report:latest --benchmark

# Run per release — no cron needed

Updating

docker pull ghcr.io/olib-ai/owl-detection-report:latest