feat: Web Bot Auth cryptographic verification (0.14.0) - #22
Merged
Conversation
Cloudflare's post introducing this mechanism is titled 'Forget IPs: using cryptography to verify bot and agent traffic'. The published-IP-range check shipped in 0.11 is precisely the method it retires, so the library should implement the successor rather than defend the predecessor. Web Bot Auth is an RFC 9421 HTTP Message Signatures profile — Ed25519 per request, a Signature-Agent header naming the key directory, keys at /.well-known/http-message-signatures-directory. Backed by Cloudflare, Amazon, Akamai and OpenAI, IETF working group chartered 2026. It dominates ranges on every axis that bit us: any agent that signs rather than four vendors, no freshness problem, no false 'spoofed' from a stale list, and it covers agents running on a user's own machine — the case ranges structurally cannot. Adds combinedVerifier(), which prefers a signature and falls back to ranges, so adoption shifts the mix with no change at the call site. Two deliberate rules: - A present-but-invalid signature returns 'spoofed' even when the client IP sits in a published range. Otherwise a forged signature could be laundered into 'verified' by the weaker check. - Unsigned traffic is 'unverifiable', never 'spoofed'. Most agents do not sign yet; treating silence as forgery would mislabel nearly all real traffic. Unsigned requests return before any I/O — the common path today costs nothing. Signed ones fetch the signer's directory once per origin and cache it, keyed by origin, for an hour. Structured-field parsing is scoped to the shapes this profile emits rather than a full RFC 8941 implementation, and returns null rather than guessing: a malformed header must never read as a valid signature. Tests 246 -> 262. The suite generates a real Ed25519 keypair and signs the actual signature base, then checks replay onto another path, a tampered signature, a key absent from the directory, expiry both by `expires` and by maxAge, non-https signers, an unreachable directory, allowedSigners, that unsigned traffic triggers no fetch, and that the directory is cached. Fixture signatures would only have encoded whatever the parser happens to do.
Gdewilde
added a commit
that referenced
this pull request
Aug 2, 2026
…ders Over 2.5 million sites answer bulk AI crawling with robots.txt Disallow. That leaves money on the table and only works if the crawler cooperates. The alternative is to let them train and price it. Pricing only works if you can tell training from retrieval, because they have opposite economics. A GPTBot fetch is corpus collection you get nothing back for. A ChatGPT-User fetch is a person asking about you, and billing that is billing your own distribution channel. agentPolicy already draws the line; this turns a 'charge' decision into the HTTP challenge. 402 training GPTBot/1.1 serve retrieval ChatGPT-User/1.0 402 training ClaudeBot/1.0 serve retrieval Claude-User (claude-code/2.1) serve search Googlebot/2.1 That distinction is the differentiator. Gateway pay-per-crawl charges crawls indiscriminately; charging the half that sends you demand is self-harm. paymentRequired() emits x402's wire format — 402 with a base64 PAYMENT-REQUIRED header — and hasPaymentPayload()/paymentPayload() read the client's PAYMENT-SIGNATURE retry. The default Content-Signal is 'search=yes, ai-input=yes, ai-train=paid', inverting the library's own ai-train=no default: the premise is that training is for sale, not forbidden. Scope, deliberately: this emits challenges and reads headers. It settles nothing. Settlement belongs to an x402 facilitator or Stripe's MPP — holding money would drag PCI scope into something meant to drop into middleware. The library never invents an amount, network or asset; those are all caller-supplied. 'meter' returns null rather than a gate, because metering is an accounting concern and the request should still be served while trackVisit records it. 'block' returns 403, not a price: a failed identity check is not a negotiation. Tests 246 -> 257, including non-ASCII in the challenge (btoa is Latin-1 only and a naive encoder throws), refusing to emit a challenge with no way to pay, and that retrieval and search stay free under a charge-training policy. NOTE: branched from main, so it does not include #22 (Web Bot Auth, 0.14.0). Merge #22 first; this is versioned 0.15.0 on that assumption.
Gdewilde
added a commit
that referenced
this pull request
Aug 2, 2026
Two things, both found by running the composed stack rather than the units.
paymentGate could not use Web Bot Auth at all.
Verification is necessarily async — verifyWebBotAuth fetches the signer's key
directory — but agentPolicy is synchronous, so combinedVerifier() could not be
passed to it. In JS that failed silently: agentPolicy read `.verdict` off a
promise, got undefined, and never applied the spoofed check. In TS it was a
compile error, which is better but still means the package's two headline
features could not be used together.
Caught by an integration check: ClaudeBot from a DigitalOcean address returned
402 charge when it should have been 403 block. Unit tests all passed either
way, because each layer was correct alone.
paymentGate is already async, so it now awaits the verifier and passes the
resolved verification to agentPolicy via a new `verification` option.
agentPolicy stays synchronous, which is worth keeping for callers who only
want UA classification.
Verified across the composed stack:
402 training charge GPTBot
serve retrieval allow ChatGPT-User
serve training charge GPTBot, paid
403 training block ClaudeBot from an unpublished IP
402 training charge ClaudeBot from a real Anthropic IP
serve search allow Googlebot
Payments are marked experimental.
The protocols are weeks old and moving. x402 and MPP are both live but their
specs are unstable, MPP had not publicly pinned a settlement-confirmation
header at the time of writing, and no agent in our own production traffic has
yet presented a payment credential. Detection, verification and policy are
stable and should be treated as such; this surface should not. Stated in
payments.ts, gateway.ts and the README rather than only in a PR description,
since that is where someone will actually read it.
Rebased onto 0.14.0 (Web Bot Auth) and resolved the package.json version
conflict to 0.15.0. Tests 294 -> 310 after picking up #22's suite.
Gdewilde
added a commit
that referenced
this pull request
Aug 2, 2026
…ders (#23) * feat: charge LLMs to train, without charging the ones sending you readers Over 2.5 million sites answer bulk AI crawling with robots.txt Disallow. That leaves money on the table and only works if the crawler cooperates. The alternative is to let them train and price it. Pricing only works if you can tell training from retrieval, because they have opposite economics. A GPTBot fetch is corpus collection you get nothing back for. A ChatGPT-User fetch is a person asking about you, and billing that is billing your own distribution channel. agentPolicy already draws the line; this turns a 'charge' decision into the HTTP challenge. 402 training GPTBot/1.1 serve retrieval ChatGPT-User/1.0 402 training ClaudeBot/1.0 serve retrieval Claude-User (claude-code/2.1) serve search Googlebot/2.1 That distinction is the differentiator. Gateway pay-per-crawl charges crawls indiscriminately; charging the half that sends you demand is self-harm. paymentRequired() emits x402's wire format — 402 with a base64 PAYMENT-REQUIRED header — and hasPaymentPayload()/paymentPayload() read the client's PAYMENT-SIGNATURE retry. The default Content-Signal is 'search=yes, ai-input=yes, ai-train=paid', inverting the library's own ai-train=no default: the premise is that training is for sale, not forbidden. Scope, deliberately: this emits challenges and reads headers. It settles nothing. Settlement belongs to an x402 facilitator or Stripe's MPP — holding money would drag PCI scope into something meant to drop into middleware. The library never invents an amount, network or asset; those are all caller-supplied. 'meter' returns null rather than a gate, because metering is an accounting concern and the request should still be served while trackVisit records it. 'block' returns 403, not a price: a failed identity check is not a negotiation. Tests 246 -> 257, including non-ASCII in the challenge (btoa is Latin-1 only and a naive encoder throws), refusing to emit a challenge with no way to pay, and that retrieval and search stay free under a charge-training policy. NOTE: branched from main, so it does not include #22 (Web Bot Auth, 0.14.0). Merge #22 first; this is versioned 0.15.0 on that assumption. * feat(payments): speak MPP as well as x402 The first cut hardcoded x402's framing while the PR text claimed settlement could go to 'an x402 facilitator or Stripe's MPP'. It could not — MPP uses a different wire format. x402 PAYMENT-REQUIRED: <base64 JSON> -> PAYMENT-SIGNATURE MPP WWW-Authenticate: Payment id="…" -> Authorization: Payment … MPP reuses standard HTTP authentication framing rather than defining its own headers, which means the two do not collide: a single 402 can advertise both and let the agent take whichever it speaks. paymentRequired() does exactly that when handed both challenges. paymentPayload() now returns { protocol, value } instead of a bare string, and checks the `Payment` auth-scheme before treating an Authorization header as a credential. Without that check a site behind ordinary Bearer or Basic auth would look like every request had already paid — a security-relevant confusion, so there is a test for it. WWW-Authenticate values are quoted and escaped per RFC 9110, and appended rather than set, since the header legitimately carries multiple challenges. withSettlement() takes an optional header name. x402 defines PAYMENT-RESPONSE; MPP's public spec did not pin a settlement-confirmation header at the time of writing, so the caller names what their provider expects rather than the library inventing one. Tests 257 -> 263. * fix: agentIntent and agentPolicy disagreed on every HTTP client Both are exported from the package root, and they returned different intents for the same user agent. The 'tooling' promotion for HTTP-library UAs lived only inside agentPolicy, so: agentIntent('curl/8.4.0') -> 'unknown' agentPolicy(req(curl)).intent -> 'tooling' Six of six HTTP-client UAs disagreed. A caller reaching for the obviously-named function got the wrong answer with no signal that a second source existed. Found it by hitting it: the site's /api/whoami called both and rendered "intent: unknown" beside "reason: coding agent or HTTP client" — visibly self-contradictory. I patched that call site and moved on, which left the trap exported for everyone else. agentIntent now checks isHttpClient itself, which it can do from the UA alone, and agentPolicy reads intent from it rather than re-deriving. One source of truth instead of two that happened to agree most of the time. Pinned with a 19-UA corpus asserting agentIntent(ua) === agentPolicy(req).intent for every entry. That invariant is the actual fix; the promotion moving is just how it is satisfied. Tests 274 -> 294. * fix: let paymentGate await an async verifier; mark payments experimental Two things, both found by running the composed stack rather than the units. paymentGate could not use Web Bot Auth at all. Verification is necessarily async — verifyWebBotAuth fetches the signer's key directory — but agentPolicy is synchronous, so combinedVerifier() could not be passed to it. In JS that failed silently: agentPolicy read `.verdict` off a promise, got undefined, and never applied the spoofed check. In TS it was a compile error, which is better but still means the package's two headline features could not be used together. Caught by an integration check: ClaudeBot from a DigitalOcean address returned 402 charge when it should have been 403 block. Unit tests all passed either way, because each layer was correct alone. paymentGate is already async, so it now awaits the verifier and passes the resolved verification to agentPolicy via a new `verification` option. agentPolicy stays synchronous, which is worth keeping for callers who only want UA classification. Verified across the composed stack: 402 training charge GPTBot serve retrieval allow ChatGPT-User serve training charge GPTBot, paid 403 training block ClaudeBot from an unpublished IP 402 training charge ClaudeBot from a real Anthropic IP serve search allow Googlebot Payments are marked experimental. The protocols are weeks old and moving. x402 and MPP are both live but their specs are unstable, MPP had not publicly pinned a settlement-confirmation header at the time of writing, and no agent in our own production traffic has yet presented a payment credential. Detection, verification and policy are stable and should be treated as such; this surface should not. Stated in payments.ts, gateway.ts and the README rather than only in a PR description, since that is where someone will actually read it. Rebased onto 0.14.0 (Web Bot Auth) and resolved the package.json version conflict to 0.15.0. Tests 294 -> 310 after picking up #22's suite. * test: integration suite for the composed stack, wired into CI Every bug found in this library after the unit suite went green was a composition bug — each layer correct alone, contradicting the next: - agentIntent returned 'unknown' where agentPolicy returned 'tooling', for every HTTP-library UA. Both exported, both passing their own tests. - paymentGate silently dropped an async verifier, so a spoofed ClaudeBot was charged (402) instead of blocked (403). All 310 unit tests passed. Unit tests structurally cannot see that class of defect, so this adds a suite that can. Two parts: 1. An end-to-end table — 12 realistic requests in, HTTP status, intent, action and emitted bot_name out. One row per behaviour worth guaranteeing, including the ones the design exists to prevent. 2. Cross-layer invariants — properties that must hold *between* layers whatever each does internally: - agentIntent agrees with the intent agentPolicy reports - the emitted event never contradicts the classifier - an async verifier actually reaches the decision - a spoofed verdict always blocks and never merely prices - retrieval is never gated, under any policy configuration - unverifiable never becomes spoofed anywhere in the stack - no raw IP is emitted unless captureIp is set, whatever else is enabled Verified the suite earns its place by reintroducing both bugs: bug 1 (agentIntent) unit: 10 failed integration: 1 failed bug 2 (async verifier) unit: 310 PASSED integration: 3 failed Bug 2 is the argument for the file: invisible to the unit suite, caught here. Split npm scripts into test:unit and test:integration, and CI runs them as separate steps so "unit passes, integration fails" is legible at a glance rather than buried in one combined run. Tests 310 -> 329. * feat(payments): metering as the primary path, plus bulk entitlements The first cut led with per-request 402, which is what x402 and MPP define and the wrong shape for a training sweep. On one production site training traffic is ~199,000 requests a month. Charging each one means three times the traffic once pay-and-retry is added, 199,000 settlements whose per-transaction cost exceeds any sane per-page price, and — decisively — no crawler in the wild retries a 402. Per-request charging is blocking with extra steps, which is the outcome this whole design argues against. Two workable shapes, both now supported. METERING, first-class rather than a side effect of onDecision. A Meter interface, and paymentGate calls it for every 'meter' decision: serve the request, count the unit, bill out of band. No crawler cooperation, no protocol dependency, works today, and it produces the only number worth taking into a licensing conversation. Errors are swallowed — a metering failure must not become a failed response. Retrieval and search are never metered. ENTITLEMENTS, so a sweep is sold a licence instead of a page. entitlementGateway takes a store and a BulkOffer. One 402 advertises the offer, one settlement issues a credential, and every later request presents it, is served directly, and decrements quota. One settlement per licence rather than per page. Unmetered licences are usable and still call consume(), so you can count without capping. Unknown, expired and exhausted credentials return an identical challenge — distinguishing them would make the endpoint an oracle for probing quota state, and there is a test asserting the bodies match. No storage ships with it beyond an in-memory store marked test-only: quota state is money, and it belongs in the caller's KV or database, not in a library that runs per-instance at the edge. x-quota-remaining is opt-in and documented as belonging to neither protocol. README and the site section are re-pitched around metering. Presenting per-request charging as the headline overstated what is practical. Tests 329 -> 341. * feat: Vercel WAF rule recommender, and a payments testing guide FIREWALL RECOMMENDER recommendFirewallRules() turns aggregated traffic into staged Vercel WAF proposals, and firewallScript() renders them as commented bash. It proposes; it never enforces. Every rule is emitted with action 'log', because a rule's blast radius is unknowable until real traffic hits it, and Vercel stages rule changes as drafts anyway — nothing is live until a human publishes. The generated script deliberately stops at `vercel firewall diff` and prints the publish command rather than running it. Two invariants, both derived from measurement rather than preference: - Retrieval agents and search crawlers are never proposed for blocking, and a 'bypass' rule protecting them is emitted first. Vercel evaluates rules top to bottom, so without that ordering a user-agent rule below would swallow the agents that bring readers — 60% of AI traffic on one production site. There is a test asserting no enforcing rule's conditions ever mention ChatGPT-User, Claude-User, Perplexity-User or Googlebot. - Training crawlers get rate limits, not denials. Removing yourself from future training sets is a discoverability decision, not a default, and the caveat says so. Only failed verification earns a proposed deny. Every recommendation carries evidence, a risk rating and a caveat: the datacenter-ASN rule is marked high risk because corporate VPNs, privacy relays and some mobile carriers egress from hosting ASNs, and a challenge page breaks API clients outright. PAYMENTS TESTING GUIDE docs/TESTING-PAYMENTS.md, four levels cheapest first: pure functions with no keys, curl against a running app, Web Bot Auth signatures, then real settlement via Stripe MPP or an x402 facilitator. The level-1 script was run verbatim from a clean install of the packed package and its documented output is the actual output, not an illustration. It closes with what none of it tests, which matters more than the checklist: no real crawler retries a 402 today, so charging per request is functionally blocking; memoryEntitlementStore has no atomic decrement so concurrent regions can oversell a licence; and no public price exists for a training crawl. Tests 341 -> 356. * fix(posthog): mirror the UA and IP to PostHog's canonical properties PostHog runs its own user-agent and GeoIP enrichment, but only off specific property names. We sent the user agent under our own key, so on 4,904 events in one production project PostHog recorded: $virt_traffic_category = "no_user_agent" $virt_bot_name = "" $virt_bot_operator = "" Its entire bot taxonomy sat dormant. Confirmed against PostHog's docs: every classification function reads properties.$raw_user_agent — getTrafficCategory(properties.$raw_user_agent) -> ai_crawler, ai_search, ... getBotName(properties.$raw_user_agent) -> 'ChatGPT', 'Googlebot', ... and GeoIP reads properties.$ip. Their own Vercel log-drain source emits both for exactly this reason. The adapter now mirrors what it already carries. No new data is collected: the user agent is the same string already on the event under `user_agent`, and $ip is mirrored only when the caller opted into captureIp — adding it otherwise would put a raw address on an event they deliberately anonymised. Worth having because it is a free second opinion. PostHog classifies from far more traffic than we see, so disagreement between bot_name and getBotName($raw_user_agent) is a cheap signal for where our patterns are wrong. The GeoIP fix matters too: without $ip, PostHog geolocates whichever edge PoP relayed the event rather than the client, which is approximately right and quietly not authoritative. I had previously dismissed the $virt_* family as unusable "because events arrive server-side from middleware". That was the wrong mechanism — it is unusable because we send the UA under the wrong key, which is fixable in one place. Tests 356 -> 360.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cloudflare's post introducing this mechanism is titled "Forget IPs: using cryptography to verify bot and agent traffic". The published-IP-range check we shipped in
0.11is precisely the method it retires — so the library should implement the successor rather than defend the predecessor.What it is
An RFC 9421 HTTP Message Signatures profile: Ed25519 per request, a
Signature-Agentheader naming the key directory, public keys at/.well-known/http-message-signatures-directory. Backed by Cloudflare, Amazon, Akamai and OpenAI, with an IETF working group chartered in 2026. Newly relevant because Cloudflare and GoDaddy are pushing it into mainstream hosting.Why it dominates ranges
spoofedEvery caveat in
verify.ts— the freshness warning, theclient-side-agentcarve-out, thex-forwarded-forprecondition — exists because IP verification is weak. Signatures dissolve all three.Usage
Prefers the signature, falls back to ranges. As signing adoption grows the mix shifts with no change at the call site.
Two rules worth reviewing carefully
A present-but-invalid signature is decisive — it returns
spoofedeven when the client IP sits in a published range. Otherwise a forged signature could be laundered intoverifiedby the weaker check. There's a test for exactly this.Unsigned traffic is
unverifiable, neverspoofed. Most agents don't sign yet; treating silence as forgery would mislabel nearly all real traffic.Cost
Unsigned requests return before any I/O — verified by a test asserting
fetchis never called. That's the overwhelmingly common path today. Signed requests fetch the signer's directory once per origin and cache it for an hour.Tests: 246 → 262
The suite generates a real Ed25519 keypair and signs the actual RFC 9421 signature base, rather than using fixtures — fixtures would only encode whatever my parser happens to do. It then checks: replay onto a different path, a tampered signature, a key absent from the directory, expiry via both
expiresandmaxAge, non-https signer origins, an unreachable directory,allowedSigners, no-fetch-on-unsigned, and directory caching.One test initially failed and the test was wrong: it reused one signer origin with two different key directories, and the (correctly origin-keyed) cache served the first. Worth knowing that cache semantics are origin-scoped.
Scope
Structured-field parsing covers the shapes this profile emits rather than implementing all of RFC 8941, and returns
nullrather than guessing — a malformed header must never read as a valid signature.Bundle: root unchanged at 11,474 B;
verifygrows 14,167 → 18,986 B, paid only by consumers that import it.🤖 Generated with Claude Code