feat: charge LLMs to train, without charging the ones sending you readers - #23
Merged
Conversation
…ders Over 2.5 million sites answer bulk AI crawling with robots.txt Disallow. That leaves money on the table and only works if the crawler cooperates. The alternative is to let them train and price it. Pricing only works if you can tell training from retrieval, because they have opposite economics. A GPTBot fetch is corpus collection you get nothing back for. A ChatGPT-User fetch is a person asking about you, and billing that is billing your own distribution channel. agentPolicy already draws the line; this turns a 'charge' decision into the HTTP challenge. 402 training GPTBot/1.1 serve retrieval ChatGPT-User/1.0 402 training ClaudeBot/1.0 serve retrieval Claude-User (claude-code/2.1) serve search Googlebot/2.1 That distinction is the differentiator. Gateway pay-per-crawl charges crawls indiscriminately; charging the half that sends you demand is self-harm. paymentRequired() emits x402's wire format — 402 with a base64 PAYMENT-REQUIRED header — and hasPaymentPayload()/paymentPayload() read the client's PAYMENT-SIGNATURE retry. The default Content-Signal is 'search=yes, ai-input=yes, ai-train=paid', inverting the library's own ai-train=no default: the premise is that training is for sale, not forbidden. Scope, deliberately: this emits challenges and reads headers. It settles nothing. Settlement belongs to an x402 facilitator or Stripe's MPP — holding money would drag PCI scope into something meant to drop into middleware. The library never invents an amount, network or asset; those are all caller-supplied. 'meter' returns null rather than a gate, because metering is an accounting concern and the request should still be served while trackVisit records it. 'block' returns 403, not a price: a failed identity check is not a negotiation. Tests 246 -> 257, including non-ASCII in the challenge (btoa is Latin-1 only and a naive encoder throws), refusing to emit a challenge with no way to pay, and that retrieval and search stay free under a charge-training policy. NOTE: branched from main, so it does not include #22 (Web Bot Auth, 0.14.0). Merge #22 first; this is versioned 0.15.0 on that assumption.
The first cut hardcoded x402's framing while the PR text claimed settlement
could go to 'an x402 facilitator or Stripe's MPP'. It could not — MPP uses a
different wire format.
x402 PAYMENT-REQUIRED: <base64 JSON> -> PAYMENT-SIGNATURE
MPP WWW-Authenticate: Payment id="…" -> Authorization: Payment …
MPP reuses standard HTTP authentication framing rather than defining its own
headers, which means the two do not collide: a single 402 can advertise both
and let the agent take whichever it speaks. paymentRequired() does exactly that
when handed both challenges.
paymentPayload() now returns { protocol, value } instead of a bare string, and
checks the `Payment` auth-scheme before treating an Authorization header as a
credential. Without that check a site behind ordinary Bearer or Basic auth
would look like every request had already paid — a security-relevant confusion,
so there is a test for it.
WWW-Authenticate values are quoted and escaped per RFC 9110, and appended
rather than set, since the header legitimately carries multiple challenges.
withSettlement() takes an optional header name. x402 defines PAYMENT-RESPONSE;
MPP's public spec did not pin a settlement-confirmation header at the time of
writing, so the caller names what their provider expects rather than the
library inventing one.
Tests 257 -> 263.
Both are exported from the package root, and they returned different intents
for the same user agent. The 'tooling' promotion for HTTP-library UAs lived
only inside agentPolicy, so:
agentIntent('curl/8.4.0') -> 'unknown'
agentPolicy(req(curl)).intent -> 'tooling'
Six of six HTTP-client UAs disagreed. A caller reaching for the obviously-named
function got the wrong answer with no signal that a second source existed.
Found it by hitting it: the site's /api/whoami called both and rendered
"intent: unknown" beside "reason: coding agent or HTTP client" — visibly
self-contradictory. I patched that call site and moved on, which left the trap
exported for everyone else.
agentIntent now checks isHttpClient itself, which it can do from the UA alone,
and agentPolicy reads intent from it rather than re-deriving. One source of
truth instead of two that happened to agree most of the time.
Pinned with a 19-UA corpus asserting agentIntent(ua) === agentPolicy(req).intent
for every entry. That invariant is the actual fix; the promotion moving is just
how it is satisfied.
Tests 274 -> 294.
Two things, both found by running the composed stack rather than the units.
paymentGate could not use Web Bot Auth at all.
Verification is necessarily async — verifyWebBotAuth fetches the signer's key
directory — but agentPolicy is synchronous, so combinedVerifier() could not be
passed to it. In JS that failed silently: agentPolicy read `.verdict` off a
promise, got undefined, and never applied the spoofed check. In TS it was a
compile error, which is better but still means the package's two headline
features could not be used together.
Caught by an integration check: ClaudeBot from a DigitalOcean address returned
402 charge when it should have been 403 block. Unit tests all passed either
way, because each layer was correct alone.
paymentGate is already async, so it now awaits the verifier and passes the
resolved verification to agentPolicy via a new `verification` option.
agentPolicy stays synchronous, which is worth keeping for callers who only
want UA classification.
Verified across the composed stack:
402 training charge GPTBot
serve retrieval allow ChatGPT-User
serve training charge GPTBot, paid
403 training block ClaudeBot from an unpublished IP
402 training charge ClaudeBot from a real Anthropic IP
serve search allow Googlebot
Payments are marked experimental.
The protocols are weeks old and moving. x402 and MPP are both live but their
specs are unstable, MPP had not publicly pinned a settlement-confirmation
header at the time of writing, and no agent in our own production traffic has
yet presented a payment credential. Detection, verification and policy are
stable and should be treated as such; this surface should not. Stated in
payments.ts, gateway.ts and the README rather than only in a PR description,
since that is where someone will actually read it.
Rebased onto 0.14.0 (Web Bot Auth) and resolved the package.json version
conflict to 0.15.0. Tests 294 -> 310 after picking up #22's suite.
Gdewilde
force-pushed
the
feat/pay-to-train
branch
from
August 2, 2026 10:12
9e664ea to
bb7f2d3
Compare
Every bug found in this library after the unit suite went green was a
composition bug — each layer correct alone, contradicting the next:
- agentIntent returned 'unknown' where agentPolicy returned 'tooling', for
every HTTP-library UA. Both exported, both passing their own tests.
- paymentGate silently dropped an async verifier, so a spoofed ClaudeBot was
charged (402) instead of blocked (403). All 310 unit tests passed.
Unit tests structurally cannot see that class of defect, so this adds a suite
that can. Two parts:
1. An end-to-end table — 12 realistic requests in, HTTP status, intent, action
and emitted bot_name out. One row per behaviour worth guaranteeing, including
the ones the design exists to prevent.
2. Cross-layer invariants — properties that must hold *between* layers whatever
each does internally:
- agentIntent agrees with the intent agentPolicy reports
- the emitted event never contradicts the classifier
- an async verifier actually reaches the decision
- a spoofed verdict always blocks and never merely prices
- retrieval is never gated, under any policy configuration
- unverifiable never becomes spoofed anywhere in the stack
- no raw IP is emitted unless captureIp is set, whatever else is enabled
Verified the suite earns its place by reintroducing both bugs:
bug 1 (agentIntent) unit: 10 failed integration: 1 failed
bug 2 (async verifier) unit: 310 PASSED integration: 3 failed
Bug 2 is the argument for the file: invisible to the unit suite, caught here.
Split npm scripts into test:unit and test:integration, and CI runs them as
separate steps so "unit passes, integration fails" is legible at a glance
rather than buried in one combined run.
Tests 310 -> 329.
The first cut led with per-request 402, which is what x402 and MPP define and the wrong shape for a training sweep. On one production site training traffic is ~199,000 requests a month. Charging each one means three times the traffic once pay-and-retry is added, 199,000 settlements whose per-transaction cost exceeds any sane per-page price, and — decisively — no crawler in the wild retries a 402. Per-request charging is blocking with extra steps, which is the outcome this whole design argues against. Two workable shapes, both now supported. METERING, first-class rather than a side effect of onDecision. A Meter interface, and paymentGate calls it for every 'meter' decision: serve the request, count the unit, bill out of band. No crawler cooperation, no protocol dependency, works today, and it produces the only number worth taking into a licensing conversation. Errors are swallowed — a metering failure must not become a failed response. Retrieval and search are never metered. ENTITLEMENTS, so a sweep is sold a licence instead of a page. entitlementGateway takes a store and a BulkOffer. One 402 advertises the offer, one settlement issues a credential, and every later request presents it, is served directly, and decrements quota. One settlement per licence rather than per page. Unmetered licences are usable and still call consume(), so you can count without capping. Unknown, expired and exhausted credentials return an identical challenge — distinguishing them would make the endpoint an oracle for probing quota state, and there is a test asserting the bodies match. No storage ships with it beyond an in-memory store marked test-only: quota state is money, and it belongs in the caller's KV or database, not in a library that runs per-instance at the edge. x-quota-remaining is opt-in and documented as belonging to neither protocol. README and the site section are re-pitched around metering. Presenting per-request charging as the headline overstated what is practical. Tests 329 -> 341.
FIREWALL RECOMMENDER recommendFirewallRules() turns aggregated traffic into staged Vercel WAF proposals, and firewallScript() renders them as commented bash. It proposes; it never enforces. Every rule is emitted with action 'log', because a rule's blast radius is unknowable until real traffic hits it, and Vercel stages rule changes as drafts anyway — nothing is live until a human publishes. The generated script deliberately stops at `vercel firewall diff` and prints the publish command rather than running it. Two invariants, both derived from measurement rather than preference: - Retrieval agents and search crawlers are never proposed for blocking, and a 'bypass' rule protecting them is emitted first. Vercel evaluates rules top to bottom, so without that ordering a user-agent rule below would swallow the agents that bring readers — 60% of AI traffic on one production site. There is a test asserting no enforcing rule's conditions ever mention ChatGPT-User, Claude-User, Perplexity-User or Googlebot. - Training crawlers get rate limits, not denials. Removing yourself from future training sets is a discoverability decision, not a default, and the caveat says so. Only failed verification earns a proposed deny. Every recommendation carries evidence, a risk rating and a caveat: the datacenter-ASN rule is marked high risk because corporate VPNs, privacy relays and some mobile carriers egress from hosting ASNs, and a challenge page breaks API clients outright. PAYMENTS TESTING GUIDE docs/TESTING-PAYMENTS.md, four levels cheapest first: pure functions with no keys, curl against a running app, Web Bot Auth signatures, then real settlement via Stripe MPP or an x402 facilitator. The level-1 script was run verbatim from a clean install of the packed package and its documented output is the actual output, not an illustration. It closes with what none of it tests, which matters more than the checklist: no real crawler retries a 402 today, so charging per request is functionally blocking; memoryEntitlementStore has no atomic decrement so concurrent regions can oversell a licence; and no public price exists for a training crawl. Tests 341 -> 356.
PostHog runs its own user-agent and GeoIP enrichment, but only off specific property names. We sent the user agent under our own key, so on 4,904 events in one production project PostHog recorded: $virt_traffic_category = "no_user_agent" $virt_bot_name = "" $virt_bot_operator = "" Its entire bot taxonomy sat dormant. Confirmed against PostHog's docs: every classification function reads properties.$raw_user_agent — getTrafficCategory(properties.$raw_user_agent) -> ai_crawler, ai_search, ... getBotName(properties.$raw_user_agent) -> 'ChatGPT', 'Googlebot', ... and GeoIP reads properties.$ip. Their own Vercel log-drain source emits both for exactly this reason. The adapter now mirrors what it already carries. No new data is collected: the user agent is the same string already on the event under `user_agent`, and $ip is mirrored only when the caller opted into captureIp — adding it otherwise would put a raw address on an event they deliberately anonymised. Worth having because it is a free second opinion. PostHog classifies from far more traffic than we see, so disagreement between bot_name and getBotName($raw_user_agent) is a cheap signal for where our patterns are wrong. The GeoIP fix matters too: without $ip, PostHog geolocates whichever edge PoP relayed the event rather than the client, which is approximately right and quietly not authoritative. I had previously dismissed the $virt_* family as unusable "because events arrive server-side from middleware". That was the wrong mechanism — it is unusable because we send the UA under the wrong key, which is fixable in one place. Tests 356 -> 360.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Over 2.5 million sites answer bulk AI crawling with
Disallowin robots.txt. That leaves money on the table, and it only works if the crawler cooperates.The alternative: let them train, and price it.
Why almost nobody can do this correctly
Pricing only works if you can tell training from retrieval, because they have opposite economics:
GPTBot— corpus collection. You get nothing back per fetch.ChatGPT-User— a person just asked about you. Billing that is billing your own distribution channel.agentPolicy()already draws that line. This turns a'charge'decision into the HTTP challenge:That is the differentiator. Gateway pay-per-crawl charges crawls indiscriminately. Charging the half that sends you demand is self-harm — and on our own traffic that half is 60%.
What it does
paymentRequired()emits x402's wire format — a 402 with a base64PAYMENT-REQUIREDheader — andhasPaymentPayload()/paymentPayload()read the client'sPAYMENT-SIGNATUREretry.withSettlement()attaches a facilitator's result.The default
Content-Signalissearch=yes, ai-input=yes, ai-train=paid, deliberately inverting the library's ownai-train=nodefault. The premise is that training is for sale, not forbidden.Scope, deliberately narrow
This emits challenges and reads headers. It settles nothing. Settlement belongs to an x402 facilitator or Stripe's MPP — holding money would drag PCI scope into something meant to drop into middleware. The library never invents an amount, network or asset; all caller-supplied.
Two behaviours worth reviewing:
'meter'returnsnull, not a gate. Metering is accounting; the request is still served whiletrackVisitrecords it.'block'returns 403, not a price. A failed identity check is not a negotiation.Tests: 246 → 310
Includes #22's suite after the rebase, plus a 19-UA corpus pinning
agentIntent(ua) === agentPolicy(req).intent, and non-ASCII in the challenge (btoais Latin-1 only — a naive encoder throws onEntraînement — 訓練 🤖), refusing to emit a challenge with no way to pay, and that retrieval and search stay free under a charge-training policy.Merge order — resolved
Rebased onto
mainnow that #22 (Web Bot Auth,0.14.0) has merged. Thepackage.jsonconflict was the two branches both setting a version; resolved to0.15.0. Mergeable, CI green on Node 20/22/24.🤖 Generated with Claude Code
Two bugs found after the first review, both by composing rather than unit-testing
1.
paymentGatecould not use Web Bot Auth at all.Verification is necessarily async (it fetches the signer's key directory) but
agentPolicyis sync, socombinedVerifier()couldn't be passed to it. In JS that failed silently —agentPolicyread.verdictoff a promise, gotundefined, and never applied the spoofed check. In TS it was a compile error, which is better, but still meant the package's two headline features couldn't be used together.Caught by an integration check: ClaudeBot from a DigitalOcean address returned
402 chargewhen it should have been403 block. Every unit test passed either way, because each layer was correct in isolation.paymentGateis already async, so it now awaits the verifier and passes the resolved result toagentPolicyvia a newverificationoption.agentPolicystays synchronous for callers who only want UA classification.2.
agentIntentandagentPolicydisagreed on every HTTP client.Both exported from the package root; the
toolingpromotion lived only insideagentPolicy, soagentIntent('curl/8.4.0')returned'unknown'while the policy returned'tooling'. Six of six HTTP-client UAs disagreed.Surfaced by the site's
/api/whoamipanel rendering "intent: unknown" beside "reason: coding agent or HTTP client". I first patched only that call site, which left the trap exported for everyone else. NowagentIntentchecksisHttpClientitself andagentPolicyreads from it — one source of truth, pinned by a 19-UA corpus test.