Close the intent gap and catch distributed scrapers (0.17.0)

[Gdewilde]

Both findings came out of a check on live openbankingtracker.com traffic, not from review.

PerplexityBot was in no intent bucket

Perplexity-User was in the retrieval list. The crawler itself was in none of the three, so it classified as unknown — which means it landed in no generated firewall rule at all: not protected by the retrieval/search bypass, not bounded by the training rate limit. 42 requests a day on that one site.

It goes to search. The Bot suffix reads like corpus collection, but Perplexity documents it as the crawler behind their search results and states it does not feed foundation-model training. Blocking it costs citations — the same shape of loss as blocking Googlebot.

Every link unfurler was unclassified

facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, TelegramBot, WhatsApp, redditbot. The protective bypass keys on retrieval|search, so none of them were covered. The first time anyone promotes a deny rule, link previews break silently everywhere the site gets shared.

They get a new preview intent rather than joining retrieval. Retrieval is the demand signal this library sells; a Slack unfurl is not an assistant answering someone's question, and counting it as one inflates the single number the split exists to measure.

PREVIEW is tested after SEARCH so Applebot stays a search crawler — Apple uses one token for both. There's a test pinning that ordering.

The recommender had no rule for distributed abuse

Rule 3 asks "is any single address abusive?" and a rotating proxy pool is built precisely so the answer is no:

34 addresses, 11 countries, one user agent each, none above 100 requests a day — every one invisible to a per-address threshold, while collectively sweeping the site and inflating its GA4 user count.

Volume cannot separate that from real readers, so the new rule keys on rate, via an optional spanSeconds on TrafficObservation. The address that made the case fetched 16 distinct pages in one second.

It is safe to key on rate only because the condition excludes static assets — the WAF counts every asset request and the middleware producing the observations does not, so a naive limit on Mozilla would throttle a real visitor on their first page view. That exclusion is asserted by a test, and the test was mutation-checked (removing the exclusions fails it).

Unfurlers get their own bypass rule at medium risk rather than riding the low one: a bypass skips managed rulesets too, and these tokens are unverifiable and heavily forged. Labelling that low alongside vendors who publish IP ranges would understate what it hands out. firewallScript now reorders every bypass rule to the top instead of assuming there is one.

Deliberately not fixed

SEO crawlers (AhrefsBot, SemrushBot, Screaming Frog) and scraping tools (Firecrawl, Scrapy) still classify as unknown. They are not AI agents, blocking them is a legitimate choice, and inventing a bucket to make the coverage number look complete would be worse than the gap.

Verified against the traffic that prompted it

Same 24h window, before and after — 2 rules become 3:

1. Allow retrieval and search agents   [bypass, low]
   1,179 requests across 6 vendors (ChatGPT, Perplexity, Apple, DuckDuckGo, Bing, Claude)

2. Burst limit page navigations        [log, medium]     <- new
   16 addresses across 11 countries, 383 requests, none individually above the
   abuse threshold; 4 of 7 measurable slices exceeded 30 requests/60s,
   peaking at 1,860/min

3. Rate limit training crawlers        [log, medium]
   10,107 training requests from 5 vendors

Perplexity moved out of unknown and into the protective bypass.

Notes

• Minor, not patch: AgentIntent gains a union member, so an exhaustive switch over it now fails to compile.
• firewall.js budget 3200 → 4300 gz, raised deliberately rather than with --update. It never runs in middleware, so size buys correctness cheaply there. Root entry unchanged at 94% of its budget.
• 388 tests pass (28 new), typecheck clean, size budget green.

🤖 Generated with Claude Code