synthetic-traffic: require evidence before paging on an all-failed tick

[AquiGorka]

What broke

On 2026-08-03 19:44Z the engine paged Discord with every action this tick failed (1/1) — platform down or config broken? while it was healthy: deposits and sends completed on the ticks either side. That tick planned exactly ONE action, a deposit to meli-mercosur:BR whose bundle FAILED on-chain, and failures === batch.length && batch.length > 0 has no floor on the denominator, so one transient failure reads identically to a total outage.

Task: theahaco-submodule-0024.

What changed

Alert policy moves into synthetic-traffic/alerts.ts as a pure fold over tick outcomes, unit-tested; main.ts counts the tick and hands the numbers to it.

• Floor. A tick pages on its own only once it ran MIN_ALL_FAILED_ACTIONS = 4 real actions. The planner samples Poisson arrivals per provider, so off-peak ticks routinely hold 1-2 actions; 4 independent actions failing back to back is not plausible transient noise, and it is still well under the 40-action cap so a genuine outage at any hour of the day clears the bar within a tick or two.
• Streak. Thinner all-failed ticks page after ALL_FAILED_TICKS = 3 consecutive ones (~15 min at the deployed 5-min cadence). A sustained outage during a quiet hour still pages promptly; an isolated transient never gets to a second tick, since any non-all-failed tick resets the streak.
• Re-alert throttle. Previously an outage paged every single tick. Now: once when the condition trips, then every ALL_FAILED_REALERT_TICKS = 12 (~hourly) while it holds, mirroring the stuck-tick alert.
• Denominator fix. Aggregator-payment failures incremented failures but were absent from batch.length, so 2 clean batch actions next to 2 failed aggregator payments paged as "2/2" — a second false-positive path. attempted now counts both loops, and only actions the engine actually ran (planned-but-skipped actions, missing actor/receiver, are excluded from both sides rather than padding the denominator).

Unchanged on purpose: intentional seasoning failures (actFail -> injectFailingBundle) still never increment failures — they submit a bundle that FAILS on-chain and return normally, so the alert has never fired on on-purpose errors. The stuck-streak path (STUCK_ALERT_TICKS / REALERT_TICKS) is untouched. The two paths cannot double-page: a tick that throws never reaches the all-failed evaluation, and a tick that completes never counts toward the stuck streak.

Verification

deno task test (new test task; @std/assert added to imports) — 9 tests, all passing:

thin tick with one real failure stays quiet (the 19:44Z page) ... ok
isolated thin failure between healthy ticks never pages ... ok
broad tick pages immediately ... ok
sustained thin failures page on the third consecutive tick ... ok
a healthy tick resets the streak ... ok
a single success in a wide tick is not an outage ... ok
an ongoing outage re-pages hourly, not every tick ... ok
empty ticks are not failures ... ok
aggregator-only failures cannot make a healthy batch read as 100% ... ok
ok | 9 passed | 0 failed

The two cases that matter, end to end:

• The 19:44Z tick — attempted=1, failures=1 -> streak 1, below the floor, below the streak requirement -> silent. The next tick's deposit succeeds -> streak resets. No page, engine keeps running.
• A real outage — platform down, tick runs 7 actions, all throw -> 7 >= 4 -> pages on the first tick, every action this tick failed (7/7). If the outage starts during a quiet hour with 1-2 action ticks, the third consecutive all-failed tick pages every action failed for 3 consecutive ticks (latest 2/2) about 15 minutes in, then hourly until it clears.

deno task check, deno lint (clean for the touched files; the pre-existing no-unused-vars in aggregators.ts is untouched) and deno fmt --check all pass.

Notes

• The streak lives in the process, not the state file, so a restart clears it. A crash-looping engine is what the stuck-tick alert covers, and SYNTRAF_ONCE=true smoke runs keep the broad-tick page while never accumulating a streak.
• No env vars added; thresholds are constants in alerts.ts.
• Redeploy is manual and yours: fly deploy -c synthetic-traffic/fly.testnet.toml --dockerfile synthetic-traffic/Dockerfile from the repo root. Nothing auto-deploys on merge.