Skip to content

perf: known_routes retention, digest identity, and a working batched reconfirm - #98

Merged
MrAlders0n merged 8 commits into
MeshCore-Beacon:devfrom
MrAlders0n:fix/known-routes-retention
Aug 12, 2026
Merged

perf: known_routes retention, digest identity, and a working batched reconfirm#98
MrAlders0n merged 8 commits into
MeshCore-Beacon:devfrom
MrAlders0n:fix/known-routes-retention

Conversation

@MrAlders0n

@MrAlders0n MrAlders0n commented Aug 7, 2026

Copy link
Copy Markdown
Member

What this PR does

known_routes has no retention — nothing ever deletes from it. The MeshMapper deployment hit 32M rows / 28 GB in 24 days and the hourly ReconfirmRoutes seq-scans the whole table, which had grown to 31 minutes per run — a core pinned for half of every hour, getting worse every day.

Three changes:

  • Retention. Routes age out routes.retention (default 14d) after last_seen; routes seen fewer than min_observations times only get routes.grace (7d) — that noise floor matters because a single flood packet mints a dozen one-off rows that are never observed again. New observations bump last_seen, so routes the mesh still uses never expire. All configurable, documented in config.yaml.example.

  • Route identity is a 16-byte digest now. The unique key was the whole ordered UUID[], so a 10-hop route cost ~380 bytes of index per entry. Migration 024 rebuilds the table keyed on (iata, path_key) where path_key is an md5 of node_ids (Go and SQL compute it identically; there's a test pinning both to the same vector). id stays for the API, just unindexed.

  • Reconfirm is batched, and the ambiguity check actually works. The old check compared 1–3 byte hop prefixes against the 4-byte prefix_4 column — bytea equality needs equal length, so it has never matched a row. It's now length-aware against a pre-aggregated ambiguous-prefix set, and each tick checks the 750K least-recently-confirmed rows instead of everything, so the cost stays flat as the table grows. The retention delete runs in the same task right before it, so the table's two writers never overlap.

Numbers from my prod after deploying:

  • reconfirm: 31 min → under 3 min per hourly tick
  • table: 29 GB → 14 GB, ~16M live rows, and flat instead of +1.2 GB/day
  • first tick after upgrade cleared the 18M-row backlog in 54s

Notes for upgrading

The migration copies every row — it bakes in no retention policy, so your configured (or default) values apply on the first tick after startup. The copy is the startup cost: ~38 minutes at 34M rows on my box, proportionally less on smaller tables. Once it commits there's no going back to the old binary (the old unique index is gone), so take a backup first.

Expect route counts to drop after the first few ticks: expired and genuinely-ambiguous routes get removed for the first time (about 5% of surviving rows here were stale entries the broken check should have been evicting all along).

@MrAlders0n
MrAlders0n requested a review from 446564 as a code owner August 7, 2026 12:37
@MrAlders0n

Copy link
Copy Markdown
Member Author
image

This is MeshMappers instance CPU after deploying the PR

@MrAlders0n
MrAlders0n merged commit 7d759d0 into MeshCore-Beacon:dev Aug 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants