Skip to content

fix(testnet): recover safely from source reorgs - #151

Merged
sprites0 merged 11 commits into
node-builderfrom
fix/pseudo-peer-recover-evicted-blocks
Nov 10, 2026
Merged

sprites0 merged 11 commits into
node-builderfrom
fix/pseudo-peer-recover-evicted-blocks

Conversation

@sprites0

@sprites0 sprites0 commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Enable Hyperliquid testnet nodes to recover from source reorgs within a 256-block window while preserving immediate finality on mainnet.

Changes

  • keep a 256-block reorg window on testnet (998)
  • continue finalizing every selected head on mainnet (999)
  • refresh stale cached blocks and maintain consistent hash indexes
  • reject stale queued responses during source reorgs
  • preserve finalization state during the reorg-depth transition
  • read sync blocks and receipts from one consistent database snapshot
  • resolve prefetched parent hashes and recover evicted blocks by hash

Safety

Reorg support is selected from the chain ID. Mainnet and unknown chains use immediate finality; only testnet receives the 256-block reorg window.

Testing

  • consensus finalization-policy tests
  • block-import service tests
  • source reorg tests
  • appended reorg coverage

@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 4 times, most recently from 90b1f85 to 18be869 Compare September 1, 2026 05:35
@sprites0 sprites0 changed the title fix(pseudo-peer): recover evicted blocks on hash lookup fix(testnet): recover safely from source reorgs Sep 1, 2026
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 4 times, most recently from b43b8ef to 4e02d2b Compare September 3, 2026 15:21
@sprites0
sprites0 marked this pull request as ready for review September 3, 2026 15:27
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 4 times, most recently from 5b1676c to 6a8b71c Compare September 4, 2026 07:55
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 3 times, most recently from ec9a551 to 0916499 Compare November 10, 2026 08:13
sprites0 and others added 11 commits November 10, 2026 04:54
Keep enough canonical history available for deep reorgs, avoid prematurely finalizing imported heads, and serve blocks with receipts from one consistent database snapshot.
Reth can begin a falling header request at a child's parent, but the reorg cache refactor stopped recording that parent mapping and caused repeated empty responses.

Index each fetched block's parent at the preceding height and cover the lookup with a focused regression test.
get_by_hash only served from the `blocks` LRU, which holds
BLOCK_CACHE_LIMIT (100k) entries. The hash index holds ten times that
(HASH_INDEX_LIMIT, 1M) and falls back to the node's HeaderNumbers table
on a miss, so a hash stays resolvable long after its body has been
evicted. Every peer request for such a hash failed with "Block not
cached for hash", even though the block was still reachable by number.

Resolve the hash to a height and refetch instead: serve from
gap_blocks when the height is one of the hardcoded testnet gaps,
otherwise call the source under a 30s timeout so a stalled fetch cannot
wedge the request loop.

Recovery is only allowed to install a block that is provably the one
asked for. The source is a by-number API, so it can legitimately return
a different block than requested after a reorg; three checks reject
that. The returned block must be at the requested height and carry the
requested hash, and — under the canonical_hashes write lock — must not
contradict a hash already recorded for that height. The last check is
what closes the race: an in-flight recovery that finishes after
cache_block has refreshed the height loses, rather than overwriting the
newer canonical block with the one it happened to fetch.

The staleness check in BlockImport::poll was using get_by_hash's error
as its signal, which is wrong in both directions now. It would trigger
a source fetch on what is only a liveness test, and a hash that reorged
away but still resolves through the DB index would come back Ok and be
treated as live. Add is_cached_canonical_hash, which answers exactly
that question with no I/O: the body is cached and its hash is still
canonical at its height.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Z9cmqnwb9tL9kxkGM5Ccb
canonical_hashes gained an entry for every cached block and was only
trimmed by invalidate_above, which mainnet effectively never reaches,
so a node syncing from genesis kept one entry per height. Evicting a
block from the LRU now drops its canonical entry as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch from 0916499 to 2bfab61 Compare November 10, 2026 09:58
@sprites0
sprites0 merged commit 7a03cea into node-builder Nov 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant