Skip to content

Recover peer traffic sooner after path failure - #2642

Open
apkd wants to merge 1 commit into
zerotier:devfrom
apkd:fix/peer-recovery
Open

apkd wants to merge 1 commit into
zerotier:devfrom
apkd:fix/peer-recovery

Conversation

@apkd

@apkd apkd commented Nov 2, 2026

Copy link
Copy Markdown

Note

This pull request was prepared using AI. I've reviewed the patch and tried to keep it clean, but I admit the ZeroTier codebase is outside my expertise.

After a network change, ZeroTier can keep sending to a peer's old address for about four minutes, even while a root relay is reachable. I reproduced this when moving a laptop from a shared Wi-Fi network to a hotspot. This change reduced the reconnection time for my SSH sessions down to ~30-40s.

This change gives ordinary heartbeats time to confirm a connection before sending extra checks. If a check goes unanswered for one heartbeat interval, traffic uses another recently active direct path or an available relay. Retries use HELLO because peers may ignore ECHO. Valid direct traffic makes the path usable again.

Low-bandwidth mode uses the existing recovery behavior. Bonded connections use their own recovery.

Tradeoffs

  • Quiet paths can need extra checks. A tracked ECHO exchange uses 137 bytes over IPv4, including UDP/IP headers: 16 bytes more than an ordinary ECHO. Heartbeats on paths receiving valid traffic keep their usual size.
  • HELLO retries use more bandwidth and processing than ECHO.
  • Lost or ignored replies can send healthy traffic through a relay, increasing latency and relay load until a usable direct path responds.
  • Tracking unanswered probes adds about 512 bytes of memory per peer on typical 64-bit builds.

Tests

One regression checks direct delivery, recovery through a relay after path loss, and return to direct delivery. It fails on unpatched upstream code.

  • Full Linux AddressSanitizer suite: passed.
  • Temporary real-node measurements added no healthy-traffic packets or bytes in either one-way or two-way workloads. Simulated failure recovery took 23–32 seconds, compared with 233–242 seconds upstream. Recovery time depends on traffic and timing.
  • With a healthy peer ignoring ECHO, HELLO retries reduced relayed messages from 270 to 10 out of 600. All messages arrived; control traffic increased by 1,880 bytes over ten minutes.

Saved peer paths can delay traffic for several minutes after a network change. Check quiet paths and use an available relay when replies stop. Reuse ordinary heartbeats and limit faster recovery to normal mode. Retry unanswered checks with HELLO to support peers that ignore ECHO. Skip redundant checks for bonded peers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant