Skip to content

fix(windows): allow multiple in-flight tap writes to stop routing stalls (#1428) - #2635

Open
NullP0interEx wants to merge 1 commit into
zerotier:devfrom
NullP0interEx:fix/windows-tap-concurrent-writes
Open

NullP0interEx wants to merge 1 commit into
zerotier:devfrom
NullP0interEx:fix/windows-tap-concurrent-writes

Conversation

@NullP0interEx

Copy link
Copy Markdown

Fixes #1428.

Problem

On Windows hosts that forward traffic between a ZeroTier network and a LAN, all inbound ZeroTier traffic of that host stalls for 1 to 3 s every time something reaches for a LAN address that does not answer ARP. With enough such traffic the stalls overlap and the ZeroTier interface stops passing traffic completely. #1428 describes this in detail.

Root cause

The analysis in #1428 suspected the driver's NBL handling. The driver does play a part, but the stall comes from the combination of driver and service:

  1. Driver (windows/TapDriver6/rxpath.c, IRP_MJ_WRITE): every frame the service writes is wrapped in an NBL that points straight at the IRP's MDL. The driver calls IoMarkIrpPending and NdisMIndicateReceiveNetBufferLists(..., ReceiveFlags 0). The IRP is only completed when NDIS hands the NBL back through MiniportReturnNetBufferLists. Without NDIS_RECEIVE_FLAGS_RESOURCES the stack may keep the NBL as long as it needs.
  2. Windows stack: when a forwarded IPv4 packet goes to a directly connected address with no ARP entry, Windows queues the packet while it resolves the address: 1 s per attempt, up to 3 attempts. The NBL stays with the stack for that whole time.
  3. Service (osdep/WindowsEthernetTap.cpp, threadMain): the tap I/O thread has exactly one OVERLAPPED for writes and one writeInProgress flag. While that one write is pending, the loop continues and nothing else from _injectPending gets written.

Together: one forwarded frame to a dead LAN IP holds the single write slot for up to 3 s. Every other inbound frame, including unrelated traffic to the host itself, waits behind it in _injectPending. The encrypted packets arrive at the physical NIC on time, but the ZeroTier adapter delivers them seconds later. In the captures below the gap is up to 7 s.

Fix

Allow up to ZT_WINDOWS_TAP_WRITE_SLOTS (16) writes to the tap device at once:

  • Each slot has its own OVERLAPPED, event and buffer. The frame is copied from _injectPending into the slot's buffer before WriteFile, because the driver keeps referencing that memory (via the MDL) until the IRP completes.
  • The wait set is built from the hEvents of the slots in flight, plus the inject semaphore and the read event as before.
  • Completed slots are freed and reused. Free slots are filled in queue order.
  • Ordering is preserved. The driver calls NdisMIndicateReceiveNetBufferLists synchronously inside the IRP_MJ_WRITE dispatch routine, so frames reach the stack in the order they were written. Only the IRP completion is deferred.
  • On shutdown or reopen, the thread waits for outstanding writes with GetOverlappedResult(..., TRUE) before the buffers and events go away. The driver marks these IRPs as not cancellable, so CancelIo does not complete them.

A frame that the stack holds now occupies only its own slot, and everything else keeps flowing.

The change is service-only. The driver and its signature are untouched, so it can ship in a normal release.

Remaining limitation

If more than 16 frames to unresolvable addresses arrive within about 3 s, all slots are taken and the old behaviour returns for the rest of that window. A complete fix in the driver would be to set NDIS_RECEIVE_FLAGS_RESOURCES, or to copy into a driver-owned buffer and complete the IRP right away. That needs a re-signed driver. This PR removes the problem in practice and is independent of a driver change. The slot count is a single define if a larger value is preferred.

Test

Setup: Windows 10 Pro 22H2 (19045) workstation acting as the LAN router of a ZeroTier network. IPEnableRouter=1, RemoteAccess service, no NAT, managed route 192.168.3.0/24 via this host, LAN router with a static route back. ZeroTier 1.16.2, tap driver 3.0.0.1.

Build: tag 1.16.2 plus this patch, built with VS Build Tools 2022 (v143) and Rust stable, msbuild windows\ZeroTierOne.sln /p:Configuration=Release /p:Platform=x64 /t:ZeroTierOne. Only zerotier-one_x64.exe was replaced, the driver was left as installed.

Measurement: 120 ICMP echo requests at 0.5 s intervals from a macOS peer to the router's ZeroTier IP. At the time, the router had several ARP entries in Unreachable state on the LAN side. The runs alternated between the official binary and the patched one, restarting the service each time:

Run Binary RTT > 100 ms RTT > 500 ms max RTT
1 official 1.16.2 85 / 120 83 / 120 7383 ms
2 patched 0 / 120 0 / 120 50 ms
3 official 1.16.2 72 / 119 70 / 119 6971 ms
4 patched 1 / 120 0 / 120 486 ms

With the official binary the RTTs form a falling sawtooth (7383, 6882, 6377, 5875 ...): the replies are held and then released together. With the patch the RTTs stay at the path's normal 20 to 30 ms. Runs 1 and 3 show that a service restart alone does not change anything.

Before this, a pktmon capture on the router showed the encrypted reply reaching the physical NIC 1 ms after the request, while the decrypted frame was only indicated on the ZeroTier adapter 2.9 s later.

The patched binary is now also deployed on two Windows Server 2019 VMs (not routing). I will report back here if anything shows up in longer operation.

The tap driver keeps every IRP_MJ_WRITE pending until NDIS returns the
indicated NET_BUFFER_LIST (rxpath.c, ReceiveFlags 0). WindowsEthernetTap
only ever had one overlapped write outstanding, so a single frame that the
Windows stack holds on to blocked every following inbound frame.

The stack holds a frame for up to ~3 s when it forwards it to a directly
connected address it has to resolve via ARP first (1 s per attempt, up to
3 attempts). On a Windows host that routes between ZeroTier and a LAN this
stalls all ZeroTier traffic of that host for 1-3 s whenever something
reaches for an IP that does not answer, and repeated requests stack up
into a complete outage.

Keep up to ZT_WINDOWS_TAP_WRITE_SLOTS (16) writes in flight, each with its
own OVERLAPPED and buffer, so a held frame only occupies its own slot.
Frames are still submitted in queue order; the driver indicates each one
synchronously inside IRP_MJ_WRITE, only the completion is deferred. On
shutdown wait for outstanding writes before the buffers go away, as the
driver marks these IRPs non-cancellable.

Fixes zerotier#1428
@CLAassistant

CLAassistant commented Oct 28, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Intermittent Network Latency/Outages on Windows Devices that Route (Forward) Traffic

2 participants