Skip to content

tailcat: test netstack Close hang, shorten TCP TIME-WAIT to 5s - #186

Merged
bradfitz merged 4 commits into
mainfrom
bradfitz/netstack-close-hang
Nov 18, 2026
Merged

bradfitz merged 4 commits into
mainfrom
bradfitz/netstack-close-hang

Conversation

@bradfitz

Copy link
Copy Markdown
Member
  • go.mod: bump tailscale.com
  • tailcat: test Close with more TIME-WAIT endpoints than netstack queues
  • tailcat: don't hold Client.startMu while closing the backend
  • tailcat: shorten netstack TCP TIME-WAIT from 60s to 5s

Fixes #151

The Close hang itself was already fixed on main by #161, which bumped
tailscale.com past tailscale/tailscale#21668. This PR adds a
regression test for it and two related fixes from the issue.

Commits:

  • go.mod: bump tailscale.com to the latest. This also bumps gVisor and
    wireguard-go, and brings the new exported
    netstack.Impl.SetTransportProtocolOption.

  • Add TestCloseWithManyTimeWaitEndpoints. It makes 600 short
    connections, waits for all 600 server endpoints to reach TIME-WAIT
    (more than netstack's 512-packet outbound queue), and checks that
    Server.Close and Client.Close both return. Against the tailscale.com
    from before go.mod: include Tailscale netstack close fix #161 (a0e471a35b8f), Close hangs and the test fails. It
    passes in about 2s now.

  • Client.Close no longer holds startMu while closing the backend, so a
    slow netstack shutdown can't block other uses of the Client.
    locoBackend.Close is already idempotent via its sync.Once.

  • Shorten the netstack TCP TIME-WAIT from gVisor's 60s default to 5s,
    on both Server and Client. While an endpoint is in TIME-WAIT, gVisor's
    TCP forwarder drops a new SYN on the same 4-tuple
    (SYN in TIME_WAIT is dropped when the stack accepts through tcp.Forwarder gvisor/gvisor#15013), so dials that reuse a recent ephemeral port
    stall. The new test sees about 5 such stalls in 600 dials with the
    60s default. Each tailcat netstack carries one WireGuard tunnel to
    one peer, so the delayed duplicate segments TIME-WAIT guards against
    are very unlikely, and 5s still covers a few retransmissions of the
    peer's FIN. This cuts both the stall rate and the TIME-WAIT buildup
    by 12x.

Not done here, from the issue's list:

  • Closing the rest of the backend if netstack Close still hangs. With
    the upstream fix, this would only guard against some future hang,
    and closing the engine under a still-running netstack Close is
    untested.

  • An exported abortive (linger 0) close. Shorter TIME-WAIT makes the
    SYN drops much rarer, and an exported API would depend on reflection
    into gonet internals. Worth revisiting if stalls are still reported.

This also bumps gVisor and wireguard-go. The newer tailscale.com
adds netstack.Impl.SetTransportProtocolOption, which an upcoming
change uses to set TCP options without reaching into netstack's
unexported gVisor stack.

Updates #151

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iaac79ae8a968ec94d537cc7211e7e4899f3fe0cc
Before tailscale/tailscale#21668, netstack.Impl.Close blocked forever
once more TCP endpoints needed a reset than its 512-packet outbound
queue held, leaking the WireGuard engine, the DERP connection and
their goroutines. Endpoints stay in TIME-WAIT for 60 seconds, so a
few hundred recent short connections were enough. go.mod already has
the fix; add a test so it stays fixed.

The test hangs in Close against the pre-fix tailscale.com
(a0e471a35b8f) and passes in about 2s now.

Updates #151

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I55d3c83555abecf45250de51da935d8f726fb1fe
Client.Close held startMu across locoBackend.Close, so a slow or
stuck netstack shutdown blocked every concurrent use of the Client
that needs startMu, not just Close. locoBackend.Close is already
idempotent via its sync.Once, so only the read of c.lb needs the
lock.

Updates #151

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Iaa27e7c50497f633bf59ea5802073665f5d4e9d8
gVisor keeps closed TCP endpoints in TIME-WAIT for 60 seconds. Each
one costs memory, and while one exists, gVisor's TCP forwarder drops
a new SYN on the same 4-tuple (gvisor/gvisor#15013), so a dial that
picks a recently used ephemeral port stalls until the endpoint
expires. The issue reporter measured stalls on 2.4% of dials at 20
dials/s and 36% at 300/s, and the new Close test hits about 5 such
stalls in 600 dials.

Each tailcat netstack carries one WireGuard tunnel to one peer, so
delayed duplicate segments long after a close, which the 60 second
TIME-WAIT guards against, are very unlikely. Five seconds still
answers a few retransmissions of the peer's FIN if our last ACK is
lost, and cuts the stall rate and TIME-WAIT buildup by 12x.

Use the new netstack.Impl.SetTransportProtocolOption rather than
the tcpipStackOf reflection.

Fixes #151

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ib806d46bd0962c07ef7f494dbb287bb1dd659904
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Close can hang and leak the WireGuard engine and DERP connection: bump tailscale.com past the netstack close fix, and close the backend regardless

1 participant