Repository navigation
tailcat: test netstack Close hang, shorten TCP TIME-WAIT to 5s - #186
Merged
Merged
Conversation
This also bumps gVisor and wireguard-go. The newer tailscale.com adds netstack.Impl.SetTransportProtocolOption, which an upcoming change uses to set TCP options without reaching into netstack's unexported gVisor stack. Updates #151 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: Iaac79ae8a968ec94d537cc7211e7e4899f3fe0cc
Before tailscale/tailscale#21668, netstack.Impl.Close blocked forever once more TCP endpoints needed a reset than its 512-packet outbound queue held, leaking the WireGuard engine, the DERP connection and their goroutines. Endpoints stay in TIME-WAIT for 60 seconds, so a few hundred recent short connections were enough. go.mod already has the fix; add a test so it stays fixed. The test hangs in Close against the pre-fix tailscale.com (a0e471a35b8f) and passes in about 2s now. Updates #151 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: I55d3c83555abecf45250de51da935d8f726fb1fe
Client.Close held startMu across locoBackend.Close, so a slow or stuck netstack shutdown blocked every concurrent use of the Client that needs startMu, not just Close. locoBackend.Close is already idempotent via its sync.Once, so only the read of c.lb needs the lock. Updates #151 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: Iaa27e7c50497f633bf59ea5802073665f5d4e9d8
gVisor keeps closed TCP endpoints in TIME-WAIT for 60 seconds. Each one costs memory, and while one exists, gVisor's TCP forwarder drops a new SYN on the same 4-tuple (gvisor/gvisor#15013), so a dial that picks a recently used ephemeral port stalls until the endpoint expires. The issue reporter measured stalls on 2.4% of dials at 20 dials/s and 36% at 300/s, and the new Close test hits about 5 such stalls in 600 dials. Each tailcat netstack carries one WireGuard tunnel to one peer, so delayed duplicate segments long after a close, which the 60 second TIME-WAIT guards against, are very unlikely. Five seconds still answers a few retransmissions of the peer's FIN if our last ACK is lost, and cuts the stall rate and TIME-WAIT buildup by 12x. Use the new netstack.Impl.SetTransportProtocolOption rather than the tcpipStackOf reflection. Fixes #151 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: Ib806d46bd0962c07ef7f494dbb287bb1dd659904
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #151
The Close hang itself was already fixed on main by #161, which bumped
tailscale.com past tailscale/tailscale#21668. This PR adds a
regression test for it and two related fixes from the issue.
Commits:
go.mod: bump tailscale.com to the latest. This also bumps gVisor and
wireguard-go, and brings the new exported
netstack.Impl.SetTransportProtocolOption.
Add TestCloseWithManyTimeWaitEndpoints. It makes 600 short
connections, waits for all 600 server endpoints to reach TIME-WAIT
(more than netstack's 512-packet outbound queue), and checks that
Server.Close and Client.Close both return. Against the tailscale.com
from before go.mod: include Tailscale netstack close fix #161 (a0e471a35b8f), Close hangs and the test fails. It
passes in about 2s now.
Client.Close no longer holds startMu while closing the backend, so a
slow netstack shutdown can't block other uses of the Client.
locoBackend.Close is already idempotent via its sync.Once.
Shorten the netstack TCP TIME-WAIT from gVisor's 60s default to 5s,
on both Server and Client. While an endpoint is in TIME-WAIT, gVisor's
TCP forwarder drops a new SYN on the same 4-tuple
(SYN in TIME_WAIT is dropped when the stack accepts through
tcp.Forwardergvisor/gvisor#15013), so dials that reuse a recent ephemeral portstall. The new test sees about 5 such stalls in 600 dials with the
60s default. Each tailcat netstack carries one WireGuard tunnel to
one peer, so the delayed duplicate segments TIME-WAIT guards against
are very unlikely, and 5s still covers a few retransmissions of the
peer's FIN. This cuts both the stall rate and the TIME-WAIT buildup
by 12x.
Not done here, from the issue's list:
Closing the rest of the backend if netstack Close still hangs. With
the upstream fix, this would only guard against some future hang,
and closing the engine under a still-running netstack Close is
untested.
An exported abortive (linger 0) close. Shorter TIME-WAIT makes the
SYN drops much rarer, and an exported API would depend on reflection
into gonet internals. Worth revisiting if stalls are still reported.