CASSANDRA-21558: Fix client connections surviving nodetool disablebinary - #5009
Open
Runtian wants to merge 1 commit into
Open
CASSANDRA-21558: Fix client connections surviving nodetool disablebinary#5009Runtian wants to merge 1 commit into
Runtian wants to merge 1 commit into
Conversation
A client channel is only registered in Server.ConnectionTracker.allChannels when the STARTUP message is handled, not when the TCP connection is accepted. Server.stop() closes only the channels present in that group at the instant it runs, and DefaultChannelGroup does not reject later additions, so a client that completed its TCP handshake before disablebinary but sent STARTUP after it is never closed by anything. The result is a fully usable CQL session on a node that reports isNativeTransportRunning() == false. In a two datacenter cluster this surfaces as a client outage: after an operator drains a DC by disabling the binary protocol and then removes that DC from a keyspace's replication, the surviving session is pinned to a coordinator whose local DC has RF 0, so every LOCAL_QUORUM query on it fails with UnavailableException while the rest of the cluster is healthy. ConnectionTracker now latches itself closed in closeAll() and closes any channel that registers afterwards. closeAll() sets the flag before sweeping and addConnection() reads it after inserting into allChannels, so either the late registration observes the shutdown or the sweep observes the channel; there is no interleaving where both miss each other. DefaultChannelGroup's own stayClosed flag cannot be used for this because it never resets, which would leave the server unable to bind again after a disablebinary/enablebinary cycle. Server.start() therefore reopens the tracker before binding, which is early enough that clients accepted before isRunning is set are not wrongly rejected. stop() is also made synchronized to match start(), closing a pre-existing race where a stop() in flight could sweep away a bind channel a concurrent start() had just installed, leaving a node that reports the transport as running while accepting nothing. CASSANDRA-21558
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CASSANDRA-21558
Problem
A client channel is only registered in
Server.ConnectionTracker.allChannelswhen the STARTUP message is handled, not when the TCP connection is accepted.Server.stop()closes only the channels present in that group at the instant it runs, andDefaultChannelGroupdoes not reject later additions, so a client that completed its TCP handshake beforedisablebinarybut sent STARTUP after it is never closed by anything.The result is a fully usable CQL session on a node that reports
isNativeTransportRunning() == false. In a two datacenter cluster this surfaces as a client outage: after an operator drains a DC by disabling the binary protocol and then removes that DC from a keyspace's replication, the surviving session is pinned to a coordinator whose local DC has RF 0, so everyLOCAL_QUORUMquery on it fails withUnavailableExceptionwhile the rest of the cluster is healthy.Fix
ConnectionTrackernow latches itself closed incloseAll()and closes any channel that registers afterwards.closeAll()sets the flag before sweeping andaddConnection()reads it after inserting intoallChannels, so either the late registration observes the shutdown or the sweep observes the channel; there is no interleaving where both miss each other.DefaultChannelGroup's ownstayClosedflag cannot be used for this because it never resets, which would leave the server unable to bind again after adisablebinary/enablebinarycycle.Server.start()therefore reopens the tracker before binding, which is early enough that clients accepted beforeisRunningis set are not wrongly rejected.stop()is also madesynchronizedto matchstart(), closing a pre-existing race where astop()in flight could sweep away a bind channel a concurrentstart()had just installed, leaving a node that reports the transport as running while accepting nothing.Testing
New in-JVM dtest
DisableBinaryConnectionRaceTestwith 3 tests:connectionCompletingHandshakeAfterDisableBinaryMustBeClosed— the race itself, with a control asserting a brand new client cannot connect afterdisablebinary.leakedConnectionFailsLocalQuorumAfterLocalDcReplicationRemoved— the two-DC operational consequence, with controls proving the cluster is otherwise healthy.nativeTransportCanBeRestartedAfterDisableBinary—enablebinarystill works, i.e. the latch resets.Also run:
NativeTransportServiceTest(13 tests, covers concurrent start/stop/destroy) andCQLConnectionTest, both passing.