Skip to content

CASSANDRA-21558: Fix client connections surviving nodetool disablebinary - #5009

Open
Runtian wants to merge 1 commit into
apache:cassandra-4.1from
Runtian:CASSANDRA-21558-4.1
Open

CASSANDRA-21558: Fix client connections surviving nodetool disablebinary#5009
Runtian wants to merge 1 commit into
apache:cassandra-4.1from
Runtian:CASSANDRA-21558-4.1

Conversation

@Runtian

@Runtian Runtian commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

CASSANDRA-21558

Problem

A client channel is only registered in Server.ConnectionTracker.allChannels when the STARTUP message is handled, not when the TCP connection is accepted. Server.stop() closes only the channels present in that group at the instant it runs, and DefaultChannelGroup does not reject later additions, so a client that completed its TCP handshake before disablebinary but sent STARTUP after it is never closed by anything.

The result is a fully usable CQL session on a node that reports isNativeTransportRunning() == false. In a two datacenter cluster this surfaces as a client outage: after an operator drains a DC by disabling the binary protocol and then removes that DC from a keyspace's replication, the surviving session is pinned to a coordinator whose local DC has RF 0, so every LOCAL_QUORUM query on it fails with UnavailableException while the rest of the cluster is healthy.

Fix

ConnectionTracker now latches itself closed in closeAll() and closes any channel that registers afterwards. closeAll() sets the flag before sweeping and addConnection() reads it after inserting into allChannels, so either the late registration observes the shutdown or the sweep observes the channel; there is no interleaving where both miss each other.

DefaultChannelGroup's own stayClosed flag cannot be used for this because it never resets, which would leave the server unable to bind again after a disablebinary/enablebinary cycle. Server.start() therefore reopens the tracker before binding, which is early enough that clients accepted before isRunning is set are not wrongly rejected.

stop() is also made synchronized to match start(), closing a pre-existing race where a stop() in flight could sweep away a bind channel a concurrent start() had just installed, leaving a node that reports the transport as running while accepting nothing.

Testing

New in-JVM dtest DisableBinaryConnectionRaceTest with 3 tests:

  • connectionCompletingHandshakeAfterDisableBinaryMustBeClosed — the race itself, with a control asserting a brand new client cannot connect after disablebinary.
  • leakedConnectionFailsLocalQuorumAfterLocalDcReplicationRemoved — the two-DC operational consequence, with controls proving the cluster is otherwise healthy.
  • nativeTransportCanBeRestartedAfterDisableBinaryenablebinary still works, i.e. the latch resets.

Also run: NativeTransportServiceTest (13 tests, covers concurrent start/stop/destroy) and CQLConnectionTest, both passing.

A client channel is only registered in Server.ConnectionTracker.allChannels when
the STARTUP message is handled, not when the TCP connection is accepted.
Server.stop() closes only the channels present in that group at the instant it
runs, and DefaultChannelGroup does not reject later additions, so a client that
completed its TCP handshake before disablebinary but sent STARTUP after it is
never closed by anything.

The result is a fully usable CQL session on a node that reports
isNativeTransportRunning() == false. In a two datacenter cluster this surfaces as
a client outage: after an operator drains a DC by disabling the binary protocol
and then removes that DC from a keyspace's replication, the surviving session is
pinned to a coordinator whose local DC has RF 0, so every LOCAL_QUORUM query on
it fails with UnavailableException while the rest of the cluster is healthy.

ConnectionTracker now latches itself closed in closeAll() and closes any channel
that registers afterwards. closeAll() sets the flag before sweeping and
addConnection() reads it after inserting into allChannels, so either the late
registration observes the shutdown or the sweep observes the channel; there is no
interleaving where both miss each other.

DefaultChannelGroup's own stayClosed flag cannot be used for this because it
never resets, which would leave the server unable to bind again after a
disablebinary/enablebinary cycle. Server.start() therefore reopens the tracker
before binding, which is early enough that clients accepted before isRunning is
set are not wrongly rejected.

stop() is also made synchronized to match start(), closing a pre-existing race
where a stop() in flight could sweep away a bind channel a concurrent start() had
just installed, leaving a node that reports the transport as running while
accepting nothing.

CASSANDRA-21558
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants