CASSANDRA-21575: Keep message ids unsigned so they do not inflate every message header - #5023
Open
SEPURI-SAI-KRISHNA wants to merge 1 commit into
Open
Conversation
Message.nextId() draws from a shared AtomicInteger and widens the result to a long. Once the counter passes Integer.MAX_VALUE it wraps to Integer.MIN_VALUE and the negative int sign-extends, so every id for the next 2^31 messages is a negative long. Message.Serializer writes ids as unsigned vints, and a negative long has all its high bits set, so it always encodes at the maximum width of 9 bytes rather than at most 5 - an extra 4 bytes on the header of every internode message. Mask the counter so ids stay in [0, 2^32). That covers the same number of distinct values, keeps them non-negative, and bounds the encoding at 5 bytes. Ids are opaque correlation keys, used only for matching responses to callbacks, so nothing depends on their sign and the wire format is unchanged. ForwardingInfo, the FORWARD_TO parameter used for inter-DC write forwarding, writes ids with the 64-bit form and sizes them the same way, but reads them back with the 32-bit form. That is the only 32-bit read of a message id in the tree, and it is harmless today only because ids happen to fit in an int. It has to be fixed in the same commit: once ids span the full unsigned 32-bit range it would throw VIntOutOfRangeException and break inter-DC forwarding. The existing ForwardingInfoTest only ever used ids 44-49, which is why the width mismatch was never exercised. patch by Sepuri Sai Krishna; reviewed by TBD for CASSANDRA-21575 Assisted-by: Claude Code:claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CASSANDRA-21575
Problem
Message.nextId()draws from a sharedAtomicIntegerand widens the result to along:Once the counter passes
Integer.MAX_VALUEit wraps toInteger.MIN_VALUE, and the negativeintsign-extends into a negative
long.Message.Serializerwrites the id as an unsigned vint, and anegative long has all its high bits set, so it always encodes at the maximum width:
12147483647-2147483648(first wrap)-1That is an extra 4 bytes on the header of every internode message, sustained for the 2³¹ messages
until the counter cycles back through positive, and again on every subsequent wrap. A node sending
50k messages/sec reaches the first wrap in about 12 hours, so this is steady state for any
long-running cluster, not an edge case.
Fix
Mask the counter so ids stay in
[0, 2^32). Same number of distinct values, always non-negative,encoding bounded at 5 bytes. Ids are opaque correlation keys — used only for matching responses to
callbacks in
RequestCallbacks— so nothing depends on their sign, and the wire format itself isunchanged.
Why
ForwardingInfois in the same commitForwardingInfois theFORWARD_TOparameter used for inter-DC write forwarding. It writes ids withthe 64-bit form and sizes them the same way, but reads them back with the 32-bit form:
This is the only 32-bit read of a message id in the tree; everything else uses
readUnsignedVInt/getUnsignedVInt. It is harmless today only by accident — ids always fit inintbecause they come from an
intcounter, so thecheckedCastinsidereadUnsignedVInt32succeedseven for sign-extended negative values.
It is therefore a blocker for the fix, not a separate cleanup. Once ids span the full unsigned 32-bit
range this read throws and inter-DC forwarding breaks:
Tests
The existing
ForwardingInfoTestonly ever used ids44..49, which is why the width mismatch wasnever exercised.
ForwardingInfoTest.testLargeMessageIdsRoundTrip— round-trips ids across the whole unsigned32-bit range including
Integer.MAX_VALUE + 1and0xFFFFFFFF, for every supported messagingversion, asserting
serializedSizematches the bytes written. Verified to fail without theForwardingInfochange:VIntOutOfRangeException: 2147483648 is out of range for a 32-bit integer.MessageTest.testIdsRemainUnsignedAcrossCounterWrap— ids stay non-negative, stay within 32unsigned bits, and encode within 5 bytes across the wrap boundary.
MessageTest.testLargeIdRoundTrips— message headers carrying ids from the upper half of therange round-trip for every supported messaging version.
Green locally:
ant jar,ant checkstyle checkstyle-test(2907 files), andMessageTest(11),ForwardingInfoTest(2),MessageSerializationPropertyTest(2),MessageDeliveryTest(4).Branches
Both defects are present from 4.0 onwards — on 4.0/4.1 the narrowing is spelled
Ints.checkedCast(in.readUnsignedVInt()). This PR targets trunk; happy to prepare backports forwhichever branches you want it on.
Prepared with AI assistance (Claude Opus 5); the commit carries an
Assisted-by:trailer.