Skip to content

CASSANDRA-21575: Keep message ids unsigned so they do not inflate every message header - #5023

Open
SEPURI-SAI-KRISHNA wants to merge 1 commit into
apache:trunkfrom
SEPURI-SAI-KRISHNA:sai-krishna/CASSANDRA-21575/trunk
Open

CASSANDRA-21575: Keep message ids unsigned so they do not inflate every message header#5023
SEPURI-SAI-KRISHNA wants to merge 1 commit into
apache:trunkfrom
SEPURI-SAI-KRISHNA:sai-krishna/CASSANDRA-21575/trunk

Conversation

@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor

CASSANDRA-21575

Problem

Message.nextId() draws from a shared AtomicInteger and widens the result to a long:

private static final AtomicInteger nextId = new AtomicInteger(0);

private static long nextId()
{
    long id;
    do
    {
        id = nextId.incrementAndGet();
    }
    while (id == NO_ID);

    return id;
}

Once the counter passes Integer.MAX_VALUE it wraps to Integer.MIN_VALUE, and the negative int
sign-extends into a negative long. Message.Serializer writes the id as an unsigned vint, and a
negative long has all its high bits set, so it always encodes at the maximum width:

id wire bytes
1 1
2147483647 5
-2147483648 (first wrap) 9
-1 9

That is an extra 4 bytes on the header of every internode message, sustained for the 2³¹ messages
until the counter cycles back through positive, and again on every subsequent wrap. A node sending
50k messages/sec reaches the first wrap in about 12 hours, so this is steady state for any
long-running cluster, not an edge case.

Fix

Mask the counter so ids stay in [0, 2^32). Same number of distinct values, always non-negative,
encoding bounded at 5 bytes. Ids are opaque correlation keys — used only for matching responses to
callbacks in RequestCallbacks — so nothing depends on their sign, and the wire format itself is
unchanged.

Why ForwardingInfo is in the same commit

ForwardingInfo is the FORWARD_TO parameter used for inter-DC write forwarding. It writes ids with
the 64-bit form and sizes them the same way, but reads them back with the 32-bit form:

out.writeUnsignedVInt(ids[i]);              // serialize
size += computeUnsignedVIntSize(ids[i]);    // serializedSize
ids[i] = in.readUnsignedVInt32();           // deserialize  <- mismatch

This is the only 32-bit read of a message id in the tree; everything else uses
readUnsignedVInt/getUnsignedVInt. It is harmless today only by accident — ids always fit in int
because they come from an int counter, so the checkedCast inside readUnsignedVInt32 succeeds
even for sign-extended negative values.

It is therefore a blocker for the fix, not a separate cleanup. Once ids span the full unsigned 32-bit
range this read throws and inter-DC forwarding breaks:

id=2147483648   ForwardingInfo.deserialize: VIntOutOfRangeException
id=4294967295   ForwardingInfo.deserialize: VIntOutOfRangeException

Tests

The existing ForwardingInfoTest only ever used ids 44..49, which is why the width mismatch was
never exercised.

  • ForwardingInfoTest.testLargeMessageIdsRoundTrip — round-trips ids across the whole unsigned
    32-bit range including Integer.MAX_VALUE + 1 and 0xFFFFFFFF, for every supported messaging
    version, asserting serializedSize matches the bytes written. Verified to fail without the
    ForwardingInfo change: VIntOutOfRangeException: 2147483648 is out of range for a 32-bit integer.
  • MessageTest.testIdsRemainUnsignedAcrossCounterWrap — ids stay non-negative, stay within 32
    unsigned bits, and encode within 5 bytes across the wrap boundary.
  • MessageTest.testLargeIdRoundTrips — message headers carrying ids from the upper half of the
    range round-trip for every supported messaging version.

Green locally: ant jar, ant checkstyle checkstyle-test (2907 files), and MessageTest (11),
ForwardingInfoTest (2), MessageSerializationPropertyTest (2), MessageDeliveryTest (4).

Branches

Both defects are present from 4.0 onwards — on 4.0/4.1 the narrowing is spelled
Ints.checkedCast(in.readUnsignedVInt()). This PR targets trunk; happy to prepare backports for
whichever branches you want it on.

Prepared with AI assistance (Claude Opus 5); the commit carries an Assisted-by: trailer.

Message.nextId() draws from a shared AtomicInteger and widens the result to a long. Once the counter
passes Integer.MAX_VALUE it wraps to Integer.MIN_VALUE and the negative int sign-extends, so every id
for the next 2^31 messages is a negative long. Message.Serializer writes ids as unsigned vints, and a
negative long has all its high bits set, so it always encodes at the maximum width of 9 bytes rather
than at most 5 - an extra 4 bytes on the header of every internode message.

Mask the counter so ids stay in [0, 2^32). That covers the same number of distinct values, keeps them
non-negative, and bounds the encoding at 5 bytes. Ids are opaque correlation keys, used only for
matching responses to callbacks, so nothing depends on their sign and the wire format is unchanged.

ForwardingInfo, the FORWARD_TO parameter used for inter-DC write forwarding, writes ids with the
64-bit form and sizes them the same way, but reads them back with the 32-bit form. That is the only
32-bit read of a message id in the tree, and it is harmless today only because ids happen to fit in
an int. It has to be fixed in the same commit: once ids span the full unsigned 32-bit range it would
throw VIntOutOfRangeException and break inter-DC forwarding.

The existing ForwardingInfoTest only ever used ids 44-49, which is why the width mismatch was never
exercised.

patch by Sepuri Sai Krishna; reviewed by TBD for CASSANDRA-21575

Assisted-by: Claude Code:claude-opus-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant