Skip to content

fix: handle channel removal gracefully in processCompletedReceives - #3580

Open
VimalN2005 wants to merge 1 commit into
AutoMQ:mainfrom
VimalN2005:fix/channel-removal-completed-receives-2094
Open

fix: handle channel removal gracefully in processCompletedReceives#3580
VimalN2005 wants to merge 1 commit into
AutoMQ:mainfrom
VimalN2005:fix/channel-removal-completed-receives-2094

Conversation

@VimalN2005

Copy link
Copy Markdown

Fixes #2094

Problem

When a client disconnects abruptly, times out, or closes during request processing:

  1. The channel is closed and removed from Selector.
  2. Any completed receives buffered during the same poll loop still get processed in processCompletedReceives.
  3. In SocketServer.scala, openOrClosingChannel(receive.source) returns None, which previously threw IllegalStateException(s"Channel ${receive.source} removed from selector before processing completed receive").
  4. This exception was caught and logged as a loud, severe ERROR stack trace by processChannelException, flooding broker logs in production.
  5. More importantly, receive.close() was never invoked on the orphaned receive, causing any allocated buffer in MemoryPool to leak until GC.

Solution

  1. Graceful Handling in SocketServer.scala:
    • In processCompletedReceives(), when openOrClosingChannel returns None, log at debug level instead of throwing IllegalStateException.
    • Explicitly invoke CoreUtils.swallow(receive.close(), this, Level.WARN) to safely release receive.payload buffer back to MemoryPool.
    • Clean up any associated channelContexts via removeChannelContext(receive.source).
  2. Unit Test in SocketServerTest.scala:
    • Added testCompletedReceiveWithRemovedChannel() verifying that an orphaned completed receive whose channel was removed from selector is processed cleanly without throwing an IllegalStateException or recording uncaught exceptions, keeping the processor healthy.

Verification

  • Ran unit test locally:
    ./gradlew :core:test --tests "kafka.network.SocketServerTest.testCompletedReceiveWithRemovedChannel" -x checkstyleTest -x checkstyleMain -x spotbugsMain -x spotbugsTest

…ceives (AutoMQ#2094)

- Downgrade IllegalStateException to debug logging when channel is removed before processing completed receive
- Safely close NetworkReceive to release buffers back to MemoryPool
- Clean up channelContexts for disconnected connections
- Add unit test testCompletedReceiveWithRemovedChannel
@CLAassistant

CLAassistant commented Sep 5, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@VimalN2005 VimalN2005 changed the title fix(network): handle channel removal gracefully in processCompletedReceives (#2094) fix: handle channel removal gracefully in processCompletedReceives Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ERROR log "Channel xxxx removed from selector before processing completed receive"

2 participants