Skip to content

fix(bedrock): deliver streaming chunks individually instead of in batches#1759

Open
shivaber wants to merge 1 commit intostrands-agents:mainfrom
shivaber:fix/bedrock-streaming-batch-1523
Open

fix(bedrock): deliver streaming chunks individually instead of in batches#1759
shivaber wants to merge 1 commit intostrands-agents:mainfrom
shivaber:fix/bedrock-streaming-batch-1523

Conversation

@shivaber
Copy link

Description

Fix BedrockModel.stream() delivering chunks in batches instead of individually.

Replace call_soon_threadsafe(queue.put_nowait) with run_coroutine_threadsafe(queue.put()).result() and use a bounded queue (maxsize=1). This blocks the producer thread until each event is delivered to the event loop, preventing callback batching where all chunks were processed in a single event loop iteration.

Root cause: When Bedrock sends chunks in a burst (e.g., multiple chunks in one TCP segment), call_soon_threadsafe scheduled all put_nowait callbacks at once. The event loop processed them all before the queue.get() waiter ran, causing queue depths of 5-6 items and delivering all chunks simultaneously.

After fix: Queue depth is always 0 after each get(), ensuring chunks are yielded one at a time for proper real-time streaming.

Related Issues

Closes #1523

Type of Change

Bug fix

Testing

  • Added test_stream_delivers_chunks_individually regression test that verifies queue depth is 0 after every get() when chunks arrive in a burst

  • All 110 existing bedrock tests pass

  • Verified with reproduction script: before fix queue depth reached 5-6, after fix always 0

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
    ⠋- [x] I have added an appropriate example to the documentation to outline the feature, o r no new docs are needed
    Thinking...- [x] My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…ches

Replace call_soon_threadsafe(queue.put_nowait) with
run_coroutine_threadsafe(queue.put()).result() and use a bounded queue
(maxsize=1). This blocks the producer thread until each event is
delivered to the event loop, preventing callback batching where all
chunks were processed in a single event loop iteration.

Previously, when Bedrock sent chunks in a burst (e.g., multiple chunks
in one TCP segment), call_soon_threadsafe scheduled all put_nowait
callbacks at once. The event loop processed them all before the
queue.get() waiter ran, causing queue depths of 5-6 items and
delivering all chunks simultaneously to the consumer.

With the fix, queue depth is always 0 after each get(), ensuring
chunks are yielded one at a time for proper real-time streaming.

Closes strands-agents#1523
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] BedrockModel streaming delivers chunks in batches sometimes instead of individual stream events

1 participant