Skip to content

Add streaming and pcm response_format to /v1/audio/speech - #1

Open
ashharadnan wants to merge 3 commits into
phattbeats:mainfrom
ashharadnan:main
Open

ashharadnan wants to merge 3 commits into
phattbeats:mainfrom
ashharadnan:main

Conversation

@ashharadnan

@ashharadnan ashharadnan commented Oct 2, 2026 •

Copy link
Copy Markdown

Ports devnen#176 (streaming + pcm response_format on the OpenAI-compatible /v1/audio/speech endpoint) onto this AMD fork, adapted to fork reality:

  • /tts untouched: this fork's CustomTTSRequest has no stream field and upstream's /tts restructure conflicted, so only the OpenAI endpoint gained streaming.
  • Two fork-specific import fixes: server.py here never imported asyncio (streaming generator calls get_running_loop/run_in_executor) or struct (streaming WAV header). Both added.
  • New config knobs: openai_stream_chunk_size (default 50), openai_stream_by_default.
  • README updated to describe /v1/audio/speech streaming only.

Tested on RX 6800 XT / gfx1030, ROCm 7.2 wheels, torch 2.9.1, Turbo model (warm):

  • stream=true, response_format=pcm, chunk_size=50: first audio ~0.9-1.0 s vs ~6.5-7.0 s non-streaming (same total wall time, ~8 s).
  • stream=true, response_format=wav: streaming WAV header + PCM16 frames verified.
  • Omitting stream: byte-identical legacy behavior.
  • Clause-fallback: 109-char single sentence splits correctly at a comma.
  • py_compile clean on server.py/config.py/utils.py.

Note: stream outputs engine-native 24 kHz PCM regardless of configured sample_rate; clients must wrap with their own header for pcm.

Known caveat (raw streaming wav): when stream=true and response_format=wav, the served WAV header carries the 0xFFFFFFFF wildcard for both the RIFF and data sizes, and it is never patched when the stream closes. This is by design - audio sizes aren't known upfront in streaming mode - but it means the raw response is not a well-formed file for direct playback: players that trust WAV duration metadata will show a bogus duration (observed ~89,478 s) or refuse the clip. Clients consuming raw streaming wav must ignore the header and rely on the stream end or accumulated chunk length instead; use pcm if you intend to write your own header client-side, as done in the reference client for this PR's tests.

)

Adapted to the AMD fork: /tts endpoint and CustomTTSRequest left untouched (fork never had stream on /tts; upstream restructure dropped in conflict resolution). README corrected to match fork reality. Fixed two fork-specific NameErrors the PR missed: server.py never imported asyncio (streaming generator) or struct (WAV header path).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant