Skip to content

fix(pipeline): start endpoint before encoders to fix RTMP audio race condition#294

Open
theermite wants to merge 1 commit into
ThibaultBee:mainfrom
theermite:fix/twitch-audio-race-hoso
Open

fix(pipeline): start endpoint before encoders to fix RTMP audio race condition#294
theermite wants to merge 1 commit into
ThibaultBee:mainfrom
theermite:fix/twitch-audio-race-hoso

Conversation

@theermite
Copy link
Copy Markdown

Summary

Fix a race condition in EncodingPipelineOutput.startStreamUnsafe() that prevents RTMP ingest endpoints (Twitch confirmed) from ever receiving the AAC audio sequence header (csd-0 / AACPacketType=0), causing the audio track to be silently dropped while video flows normally.

Symptom observed

  • StreamPack 3.1.2 → RTMP → live.twitch.tv
  • Video reaches Twitch correctly (e.g. H.264/AVC 1920x858 @ 30fps, stable bitrate)
  • Twitch Inspector reports Audio Track | Codec [empty] | Sample Rate [empty] for the entire stream
  • Reproducible on Android 15 / Realme GT Neo5 (ColorOS), MIC source, AAC LC, 44 100 Hz mono
  • No exception thrown to caller; only a single W log line Error while writing FLV data appears in logcat

Root cause

In core/src/main/java/io/github/thibaultbee/streampack/core/pipelines/outputs/encoding/EncodingPipelineOutput.kt, startStreamUnsafe() currently launches the audio and video encoder coroutines before calling endpointInternal.startStream():

```kotlin
streamEventListener?.onStartStream()

val audioEncoderJob = audioEncoderInternal?.let {
coroutineScope.launch { it.startStream() } // <-- MediaCodec.start() inside
}
val videoEncoderJob = videoEncoderInternal?.let {
coroutineScope.launch { it.startStream() }
}
audioEncoderJob?.join()
videoEncoderJob?.join()

endpointInternal.startStream() // <-- RtmpClient.publish() inside, runs LAST
```

But:

  1. audioEncoderInternal.startStream() calls MediaCodec.start() internally. The AAC encoder emits its sequence header (csd-0, an AudioSpecificConfig of 2 bytes for AAC LC mono 44 100) within milliseconds of start.
  2. The output consumer launched in init { } (around lines 258-272) forwards every encoded frame to endpointInternal.write(closeableFrame, audioStreamId) as soon as audioStreamId != null.
  3. RtmpEndpoint.write() silently catches all Throwable:
    ```kotlin
    catch (t: Throwable) {
    Logger.w(TAG, "Error while writing FLV data: $t")
    }
    ```
  4. So when the csd-0 frame arrives at the endpoint, RtmpClient.publish() has not yet handshaked. The write fails, the exception is swallowed, and AudioFlvStream.sentSequenceStart flips to true on that first frame (it tracks "I already tried to send the header"). The codec config is never re-emitted.
  5. The RTMP server (Twitch ingest in our case) never receives an audio codec descriptor and reports an empty Audio Codec / Sample Rate for the whole session.

Video is unaffected because by the time the video encoder produces a keyframe (later than the AAC csd-0), endpoint.startStream() has usually completed.

Fix

Move endpointInternal.startStream() to before the encoder coroutines launch, so the RTMP publish() handshake and onMetaData write are guaranteed complete before any encoded frame can be produced.

```kotlin
streamEventListener?.onStartStream()

endpointInternal.startStream() // <-- moved up

val audioEncoderJob = audioEncoderInternal?.let { ... }
val videoEncoderJob = videoEncoderInternal?.let { ... }
audioEncoderJob?.join()
videoEncoderJob?.join()
```

A descriptive inline comment is added at the new position explaining the race.

Validation

Tested live against Twitch ingest (live.twitch.tv) from the Hoso Android app (https://github.com/theermite/Hoso), Realme GT Neo5 / Android 15 / ColorOS, StreamPack 3.1.2 + this single patch (Gradle composite build):

Before patch (StreamPack 3.1.2 stock):

  • Twitch Inspector → Audio Track: Live | Codec [empty] | Sample Rate [empty]

After patch:

  • Twitch Inspector → Audio Track: Live | Codec AAC | Sample Rate 44 100 Hz
  • Test De Configuration: Excellent
  • Stream stable, no degradation on video side (still 1920x858 H.264 @ 49.82 fps avg, 1857 kbps avg)

Scope

  • Single file changed: core/.../EncodingPipelineOutput.kt
  • 15 lines added (mostly the explanatory comment), 2 lines moved
  • No public API change
  • No behavior change for non-RTMP endpoints (the reordering is harmless for endpoints whose startStream() is cheap or whose write() does not depend on a prior handshake)

Test plan

  • Reproduce the bug on stock StreamPack 3.1.2 against Twitch RTMP ingest
  • Apply the patch
  • Verify Twitch Inspector reports AAC | 44100 Hz for the audio track
  • Verify video track unaffected
  • Verify stream stability over 30+ seconds
  • Maintainers: run the existing StreamPack test suite
  • Maintainers: validate against SRT endpoint (should be unaffected — SRT startStream is also fast, but the principle "endpoint ready before encoders" is structurally safer regardless)

🤖 Patch authored with the help of Claude Code on a real-world debugging session for Hoso.

EncodingPipelineOutput.startStreamUnsafe() launched the audio and video
encoder coroutines BEFORE calling endpointInternal.startStream(). On
RTMP endpoints this meant the AAC encoder's sequence header (csd-0,
AACPacketType=0) was emitted by MediaCodec within milliseconds of
start, forwarded by the init{} output consumer to the endpoint, while
the underlying RtmpClient.publish() handshake had not yet run.

RtmpEndpoint.write() silently catches all Throwables, so the failed
write of the AAC sequence header surfaced only as a single W log. The
AudioFlvStream.sentSequenceStart flag was set to true on that first
frame, so the codec config was never re-emitted. Twitch (and any RTMP
ingest) then never received an audio codec descriptor, and reported an
empty Audio Codec / SampleRate in Inspector for the whole stream while
video flowed normally.

Moving endpointInternal.startStream() to BEFORE the encoder coroutines
launch guarantees the RTMP publish() handshake and onMetaData write
complete before any encoded frame can be produced.

Observed on:
- StreamPack 3.1.2
- Android 15 / ColorOS (Realme GT Neo5)
- Twitch RTMP ingest live.twitch.tv

Co-Authored-By: Takumi "IA Dev Partner"
@ThibaultBee
Copy link
Copy Markdown
Owner

Hi,
Thanks for reporting.
However I am afraid this fix could bring regression.
Have you tried running the dev_v3_2 branch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants