Repository navigation
fix(mux-h2): release the slot of a frontend stream Sōzu resets - #1922
Merged
Merged
Conversation
A frontend HTTP/2 stream that Sōzu reset after a stream error the client provoked kept its stream slot, and the two pool buffers the slot holds, until the connection closed. `ConnectionH2::reset_stream` left the slot `StreamState::Unlinked` and every caller then dropped the wire id with `ConnectionH2::remove_dead_stream`; every path that recycles a frontend slot while the connection lives reaches it through the stream table, so `Context::create_stream` never reused it. The slot left the MAX_CONCURRENT_STREAMS count with its wire id, and the emitted-RST budgets sit far above the pool size, so nothing bounded the growth: 499 requests with a `:path` carrying a fragment, on one connection, took every buffer of a worker's default 1000-buffer pool, and the worker then refused new streams and sessions on all its listeners for as long as that connection stayed open. The slot stopped being recycled when the wire id started being dropped right after the reset (0c4a739): until then the stream stayed in the table and the write pass retired it once its RST_STREAM was flushed. On a frontend connection the reset now logs the request, when no line has logged it yet, and recycles the slot before the caller drops the wire id, as a RST_STREAM from the client and a stream timeout already did. This holds whatever state the slot was in, a stream already answered by a default answer included. A backend connection is unchanged: the slot belongs to the frontend stream, which still owes its client an answer. A request refused at its HEADERS is therefore logged when it is reset, tagged `H2::Reset`, instead of when the session closes, tagged `session close`. A debug invariant, checked after every frame and at the end of every read and write pass, now states the ownership rule: every slot of an HTTP/2 frontend session that is not `Recycle` is named by exactly one wire id. Four test fixtures that dropped a wire id without recycling its slot, or linked a slot they never registered, are aligned with it; their assertions are unchanged. The refusal of an oversized CONTINUATION block drops a wire id the same way, but the header-size flood check ends the connection first; a test pins that order. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>
This was referenced Nov 12, 2026
FlorentinDUBOIS
added a commit
that referenced
this pull request
Nov 12, 2026
main gained #1922 and an actions bump. Git merged every file on its own; CHANGELOG.md is rearranged by hand so the two Security entries under Unreleased share one heading, main's first. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A frontend HTTP/2 stream that Sōzu reset after a stream error the client provoked never gave its stream slot back. The slot, and the two pool buffers it holds, stayed pinned until the connection closed. This PR makes the reset recycle the slot.
Security impact
h2, orh2c_prior_knowledge). The backend protocol does not matter. HTTP/1.1 frontends are not affected: such a connection owns one slot for its whole life.max_buffers = 1000, 499 requests carrying a:pathwith a fragment, on one connection, take every buffer of the worker's pool. The worker then answers new HTTP/2 streams withRST_STREAM(REFUSED_STREAM)and cannot allocate buffers for new HTTP/1.1 or TCP sessions, on all its listeners, for as long as that connection stays open. It recovers when the connection closes.MAX_CONCURRENT_STREAMScount together with its wire id, and the emitted-RST_STREAMbudgets (glitch cap 2000 per window, lifetime floor 10 000) sit far above the pool size.RST_STREAMon a frontend: a malformed header block, a body that does not match itscontent-length, a zero-increment or overflowing streamWINDOW_UPDATE, a self-dependentPRIORITY, a frame on a half-closed stream, malformed trailers.Cause
ConnectionH2::reset_streamleft the slotStreamState::Unlinked, and each of its 11 callers then dropped the wire id withConnectionH2::remove_dead_stream. Every path that recycles a frontend slot while the connection lives reaches it through the stream table, soContext::create_streamnever reused the slot.The slot stopped being recycled in 0c4a739, which started dropping the wire id right after the reset. Until then the stream stayed in the table and the write pass retired it once its
RST_STREAMwas flushed (ConnectionH2::try_recycle_server_stream).The fix
On a frontend connection (
Position::Server), the reset now logs the request when no line has logged it yet, then moves the slot toStreamState::Recyclebefore the caller drops the wire id.ConnectionH2::handle_rst_stream_frameandConnectionH2::cancel_timed_out_streamsalready did the same. It applies whatever state the slot was in, a stream already answered by a default answer included.A backend connection (
Position::Client) is unchanged: the slot belongs to the frontend stream, which still owes its client an answer.A debug invariant,
ConnectionH2::check_slot_ownership, now states the rule: every slot of an HTTP/2 frontend session that is notRecycleis named by exactly one wire id. It runs after every frame and at the end of every read and write pass, so the unit and e2e suites now fail on any other site that orphans a slot. None tripped it.Behaviour change
A request refused at its HEADERS is logged when it is reset, with message
H2::Reset, instead of when the session closes, with messagesession close. Its timings end at the reset.doc/observability.mdand the CHANGELOG say so.Tests
Each new test was seen failing before the fix.
peer_provoked_reset_recycles_stream_slot(600 malformed HEADERS, default pool)499 of 499 slots do, pinning 998 pool bufferscontent_length_reset_recycles_stream_slot50 of 50 slots do, pinning 100 pool buffersreset_of_an_unlinked_stream_recycles_its_slotleft: Unlinked right: Recyclea_backend_stream_reset_leaves_the_frontend_slot_to_the_frontendPosition::Servergate is removeda_stream_rejected_at_its_headers_is_logged_once_when_it_is_resetsession closeH2::Reseta_reset_after_a_silent_close_does_not_log_the_request_againleft: 2 right: 1when the log guard is removedtest_h2_malformed_requests_release_their_stream_slotsprotocol_error=499 refused_stream=101, no answer on either connectiontest_h2_content_length_resets_release_their_stream_slotsThe two e2e tests run a real worker and backend: 600 resets on one connection, then a request served on that connection and on a second one.
Existing tests: four fixtures dropped a wire id without recycling its slot, or linked a slot they never registered. They are aligned with the invariant in their setup only; no assertion changed, and each test was seen failing again under its documented mutation. The
stream.rshelper for rejected-request log lines now emits and expectsH2::Resetinstead ofsession close; it checks the same request line.Commands run
All on the pushed commit, exit code 0:
cargo +nightly fmt --all -- --checkcargo clippy --all-targets --locked -- -D warningscargo clippy --all-targets --all-features --locked -- -D warningscargo check --release --locked -p sozu-libRUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p sozu-lib --lockedpython3 .github/scripts/check_doc_citations.py --base 072b610e7cargo test --workspace --locked --no-fail-fast -- --skip tests::fuzz_tests::: 3469 passed, 0 failed, 56 ignored (run before a final comment-only edit;cargo test -p sozu-lib --locked --lib protocol::muxwas rerun after it)The change was reviewed by a second, independent reviewer that saw only the diff. The Codex cross-check was not run.
Docs
lib/src/protocol/mux/LIFECYCLE.md(§3.1, §3.2, §4.3, §8.2, §9),doc/h2_mux_internals.md,doc/observability.md,CHANGELOG.md.Not in this PR
ConnectionH2::handle_continuation_header_statedrops a wire id the same way, but it cannot run: the header-size flood check ends the connection withGOAWAYfirst.an_oversized_header_block_ends_the_connection_before_its_refusalpins that order. The dead branch, and the prose that describes it as live, are left for a follow-up.a_removed_half_written_stream_still_completes_its_frameand its sibling documents a failure message that an assertion added in fix(mux-h2): give back the send credit of DATA prepared and dropped unsent #1646 now pre-empts. Not caused by this change.