Skip to content

fix(mux-h2): release the slot of a frontend stream Sōzu resets - #1922

Merged
FlorentinDUBOIS merged 1 commit into
mainfrom
fix/h2-reset-stream-slot-recycle
Nov 12, 2026
Merged

FlorentinDUBOIS merged 1 commit into
mainfrom
fix/h2-reset-stream-slot-recycle

Conversation

@FlorentinDUBOIS

Copy link
Copy Markdown
Collaborator

Summary

A frontend HTTP/2 stream that Sōzu reset after a stream error the client provoked never gave its stream slot back. The slot, and the two pool buffers it holds, stayed pinned until the connection closed. This PR makes the reset recycle the slot.

Security impact

  • Class: denial of service by resource exhaustion (CWE-404, CWE-770), unauthenticated, remote.
  • Affected: every release from 2.0.0 to 2.3.4, HTTP/2 frontends only (TLS ALPN h2, or h2c_prior_knowledge). The backend protocol does not matter. HTTP/1.1 frontends are not affected: such a connection owns one slot for its whole life.
  • Effect: each reset orphans one slot. With the default max_buffers = 1000, 499 requests carrying a :path with a fragment, on one connection, take every buffer of the worker's pool. The worker then answers new HTTP/2 streams with RST_STREAM(REFUSED_STREAM) and cannot allocate buffers for new HTTP/1.1 or TCP sessions, on all its listeners, for as long as that connection stays open. It recovers when the connection closes.
  • Why nothing bounded it: the orphaned slot leaves the MAX_CONCURRENT_STREAMS count together with its wire id, and the emitted-RST_STREAM budgets (glitch cap 2000 per window, lifetime floor 10 000) sit far above the pool size.
  • Triggers: any stream error Sōzu answers with RST_STREAM on a frontend: a malformed header block, a body that does not match its content-length, a zero-increment or overflowing stream WINDOW_UPDATE, a self-dependent PRIORITY, a frame on a half-closed stream, malformed trailers.

Cause

ConnectionH2::reset_stream left the slot StreamState::Unlinked, and each of its 11 callers then dropped the wire id with ConnectionH2::remove_dead_stream. Every path that recycles a frontend slot while the connection lives reaches it through the stream table, so Context::create_stream never reused the slot.

The slot stopped being recycled in 0c4a739, which started dropping the wire id right after the reset. Until then the stream stayed in the table and the write pass retired it once its RST_STREAM was flushed (ConnectionH2::try_recycle_server_stream).

The fix

On a frontend connection (Position::Server), the reset now logs the request when no line has logged it yet, then moves the slot to StreamState::Recycle before the caller drops the wire id. ConnectionH2::handle_rst_stream_frame and ConnectionH2::cancel_timed_out_streams already did the same. It applies whatever state the slot was in, a stream already answered by a default answer included.

A backend connection (Position::Client) is unchanged: the slot belongs to the frontend stream, which still owes its client an answer.

A debug invariant, ConnectionH2::check_slot_ownership, now states the rule: every slot of an HTTP/2 frontend session that is not Recycle is named by exactly one wire id. It runs after every frame and at the end of every read and write pass, so the unit and e2e suites now fail on any other site that orphans a slot. None tripped it.

Behaviour change

A request refused at its HEADERS is logged when it is reset, with message H2::Reset, instead of when the session closes, with message session close. Its timings end at the reset. doc/observability.md and the CHANGELOG say so.

Tests

Each new test was seen failing before the fix.

Test Without the fix With the fix
peer_provoked_reset_recycles_stream_slot (600 malformed HEADERS, default pool) 499 of 499 slots do, pinning 998 pool buffers pass
content_length_reset_recycles_stream_slot 50 of 50 slots do, pinning 100 pool buffers pass
reset_of_an_unlinked_stream_recycles_its_slot left: Unlinked right: Recycle pass
a_backend_stream_reset_leaves_the_frontend_slot_to_the_frontend fails when the Position::Server gate is removed pass
a_stream_rejected_at_its_headers_is_logged_once_when_it_is_reset line tagged session close one line, H2::Reset
a_reset_after_a_silent_close_does_not_log_the_request_again left: 2 right: 1 when the log guard is removed pass
e2e test_h2_malformed_requests_release_their_stream_slots protocol_error=499 refused_stream=101, no answer on either connection pass
e2e test_h2_content_length_resets_release_their_stream_slots first failure at reset 499 pass

The two e2e tests run a real worker and backend: 600 resets on one connection, then a request served on that connection and on a second one.

Existing tests: four fixtures dropped a wire id without recycling its slot, or linked a slot they never registered. They are aligned with the invariant in their setup only; no assertion changed, and each test was seen failing again under its documented mutation. The stream.rs helper for rejected-request log lines now emits and expects H2::Reset instead of session close; it checks the same request line.

Commands run

All on the pushed commit, exit code 0:

  • cargo +nightly fmt --all -- --check
  • cargo clippy --all-targets --locked -- -D warnings
  • cargo clippy --all-targets --all-features --locked -- -D warnings
  • cargo check --release --locked -p sozu-lib
  • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p sozu-lib --locked
  • python3 .github/scripts/check_doc_citations.py --base 072b610e7
  • cargo test --workspace --locked --no-fail-fast -- --skip tests::fuzz_tests::: 3469 passed, 0 failed, 56 ignored (run before a final comment-only edit; cargo test -p sozu-lib --locked --lib protocol::mux was rerun after it)

The change was reviewed by a second, independent reviewer that saw only the diff. The Codex cross-check was not run.

Docs

lib/src/protocol/mux/LIFECYCLE.md (§3.1, §3.2, §4.3, §8.2, §9), doc/h2_mux_internals.md, doc/observability.md, CHANGELOG.md.

Not in this PR

  • The refusal of an oversized CONTINUATION block in ConnectionH2::handle_continuation_header_state drops a wire id the same way, but it cannot run: the header-size flood check ends the connection with GOAWAY first. an_oversized_header_block_ends_the_connection_before_its_refusal pins that order. The dead branch, and the prose that describes it as live, are left for a follow-up.
  • The "TO SEE THIS RED" recipe of a_removed_half_written_stream_still_completes_its_frame and its sibling documents a failure message that an assertion added in fix(mux-h2): give back the send credit of DATA prepared and dropped unsent #1646 now pre-empts. Not caused by this change.

A frontend HTTP/2 stream that Sōzu reset after a stream error the client
provoked kept its stream slot, and the two pool buffers the slot holds,
until the connection closed. `ConnectionH2::reset_stream` left the slot
`StreamState::Unlinked` and every caller then dropped the wire id with
`ConnectionH2::remove_dead_stream`; every path that recycles a frontend
slot while the connection lives reaches it through the stream table, so
`Context::create_stream` never reused it. The slot left the
MAX_CONCURRENT_STREAMS count with its wire id, and the emitted-RST
budgets sit far above the pool size, so nothing bounded the growth: 499
requests with a `:path` carrying a fragment, on one connection, took
every buffer of a worker's default 1000-buffer pool, and the worker then
refused new streams and sessions on all its listeners for as long as
that connection stayed open.

The slot stopped being recycled when the wire id started being dropped
right after the reset (0c4a739): until then the stream stayed in the
table and the write pass retired it once its RST_STREAM was flushed.

On a frontend connection the reset now logs the request, when no line
has logged it yet, and recycles the slot before the caller drops the
wire id, as a RST_STREAM from the client and a stream timeout already
did. This holds whatever state the slot was in, a stream already
answered by a default answer included. A backend connection is
unchanged: the slot belongs to the frontend stream, which still owes
its client an answer.

A request refused at its HEADERS is therefore logged when it is reset,
tagged `H2::Reset`, instead of when the session closes, tagged
`session close`.

A debug invariant, checked after every frame and at the end of every
read and write pass, now states the ownership rule: every slot of an
HTTP/2 frontend session that is not `Recycle` is named by exactly one
wire id. Four test fixtures that dropped a wire id without recycling
its slot, or linked a slot they never registered, are aligned with it;
their assertions are unchanged.

The refusal of an oversized CONTINUATION block drops a wire id the same
way, but the header-size flood check ends the connection first; a test
pins that order.

Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>
@FlorentinDUBOIS FlorentinDUBOIS self-assigned this Nov 12, 2026
@FlorentinDUBOIS
FlorentinDUBOIS merged commit 68f3484 into main Nov 12, 2026
46 checks passed
FlorentinDUBOIS added a commit that referenced this pull request Nov 12, 2026
main gained #1922 and an actions bump. Git merged every file on its own; CHANGELOG.md is rearranged by hand so the two Security entries under Unreleased share one heading, main's first.

Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant