Skip to content

Phase 8c: asynchronous transport — the async-http adapter and the per-gem floor - #96

Merged
Wahbeh-Mohammad merged 7 commits into
mainfrom
32-phase-8c-asynchronous-transport
Sep 22, 2026
Merged

Wahbeh-Mohammad merged 7 commits into
mainfrom
32-phase-8c-asynchronous-transport

Conversation

@Wahbeh-Mohammad

Copy link
Copy Markdown
Contributor

Part of #32. First PR of phase 8c's three-PR stack — the code — built off main at a7cfeb6 (phases 0–7 and 8a) concurrently with 8b #31, reviewed and approved there, and reconciled onto 8b's tree (5755267) once 8b approved first — rebased with every commit preserved, re-proven at the rebased tip; this PR's base is 8b's docs branch, so the stack lands in order. The first transport that runs inside a reactor, and the asynchronous half of the wire-boundary re-validation phase 1 postponed.

What lands

dexpace-transport-async_http becomes a real gem — 61 files — TRANSPORT-7, -8, -9, -12, -13, -21, -23, ASYNC-6, -21, -22 (nine ✅, ASYNC-21 N/A with its one honourable property asserted on ResponseBody), plus eleven cross-reference rows (HTTP-17/18/XCUT-18, XCUT-14, TRANSPORT-1–-29's second driver, ASYNC-7, OBS-19, SEAM-5/6/16, SEAM-24, CFG-7, NFR-2/3/11/13, NFR-10/14).

  • Dexpace::Transport::AsyncHTTP — the entry file's six constants (DEFAULT_TIMEOUT_SECONDS, DEFAULT_CONNECTION_LIMIT, MAX_ORIGINS, REGISTRY_KEY, FRAMING_HEADERS, ALPN_PROTOCOLS), .build(timeout:, logger:, drop_policy:, connection_limit:, ssl_context:, configuration:), .using(client, logger:, drop_policy:), .default and the require-time AsyncTransport.register(:async_http, …, core: "~> 0.0"); ten files under async_http/ — adapter.rb and drop_policy.rb public, the eight private_constants clients, endpoints, errors, exchange, request_body, request_mapper, response_body, response_mapper — every one mirrored in sig/ and every one but exchange.rb in test/.
  • The client map is keyed by (reactor, origin) (P8-92) — Fiber.scheduler identity at the call, MAX_ORIGINS (32) capping pairs, the drain evicting closed reactors first and releasing every evicted client; the cross-check's hang (one Async::HTTP::Client shared by two threads each in their own Sync) is what forced it.
  • The cancellation bridge is queue-marshalled (P8-91): the token's hook and Future#cancel settle the pivot and push the reason; a transient watcher task pops it on the reactor's thread and cancels the in-flight exchange or closes the delivered response; both hooks push through Exchange#signal, which swallows the ClosedQueueError of a push after the exchange's own end (round 2's R2-1). Async::Task#cancel from a foreign OS thread raises NoMethodError and cancels nothing — measured on 3.3.12, 3.4.10 and 4.0.6.
  • Both-protocol token predicate (P8-40): HeaderSyntax.token? applied before dispatch over HTTP/1.1, plaintext h2 and TLS h2; the seventeen delimiter bytes HTTP-17 admits and the grammar refuses are the antecedent. DropPolicy — EVERY / ONCE_PER_NAME / QUIET, the folded-name latch bounded at 64 — lands phase 5b's postponed OBS-19. RequestMapper#revalidate! is the second wire-boundary call site; no Content-Type is invented (P8-97).
  • The lazy ResponseBody — pull-per-demand, one native read per yield, a memoised #source, mid-stream failures as StreamError, and the native close through close_quietly because async-http 0.105.0 double-releases an unread h2 body (P8-101).
  • dexpace-conformance gains groups six and seven, Asynchronous (TRANSPORT-7, -9, -21, -23) and HeaderDrops (TRANSPORT-12, -13) — 28 → 34 assertions in seven groups, PREAMBLE naming TRANSPORT-8 as the third thing a green run does not prove — and Scripts writes Connection: close on every head with a close: keyword for the one keep-alive fixture (P8-96), with every 8a count unchanged.
  • The repository — the per-gem Ruby floor (VERSIONS' ruby floor:dexpace-transport-async_http 3.3, DexpaceVersions.ruby_floor(gem)/.gem_supported?, gates:versions and gates:gemspec_audit reading it, the Gemfile/test:gems/gates:clean_bundle skips, two per_gem_floor_ahead fixtures); the Steepfile's :async_http target relaxed as :serde_json is, with library "openssl", "uri" and the rbs_collection.yaml row ignoring the stale async/2.12 by name (R0-2); test/support/async_http_warmup.rb parking Ruby 4.0's once-per-process IO::Buffer warning (P8-98); one core widening, Configuration::Keys::TRANSPORT_CONNECTION_LIMIT, with its two pins flipped, core's four registration-invalidated pins moved to a child process, and both surface manifests regenerated (core 1,335 → 1,336, async_http 2 → 25).

Decisions taken in the open, against the plan's text

Ledger rows P8-91–P8-102 in the design's As-built addendum and the checklist's "Deviations from the plan" (forty-three items). Beyond those above: TRANSPORT-8 lives in this gem's own suite (the runtime-originated cancel has no portable antecedent) and six portable assertions plus the PREAMBLE sentence are the substitute (P8-93); a response does not outlive the reactor that produced it, so the driver's foreign-thread settle materialises the body inside its reactor (P8-94); the floor is one VERSIONS row read by gemspec, gates and loaders (P8-95); P8-37 as built retires every pooled resource before pool.close, because pool.close alone drains and waits exactly as Client#close does (P8-100); the public surface is .build/.using over .owning/.borrowing, DropPolicy public, no _Client interface (P8-102). Two of the design's thirteen facts are stale on async 2.46.0 — cause: drops a Symbol, and Kernel#Async inside a task is the task's child — corrected in a new knowledge-note entry and routed to phase 10's inbound list.

Layering

Each tip is green under every gate on its own tree — all eighteen. This branch is green on the SimpleCov floor too: 97.57% on 4.0.6 at the reconciled tip (3,833 runs, 73,334 assertions, 0 failures, 3 skips — the net_http driver's TRANSPORT-18 vacuity plus the two new groups' rows measured vacuous there; 97.52% / 3,713 / 3 at the reviewed tip, where Exchange#signal's rescue branch was the three uncovered lines); the 3.2.11 matrix row is green with the gem absent by its floor (3,824 runs, 3 skips, 95.62%, gates:clean_bundle "5 gem(s)"). The tests PR takes the same tree to 99.88% — 4,020 runs / 74,125 assertions / 7 skips: the net_http driver's three and the async driver's four (two TRANSPORT-14 assertions and one TRANSPORT-27 waived by id, TRANSPORT-18 vacuous).

Verification

  • Independent review, four rounds by four fresh reviewers with a fix round between each — round 0 1 / 2 / 4; round 1 0 / 1 / 2; round 2 0 / 2 / 2; round 3 approve, 0 / 0 / 3. The one real code change from review: Exchange#signal swallowing a hook's push after the exchange's own end (round 2's R2-1); everything else code-side was a comment or configuration correction (the rbs-collection row and Steepfile comment, round 0's R0-2; the gemspec's ~> 0.104 comment, round 1's R1-3; the portable row's race comment, round 1's R1-1). Round 3's three nits were documentation, folded into the reconcile pass's chore commit.
  • Mutations: 42, 45, 51 and 55 rows by the four reviewers on 4.0.6 and 3.3.12 (an a/b split is two; the prose counts named mutations), 37 / 40 / 45 / 52 red; the recorded equivalents are 14 (Kernel#Async is the current task's child on 2.46.0), 29 (a raise inside #dispatch's fence is still a settlement) and 46 (the cause: wrap is read back by nothing), with row 18 red since the round-2 hook-race case caught it. Forty-three guards run red and recorded in the checklist.
  • Reconcile re-proof (rebased tips c05ada2 / a56336e / 47c6afc, on 8b's tree): all eighteen gates individually at the code tip on 4.0.6 and the matrix set on 3.2.11; the full default task at the tests and docs tips; three extra seeds identical; 8b's composed_transport_test.rb green by name (10 runs / 45 assertions) against the Connection: close fixture; surface:regenerate a no-op; probe clean.

Known follow-ups from the reviews (not blocking a gate)

  • Recorded equivalents: guard 14 (measured — Kernel#Async delegates to Task.current.async), guard 29 (equivalent by construction), guard 46 (the cause: wrap names the reason for a debugger and nothing reads it back); guard 18 is red through the round-2 case and its close-count measurement stands separately.
  • The driver reports four skips where the plan said three — two assertions carry TRANSPORT-14 and a waiver is by id; a per-assertion waiver protocol is 8a's design's to change.
  • The portable TRANSPORT-7 row proves its delivered-body clause by chance against a streaming adapter (the cancel races the client's head parse; half the runs on 4.0.6, a third on 3.3.12 against the mutant) — stated in the row's comment, the checklist and the design addendum, and routed to phase 10's inbound list; the adapter's own body-path test is the deterministic proof.
  • AsyncHTTPServerFixture::QuietServer swallows a peer's mid-head EOFError from async-http's server (seen once in ~100 whole-suite runs; upstream report on the same inbound list); the three configuration-default pins build through AsyncHTTPHermeticConfiguration because Configuration.build reads the real environment.
  • Routed elsewhere: docs/first-release.md carries the gates:bounded_map status (the map is bounded, the gate does not exist until phase 9), the conformance line and the P8-9 box; OBS-29 has no route and says so; the 3.4.10 row's scratch bundle resolved the installed openssl 4.0.2 rather than a fresh resolve.
  • The design's facts 8 and 10 and async-http's Console noise are phase 10's inbound bullets, dated 2026-09-21.

@Wahbeh-Mohammad Wahbeh-Mohammad added type:feature New capability or enhancement area:transport Transport, async model, seams: TRANSPORT-* ASYNC-* SEAM-* labels Sep 22, 2026
@Wahbeh-Mohammad
Wahbeh-Mohammad changed the base branch from 31-phase-8b-async-runtime-adapter-docs to main September 22, 2026 10:37
dexpace-transport-async_http alone needs Ruby >= 3.3, because async-http
and its whole closure declare it (phase 8c's P8-36). The floor is one
row in VERSIONS, `ruby floor:dexpace-transport-async_http 3.3`, and
DexpaceVersions.ruby_floor(gem) with a KeyError fallback to the global
floor is what the gemspecs, gates:versions and gates:gemspec_audit read.
DexpaceVersions.gem_supported?(gem, ruby) is what the Gemfile, test:gems
and gates:clean_bundle consult, so a row below a gem's floor skips that
gem with the count printed instead of failing on Bundler's refusal.

Two gate fixtures, per_gem_floor_ahead under versions/ and
gemspec_audit/, are the deliberately failing inputs: a gemspec whose
floor is below its VERSIONS row.
Phase 8c's reference asynchronous transport over async-http ~> 0.104:
.build over a client map keyed by (reactor, origin) and bounded at
MAX_ORIGINS, .using over a caller's own client with retries already
zero, the Adapter behind both with its Closeable latch, one exchange
task per call under the caller's own task and one with_timeout budget,
the queue-marshalled cancellation bridge whose transient watcher acts on
the reactor's thread (a Task#cancel from a foreign OS thread raises and
cancels nothing), the RFC 7230 token predicate applied before dispatch
on both protocols with DropPolicy's once-per-name, bounded reporting,
the ten framing headers never copied, the wire-boundary re-validation's
second call site, the lazy pull-shaped ResponseBody whose native close
goes through close_quietly, the classifier that asks the token first,
the lenient inbound mapper, the default TLS context offering h2 by ALPN,
and the require-time registration under :async_http.

Core gains Configuration::Keys::TRANSPORT_CONNECTION_LIMIT; the two key
pins move, and the async seam's four in-process bare-require pins become
a child-process suite, as 8a's registration did to the sync seam's. The
:async_http Steep target relaxes UnknownConstant to :information as
:serde_json's does, since nothing in the closure ships a sig/. The
surface manifests are regenerated once: core gains the one key, this
gem its twenty-two public rows. test/support/async_http_warmup.rb
spends Ruby 4.0's once-per-process IO::Buffer warning at test-helper
load, before the fatal-warning hook can see it.
Phase 8c's portable rows join dexpace-conformance as two private groups:
Asynchronous (TRANSPORT-7, 9, 21, 23) and HeaderDrops (TRANSPORT-12,
13), each written against the suite contract's primitives alone, so the
suite is thirty-four assertions in seven groups; TRANSPORT-8 stays an
adapter's own row, which PREAMBLE now says beside 8a's two omissions.

Every head a script writes carries Connection: close and write_response
takes close: (true by default): WireServer closes after one exchange,
and without the header a client that pools keep-alive connections
re-used one the server had already closed and read EOF on its next
request one time in two against the async adapter. Net::HTTP builds a
client per call and reads the header as nothing; the one keep-alive
fixture passes close: false for its first response and every 8a count
is unchanged. The two size pins (lifecycle_test.rb, 8a's driver) move
to thirty-four, with TRANSPORT-12 and 13 vacuous by measurement there.
Phase 8c, review round 0's R0-2. The :async_http Steep target's comment and
the phase record said ruby/gem_rbs_collection carries nothing for this
gem's closure, and that the plan's `- name: async / ignore: true` row would
ignore nothing. Measured otherwise: the collection carries gems/async/2.12,
whose Task declares #stop and neither #cancel nor .current?, and `rbs
collection install` installs it the moment the workspace gem's own
`ignore: true` is lifted -- rbs cuts its dependency walk at an ignored gem,
which is the only reason the walk never reached async on the committed
tree. With the row in place and the workspace gem un-ignored the walk
reaches the closure (38 gems) and still installs no async; with the gem
ignored the lock is unchanged and steep stays green. The row is the
manager's decision of 2026-09-21 restored on its true reason, and the
Steepfile comment now states that reason.
The gemspec's comment said `~> 0.104` admits every 0.104.x and 0.105.x
release. A two-segment pessimistic constraint is `>= 0.104, < 1`: it
admits every 0.x release from 0.104 on, 0.200.0 and 0.999.9 included
(measured with Gem::Requirement#satisfied_by? on 4.0.6). The constraint
is the one the design chose; only the comment misdescribed it (review
round 1, R1-3). No code changes.
The TRANSPORT-7 row's comment said a streaming adapter surfaces the
cancellation from the body read. It does only when the cancel lands
after the consumer's read has blocked: the server's signal fires as the
head leaves its socket, and the cancel reaches the adapter either
before it has checked its token on the delivered head (the send
surfaces it, the in-flight path) or after (the read does). Measured
against the async-http adapter with its delivered-response close
deleted (review round 1, mutation 37): half the runs passed through the
send path and half hung in the read. The row proves the in-flight
clause on every adapter and the delivered-body clause only when the
race falls that way; the adapter's own suite pins the body path with
the consumer signalling from inside the read, and its driver bounds
`around:` (the tests branch, R1-1). The contract has no primitive that
tells the two apart for an eager and a streaming adapter alike, so the
comment states the race rather than the row pretending to settle it.
Comment only; no code changes.
A token cancel in flight while the exchange finishes raised
ClosedQueueError out of Cancellation::Source#cancel on the canceller's
thread (review round 2, R2-1). Source#cancel and Completer#settle both
steal their hook list under their mutex, flip the state and run the
hooks outside it; an exchange whose check-after-resume saw the flag in
that window settled the pivot cancelled and closed its queue through
release_watch, whose detach reached a list the source no longer held,
and the adapter's hook then pushed onto the closed queue. Hooks.notify
hands the first hook failure back to the caller, so an ordinary
caller's own hook registered first -- or the canceller merely being
descheduled between the flip and the push -- turned a legitimate cancel
into a raise on the cancelling thread while the future read cancelled
(reproduced deterministically on 4.0.6 and 3.3.12).

Both pushes now go through Exchange#signal, which rescues
ClosedQueueError: a hook that runs after the exchange ended has nothing
left to do, the pivot is settled cancelled with the reason and the
token reads cancelled, so the raise is swallowed there and nowhere
else. A closed? check first would be the same race one instruction
later. The watcher's comment also states that the CancelledError
wrapped into Task#cancel(cause:) names the reason for whoever reads the
task and is read back by nothing in the adapter -- the pivot is settled
before the watcher acts (R2-4). The sig mirror gains the private
method.
@Wahbeh-Mohammad
Wahbeh-Mohammad force-pushed the 32-phase-8c-asynchronous-transport branch from c05ada2 to 1dcaeca Compare September 22, 2026 11:04
@Wahbeh-Mohammad
Wahbeh-Mohammad merged commit 11ec324 into main Sep 22, 2026
5 checks passed
@Wahbeh-Mohammad Wahbeh-Mohammad mentioned this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:transport Transport, async model, seams: TRANSPORT-* ASYNC-* SEAM-* type:feature New capability or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant