io_uring event engine — Stage 2: completion-mode accept, tiers, hardening#31
io_uring event engine — Stage 2: completion-mode accept, tiers, hardening#31andypost wants to merge 11 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates the Stage 1 (poll-mode readiness bridge) and Stage 2 (completion-mode accept) of the new io_uring event engine, sharing several edge-trigger and signalfd helpers with the epoll engine. It also adds robust runtime fallback to epoll upon io_uring resource exhaustion and environment overrides for operational safety. The review feedback correctly identifies two instances (in nxt_epoll_signalfd_handler and nxt_io_uring_internal_cqe) where nxt_errno is checked or logged after executing nxt_debug macros, which risks overwriting the active error code; saving nxt_errno immediately after the system call is recommended.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| for ( ;; ) { | ||
| n = read(ev->fd, &sfd, sizeof(struct signalfd_siginfo)); | ||
|
|
||
| nxt_debug(task, "read signalfd(%d): %d", ev->fd, n); | ||
| nxt_debug(task, "read signalfd(%d): %d", ev->fd, n); | ||
|
|
||
| if (n != sizeof(struct signalfd_siginfo)) { | ||
| nxt_alert(task, "read signalfd(%d) failed %E", ev->fd, nxt_errno); | ||
| return; | ||
| } | ||
| if (n != sizeof(struct signalfd_siginfo)) { | ||
| if (n == -1 && nxt_errno == NXT_EAGAIN) { | ||
| return; | ||
| } | ||
|
|
||
| nxt_alert(task, "read signalfd(%d) failed %E", ev->fd, nxt_errno); | ||
| return; | ||
| } |
There was a problem hiding this comment.
In nxt_epoll_signalfd_handler, nxt_errno is checked after calling the nxt_debug macro. Since nxt_debug can internally invoke system calls or library functions that modify errno, there is a risk that nxt_errno is overwritten before it is checked for NXT_EAGAIN.\n\nTo prevent this, the value of nxt_errno should be saved immediately after the read system call, before any logging or debugging macros are executed.
for ( ;; ) {\n nxt_err_t err;\n\n n = read(ev->fd, &sfd, sizeof(struct signalfd_siginfo));\n err = (n == -1) ? nxt_errno : 0;\n\n nxt_debug(task, \"read signalfd(%d): %d\", ev->fd, n);\n\n if (n != sizeof(struct signalfd_siginfo)) {\n if (n == -1 && err == NXT_EAGAIN) {\n return;\n }\n\n nxt_alert(task, \"read signalfd(%d) failed %E\", ev->fd, err);\n return;\n }\nThere was a problem hiding this comment.
Fixed in 788e4c1: errno is saved right after read() and the EAGAIN check / alert use the saved value.
| if (iou->neventfd++ >= 0xFFFFFFFE) { | ||
| iou->neventfd = 0; | ||
|
|
||
| n = read(iou->eventfd.fd, &value, sizeof(uint64_t)); | ||
|
|
||
| nxt_debug(&engine->task, "read(%d): %z events:%uL", | ||
| iou->eventfd.fd, n, value); | ||
|
|
||
| if (n != sizeof(uint64_t)) { | ||
| nxt_alert(&engine->task, "read eventfd(%d) failed %E", | ||
| iou->eventfd.fd, nxt_errno); | ||
| } | ||
| } |
There was a problem hiding this comment.
In nxt_io_uring_internal_cqe, nxt_errno is checked after calling the nxt_debug macro. Since nxt_debug can internally invoke system calls or library functions that modify errno, there is a risk that nxt_errno is overwritten before it is logged in nxt_alert.\n\nTo prevent this, the value of nxt_errno should be saved immediately after the read system call, before any logging or debugging macros are executed.
if (iou->neventfd++ >= 0xFFFFFFFE) {\n nxt_err_t err;\n\n iou->neventfd = 0;\n\n n = read(iou->eventfd.fd, &value, sizeof(uint64_t));\n err = (n != sizeof(uint64_t)) ? nxt_errno : 0;\n\n nxt_debug(&engine->task, \"read(%d): %z events:%uL\",\n iou->eventfd.fd, n, value);\n\n if (n != sizeof(uint64_t)) {\n nxt_alert(&engine->task, \"read eventfd(%d) failed %E\",\n iou->eventfd.fd, err);\n }\n }\nThere was a problem hiding this comment.
Fixed in 788e4c1: errno is saved right after read(), before nxt_debug(), and the alert prints the saved value.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7f811eec3a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (engine->u.io_uring.tier >= NXT_IOU_TIER_ACCEPT) { | ||
| nxt_io_uring_arm_accept(engine, ev, slot); | ||
| return; |
There was a problem hiding this comment.
When the listener is disabled for EMFILE/max_connections, nxt_conn_listen_timer_handler() later calls nxt_fd_event_enable_accept() and then immediately invokes the cached lev->accept syscall path. In the ACCEPT tier this branch arms an IORING_OP_ACCEPT, so the timer path now has both a pending io_uring accept and a synchronous accept/accept4; because completion-mode CQEs never initialized lev->ready, that synchronous path decrements it from zero and can drain the backlog through the legacy loop until EAGAIN. This only happens after the 100 ms backpressure/resource retry, but it defeats the one-pending-accept invariant and can make a recovering worker monopolize accepts.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 788e4c1. The exact failure was worse than a plain invariant break: nxt_conn_io_accept() decrements lev->ready from zero (never initialized in completion mode), the wrapped uint32 latches read_ready = 1, and the legacy loop then drains the whole backlog on the recovering worker until EAGAIN — alongside the IORING_OP_ACCEPT the timer's enable_accept just armed. The fix installs an ACCEPT-tier conn_io.accept handler whose only job is to guarantee a pending oneshot accept exists (enable_accept is idempotent), so connections keep arriving one CQE at a time and stay distributed across workers. Validated by soaking a 300-connection burst against a 128-fd rlimit: 470 backpressure/timer cycles, full recovery, 100% success afterwards, no legacy-loop drains.
7f811ee to
788e4c1
Compare
|
Review addressed in 788e4c1: ACCEPT-tier |
|
check-whitespace was failing on a blank line at EOF introduced inside the dedup-refactor commit; since the workflow runs |
788e4c1 to
64c8cb5
Compare
77ed9c2 to
962b72d
Compare
Introduce the Stage-2 tier ladder (POLL < ACCEPT < RECV < OPT) and resolve
the kernel's capability once, from a cached throwaway-ring probe, never from
uname:
- multishot POLL_ADD (floor, already required by Stage 1)
- multishot accept functional probe: arm IORING_ACCEPT_MULTISHOT on a
throwaway abstract-namespace listener; an immediate
-EINVAL completion means unsupported, a short timeout
with no CQE means armed-and-waiting => supported.
- buf_ring + multishot recv functional probe: register a tiny buffer ring
and arm a multishot recv with buffer-select on a
socketpair; a delivered (non -EINVAL) completion means
supported.
NXT_IO_URING_FORCE_TIER now accepts none|create|poll|accept|recv|opt and caps
the resolved tier (never above kernel support) so every degraded path is
exercisable on a modern kernel. The behaviour at ACCEPT/RECV is still the
Stage-1 poll path in this commit; the tier byte only gates which completion
paths later commits light up.
Add SINGLE_ISSUER + DEFER_TASKRUN with an -EINVAL setup-flag fallback probe,
gated behind an explicit NXT_IO_URING_SETUP_OPT opt-in and OFF by default.
The kernel binds a ring's permitted submitter task at io_uring_setup() time
(verified on 7.0: -EEXIST on a submit from any other task, even for
SINGLE_ISSUER without DEFER_TASKRUN). Unit creates each router worker's
engine on the router-main thread and only later runs poll() on the spawned
worker thread, so enabling the flags by default busy-loops every worker's
poll() on -EEXIST. The flags are safe only where an engine is polled on its
creating thread (main/controller/application processes). create() also no
longer submits any probe SQE on the engine's own ring, so the first submit --
and thus the SINGLE_ISSUER binding -- always happens in poll() on the owning
thread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 1 disarmed a direction (POLL_REMOVE) whenever a CQE landed on a BLOCKED direction and re-armed (POLL_ADD) on the next enable_*, ~1 SQE pair per read -- a measured contributor to the router CPU/req regression on keepalive/proxy paths (results-stage1.md). Multishot poll is edge-like: the kernel posts one CQE per new wait-queue wakeup and never re-reports steady-state readiness (design §1.7a). A BLOCKED direction therefore cannot busy-loop, so the disarm was defensive churn. Keep the multishot poll armed and merely latch read_ready/write_ready without dispatching, exactly as epoll leaves a blocked fd in its set (block_read makes no syscall). The subsequent enable_read/enable_write becomes a pure state write with zero SQEs. Invariants preserved: dispatch is still gated on state (a BLOCKED direction is excluded from the work-queue add) and on the per-drain read_seq/write_seq dedupe, so no double dispatch; generation counters still reject stale CQEs; the !F_MORE re-arm still fires for a BLOCKED direction whose multishot the kernel dropped, keeping it armed. The conn read/write discipline consumes the latched readiness on its next voluntary read/write. Verified: keepalive, proxy-keepalive (c64) and close-mode proxy loads all clean (100% 200s, zero alerts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At tier >= ACCEPT, listen sockets are armed with IORING_ACCEPT_MULTISHOT
instead of a re-armed POLL_ADD. One SQE yields a CQE per accepted connection
with the new fd in res, and -- because accept is a *consuming* wake-one
operation -- N worker rings sharing a listen fd behave like N threads blocked
in accept(2): each connection completes on exactly one ring. This structurally
eliminates the cross-worker thundering herd that POLL_ADD (no EPOLLEXCLUSIVE
analogue) suffered in Stage 1, the dominant measured accept-path regression
(results-stage1.md). Measured on 8 workers under close-mode load: voluntary
context switches per accepted connection drop from ~2.5 (all workers wake) to
~0.01 (one worker consumes each connection), a ~270x reduction.
Design decisions:
- sockaddr: multishot accept's addr buffer is shared and overwritten
asynchronously as connections arrive, so it is unreliable per-CQE. The
remote address is filled with getpeername() on each accepted fd. This
trades the removed accept4() syscall for a getpeername() (net syscall count
roughly even); the herd elimination, not a syscall saving, is the win.
- accept flags: SOCK_NONBLOCK, matching Unit's accept4(..., SOCK_NONBLOCK)
contract (the existing accept4 path passes no CLOEXEC, so neither do we),
letting the accepted fd skip the per-conn nonblocking fixup.
- backpressure: cancel-based, reusing the existing accept machinery
unchanged. When a conn slot cannot be allocated (max_connections),
nxt_conn_accept_next -> _close_idle already calls disable_read, which now
cancels the multishot accept (ASYNC_CANCEL by the arming's user_data), and
the 100 ms listen timer's enable_accept re-arms it. No new backpressure
logic; deterministic and reversible.
- each accept CQE feeds nxt_conn_accept directly; its poll-mode re-arm tail
is inert because lev->socket.read_ready is never set in multishot mode.
- EMFILE/ENFILE/etc: the multishot terminates (!F_MORE) with res < 0 and is
routed through nxt_conn_accept_error -> 100 ms timer re-arm, unchanged.
- !F_MORE with a live listener re-arms the multishot accept.
Invariant preservation:
- user_data gains a self-describing ACCEPT bit (generation narrowed 14->13
bits, still 8192 values). An accept CQE is routed and, when stale
(generation mismatch after cancel/close/fd-reuse), still recognised so its
already-accepted fd is CLOSED, never leaked -- the accept analogue of the
recv "a stale CQE must return its buffer" rule.
- the per-direction generation counter still rejects stale accept CQEs; the
disable/close path bumps it and cancels the accept before Unit closes the
listen fd.
- a connection the kernel accepted but that cannot be serviced (over the
limit, or queued before a cancel) is closed, so no fd leaks.
Tier POLL keeps the Stage-1 re-armed POLL_ADD listener. Verified: accept-heavy
close-mode load at c8/c64 on both direct and proxy listeners is 100% clean;
getpeername fills the real client address (access log shows 127.0.0.1, not
0.0.0.0); full tier matrix (none/create/poll/accept/recv/opt) clean at c8/c64.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kernel buf_ring + multishot recv support is probed and cached, but the
operational tier is clamped to ACCEPT and the completion-mode recv read path
is not wired in this change. This is a deliberate, documented deferral -- the
in-tree note (near the tier defines) records the three coupled constraints that
must be resolved together first, none of them small:
1. Scope (§2.4): recv must cover data conns only, never UDS port sockets
(SCM_RIGHTS/cred cmsgs that provided-buffer recv cannot carry). But both
arm read through the same engine enable_read and nxt_fd_event_t has no
type tag to separate them -- and §1.7 forbids adding one. So recv has to
be driven from a completion-mode nxt_conn_io_t, not enable_read.
2. Buffer lifetime (OQ2): the zero-copy win needs a ring buffer handed to the
pipeline as c->read, but the router can pin c->read across the app
round-trip and starve the ring; the safe answer is copy-out at the parse
boundary, and a naive copy-out-everywhere model adds a userspace copy that
makes it net-negative. A lifecycle slip is a soak signal-11.
3. Read discipline: Unit block_read()s after each read while multishot recv
delivers continuously, so a blocked conn accumulates ring buffers until
ENOBUFS and the degrade-to-poll+recv path must be robust.
Since a subtly wrong recv path would break the DEFAULT build's soak with
signal-11 -- which the acceptance gate forbids -- wiring it is left to a
focused follow-up with its own soak gate. The ACCEPT tier already delivers
the dominant measured win (thundering-herd removal). The RECV capability stays
detected so the follow-up only flips the clamp and lights up the tier >= RECV
branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The multishot accept starved all but one router worker. A multishot accept is a PERSISTENT wake-one registration: with N worker rings arming it on the same shared listen fd, the kernel keeps completing every incoming connection into the same (first-armed) ring's CQ regardless of that worker's busyness -- observed on a debug build as ONE thread taking all 32 connections of a c=32 keepalive load while 7 workers idled, capping throughput at a single worker. EPOLLEXCLUSIVE avoids this because it wakes only workers actually WAITING in epoll_wait, so busy workers shed load organically. Replace it with a ONESHOT IORING_OP_ACCEPT re-armed after each completion. Each ring keeps at most one pending accept, so the set of pending accepts is the set of workers that have reached poll(): the kernel distributes connections among them like N threads blocked in accept(2) -- herd-free (accept is consuming) AND load-balanced (a saturated worker re-arms late, naturally yielding to idle workers). IORING_OP_ACCEPT evaluates the backlog at submission and completes immediately when it is non-empty, so bursts cannot stall (level-equivalent, contract item #5). This also lowers the accept-tier kernel floor: the probe now tests plain IORING_OP_ACCEPT, no multishot flag. Re-arm placement: measured both candidates on a debug build under c=64 keepalive and close-mode loads. (a) inline in the accept CQE handler and (b) a work item behind the listen handler give identical per-thread accept distributions (8 workers x 11-14% each) because the SQ ring is only flushed at the top of poll(), so both re-arms reach the kernel in the same submit -- after the worker's queued work has run. (a) is kept as the simpler form with no extra work item. Kept from the multishot design: the SOCK_NONBLOCK accept flag, the getpeername() sockaddr strategy, the self-describing ACCEPT user_data bit, generation-based stale-CQE rejection, cancel-based backpressure via disable_read + the 100 ms listen timer, and the EMFILE error path. A stale oneshot accept CQE still closes the fd it carries: such a CQE arises when the kernel completed the accept before a disable/close-issued ASYNC_CANCEL took effect -- the connection was already consumed from the backlog, the disable's generation bump marks the CQE stale, and the handler close(2)s the fd before the generation check returns (comment in nxt_io_uring_accept_cqe). New in the oneshot flow: benign accept errors (EAGAIN/ECONNABORTED) now re-arm immediately -- under multishot the registration survived such errors, but a oneshot arming is consumed even by an error CQE and the listener would otherwise strand. Verified (kernel 7.0, default tier ACCEPT): - distribution: c=64 keepalive AND close-mode both spread across all 8 workers at 11-14% each (was: one worker 100%); - throughput sanity vs FORCE_TIER=poll: keepalive c=64 -8%, proxy c=64 +8..14% (was -58%/-84%) -- starvation gone; - tier matrix none/create/poll/accept/recv/opt at c8 and c64: 100% 200s, zero alerts; soak 40x c8 + 10x c64: zero bad rounds, zero signal-11; SIGTERM under load x10 clean; pytest subset green on io_uring-default (50 passed) and plain (5 passed) builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five verified bugs, none yet observed in testing but all reachable:
1. Generation-wrap fd death. The stale-CQE comparisons matched the 13-bit
generation from user_data against the raw free-running uint16_t slot
counter: once a slot generation passed 8191, EVERY completion for that fd
was rejected as stale, leaving it permanently dead. All three compare
sites now reduce the slot counter with the new nxt_iou_gen() macro; a
comment documents the modulo-2^13 wrap semantics (a mis-accept would need
exactly 8192 disable/enable cycles while one CQE is in flight -- CQEs are
bounded by one CQ-drain window).
2. Ring fd + mmap leak per failed create. setup() could return NXT_ERROR at
the tier/feature gates AFTER io_uring_queue_init_params() succeeded but
BEFORE iou->tier was assigned, and free() keyed queue_exit on the tier.
Ring liveness is now tracked by a dedicated ring_inited flag set at init
and consumed by free().
3. Duplicate user_data registrations from oneshot_read/oneshot_write
(reachable via the sendfile job path re-arming a multishot-enabled conn):
they armed without checking for a live arming -- registering a second
poll under the SAME user_data since the generation was not bumped -- and
set the opposite direction INACTIVE without removing its armed poll,
leaking it. Both now remove any live arming first (the remove bumps the
generation, invalidating in-flight completions) before arming the oneshot.
4. Zeroed client address after peer RST. The getpeername() result was
discarded, so a peer that dropped the connection between the kernel's
accept and the CQ drain produced a conn with sa_family 0 flowing into
access logs and ACL matching. Such connections are now dropped: close
the fd, keep the pre-allocated conn for the next completion, keep
accepting -- the same disposition as the poll path's accept4() failing
for a torn-down peer.
5. Negative-fd table corruption. nxt_io_uring_slot() cast fd to uint32_t
for the grow check, so fd -1 would pass it, shrink the table to a single
slot and index it out of bounds. Negative fds now return NULL, which all
callers tolerate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tup flags
Three hardening fixes from the review:
1. Stranded queued signals. The signalfd handler read exactly ONE siginfo
per dispatch, but the engine's multishot poll is edge-like and the
per-drain dedupe collapses multiple CQEs into a single handler run, so a
second queued signal (e.g. SIGTERM behind SIGCHLD) stayed in the signalfd
until an unrelated future signal produced a new wakeup. The handler now
drains to EAGAIN, dispatching the signal handler once per siginfo.
2. Seccomp exposure of the startup probe. Every --io-uring build performs
one io_uring_setup(2) at registration; a seccomp policy answering
io_uring syscalls with SCMP_ACT_KILL delivers an uncatchable SIGSYS and
kills the binary at startup (SCMP_ACT_ERRNO policies were already
handled). That cannot be prevented from userspace, so minimise exposure:
a NXT_IO_URING=0|off environment kill-switch is now checked BEFORE any
io_uring syscall in both the registration probe and create(), and the
caveat is documented in the engine header comment and the --io-uring
help text. Verified under strace: NXT_IO_URING=0 starts on epoll_edge
with zero io_uring_setup calls.
3. Signals require signalfd. Without NXT_HAVE_SIGNALFD the engine used the
sigwait()-thread fallback whose signal() carries a signo -- which the
ring's eventfd doorbell drops -- so Unix signals would be silently lost.
The registration probe (and, belt-and-braces, create()) now refuses the
engine on such builds, keeping epoll as the default.
Also REMOVE the NXT_IO_URING_SETUP_OPT machinery (SINGLE_ISSUER +
DEFER_TASKRUN with the EINVAL retry): the kernel binds the ring's permitted
submitter task at io_uring_setup() time, and Unit creates router-worker
engines on the router-main thread while poll() runs on the worker thread, so
the flags busy-loop every worker on -EEXIST. The opt-in was a trap for
exactly the deployments that would reach for it; the §2.5 findings stay
recorded in a comment at the setup site, to be revisited only if ring
creation moves onto the polling thread. The registration probe now also
reuses the cached kernel-tier probe instead of standing up a second
throwaway ring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ade path Extract the io_uring->epoll degrade blocks duplicated between nxt_event_engine_create() and nxt_event_engine_change() into one static nxt_event_engine_fallback() helper (also removing their divergent exit styles), and fix the unhandled double-failure state in change(): When change() had already freed the old facility and both the requested create() and the epoll fallback failed, engine->event still held the OLD facility's vtable over a freed/half-written union; a later nxt_event_engine_free() or poll would dispatch stale function pointers into dead state. The double-failure path now zeroes both the vtable copy and the engine union, and nxt_event_engine_free() NULL-guards event.free for exactly this case. The helper also zeroes the union before the fallback create claims its arm (belt-and-braces; the failed create's own cleanup contract already zeroes it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Apply the review's verified cleanups (net -265 lines):
- Share the four helpers the io_uring engine had copied from epoll, which
it can reuse verbatim because multishot poll is equally edge-like:
nxt_epoll_test_accept4 + the accept4 conn-io variant, the edge-mode
recvbuf EOF shim, and the signalfd plumbing. The signalfd creation is
extracted into a shared nxt_epoll_signalfd_create() (engine-specific
registration stays local: epoll_ctl vs enable_read), and the shared
handler drains to EAGAIN -- required by io_uring's collapsed edge
delivery, equally correct under epoll's level-triggered registration.
Declarations live in nxt_event_engine.h beside the epoll externs.
- Remove the RECV-tier probe machinery: the completion-mode recv path is
not implemented, so probing buf_ring + multishot recv at every startup
bought nothing. The tier defines and FORCE_TIER names stay (documented
as reserved; "recv"/"opt" resolve to the full implemented tier), and the
deferral design note stays; the follow-up reintroduces a fresh probe
with the actual implementation.
- Probe IORING_OP_ACCEPT with io_uring_get_probe_ring()/
io_uring_opcode_supported() instead of arming a throwaway listener and
waiting 20 ms: accept is a first-class opcode (unlike multishot poll's
flag bit), so the opcode probe suffices and the startup delay goes away.
- Adopt epoll's eventfd counter trick: read the doorbell eventfd once per
2^32-2 posts instead of a drain loop per wakeup; multishot poll posts a
CQE per write()-side wakeup regardless of the accumulated counter, like
EPOLLET, so skipping the read never suppresses future doorbells.
- Drop the nsubmitted counter: close() now calls io_uring_submit()
unconditionally -- liburing skips the io_uring_enter() syscall when the
SQ ring is empty, so the counter only duplicated ring state.
- Pass the caller's resolved slot into nxt_io_uring_arm()/_arm_accept()
instead of a second table lookup (nothing between the lookups can
realloc the table).
- Register the appended (non-default) io_uring engine service through the
existing nxt_service_add(), keeping its duplicate check; the prepend
branch stays bespoke since ordering is the whole point there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The verified nxt_event_interface_t contract (with the readiness-vs-completion mismatch inventory), the io_uring engine design including the findings made during bring-up (edge-like multishot poll delivery, shared-listener accept distribution, task-bound SINGLE_ISSUER rings), and the A/B benchmark method and results for both stages against the epoll engine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ring - Install an ACCEPT-tier conn_io accept handler: the generic nxt_conn_io_accept() drains the backlog with synchronous accept(2) batches driven by lev->ready, which completion mode never initializes. Invoked from the 100 ms listen-timer retry path it decremented lev->ready from zero, latched read_ready off the wrapped counter and looped the legacy accept path until EAGAIN -- monopolizing the backlog on the recovering worker while the IORING_OP_ACCEPT that the timer's enable_accept had just armed was also pending. The replacement only guarantees a pending oneshot accept exists (enable_accept is idempotent); connections keep arriving one CQE at a time, distributed across workers. Verified with a 300-connection burst against a 128-fd rlimit: 470 backpressure cycles, full recovery, no legacy-loop drains. - Save errno before nxt_debug() in the shared signalfd handler and the eventfd doorbell read: logging can clobber errno, misclassifying EAGAIN or printing a bogus error code. - Extend the configure feature test with io_uring_prep_cancel64(), which the ACCEPT tier uses to cancel pending oneshot accepts. - Drop a duplicated nxt_iou_gen definition (the Stage 1 backport and this branch's review-fix commit both carried one). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Rebased onto the updated stage1 base (962b72d, itself rebased onto post-1.36.0 master with three rounds of review fixes folded in). The stage1 fixes were propagated into stage2's own architecture where they applied — most notably: stage2's completion-mode accept had the same missing-SOCK_CLOEXEC issue as stage1's accept4 shim (its comment even documented the now-false premise); fixed inside the commits that introduced it, and runtime-verified via /proc fdinfo that every accepted socket carries O_CLOEXEC. The arm-failure escalation now covers all 13 arm sites including the oneshot IORING_OP_ACCEPT re-arms, the error-only single-dispatch policy and the delete/disable slot-armed gate survive the dispatch rework (an armed accept is covered by the gate via read_armed, and remove() cancels it with ASYNC_CANCEL by exact user_data), and stage1's and stage2's NXT_IO_URING kill switches were deduplicated into the env_disabled helper ('0' or 'off') checked in both the probe and create(). Both build variants clean, test subset green on each, tier smokes (tier 2 / FORCE_TIER=poll / FORCE_TIER=none / NXT_IO_URING=0) all verified. New head bd3f296. |
64c8cb5 to
bd3f296
Compare
1 participant
io_uring event engine — Stage 2: completion-mode accept, feature tiers, review hardening
Stacked on #30 (Stage 1). Merge that first; this PR then rebases to master.
What's here
IORING_OP_ACCEPTre-armed per completion for listen sockets (tier ACCEPT, kernel 5.19+ probed at runtime). Accept completions replace the readiness+accept4()dance and — because each worker ring keeps at most one pending accept — the kernel distributes connections among waiting workers like N threads blocked inaccept(2): no cross-worker thundering herd (Stage 1's poll listener woke every worker per connection), no worker monopolization (a persistent multishot accept funnels every connection into the first-armed ring; measured and documented indocs/io_uring/design.md§2.1a).io_uring_get_probe, functional multishot-poll test, EINVAL setup-flag fallback — never version parsing) andNXT_IO_URING_FORCE_TIERfor exercising every degraded path on a new kernel. RECV-tier groundwork (buffer-ring multishot recv) is deliberately deferred; the three constraints a follow-up must solve are documented in-tree.NXT_IO_URING=0kill-switch before any io_uring syscall (seccomp-KILL hosts), oneshot-arming duplicate-registration fix, dropped-connection handling ongetpeername()failure, negative-fd guard, engine-change double-failure hardening, and deduplication against the epoll engine's helpers (shared accept4 test/conn-io, recvbuf EOF shim, signalfd setup+handler; net −265 lines).docs/io_uring/: the verified engine contract, the design doc with all bring-up findings, and the benchmark method + results for both stages.Benchmarks (5 interleaved rounds vs epoll, oha + ab, noise floor from a null A/A run)
docs/io_uring/results-stage1.md,results-stage2.md.Net standing is parity-minus on CPU with the projected wins in the deferred RECV tier — this lands as opt-in experimental (epoll remains the default in every flavor except the explicit
--io-uring-defaultA/B build).Testing
Tier-matrix functional runs (all tiers × c8/c64, zero errors), soak (0 failures, 0 crashes), SIGTERM-under-load 10/10, full pytest on the io_uring-default build (149 passed; 2 failures environmental, identical on the epoll build).
🤖 Generated with Claude Code