internals.md

elfuse Internals

This document is the canonical technical reference for the runtime. It maps the source tree, then walks each subsystem from guest entry to syscall return: memory layout, the EL1 shim and HVC protocol, page-table management, syscall translation, threads, fork/clone, signals, ptrace, dynamic linking, and the GDB stub.

It is aimed at contributors. For a high-level overview see ../README.md; for command-line use see usage.md; for the validation flow see testing.md.

Runtime Model

elfuse runs one Linux guest process inside one Hypervisor.framework VM owned by one macOS host process. Guest code executes at EL0. A small EL1 shim (src/core/shim.S) handles exceptions, provides the syscall trap path, and cooperates with the host through a compact HVC protocol.

aarch64 guests execute their own instructions natively on the CPU. x86_64 guests (both static and dynamic) execute through Apple's Rosetta translator hosted inside the same VM at link address 0x800000000000 (see x86_64-via-Apple-Rosetta). Both architectures share the same EL1 shim, syscall surface, and host-side handlers.

Lifecycle In Five Steps

The whole runtime fits into a load -> boot -> run -> translate -> return loop:

1. Load. src/main.c parses options; src/core/elf.c parses the guest ELF and any PT_INTERP; src/core/guest.c reserves a demand-paged guest address space (up to 1 TiB IPA on M3+); the initial Linux stack is built by src/core/stack.c.
2. Boot. src/core/bootstrap.c installs the EL1 shim (src/core/shim.S) in the runtime infrastructure reserve, seeds page tables, and enters EL0. For x86_64 guests, Rosetta is loaded alongside as a co-resident aarch64 binary at 0x800000000000.
3. Run. The guest executes natively on an HVF vCPU. Pure computation never leaves the guest.
4. Translate. A guest SVC #0 traps into the shim and is forwarded over HVC #5 to the host. src/syscall/syscall.c dispatches into focused domain handlers (mem.c, fs.c, net.c, signal.c, ...) that translate errno values, flag layouts, struct shapes, and socket address formats between Linux and macOS.
5. Return. The host writes the result back into the vCPU, signals any required TLBI (see Dynamic Page-Table Extension And TLBI), and the shim ERETs back to EL0.

Threads enter the same loop on their own vCPU (src/runtime/thread.c). fork clones the loop in a new posix_spawn-ed elfuse process and transfers state through IPC (src/runtime/forkipc.c). execve reloads the ELF inside the existing VM and restarts the loop (src/syscall/exec.c).

Boot sequence:

1. src/main.c parses options and prepares guest bootstrap state.
2. src/core/elf.c parses the ELF image and any PT_INTERP interpreter.
3. src/core/stack.c builds a Linux-style initial stack (argc, argv, envp, auxv).
4. src/core/guest.c reserves guest memory, page tables, and semantic regions.
5. src/core/shim.S enters guest EL0 and forwards traps through HVC exits.
6. src/syscall/syscall.c dispatches Linux syscalls into domain handlers.
7. src/syscall/proc.c owns the main vCPU run loop and stop/exit integration.

Source Map

Top-level areas:

• src/core/: guest memory, ELF loading, bootstrap, initial stack, vDSO, EL1 shim assembly, and the embedded Rosetta translator host
• src/syscall/: Linux syscall handlers and translation boundaries
• src/runtime/: thread table, futexes, procfs helpers, fork/clone IPC, argv/comm rewriting for prctl PR_SET_NAME
• src/debug/: crash reporting and built-in GDB RSP stub

Key files:

File	Role
src/main.c	CLI, bootstrap, first vCPU creation, debugger startup
src/core/guest.c	guest memory reservation, region tracking, page tables
src/core/elf.c	ELF parsing, PT_LOAD, PT_INTERP, loader decisions
src/core/stack.c	Linux initial stack and auxv construction
src/core/shim.S	EL1 shim, exception vectors, HVC protocol
src/core/shim-globals.c	EL1-only shim_data cache (identity slots, urandom ring, attention bits)
src/core/vdso.c	synthetic vDSO (CNTVCT clock_gettime fast path, SVC trampolines)
src/core/rosetta.c	x86_64-via-Rosetta translator host, AOT cache, kbuf alias
src/core/sysroot.c	--sysroot / --create-sysroot provisioning
src/syscall/syscall.c	syscall dispatch and shared wrapper helpers
src/syscall/mem.c	brk, mmap, mprotect, mremap, madvise, msync
src/syscall/fs.c, fs-stat.c, fs-xattr.c	filesystem syscalls
src/syscall/io.c, poll.c, fd.c, fdtable.c	I/O, polling, FD lifecycle and table
src/syscall/path.c	centralized guest-to-host path resolution
src/syscall/casefold.c	guest/host filename encoding for case-folding volumes (see filenames.md)
src/syscall/casefold-walk.c	case-exact path resolution against the sysroot
src/syscall/fuse.c	guest-internal FUSE transport and minimal VFS
src/syscall/inotify.c	inotify via kqueue EVFILT_VNODE
src/syscall/sysvipc.c	System V shared memory and semaphores
src/syscall/signal.c	signal delivery, rt_sigframe, rt_sigaction
src/syscall/time.c	clocks, timers, setitimer, clock-ID translation
src/syscall/sys.c	uname, sysinfo, getrandom, prlimit64
src/syscall/net.c, net-abi.c, net-absock.c, net-msg.c, net-sockopt.c, netlink.c	sockets, SCM_RIGHTS, abstract Unix sockets, netlink
src/syscall/usbdev.c	usbdevfs fds over IOKit (see USB Device Passthrough)
src/syscall/translate.c	errno and shared AT_* flag translation
src/syscall/proc.c	vCPU run loop, wait4, ptrace coordination, HVC #6 routing
src/syscall/exec.c	execve: ELF reload, interpreter resolve, vCPU restart
src/runtime/forkipc.c, fork-state.c	fork/clone state transfer over the fork IPC channel
src/runtime/thread.c, futex.c	guest thread table, futex wait queues
src/runtime/procemu.c	/proc, /dev, and selected pseudo-files
src/runtime/usb-sysfs.c	synthetic /dev/bus/usb and /sys/bus/usb trees from the IOKit registry
src/runtime/proctitle.c	argv / comm rewriting for prctl PR_SET_NAME
src/debug/gdbstub.c, gdbstub-rsp.c, gdbstub-reg.c	GDB RSP stub

Generic Dynamic Containers

src/utils.h provides the raw dynamic_array_t used by the procfs VMA snapshot and the string builder. It is header-only: every operation is a static inline, so there is no container object to link. Capacity is measured in element slots, while count is the number of logical elements. The allocation is one contiguous block of capacity * element_size bytes; both the count addition and the multiplication are checked before a growth. Arithmetic overflow reports EOVERFLOW, invalid arguments report EINVAL, and an allocation failure reports ENOMEM. Growth is transactional: on any failure the old pointer, count, and capacity remain valid.

The generated typed facades own only their array storage. Elements are copied as trivially-copyable bytes, so the container does not call destructors and does not manage pointers or other resources held by an element. destroy frees the contiguous block and restores the zero state. A facade can therefore be declared as {0} and initialized lazily on its first operation.

string_builder_t wraps a raw dynamic_array_t of char. The count is the C-string length and excludes the terminator, so every path that grows the builder reserves one byte beyond the payload and restores data[count] == '\0'. The surviving surface is init, destroy, length, data_const, append_n, and appendf; the plain-text append, reserve, and the capacity accessor went with the header-only conversion because nothing outside the tests used them. appendf renders into a scratch buffer before touching storage, which is what makes a format string or argument aliasing the builder's own data safe, and it commits only the prefix through the first NUL, matching standard C string semantics.

Hypervisor.framework Constraints

Apple HVF imposes a handful of constraints that shape the rest of the design:

• W^X is enforced even with SCTLR.WXN=0. A page-table entry cannot be both writable and executable. Use RW for data and RX for code.
• HVF returns SCTLR=0x0 by default; RES1 bits must be set explicitly. The shim uses SCTLR_RES1 (0x30D00980) plus the desired bits.
• The MMU must be enabled from inside the vCPU (via HVC #4 from the shim), not by the host before hv_vcpu_run(). Setting SCTLR.M=1 from the host before vCPU entry causes permission faults on the first instruction fetch.
• GUEST_IPA_BASE must be 0. ELF binaries use absolute addresses from their link address (e.g., 0x400000); a non-zero IPA base produces translation faults.
• System registers cannot be set via MSR from the guest because HCR_EL2.TSC=1 traps all MSR writes. Boot-time sysreg installation (RES1 bits, MMU enable, TTBR0, etc.) goes through HVC #4 from the EL1 shim. Runtime EL0 sysreg traps -- MSR TPIDR_EL0 and similar -- are handled by the HVC #12 system-instruction trap path.
• Only HV_SYS_REG_* constants from Hypervisor.framework may be used for register IDs.
• hv_vcpu_t value zero is a valid handle (normally the first vCPU in a VM), not an invalid sentinel. guest_t and thread_entry_t therefore track handle ownership with a separate vcpu_valid flag; signal preemption, quiesce, ptrace, and teardown must consult that flag before calling HVF.
• Cross-thread vCPU register access is unreliable; all register access must happen on the owning thread. This drives the snapshot protocol used by both ptrace and the GDB stub.
• HVF allows only one VM per host process. This is the reason fork is implemented through posix_spawn plus IPC state transfer.

Memory Layout

Guest memory is identity-mapped (guest VA == guest IPA). Large areas use 2 MiB block descriptors by default; mappings that need mixed permissions are split to 4 KiB L3 pages (see Page Table Splitting).

Low, fixed addresses:

0x00400000   - varies:       ELF LOAD segments (PIE_LOAD_BASE for ET_DYN)
0x01000000:                  brk base (16 MiB)
0x07800000   - 0x07800FFF:   Stack guard page (PROT_NONE, dynamic position)
0x07801000   - 0x07FFFFFF:   Stack (8 MiB, 4 x 2 MiB blocks, RW, grows down)
0x10000000   - 0x101FFFFF:   mmap RX region (initial 2 MiB, pre-mapped RX)
0x10200000   - mmap_limit:   mmap RX growth area
0x000200000000 - 0x0002001FFFFF: mmap RW region (initial 2 MiB at 8 GiB, RW)
0x000200200000 - mmap_limit:     mmap RW growth area

Within-32-bit values are rendered with 8-digit padding; values that extend above 32 bits use 12-digit padding so a reader can tell the range class at a glance.

High addresses, anchored to interp_base (computed from guest_size, see below). The runtime infrastructure reserve is a 16 MiB region placed at [interp_base - INFRA_RESERVE, interp_base) -- in the dead zone above mmap_limit -- so guest binaries keep the low addresses their link scripts expect. It is present regardless of --sysroot; offsets are relative to its base infra = interp_base - 16 MiB (exact values in src/core/guest.h):

infra + 0x000000 - 0x00FFFF:  null guard (64 KiB, unmapped)
infra + 0x010000 - 0xDF5FFF:  page table pool (~13.9 MiB, RW)
infra + 0xDF6000 - 0xDFFFFF:  shim code slot (40 KiB, RX). Shares the PT
                              pool's tail 2 MiB L2 block, so that block
                              splits to 4 KiB L3 pages (mixed RX/RW).
infra + 0xE00000 - 0xFFFFFF:  shim data + EL1 stack (2 MiB L2 block, RW;
                              ends at interp_base)
interp_base      - varies:    Dynamic linker (g->interp_base, --sysroot only)

The reserve is demand-paged (MAP_ANON), so its unused page-table-pool pages cost no host RAM despite the generous virtual reservation. The pool holds ~3558 L3 pages (~7 GiB of split address space), enough for the many V8 isolates a Node worker_threads pool or cluster spins up.

The guest size is determined by the VM's configured IPA width (capped at 40-bit / 1 TiB):

• 36-bit IPA (64 GiB) -- native AArch64 on Apple M2: mmap_limit ≈ 56 GiB, interp_base ≈ 60 GiB
• 40-bit IPA (1 TiB) -- native AArch64 on Apple M3 and later: mmap_limit ≈ 1016 GiB, interp_base ≈ 1020 GiB

Both mmap_limit and interp_base are computed at runtime from guest_size and stored in guest_t. macOS demand-pages physical memory on first touch, so the reservation costs no RAM until the guest writes to a page.

The mmap RW region starts at 8 GiB to match real Linux kernel address-space layout, where mmap allocations sit well above text/data/brk.

For address spaces larger than 512 GiB, the L0 page table needs multiple entries (each covers 512 GiB). The page-table builders compute the L0 index from the actual IPA and allocate L1 tables on demand per L0 slot.

Page Table Splitting

A 2 MiB L2 block descriptor cannot mix RW and RX permissions, but real shared libraries combine .text (RX) and .data (RW) within a single 2 MiB range. guest_split_block() in src/core/guest.c converts an L2 block descriptor into a table descriptor pointing to an L3 table of 512 × 4 KiB page entries, each with independent permissions.

Splitting is triggered by:

• sys_mmap with MAP_FIXED, when the fixed address lands in a block whose permissions differ from the request (typical case: the dynamic linker overlaying .data RW onto a library .text RX block).
• sys_mprotect, when changing permissions for a sub-block range, e.g. RELRO finalization.

guest_update_perms() orchestrates the full workflow: it checks whether a block needs splitting, splits it if so, then updates the affected L3 page entries. Whole-block permission changes are done in place without splitting.

mmap itself uses a gap-finding allocator that walks the sorted region array to find free address space. PROT_EXEC requests go to the RX region (MMAP_RX_BASE = 0x10000000); other requests go to the RW region (MMAP_BASE = 0x200000000). Address hints are honored when possible. This arrangement makes .text and .data land in different 2 MiB blocks where practical, and L3 splitting handles the residual cases where they share a block.

Dynamic Page-Table Extension And TLBI

When sys_mmap or sys_brk needs memory beyond the currently mapped page tables, the host calls guest_extend_page_tables() to add new L2 entries. This is safe because the vCPU is paused while servicing HVC #5. The host accumulates the smallest sufficient TLBI request into a per-vCPU _Thread_local cpu_tlbi_req slot (see src/core/guest.h) and translates it into X8, X9, X10, and X11 on return from HVC #5:

X8 == 0  TLBI_NONE        no flush; restore GPRs (keep X0); ERET
X8 == 1  TLBI_BROADCAST   TLBI VMALLE1IS + DSB ISH + ISB
                          -> restore GPRs (keep X0); ERET
X8 == 2  drop-frame       discard the saved GPR frame
                          (`add sp, sp, #256`) and ERET on the rebuilt
                          EL0 register state, reloading only `X8` from
                          the frame's own slot (`[sp, #64]`), since the
                          marker occupies that register. Set by
                          `rt_sigreturn` (which writes the whole
                          register set directly into the vCPU) and by
                          signal delivery on the syscall-return path
                          (so handler PC/SP/LR/args installed by the
                          host are not overwritten by the stale shim
                          frame on ERET). Always issues `IC IALLU`.
X8 == 3  TLBI_RANGE       loop TLBI VAE1IS over `X9` (start VA),
                          `X10` (page count); 4 KiB granule. Used for
                          up to `TLBI_SELECTIVE_MAX_PAGES = 16` pages.
X8 == 4  TLBI_RANGE_LARGE single-shot TLBI RVAE1IS with the encoded
                          operand in `X9`. Used for 17..64 pages when
                          `FEAT_TLBIRANGE` is available
                          (`g_tlbi_range_supported`). Above 64 pages,
                          or when the feature is absent, the host
                          upgrades to broadcast (`X8 == 1`).
X11      icache hint      Set to `1` when the request transitions a
                          page to executable (W^X swap); the shim then
                          issues an `IC` invalidate alongside the
                          chosen TLBI flavor.

TLBI VAE1IS retires any 2 MiB block entry containing the VA (ARM ARM B2.2.5.6), so guest_split_block() callers no longer issue a separate broadcast after the split lands.

X8 == 2 is the generic drop-saved-frame marker: the host has rebuilt EL0 register state directly into the vCPU and the saved syscall frame on the EL1 stack is stale, so the shim drops the frame and ERETs without restoring GPRs. Two call sites write it, both in src/syscall/signal.c:

• signal_rt_sigreturn, after restoring the saved sigframe.
• deliver_signal_locked, when a signal is delivered on the syscall-return path; without the marker the shim would overwrite the handler PC, SP, LR, and arg-register state with the stale syscall frame on ERET.

sys_execve writes no marker: it re-enters through the shim's MMU-off _start, which pops no frame. The HVC #5 epilogue and both HVC #9 W^X tails branch to exec_drop_frame on the marker; handle_brk (HVC #10) drops its frame on every return.

Linux preserves X1-X30 across SVC #0, so the marker must not reach EL0: a resumed SVC that has not executed yet would run as syscall 2. Those tails therefore reload X8 from the frame's X8 slot before the pop. The slot holds the X8 the exception was taken with unless the host published another there: signal_rt_sigreturn publishes the one it restored, and the BRK ptrace stop the one its tracer left. signal_rt_sigreturn also parks that value for a signal delivered later in the same epilogue, which would otherwise snapshot the marker as the guest's X8; the run loop drops the record before every hv_vcpu_run(), and an inline ptrace stop that moves the PC re-keys it. The TLBI kinds on the ordinary syscall-return tail are not covered: a signal delivered there still records the wire values as X8-X11.

Important: signal_rt_sigreturn returns SYSCALL_EXEC_HAPPENED to bypass the normal syscall dispatch epilogue, as sys_execve does. deliver_signal_locked runs from inside the epilogue. Any future code path that rebuilds EL0 register state on the syscall-return path must write X8 = 2 the same way, and publish the guest's X8 if it differs from the one the frame was entered with.

EL1 Shim And HVC Protocol

Vectors enter EL1 from EL0 traps and forward them to the host through HVC. DC ZVA is emulated inside the shim (it zeroes 64 bytes at the cache-line- aligned address from the Rt register); HVF traps DC ZVA via HCR_EL2.TDZ=1.

HVC #	Purpose	Registers
#0	Normal exit	X0 = exit code
#2	Bad exception	X0=ESR, X1=FAR, X2=ELR, X3=SPSR, X5=vector
#4	Set boot system register	X0 = reg ID (0–8), X1 = value (used by the shim during boot to install RES1 bits and enable the MMU)
#5	Syscall forward	X0–X5 = args, X8 = syscall number on entry; on return X8 carries the TLBI kind (0 = none, 1 = broadcast, 3 = selective range with X9 = VA + X10 = page count, 4 = single-shot TLBI RVAE1IS with encoded operand in X9). X8 = 2 is the generic drop-saved-frame marker -- set when the host has rebuilt EL0 state directly (by rt_sigreturn and by signal delivery on the syscall-return path) so the shim discards the saved syscall frame on ERET, reloading only X8 from it. X11 is the icache-flush hint (set to 1 when the request transitions a page to executable, so the shim issues IC alongside the chosen TLBI)
#6	Embedder extension	X8 = call number, X0–X7 = args; routed to g->hvc6_handler if set, no-op otherwise. Handler may request a vCPU yield via proc_request_hvc6_yield()
#7	MRS trap (read sysreg)	host reads register from ESR ISS; returns value in X0
#9	W^X toggle	X0 = FAR, X1 = type (0 = exec→RX, 1 = write→RW); on return X8 = 2 when the host answered with a SIGSEGV delivery instead of a flip
#10	BRK from EL0	SIGTRAP delivery / ptrace-stop; GPRs in frame
#11	EL0 fault	SIGSEGV/SIGILL delivery; GPRs in frame
#12	EL0 system-instruction trap	cache maintenance logging (DC CVAU, IC IVAU, …) and MSR TPIDR_EL0 emulation
#13	Ptrace interrupt	the shim restores the saved SVC frame before this stop, so ptrace snapshots architectural registers

Critical Vector-Entry Rule

Vector entry stubs that lead to svc_handler MUST NOT clobber any GPR. The Linux syscall ABI preserves all registers except X0 across SVC #0, and musl/glibc rely on this for scratch registers (X9–X15). If a vector entry writes to any GPR (e.g., mov x5, #offset) before svc_handler saves registers, the saved value is wrong and the EL0 caller's register state is corrupted after ERET.

Only bad_exception vectors may clobber X5 (they halt, so preservation is unnecessary).

Syscall Translation Boundary

Linux user-space compatibility comes from explicit translation at the syscall boundary. src/syscall/syscall.c routes syscall numbers into focused domain files. The translation layer is responsible for:

• errno translation
• flag translation
• Linux/macOS structure layout adaptation
• guest memory copying for pointer arguments
• Linux-compatible descriptor and signal semantics

errno

macOS and Linux errno values diverge starting around 35. linux_errno() in src/syscall/translate.c maps via switch. Notable mappings:

Linux	macOS
EAGAIN (11)	EAGAIN (35)
ENOSYS (38)	ENOSYS (78)
ENAMETOOLONG (36)	ENAMETOOLONG (63)
ELOOP (40)	ELOOP (62)

AT_* Flags

AT_* flag bits differ between Linux and macOS:

Flag	Linux	macOS
AT_SYMLINK_NOFOLLOW	0x100	0x20
AT_SYMLINK_FOLLOW	0x400	0x40
AT_REMOVEDIR	0x200	0x80

Most AT_* flags go through translate_at_flags() in src/syscall/translate.c before issuing macOS calls. faccessat is a special case: Linux defines AT_EACCESS at the same bit (0x200) as AT_REMOVEDIR, so faccessat paths instead use translate_faccessat_flags() (also in src/syscall/translate.c), which interprets that bit as AT_EACCESS.

open Flag Values

Aarch64-Linux open flags differ from x86_64. From asm-generic/fcntl.h:

Flag	Value (octal)	Value (hex)
O_DIRECTORY	040000	0x4000
O_NOFOLLOW	0100000	0x8000
O_DIRECT	0200000	0x10000
O_LARGEFILE	0400000	0x20000 (no-op on LP64)
O_CLOEXEC	02000000	0x80000

Clock IDs

Linux CLOCK_MONOTONIC = 1, macOS CLOCK_MONOTONIC = 6. See translate_clockid() in src/syscall/time.c. Other clock IDs are translated similarly.

Sockets

Socket syscalls are translated in src/syscall/net.c and friends:

• AF_INET6 differs: Linux 10, macOS 30.
• sockaddr has no sa_len byte on Linux but does on macOS. All conversions go through linux_to_mac_sockaddr() and mac_to_linux_sockaddr().
• Linux ORs SOCK_NONBLOCK (0x800) and SOCK_CLOEXEC (0x80000) into the type argument; both bits must be extracted before calling socket().
• SOL_SOCKET option numbers (SO_TYPE, SO_SNDBUF, SO_RCVBUF, …) differ between platforms and are remapped per option.

Netlink sockets are emulated in src/syscall/netlink.c: NETLINK_ROUTE answers the RTM_GETLINK / RTM_GETADDR dumps getifaddrs issues, and NETLINK_KOBJECT_UEVENT sockets are accepted as silent sockets -- no uevent is ever synthesized, so the fd never becomes readable, which is exactly a real Linux kernel with no hotplug activity -- because libusb-style hotplug monitors must open, bind, and set SO_PASSCRED on one before they will initialize at all.

socket() validates its type argument the way __sock_create() and netlink_create() do, in that order: an unknown bit outside SOCK_NONBLOCK|SOCK_CLOEXEC is EINVAL, and a base type other than SOCK_RAW or SOCK_DGRAM is ESOCKTNOSUPPORT -- before the protocol number is looked at, so socket(AF_NETLINK, SOCK_STREAM, 99) reports the type error and not the family one.

SOL_SOCKET options on a netlink fd follow sk_setsockopt / sk_getsockopt rather than being accepted and forgotten: whatever can be set can be read back, SO_RCVBUF and SO_SNDBUF clamp-double-floor at the two different SOCK_MIN_* floors, SO_RCVTIMEO really does bound a blocking receive, and the options Linux refuses (SO_SNDLOWAT, a privileged SO_DEBUG, a non-inet SO_REUSEPORT, a short SO_LINGER or timeout optlen) are refused with the same errno.

SO_RCVTIMEO and SO_SNDTIMEO are stored as sk_rcvtimeo / sk_sndtimeo are: a jiffy count with HZ pinned at 1000, which gives sock_set_timeout()'s three states rather than two. {0, 0} and any tv_sec past MAX_SCHEDULE_TIMEOUT/HZ - 1 mean wait forever; a negative tv_sec means do not wait at all, so a blocking receive under it reports EAGAIN at once; anything else is that many milliseconds, rounded up from a sub-jiffy tv_usec. Both of the first two read back as {0, 0}, exactly as sock_get_timeout() reports them, so only a receive can tell them apart. The deadline such a timeout produces is added to the clock with saturation, since the largest one Linux accepts is within seconds of overflowing a 64-bit millisecond count.

An interrupted receive with a finite timeout forbids the SVC restart only when the interruption is one the guest can see. io_wait_fd_timed_or_- interrupted() also reports EINTR for the dispatcher's own execve handoff, which no signal accompanies; forbidding there turned a sibling's execve into an EINTR on a recvmsg. The restarted call instead resumes the deadline the interrupted attempt was using, which is what Linux's restart_block carries.

SOL_NETLINK is emulated under the same contract bind() gives -- the membership and the flags are recorded, and no multicast traffic is ever synthesized to deliver on them. NETLINK_ADD_MEMBERSHIP / NETLINK_DROP_MEMBERSHIP range-check the group and return 0; NETLINK_LIST_MEMBERSHIPS reads the resulting bitmap back; NETLINK_PKTINFO, NETLINK_BROADCAST_ERROR, NETLINK_NO_ENOBUFS, NETLINK_CAP_ACK, NETLINK_EXT_ACK and NETLINK_GET_STRICT_CHK set and read back as flags (a libnl or libmnl monitor sets several of these on the way up and treats a refusal as a broken socket); NETLINK_LISTEN_ALL_NSID wants CAP_NET_ADMIN and reports EPERM; the removed mmap ring options and anything else report ENOPROTOOPT.

Netlink socket state is keyed by guest fd number, and a guest fd number outlives its socket: fd_cleanup_entry() runs the netlink teardown after the number is already back in the fd table's free pool, so a concurrent socket() can be handed it while the previous slot is still live. Slots therefore carry an allocation generation -- lookups answer with the newest slot for a number and teardown retires the oldest -- which pairs each teardown with the socket that asked for it whatever order the threads arrive in.

Stack Alignment

The Linux initial stack must have SP 16-byte aligned and pointing directly at argc. Total 8-byte words on the structured area are 35 + extra + argc + envc, where:

• 35 covers the fixed scaffolding: 15 base auxv entries (30 words) + the AT_NULL auxv terminator (2 words) + the envp NULL terminator (1) + the argv NULL terminator (1) + argc itself (1) = 35.
• extra starts at 4 because AT_EXECFN and AT_BASE are always emitted (+2 words each). It adds another 2 for AT_SYSINFO_EHDR when a vDSO is present, and another 2 for AT_EXECFD when binfmt_misc passes one.
• argc and envc are the user-provided argument and environment counts.

If that total is odd, one padding word is pushed before auxv. Padding goes above the structured area, never below. Post-push masking (sp &= ~15) breaks because it inserts a gap between SP and argc. See build_linux_stack() in src/core/stack.c for the full layout.

mmap Notes

Aligned file-backed MAP_SHARED (fixed or non-fixed) installs a real host mmap(MAP_FIXED|MAP_SHARED, fd) overlay onto the guest slab so the kernel page cache keeps the mapping coherent with the file (and with peer overlays). The slab is tracked as a sorted list of 2 MiB-aligned hvf_segment_t entries; each overlay request splits, unmaps, and re-mmaps the host file at the exact host VA, then re-maps the segment. Apple Silicon enforces 16 KiB host pages, so the gap-finder advances to the next host-page boundary after each allocation.

The snapshot pread emulation (zero first for pages beyond EOF, then overlay file content) is the fallback for the cases where the overlay path cannot be used: misaligned MAP_FIXED, MAP_PRIVATE file-backed mappings, and any time the host slab cannot accept a fresh overlay at the requested VA.

MAP_SHARED|MAP_ANONYMOUS is promoted before fork to a memfd-style overlay (mmap_fork_prepare_anon_shared_locked) and reattached in the child via SCM_RIGHTS (mmap_fork_restore_overlays), so cross-fork shared anonymous memory stays coherent. Both promotion and overlay are disabled for Rosetta because HVF caches host VA-to-PA at hv_vm_map time. Validation: tests/test-msync.c, tests/test-cross-fork-mapshared.c.

Threads And Futexes

Guest threads map 1:1 to host pthreads. Each guest thread owns one vCPU and shares the same guest address space. HVF supports multiple vCPUs per VM, each bound to the host thread that created it; multiple vCPUs share guest physical memory via hv_vm_map(). Up to MAX_THREADS = 64 guest threads are supported per VM.

Thread Table

src/runtime/thread.c and thread.h:

• thread_entry_t per thread holds the vCPU handle plus its explicit validity, host pthread, per-thread signal mask, clear_child_tid (for CLONE_CHILD_CLEARTID), and the thread's SP_EL1 exception stack.
• _Thread_local current_thread gives O(1) access from syscall handlers.
• Each thread receives a 4 KiB EL1 exception stack carved out of the shim data region.

Futex

src/runtime/futex.c implements:

• The classic ops: FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, FUTEX_CMP_REQUEUE, FUTEX_WAKE_OP.
• A subset of priority-inheritance ops: FUTEX_LOCK_PI, FUTEX_UNLOCK_PI, FUTEX_TRYLOCK_PI. Priority semantics are not actually inherited -- the ops behave as ordinary mutex acquire/release, which is enough for glibc and musl to make forward progress.
• futex_waitv (syscall 449) for batch waits across up to 128 futex addresses.
• Robust-list cleanup: set_robust_list / get_robust_list are wired up in src/syscall/syscall.c, and robust_list_walk() releases owned futexes on thread exit, marking them FUTEX_OWNER_DIED.

Wait queues live in a hash table keyed by guest virtual address, with a per-bucket mutex. Each waiter has its own condition variable for precise wakeup. Thread exit calls futex_wake_one() on clear_child_tid, which is how pthread_join() waits via the TID address.

Atomicity: the bucket mutex is held across the futex word read and the waiter enqueue, so the compare-and-wait is a single critical section.

Lock Map

Resource	Lock	File
mmap/brk allocators + page tables	mmap_lock (order 1)	src/syscall/mem.c
FD table	fd_lock (order 3)	src/syscall/fdtable.c
Special FDs (timerfd, eventfd, signalfd)	sfd_lock (order 5a)	src/syscall/fd.c
Thread table	pthread_mutex	src/runtime/thread.c
Futex wait queues	pthread_mutex (per bucket)	src/runtime/futex.c
FUSE (sessions, file/dir state)	global fuse_lock + per-session session->lock	src/syscall/fuse.c
Sysroot snapshot	pthread_mutex	src/syscall/proc-state.c
Synthetic USB tree (scratch dirs + device model)	usb_lock (leaf)	src/runtime/usb-sysfs.c
usbdevfs fd side table	usbdev_table_lock + per-entry lock	src/syscall/usbdev.c
usbdevfs event-thread startup (runloop + notify port)	usbdev_loop_lock (leaf)	src/syscall/usbdev.c

Lock ordering is documented inline in those files (mmap_lock is order 1, fd_lock is order 3, sfd_lock is order 5a) so callers that need multiple locks acquire them in the right sequence. See the lock-ordering comment block in src/syscall/internal.h for the authoritative list.

Page-table consistency is preserved by the mmap_lock plus TLB broadcasts via TLBI VMALLE1IS from any vCPU; hardware coherency is verified by tests/test-multi-vcpu.

Thread Group Exit

exit_group sets a global exit_group_requested flag, calls thread_for_each(thread_force_exit_cb) which invokes hv_vcpus_exit() on all worker vCPUs to break them out of hv_vcpu_run(), and joins worker threads with a timeout so CLEARTID cleanup can complete.

Not Implemented

• True priority inheritance for the PI futex ops. The ops are wired up but behave as ordinary mutexes; this matches what glibc and musl need for forward progress, not real RT scheduling.
• CPU affinity (sched_setaffinity) returns the all-CPUs mask.

Fork, Clone, And execve

macOS HVF allows only one VM per process, so process-style fork cannot clone the live VM. elfuse implements it through posix_spawn plus IPC state transfer:

1. Parent creates a socketpair(AF_UNIX, SOCK_STREAM).
2. Parent posix_spawns a new elfuse --fork-child <fd> process.
3. Parent serializes VM state over IPC (see paths below).
4. Child receives state, creates its own VM, restores registers directly into EL0 (bypassing the shim _start so callee-saved GPRs survive), and enters the vCPU loop with X0 = 0 (the child return from clone).
5. Before spawning, the parent allocates a namespace-wide guest PID and reserves both a local process-table slot and shared lifecycle entry. After IPC succeeds it commits the child's host PID, releases the child into guest code, and returns the child PID. Allocation or admission failure returns EAGAIN without allowing an untracked child to run.

Process Lifecycle And Guest PID 1

The first guest process in an elfuse invocation has guest PID 1 and therefore becomes the fallback parent for orphaned descendants when no living child subreaper is closer. elfuse does not insert a hidden init process and does not automatically discard an adopted child's exit status merely because its new parent is PID 1. As on Linux, the new parent must consume that status with a wait*() call, or explicitly select no-zombie semantics with SIGCHLD = SIG_IGN or SA_NOCLDWAIT.

The invocation-scoped lifecycle registry keeps the Linux wait-format terminal status (including signal and core-dump bits) and the exiting process's resource usage until the new parent consumes it. A parent already blocked in wait*() periodically imports newly adopted descendants; adoption of an already exited child also sends SIGCHLD to wake the adopter. Both the per-process wait table and the invocation-wide lifecycle registry grow geometrically with the actual fork-family population; the on-disk registry serializes only its live records. An empty newly created registry is initialized on first use; a nonempty record set that cannot be read or validated fails closed instead of being overwritten as empty. Fork admission is reserved before the host helper starts, so allocation or registry failure fails the new fork instead of silently dropping a waitable child. Pre-spawn registry reservations are not imported as adopted children, and a matching local reserved slot remains authoritative through the registry-publish/local-commit window.

An application runtime used directly as guest PID 1 may not perform that reaper role. In that case, adopted terminal statuses remain in elfuse's invocation-scoped lifecycle registry, and a direct macOS child of the PID 1 elfuse process may also remain a host zombie until the guest waits, changes its SIGCHLD disposition, or exits. Guest waitability and host-process cleanup are separate concerns: a future host-only reaper could collect the macOS process while retaining its exit status for a later guest wait, but no such background reaper is currently provided.

CoW Fork Path

When g->shm_fd >= 0 the guest memory is file-backed (mkstemp + unlink, MAP_SHARED). Fork takes an APFS fclonefileat snapshot of the backing file and sends the snapshot fd over SCM_RIGHTS:

• The snapshot decouples the parent and child: subsequent parent writes cannot be observed by the child even before the child remaps. APFS copy-on-write keeps the snapshot cheap.
• Parent stays on MAP_SHARED and does NOT remap -- HVF caches the host VA->PA mapping from hv_vm_map, and a MAP_FIXED remap does not update Stage-2, so a remapping parent would observe stale pages.
• Child maps the snapshot fd MAP_PRIVATE, producing an instant CoW clone with zero data copy.
• The IPC header sets has_shm = 1 and num_regions = 0, skipping memory serialization entirely.
• Child calls guest_init_from_shm() instead of guest_init(), and must restore g->ttbr0 from the IPC header -- guest_init_from_shm zeroes the struct, and without ttbr0 page-table walks fail for all high VAs.
• MAP_SHARED|MAP_ANONYMOUS regions are promoted to a memfd-style overlay before fork (mmap_fork_prepare_anon_shared_locked) and reattached in the child via SCM_RIGHTS (mmap_fork_restore_overlays) so cross-fork shared-memory coherence is preserved.

This path is roughly 50x faster than the legacy IPC copy path on large guest memories.

The CoW path is disabled when hosting Rosetta because HVF caches the host VA->PA mapping from hv_vm_map, and Rosetta's translated code touches the parent's slab in ways the snapshot model cannot intercept. Rosetta forks fall back to the legacy IPC copy path.

macOS rejects MAP_PRIVATE on shm_open fds (EINVAL), so the backing file is created via mkstemp + unlink.

Legacy IPC Copy Path

When g->shm_fd < 0, the parent serializes used memory regions in 1 MiB chunks over the socket pair, the child calls guest_init() and receives the data into fresh guest memory. CLOEXEC semantics follow POSIX: all FDs (including those marked CLOEXEC) are inherited across fork. CLOEXEC takes effect at execve (see src/syscall/exec.c step 4).

clone(CLONE_VM)

sys_clone_vm() in src/runtime/forkipc.c handles CLONE_VM without CLONE_THREAD. Unlike sys_clone_thread(), VM-clone children are waitable via wait4 and have exit semantics (exit_signal, vm_exit_status). Unlike the posix_spawn fork path, they share the same guest_t*, the same guest memory, and the same page tables.

clone(CLONE_THREAD)

In src/runtime/forkipc.c:

1. hv_vcpu_create() and per-thread SP_EL1 allocation.
2. Set the child SP, TPIDR_EL0 (TLS), and copy the parent's signal mask.
3. pthread_create() running vcpu_run_loop() for the child vCPU.
4. Return the child TID to the parent; the child runs with X0 = 0.

execve

sys_execve in src/syscall/exec.c reloads the ELF, resolves the dynamic interpreter for dynamically-linked targets through path_translate_at() like any other guest path, rebuilds page tables, and restarts the vCPU. Signal handlers are reset to SIG_DFL per POSIX: SIG_IGN stays SIG_IGN, and pending and blocked masks are preserved. This happens in signal_reset_for_exec() after guest_reset.

Signals

Signals are fully implemented in src/syscall/signal.c. The signal frame matches Linux arch/arm64/kernel/signal.c:setup_rt_frame() so the C library's __restore_rt → rt_sigreturn (syscall 139) restores state correctly. Both musl and glibc use this mechanism.

Key points:

• signal_deliver() builds linux_rt_sigframe_t on the guest stack, redirects the vCPU PC to the handler, and sets X0 = signum, X30 = sa_restorer.
• signal_rt_sigreturn() restores all 31 GPRs, SP, PC, and PSTATE from the frame.
• SIGPIPE is queued automatically when write, writev, or pwrite64 returns EPIPE.
• Guest ITIMER_REAL is emulated internally rather than being forwarded to host setitimer, because macOS shares alarm() and setitimer(ITIMER_REAL) as a single timer, and elfuse already needs alarm() for its per-iteration vCPU watchdog.
• signal_check_timer() is called from the vCPU loop after each syscall.
• After SYSCALL_EXEC_HAPPENED, the vCPU loop verifies that ELR_EL1 is non-zero -- a defensive check against HVF register-sync bugs.
• Each thread_entry_t carries its own blocked mask. rt_sigprocmask operates on current_thread->blocked, and child threads inherit the parent's mask at clone time.

Ptrace And clone(CLONE_VM) Tracing

clone(CLONE_VM) produces an inferior that shares guest memory; the tracer attaches via PTRACE_SEIZE. BRK instructions trigger ptrace-stops, after which the tracer reads and writes registers through the snapshot protocol.

Operations

In src/syscall/proc.c:

• PTRACE_SEIZE -- attach without stopping; sets ptraced = 1.
• PTRACE_CONT -- resume the stopped tracee, optionally injecting a signal.
• PTRACE_INTERRUPT -- force the tracee into ptrace-stop via hv_vcpus_exit(). A stop taken on a syscall return whose tail restores the saved SVC frame goes through HVC #13, so ptrace snapshots the architectural GPR set rather than shim scratch. The tails that rebuild EL0 state instead (X8 = 2) already hold that set live, bar the X8 the shim reloads from the saved frame, so the host stops on them directly; an execve re-entry leaves the stop owed for the new image's first syscall.
• PTRACE_GETREGSET / PTRACE_SETREGSET (NT_PRSTATUS) -- read or write the tracee's register snapshot. Writes are applied on resume.

Snapshot Protocol

Cross-thread HVF register access is unreliable, so every actor uses a snapshot:

1. Tracee snapshots its own vCPU registers into ptrace_regs before entering ptrace-stop.
2. ptrace_stopped = 1 and the tracer is signaled via ptrace_cond.
3. Tracer reads and writes the snapshot via GETREGSET / SETREGSET.
4. Tracer issues PTRACE_CONT; the tracee clears ptrace_stopped and is signaled via resume_cond.
5. Tracee applies any dirty register changes back to the vCPU and resumes.

BRK Flow

1. Guest executes BRK → shim forwards via HVC #10.
2. If current_thread->ptraced, the tracee calls thread_ptrace_stop(SIGTRAP).
3. Snapshot, broadcast, wait, resume as above.
4. wait4 returns WIFSTOPPED(status) with WSTOPSIG == SIGTRAP to the tracer.

wait4 Integration

sys_wait4() first checks for ptraced or VM-clone children via thread_ptrace_wait() before falling through to the process table for regular fork children, so ptrace-stops and VM-exit states are reported without racing the regular wait path.

Guest-Internal FUSE

src/syscall/fuse.c implements /dev/fuse, mount(..., "fuse", ...), and the minimal VFS dispatch entirely inside the guest VM. Guest libfuse programs (sshfs, ntfs-3g, AppImage runtimes) run without macFUSE, FUSE-T, or FSKit on the host.

Key shape:

• A global fuse_lock plus per-session locks. Sessions are refcounted, so in-flight reads and writes pin the lock against daemon exit.
• Per-fd alias bindings for dup, dup3, and fcntl(F_DUPFD).
• Per-file io_in_progress plus io_cond serializes read and getdents64 against the offset field (matching Linux f_pos_lock). lseek waits on io_in_progress; pread skips it.
• Synchronous FUSE_INIT in sys_mount; negative hdr.error propagates back. Mount tombstones on daemon death and surfaces -ENOTCONN.
• O_PATH support, NAME_MAX-bounded getdents64, 8 MiB FUSE_FRAME_CAP on daemon writes, and procfs integration (/proc/{mounts,filesystems,self/mountinfo}).
• fuse_materialize_path so execve can load FUSE-backed binaries.
• The wait path honors SA_RESTART and ignored signals via the signal_pending_interruption(restart_out) helper.

Validation lives in make test-fuse-alpine, which exercises /dev/fuse plus mount("fuse") against the staged Alpine musl sysroot fixture.

USB Device Passthrough

src/syscall/usbdev.c implements the usbdevfs character device (/dev/bus/usb/BBB/DDD) on top of IOKit's IOUSBDeviceInterface650 and IOUSBInterfaceInterface800 plugin interfaces, so Linux USB tools drive real devices attached to the Mac. macOS arbitrates that access per IOKit object and dynamically: USBInterfaceOpen refuses an interface exactly while some driver holds that interface open, and grants it while that driver is idle. An interface bound to an idle Apple class driver therefore does open -- the ESP32-S3's CDC data interface opens whenever nothing holds /dev/cu.usbmodem1101 and refuses while something does -- so the layer attempts the open and maps what IOKit answers rather than deciding from the presence of a bound driver.

The fd model: opening a node constructs a typed FD_USBDEV fd. The open consults the model, not the hardware: stat, access and an O_PATH open of the same name are all answered from the model, so requiring a live IOKit service here would make a plain O_RDONLY the one entry point that reported ENODEV for a node the other three describe. The service is resolved on first use instead, and every operation that needs the wire reports ENODEV when it is not there.

Per-fd state (the IOKit device handle, claimed interfaces, and the endpoint-to-pipe map) lives in a side table keyed by the guest fd and the fd generation, so a concurrently closed and reallocated guest fd cannot reach another open's state. A lookup pins its entry under usbdev_table_lock and then takes the per-entry lock with the table lock released: taking the two nested meant a thread blocked on one fd's transfer held the table lock on every other thread's behalf, and one transfer with usbdevfs's documented "unlimited" timeout == 0 wedged every usbdevfs fd in the process, close() included. Teardown runs from the fd-cleanup hook, which is handed a bare fd number after the fd-table slot is already free, so a sibling thread's open can already hold the same number; among the entries that answer to it the closing one is the one with the smaller generation, since fd_alloc stamps a globally monotonic counter. fd_alloc also publishes the guest fd before the side table can bind it, so a close landing in that window would find no entry to tear down; the open rereads the fd's generation once its entry is findable and retires the entry itself if the close has already been and gone.

That leaves two windows around the bind, and the entry keeps one identity across both. Before the bind its guest_fd is still -1, so a close finds nothing; after it, the entry is findable and therefore also freeable, so a close can reap it, the last unref can free the slot, and a sibling open can bind its own live fd there before the recheck runs. The generation is the one fd_alloc_from reports from inside the allocating fd_lock section, not a later read of the slot -- read back after the slot was publishable it is whatever the number carries by then, which in the first window is a reopening thread's stamp, and two entries then hold the same pair. And the retire path claims the whole tuple the caller allocated (used and alive, this fd number, this generation) rather than testing that the slot is not dead, which in the second window claims the sibling's entry instead. The tuple is sufficient because slot storage is static, so the read is always defined; all four fields are written and read under usbdev_table_lock, so they are read as one value; and fd_next_generation is globally monotonic, so each entry carries its own allocation's stamp and no two live entries can present the same pair.

The entry's host fd is a pipe read end reserved for readiness signaling. read() serves the descriptors blob at a per-open file position, byte-identical to the sysfs descriptors attribute; SEEK_END is EINVAL, as on Linux usbfs; fstat reports a character device, major 189, answered from the entry's own identity ahead of the /dev/bus path stamp, the way a FUSE descriptor is answered by the layer that owns it. The whole write family -- write, writev, pwrite, pwritev -- is EBADF without FMODE_WRITE and EINVAL otherwise, the order vfs_write checks FMODE_WRITE and then FMODE_CAN_WRITE; none of them may fall through to the readiness pipe behind the fd. The two capability bits are derived once, the way OPEN_FMODE derives them -- (flags + 1) & O_ACCMODE, not a comparison against O_RDONLY -- and the four gates that need them (read, pread, the write family, and the ioctl surface) read that. Access mode 3 is the case the comparison gets wrong: open(2) takes it and ACC_MODE(3) asks this 0666 node for read plus write, which it grants, so Linux hands back a descriptor carrying neither FMODE_READ nor FMODE_WRITE and refuses everything on it. dup of the fd is refused with EBADF (the side table is keyed by the guest fd and IOKit plugin handles are process-local).

The ioctl surface at this layer is whatever usbdev_ioctl dispatches, and it is written down in exactly one place a reader can check: tests/usbdev-ioctl-departed.tbl carries a row per request and scripts/gen-usbdev-ioctl-departed.py fails when the table and the dispatch disagree. Two of those requests are the synchronous CONTROL and BULK transfers, which bounce guest data through host buffers around DeviceRequestTO and ReadPipeTO / WritePipeTO; they are synchronous in the guest too, in that the guest thread blocks in the ioctl for the duration of the request.

errno fidelity is the design rule, because libusb and friends branch on exact values, and so is the order the answers are decided in, which is just as observable: check_ctrlrecip runs before the wLength cap and findintfep before the transfer-length checks, so a request naming an endpoint or interface the device does not have is ENOENT however long it is. The reserved-bit test on an endpoint address lives in the one lookup BULK, CLEAR_HALT, RESETEP and the control endpoint recipient all go through, so the four cannot disagree about it. The interface-number bound is 64, 8 * sizeof(unsigned long) as in claimintf; between 64 and 32 an interface is merely absent, which is a question about the device.

Every argument that comes off the wire is bounded where it enters, not only where it is used. bInterfaceNumber is a device-supplied byte with the whole 0..255 range behind it, and the endpoint-owner lookup reports it back as the EINVAL checkintf reports rather than handing a 64-entry array an index of 200. An endpoint address arrives as a 32-bit word and is tested as one, since narrowing it to a byte first resolves the transfer to an endpoint the caller did not name; the control path is the exception Linux itself makes, masking wIndex to its low byte before the lookup. An altsetting above 255 matches no bAlternateSetting and is EINVAL, after the implicit claim, the order proc_setintf decides them in.

The other half of errno fidelity is that a failure keeps the errno of whatever failed. ENODEV from the model, meaning nothing answers to this address, becomes the ENOENT open(2) owes for a name with nothing behind it; every other failure on the open path -- an allocation, a scratch-tree mkdir, a pipe the host would not give -- reports itself, so a host out of memory is not described to the guest as a missing node.

Every ioctl needs FMODE_WRITE and is EPERM without it (Linux's gate in devio.c). FIONBIO and FIOASYNC are the exception, because they are not part of this file's ioctl set at all: do_vfs_ioctl answers both for every file before it reaches f_op->unlocked_ioctl, so they never meet that gate. They are answered where FIOCLEX and FIONCLEX already are. FIONBIO edits only the guest flag word, the way F_SETFL does -- the readiness pipe behind the fd stays nonblocking whatever the guest asks -- and always reports 0. FIOASYNC reports 0 when the requested FASYNC state already holds and ENOTTY when it would change, because usbdev_file_operations declares no .fasync.

Transfer memory is one allowance for everything in flight, not a per-call size cap: Linux charges against a module-global usbfs_memory_mb (16 MB) and refunds when the transfer settles, so a request of exactly the allowance never fits and concurrent transfers across different fds contend for the same total. What is charged depends on the path and not on the caller: do_proc_bulk and proc_do_submiturb book len + sizeof(struct urb), while do_proc_control books a fixed PAGE_SIZE + sizeof(struct urb) + sizeof(struct usb_ctrlrequest) whatever wLength says, because the kernel bounces every control transfer through one whole page. An atomic counter carries all three here, charged where Linux charges each and refunded on every exit including the error arms. CONTROL used to be outside it, which is the one way a guest could keep transferring after the allowance was full. A claim that IOKit answers kIOReturnExclusiveAccess is EBUSY, what Linux reports for an interface held by a kernel driver. GETDRIVER names the bound Apple driver, or usbfs for an interface any usbfs fd on this device holds, or reports ENODATA; a user-client child is not a driver, so another usbfs consumer's USBInterfaceOpen is not reported as one. It and the three claim/connect ops reach the interface through one CreateInterfaceIterator, and SETCONFIGURATION opens that same enumeration to ask whether any interface has a host driver bound. The usbfs answer is the one that has to be made to reach it as well: a claim is this layer's own bookkeeping, which knows nothing about the device and stays true after it leaves, so each of those ops opens the enumeration for the device question before the claim answers rather than instead of it. A device-gone answer from the call is about the device, so each reports ENODEV and stamps the fd, rather than reporting the interface missing from a device that is missing entirely or -- for SETCONFIGURATION, which is asking about drivers and not about one interface -- reading an enumeration that never ran as "no driver is bound" and changing the configuration of a device that had departed. Linux gets the same answer one level up, from the connected() gate usbdev_do_ioctl runs before any of them. Transfer errors map per devio.c: a stall is EPIPE, a timeout ETIMEDOUT, a vanished device ENODEV. kIOReturnAborted maps to EINTR with syscall_restart_forbid(), because the transfer was already on the wire and a dispatcher restart would send it twice.

Known gaps at this stage, each printed as an XFAIL by tests/test-usbdev-ioctl.c rather than only written down here:

• GET_CAPABILITIES reports ZERO_PACKET | REAP_AFTER_DISCONNECT, the two the URB engine below honors. BULK_CONTINUATION stays clear because the flag is accepted without its error-cascade unlink, and the two remaining bits describe URB splitting IOKit does not expose.
• Two ioctls on one fd serialize, because the entry lock is held across the blocking transfer. Linux drops the device lock around the URB wait.
• GETDRIVER reports the IOKit class name (AppleUSBACMControl) where Linux reports the driver's name (cdc_acm), and DISCONNECT_CLAIM's name filters compare against it.
• RESET kills the device's URBs and clears the claimed pipes' stalls instead of re-enumerating the port, which would destroy the handles. The URB kill is the half usb_reset_device does have; the re-enumeration is the half it does not get. A per-pipe stall clear that fails is logged, not returned: the clears are the substitute, and refusing a RESET Linux performs would be the worse answer.
• CLEAR_HALT and RESETEP cancel whatever is outstanding on the endpoint, so a queued async URB there reaps ECONNRESET. Linux's check_reset_of_active_ep only warns and leaves the queue alone, and IOKit exposes no stall clear that does not abort the pipe. libusb calls libusb_clear_halt between transfers, so this is reachable in ordinary use.
• The per-endpoint abort shutter (ep_aborting) is an invariant of the paths that abort deliberately, not of the slot: DISCARDURB and every wholesale kill raise it, CLEAR_HALT and RESETEP do not, because ClearPipeStallBothEnds aborts the pipe as a side effect and IOKit exposes no variant that does not. A queued follower can therefore be started behind a stall clear's abort that is still in flight. Linux has no such shutter to keep; the guest-visible half of the gap is the ECONNRESET row above.
• A printer's GET_DEVICE_ID names its interface in the high byte of wIndex and an alt setting in the low one, and check_ctrlrecip lets that one request through untouched: it returns before the index &= 0xff when the request type is 0xa1, the request is 0, and usb_find_alt_setting(actconfig, index >> 8, index & 0xff) has class USB_CLASS_PRINTER. Both control paths here read wIndex & 0xff as the interface number for every non-vendor interface recipient, so on a printer such a request implicitly claims the alt setting's number instead of the interface's. Demonstrating it needs a printer attached, so it is recorded rather than modeled.
• A URB's signr, like DISCSIGNAL's, is accepted, reported as success and never delivered: elfuse has no async guest-signal injection from the event thread.
• DISCARDURB waits for IOKit's abort to settle, so a REAPURBNDELAY issued straight afterwards finds the URB the way it does on Linux, but the wait is bounded at 2 s rather than unbounded. The kernel call this matches is the one proc_unlinkurb actually makes, usb_kill_urb, which is synchronous ("upon return all completion handlers will have finished") -- not usb_unlink_urb, the asynchronous unlink, which proc_unlinkurb does not call. Waiting is therefore the faithful half and the 2 s ceiling is the divergence: Linux never gives up, so libusb_cancel_transfer on a wedged endpoint blocks there for as long as the transfer lives, where here it returns 0 after 2 s with the record still flagged and still reapable when its completion arrives. An abort that is refused outright is a different answer and gets one: kIOReturnNoDevice from AbortPipe or USBDeviceAbortPipeZero means the user client has no device behind it, so it aborted nothing and no completion is coming. Both used to be cast to void, which spent the whole 2 s waiting for a callback that could not arrive, returned 0 to the guest with the fd unstamped, and left the URB in the pending list where the next REAPURB waited on it for ever. It now stamps the fd, the way every other op's device-gone answer does, and hands the record back as an orphan -- reapable at once, with its buffer and handles pinned until a callback that may never come.
• SETINTERFACE and SETCONFIGURATION report EBUSY when that same 2 s drain expires with transfers still outstanding. Linux cannot reach this: usb_kill_urb does not give up, so the kill always finishes and the change always proceeds. Both retire the handles the survivors still reference -- the interface's pipeRefs, and the whole pipe table -- so the two decide it once, in usbdev_drain_for_change, rather than each on its own. A drain that expires settles the slot's own counters, unlinks the survivors, hands each of them back on the completed list with the ENOENT usb_kill_urb leaves, pins the handle it still references -- the interface's, or the device's for an ep0 record, which names no interface -- and then restarts every endpoint FIFO exactly as the successful path does: draining shut all of them, including the ones the kill did not match, and an endpoint whose leader completed inside the window would otherwise be left with a queued URB, none in flight and nothing to restart it. The process-wide byte budget is not settled there -- it accounts live memory, and a survivor's buffer is still owned by an in-flight IOKit transfer, so its charge is given back where the buffer is freed, in the late callback. USBDEVFS_RESET deliberately proceeds instead: a wedged transfer is the state a guest issues RESET to escape, and a stall clear leaves the pipeRefs where they are.
• A short bulk OUT is EIO: WritePipeTO reports no length, and reporting the requested count would spell a partial write as a complete one.
• The side table holds 32 open fds per process and reports ENOMEM past that; Linux allocates a usb_dev_state per open and has no such limit.
• USBDEVFS_IOCTL DISCONNECT of an interface another usbfs fd holds is EBUSY; Linux releases that claim and answers 0.

The URB Engine

SUBMITURB validates the URB type and flags the way devio.c does, copies the guest buffer into a host bounce buffer on the vCPU thread, and queues the URB per endpoint. At most one URB per endpoint is in flight at IOKit at a time; later submissions queue inside elfuse and are started from the completion callback. The queue exists because cancellation granularity differs: DISCARDURB must kill exactly one URB, but IOKit's AbortPipe (USBDeviceAbortPipeZero for ep0) aborts every outstanding transfer on the pipe.

"Per endpoint" is a list and not a filter. Each pending record sits in the fd-wide pending list and in its endpoint's own intrusive FIFO at once, and that FIFO's head is the record in flight there when the endpoint has one and the next to start otherwise -- so append, unlink and restart are each O(1). Reading the same three answers out of the fd-wide list was not: every completion walked it twice, once to unlink and once to find its endpoint's next record, and every submit walked it a third time to decide whether the endpoint was busy, all under async_lock on the one event thread that carries every fd's completions. Measured on the loopback fixture as the median of 31 submit-to-reap round trips on one endpoint against a second endpoint's pending depth, before: 0.017 ms at depth 0, 0.334 at 20k, 1.848 at 80k and 2.688 at 100k. After: 0.016, 0.025, 0.020 and 0.021 ms. Linux is O(1) both ways (list_move_tail, devio.c:634). The fd-wide list stays, because five callers genuinely need every record and not one endpoint's: the wholesale abort, the drain's still-busy scan, the drain timeout's orphaning, DISCARDURB's lookup by user pointer, and the reap's "is anything still out" test. The three extra link words take the record from 136 to 160 bytes, which the process-wide byte budget pays for: 104857 zero-length URBs now fit in the 16 MB allowance where 123361 did.

One URB on the wire is not by itself enough. DISCARDURB drops async_lock to issue the abort, and in that window the target can complete normally and the completion callback can start the queued follower, which the abort then cancels instead -- measured against the attached board at a handful per hundred thousand at natural rates, and reproducibly there with the window widened, as the discarded URB reaping success and its innocent successor reaping ECONNRESET. The rate is the board's: the loopback fixture retargets an aborted transfer's timer under its own lock and cannot start a follower into an abort that is still running, so it shows this at no rate at all. The endpoint's FIFO therefore stays shut for the duration of the abort, and DISCARDURB waits for the record to leave the pending list before it returns. With both, a discarded URB reaps ENOENT and any other abort reaps ECONNRESET, mirroring usb_kill_urb against an async unlink.

Two caps that used to be elfuse's own are gone. The 16 MB budget is one process-wide byte count, the way usbfs_memory_usage is one kernel-wide static, and there is no URB-count cap: a 257th eight-byte URB on one fd used to be ENOMEM against a 16 MB budget, which is exactly the ring depth libusb's async API builds for bulk streaming. A URB's record is charged alongside its buffer, so zero-length URBs are bounded too.

Completions land on one lazily-started host thread driving a CFRunLoop -- the first CFRunLoop in the codebase -- fed by CreateDeviceAsyncEventSource / CreateInterfaceAsyncEventSource, which is libusb's own darwin model. The teardown discipline is inherited from netlink's blocking-recv rule (netlink.c): no host thread ever touches guest memory. The callback moves only usbdev-owned host memory (fd slots, URB records, the completion pipe); copy-in at submit and copy-out at reap both run on the vCPU thread. That is what makes the thread safe across execve and guest teardown, so it is started once and never joined.

REAPURB blocks in io_wait_fd_or_interrupted on the completion pipe and copies the result out on the vCPU thread. What that pipe carries is a level, not a running count: it holds exactly one byte while the fd has a completion to hand back or has been disconnected, and none otherwise, restored under async_lock by every path that can move either term. A byte per completion would not survive the absence of a URB-count cap -- a zero-length URB costs only its record against the 16 MB budget, so far more records fit than the pipe has room for bytes, and a byte that does not fit would leave its record reapable with nothing readable behind it. The actual_length it reports is clamped to the buffer the guest submitted, the way every Linux HCD bounds urb->actual_length by transfer_buffer_length; IOKit's transferred count is a device-supplied number and is not passed through. A signal interrupts it with EINTR and syscall_restart_forbid(), because Linux's reap path does not restart. REAPURBNDELAY reports EAGAIN when nothing has completed. ZERO_PACKET on a maxpacket-multiple OUT emits the trailing zero-length write from the completion callback, and SHORT_NOT_OK turns a short IN into EREMOTEIO at completion.

Poll Semantics

usbfs readiness is inverted relative to a pipe: POLLOUT means "completed URBs are reapable", and POLLIN is never signaled. The completion pipe's read end raises host POLLIN, so ppoll and pselect6 remap it to the guest-visible POLLOUT | POLLWRNORM, and epoll registers EVFILT_READ on the pipe but reports EPOLLOUT, through a per-registration flag. The host interest is always armed, whatever events the guest asked for: a disconnected device raises the unmaskable POLLERR | POLLHUP, and the pipe is the only wake source that reaches a read-only waiter. Disconnect raises the readiness level and nothing lowers it again, which is what a sticky POLLERR | POLLHUP needs. Because the level is one token rather than a count, an EPOLLET registration is edged once per rise rather than once per completion. A consumer that reaps to EAGAIN sees no difference, which is the contract EPOLLET states and the one tests/test-epoll-edge.c pins; one that reaps a single URB per wake and re-arms would wait here where Linux, whose usbfs wakes the wait queue once per completion, fires again. Stated rather than tested: reaching it needs a guest that breaks the EPOLLET contract. A wake that maps to nothing guest-visible re-blocks: the woken entry's host interest is withdrawn for the rest of the call (unreaped completions keep the pipe readable, so leaving it armed would busy-spin), the wait resumes in bounded slices that re-check the lock-free disconnect map, and epoll additionally re-adds a fired EV_ONESHOT knote disabled so a wake the guest never saw cannot consume an EPOLLONESHOT arm. epoll_wait therefore never returns 0 before its timeout, matching Linux ep_poll. Undoing those mutes on the way out reads the registration and applies the knote change in one step under the instance lock: with the lock dropped in between, a concurrent EPOLL_CTL_MOD could re-arm the registration after the delete had been decided on, and the EV_DELETE then retired the knote that MOD had just installed, leaving an active registration with nothing behind it.

EPOLLPRI is not defined in this layer and nothing below produces the conditions Linux raises it for -- socket out-of-band data, a sysfs attribute poke -- so it is left unmodeled rather than half-wired onto EVFILT_EXCEPT, which would be a guess about which of the two the guest meant. A registration naming EPOLLPRI alone arms no filter and so hears nothing at all, the otherwise unmaskable EPOLLERR | EPOLLHUP included, since both reach the guest only through a knote that fired.

Both readiness masks are read as pairs, on every registration and not only on a usbfs one. eventpoll has no mask of its own: ep_item_poll masks the file's answer by epi->event.events, so a registration naming only EPOLLRDNORM or only EPOLLWRNORM is a legal registration Linux both wakes and reports back in the spelling it was asked in. Every readiness source below raises the bit alongside its EPOLLIN or EPOLLOUT twin -- a pipe, a socket and a tty on the read side, and on the write side a pipe with room, a writable TCP socket, and a unix or datagram socket, which add EPOLLWRBAND as well -- so this layer arms EVFILT_READ for either read bit and EVFILT_WRITE for either write bit, at ADD, at MOD and at DEL, and reports back the intersection rather than the canonical spelling. tests/test-epoll.c pins both halves. Two asymmetries are deliberate. Linux EPOLLWRBAND (0x200) is in neither pair, and neither is its poll twin: POLL_WRITE_EVENTS is POLLOUT | POLLWRBAND, but that token is the macOS POLLWRBAND, which is 0x100 -- the value Linux spells POLLWRNORM -- so the two halves arm on the same two Linux bits, 0x104, and Linux POLLWRBAND is absent from both. And a registration naming no read bit at all still has its read filter armed for EPOLLRDHUP, so that one arm reports the EPOLLIN it always did rather than an entry with no events in it. A usbfs fd is unaffected by either: its read filter is armed whatever the mask says, and usbdev_poll raises neither read bit ever.

Disconnect And Fork

An IOServiceAddInterestNotification on the runloop (or kIOReturnNoDevice / kIOReturnNotAttached from any op -- every op translates its IOKit status through usbdev_ioret_op, the synchronous transfers and the setup calls included, so an op that sees either of those two codes originates the disconnect rather than only reporting it) marks every usbfs fd open on that device disconnected, not only the one that noticed: poll reports POLLERR | POLLHUP, REAPURB hands back every URB and then reports ENODEV, and once an fd carries that mark every other usbdevfs ioctl on it reports ENODEV too -- the usbfs disconnect contract. Only usbdevfs: the requests do_vfs_ioctl answers before f_op->unlocked_ioctl never reach this layer's gate on Linux and do not here either, so a marked fd still answers FIONBIO 0 and FIOQSIZE ENOTTY. Not meeting the gate is not the same as matching Linux, though: this layer models none of those requests and answers ENOTTY to all ten, where Linux agrees on two and answers from the superblock, from CAP_SYS_ADMIN or from the argument for the other eight. check_vfs_ioctls in tests/test-usbdev-ioctl.c carries both values for each, drives them on a read-only and on a writable fd, and prints the eight as XFAIL. The scope is the mark, not the device: an fd that has not been told yet answers per request, and which requests answer what is tests/usbdev-ioctl-departed.tbl rather than a sentence here. tests/test-usbdev-ioctl-departed.c drives both halves, the marked fd's whole usbdevfs surface and the requests beside it that are not usbdevfs included. The cross-fd half of it is what usbdev_remove does by walking udev->filelist, and it has to be a walk here for the same reason: a device is gone for every consumer of it at once, and a second fd on the node has nothing of its own that would find out. usbdev_ioret_device_gone, the predicate that decides origination, is deliberately narrower than the ENODEV row of ioret_neg_errno, which also carries kIOReturnNotOpen. That code is what IOKit answers for a handle nobody has opened yet, so a control request that arrives before the lazy USBDeviceOpen draws it from a device that is plainly still attached: it answers ENODEV, the way Linux answers for a device it cannot reach, and it does not stamp. Widening the predicate to cover it would mark a live fd gone on its first control request.

"Every URB" is what CAP_REAP_AFTER_DISCONNECT promises, stated as one invariant: after a disconnect every URB still in flight comes back before any reap answers ENODEV, whichever reap flavor asked first. usbdev_remove delivers it for free by running destroy_all_async -- a synchronous usb_kill_urb each -- before it wakes the reapers, so a Linux reap cannot observe the disconnect until the pending list is already empty. IOKit completes nothing of its own when a device terminates, so the first reap that finds the completion list empty on a disconnected fd issues that kill itself, and its aborts land asynchronously. The window Linux does not have -- disconnected, aborts issued, URBs not back yet -- is real here, so ENODEV is decided by the pending list being empty and never by a flag saying some earlier pass did the work. A blocking REAPURB waits for the kill to settle, latch or no latch. REAPURBNDELAY issues the same aborts, does not wait for them, and answers EAGAIN -- proc_reapurbnonblock's other arm -- until they land: a non-blocking ioctl that sat out the two-second drain deadline holding the fd's async_lock against every SUBMITURB and DISCARDURB was the worse half of the contract, but so was answering ENODEV with the URBs still in flight, which is what a one-shot flag doing both jobs did to every reap behind it.

Both flavors reach the end of that conversation, and reach it inside one drain deadline of the aborts. A wire can answer no abort at all, and then what empties the pending list is the deadline: the blocking reap spends it in the kill's wait and orphans the survivors on the way out, and the non-blocking one measures the same deadline instead of waiting it out and orphans them itself. Without that second half only the blocking flavor had a ceiling, so a libusb_handle_events_timeout(0) loop -- non-blocking reaps and nothing else, the common shape -- got EAGAIN for ever against a wedged endpoint while poll held POLLERR|POLLHUP, where Linux answers connected(ps) ? -EAGAIN : -ENODEV and the application tears down. What the deadline gives up on is the wire answering, not the URB: the record is handed back on the same pass that gives up on it, carrying the ENOENT destroy_all_async's usb_kill_urb leaves, and stays alive behind that hand-back until IOKit's callback arrives -- whichever of the reap and the callback comes second frees it. That is the last place CAP_REAP_AFTER_DISCONNECT did not hold: the deadline used to unlink the record, mark it and drop it, so the pass that gave up answered ENODEV with the URB pointer never returned.

The refcon that carries the disconnect notification packs the slot index in a field sized from the slot table, not in a hand-written four bits: with 32 slots and four bits, slot 16+k decoded as slot k, so half the table never saw a disconnect and the other half could be marked gone while attached. Across fork the fd is dropped and the child sees EBADF, like FD_NETLINK and FD_INOTIFY today: IOKit plugin handles are Mach-port-backed and cannot cross the posix_spawn that implements fork.

Testing The Engine Without Hardware

IOKit publishes no loopback device, so the async engine had no in-tree lane at all: ELFUSE_USB_FIXTURE's devices have no IOKit service behind them and stop at SUBMITURB's argument gate. ELFUSE_USB_FIXTURE=loopback adds one that does, in the USB_LOOPBACK_FIXTURE=1 build that links the model rather than the stub (below), by substituting at the narrowest place that leaves every layer above it real: the two COM vtables. Every wire call in usbdev.c goes through IOUSBDeviceInterface650 ** or IOUSBInterfaceInterface800 ** as (*h)->Method(h, ...), so src/syscall/usbdev-fixture.c hands back an object whose first member is a vtable of the same shape and nothing above it changes. The URB records, the per-endpoint FIFO, the completion callback, urb_status, the ZLP predicate, the readiness and disconnect maps, REAPURB, the drain and all of poll.c are the same code that runs against a board. Completions arrive from a one-shot CFRunLoopTimer on the event thread, which is where IODispatchCalloutFromCFMessage would have delivered them.

What the fixture does is a script rather than a flag: ELFUSE_USB_LOOPBACK=ep02:delay(80),ok;ep81:short(8) and the rest of the vocabulary in src/syscall/usbdev-fixture.c name the IOReturn each outcome stands for, and a guest can rewrite the script, read back a log of what crossed the seam and terminate the device through vendor control requests. The log is what makes the ZERO_PACKET trailing packet observable rather than inferred.

The seam is five if (u->fake) branches, one has-device probe and one bind call in usbdev.c, all behind a mode resolved once per process, and the flag is set only for the one location the fixture models -- so the other fixture modes, and every real device, take the paths they took before. make test-usbdev-ioctl-loopback is the standing check on that: the fd-contract lane must answer the same thing with the loopback device present as without it.

A device that can be made to leave on demand is what the third loopback lane needs. make test-usbdev-ioctl-departed terminates the fixture's device once and then drives every usbdevfs ioctl the layer implements against it, asserting four things per request: the ioctl(2) return, the errno behind it, the poll revents left on the fd that asked, and the revents on another fd open on the same node, which is what says the disconnect was recorded against the device rather than against one caller. The surface is not a list anyone maintains: scripts/gen-usbdev-ioctl-departed.py reads it out of usbdev_ioctl's own dispatch and refuses to emit the lane's vectors unless every request it dispatches has a row in tests/usbdev-ioctl-departed.tbl saying what Linux answers and why -- so an ioctl added to the layer fails make check until its departed-device answer is recorded. make check-usbdev-departed is that gate on its own.

The default arm is in the join too, and had to be added to it: a join built from case labels alone covers every arm except the one catching what no label matches, which is how that arm's ENOTTY on a departed device survived the table built to find exactly that kind of answer. A row may name a USBDEVFS_ request usbdev.c defines and dispatches nowhere, the generator refuses to emit while the arm exists with no row driving it, and three rows do.

A row whose elfuse answer differs from Linux's carries both values and is an XFAIL: the lane fails if the difference widens and fails if it quietly closes, so a deliberate gap stays a recorded measurement instead of becoming a sentence in a comment. The lane runs three times because one process can hold only one departed device: the first correct ENODEV stamps every fd open on the node, so the rows that need a claim taken before the device left cannot share a process with the rows that take it away.

None of that is in the shipped binary. src/syscall/usbdev-fixture.c is a translation unit under src/ that only an assertion has a use for, so the default build links src/syscall/usbdev-fixture-stub.c in its place: the same seven entry points, answering false and -ENODEV, 44 bytes of text against the model's 11 KB. USB_LOOPBACK_FIXTURE=1 swaps the two, and it names the binary as well: mk/config.mk points ELFUSE_BIN at $(ELFUSE_LOOPBACK_BIN) for that flavor, so the fixture build writes build/elfuse-loopback and a plain make leaves build/elfuse without the model. The name is what carries that, because nothing else does. The switch changes SRCS and not CFLAGS, and the stale-object guard in mk/common.mk is keyed on CFLAGS alone, so while both flavors wrote one path a fixture build left the model in build/elfuse and the next make elfuse answered "Nothing to be done" -- measured on the tree as it stood before this split, at 834288 bytes with nm finding _usbdev_fixture_lock, against 815984 clean, until a make clean. That sequence cannot be run again to re-measure, which is what the split is for, so those two figures stay dated to the pre-split tree and are the only place this series records them. What reproduces on the tree as it stands is the pair the paths now keep apart: a clean build/elfuse is 815984 bytes with the symbol 0 times, and build/elfuse-loopback is 834304 bytes with it once. Both are link outputs, so they hold only for the compiler that produced them: Homebrew clang 22.1.8, reached through /opt/homebrew/opt/llvm/bin on PATH, which is the same toolchain the format gate already requires. Built with Apple clang 17.0.0 from /usr/bin instead -- which is what CC := clang in mk/toolchain.mk finds when that directory is not on PATH -- the same two commits give 817776 and 836160, deterministic: +1792 on the default flavor and +1856 on the loopback one, a gap per flavor rather than one constant. So a byte count that does not match here is a different compiler before it is a different tree, and the two symbol counts, 0 and 1, are what hold under either. .ci/check-usb-fixture-bin.sh fails if the two paths are ever equal again. Which object defines the seam is the whole difference between the two builds: not one branch in usbdev.c is conditionally compiled, so the fixture cannot drift into code the default build never compiles, and the default build still pays the branch that keeps the fixture off every path it must not touch.

Deviations From Linux

usbfs behavior	elfuse behavior
RESET re-enumerates the device	asks the device first, then kills the URBs and clears the claimed pipes' stall state and returns 0; USBDeviceReEnumerate would tear down every open plugin handle
every ioctl answers ENODEV once the device is gone, from usbdev_do_ioctl's connected() gate	some answer from the open-time model or from this layer's own bookkeeping instead, on an fd that has not yet been told the device left. Which ones, and what each answers, is one row per request in tests/usbdev-ioctl-departed.tbl; the list is not repeated here, because it was written down wrong twice
isochronous URBs	EINVAL
DISCSIGNAL delivers a signal on disconnect	signal and context stored, never delivered
sync BULK on an interrupt endpoint works (converted to an interrupt URB)	EINVAL; the conversion exists only on the async URB path
BULK_CONTINUATION unlinks the rest of the cascade on error	flag accepted, no cascade unlink
dup of a usbfs fd works	EBADF

procfs And Device Emulation

src/runtime/procemu.c intercepts a focused set of guest-visible paths under /proc, /dev, and a few Linux-expected compatibility files:

• Many procfs files are synthesized from host-side runtime state.
• /proc/self/* is backed by internal region, FD-table, and process data.
• Synthetic proc directories support common traversal patterns used by BusyBox ps, uptime, and top.
• Guest cwd handling preserves a virtual /proc working directory even though the host operates on synthetic backing directories.

Reported Kernel Identity

Two surfaces tell the guest which kernel it is running on: uname(2), served from a static linux_utsname_t in src/syscall/sys.c, and /proc/version, emitted as a literal by src/runtime/procemu.c. Both spell the release through GUEST_KERNEL_RELEASE and GUEST_KERNEL_VERSION in src/syscall/sys.h, so the two cannot disagree; tests/test-proc.c asserts that the release uname reports appears in the /proc/version banner.

The reported release is 6.18.0, the 6.18 LTS baseline; it tracks no test image. elfuse emits no /proc/sys/kernel/osrelease, so glibc's dl_discover_osversion falls back to uname(2) and reads this value at every dynamic startup. It refuses a kernel only when the release is below its build-time floor, so a high baseline is the safe side.

The release feeds version-gated feature detection. src/syscall/dispatch.tbl is the implemented set with two exceptions: pidfd_getfd and userfaultfd are registered there, but their handlers in src/syscall/syscall.c return -ENOSYS, and tests/test-pidfd.c and tests/test-userfaultfd.c pin them at that stub. Anything absent from the table gets -ENOSYS from the dispatch default, whatever number uname prints. Calls a 6.18 kernel provides that are absent: io_uring_*, mseal, cachestat, process_madvise, listmount/statmount, landlock_*, and file_getattr/file_setattr. Guests probe for a syscall and handle ENOSYS.

The LINUX_2.6.39 tag in the synthetic vDSO (src/core/vdso.c) is the frozen ELF symbol-version name the real arm64 vDSO exports; a genuine 6.18 kernel emits it too.

Three things read the release automatically. tests/test-proc.c runs on both the elfuse and the qemu aarch64 lane and checks only that the banner and uname agree, so on the qemu lane it pins the real Alpine kernel to itself. tests/test-comprehensive.c asserts the release is at least 6.18 behind a uts.nodename == "elfuse" gate, which holds on the elfuse lanes and not on the qemu lane, whose initramfs hostname is elfuse-qemu. The third reader is silent: build/bench-hot-guard-glibc is the one aarch64 glibc binary in make check, built only when the cross-toolchain sysroot is present, and a release below glibc's floor would abort its startup rather than fail an assertion. The Alpine tree tests/fetch-fixtures.sh builds is musl throughout and does no version gating.

/proc/self/smaps and /proc/<pid>/smaps are generated from the same tracked VMA list as /proc/self/maps. The complete field set currently emitted for each VMA is the maps header followed by these 24 fields, in this order:

Size, KernelPageSize, MMUPageSize, Rss, Pss, Pss_Dirty,
Shared_Clean, Shared_Dirty, Private_Clean, Private_Dirty, Referenced,
Anonymous, KSM, LazyFree, AnonHugePages, ShmemPmdMapped, FilePmdMapped,
Shared_Hugetlb, Private_Hugetlb, Swap, SwapPss, Locked, THPeligible,
VmFlags

Size is the VMA length in KiB. KernelPageSize and MMUPageSize are reported as 4 KiB. In a fork child, writable private anonymous VMAs that existed in the parent's CoW snapshot report their full VMA size for Shared_Dirty, Rss, Pss, and Pss_Dirty, keeping those coarse counters internally consistent. Newly-created VMAs are excluded from that signal. Every other numeric counter, including THPeligible, is emitted as a stable zero. ProtectionKey (a Linux pkeys field added in Linux 4.9) is intentionally omitted because the macOS host has no equivalent; consumers comparing against a real Linux kernel should treat it as optional. VmFlags is evidence-based and contains only the permission/sharing flags represented by the tracked VMA (rd, wr, ex, sh, and nr when applicable); no untracked kernel flags are invented. A VMA with no such evidence (for example PROT_NONE) still prints the field's separating space as VmFlags: . These are coarse VMA-level values, not host page-residency, dirty-bit, or proportional-sharing accounting, so they are suitable for fork-safety checks but not precise memory profiling. The fork-child marker is tracked per VMA, so writable private anonymous mappings created after fork are excluded from the compatibility signal. smaps_rollup is not implemented.

Synthetic proc directories have explicit snapshot boundaries. The backing trees reached by opening /proc or /proc/self are materialized once, on the first access, and their initial stat, status, cmdline, maps, and smaps files remain fixed for the process lifetime. An absolute open of one of those proc paths is intercepted directly and generates fresh content; opening the same name through an already-open synthetic directory reads that directory's snapshot. /proc/self/fd and /proc/self/fdinfo are rebuilt into an independent scratch directory on every open, so each directory fd sees the guest-fd table as it existed at that open and concurrent enumerations cannot mutate one another. /proc/self/task is repopulated from the current thread set whenever the directory is opened. These boundaries are intentional: a directory stream is stable while it is being read, while dynamic task and fd listings refresh only when a new directory is opened. The synthetic /sys/devices/system/cpu tree follows the one-shot rule: its CPU count, cpumask files, and cpuN directories are captured on first access and then remain fixed.

Synthetic USB Device Tree

src/runtime/usb-sysfs.c materializes two more scratch-directory trees, built from an IOKit enumeration of the USB registry that opens no device: /sys/bus/usb/devices and /dev/bus/usb. Device directories are named <bus>-<ports> (for example 2-1.4), interface directories <bus>-<ports>:<config>.<interface>, following the Linux sysfs layout that libusb and nusb walk. busnum comes from the IOKit locationID's top byte plus one (macOS numbers controllers from 0, Linux busses from 1), devnum from the device's USB address, and the port path from the locationID nibbles. Each device directory carries the attribute files those enumerators read (busnum, devnum, idVendor, idProduct, speed, the manufacturer / product / serial strings when the device supplies them, the dev major:minor, and the rest of the descriptor fields), a uevent file, a subsystem symlink, and the binary descriptors blob.

Two invariants matter to consumers. The descriptors attribute and the matching /dev/bus/usb/BBB/DDD node read byte-identically, because both serve one stored blob per device -- the rule Linux keeps between sysfs and the usbfs read() view. And statfs / fstatfs on /sys paths report SYSFS_MAGIC, without which libudev refuses to trust the tree and libusb falls back to a usbfs directory scan.

The /dev/bus/usb/BBB/DDD nodes are 0444 placeholder files on disk (the /dev/pts placeholder pattern); the stat intercept reports them as character devices, major 189, minor (bus - 1) * 128 + (dev - 1), and the open intercept diverts an open away from the placeholder into the usbdevfs fd constructor (see USB Device Passthrough).

One layout deviation is deliberate: the /sys/bus/usb/devices entries are real directories, not symlinks into /sys/devices/..., so realpath() of an entry canonicalizes to itself. libusb opens attributes relative to the entry and nusb canonicalizes the entry path; both tolerate this.

The tree follows the /sys/devices/system/cpu one-shot rule: it is built lazily under usb_lock on first access and then stays fixed for the process. Hotplug support, once the uevent layer can observe attach and detach, would discard the tree so that the next access re-enumerates; until then a replugged device is not picked up within a run.

A second, smaller deviation is in /dev/bus/usb: the device nodes are placeholder files, so getdents64 reports them with d_type DT_REG where Linux reports DT_CHR. stat() is correct -- the intercept fills S_IFCHR with major 189 and the right minor, which is what libusb, nusb and lsusb consult -- so only a scanner that filters on d_type alone, without ever calling stat, would skip the nodes. No such consumer is known; creating real character devices needs privileges elfuse does not ask for, so the placeholder stays and the deviation is recorded here rather than papered over.

One boundary is worth stating plainly: there are no usbN root-hub entries. macOS publishes no root hub as a USB device -- the registry match returns downstream devices only -- so the tree carries no bus entry and no parent link from a device up to it. Enumeration is unaffected (libusb leaves the parent NULL, nusb skips port-less names), but topology is: a lsusb -t listing shows the devices without the bus rows above them.

The configuration attribute is the other one. It holds the string named by the active configuration's iConfiguration, and a string descriptor is fetched over an ep0 control transfer on an open device -- which this layer does not do. IOKit does not offer a way around it: an IOUSBHostDevice registry entry publishes iManufacturer, iProduct and iSerialNumber alongside the three strings the family caches for them, but no iConfiguration and no configuration string. So the file is emitted empty, and that is not a claim the configuration has no string: Linux's usb_cache_string returns NULL "if the index is 0 or the string could not be read", and configuration_show emits nothing in both cases, so an empty configuration on Linux means "no cached string" and every device here takes the second of those two routes to it. The file stays present rather than being omitted, because dev_attr_grp carries no .is_visible hook and so every Linux USB device has all four of bNumInterfaces, bmAttributes, bMaxPower and configuration; a value the kernel cannot supply is a zero-length read, never an ENOENT. The same rule is why those first three are emitted even for a device whose descriptor blob carries no usable configuration at all. The fidelity gap that remains is narrow and one-directional: a device whose iConfiguration is non-zero and whose string descriptor is readable shows that string on Linux and shows nothing here.

Related implementation: src/runtime/procemu.c, src/syscall/path.c, src/syscall/fs.c, src/syscall/proc-state.c, src/runtime/usb-sysfs.c.

Ownership Of /sys And /dev/bus Names

The layer synthesizes exactly one subtree on each side, /sys/bus/usb and /dev/bus/usb, on top of a /sys and a /dev/bus that a sysroot supplies. Which of the two answers a name is one decision, taken once in classify_and_normalize, and every entry point -- open, stat, lstat, readlink, access, getdents64, statfs, chdir -- answers from it. An entry point that re-derives the decision is how four regressions arrived, each one a shadow: the layer claiming a name it does not serve and reporting ENOENT for a file the sysroot really has.

The classes and who answers them:

Class	Path	Answered by
USB_PATH_SYS	/sys[/suffix]	the layer, if the folded suffix resolves in the scratch tree; otherwise the backing, unless the suffix is under bus/usb, where an absence is authoritative
USB_PATH_DEV_BUS	/dev/bus	both: a union listing, synthetic usb plus the backing's names
USB_PATH_DEV_USB	/dev/bus/usb	the layer
USB_PATH_DEV_BUSNUM	/dev/bus/usb/BBB	the layer
USB_PATH_DEV_NODE	/dev/bus/usb/BBB/DDD	the layer
USB_PATH_DEV_NODE_SUB	a node used as a directory	the layer (ENOTDIR once the node exists)
USB_PATH_DEV_ABSENT	under /dev/bus/usb, no such device	the layer, ENOENT
USB_PATH_DEV_FOREIGN	under /dev/bus, a bus we do not model	the backing
USB_PATH_NONE	anything else, and a name that folds above its root	the backing

Only PROC_NOT_INTERCEPTED means "ask the backing". A name the layer claims and then fails to serve is an answer, not a fall-through: taking the failure for one let access(2), and then statfs(2), answer from the backing while open and stat reported ENOENT for the same path.

. and .. are folded before ownership is decided, on both halves. On /sys the fold is lexical and is the ours/not-ours gate and nothing else: the served path is built by usb_sys_resolve_suffix, which resolves symlinks and applies each .. to what the previous component resolved to, as Linux does, so <dev>/subsystem/.. names /sys/bus. On /dev/bus a .. after a usbfs node is not applied, since Linux applies .. only to a directory, and the name answers ENOTDIR once the node exists. A .. after any other name is applied without checking that the name exists, so /dev/bus/usb/099/../001/001 is served where Linux answers ENOENT. A relative name is folded the same way up to a usbfs node, so 099/../../../null from a /dev/bus/usb cwd skips the check the climb rewrite makes for the absolute spelling. Deciding ownership on the unfolded spelling splits the two halves apart in both directions: /dev/bus/usb/../other/f reads as a malformed device number and is claimed, and /dev/bus/other/../usb/001/002 reads as a foreign bus and is disowned. A suffix that folds away above its own root leaves as USB_PATH_NONE.

A backing cannot apply such a .. on /dev/bus: the directories are the layer's, and the macOS host has no /dev/bus at all. So a /dev/bus name that climbs out over the layer's own directories is first rewritten in the path layer to /dev followed by what comes after the .. that climbed. A further .. is applied to /dev and stops at /, so /dev/bus/../../proc/self/status reaches the /proc intercept. When the climb ends the name, the rewrite ends in ., as in /dev/., so rmdir and rename fail rather than act on the directory reached, with EINVAL on a sysroot where Linux gives ENOTEMPTY and EBUSY for a final ...

Filesystem Identity Of A Descriptor

Two questions have one answer here: what statfs reports for a name, and what fstatfs reports for a descriptor opened by that name. systemd's sd-device gates every enumerated syspath on fstatfs(fd) == SYSFS_MAGIC and libusb gates on statfs("/sys"), so a build where the two disagree is one where a device enumerates through one library and not the other.

The rules:

• /sys, whichever side served it, reports SYSFS_MAGIC (0x62656572). That covers the scratch-dir backed names, where the host fstatfs would leak the /tmp filesystem's magic, and the ones that fell through to a sysroot's own /sys, where it would leak the sysroot's.
• /dev/bus reports devtmpfs, on both entry points.
• .. is folded and a relative name is resolved against the cwd before either entry point decides, so the two cannot be handed different spellings of one object. A .. after a usbfs node is the exception, and statfs asks /dev/bus before its /sys fold so that it answers ENOTDIR as stat does.

A descriptor's identity comes from the virtual path stamped on its slot when this layer served the open, and from the descriptor's own host path mapped back through the sysroot when there is no stamp -- a fall-through open stamps nothing, which is exactly the case that used to make statfs("/sys/class") report SYSFS_MAGIC while fstatfs on the fd it had just opened reported the sysroot's filesystem.

fstat answers from the stamp for O_PATH descriptors and for /sys and /dev/bus names, because the object behind a /dev/bus/usb/BBB/DDD node is a placeholder file: libusb fstats the node it just opened and refuses anything that is not a character device, so the descriptor has to report the character device the path side described. /proc stays O_PATH-only there: its descriptors are real host files whose contents are the answer, and stamping their type would misdescribe the object the guest is reading.

The stamp and the descriptor are read in one fd_lock window (host_fd_ref_open_entry). They are two facts about one open file description, and read separately a close and reopen between them gives the stamp of one and the descriptor of another.

Union Directory Listings

A directory that exists on both sides -- /sys, /dev/bus -- lists the union. sys_getdents64 walks the synthetic (primary) stream first and then the names drained out of the backing; the drain opens the backing directory and closes it before returning, so elfuse keeps its promise of one host descriptor per guest fd and a union directory costs no more than a plain one.

What the guest is told when part of the listing cannot be delivered is the contract that matters, because the failure mode is silent: a listing that lost a name and still ends at 0 is an end-of-directory the guest cannot tell from a real one, and having read it, it never asks again.

• A failure that costs the stream names it can never produce again -- a drain that failed part-way, a primary readdir that stopped -- is recorded in listing_errno. It is sticky: every later call on that fd fails the same way. Linux has no case that re-delivers an error on a directory stream, so there is no shape to copy; clearing it would put the next call back on the silent path, and the stream has nothing truthful left to return.
• Linux's two-part shape is followed for reporting one: a call that has already written entries returns their count and leaves the error for the next call; a call that has written nothing returns -1 with the errno.
• A buffer too small to hold one entry is EINVAL, not 0. Measured on Linux over a directory holding a twelve-byte name: getdents64 with 8- and 16-byte buffers returns EINVAL immediately, and with 24 bytes it delivers the names that fit and then reports.
• No exit from the walk may consume an entry without putting it back. The entry-does-not-fit path and the guest-write fault path both rewind the primary with seekdir; the backing side rewinds by not advancing its cursor.
• A host name too long for Linux NAME_MAX is skipped, and the rest of the stream is delivered. This is an elfuse compatibility policy with no Linux counterpart: Linux enforces NAME_MAX at the filesystem layer, so no oversize entry ever reaches getdents64 there. macOS APFS accepts them, and aborting the stream truncated ls / find listings against APFS trees.

dir_stream_t is the wrapper around the DIR*. It is refcounted and carries its own lock, so an in-flight getdents64 pins it against a sibling's close(); dup, dup2 and F_DUPFD share one wrapper rather than opening a second, which is what makes two guest fds on one open file description share one position and one union state. A forked child's inherited fds share one wrapper per inherited open file description for the same reason. The wrapper owns the descriptor it was opened on -- only closedir() gives it back -- so an alias names the descriptor the shared wrapper holds and the redundant one is closed.

A stream built for an inherited descriptor has backing_private set: the primary is shared with the parent through the description and must be read, and the backing belongs to whichever stream first ran out of primary and must not be drained twice. docs/testing.md records the one state this cannot deliver whole.

Limits Of The IOKit Mapping

This tree enumerates devices; the pieces that follow open them and move data. It is worth stating where the approach stops, because three of the limits are macOS policy that no amount of translation moves, and the rest cost something permanent. The short version: the layer is complete for devices macOS does not claim, honest but limited for the ones it does, and structurally unable to present bus topology.

Driver arbitration decides which devices are reachable at all. Once a class driver matches an interface, USBInterfaceOpen returns kIOReturnExclusiveAccess, and an unprivileged process has no way past it. On the ESP32-S3 used to develop this: interface 0 (CDC control) is held by AppleUSBACMControl and always refuses, interface 1 (CDC data) refuses whenever anything holds a /dev/cu.usbmodem* node, and interface 2 (a vendor JTAG class with no matching driver) opens freely. The documented escape, USBDeviceReEnumerate with kUSBReEnumerateCaptureDeviceMask, needs root or the com.apple.vm.device-access entitlement; an unentitled non-root caller gets a success return that changes nothing, the registry ID and the driver binding both intact. Mass storage is excluded from capture outright. So the layer serves vendor and bulk devices completely, while CDC and HID stay reachable only through the interfaces macOS already exposes -- /dev/cu.* and IOHIDManager. This is why the ttyACM / ttyUSB aliasing is a companion to the USB layer rather than a workaround for it.

Cancellation granularity differs in a way that costs throughput. USBDEVFS_DISCARDURB cancels one URB; IOKit's AbortPipe cancels everything outstanding on the pipe. Preserving the Linux semantics means queueing inside elfuse and handing IOKit one transfer per endpoint at a time, which gives up the pipelining a bulk-heavy workload depends on. The alternative -- report the collateral cancellations as -ECONNRESET and let the guest resubmit, which libusb and nusb both do -- lets a cancel on one transfer disturb unrelated ones. Neither choice is free, and no macOS API removes the tradeoff.

Reset is an unplug. ResetDevice has been a no-op since OS X 10.11, so the only real reset is USBDeviceReEnumerate, which tears down the user client and re-runs matching. Linux USBDEVFS_RESET keeps the fd valid and the device numbering stable. Emulating it means holding busnum and devnum steady across a registry identity that changed underneath, waiting for the re-attach, and diffing descriptors afterward -- and distinguishing "came back different" from "never came back", since firmware update flows (DFU) are both the workflow that leans on reset and the one where the descriptors legitimately change.

Identity is topological, not device-based. A locationID encodes the port path, so unplugging one device and plugging another into the same port yields the same value. The enumeration cross-checks idVendor, idProduct and the serial string against the cached model and answers -ENODEV on a mismatch, which is safe but not complete: the model is a one-shot snapshot with no hotplug observer, so a device attached mid-run is unusable until the process restarts. Real hotplug would rewrite IOKit attach and detach notifications as uevents on the netlink socket -- a fair amount of machinery for something libusb tolerates the absence of.

Bus topology has no macOS source, the boundary noted above: macOS publishes no root hub as a USB device, so there is no usbN entry, no parent link, and lsusb -t loses its bus rows. A plausible root hub could be synthesized, but it would be a device no registry entry backs, and its maxchild, port numbering and /sys/bus/usb/devices/usb1 node would be invention too. The tree stays short rather than carrying a fabricated node.

Two narrower ones. Isochronous transfers need explicit frame scheduling through GetBusFrameNumber, where usbfs lets URB_ISO_ASAP hand the timing to the controller, so audio and video capture do not map cleanly. And the permission models do not correspond: Linux gates usbfs through udev rules and file permissions, macOS gates IOKit through driver matching and entitlements, and there is no single spelling of "let this program use this device" that means the same on both.

Path Resolution

A guest names files the way Linux does, from a namespace rooted at the guest's /. The host has its own root, and with --sysroot the guest's tree is a subdirectory of it. Every path-taking syscall therefore asks two questions before it can act, and path_translate_at() in src/syscall/path.c answers both in one place so no two syscalls can answer them differently for the same name:

1. Does the sysroot claim this path? A path it holds resolves there; one it does not falls back to the host filesystem, except for the guest system directories and the temp roots, which resolve in the sysroot whether or not it holds them.
2. What host spelling does it get? That is the sysroot prefix plus the guest path, adjusted so the host kernel's own resolution lands where Linux's would.

Three resolvers in src/syscall/proc-state.c do the work: proc_resolve_sysroot_path() for a following lookup, its _nofollow_ sibling, and proc_resolve_sysroot_create_path() for a path whose final component may not exist yet. path_translate_at() picks one by flags; sys_path_has_symlink() calls the nofollow form directly for absolute paths in the openat2(RESOLVE_NO_SYMLINKS) precheck described below, and walks relative paths from their descriptor with the same clamp applied in the walk.

Folding // And .

Linux steps over // runs and . components in the walk every path syscall shares, so //sys/bus, /./sys/bus and /sys/./bus are /sys/bus to all of them. The intercepts match literal prefixes, so path_translate_at() folds an absolute name once, before any of them reads it.

Two things stay as written. .. is not folded, because Linux applies it to what the component before it resolved to. And a . that is the last component keeps its place, because that is where Linux gives it a meaning of its own: rmdir("d/.") is EINVAL where rmdir("d/") removes d.

What a final . or a trailing slash means to a lookup is a requirement: the name has to resolve to a directory, following a final symlink to get there. The host walk applies it for itself. The intercepts match names literally, so proc_intercept_open(), proc_intercept_stat_at() and proc_intercept_readlink() take the requirement off the name once, dispatch on the bare name, and enforce it on the answer: a served directory answers as itself, anything else answers ENOTDIR, and readlink of a served directory answers EINVAL. The gates in path.c read the bare name for the same reason.

tests/test-path-fold.c holds every respelling of a name to the answer its canonical spelling gets, and both endings to what Linux makes of them.

Clamping .. At The Guest Root

A guest resolves .. against its own root, and Linux clamps it there: /.. names /, and no number of .. components reaches the directory above (path_resolution(7)). The sysroot is an ordinary directory on the host, so the host kernel has somewhere to go, and a guest path handed over unchanged climbs straight out of the tree.

clamp_dotdot_at_guest_root() rewrites exactly the components that would do that, and nothing else:

guest path	host spelling below the sysroot	why
/.., /../..	/	the clamp, applied once per escaping component
/../etc/hosts	/etc/hosts	the escaping .. is dropped, the rest is untouched
/a/../b	/a/../b	interior, so the host resolves it
/a/../../etc/x	/a/../etc/x	one .. is interior, the second escapes
/file/., /file/	unchanged	the trailing form still demands a directory

Interior .. survives on purpose. Collapsing it would spell a path whose popped components the host never looks at, and their existence and type are part of the answer Linux owes: /absent/../b is ENOENT, /file/../b is ENOTDIR, rmdir("/a/b/..") fails rather than removing /a, and a .. past a symlink keeps that link in the path where the no-symlinks precheck can see it. Only the host walk can report those, so only the host walk decides them.

Two spellings of a path therefore travel together: the clamped one, which is probed and handed to the syscall, and the fully collapsed one from lexical_normalize_absolute_path(), which is used solely to classify the path as a guest system directory or a temp root, where /usr/../home/x must be judged as /home/x rather than by its leading component.

Relative Paths And The Containment Recheck

The resolvers key off a leading /, so a relative path reaches them unchanged: they have no dirfd to rebuild a location from. That would leave openat(dirfd, name) to the host kernel, whose resolution is not confined to the sysroot, so path_check_relative_sysroot_containment() reconstructs the absolute guest path from the descriptor's guest base path and runs it back through the same resolver.

A reconstructed path that climbs above the guest root needs more than a verdict. Clamping makes the guest's own answer well defined, but the host still holds a descriptor from which .. leads out of the sysroot, and the recheck cannot express "contained, but not by that route": a path the sysroot does not claim looks the same as one that never left. So when the reconstruction clamps, the recheck hands back the absolute host path it resolved and the caller uses that instead of the guest's relative spelling. POSIX has the kernel ignore dirfd for an absolute path, so the descriptor drops out of the resolution entirely, which is the point.

A climbed path is the one case where the recheck's answer is the resolution itself rather than a discarded verdict, so it also honors the caller's create intent: missing sysroot parents are materialized exactly as they would be for the absolute spelling.

Why The No-Symlinks Precheck Has No Component Budget

openat2(RESOLVE_NO_SYMLINKS) must fail with ELOOP if any component of the path is a symlink. macOS has no equivalent flag, so sys_path_has_symlink() walks the path itself, one component at a time, and reports ELOOP at the first S_ISLNK.

An absolute path is walked in its host spelling, which the resolvers have already clamped. A dirfd-relative path is walked from the descriptor, so the clamp happens in the walk: the descriptor's guest depth seeds a counter, and a .. that would climb above the guest root stays in place instead of stepping onto the sysroot's host parent, whose entries, macOS's own symlinks among them, are not the guest's to trip over.

That walk deliberately carries no MAXSYMLINKS counter. It never follows a link, so nothing can accumulate against a link budget; the only thing a per-component counter could reject is a link-free path deeper than the limit, which Linux resolves without complaint, since path_resolution(7) caps links followed rather than components walked. The counter that does matter lives in path_openat2_crosses_mount(), which follows links to answer RESOLVE_NO_XDEV and charges MAXSYMLINKS once per link actually crossed.

Related implementation: src/syscall/path.c (path_translate_at, path_check_relative_sysroot_containment, sys_path_has_symlink, path_openat2_crosses_mount), src/syscall/proc-state.c (clamp_dotdot_at_guest_root, sysroot_seed_host_path, the three resolvers). Validation: make test-sysroot-dotdot and make test-sysroot-openat2-walk, plus the openat2 cases in tests/test-syscall-fidelity.c, which make test-matrix runs against both elfuse and a reference kernel.

POSIX Shared Memory (/dev/shm)

Linux exposes POSIX shared memory through /dev/shm, a tmpfs the C library opens by name: shm_open("/foo", ...) opens /dev/shm/foo. macOS has no /dev/shm, so elfuse backs it with a per-UID host directory, /tmp/elfuse-shm-<uid>/<name>. dev_shm_resolve_path() in src/runtime/procemu.c (exported as proc_dev_shm_resolve) is the single source of truth for that mapping and gates the name. This is a different mechanism from System V shared memory (shmget/shmat), which lives in src/syscall/sysvipc.c.

One Redirect, One Resolution

Every path syscall resolves guest paths through path_translate_at() in src/syscall/path.c. For a /dev/shm/<leaf> path it rewrites host_path into the backing directory and records the fact in tx->is_dev_shm. If each syscall applied the redirect on its own, any that missed it would fall through to the sysroot while its peers used the backing directory, so /dev/shm/foo would resolve two ways for the same program:

/dev/shm/foo
  open  -> /tmp/elfuse-shm-<uid>/foo   (backing dir, created)
  chmod -> <sysroot>/dev/shm/foo       (absent -> ENOENT)

Resolving in path_translate_at instead means chmod, chown, truncate, utimensat, rename, link, symlink, mknod, readlink, mkdir, statfs, and the xattr calls all inherit the backing path from one place, so an open followed by any of them on the same name stays consistent.

The early return also takes shm objects out of sysroot resolution entirely, so the /tmp/elfuse-shm-<uid> backing directory is reached as a host path and is unaffected by the sysroot's redirect of guest /tmp.

Only a non-empty flat leaf is redirected. Bare /dev/shm and /dev/shm/ stay on the sysroot path so the synthetic-directory intercepts keep answering for them, and statfs on a shm leaf or on /dev/shm reports TMPFS_MAGIC synthetically rather than the host filesystem's type. Because the backing path is absolute, two inline helpers in src/syscall/path.h adapt the *at() calls: path_translation_dirfd() returns AT_FDCWD (POSIX ignores dirfd for an absolute path), and path_translation_at_flags() forces the nofollow flag described next.

The Never-Follow Invariant

On Linux /dev/shm is an in-namespace tmpfs, so a symlink planted at a shm leaf resolves inside that namespace. elfuse's backing store is a plain host directory, so the same symlink would resolve onto the host filesystem, which is a sandbox escape. A symlink leaf is never legitimate anyway: glibc's shm_open (sysdeps/posix/shm_open.c) opens objects with O_NOFOLLOW. So every shm operation acts on the leaf itself, never the target it points at. Because the resolver hands back an absolute host path that bypasses the sysroot, that duty is spread across the syscall families, one mechanism each:

Operation family	Never-follow mechanism
*at() metadata (chmod, chown, stat, utimensat, access)	path_translation_at_flags() adds AT_SYMLINK_NOFOLLOW
open for truncate/chdir	shm_open_leaf() opens O_NOFOLLOW
proc open	O_NOFOLLOW
xattr get/set/list/remove	XATTR_NOFOLLOW
stat	lstat, not stat
linkat	clears AT_SYMLINK_FOLLOW
statfs	nofollow lstat existence probe, then a synthetic reply

The name gate lives with the resolver. A POSIX shm name is always a single flat component: glibc's __shm_get_name (posix/shm-directory.c) strips the leading slash and rejects an empty name or any embedded / with EINVAL. dev_shm_resolve_path() enforces the same shape, additionally rejects the .. component (a flat name like a..b is fine), and returns EACCES for a rejected name (ENAMETOOLONG if the backing path overflows).

Related implementation: src/runtime/procemu.c (dev_shm_resolve_path), src/syscall/path.c and path.h (path_translate_at, is_dev_shm, path_translation_dirfd, path_translation_at_flags), and the metadata handlers in src/syscall/fs.c, fs-stat.c, and fs-xattr.c. Validation: tests/test-dev-shm-paths.c.

Dynamic Linking

elfuse supports dynamically linked aarch64-linux ELF binaries via --sysroot:

elfuse --sysroot /path/to/sysroot ./my-dynamic-program

How it works:

1. elf_load() parses PT_INTERP to find the interpreter path (/lib/ld-linux-aarch64.so.1 for glibc, /lib/ld-musl-aarch64.so.1 for musl).
2. The interpreter is loaded as ET_DYN at g->interp_base (computed dynamically: 60 GiB for 36-bit IPA, 1020 GiB for 40-bit IPA).
3. build_linux_stack() passes AT_BASE (interpreter load address) and AT_EXECFN (the execve filename, supplied by the caller) in the auxiliary vector. Linux takes AT_EXECFN from bprm->filename, so it stays the program the guest asked for even when argv[0] is an alternate name or the rosetta binfmt_misc argv prepends the translator.
4. The entry point becomes interp_entry + load_base; the dynamic linker takes over from there.
5. Guest absolute paths reach the host through path_translate_at() (src/syscall/path.c), the single forward resolver every path-taking handler uses; with --sysroot set it dispatches each path between the sysroot and the host on existence. The temp roots (/tmp, /var/tmp, any .ccache directory) and the guest system directories are exempt from that dispatch and resolve in the sysroot either way, so lookup and removal cannot disagree about where a path lives. filenames.md covers how a name is spelled once it lands on the sysroot volume.

The sysroot is inherited by fork children via IPC state transfer. sys_execve also loads the interpreter for dynamically linked targets, so tools that execve dynamic children (env, nice, nohup) work correctly. The two loaders resolve PT_INTERP in different orders. Guest-issued execs route it through path_translate_at() like any other guest path. The initial process is loaded by the core bootstrap (load_interpreter() in src/core/bootstrap.c), which probes elf_resolve_interp() (src/core/elf.c) first, a literal sysroot concatenation plus a /lib/<basename> fallback, and only when both probes miss does it fall through to path_translate_at(), materializing a FUSE interpreter and refusing one in /dev/shm.

Known Limitations

None specific to the aarch64-linux dynamic-linker path. Limitations in guest path handling are covered by filenames.md, which records what the sysroot volume can and cannot represent.

x86_64-via-Apple-Rosetta

src/core/rosetta.c hosts Apple's Rosetta Linux translator inside the same VM that runs aarch64 guests, so statically linked x86_64-linux ELFs execute through a translator the host process owns rather than an external translation service. The guest architecture is auto-detected from the ELF header; opt out via --no-rosetta or ELFUSE_NO_ROSETTA=1.

Address-space layout:

• The translator lives in the primary buffer at a low guest physical address but is mapped at its link address 0x800000000000 via a non-identity page-table entry. This works around the 36-bit Stage-2 IPA cap on M1 / M2.
• A 256 MiB kernel buffer (kbuf) at g->kbuf_gpa is aliased at KBUF_VA_BASE = 0xFFFFFFFFF0000000 under TTBR1 and at KBUF_USER_VA = KBUF_VA_BASE & 0x0000FFFFFFFFFFFF under TTBR0, so Rosetta's TaggedPointer extraction resolves both views to the same physical pages. The kbuf is always RW; nothing executable is installed there.
• M5 hosts bisect the primary slab from 1 TiB to 256 GiB or 64 GiB on hv_vm_map failure while keeping the IPA width at 48 so the high-VA Stage-2 entries remain reachable.

Runtime integration:

• The VZ ioctls (CHECK 0x80456125, CAPS 0x80806123, ACTIVATE 0x6124) are trapped when g->is_rosetta is set.
• /proc/self/exe is redirected to ROSETTA_PATH (the binfmt-misc convention Rosetta expects).
• The rosettad SCM_RIGHTS bridge implements an SHA-256-keyed AOT cache under $HOME/.cache/elfuse-rosettad/. First launch warms the cache; subsequent launches reuse translations.

Fork interaction: the CoW shm fast path is disabled for Rosetta because HVF caches host VA-to-PA at hv_vm_map time. Rosetta forks use the legacy IPC copy path.

Dynamic linking under Rosetta is supported. For x86_64 guests with a PT_INTERP header, elfuse does not load the ELF segments or the interpreter from the host side; bootstrap_prepare deliberately skips the aarch64 loader path (src/core/bootstrap.c:415-462) and instead hands control to Rosetta. The translator then opens the guest ELF through Linux syscalls, reads PT_INTERP, and mmaps /lib64/ld-linux-x86-64.so.2 (or the musl equivalent) from the sysroot. The translated dynamic linker loads shared libraries via the same path. Coverage lives in tests/test-rosetta-glibc.sh, which exercises direct loader bring-up, explicit ld.so invocation, ld.so --list, runtime dlopen, initial-exec TLS, general-dynamic TLS via dlopen, and per-pthread TLS.

Boundaries:

• --gdb is rejected because the stub serves the aarch64 view Rosetta produces, not the original x86_64 architectural state.
• Two Rosetta-internal divergences are tracked in the acceptance audit rather than papered over: SA_RESETHAND is shadowed by Rosetta's own signal-handler state, and clone(..., CLONE_SETTLS, tls=0, ...) can hang.

GDB Stub

src/debug/ is split by role:

• gdbstub.c -- session lifecycle, stop/resume flow, packet dispatch
• gdbstub-rsp.c -- RSP packet transport and hex helpers
• gdbstub-reg.c -- register snapshot layout, restore flow, target.xml

The stub runs in all-stop mode. Because Hypervisor.framework register access must happen on the owning thread, the stopped vCPU snapshots its own state; the GDB-handler thread reads and updates the snapshot, and the owning thread restores the modified state on resume -- the same pattern used by ptrace.

The split mirrors the architectural boundary: transport and encoding are independent of guest execution; register layout is independent of socket I/O; stop/resume sequencing remains tightly coupled to process and thread state.

Language Choice

elfuse is C11 plus a small amount of aarch64 assembly. This section records why, against the specifics of this codebase rather than in the abstract.

Where The Safety Boundary Falls

The central data structure is a slab of guest memory obtained from mmap, which the guest rewrites at will from another vCPU thread. Host access goes through bounds-checked accessors over that slab: guest_ptr, guest_ptr_w, guest_ptr_avail, guest_ptr_bound, guest_read, and guest_write (see Memory Layout), each resolving a guest-controlled address against the recorded mappings.

Rust would express the same shape as a safe wrapper over an unsafe core, and would enforce that callers stay on the safe side, which C leaves to convention. That enforcement is a real gain. What it does not do is validate the checks inside the wrapper, and those checks are where the defects below live: the interior of the guest slab, the page-table walk itself, and syscall arguments arriving as guest-controlled integers.

What Defects Surface In Practice

One worked example, from the loader. A PT_LOAD whose p_vaddr + p_memsz overflows is rejected where it is parsed, by the checked elf_add_no_wrap (src/core/elf.c), and verify-elf discharges that arithmetic as a proof obligation. Saturating load_max to UINT64_MAX is the alternative that does not hold: every consumer adds a load base to it before comparing against guest_size, so the clamp relocates the wrap into the consumer, where the bound check it was meant to trip reads elf_end > guest_size and passes. Where the image lands is a separate question, checked per segment by elf_place_segment under elf_check_placement, which sys_execve calls before its point of no return so an unplaceable image is a recoverable -ENOEXEC rather than a fatal exec.

The class of defect is the point. That is integer arithmetic, not memory safety. Rust's + wraps silently in release builds too, so the same checked add is required there.

elf_map_segments_fd also re-reads the header and program headers from the fd, and can disagree with the first parse: a TOCTOU on the ELF image. Its own bounds checks contain the damage, but no language rules the disagreement out. It is a property of how the loader is structured.

Verification Actually In Place

The language-independent tooling gates CI: clang-format, a banned-API and unsafe-preprocessor scan, cppcheck, and the dispatch-table consistency check on Linux; scan-build as an advisory job; clang-tidy advisory except for readability-function-size, which is named in WarningsAsErrors and fails the job when a function crosses its ceiling; an Infer run that fails on any finding; and a runtime matrix under ASAN, UBSAN, and TSAN (see Testing And Confidence). That catches memory-safety defects after the fact rather than excluding them by construction, which is the honest cost of the choice.

The formal layer proves each selected function body free of arithmetic runtime errors, assuming its stated preconditions. .github/workflows/verify.yml runs the targets a change can move, split between a proof leg and a per-target mutation matrix, with a final job that requires both halves; make verify runs the whole set locally. Each target names a function set and discharges its obligations with -wp-rte, which adds the implicit runtime-error goals (overflow, out-of-bounds, invalid dereference) that the ACSL contracts alone leave open. verify-elf covers the named arithmetic helpers in src/core/elf.c, not elf_load_fd, elf_map_segments_fd, or their call-site preconditions; the others cover the bounds math of the remaining attacker-facing parsers and packers.

Two shapes, and mk/verify.mk is the list that says which one a target is. Most name functions in a header under src/proved/, which is what lets one .c file's arithmetic be proved while the rest of that file stays unparsable to the analyzer. A few name functions in a .c file directly, so the loops are proved as written rather than as a copy of the math.

Four gates close the ways a proof can say less than it appears to:

• scripts/check-proof-targets.py (in make check): nothing lands in src/proved/ without a proof target.
• scripts/check-acsl-coverage.py: a contracted function left out of the proof set is an assumed axiom, so the target fails rather than trusting it.
• make verify-mutants: each target must reject a known-broken source, which is what catches a contract whose clauses do not bite.
• make check-contracts: rebuilds with -DELFUSE_CONTRACT_ASSERT so the expressible preconditions of proved/gva.h are checked on every call the suite makes, since src/core/guest.c does not parse under the analyzer and nothing else checks its call sites.

What the proofs do not cover: the I/O around the proved arithmetic (pread, malloc) stays test-covered, and preconditions at call sites in files the analyzer cannot parse are review-only. frama-c-stubs/ supplies the Darwin declarations the analyzer needs, held to the SDK's own values by scripts/check-stub-constants.py.

Host Interface Surface

hv_vcpu_run, hv_vcpu_set_sys_reg, Mach (host_statistics64 behind /proc/meminfo), pthreads, macOS syscalls, file descriptor passing over SCM_RIGHTS, and hosting the Apple Rosetta translator are all C ABI. The EL1 shim is aarch64 assembly (src/core/shim.S) under any host language. A safe-language port would spend a significant share of its effort on binding layers for that surface.

Testing And Confidence

elfuse uses several layers of validation:

• make check -- fast guest tests plus the BusyBox applet smoke suite, followed by scripts/check-syscall-coverage.py so any new dispatch.tbl entry without a direct or aliased test reference fails the build.
• make test-busybox -- applet coverage in isolation.
• make test-fuse-alpine -- guest-internal FUSE against the Alpine musl sysroot fixture.
• make test-gdbstub -- debugger integration.
• make test-rosetta-all -- the x86_64 acceptance sub-suites (CLI gating, failure modes, statics, Alpine pipelines, audit, JIT, glibc dynamic).
• make test-matrix -- cross-checks elfuse (aarch64), QEMU (aarch64), and elfuse (x86_64-via-Rosetta) on overlapping corpora, with per-host baselines for the Rosetta branch.
• make verify and make verify-mutants: the Frama-C WP proof targets, and the assertion that each of them rejects a known-broken source. Run the two together, and not beside a timing lane, because both fan out and the timed lanes fail under the load they create.

The rule for contributors is simple: match the validation depth to the subsystem you changed. Procfs, process state, dynamic linking, and debugging typically warrant more than make check. Touching the Rosetta path additionally requires make test-rosetta-all so the acceptance audit catches new divergences from the documented Rosetta-internal failures. See testing.md for the full target list, per-host baseline scheme, and the validation-by-change-type table.