usb: fix SuperSpeedPlus passthrough and reconnect saved devices

[giovariot]

Summary

QEMU USB passthrough fails for USB 3.2 Gen 2 (10 Gbps) devices — the external SSDs and NVMe enclosures that most users hit in #7530 — with the guest rejecting the device as Invalid ep0 maxpacket: 9. This change fixes the speed negotiation that causes it, unmounts the host volumes of a mass storage device before it is redirected so macOS never loses a mounted disk, and reconnects a saved device automatically when it appears again (for example after a firmware update restarts it).

Resolves #7530, #2728, #3975, #3995, #5154, #6291, #6721 and #6764.

Root cause

UTM's QEMU passthrough does not hand the device to QEMU directly: it uses SPICE usb-redir with libusb running in the app process (CocoaSpice/usbredirhost). The entitlements are already in place (com.apple.security.device.usb, com.apple.vm.device-access), so macOS can release the device from its kernel drivers; the failure is in the speed negotiation:

1. libusb reports a USB 3.2 Gen 2 (10 Gbps) device as LIBUSB_SPEED_SUPER_PLUS (kUSBDeviceSpeedSuperPlus).
2. usbredir 0.14.0 has no case for that value in usbredirhost_send_device_connect(), so it sends usb_redir_speed_unknown.
3. QEMU maps unknown speed to full speed (hw/usb/redirect.c) while still forwarding the device's SuperSpeed descriptor, whose bMaxPacketSize0 is 9 (the SuperSpeed encoding, 512 bytes).
4. The guest enumerates the device at a non-SuperSpeed link and rejects the descriptor: Invalid ep0 maxpacket: 9, then unable to enumerate USB device.

This matches the reports: USB 2.0 and USB 3.0 (5 Gbps) devices keep working, 10 Gbps SSDs and NVMe enclosures fail, and a USB 2.0 hub "fixes" it by forcing the host device to enumerate at high speed. The usb-redir protocol has no speed above SuperSpeed, so these devices are presented at 5 Gbps instead of 10 Gbps.

Changes

• patches/usbredir-0.14.0.patch (new): present LIBUSB_SPEED_SUPER_PLUS devices as SuperSpeed, so the guest gets a link and descriptors that match.
• patches/qemu-10.0.12-utm.patch: the existing bMaxPacketSize0 fix-up only ran when QEMU had marked the device as SuperSpeed. Run it whenever the device is presented at a non-SuperSpeed link, which also covers speeds libusb cannot report (for example 20 Gbps kUSBDeviceSpeedSuperPlusBy2 devices are reported as unknown speed).
• Services/UTMUSBDeviceUnmount.swift (new, macOS): before redirecting a device, unmount its mounted volumes with DiskArbitration so the capture does not tear the storage driver away from macOS with volumes still in use. If a volume is busy the connection is refused with a clear message instead of risking data loss. The helper also maps the "device still in use" errors from libusb to a message that tells the user what to do.
• Reconnect a saved device automatically (VMDisplayQemuDisplayController): a device in the saved auto-connect list is connected again without prompting when it is attached while the virtual machine is running. This covers devices that reset themselves, for example during a firmware update, and come back a few seconds later.

Related issues

These issues are less specific or describe a different problem and are only possibly related: #3322, #3375, #4167, #5664, #7288.

Testing

I confirmed the report with a synthetic test: without access to the com.apple.vm.device-access entitlement a local build cannot capture the physical device (on macOS 27 the device is marked NeedsDeviceAccessEntitlement = Yes, so even root cannot), which makes a hardware test impossible. The tests were run on macOS 27.0, MacBook Air M4 (Apple Silicon).

The reporter's ASM2362 enclosure (174c:2362) was used to confirm the reported values:

• libusb: speed=5 (LIBUSB_SPEED_SUPER_PLUS), bcdUSB=0x0320, bMaxPacketSize0=9
• IORegistry: USBSpeed=5, UsbLinkSpeed=10000000000
• binary comparison of the pinned usbredirhost: the current one handles only speeds below 5 (so SUPER_PLUS falls through to "unknown"); the patched one handles speeds below 6 and maps SUPER_PLUS to super.

End-to-end in QEMU, with a synthetic usb-redir device that mirrors the SSD descriptors (bcdUSB 0x0320, bMaxPacketSize0 = 9, mass storage interface) and an Alpine Linux guest on the serial console:

usbredir	QEMU	Guest result
current (unknown speed)	current	fails: new high-speed USB device, Invalid ep0 maxpacket: 9, unable to enumerate USB device
patched (super speed)	current	enumerates: new SuperSpeed USB device number 2, New USB device found, idVendor=174c, idProduct=2362
current (unknown speed)	patched	enumerates: new high-speed USB device, descriptor accepted (the fix-up rewrites bMaxPacketSize0 to 64)

The physical device itself could not be captured on this machine (see above), and the automatic reconnection path also needs a device that resets while a virtual machine is running, which was not exercised here.

Testing: Tested by the author on macOS 27.0, MacBook Air M4 (Apple Silicon), assisted by the AI agent (DeepSeek V4.1 Flash via OpenCode) which set up and ran the diagnostics and the QEMU tests under the author's supervision. The author acknowledges that this change has been tested and/or reviewed by a human in accordance with UTM's AI contribution guidelines.

[osy]

If you don't have the entitlement you can still test with sudo (QEMU command line only) or with SIP off (get entitlement without needing a profile). Either way we do not accept untested PR as per CONTRIBUTING.md.

[giovariot]

As requested, I ran the tests physically with the hardware from the report.

Setup: MacBook Air M4, macOS 27.0.1, the ASM2362 USB 3.2 Gen 2 enclosure (174c:2362, 4 TB NVMe), guest Fedora 38 aarch64 (QEMU, HVF). The test build carries the standard project signing (no restricted entitlements), so UTM was started as root (sudo) to capture the device.

• Current main reproduces the report: usb 3-3: new high-speed USB device, Invalid ep0 maxpacket: 9, then unable to enumerate USB device.
• With this branch the device enumerates at SuperSpeed: usb 4-3: new SuperSpeed USB device number 2 using xhci_hcd; lsusb -t reports Driver=uas, 5000M; the 4 TB disk attaches (sd 1:0:0:0: [sda] 7813971617 512-byte logical blocks: (4.00 TB/3.64 TiB)) and reads at ~199 MB/s (dd if=/dev/sda of=/dev/null bs=1M count=2000), with no uas_eh_abort_handler and no device resets while probing.
• Stopping the virtual machine returns the device to macOS and the disk reappears in diskutil/Finder, instead of staying captured until a physical replug.
• The only remaining messages are the ses enclosure LUN errors (Failed to get diagnostic page 0x1, Failed to bind enclosure -19), which are harmless: the ASM2362 exposes an enclosure LUN that does not respond.

Both attaching and detaching were confirmed on the physical device. The detach behaviour depends on the libusb fix (darwin_cancel_transfer() handling LIBUSB_TRANSFER_TYPE_BULK_STREAM), now included in this PR.