zephyr-cp: BLE bring-up failure after warm reset is not reproducible — mechanism real, trigger unexplained #25

Activity

1. tyeth commented on Sep 10, 2026

   

   tyeth

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Independent corroboration: 57/57. A separately written harness reached the same result before its session was interrupted.

   That harness recorded one NVM record per boot with reset reason, pre-state and elapsed time, and idled 15 s before each forced bring-up attempt — a different design from the loops in the issue body. Its log ends at:

   PROBE result n=12 rr=1 pre=1 OK dt=0ms addr=ac62b767cf2c
   

   12/12 OK, with zero OSError, zero "Hardware error, hardware code: 0", and zero asserts or halts anywhere in its 55 KB console log. Combined with the 45 boots already recorded here, that is 57 consecutive successful bring-ups on the unfixed image across two independent harnesses. Against the originally reported ~19% failure rate, p ≈ 2e-5.

   It had just started a second OpenOCD server (telnet + gdb) to begin the persistent-debugger arm when the session ended, so it never reached its own conclusion or posted to this issue.

   A better explanation of the severity, from that session's own work

   Its commit message identifies why any bring-up failure looked like a bricked board, and it is not what this issue's body assumed:

   pico-sdk's assert() is live in the Zephyr build of the cybt driver. It lowers to __assert_no_args() → abort() → kernel panic, and CONFIG_ASSERT governs only Zephyr's __ASSERT, so it does not disarm them. Three of those asserts sit on the bring-up path in cyw43_btbus_init() — after cybt_wait_bt_ready(), cybt_init_buffer() and cybt_wait_bt_awake() — and each already has an error return behind it. So a failing bring-up panics the kernel and kills USB before the code written to report the error can run.

   That is a genuine defect independent of frequency: the correct outcome is a retryable OSError in Python, not a panic. It is also distinct from CONFIG_BT_ASSERT, which is the Zephyr-side assert noted elsewhere in this project — there are two separate assert paths on this path.

   Filed as tyeth/hal_rpi_pico#5, which is deliberately scoped to the panic-vs-errno behaviour and makes no reliability claim. It is the preferable half of tyeth/hal_rpi_pico#4, whose base-address rework was written under the degradation theory this issue rejects.

   🤖 Generated with Claude Code
2. tyeth commented on Sep 10, 2026

   

   tyeth

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Fault-injection results, and a finding that undercuts the documented mechanism.

   Forced cybt_reg_read(WLAN_RAM_BASE_REG_ADDR) to return 0 while reporting success on a Pico 2 W, with bring-up forced from Python (adapter.enabled = False → pause → = True; at boot the adapter is already enabled, so enabled = True alone is a no-op that never reaches cybt_init_buffer — my first attempt at this test measured nothing for exactly that reason):

   image	base=0 injected	outcome
   baseline integration-pico2w-ble	yes	BLE works — REINIT OK ac62b767cf2c
   tyeth/hal_rpi_pico#5 (asserts → errors)	yes	BLE works — identical
   #4 + #5	no	BLE works, no regression
   #4 + #5	yes	kernel fatal halt, USB lost

   The dead board, read over SWD: pc = 0x10089522 → arch_system_halt (kernel/fatal.c:30), CFSR and HFSR both zero — a software k_fatal_halt, not a CPU fault.

   This is evidence against the mechanism recorded in this issue. The body states that a zero base makes every BTSDIO ring address 0+offset, so the first HCI command goes to backplane address 0 and the controller never responds. On this hardware, baseline tolerated an injected zero base and completed bring-up with the correct OTP address. So either the injected read is not on the re-init path in that build, or a zero base is survivable further down. Unexplained, and it means the "base=0 is fatal if trusted" premise is not demonstrated here.

   Consequences recorded on the PRs: #5 is behaviourally neutral and safe to land; #4 is held, because in the only case where it diverges from baseline it produced a dead board.

   Incidental finding, unrelated to either PR but relevant to the collector work: after enabled = False → True, the adapter comes back with a random static address (d93c1d49a2d1) instead of the OTP-derived ac62b767cf2c it has at boot. A hub whose BD_ADDR changes across a re-init cannot be found by nodes that cached it. Worth its own issue if the collector design depends on a stable address.

   🤖 Generated with Claude Code
3. tyeth commented on Sep 10, 2026

   

   tyeth

   on Sep 10, 2026

   OwnerAuthor

   More actions

   Close-out of the open questions (bench session 2026-09-10, afternoon)

   Everything below was measured on the Pico 2 W (probe E6633861A3303E38), with the Zephyr console captured on the probe UART and the post-mortems taken over SWD with arm-none-eabi-gdb against the exact ELF that was running.

   1. The zero base is real, normal, and not fatal — the mechanism section is wrong

   WLAN_RAM_BASE_REG_ADDR (0x18000d68) reads 0 natively on this board, no injection needed. The #4+#5 image without any injection logged, on every bring-up:

   [CYW43] cybt_init_buffer: WLAN_RAM_BASE unusable (0x00000000) after 50 reads
   [CYW43] hci_open(): cybt_init_buffer failed (0xb4)
   

   That is 50 genuine backplane reads, 1 ms apart, all 0. Reading RAM over SWD on the pristine image after a bring-up gives the same picture (fw_pristine.elf, addresses from the ELF):

   wlan_ram_base_addr @0x20007e4c = 0x00000000
   buf_info           @0x20007e30 = {0x0, 0x2000, 0x2004, 0x1000, 0x2008, 0x200c}
   

   So the BTSDIO rings live at backplane 0x0000..0x200c. The injection ("force the read to 0") changes nothing, which is why baseline "tolerated" it. The body's chain "base reads 0 → rings at 0+offset → first HCI command to address 0 → timeout" does not describe this hardware: base 0 is the working configuration.

   2. The "forced re-init" stimulus never reaches cybt

   common_hal_bleio_adapter_set_enabled(false) in ports/zephyr-cp/common-hal/_bleio/Adapter.c deliberately does not call bt_disable() (it only stops scan/advertising and flips a flag; the comment says "Keep the controller running"). set_enabled(true) then sees bt_is_ready() and returns. Even a real bt_disable()/bt_enable() would not re-download: cyw43_bt_ensure_up() returns early on cyw43_bt_state.bt_loaded. The injected read therefore executes exactly once per boot, inside the first import _bleio. Every "REINIT OK" recorded earlier measured a no-op.

   3. Baseline bring-up failed 3/3 today, all with the same signature

   import _bleio from the REPL on 10.3.0-59-gfb700f76bc-dirty, after an SWD reset run (no power cycle), at uptimes 9 s, 48 s and 69 s:

   BT FW download, version = CYW4343A2_001.003.016.0065.0000_Generic_SDIO_37MHz_wlbga_BU_RPI_dl_signed
   [CYW43] cybt_get_bt_buf_index: out-of-range index (attempt 1/8), re-reading   (1 to 7 repeats)
   <err> bt_hci_core: Hardware error, hardware code: 0                              (81x, in two of three runs)
   ASSERTION FAIL [err == 0] @ WEST_TOPDIR/zephyr/subsys/bluetooth/host/hci_core.c:483
           Controller unresponsive, command opcode 0x1009 timeout with err -11
   <err> os: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 0
   

   gdb on the halted board: arch_system_halt ← k_sys_fatal_error_handler ← z_fatal_error(reason=3) ← z_arm_fatal_error ← z_arm_svc (that is k_oops() from BT_ASSERT_MSG), exception-frame pc 0x10013f54 = bt_hci_cmd_send_sync+164, CFSR=0 HFSR=0. RAM at that point: bt_dev.flags=0x1 (ENABLE, never READY), bt_dev.id_count=0, bt_loaded=1, buf_info as above.

   This contradicts the "0/68 at ≥15 s" and "57/57" figures recorded here. Only warm resets were available to me; the CYW43 was never power-cycled during the session, and the bench had the second probe and the Pico W attached and running. I am not claiming a cause for the difference, only that the failure reproduces today without a debugger halt and independent of bring-up timing.

   4. Why the console looked dead (Item 3)

   Two independent causes, both now fixed in procedure:

   • The Raspberry Pi Debug Probe's UART bridge only forwards while the host asserts DTR. serial_bridge.ps1 (which worked) sets DtrEnable; uartboot.ps1, uart.ps1, reset_capture.ps1 and dualcap.ps1 (which all saw 0 bytes) do not. With DTR set, a reset produced *** Booting Zephyr OS build 3943b1cd7f47 *** and the full fatal dump on COM48.
   • COM48 = probe E6633861A3303E38 = Pico 2 W console (verified by the banner after a probe-1 reset). COM53 = probe E6654854576EB830 = Pico W console (its uptime was 3 h 42 min).
   • When no host holds the CDC port open, udc_rpi_pico: Endpoint 0x83 busy floods the console at ERR level and eats the 1 KB deferred log buffer. Keep a CDC reader attached during tests.

   Recipe (PowerShell): $p = New-Object System.IO.Ports.SerialPort 'COM48',115200,'None',8,'One'; $p.DtrEnable=$true; $p.Open() — then read. An RTT variant (CONFIG_USE_SEGGER_RTT/RTT_CONSOLE/LOG_BACKEND_RTT) also builds cleanly for this board (C:\temp\pico2w\rtt_inj_pr4plus5.elf) but was not needed once DTR was set.

   Consequences for the PRs are on tyeth/hal_rpi_pico#4 and #5; the enable-path defect that produces the random BD_ADDR is filed separately.

   🤖 Generated with Claude Code