Add hardware-in-the-loop and fault injection tests

[usmanmehmood55]

Summary

This PR adds a hardware-in-the-loop test suite for universal_ble. It uses a physical nRF52 DK as the BLE peripheral.

The Windows suite currently has 42 working tests:

• 15 baseline tests for normal BLE behavior
• 27 fault injection tests (FIT) for peripheral-side errors, awkward timing, disconnects, and recovery

The firmware lives in the separate universal_ble_hil_firmware repository. The host controls it over BLE, and USB serial is used for logs.

What the tests cover

The baseline tests cover:

• filtered scanning
• service, characteristic, and descriptor discovery
• reads and writes with exact payload verification
• writes with and without response
• notifications and indications
• subscription state
• MTU reporting
• peripheral and host disconnects
• reconnects and repeated connection cycles
• concurrent operations

The FIT suite then takes those same operations and deliberately makes the peripheral misbehave in ways still allowed by a normal BLE stack. For example, the peripheral can:

• return ATT read or write errors
• delay an operation beyond the host timeout
• disconnect while a read or write is still pending
• send notifications with gaps, duplicates, or reordered sequence numbers
• disconnect during a notification burst
• keep notifications queued while the host tears down the connection
• repeatedly force disconnect and reconnect cycles
• emit a notification while the CCC subscription write is still completing

The tests also check recovery. A failed operation is normally followed by a clean read, write, reconnect, or resubscription so the test can catch state which remains poisoned after the original error.

Most importantly, these tests exercise the full path against real hardware: Dart API, operation queue, Pigeon channel, Windows C++ implementation, WinRT, Windows Bluetooth stack, and finally the physical radio link. So they should be useful for catching future regressions which are difficult to reproduce with mocks or unit tests alone.

Bugs found

This test setup has caught two real Windows-side problems.

1. During development, notification and disconnect stress exposed the native crash and connection-lifecycle issues fixed in PR Windows: Harden BLE connection lifetime, async callbacks, and notification subscription handling #278. The tests were also used to verify those fixes against an actual peripheral.
2. Windows registered its ValueChanged handler after the CCC enable operation had completed. If a peripheral sent its first notification from the CCC callback, that notification arrived before Windows was listening and was silently lost.

Scope

These are manual hardware tests. They are not intended to run in normal PR CI, but can be extended to work so they run before every minor/major release.

They are useful when changing:

• Windows connection ownership and cleanup
• GATT read or write completion
• subscriptions and native event handlers
• disconnect behavior
• callback lifetime
• operation queueing
• other platform-specific BLE lifecycle code

The suite also contains many skipped placeholders for scenarios which still need matching firmware support and proper assertions. I can implement them gradually as the test fixture develops.

[usmanmehmood55]

@fotiDim are tests of this kind useful here?

As mentioned in the PR, they surfaced actual Windows implementation bugs:

1. Native Windows crash during overlapping notifications, disconnects, and GATT operations. Fixed in PR Windows: Harden BLE connection lifetime, async callbacks, and notification subscription handling #278.
2. First notification lost because ValueChanged was registered after enabling the CCC descriptor.
3. Descriptor operations error handling, and connection-lifetime protection.
4. GATT Service Changed events were not handled, leaving cached services stale.

I have to do these tests regardless for my app, so might be useful to have them as regression tests in major/minor releases.

[fotiDim]

@usmanmehmood55 cool stuff! One of the biggest pain points so far was reproducing obscure Windows issues so there is definitely value in it. Ideally it should be able to run as part of a GitHub Action with a local worker but this can wait. I could order a board and run tests locally prior to release. Do you have a shop to recommend?

Some questions:

• How long do the tests take to run and do they all pass now?
• What would it take to also include Linux?
• Since universal_ble can also act as a peripheral (i.e. advertise) would it make sense to rewrite the test suite so that nRF52 is not needed? Of course testing would require 2 computers or 2 BLE dongles. Note though that selecting a dongle though universal_ble is not yet implemented.

Why was the PR closed?

[usmanmehmood55]

@fotiDim Running this as a GitHub action is the goal, and can be implemented with minor effort, once I finalize the test orchestration.

The firmware can run on any nRF52 (or nRF53, nRF54) board. Usually I get mine from Mouser or DigiKey.

How long do the tests take to run and do they all pass now?

On Windows, it takes 1.5-2 minutes to run the full suite I think. On Android, the same time I think. I haven't measured it yet.

What would it take to also include Linux?

Can't say for sure, I don't have an AMD64 Linux system here. I'll try running them once I find one.

Since universal_ble can also act as a peripheral (i.e. advertise) would it make sense to rewrite the test suite so that nRF52 is not needed? Of course testing would require 2 computers or 2 BLE dongles. Note though that selecting a dongle though universal_ble is not yet implemented.

Two computers can be possible but they would only be able to test the basic functionality like connect, read/write, pair etc. They won't be able to create malformed situations, and precisely timed anomalies that an nRF52 allows me to create, and verify the error handling and recovery. Right now it gives us an independent and agnostic BLE system to test against.

I closed the PR temporarily to prevent CI from running too much, as I was doing lots of changes. I'll re-open it in a bit.

Edit: GitHub won't let me re-open because the branch had force-pushed changes. I've opened a new one with updated contents: #282