Add hardware-in-the-loop and fault injection tests - #282
Open
usmanmehmood55 wants to merge 3 commits into
Open
Conversation
Docs updated, implemented 4 more tests, improved existing tests Fixes for CI Added Android to HIL tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a hardware-in-the-loop (HIL) test suite for
universal_bleand uses it to harden the Windows implementation.The fixture is a physical nRF52 DK running purpose-built Zephyr firmware. The firmware lives in the separate universal_ble_hil_firmware repository.
The Windows suite currently has 57 working tests:
The host controls the fixture over BLE. USB serial is used for firmware logs.
What the tests cover
The baseline suite covers:
The FIT suite arms the firmware with a fault plan before starting an ordinary BLE operation. The fixture can then:
Most tests also verify recovery. After the injected failure, the test performs another read, write, reconnect, or subscription. This catches cases where the original call fails correctly but leaves the connection or operation queue in a broken state.
These tests go through the complete path: Dart API, operation queue, Pigeon, Windows C++, WinRT, Windows Bluetooth stack, the radio link, and finally the Zephyr GATT server. This gives us coverage which mocks and software-only tests cannot provide.
Windows implementation improvements
The HIL work also led to a larger review of the Windows implementation. Production changes are kept separate from the test fixture and include:
GattServicesChangedis handled. Windows rebuilds the GATT map safely, rejects stale refresh results, waits for active operations, and restores existing subscriptions.readDescriptorAPI also returns the descriptor value instead ofvoid.value.buffercan no longer expose unrelated bytes from the surrounding Pigeon message.The deterministic native lifetime logic also has software-only tests, which now run in the Windows CI job.
Bugs found
The suite had caught several real issues:
ValueChangedafter the CCC enable operation had completed.value.buffercould include unrelated envelope data.The stale connection, stale completion, descriptor, service refresh, and notification timing tests now cover these paths against the physical fixture.
Running the suite
These are manual hardware tests. They are not intended to run in normal pull-request CI because they require an nRF52 DK and exclusive access to a Bluetooth adapter.
Generated Android, Web, and Windows runner files are not committed. They can be recreated with
flutter create, which keeps the PR focused on the actual fixture, tests, and production changes.The Windows suite should be run when changing:
The Web runner provides an interactive path for the portable baseline behavior. The suite also contains skipped placeholders for faults which still need firmware support and proper assertions. Those can be implemented gradually as the fixture develops.