Driver tests based on `sysfs/bus/*/devices` aren't future proof

[r-c-n]

There are quite a few KernelCI regression reports from time to time that point to the same basic bootrr tests based on assert_device_present. These tests rely on sysfs nodes that aren't meant to be used as a stable ABI, so some changes like driver or device file renames and DT re-structuring can make these tests fail when, in fact, they shouldn't be considered as regressions.

Here's a recent example: https://groups.io/g/kernelci-results/message/39167 (this one is based on the KernelCI fork of bootrr)
Introduced by this patch: https://lore.kernel.org/all/20230120092053.182923-20-angelogioacchino.delregno@collabora.com/

Fixing these tests to comply with the current changes or to run different checks depending on the kernel version seems more like a temporary workaround than a proper fix.

Do you think there'll be a better way to test this type of things in the future? Any suggestions from your side?

Here's a related discussion with a bit more background on this: https://lore.kernel.org/lkml/5095423.31r3eYUQgx@diego/

Thanks

[r-c-n]

Here's another similar failure, this time after a patch affecting PCIe enumeration: https://groups.io/g/kernelci-results/message/38558
Patch: http://lore.kernel.org/all/1673847684-31893-12-git-send-email-hongxing.zhu@nxp.com/

[nfraprado]

I have a series on the kernel mailing list adding a kselftest that checks for device probe based on the Devicetree currently in use: https://lore.kernel.org/all/20230828211424.2964562-1-nfraprado@collabora.com . That should make most assert_device_present and assert_driver_present tests unnecessary. Since this DT kselftest relies on the DT and stable properties (compatibles) it should also be future-proof.

However that kselftest doesn't cover ACPI-based platforms, nor devices probed from discoverable busses like PCI and USB. For PCI and USB devices, checking for them statically in bootrr still seems to be the way to go. For that reason I have just opened a PR: #60 introducing a helper for testing USB devices only relying on documented ABI, as to make the tests future-proof. I intend to do the same for PCI.

@andersson Let me know if you have any feedback on the kernel series or the USB bootrr test.