SCHED-1741: Make sure we have Active or Passive check to drain nodes where there are empty libnvidia.so files (amended)

[IronhandedLayman]

Problem

Intermittently, Nvidia libraries on worker nodes can be mounted incorrectly, leading to an error state where the library exists but is either not loadable or has a size of 0. This causes problems work ML workloads running on said workers.

Solution

This patch creates an active check that confirms the existence of Nvidia libraries and their ability to load. If those libraries cannot be loaded, then the worker node drains, thus preventing future workloads from running on that node until the issue is resolved.

Testing

This patch was tested on an existing 2-node Soperator cluster running 3.0.2, both in positive (libraries existing, passed check) and negative tests (library did not exist, check failed, node drained). All cases worked as expected. Even though this is tested against a 3.0.2 soperator cluster, it is expected to work with all versions, including latest.

Release Notes

Adding a new health check to existing clusters may cause worker nodes to drain, because libraries were missing that were unexpected, or maybe not used by previous workloads (but given the list of libraries, that is a vanishingly small probability). While this is intended behavior, it may be surprising.

[rdjjke]

All nodes drained with [node_problem] prefix are being automatically replaced by Soperator automation.
But these drains show issues with hardware, not software.

This check doesn't show hardware issues, so automatic node replacement isn't needed. I suggest to change the prefix to something else, e.g. [software_problem]