Is your feature request related to a problem? Please describe.
Proceed with tuning even if ClearML (or other trackers) fail.
Currently it crashes if ClearML tracker throws an exception/error.
We ran into this in a situation where the ClearML server's certificate had expired, it was blocking all tuning jobs.
Describe the solution you'd like
Do not crash. Print a warning/error and proceed with tuning.
Describe alternatives you've considered
This can be enabled by a flag if necessary. Example: --ignore-tracker-errors
Additional context
Add any other context about the feature request here.
Is your feature request related to a problem? Please describe.
Proceed with tuning even if ClearML (or other trackers) fail.
Currently it crashes if ClearML tracker throws an exception/error.
We ran into this in a situation where the ClearML server's certificate had expired, it was blocking all tuning jobs.
Describe the solution you'd like
Do not crash. Print a warning/error and proceed with tuning.
Describe alternatives you've considered
This can be enabled by a flag if necessary. Example:
--ignore-tracker-errorsAdditional context
Add any other context about the feature request here.