Agent: a crashing long-running daemon must not take down the agent and its running tasks

[almazgimaev]

Context:

The agent runs several long-running daemons as threads (telemetry, ping, GetNewTask/GetStopTask streams, cleaners) and supervises each task in a child container.

Problem:

an unhandled exception in any one of these daemons propagated up and exited the whole agent process. Docker then restarted the agent, and startup cleanup force-removed every running task container — so a transient error in a non-critical daemon (e.g. a telemetry HTTP 500) killed in-flight training tasks and their checkpoints.

Solution:

• every daemon now runs under a resilient supervisor that catches the exception, logs it, and restarts the daemon with exponential backoff instead of crashing the agent.
• The task-stream and health-check loops also skip a single bad message without dropping the connection.
• Telemetry retries are bounded (an earlier unbounded retry blocked the supervisor thread) and shutdown stays clean.