Stop the training subprocess on every exit path

[jfrieli]

Motivation

Aborting a training cancels the trainer task, but for executor-based trainers (e.g. classification, yolov5) the python train.py subprocess was left running — still holding the GPU. The CancelledError from abort() propagated past _train's only cleanup (except TrainingError), so nothing terminated the executor. This was surfaced by the loop's new abort_training flow (zauberzeug/loop#346 + #80): once a node is on 0.20.0 and extends the executor-based TrainerLogic, abort orphans the process.

Implementation

• Move the executor cleanup in TrainerLogic._train out of the except TrainingError block into a finally, so the training subprocess is terminated on every exit path (normal return, TrainingError, CancelledError/abort, and any unexpected exception)
• Use asyncio.shield(self.executor.stop_and_wait()) so the terminate+wait completes even while the task is being cancelled
• Add test_abort_during_training_kills_executor: starts a real training subprocess, aborts mid-run, asserts executor.is_running() becomes False (no orphan)

Notes / risks

• stop() (graceful) and abort() remain distinct: finally only guarantees the subprocess is dead; whether _train returns normally (→ upload) or with CancelledError (→ ReadyForCleanup, discard) is unchanged. Covered by the existing test_stop_during_training_uploads_model / test_abort_during_training.
• The TrainingError path is behavior-preserving (executor still killed, previous_state still reset) — only the internal ordering changed.
• Relevant train-state tests pass locally. The SocketIO-dependent integration tests could not be run locally (no mock loop running here) and fail identically on the unmodified base; they should be verified in CI.

---

⬛ claude-opus-4-8[1m]

[jfrieli]

@denniswittich @NiklasNeugebauer
Ready to review