Fix diagnostics stop lifecycle

[lzy-edu]

PR Category

Train

PR Types

Bug Fixes

PR Description

This PR fixes the diagnostics stop lifecycle for GPU progress heartbeat monitoring.

• Persist the active diagnostics run ID atomically when a training run starts.
• Reuse the active run ID during action=stop, instead of generating a new ID that points to a different heartbeat directory.
• Stop the node-zero heartbeat monitor before terminating training workers.
• Coordinate nonzero nodes with node zero before stopping their local workers and GPU health collectors.
• Validate recorded diagnostic PIDs before terminating processes, use bounded graceful shutdown with a SIGKILL fallback, and clean stale lifecycle files safely.
• Preserve the original training exit code and clean the active run ID after a normal exit.
• Add unit tests covering run ID persistence, stop ID reuse, missing active state, multi-node stop ordering, PID validation, and generated run/stop scripts.