[docs] Slurm: fix multi-node example, add single-node and offline batch examples

[holtwood]

Motivation

Contributes to #3244.

The existing multi-node example expands $SLURM_NODEID in the batch shell, before srun creates its per-task environments. Both launches can therefore receive the same node rank. The existing readiness loop also checks only whether a TCP port is open.

This PR covers basic multi-node serving, single-node serving, and offline batch inference. #29991 has a separate scope around PD disaggregation and Apptainer.

Changes

• Evaluate $SLURM_PROCID inside each srun task and use Slurm %n patterns for per-node logs.
• Use configurable HTTP and distributed-init ports, a bounded /health check, and early process-exit detection.
• Add a single-node serving job with cleanup and an SSH forwarding hint.
• Add an offline job that writes a real Python file with a __main__ guard, as required by the Engine multiprocessing setup.
• Document the log-directory prerequisite and cluster-specific scheduler, network, port, and model-cache settings.

Validation

• bash -n on all three batch scripts.
• mint validate.
• mint broken-links.
• Parsed and mock-executed the offline Python example, including Engine context cleanup.
• Checked task-side rank expansion with SLURM_PROCID=0 and SLURM_PROCID=1.
• Cross-checked the Engine spawn setup, context-manager support, /health behavior, and referenced server arguments against the PR base revision.

I did not run these examples end to end on a Slurm GPU cluster. The #SBATCH resource values, partition name, network interfaces, and port availability remain site-specific.

---

CI States

Latest PR Test (Base): ✅ Run #32676790313
Latest PR Test (Extra): ❌ Run #32676790204
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.

[holtwood]

/tag-and-rerun-ci