fix(pd): notify decode of NIXL transfer failures

[zupengwang]

Motivation

NIXL's prefill transfer worker currently records transfer errors only in its local manager. The decode side receives no terminal failure signal and can wait until SGLANG_DISAGGREGATION_WAITING_TIMEOUT (300 seconds by default), unlike Mooncake and MORI.

Fixes #36033.

Changes

• Send a tagged NIXL_TRANSFER_FAILURE control message to every unique decode endpoint when the prefill transfer worker fails.
• Snapshot decode endpoints before publishing the local Failed state, so sender cleanup cannot race away the notification targets.
• Run the decode control listener in the default NIXL configuration and make received failures terminal immediately.
• Ignore malformed and late notifications so a cleared room is not recreated or polluted.
• Keep status delivery best-effort: if a decode endpoint cannot be notified, the existing waiting timeout remains the fallback.

The NIXL success/completion protocol and KV data path are unchanged.

Validation

• Regression-first baseline at d1af3c89233c475fc1bf11939d86787e6cddd58c: the 7 new focused tests failed before the implementation.
• NIXL CPU unit file: 50 passed, including active-room failure, late/malformed messages, endpoint deduplication, partial notification failure, and the transfer-worker exception path.
• Loopback TCP/ZMQ fault injection: 5 independent runs, each with 10 warmups and 100 measured notifications (500 samples total).
  • aggregate median: 0.0911 ms
  • aggregate P95: 0.1129 ms
  • aggregate P99: 0.1234 ms
  • max: 0.1855 ms
• isort 7.0.0 --check-only, ruff 0.15.1 (F401,F821,UP037), black 26.1.0 --check, codespell 2.4.1, check_no_bare_pytest_main.py, check_registered_tests.py, and git diff --check passed.

The unit run used an import-isolated harness because the local environment does not contain every optional SGLang model/quantization dependency. The harness stubs only unrelated top-level imports; the NIXL, common disaggregation, ZMQ, and test modules are the source from this commit.

Risk and checks not run

• Mixed-version deployments degrade safely: an older decode ignores the new tag and retains the existing timeout behavior.
• No full model-serving NIXL PD or multi-GPU test was run. This patch is control-plane-only; the shared host had one available GPU and no IB device, so that test would not add representative NIXL data-plane coverage.
• Upstream CI was not run. Draft PRs are intentionally blocked by the PR gate, and the author is not listed in CI_PERMISSIONS.json.
• No throughput, TTFT, ITL, or GPU-memory improvement is claimed; the measured effect is failure-detection latency only.

---

CI States

Latest PR Test (Base): ❌ Run #32662942920
Latest PR Test (Extra): ❌ Run #32662942889
Latest PR Test (AMD ROCm 7.2): ❌ Run #32662943037