weight cache: key daemon paths by GPU UUID

[TarangKhanna]

Motivation

Weight cache daemon discovery currently uses distributed rank in the Unix socket and ready file paths. Rank identifies a process within one distributed job, not a physical GPU across independent jobs.

As a result, two jobs on the same host can use the same daemon path even when their ranks run on different GPUs. For example, two TP jobs can both contain rank 0, allowing one job to discover the other job's daemon.

Change

Key daemon socket and ready file paths by physical GPU UUID instead of distributed rank.

Discovery and weight compatibility remain separate:

• Physical GPU UUID identifies the daemon associated with the GPU used by an engine rank.
• CacheConfig continues to validate that the daemon holds compatible weights before CUDA IPC import.

This preserves the existing failure semantics. If no daemon exists for the GPU, the existing fallback behavior still applies. If a daemon exists but its cached weights are incompatible, the client still reaches the existing explicit CacheConfig mismatch instead of treating the cache as absent.

The launcher already computes the local GPU id for each daemon, so it can derive the corresponding GPU UUID while preserving the existing stale file cleanup, readiness polling, timeout, and process liveness flow.

Scope

The change is intentionally limited to node local daemon discovery identity.

Config matched registry discovery, configurable runtime directories, namespaces, cross node discovery, peer copy, admission control, and cache eviction remain separate roadmap work in #33522.

Validation

• test/registered/unit/model_loader/test_weight_cache_protocol.py: 31 passed.
• Unit tests verify that different physical GPU UUIDs produce different socket and ready paths, independent of logical rank.
• Unit tests verify that automatic discovery uses the caller's physical GPU UUID rather than logical rank or an implicit GPU 0.
• Existing CacheConfig compatibility checks remain unchanged and continue to detect incompatible cached weights.
• TP1 CUDA E2E on A40 and TP2 CUDA E2E on 2x H100 SXM GPUs: daemon startup, automatic client discovery, weight import over CUDA IPC, and generation all passed.

---

CI States

Latest PR Test (Base): ❌ Run #32852938143
Latest PR Test (Extra): ❌ Run #32852937767
Latest PR Test (AMD ROCm 7.2): ❌ Run #32852937999

[liusy58]

@TarangKhanna Is this PR ready for review yet? Can we limit the scope of the codebase it modifies?

[TarangKhanna]

@TarangKhanna Is this PR ready for review yet? Can we limit the scope of the codebase it modifies?

@liusy58 Not quite yet. I’m doing one final pass to reduce the diff. The change is broader than just changing the socket key because moving from rank based identity to GPU UUID plus config also means the old rank based ready file can no longer provide the ownership/readiness coordination we need. The new path uses an identity scoped lock for ownership and the socket itself for readiness. I’m trimming the remaining lifecycle and wiring changes to only what’s required to preserve those, then I’ll mark it ready once the diff is tighter.

[liusy58]

@liusy58 Not quite yet. I’m doing one final pass to reduce the diff. The change is broader than just changing the socket key because moving from rank based identity to GPU UUID plus config also means the old rank based ready file can no longer provide the ownership/readiness coordination we need. The new path uses an identity scoped lock for ownership and the socket itself for readiness. I’m trimming the remaining lifecycle and wiring changes to only what’s required to preserve those, then I’ll mark it ready once the diff is tighter.

Thanks for your contribution. We’d like to keep the scope of the changes as tight as possible.

[TarangKhanna]

@liusy58 Not quite yet. I’m doing one final pass to reduce the diff. The change is broader than just changing the socket key because moving from rank based identity to GPU UUID plus config also means the old rank based ready file can no longer provide the ownership/readiness coordination we need. The new path uses an identity scoped lock for ownership and the socket itself for readiness. I’m trimming the remaining lifecycle and wiring changes to only what’s required to preserve those, then I’ll mark it ready once the diff is tighter.

Thanks for your contribution. We’d like to keep the scope of the changes as tight as possible.

I agree. I’ll trim this back to the discovery change and keep the existing startup behavior where possible. The remaining changes will focus on identifying daemons by physical GPU UUID plus cache config, with only the minimal loader wiring needed for that.

[TarangKhanna]

@liusy58 Ready for review now. I trimmed the change further to use physical GPU UUID only for daemon socket and ready path identity, while leaving the existing CacheConfig compatibility check and startup lifecycle unchanged. I also validated the final revision with the unit suite plus TP1 and TP2 CUDA IPC end to end tests. Thanks for the scope guidance.

[liusy58]

@TarangKhanna Will review soon.