Absence of KV Cache Eviction / Tier Management in Opal

[leipanhz]

When reviewing KV tiering in Opal, I didn't find explicit KV cache eviction policies. There are currently no DELETE or REMOVE operations implemented. Instead, OpalStorageManager handles tiering (CPU --> Local NVMe --> Distributed FS) by spilling over based on capacity: when a tier reaches full capacity, new cache is written in the next tier.

This mechanism differs significantly from vLLM's Tiering Manager (FS connector). Cache placement entiredly depends on write order rather than access frequency, and it will introduce performance gap when benchmarking Opal against other runtimes.

[animeshtrivedi]

This is indeed the issue. The tiering logic is very primitive. We are working on pushing the LRU (and friends) related policies, together with the GPU HBM kvcache implementation. I will check if we push it this week. Thanks for the comment.

[animeshtrivedi]

#25

there is a pending PR, this should be merged soon. Hopefully this should help