Problem: Currently, the request batch creation is suboptimal in two ways: 1) under memory pressure, active LLM requests are kicked out of the worker processing batch, then immediately replaced with new LLM requests, leading to KVCache trashing and lack of progress; 2) the batch construction algorithm is O(N), N = number of pending requests (waiting / pending + active). This leads to much slower simulation in cases where a worker becomes overloaded.
Desired functionality:
- Ensure the batch construction matches the stated goals: 1) prioritize decode vs prefill requests; 2) prioritize active requests vs. waiting requests; 3) in case of memory pressure, evict a single request at a time.
- Ensure batch construction is an O(K) algorithm, where K is the max number of active requests.
Problem: Currently, the request batch creation is suboptimal in two ways: 1) under memory pressure, active LLM requests are kicked out of the worker processing batch, then immediately replaced with new LLM requests, leading to KVCache trashing and lack of progress; 2) the batch construction algorithm is O(N), N = number of pending requests (waiting / pending + active). This leads to much slower simulation in cases where a worker becomes overloaded.
Desired functionality: