You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
serve: KVCache::new and OptimizedKVCache::new commit num_layers-1 full layers at construction (the form #4778 removes from StreamingKVCache) #4783
crates/aprender-serve/src/inference/kv_cache.rs lines 61-62 (KVCache::new) and 255-256 (OptimizedKVCache::new) build vec![vec![0.0; max_seq_len * hidden_dim]; num_layers] for both K and V.
The inner zero vector is allocated zeroed and untouched, but vec![x; num_layers] makes the other num_layers - 1 layers
with Clone, which copies them, so their memory is committed before the first token is written.
StreamingKVCache had this form. Three of its memory-bound tests ran out of memory in CI, were ignored under FLAKE-0, and #4778 makes that allocation lazy.
Evidence. The independent review of #4778 (noted in its PR body, not in its diff). Both sites re-read on main at 75e7b36: they match; the file was last changed in a8c6807.
Fix. The #4778 lazy form, or build each layer separately ((0..num_layers).map(|_| vec![0.0; n]).collect()), so a
layer stays untouched until it is written. Add a test in the style of #4778's.
Scope. No test is ignored for these two types today, so this is not on the 0.70.2 release path. Work starts after #4778 merges.
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.
Activity
noahgift commentedon Oct 14, 2026
merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.
noahgift commentedon Oct 14, 2026
Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.