Skip to content

serve: KVCache::new and OptimizedKVCache::new commit num_layers-1 full layers at construction (the form #4778 removes from StreamingKVCache) #4783

Closed as not planned

Description

@noahgift

crates/aprender-serve/src/inference/kv_cache.rs lines 61-62 (KVCache::new) and 255-256 (OptimizedKVCache::new) build
vec![vec![0.0; max_seq_len * hidden_dim]; num_layers] for both K and V.

The inner zero vector is allocated zeroed and untouched, but vec![x; num_layers] makes the other num_layers - 1 layers
with Clone, which copies them, so their memory is committed before the first token is written.

StreamingKVCache had this form. Three of its memory-bound tests ran out of memory in CI, were ignored under FLAKE-0, and
#4778 makes that allocation lazy.

Evidence. The independent review of #4778 (noted in its PR body, not in its diff). Both sites re-read on main at
75e7b36: they match; the file was last changed in a8c6807.

Fix. The #4778 lazy form, or build each layer separately ((0..num_layers).map(|_| vec![0.0; n]).collect()), so a
layer stays untouched until it is written. Add a test in the style of #4778's.

Scope. No test is ignored for these two types today, so this is not on the 0.70.2 release path. Work starts after
#4778 merges.

Owner aprender-wmainred. Refs #4778.

Activity

  1. added this to the backlog milestone on Oct 8, 2026
  2. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  3. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5321 (apr memory footprint: serve Qwen3.5 within 8 GB unified-memory ARM64 CUDA (Orin)), split out of #5305 by theme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions