Support stateful KV cache for multimodal decoder

[Anirudh Swaminathan (Anirudh-Swaminathan)]

The multimodal DecoderState previously hardcoded a DefaultKeyValueCache, which prevented stateful OpenVINO VLM decoders (whose KV cache is managed inside the ORT session) from working.

• Replace the fixed DefaultKeyValueCache with a CreateKeyValueCache factory result (std::unique_ptr), so a ModelManagedKeyValueCache is selected for stateful models; guard all Add/Update call sites for the null (no-cache) case.
• Add DecoderState::RewindTo to reset position inputs, KV cache, and recurrent state, mirroring DecoderOnly_State.
• Add MultiModalPipelineState::RewindTo to forward to the decoder state and reset is_prompt_ so continuous decoding re-enters the prompt stage.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates the multimodal decoder/pipeline state handling so stateful decoders (e.g., OpenVINO VLM with session-managed KV cache) can work correctly, and so generator rewind/continuous decoding can reset multimodal state consistently.

Changes:

• Replaces DecoderState’s hardcoded DefaultKeyValueCache with a std::unique_ptr<KeyValueCache> created via CreateKeyValueCache(), and adds null guards for KV cache call sites.
• Adds DecoderState::RewindTo() to rewind position inputs, KV cache, and recurrent state.
• Adds MultiModalPipelineState::RewindTo() to forward rewind to the decoder and reset is_prompt_ when rewinding to index 0.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/models/multi_modal.h	Adds RewindTo() overrides and switches decoder KV cache storage to a std::unique_ptr<KeyValueCache>.
src/models/multi_modal.cpp	Implements KV cache factory usage + null-guards, and adds rewind implementations for decoder and multimodal pipeline.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/models/multi_modal.cpp:666

• After switching DecoderState::kv_cache_ to a std::unique_ptr, there is still a remaining call site using the old value-type API: DecoderState::RunPrefillWithChunking() calls kv_cache_.Update(next_indices, length) (multi_modal.cpp:754). This will not compile (unique_ptr has no Update()), and it also bypasses the intended null-cache guard for models where CreateKeyValueCache() returns nullptr.

      model_{model},
      position_inputs_{CreatePositionInputs(*this, sequence_lengths, model_.config_->model.decoder.inputs.attention_mask)},
      kv_cache_{CreateKeyValueCache(*this)},
      recurrent_state_{CreateRecurrentState(*this)} {

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/models/multi_modal.cpp:665

• Changing kv_cache_ to a pointer leaves RunPrefillWithChunking calling kv_cache_.Update(...) at line 754. This no longer compiles; that call also needs the same null guard and kv_cache_->Update(...) used by the other update paths.

      kv_cache_{CreateKeyValueCache(*this)},

src/models/multi_modal.cpp:945

• A full rewind cannot restore the multimodal prompt stage by toggling this flag alone. The first prompt destroys vision_state_/speech_state_ (lines 927-928), while MultiModalFeatures::Update(false) replaces nonempty feature inputs with empty tensors. Consequently, RewindTo(0) followed by a new prefill silently runs without the original image/audio features. Preserve reusable encoder outputs, or recreate the encoder states and rebind the original extra inputs/features before setting is_prompt_.

  if (index == 0)
    is_prompt_ = true;