Fall back to legacy pipeline when CB model lacks PagedAttention (e.g. Gemma)#4336
Fall back to legacy pipeline when CB model lacks PagedAttention (e.g. Gemma)#4336exzile wants to merge 4 commits into
Conversation
In the async inference path (modelInferAsync, used by KServe gRPC and the C-API async endpoint), the ExecutingStreamIdGuard that holds an infer-request pool slot is std::move-d into the OV completion-callback lambda before inferRequest.start_async() is called. The slot is returned to OVInferRequestsQueue only when that lambda is destroyed, which normally happens when the async completion callback fires. If start_async() throws, the function returns immediately and the completion callback never fires, so the lambda (sole owner of the guard) is never destroyed and the infer-request slot is never returned to the pool. Under long-running concurrent load, repeated start_async failures slowly drain the pool until no idle infer request is available; requests then block and gRPC surfaces ResourceExhausted even though CPU/GPU/RAM are not saturated (matches the symptom class in openvinotoolkit#2871). Clear the callback in both start_async() catch blocks so the captured guard is destroyed and the slot is returned, mirroring the existing "set empty callback to release captures" pattern used after normal completion. The synchronous infer() path is unaffected (it uses a stack-allocated guard with RAII release on all paths). Builds clean (//src:ovms //src:ovms_test); CAPIInference.AsyncErrorHandling passes. The change is confined to the start_async error path. Signed-off-by: exzile <joeypongallo@gmail.com>
Models whose IR exports without a ScaledDotProductAttention op (e.g. Gemma) cannot undergo the SDPAToPagedAttention transformation required by the continuous batching engine. Previously, when the pipeline type was auto-selected, OVMS picked *_CB and then failed hard during LLM node initialization, taking the whole server down with a cryptic SDPAToPagedAttention error (see model_server#4178). Detect this specific failure in the continuous batching initializer and report it via a dedicated status (LLM_NODE_PAGED_ATTENTION_NOT_SUPPORTED), logged at warning level. When the pipeline type was auto-selected, fall back to the legacy pipeline (LM_CB->LM, VLM_CB->VLM). An explicitly requested *_CB pipeline still surfaces the error rather than being silently downgraded. Verified manually with OpenVINO/gemma-4-E4B-it-int4-ov: AUTO now falls back to the legacy VLM servable and serves chat completions instead of failing to start. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
What hardware was used for testing? GPU or CPU? What models and specs? |
|
@mzegla Hardware: CPU only — AMD Ryzen 9 9950X3D (16C/32T), 128 GB RAM, Windows. (The box also has an Arc Pro B70 GPU, but this change was not exercised on GPU — see the note below on why it shouldn't matter.) Software: OVMS built from main (2026.3.0, commit 9ddd45c), OpenVINO / GenAI 2026.3 (genai 2026.3.0.0-3173). Model: OpenVINO/gemma-4-E4B-it-int4-ov (INT4, ~6.1 GB) — the model from #4178. What I verified end-to-end: --pipeline_type AUTO (default) selects VLM_CB and, before this change, hard-fails with: No ScaledDotProductAttention operation observed in the graph, cannot perform the SDPAToPagedAttention transformation (sdpa_to_paged_attention.cpp). That's the gap this PR handles. One model, CPU only. The SDPAToPagedAttention transformation runs on the model graph before device compilation, so the "no SDPA op → fall back" decision is device-independent; I'd expect identical behavior on GPU, but I did not verify it there. |
2 participants
Problem
Models whose OpenVINO IR exports without a
ScaledDotProductAttentionop — e.g. Gemma / gemma4 (Gemma4ForConditionalGeneration) — cannot undergo theSDPAToPagedAttentiontransformation required by the continuous batching engine.With the default
pipeline_type: AUTO, OVMS selects a continuous-batching pipeline (LM_CB/VLM_CB) and then fails hard during LLM node initialization, taking the whole server down with a cryptic error:This is the trap behind #4178 — the documented workaround is to manually set
pipeline_type: VLM, but a user following defaults just gets a dead server.Change
LLM_NODE_PAGED_ATTENTION_NOT_SUPPORTED.ScaledDotProductAttentionsignal) and return the new status, logged at warning level instead of error.initializeGenAiServable, when the pipeline type was auto-selected, fall back to the legacy pipeline (LM_CB→LM,VLM_CB→VLM).*_CBpipeline still surfaces the error rather than being silently downgraded.Behavior after fix
Testing
Manually verified with
OpenVINO/gemma-4-E4B-it-int4-ovon OVMS built from this branch (OpenVINO/GenAI 2026.3): with defaultAUTO, the server now falls back to the legacy VLM servable and serves/v3/chat/completionssuccessfully, instead of failing to start.No automated test is included — exercising the fallback needs a no-SDPA (Gemma-class) model in the test model set, which
windows_prepare_llm_models.bat/ the export tooling does not currently produce. Happy to add coverage if a suitable small fixture model can be agreed on.Notes
This restores a working default path; it does not enable continuous batching for Gemma — that remains gated on OpenVINO core adding PagedAttention support for the Gemma architecture.
Related: #4178