Skip to content

fix(cli): enforce /v1/completions stop sequences host side for qairt - #1369

Open
Shishir Shrivastava (Shishir2405) wants to merge 1 commit into
qualcomm:mainfrom
Shishir2405:fix/qairt-completions-stop-sequences
Open

fix(cli): enforce /v1/completions stop sequences host side for qairt#1369
Shishir Shrivastava (Shishir2405) wants to merge 1 commit into
qualcomm:mainfrom
Shishir2405:fix/qairt-completions-stop-sequences

Conversation

@Shishir2405

Copy link
Copy Markdown

Summary

The qairt plugin rejects any generate call that carries stop sequences (sdk/plugins/qairt/src/llm.cpp:219), but completion.go forwarded Stop unchanged to every runtime. Any FIM autocompletion client sending a stop list (which real editor clients always do) got ErrorCode[-100016] and an empty completion on QAIRT models, while the identical request worked on llama_cpp GGUF models.

This implements the host-side enforcement the issue proposed:

  1. Stop is withheld from GenerationConfig when the resolved runtime is qairt.
  2. A new stopMatcher (cli/server/handler/stop_match.go) matches the requested stop strings against generated text as it comes in, buffering only the minimal tail needed to catch a stop sequence split across token boundaries (e.g. <fim + _suffix + >).
  3. On a match, generation is cancelled through the existing OnToken return-false path (already supported by the qairt pipeline, reported as stop_reason: "user"finish_reason: "stop" via the existing mapFinishReason).
  4. The returned text is truncated at the match, for both the streaming (SSE) and blocking response paths.

llama_cpp models are unaffected — Stop is still forwarded to the plugin exactly as before when the runtime isn't qairt.

Test plan

  • Added stop_match_test.go: table-driven tests for stopMatcher, including the exact FIM stop-sequence-split-across-tokens scenario from the issue, multiple stop sequences, earliest-match-wins, empty stop strings, and the case where a partial match never completes (must still flush the held-back text).
  • Verified the matcher logic in isolation (extracted to a throwaway module, no cgo/SDK dependency) — all cases pass, including the issue's repro shape.
  • gofmt -l clean on all three changed/added files.
  • Could not run go vet / go test ./cli/server/handler/... or bazelisk coverage in this environment — the handler package imports bindings/go, which requires the built SDK bridge (sdk/pkg-geniex/), and per _build-sdk.yml that only builds on windows-arm64 / linux-arm64 (Docker) / android-arm64, none of which were available here. I'd appreciate a CI run / local verification on a supported host before merge.

Notes

  • Scope kept to /v1/completions, matching the issue. chat.go's /v1/chat/completions path doesn't forward Stop to GenerationConfig at all today (a separate gap, if it is one) — left untouched here to keep this PR focused.
  • No new dependencies; stop_match.go only uses strings from the standard library, so go.mod/go.sum are untouched.

Closes #1341

The qairt plugin rejects any generate call that carries stop sequences
(sdk/plugins/qairt/src/llm.cpp:219), so completion.go forwarding Stop
unchanged to every runtime made /v1/completions unusable with QAIRT
models for any client that sends a stop list, which real FIM
autocompletion clients always do.

Withhold Stop from the plugin config when the runtime is qairt and
match the requested stop strings against the generated text instead,
via a new stopMatcher that buffers just enough of the tail to catch a
match split across token boundaries. A match cancels generation early
through the existing OnToken return-false path and truncates the
returned text at the match, for both the streaming and blocking
response paths.

Closes qualcomm#1341

Signed-off-by: Shishir2405 <shishirshrivastava30@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/v1/completions fails on QAIRT models because the qairt plugin rejects stop sequences

1 participant