embeddings: MAX_INPUT_TOKENS cap on the tokens taken from each input text (manticoresearch#4816)

[tudorvasinca]

Embeddings-library half of the fix for manticoresoftware/manticoresearch#4816 (the daemon half — DDL option, ALTER, SHOW CREATE TABLE, docs, CLT test — is a separate PR on manticoresearch that pins this commit in the mcl submodule).

Problem

Auto-embedding time grows superlinearly with input length, and the only limit is the model's own context window (max_position_embeddings, 32,768 tokens for Qwen/Qwen3-Embedding-0.6B). Measured on 29.0.2 (details in the issue): 500 B → 4.6 s, 2,000 B → 35.6 s, 5,000 B → 196 s per document on CPU. A single long document holds an embedding statement for minutes; clients time out, the statement keeps running, retries stack more statements and the daemon is eventually OOM-killed.

Change

A per-model cap on the number of tokens taken from each input text, max_input_tokens (0 / None = the model's own limit; the cap can only lower it):

• ModelOptions.max_input_tokens: Option<usize>; LocalModel::new applies min(model limit, cap) to every architecture's max_input_len (BERT / T5 / Causal / Quantized / ONNX), so all predict paths truncate on it — no change to the truncation code itself.
• FFI: load_model gains a trailing i32 max_input_tokens; EmbedLib.version 4 → 5 (header regenerated accordingly).
• knn::ModelSettings_t::m_iMaxInputTokens, part of the model cache key, passed through in knn/embeddings.cpp; SUPPORTED_EMBEDDINGS_LIB_VER 4 → 5.
• Unit test test_max_input_tokens_cap: cap lowers the limit, never raises it, zero = no cap, and a text plus a long tail embeds identically to the text alone under the cap (and differs without it).

Remote providers (OpenAI / Voyage / Jina) are not capped by this change — happy to add pre-truncation there in a follow-up if wanted.

Verification

• cargo check --tests, cargo fmt --check, cargo test --release --features download-ort test_max_input_tokens_cap (rust 1.98).
• End-to-end with the daemon PR on sentence-transformers/all-MiniLM-L6-v2: with a cap of 6 tokens, "a red apple on the table" and the same text followed by 300 extra words produce byte-identical vectors, a different head differs; uncapped the tail changes the vector. A ~5 KB document: 38 ms capped (64 tokens) vs 409 ms uncapped per REPLACE.

[CLAassistant]

All committers have signed the CLA.

[donhardman]

Thanks for PR. It's all good to me. I also rebased your branch to latest Qwen optimization we had in master

[tudorvasinca]

Rebased onto current master (the branch had drifted into conflict): the FFI version moved to 9 (8 was taken by the chunked-embeddings work meanwhile), and two test call sites added upstream since (LocalModel::new in local.rs, load_model in ffi_test.rs) now pass the new max_input_tokens argument. cargo check --tests, the release build and the test suite (cap test + FFI tests) are green; the daemon side of manticoresoftware/manticoresearch#4857 is rebased on top and smoke-tested against this head (CREATE/INSERT/knn with a local model, SHOW CREATE, ALTER of the cap, validation). Sorry for resetting the approval with the force-push - it was the only way to resolve the conflicts.