Add minicpm5 tool_parser for MiniCPM5 XML tool calls (draft, #4268)

[exzile]

Summary

Adds a minicpm5 tool parser so MiniCPM5 XML-style tool calls are surfaced as OpenAI-style tool_calls on /v3/chat/completions, instead of being left as raw XML in message.content with tool_calls: [].

Addresses #4268.

Note: #4268 is assigned to @przepeck. I opened this as a draft so it can either be reviewed/merged or used as a starting point — whatever is least disruptive to any work already underway. Happy to hand it off, iterate on review, or close it if it duplicates effort.

Approach

MiniCPM5's format (<function name="get_weather"><param name="city">Beijing</param></function>) is structurally almost identical to qwen3coder, so the parser is modeled directly on it:

• New src/llm/io_processing/minicpm5/minicpm5_tool_parser.{hpp,cpp} — Minicpm5ToolParserImpl state machine + Minicpm5ToolParser : BaseOutputParser.
• Unary parse() and streaming parseChunk() (same first-delta / full-delta protocol as qwen3coder).
• Attribute-style tag parsing (name="...", also tolerates '...'), <think>...</think> stripping before parsing, and schema-driven argument typing (string vs number/bool) reusing the existing ToolsSchemas_t mechanism.
• Registered minicpm5 in output_parser.cpp and getSupportedToolParserNames(); bazel target wired into output_parsers.

Tests (src/test/llm/output_parsers/minicpm5_output_parser_test.cpp)

• Unary OutputParser::parse cases (single/multiple calls, integer-typed params, mixed types, <think> stripping, content preservation, content-around-call removal) — these run in CI (tokenizer-gated).
• Impl-level state-machine cases: single/multiple calls, integer typing, newline trimming, content removal, single-quote attributes.
• Streaming robustness: char-by-char and every fragment size 1..N must yield the same calls as a single-chunk parse; two-calls-char-by-char; leading content + <think> char-by-char.
• Edge cases (no crash / no spurious calls): unterminated <function>, empty <param> value, stray < in plain prose.

All 15 impl/streaming tests pass locally (Windows build). The OutputParser-level tests are tokenizer-gated and run in CI (same as the existing qwen3coder tests).

Not yet included (intentional, for v1 follow-up / reviewer input)

• export_model.py + docs/llm/reference.md updates for the new --tool_parser minicpm5 value.
• End-to-end validation against a real openbmb/MiniCPM5-1B IR (the acceptance test in Add tool_parser minicpm5 for MiniCPM5-1B XML tool calls #4268).
• Tool-guided / XGrammar generation for MiniCPM5 (explicitly out of scope per the issue).

Glad to add the docs/export wiring and run the end-to-end check once direction is confirmed.

[exzile]

Thanks for the detailed review @przepeck — pushed updates addressing all of the comments:

• Reasoning extracted to a reasoning parser + preserved as reasoning_content (:222, :244): the tool parser no longer handles <think> (dropped the InsideThink state/tags). MiniCPM5 uses the same <think>/</think> as Qwen3, so reasoning is handled by a reasoning parser and emitted as reasoning_content, consistent with the other parsers.
• Long method split (:215): parseUntilStateChange is now a small dispatcher delegating to per-state handlers.
• Shared code with qwen3coder (:148): trimNewline, jsonTypeOf, enforceStringValue and Python-bool normalization moved to io_processing/utils; parseToolSchema/createToolsParametersTypesMap moved to base_output_parser (which owns the ParameterType/ToolsSchemas types). Both parsers use the shared versions; qwen3coder behavior is unchanged.
• Tests (:304, :1): added a test driving the public parseChunk streaming path (name delta once, arguments delta after the function closes), plus complex-content cases — embedded quotes/backslashes, code with braces/newlines, JSON-looking string values, and special/unicode characters.
• Special-token tags (:433): confirmed and fixed. In the openbmb/MiniCPM5-1B tokenizer the tool tags are special tokens — <function=18, </function>=19, <param=20, </param>=21 (all special: true) — so the default streamer skips them and the tag-scanning parser never saw them while streaming. The tool parser now overrides requiresStreamingWithSpecialTokens() to return true. Because the framework requires the paired reasoning parser to agree on that flag, and MiniCPM5's <think>/</think> are non-special tokens (so the Qwen3 reasoning logic still applies), I added a thin Minicpm5ReasoningParser subclass that flips the flag and registered minicpm5 as a reasoning parser. A deployment uses tool_parser: minicpm5 + reasoning_parser: minicpm5.

All changes compile-verified locally (parser libraries + output_parsers build clean); the full gtest run is via CI here, as my local ovms_test build is blocked by an unrelated genai-version skew in the audio module.

[exzile]

Local verification update

Built //src:ovms_test and ran the minicpm5 suite locally — 30 tests, 22 passed, 8 failed, where the 8 failures are purely an environmental missing-model issue, not a code problem:

• The 8 failing tests (Minicpm5OutputParserTest.Parse*) call tokenizer->encode(...) and fail with Tokenizer::encode is not available — openvino_tokenizer.xml was not provided or it was not loaded correctly because the tokenizer IR isn't present in my local checkout. These are the tests that need a real model and will run under CI.
• All 22 model-independent tests pass, including:
  • the Minicpm5ToolParserImplTest core + edge-case + streaming-robustness tests,
  • the public streaming-delta test (name delta once, arguments delta after </function>),
  • the complex-content tests (embedded quotes/backslashes, code with braces/newlines, JSON-looking string values, unicode/special chars),
  • and a RequiresStreamingWithSpecialTokens test confirming the tool parser and the new Minicpm5ReasoningParser both report true and the minicpm5 + minicpm5 pairing constructs without tripping the requiresStreamingWithSpecialTokens() consistency check.

So the refactor and the special-token handling are behaviorally verified for everything testable without a downloaded model; the end-to-end/encode-dependent cases are left to CI.