diff --git a/CHANGELOG.md b/CHANGELOG.md
index 96f98390c2..9db3ce6e37 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -60,6 +60,8 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel
- Eight remaining host BF16 roundings in test oracles (GLM's Metal checks and CUDA FP8 KV reference, Nemotron's training scan, LFM2's DSpark taps, Flash Next's HIP kernel tests) and the XPU FP64 GDN reference use core's shared conversion, so a negative NaN gives a NaN instead of overflowing in Debug builds; every other non-NaN input gives the same bits. No kernel or served path changes (part of #1863).
- Qwen3.8-27B on Metal gains a host GGUF family adapter for metadata, tensor names and exact affine preparation; GGUF serving remains unavailable (#1920).
- Qwen3.8-27B on Metal gains a host GGUF family adapter for metadata, tensor names and exact affine preparation, with the observed qwen35 schema and effective norm gains pinned in host tests; GGUF serving remains unavailable (#1920).
+- `POST /v1/rerank` serves `Qwen/Qwen3-Reranker-8B` on HIP in the rerank shape agreed in #1895 (`results` with `index`, `relevance_score` and `document`, `top_n`, `return_documents`, `usage.total_tokens`), taking SGLang's request fields too: each document's prompt is the model card's template token for token, and its score is sigmoid(yes - no) of the last position's label logits, read from `lm_head` rows in f64. Only an engine that declares a reranker answers; every other server answers the route with HTTP 400, and a reranker server refuses the other model routes by name. The lane host's score queue now stops at a drain's deadline: the score already running finishes and the queued ones fail without running, which also reaches `/v1/decisions` on Flash Next on CUDA (#1981, part of #1977). Thanks to @laudney.
+- `POST /v1/rerank` serves `Qwen/Qwen3-Reranker-8B` on HIP in the rerank shape agreed in #1895 (`results` with `index`, `relevance_score` and `document`, `top_n`, `return_documents`, `usage.total_tokens`), taking SGLang's request fields too: each document's prompt is the model card's template token for token, a prompt past `max_length` (default 8192) losing the end of its content as the card's tokenizer cuts it, and its score is sigmoid(yes - no) of the last position's label logits, read from `lm_head` rows in f64. Only an engine that declares a reranker answers; every other server answers the route with HTTP 400, and a reranker server refuses the other model routes by name. The lane host's score queue now stops at a drain's deadline: the score already running finishes and the queued ones fail without running, which also reaches `/v1/decisions` on Flash Next on CUDA (#1981, part of #1977). Thanks to @laudney.
- Living Weights on Qwen4 dense, step 1: the QSA backward pass on the host, checked against a binary64 reference; nothing served changes (#2021, part of #1804).
diff --git a/docs/api.md b/docs/api.md
index e044f3cff0..439dbd02cd 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -45,7 +45,7 @@ different scopes, or scopes a key the flags or environment already give, is refu
| Scope | Routes |
| --- | --- |
-| `inference` | POST `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/tokenize`, `/v1/detokenize`, `/v1/audio/speech`, `/v1/embeddings`, `/v1/decisions`, `/v1/systemone`, and the same paths without `/v1` where the server has them; GET `/v1/models`, `/models`, `/v1/voices` |
+| `inference` | POST `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/tokenize`, `/v1/detokenize`, `/v1/audio/speech`, `/v1/embeddings`, `/v1/decisions`, `/v1/systemone`, `/v1/rerank` (also by PUT), and the same paths without `/v1` where the server has them; GET `/v1/models`, `/models`, `/v1/voices` |
| `read` | GET `/metrics`, `/v1/metrics`, `/v1/load`, `/memory`, `/v1/memory`, `/stats`, `/dashboard`, `/v1/benchmark/manifest`, `/v1/models`, `/models`, `/v1/voices` |
A scoped key reaches nothing else: not the stored Responses (GET or DELETE `/v1/responses/ID`), Recall, Sliding Weights,
@@ -148,6 +148,34 @@ question id, and `usage.completion_tokens` 0. `probabilities` are a softmax over
For decisions, `chat_template_kwargs` may be omitted, null, or an object containing only
`enable_thinking: false`; other types, keys, or thinking values return HTTP 400.
+## Rerank
+
+`POST /v1/rerank` (also `/rerank`, and by `PUT` as SGLang takes it) scores each document against a query with a
+reranker checkpoint. The native HIP engine serves `Qwen/Qwen3-Reranker-8B`: the engine declares the reranker and
+its label tokens (`1_LogitScore/config.json`), and its startup line says `qwen3 reranker`. Every other server answers
+the route with HTTP 400 "this model is not a reranker", and a reranker server answers chat, completion, Responses,
+Messages, speech, decision, System One and embedding requests with HTTP 400 "this model is a reranker: use POST
+/v1/rerank".
+
+| Field | Meaning |
+| --- | --- |
+| `query` | The query: a string, a `{"text": ...}` object, or a list of text parts |
+| `documents` | One or more documents, each in the same forms as `query` |
+| `instruction` | The task the model card's prompt states; default "Given a web search query, retrieve relevant passages that answer the query". SGLang's `instruct` is read when `instruction` is absent |
+| `top_n` | Return the best `top_n` results only; default all |
+| `return_documents` | Echo each document as sent; default `true` |
+| `max_length` | The most tokens one prompt may take; default 8192, the model card's. A longer prompt keeps the template whole and loses the end of its content (the document's last tokens), as the card's tokenizer cuts it; a cap that would leave no token of the document returns HTTP 400 |
+| `return_logits`, `return_prompt_token_ids` | Add `meta_info` with `logit_yes` and `logit_no`, or the prompt's token ids, to each result; default `false` |
+
+Each document's prompt is the model card's template, token for token, with the query, the document and the
+instruction as plain text (text that spells a special token stays text). The score is
+`sigmoid(logit_yes - logit_no)` from the last position's label logits. The response is
+`{"results": [{"index", "relevance_score", "document"}], "usage": {"total_tokens"}}`, best first, ties in input order;
+`total_tokens` counts every prompt's tokens as scored, after any cut. Every prompt is cut and checked against the
+context window before the first is scored, so a refused request costs no GPU work; a prompt past the window (with a
+`max_length` above it) returns HTTP 400 with `code` `context_length_exceeded`. An image or video part returns HTTP
+400. Errors use the server's envelope, as on every route: `{"error": {"message", "type"}}`.
+
## Living Weights
With `--slide`, the native server learns facts into the served weights. [Living Weights](living-weights.md) explains
diff --git a/docs/recipes/qwen3-reranker-hip.md b/docs/recipes/qwen3-reranker-hip.md
new file mode 100644
index 0000000000..769bceee43
--- /dev/null
+++ b/docs/recipes/qwen3-reranker-hip.md
@@ -0,0 +1,91 @@
+# Qwen3-Reranker-8B on native HIP
+
+The native HIP engine serves `Qwen/Qwen3-Reranker-8B` as published (BF16
+safetensors, revision `77d193c791ed757ca307ee72715aa132723da912`) through
+`POST /v1/rerank`. It is qualified on one Radeon AI PRO R9700 (gfx1201).
+gfx11 builds run the same code; FP16 builds (RDNA2) refuse the checkpoint.
+
+| Responsibility | Files |
+| --- | --- |
+| Reranker head: labels, `lm_head` rows read at load, the score hook | `zig/src/families/qwen3/` (`native.zig`, `score.zig`, `prompt.zig`) |
+| The engine's declaration (`Info.rerank`) | `zig/src/core/engine_api.zig`, `zig/src/native/hip.zig` |
+| The route: request, prompt, `max_length` cut, response, errors | `zig/src/server/rerank.zig` |
+| The lane host's score queue at a drain | `zig/src/core/lane_host_score.zig` |
+| Pinned oracle, FP32 and BF16 references, the card-formatter check | `tools/rerank/qwen3_reranker_oracle.py` |
+
+The checkpoint is a plain Qwen3 causal LM; the prompt pass is #1978's. A
+folder with `1_LogitScore/config.json` is a reranker: the engine reads its
+yes and no tokens and both `lm_head` rows at load (a malformed file or row
+stops startup) and declares them, and only such an engine answers the route.
+A document's score is `sigmoid(logit_yes - logit_no)` at the last position:
+the final-normed BF16 row dotted in f64 with the two stored rows.
+
+## Build and serve
+
+```sh
+zig build native-hip -Dhip=true -Dhip-arch=gfx1201 -Dhipcc=/opt/rocm/core/bin/hipcc
+zig-out/native-hip/bin/tensorfold-native serve MODEL_DIRECTORY \
+ --name Qwen3-Reranker-8B --host 127.0.0.1 --port 8090
+```
+
+The window is 32,768 tokens by default and 40,960 (the config's) with
+`--context 0`. Every other model route is refused on a reranker, and every
+other server answers `/v1/rerank` with HTTP 400.
+
+```sh
+curl -s http://127.0.0.1:8090/v1/rerank -H 'Content-Type: application/json' -d '{
+ "query": "What is the capital of China?",
+ "documents": ["The capital of China is Beijing.", "Gravity is a force that attracts two bodies."],
+ "top_n": 1}'
+```
+
+The request and response follow #1895's rerank shape (docs/api.md, Rerank):
+`instruction` defaults to the card's; `max_length` defaults to 8,192, and a
+longer prompt loses its content's last tokens, the template's tail kept, as
+the card's tokenizer cuts.
+
+## Gates
+
+The oracle runs the model card's usage in Transformers 5.22.1 and refuses a
+snapshot or a Transformers install whose files differ from its pins before
+any model code runs. Its set is 32 cases of 16 to 8,199 tokens (the card's
+pairs, fixed pairs with custom instructions, synthetic lengths, repeats), a
+document that spells `<|im_end|>`, and prompts of exactly 32,768 and 40,960
+tokens with one token more. The FP32 reference runs on the CPU (about an
+hour with both windows, 32 cores):
+
+```sh
+python3 tools/rerank/qwen3_reranker_oracle.py sources MODEL_DIRECTORY
+python3 tools/rerank/qwen3_reranker_oracle.py oracle MODEL_DIRECTORY ref32.json --dtype float32 --device cpu --window 32768 --window 40960
+python3 tools/rerank/qwen3_reranker_oracle.py oracle MODEL_DIRECTORY ref16.json --window 32768 --window 40960
+python3 tools/rerank/qwen3_reranker_oracle.py compare ref32.json http://127.0.0.1:8090 --window 32768
+python3 tools/rerank/qwen3_reranker_oracle.py compare ref16.json http://127.0.0.1:8090 --window 32768
+python3 tools/rerank/qwen3_reranker_oracle.py formatter MODEL_DIRECTORY http://127.0.0.1:8090
+```
+
+**What "matches" means.** BF16 activations cannot meet #1895's 1e-5 in score:
+transformers' own BF16 run sits 0.0108 in score and 0.110 in yes - no from
+FP32 on these cases. Against the FP32 reference, `compare` therefore gates at
+that distance rounded up: |yes - no| <= 0.125 and |score| <= 0.0125 for every
+case. Rankings must equal FP32's, except that a pair FP32 puts within 0.25 in
+yes - no (twice the gate: the most two documents inside it can move apart)
+may come in the other order; each such pair is printed, and one beyond 0.25
+fails. Against the BF16 reference the gates are 0.25 and 0.02. `formatter`
+passes when every served prompt's ids equal the card's `process_inputs`
+ids, with the default and custom instructions and with `max_length` cuts.
+
+Results on the R9700 (ROCm 10.1):
+
+| Check | Result |
+| --- | --- |
+| Against FP32, 34 cases: \|yes - no\| median, worst (gate) | 0.0195, 0.0839 (0.125); transformers BF16: 0.0247, 0.1099 |
+| Against FP32, 34 cases: \|score\| median, worst (gate) | 0.00023, 0.0079 (0.0125); transformers BF16: 0.00014, 0.0108 |
+| Window prompts against FP32 (yes - no) | 32,768 tokens: 0.0064; 40,960 tokens (`--context 0`): 0.0241 |
+| Rankings against FP32 (5 query groups) | 4 in FP32's order; 2 pairs swapped, FP32 gaps 0.0077 and 0.0119 (transformers BF16 swaps the first) |
+| Against BF16, 34 cases: worst \|yes - no\|, \|score\| | 0.1081, 0.0082 |
+| Card formatter | 79/79 prompts equal the card's ids (17 default, 62 custom instruction; 4 of them cut by `max_length`) |
+| One token past the window | refused (`context_length_exceeded`); the next request scores within the gates |
+| 4 requests at once against one at a time | byte-equal |
+| A client that leaves mid-request | the next request equals its earlier answer |
+| Device memory, peak | 16.4 GB (32,768 window), 16.7 GB (40,960) |
+| 16 documents of ~260 tokens, cold, one request | 3.22 s (SGLang 0.53 s on the same card); the dot2 GEMM tile, gfx12 WMMA is the follow-up |
diff --git a/tools/rerank/qwen3_reranker_oracle.py b/tools/rerank/qwen3_reranker_oracle.py
new file mode 100644
index 0000000000..d32a28e326
--- /dev/null
+++ b/tools/rerank/qwen3_reranker_oracle.py
@@ -0,0 +1,525 @@
+"""Qwen3-Reranker reference scores and prompt ids from transformers, and a comparison against a /v1/rerank server.
+
+sources MODEL_DIR check the snapshot and Transformers against the pins (no model loads)
+oracle MODEL_DIR OUT.json [--window N ...] write the reference cases (BF16 on the GPU by default, ROCm torch
+ [--dtype float32 --device cpu] venv; FP32 on the CPU for #1895's reference); each --window adds a
+ prompt of exactly N tokens and one of N + 1
+compare OUT.json URL [--window N] post the cases to URL/v1/rerank and print the gate verdicts; with
+ --window, also the served window's cases (N + 1 must be refused)
+formatter MODEL_DIR URL the served prompt ids against the model card's own formatter, token
+ for token, with the default and with custom instructions
+"""
+
+import argparse
+import hashlib
+import importlib.metadata
+import importlib.util
+import json
+import math
+import statistics
+import sys
+from pathlib import Path
+import urllib.error
+import urllib.request
+
+# The checkpoints the receipts were made from, and sha256 of every non-weight file of each pinned snapshot.
+CHECKPOINTS = {
+ "8B": ("Qwen/Qwen3-Reranker-8B", "77d193c791ed757ca307ee72715aa132723da912"),
+ "4B": ("Qwen/Qwen3-Reranker-4B", "22e683669bc0f0bd69640a1354a6d0aebcfeede5"),
+}
+SOURCES = {
+ "8B": {
+ ".gitattributes": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930",
+ "1_LogitScore/config.json": "73e3156450564d8a98b7e47bcf5aace0f29600828b51937da545571e84db3ff3",
+ "README.md": "5f33734e77bb56c191dfd66bd8c4c04fdba6c023cddd6052871082c411f9b608",
+ "chat_template.jinja": "6f682162495ec5b39fd9005c01b6aa2a74669379fe967039f1e2cbbe8752369d",
+ "config.json": "e87a4f50c475c7276fc642eabfc986a7c8da1c8a30a4ee34b4979048b330f2be",
+ "config_sentence_transformers.json": "231aa494dcabada22d73f151120560b2a72b49e5b93e1b0a8329664f2348cf2e",
+ "generation_config.json": "81051cd3f6e77013827148d0b8a6ead93f8ac390d5ab805f849199f0af6a08db",
+ "merges.txt": "8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5",
+ "model.safetensors.index.json": "8a22e7ffd6001796256c0be7c73594835b6f907eec442fde691301ebf35087d9",
+ "modules.json": "6f13b6b4a89e577b591b2077bca40c67c26541a6740a8809267cb474f90806a9",
+ "sentence_bert_config.json": "3234ebd224d492cbe8d55d5ec80a3f408451c4db3005bafb64fe1c51c763e01e",
+ "tokenizer.json": "aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4",
+ "tokenizer_config.json": "253153d0738ceb4c668d2eff957714dd2bea0b56de772a9fdccd96cbf517e6a0",
+ "vocab.json": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910",
+ },
+ "4B": {
+ ".gitattributes": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930",
+ "1_LogitScore/config.json": "73e3156450564d8a98b7e47bcf5aace0f29600828b51937da545571e84db3ff3",
+ "README.md": "b7cf82b30d5e38cd1a02ca7cfd60c26c0e3664cb31b2c4b661550a93c9281bee",
+ "chat_template.jinja": "6f682162495ec5b39fd9005c01b6aa2a74669379fe967039f1e2cbbe8752369d",
+ "config.json": "38bff5eac700032a185745e4076eccad7aa453473cafc2a27de412cdb7b79e19",
+ "config_sentence_transformers.json": "6a153d6696f78fd588c1c728967f0b773ea869d3c6028f151ce71ebe49140762",
+ "generation_config.json": "81051cd3f6e77013827148d0b8a6ead93f8ac390d5ab805f849199f0af6a08db",
+ "merges.txt": "8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5",
+ "model.safetensors.index.json": "18a783f197360068da36d2ff9ecffce0121542a87598fd1fef3079ead1c3cc08",
+ "modules.json": "6f13b6b4a89e577b591b2077bca40c67c26541a6740a8809267cb474f90806a9",
+ "sentence_bert_config.json": "3234ebd224d492cbe8d55d5ec80a3f408451c4db3005bafb64fe1c51c763e01e",
+ "special_tokens_map.json": "76862e765266b85aa9459767e33cbaf13970f327a0e88d1c65846c2ddd3a1ecd",
+ "tokenizer.json": "aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4",
+ "tokenizer_config.json": "253153d0738ceb4c668d2eff957714dd2bea0b56de772a9fdccd96cbf517e6a0",
+ "vocab.json": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910",
+ },
+}
+VERSION = "5.22.1"
+# The Qwen3 model code the oracle runs, and tree_sha256 of the whole installed Transformers package (shared code too).
+LIBRARY = {
+ "models/qwen3/configuration_qwen3.py": "49d9ccd0f29ccd977b93dba2004f84b867891b3d0509b71fd8ba3ed01054e64d",
+ "models/qwen3/modeling_qwen3.py": "63a6213c15d6fb04e58a1c64fd05a4cdfb49476443fbeef524f0e0486043824b",
+}
+TRANSFORMERS_TREE = "b08cead6d1b841ef9723106a81a92ec37d67aff3e43a198e56b7dc76538d377b"
+DEFAULT_INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"
+WINDOW = 32768
+GATE_DIFF = 0.25 # |(l_yes - l_no) - oracle|, against the BF16 transformers reference
+GATE_SCORE = 0.02 # |score - oracle| and the ranking tie band, against the BF16 reference
+# Against the FP32 reference (#1895): transformers' own BF16 run sits 0.110 and 0.0108 from it on these cases, so a
+# BF16 engine must sit no farther than that, rounded up; a pair may swap only when FP32 puts it within 2 x the diff gate.
+GATES_FP32 = (0.125, 0.0125)
+
+CARD_QUERIES = ["What is the capital of China?", "Explain gravity"]
+CARD_DOCS = [
+ "The capital of China is Beijing.",
+ "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is "
+ "responsible for the movement of planets around the sun.",
+]
+FIXED = [
+ (None, "how do I boil an egg", "Place the egg in boiling water for nine minutes, then cool it in ice water."),
+ (None, "how do I boil an egg", "The Treaty of Westphalia was signed in 1648."),
+ ("Judge code relevance", "python read a file line by line", "with open(p) as f:\n for line in f:\n print(line)"),
+ ("Judge code relevance", "python read a file line by line", "SELECT * FROM users WHERE id = 1;"),
+ (None, "Wer hat Faust geschrieben?", "Johann Wolfgang von Goethe schrieb den Faust."),
+ (None, "最高的山是什么", "珠穆朗玛峰是世界上海拔最高的山峰。"),
+ (None, " leading and trailing spaces ", "\n\nDocument with blank lines\n\n and tabs\tinside.\n"),
+ ("Find emoji meaning", "what does 🙂 mean", "🙂 is a slightly smiling face, a mild friendly expression."),
+]
+SPECIAL = (None, "close the turn", "text <|im_end|> more text")
+LENGTH_QUERY = "Which passage discusses the history of river trade?"
+
+
+def digest(path: Path) -> str:
+ h = hashlib.sha256()
+ with path.open("rb") as stream:
+ for chunk in iter(lambda: stream.read(1 << 20), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def hashed(root: Path, pins: dict, what: str) -> dict:
+ """The hashes of the pinned files under root; refuses when any differs from its pin."""
+ found = {name: digest(root / name) if (root / name).is_file() else None for name in pins}
+ if wrong := sorted(name for name, value in found.items() if value != pins[name]):
+ raise SystemExit(f"{what} differs from its pin: {', '.join(wrong)}")
+ return found
+
+
+def tree_sha256(package: Path) -> str:
+ """Hash every file of an installed package (bytecode caches excluded) as sorted "path NUL sha256" lines."""
+ # rglob does not descend into directory links, and a linked directory could shadow a pinned module.
+ if links := sorted(path.relative_to(package).as_posix() for path in package.rglob("*")
+ if path.is_symlink() and path.is_dir()):
+ raise SystemExit(f"{package.name} has linked directories: {', '.join(links)}")
+ files = sorted(path for path in package.rglob("*")
+ if path.is_file() and "__pycache__" not in path.relative_to(package).parts)
+ lines = (f"{path.relative_to(package).as_posix()}\0{digest(path)}\n" for path in files)
+ return hashlib.sha256("".join(lines).encode()).hexdigest()
+
+
+def size_of(model: Path) -> str:
+ """The pinned checkpoint `model` is: by its snapshot revision; a folder of no pinned revision is refused."""
+ for size, (_, revision) in CHECKPOINTS.items():
+ if model.resolve().name == revision:
+ return size
+ raise SystemExit(f"{model}: not a pinned snapshot ({', '.join(f'{r}@{v}' for r, v in CHECKPOINTS.values())})")
+
+
+def pinned(model: Path) -> dict:
+ """Refuse, before any model code runs, a snapshot or Transformers install whose sources differ from the pins."""
+ size = size_of(model)
+ repo, revision = CHECKPOINTS[size]
+ model = model.resolve()
+ if links := sorted(path.relative_to(model).as_posix() for path in model.rglob("*")
+ if path.is_symlink() and path.is_dir()):
+ raise SystemExit(f"{size}: snapshot has linked directories: {', '.join(links)}")
+ extra = sorted(path.relative_to(model).as_posix() for path in model.rglob("*")
+ if path.is_file() and path.suffix != ".safetensors"
+ and path.relative_to(model).as_posix() not in SOURCES[size])
+ if extra:
+ raise SystemExit(f"{size}: snapshot has unpinned non-weight files: {', '.join(extra)}")
+ if (version := importlib.metadata.version("transformers")) != VERSION:
+ raise SystemExit(f"expected Transformers {VERSION}, found {version}")
+ package = Path(importlib.util.find_spec("transformers").origin).parent
+ if (tree := tree_sha256(package)) != TRANSFORMERS_TREE:
+ raise SystemExit(f"Transformers {VERSION} package tree differs from its pin: {tree}")
+ return {"repo": repo, "revision": revision, "checkpoint": hashed(model, SOURCES[size], f"{repo}@{revision}"),
+ "transformers": VERSION, "transformers_files": hashed(package, LIBRARY, f"Transformers {VERSION}"),
+ "transformers_tree": tree}
+
+
+def weights_sha256(model: Path) -> dict:
+ """Every shard's sha256, after the index names exactly the shards present."""
+ model = model.resolve()
+ shards = sorted(model.glob("*.safetensors"))
+ index = json.loads((model / "model.safetensors.index.json").read_text())
+ if set(index["weight_map"].values()) != {path.name for path in shards}:
+ raise SystemExit("the safetensors shards do not match the checkpoint index")
+ return {path.name: digest(path) for path in shards}
+
+
+def sources(args):
+ """Check the snapshot and the Transformers sources against the pins and print their hashes; no model loads."""
+ print(json.dumps(pinned(args.model), indent=1))
+ return 0
+
+
+def synthetic(tokenizer, tokens):
+ """Deterministic prose of about `tokens` tokens."""
+ words = ("river trade grew along the valley as merchants moved salt grain and cloth between market towns "
+ "while boats carried timber downstream and the harbor kept records of every cargo ").split()
+ out, i = [], 0
+ while True:
+ out.append(words[i % len(words)])
+ i += 1
+ if i % 64 == 0 and len(tokenizer(" ".join(out), add_special_tokens=False).input_ids) >= tokens:
+ break
+ text = " ".join(out)
+ ids = tokenizer(text, add_special_tokens=False).input_ids[:tokens]
+ return tokenizer.decode(ids)
+
+
+def padded(tokenizer, template, n):
+ """A document whose rendered prompt is exactly `n` tokens."""
+ def length(doc):
+ text = template.render(messages=messages(None, LENGTH_QUERY, doc))
+ return len(tokenizer(text, add_special_tokens=False).input_ids)
+ want = n - length("x") + 1
+ for _ in range(64):
+ doc = synthetic(tokenizer, max(want, 1))
+ got = length(doc)
+ if got == n:
+ return doc
+ want += n - got
+ raise SystemExit(f"--window {n}: no document makes the prompt exactly {n} tokens")
+
+
+def messages(instruct, query, document):
+ msgs = []
+ if instruct:
+ msgs.append({"role": "system", "content": instruct})
+ msgs += [{"role": "query", "content": query}, {"role": "document", "content": document}]
+ return msgs
+
+
+def oracle(args):
+ pins = pinned(args.model) # before any model code runs
+ out_path = Path(args.out)
+ if out_path.exists():
+ raise SystemExit(f"{out_path} exists: write each reference to a new file")
+ weights = weights_sha256(args.model) # the shards hashed before they load
+ if any(n < 256 for n in args.window):
+ raise SystemExit("--window: the template alone takes about 80 tokens; give 256 or more")
+ import jinja2
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ with open(f"{args.model}/chat_template.jinja") as f:
+ template = jinja2.Environment(trim_blocks=False, lstrip_blocks=False).from_string(f.read())
+ with open(f"{args.model}/1_LogitScore/config.json") as f:
+ labels = json.load(f)
+ yes, no = labels["true_token_id"], labels["false_token_id"]
+ tok = AutoTokenizer.from_pretrained(args.model, local_files_only=True, trust_remote_code=False)
+ dtype = {"bfloat16": torch.bfloat16, "float32": torch.float32}[args.dtype]
+ model, loading = AutoModelForCausalLM.from_pretrained(args.model, dtype=dtype, local_files_only=True,
+ trust_remote_code=False, output_loading_info=True)
+ keys = ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
+ if errors := {key: loading[key] for key in keys if loading.get(key)}:
+ raise SystemExit(f"the checkpoint did not load completely: {errors}")
+ if (model.config.tie_word_embeddings
+ and model.get_input_embeddings().weight.data_ptr() != model.get_output_embeddings().weight.data_ptr()):
+ raise SystemExit("the model did not tie its input and output embeddings")
+ model = model.to(args.device).eval()
+ head = model.lm_head.weight.detach()
+ rows = head[[yes, no]].double().cpu()
+
+ cases = []
+ for q in CARD_QUERIES:
+ for d in CARD_DOCS:
+ cases.append({"instruct": None, "query": q, "document": d, "kind": "card"})
+ for ins, q, d in FIXED:
+ cases.append({"instruct": ins, "query": q, "document": d, "kind": "fixed"})
+ for n in (16, 128, 512, 2048, 8192):
+ for variant in range(3 if n <= 512 else 1):
+ d = synthetic(tok, n + 7 * variant)
+ cases.append({"instruct": None, "query": LENGTH_QUERY, "document": d, "kind": f"len{n}"})
+ while len(cases) < 32:
+ k = len(cases)
+ cases.append({"instruct": None, "query": CARD_QUERIES[k % 2], "document": CARD_DOCS[(k + 1) % 2] * (1 + k % 5),
+ "kind": "repeat"})
+ ins, q, d = SPECIAL
+ special = {"instruct": ins, "query": q, "document": d, "kind": "special", "score_skip": True}
+ extra = [special]
+ for n in args.window:
+ doc = padded(tok, template, n)
+ extra.append({"instruct": None, "query": LENGTH_QUERY, "document": doc, "kind": f"window{n}"})
+ extra.append({"instruct": None, "query": LENGTH_QUERY, "document": doc + " river", "kind": f"over{n}",
+ "score_skip": True})
+
+ out = []
+ with torch.no_grad():
+ for c in cases + extra:
+ text = template.render(messages=messages(c["instruct"], c["query"], c["document"]))
+ ids = tok(text, add_special_tokens=False).input_ids
+ rec = dict(c, text=text, ids=ids, tokens=len(ids))
+ if not c.get("score_skip"):
+ t = torch.tensor([ids], device=args.device)
+ last = model.model(input_ids=t).last_hidden_state[0, -1]
+ logits = model.lm_head(last)
+ own = model(input_ids=t, logits_to_keep=1).logits[0, -1] # one row: 40k rows of logits are 12 GB
+ # a one-row lm_head and the full forward's head take different kernels: equal within the dtype's steps
+ gap = float((logits.float() - own.float()).abs().max())
+ assert gap <= 0.25 and int(logits.argmax()) == int(own.argmax()), f"last row is not final-normed: {gap}"
+ rec["head_gap"] = gap
+ row = last.double().cpu()
+ ly, ln = float(row @ rows[0]), float(row @ rows[1])
+ rec["diff"] = ly - ln
+ rec["score"] = 1.0 / (1.0 + math.exp(-(ly - ln)))
+ by, bn = float(own[yes].float()), float(own[no].float())
+ rec["own_diff"] = by - bn # the reference model's own head, at its dtype
+ rec["own_score"] = 1.0 / (1.0 + math.exp(-(by - bn)))
+ out.append(rec)
+ print(f"{c['kind']:8} tokens={len(ids):6} score={rec.get('score', float('nan')):.6f}", file=sys.stderr)
+ partial = out_path.with_name(out_path.name + ".partial")
+ with open(partial, "w") as f:
+ json.dump({"model": str(args.model), "dtype": args.dtype, "device": args.device,
+ "torch": torch.__version__, "sources": pins, "weights_sha256": weights,
+ "yes": yes, "no": no, "cases": out}, f, ensure_ascii=False, indent=1)
+ partial.replace(out_path) # whole or not at all
+
+
+# The model card's formatter, verbatim (README "Transformers Usage"): prefix, content and suffix encoded apart.
+CARD_PREFIX = ("<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the "
+ "Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n")
+CARD_SUFFIX = "<|im_end|>\n<|im_start|>assistant\n\n\n\n\n"
+CUSTOM = ["Judge code relevance", "Given a question in German, find the passage that answers it",
+ " An instruction with spaces, a tab\tand\nnew lines "]
+
+
+def card_ids(tok, instruction, query, doc, max_length=8192):
+ """The ids the card's process_inputs builds for one pair: its content cut to max_length less prefix and suffix."""
+ if instruction is None:
+ instruction = DEFAULT_INSTRUCT
+ prefix = tok.encode(CARD_PREFIX, add_special_tokens=False)
+ suffix = tok.encode(CARD_SUFFIX, add_special_tokens=False)
+ content = f": {instruction}\n: {query}\n: {doc}"
+ cut = tok(content, truncation="longest_first", max_length=max_length - len(prefix) - len(suffix)).input_ids
+ return prefix + cut + suffix
+
+
+def formatter(args):
+ """Every case's served ids equal the card formatter's, with the default instruction, each custom one sent as
+ #1895's `instruction`, and once as SGLang's `instruct`."""
+ pinned(args.model)
+ from transformers import AutoTokenizer
+ tok = AutoTokenizer.from_pretrained(args.model, local_files_only=True, trust_remote_code=False)
+ pairs = [(q, d) for q in CARD_QUERIES for d in CARD_DOCS] + [(q, d) for _, q, d in FIXED]
+ pairs += [(LENGTH_QUERY, synthetic(tok, n)) for n in (16, 512, 2048)]
+ cases = [(None, q, d, None) for q, d in pairs]
+ cases += [(ins, q, d, "instruction") for ins in CUSTOM for q, d in pairs]
+ cases += [(CUSTOM[0], q, d, "instruct") for q, d in pairs]
+ cases = [c + (None,) for c in cases]
+ # max_length cuts the content's end as the card's tokenizer does: by default (8,192) and as sent
+ cases += [(None, LENGTH_QUERY, synthetic(tok, 9000), None, None),
+ (CUSTOM[1], LENGTH_QUERY, synthetic(tok, 9000), "instruction", None),
+ (None, LENGTH_QUERY, synthetic(tok, 2048), None, 512),
+ (CUSTOM[0], CARD_QUERIES[1], CARD_DOCS[1], "instruction", 70)]
+ bad = 0
+ for ins, q, d, field, cap in cases:
+ body = {"query": q, "documents": [d], "return_documents": False, "return_prompt_token_ids": True}
+ if field:
+ body[field] = ins
+ if cap:
+ body["max_length"] = cap
+ meta = post(args.url + "/v1/rerank", body)["results"][0]["meta_info"]
+ want = card_ids(tok, ins, q, d, cap or 8192)
+ if meta["prompt_token_ids"] != want:
+ bad += 1
+ print(f"FAIL: {field or 'default'} {ins!r} {q!r}: served ids differ from the card's ({len(want)} tokens)")
+ print(f"formatter: {len(cases) - bad}/{len(cases)} prompts equal the card's ids token for token "
+ f"(default instruction: {len(pairs) + 2}, custom: {len(cases) - len(pairs) - 2}; cut to max_length: 4) "
+ f"verdict={'PASS' if bad == 0 else 'FAIL'}")
+ return 0 if bad == 0 else 1
+
+
+def post(url, body):
+ req = urllib.request.Request(url, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
+ with urllib.request.urlopen(req, timeout=600) as r:
+ return json.load(r)
+
+
+def refusal(url, case):
+ """The 400 message a case gets, or None when the server scored it."""
+ body = {"query": case["query"], "documents": [case["document"]], "max_length": case["tokens"]} # never cut
+ try:
+ post(url + "/v1/rerank", body)
+ return None
+ except urllib.error.HTTPError as e:
+ return json.load(e).get("error", {}).get("message", "") if e.code == 400 else f"HTTP {e.code}"
+
+
+class Gate:
+ """The receipt's verdict: every failed check is printed, and any one fails the run."""
+
+ def __init__(self, fp32):
+ self.ok = True
+ self.worst_diff = self.worst_score = self.worst_own = 0.0
+ self.diffs, self.scores = [], []
+ self.fp32 = fp32
+ self.gates = GATES_FP32 if fp32 else (GATE_DIFF, GATE_SCORE)
+ self.swaps = 0
+
+ def fail(self, why):
+ self.ok = False
+ print(f"FAIL: {why}")
+
+ def scored(self, url, c):
+ """One case through the route: ids equal HF's, finite logits and score within the gates."""
+ body = {"query": c["query"], "documents": [c["document"]], "return_documents": False,
+ "return_logits": True, "return_prompt_token_ids": True, "max_length": c["tokens"]}
+ if c["instruct"]:
+ body["instruct"] = c["instruct"]
+ items = post(url + "/v1/rerank", body)["results"]
+ if len(items) != 1:
+ return self.fail(f"{c['kind']}: {len(items)} results for one document")
+ meta = items[0]["meta_info"] or {}
+ if meta.get("prompt_token_ids") != c["ids"]:
+ return self.fail(f"{c['kind']}: ids differ from HF's ({c['query']!r})")
+ values = (meta.get("logit_yes"), meta.get("logit_no"), items[0]["relevance_score"], c["diff"], c["score"])
+ if not all(isinstance(v, (int, float)) and math.isfinite(v) for v in values):
+ return self.fail(f"{c['kind']}: a non-finite logit or score {values}")
+ diff = meta["logit_yes"] - meta["logit_no"]
+ got = items[0]["relevance_score"]
+ self.worst_diff = max(self.worst_diff, abs(diff - c["diff"]))
+ self.diffs.append(abs(diff - c["diff"]))
+ self.scores.append(abs(got - c["score"]))
+ self.worst_score = max(self.worst_score, abs(got - c["score"]))
+ self.worst_own = max(self.worst_own, abs(got - c.get("own_score", c.get("bf16_score")))) # older files: bf16_
+ print(f"{c['kind']:12} tokens={c['tokens']:6} ref={c['score']:.6f} got={got:.6f} "
+ f"diff={diff:+.4f} ref_diff={c['diff']:+.4f}")
+
+ def special(self, url, c):
+ """The masked case: TensorFold's ids must differ from HF's (request text stays plain)."""
+ items = post(url + "/v1/rerank", {"query": c["query"], "documents": [c["document"]],
+ "return_prompt_token_ids": True, "max_length": 1 << 20})["results"]
+ meta = items[0].get("meta_info") if len(items) == 1 else None
+ ids = meta.get("prompt_token_ids") if isinstance(meta, dict) else None
+ if not isinstance(ids, list) or not ids or not all(type(i) is int and i >= 0 for i in ids):
+ return self.fail(f"special: no prompt ids came back ({items!r:.200})")
+ if ids == c["ids"]:
+ self.fail("special: request text was read as a special token")
+ else:
+ print("special: ids differ from HF's, as intended (request text masked)")
+
+ def ranking(self, url, ins, q, group):
+ """One request of a query's documents: a full permutation, no pair against the oracle beyond the tie band."""
+ body = {"query": q, "documents": [c["document"] for c in group], "return_documents": False,
+ "max_length": max(c["tokens"] for c in group)}
+ if ins:
+ body["instruct"] = ins
+ order = [r["index"] for r in post(url + "/v1/rerank", body)["results"]]
+ if sorted(order) != list(range(len(group))):
+ return self.fail(f"ranking: {q!r}: indices {order} are not a permutation of {len(group)} documents")
+ for pos, a in enumerate(order):
+ for b in order[pos + 1:]:
+ if self.fp32:
+ gap = group[b]["diff"] - group[a]["diff"]
+ if gap > 0:
+ self.swaps += 1
+ print(f"ranking: {q!r}: {a} before {b}, FP32 puts {b} ahead by {gap:.4f} in yes - no")
+ if gap > 2 * self.gates[0]:
+ self.fail(f"ranking: {q!r}: {a} before {b} against the oracle by {gap:.4f}")
+ elif group[b]["score"] - group[a]["score"] >= GATE_SCORE:
+ self.fail(f"ranking: {q!r}: {a} before {b} against the oracle")
+
+
+def compare(args):
+ with open(args.ref) as f:
+ ref = json.load(f)
+ cases = ref["cases"]
+ g = Gate(ref.get("dtype") == "float32")
+ plain = [c for c in cases if c["kind"] not in ("special",) and not c["kind"].startswith(("window", "over"))]
+ specials = [c for c in cases if c["kind"] == "special"]
+ if len(plain) < 32 or len(specials) != 1 or not all("score" in c and "ids" in c for c in plain):
+ print(f"FAIL: the reference needs 32 scored cases and the special one; it has {len(plain)} and {len(specials)}")
+ return 1
+ for c in plain:
+ g.scored(args.url, c)
+ for c in cases:
+ if c["kind"] == "special":
+ g.special(args.url, c)
+ if args.window:
+ inside = [c for c in cases if c["kind"] == f"window{args.window}" and c["tokens"] == args.window]
+ over = [c for c in cases if c["kind"] == f"over{args.window}" and c["tokens"] == args.window + 1]
+ if len(inside) != 1 or len(over) != 1:
+ g.fail(f"the reference has no window cases of exactly {args.window} and {args.window + 1} tokens")
+ else:
+ g.scored(args.url, inside[0])
+ why = refusal(args.url, over[0])
+ print(f"over{args.window}: {over[0]['tokens']} tokens: {why}")
+ if why is None or "does not fit the context length" not in why:
+ g.fail(f"over{args.window}: not refused with the window message")
+ g.scored(args.url, plain[0]) # the next request after the refusal scores within the gates
+ by_query = {}
+ for c in plain:
+ by_query.setdefault((c["instruct"], c["query"]), []).append(c)
+ for (ins, q), group in by_query.items():
+ if len(group) > 1:
+ g.ranking(args.url, ins, q, group)
+ gate_diff, gate_score = g.gates
+ if g.worst_diff > gate_diff:
+ g.fail(f"worst |diff| {g.worst_diff:.4f} > {gate_diff}")
+ if g.worst_score > gate_score:
+ g.fail(f"worst |score| {g.worst_score:.5f} > {gate_score}")
+ if g.fp32:
+ print(f"ranking: {g.swaps} pair(s) in another order than FP32's, each within {2 * gate_diff} in yes - no")
+ def spread(v): # median, p99 (nearest rank) and worst of the scored cases
+ if not v:
+ return "no case scored"
+ v = sorted(v)
+ return f"median {statistics.median(v):.5f} p99 {v[math.ceil(0.99 * len(v)) - 1]:.5f} worst {v[-1]:.5f}"
+ print(f"reference {ref.get('dtype', 'bfloat16')}: |diff| {spread(g.diffs)}; |score| {spread(g.scores)}")
+ print(f"worst |diff|={g.worst_diff:.4f} (gate {gate_diff}) worst |score|={g.worst_score:.5f} (gate {gate_score}) "
+ f"worst |score - the reference's own-head score|={g.worst_own:.5f} (recorded) verdict={'PASS' if g.ok else 'FAIL'}")
+ return 0 if g.ok else 1
+
+
+def main():
+ p = argparse.ArgumentParser()
+ sub = p.add_subparsers(dest="cmd", required=True)
+ src = sub.add_parser("sources")
+ src.add_argument("model", type=Path)
+ o = sub.add_parser("oracle")
+ o.add_argument("model", type=Path)
+ o.add_argument("out")
+ o.add_argument("--window", type=int, action="append", default=[])
+ o.add_argument("--dtype", choices=("bfloat16", "float32"), default="bfloat16")
+ o.add_argument("--device", default="cuda")
+ c = sub.add_parser("compare")
+ c.add_argument("ref")
+ c.add_argument("url")
+ c.add_argument("--window", type=int, default=0)
+ fm = sub.add_parser("formatter")
+ fm.add_argument("model", type=Path)
+ fm.add_argument("url")
+ args = p.parse_args()
+ if args.cmd == "formatter":
+ return formatter(args)
+ if args.cmd == "sources":
+ return sources(args)
+ if args.cmd == "oracle":
+ oracle(args)
+ return 0
+ return compare(args)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/zig/build/cuda.zig b/zig/build/cuda.zig
index 0e6e7ef813..369d62ce8a 100644
--- a/zig/build/cuda.zig
+++ b/zig/build/cuda.zig
@@ -419,12 +419,12 @@ pub fn hostTests(b: *std.Build, draft_ids: *std.Build.Module, build_options: *st
const eos_policy = b.step("test-eos-policy-host", "Shared EOS, real HfText and HTTP policy without a GPU");
eos_policy.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"EOS policy:"} })).step);
eos_policy.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = mods.tokenizer, .filters = &.{"EOS policy:"} })).step);
- const eos_family = kolibri1Module(b, host, .debug, cuda, mods.core, mods.lanes);
- eos_policy.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = eos_family, .filters = &.{"EOS policy:"} })).step);
+ eos_policy.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = kolibri1Module(b, host, .debug, cuda, mods.core, mods.lanes), .filters = &.{"EOS policy:"} })).step);
b.step("test-dory-cli-host", "Dory CUDA loop option forwarding and invalid-value refusals").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"--device and --segments:"} })).step);
b.step("test-dory-chat-host", "Dory chat template, EOS, reasoning split and qwen3_coder call on the CPU").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"Dory chat:"} })).step);
b.step("test-embeddings-protocol-host", "Embedding request and response contracts without an admitted model or GPU").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"embeddings protocol:"} })).step);
b.step("test-embeddings-http-host", "Embedding route capability and borrowed request lifetime without a model or GPU").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"embeddings HTTP:"} })).step);
+ @import("rerank_host.zig").steps(b, server_tests);
@import("safetensors_header_host.zig").steps(b, mods.core, step);
b.step("test-runtime-capabilities-host", "Optional library discovery in actual capabilities JSON").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"runtime library:"} })).step);
b.step("test-vision-cli-host", "Image token budgets through parsing and engine admission options without a GPU").dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"vision image budget:"} })).step);
diff --git a/zig/build/rerank_host.zig b/zig/build/rerank_host.zig
new file mode 100644
index 0000000000..f83a3cc7ba
--- /dev/null
+++ b/zig/build/rerank_host.zig
@@ -0,0 +1,8 @@
+//! test-rerank-host: the /v1/rerank route and the reranker's labels on the CPU; TF_QWEN3_RERANK_* checks real ids.
+const std = @import("std");
+
+pub fn steps(b: *std.Build, server_tests: *std.Build.Module) void {
+ const run = b.addRunArtifact(b.addTest(.{ .root_module = server_tests, .filters = &.{"rerank:"} }));
+ run.has_side_effects = true; // TF_QWEN3_RERANK_DIR and _REF are not cached inputs
+ b.step("test-rerank-host", "Reranker labels and the /v1/rerank route; TF_QWEN3_RERANK_* checks real ids").dependOn(&run.step);
+}
diff --git a/zig/src/core/engine_api.zig b/zig/src/core/engine_api.zig
index 3c78d58e27..ff5c91bd89 100644
--- a/zig/src/core/engine_api.zig
+++ b/zig/src/core/engine_api.zig
@@ -144,6 +144,8 @@ pub const Sink = struct {
};
pub const Benchmark = @import("benchmark_info.zig").Snapshot;
+/// A reranker's label tokens: a prompt's relevance score is sigmoid(yes - no) of their last-row logits.
+pub const Rerank = struct { yes: u32, no: u32 };
/// What the engine serves, fixed once it has loaded.
pub const Info = struct {
@@ -198,6 +200,8 @@ pub const Info = struct {
decode_share: ?f64 = null,
/// The engine scores decisions (`score`) and writes no text: the server refuses text requests.
decides: bool = false,
+ /// The engine is a reranker (it scores through `score`): the server answers POST /v1/rerank only when set.
+ rerank: ?Rerank = null,
/// Decisions may carry pictures (`Decision.pictures`).
pictures: bool = false,
/// Image input (--vision); null: text only.
diff --git a/zig/src/core/lane_host_score.zig b/zig/src/core/lane_host_score.zig
index fc00d982cf..9318f1fdd2 100644
--- a/zig/src/core/lane_host_score.zig
+++ b/zig/src/core/lane_host_score.zig
@@ -47,6 +47,14 @@ pub fn answer(h: *LaneHost) void {
defer h.gpa.free(list);
const backend = h.core.backend;
for (list) |a| {
+ // a drain's deadline stops the queue: the started score finishes, the rest fail without running
+ h.lock();
+ const closing = h.closing;
+ h.unlock();
+ if (closing) {
+ a.failed = true;
+ continue;
+ }
const run = backend.vtable.score orelse {
a.failed = true;
continue;
diff --git a/zig/src/core/lane_host_score_test.zig b/zig/src/core/lane_host_score_test.zig
index 12d91eb3c8..043ae3e295 100644
--- a/zig/src/core/lane_host_score_test.zig
+++ b/zig/src/core/lane_host_score_test.zig
@@ -3,6 +3,7 @@ const std = @import("std");
const lanes = @import("lanes");
const api = @import("engine_api.zig");
const LaneHost = @import("lane_host.zig").LaneHost;
+const scoring = @import("lane_host_score.zig");
/// The fake backend's entries with a score of its own: label i scores prompt.len + i, the mass 7.5.
const Scorer = struct {
@@ -62,3 +63,48 @@ test "a backend's score runs on the engine thread, a failure comes back as Faile
try std.testing.expectError(error.Failed, engine.score(&.{1}, &.{2}, logits[0..1], false));
try std.testing.expectEqual(@as(usize, 2), Scorer.calls);
}
+
+/// Closes its host during the first score, as a drain's halt does while the engine thread answers a queue.
+const Closer = struct {
+ var host: ?*LaneHost = null;
+ var calls: usize = 0;
+
+ fn score(_: *anyopaque, _: []const u32, _: []const u32, logits: []f64, _: bool) anyerror!f64 {
+ calls += 1;
+ const h = host.?;
+ h.lock();
+ h.closing = true;
+ h.unlock();
+ @memset(logits, 1);
+ return 0;
+ }
+};
+
+test "a queue answered while the host closes runs the started score and fails the rest" {
+ const gpa = std.testing.allocator;
+ var cfg = try lanes.Config.init(gpa, .{}, 1, 0);
+ defer cfg.deinit(gpa);
+ var target: lanes.fake.Fake = .{ .gpa = gpa };
+ defer target.deinit();
+ const plain = target.backend();
+ var vt = plain.vtable.*;
+ vt.score = Closer.score;
+ var clock: lanes.fake.FixedClock = .{};
+ var core = lanes.Engine.init(gpa, &cfg, .{ .ptr = plain.ptr, .vtable = &vt }, clock.clock());
+ defer core.deinit();
+ var host = LaneHost.init(gpa, std.testing.io, &core, .{ .lanes = 1 });
+ defer host.scores.asks.deinit(gpa);
+ Closer.host = &host;
+ Closer.calls = 0;
+ var logits: [3][1]f64 = .{ .{0}, .{0}, .{0} };
+ var asks: [3]scoring.Ask = undefined;
+ for (&asks, &logits) |*ask, *row| {
+ ask.* = .{ .prompt = &.{1}, .labels = &.{2}, .logits = row, .mass = false };
+ try host.scores.asks.append(gpa, ask);
+ }
+ scoring.answer(&host);
+ try std.testing.expectEqual(@as(usize, 1), Closer.calls);
+ try std.testing.expect(!asks[0].failed and asks[0].done);
+ try std.testing.expectEqual(@as(f64, 1), logits[0][0]);
+ for (asks[1..]) |ask| try std.testing.expect(ask.failed and ask.done);
+}
diff --git a/zig/src/families/qwen3/native.zig b/zig/src/families/qwen3/native.zig
index 7efa2b9540..969a2757e3 100644
--- a/zig/src/families/qwen3/native.zig
+++ b/zig/src/families/qwen3/native.zig
@@ -5,6 +5,7 @@ const lanes = @import("lanes");
const api = @import("engine_api");
const engine = @import("engine.zig");
const prompt = @import("prompt.zig");
+const score = @import("score.zig");
const Engine = engine.Engine;
const Allocator = std.mem.Allocator;
@@ -39,6 +40,10 @@ pub const Loaded = struct {
embeddings: ?api.EmbeddingInfo = null,
/// The window the engine was sized to (the asked one, or less where the budget shrank it).
window: ?usize = null,
+ /// A reranker: the engine scores labels (the score hook) and serves no embeddings.
+ decides: bool = false,
+ /// The reranker's label tokens, from `1_LogitScore/config.json`.
+ rerank: ?api.Rerank = null,
};
pub const Refused = error{Refused};
@@ -52,6 +57,8 @@ fn refuse(a: Allocator, problem: *[]const u8, comptime fmt: []const u8, args: an
pub const Head = union(enum) {
/// sentence-transformers last-token pooling, unit-normed when modules.json lists Normalize.
embedding: struct { normalize: bool },
+ /// A reranker: `1_LogitScore` names the yes and no tokens its last row's logits score.
+ rerank: api.Rerank,
};
const pooling_modes = [_][]const u8{
@@ -61,6 +68,7 @@ const pooling_modes = [_][]const u8{
/// The head from `1_Pooling/config.json` and `modules.json`, or a refusal naming what the folder lacks.
pub fn headOf(a: Allocator, io: std.Io, dir: []const u8, problem: *[]const u8) (Refused || Allocator.Error)!Head {
+ if (try readJson(a, io, dir, "1_LogitScore/config.json", problem)) |v| return .{ .rerank = try labelsOf(a, v, problem) };
const pooling = try readJson(a, io, dir, "1_Pooling/config.json", problem) orelse return refuse(
a,
problem,
@@ -86,6 +94,21 @@ pub fn headOf(a: Allocator, io: std.Io, dir: []const u8, problem: *[]const u8) (
return .{ .embedding = .{ .normalize = normalize } };
}
+/// The yes and no tokens `1_LogitScore/config.json` names (the vocabulary bound is checked once the config is read).
+fn labelsOf(a: Allocator, v: std.json.Value, problem: *[]const u8) (Refused || Allocator.Error)!api.Rerank {
+ const name = "1_LogitScore/config.json";
+ if (v != .object) return refuse(a, problem, "{s} is not a JSON object", .{name});
+ var ids: [2]u32 = undefined;
+ for ([_][]const u8{ "true_token_id", "false_token_id" }, &ids) |key, *id| {
+ const x = v.object.get(key) orelse return refuse(a, problem, "{s} has no {s}", .{ name, key });
+ if (x != .integer or x.integer < 0 or x.integer > std.math.maxInt(u32))
+ return refuse(a, problem, "{s}: {s} is not a token id", .{ name, key });
+ id.* = @intCast(x.integer);
+ }
+ if (ids[0] == ids[1]) return refuse(a, problem, "{s} names token {d} for both answers", .{ name, ids[0] });
+ return .{ .yes = ids[0], .no = ids[1] };
+}
+
/// The folder's JSON file `name`, null when it is absent; unreadable or malformed JSON is refused.
fn readJson(a: Allocator, io: std.Io, dir: []const u8, name: []const u8, problem: *[]const u8) (Refused || Allocator.Error)!?std.json.Value {
const path = try std.fs.path.join(a, &.{ dir, name });
@@ -123,6 +146,8 @@ const Owned = struct {
e: *Engine,
p: prompt.Prompt,
startup: []const u8,
+ /// The checkpoint folder, kept for the reranker's head rows.
+ dir: []const u8,
};
fn release(ptr: *anyopaque) void {
@@ -130,6 +155,7 @@ fn release(ptr: *anyopaque) void {
own.p.deinit();
own.e.deinit();
own.gpa.free(own.startup);
+ own.gpa.free(own.dir);
own.gpa.destroy(own);
}
@@ -154,10 +180,34 @@ fn openEngine(a: Allocator, gpa: Allocator, io: std.Io, dev: hip.Device, dir: []
return refuse(a, problem, "the native HIP engine cannot allocate its scratch ({s})", .{@errorName(err)});
const own = try gpa.create(Owned);
errdefer gpa.destroy(own);
- own.* = .{ .gpa = gpa, .e = e, .p = prompt.Prompt.init(gpa, e, head.embedding.normalize), .startup = undefined };
+ const normalize = switch (head) {
+ .embedding => |h| h.normalize,
+ .rerank => false,
+ };
+ const labels: ?api.Rerank = switch (head) {
+ .embedding => null,
+ .rerank => |l| l,
+ };
+ if (labels) |l| if (l.yes >= e.m.spec.vocab or l.no >= e.m.spec.vocab)
+ return refuse(a, problem, "1_LogitScore/config.json: tokens {d} and {d} are not both in this {d}-token vocabulary", .{ l.yes, l.no, e.m.spec.vocab });
+ const kept = try gpa.dupe(u8, dir);
+ errdefer gpa.free(kept);
+ own.* = .{ .gpa = gpa, .e = e, .p = prompt.Prompt.init(gpa, e, normalize), .startup = undefined, .dir = kept };
+ errdefer own.p.deinit();
+ if (labels) |l| {
+ own.p.heads = score.Heads.of(&e.m, io, own.dir);
+ own.p.table.score = prompt.Prompt.scoreLabels;
+ // both label rows read at load: a bad head fails startup, and the first request reads nothing
+ for ([_]u32{ l.yes, l.no }) |id| _ = own.p.heads.?.row(gpa, id) catch |err|
+ return refuse(a, problem, "the native HIP engine cannot read lm_head row {d} ({s})", .{ id, @errorName(err) });
+ }
const shrunk: []const u8 = if (window < o.window) " (shrunk from the asked window to fit the budget)" else "";
- own.startup = try std.fmt.allocPrint(gpa, "HIP {s} device {d}: qwen3 embedding, model {d:.2} GiB; {d}-token passes{s}, {d:.2} GiB scratch, of {d:.1} GiB left after a {d:.1} GiB reserve; vectors {s}", .{
- hip.kernels.arch, dev.index, gibs(e.weights), window, shrunk, gibs(e.scratchBytes(window) catch 0), gibs(room), gibs(pool.reserve), if (head.embedding.normalize) "unit-normed" else "as pooled",
+ own.startup = try std.fmt.allocPrint(gpa, "HIP {s} device {d}: qwen3 {s}, model {d:.2} GiB; {d}-token passes{s}, {d:.2} GiB scratch, of {d:.1} GiB left after a {d:.1} GiB reserve; {s}", .{
+ hip.kernels.arch, dev.index, if (head == .rerank) "reranker" else "embedding", gibs(e.weights), window, shrunk, gibs(e.scratchBytes(window) catch 0), gibs(room), gibs(pool.reserve),
+ switch (head) {
+ .embedding => |h| if (h.normalize) "vectors unit-normed" else "vectors as pooled",
+ .rerank => "label scores from lm_head rows read at load, on the host",
+ },
});
const spec = e.m.spec;
return .{
@@ -169,8 +219,10 @@ fn openEngine(a: Allocator, gpa: Allocator, io: std.Io, dev: hip.Device, dir: []
.streams = 1,
.startup = own.startup,
.text_generation = false,
- .embeddings = .{ .dimensions = @intCast(spec.hidden), .vocabulary = @intCast(spec.vocab), .max_batch_tokens = @intCast(window), .max_inputs = @intCast(engine.max_inputs), .mrl_min = @min(32, @as(u32, @intCast(spec.hidden))) },
+ .embeddings = if (head == .rerank) null else .{ .dimensions = @intCast(spec.hidden), .vocabulary = @intCast(spec.vocab), .max_batch_tokens = @intCast(window), .max_inputs = @intCast(engine.max_inputs), .mrl_min = @min(32, @as(u32, @intCast(spec.hidden))) },
.window = window,
+ .decides = head == .rerank,
+ .rerank = labels,
};
}
@@ -230,3 +282,39 @@ test "a pooling folder declares the embedding head: last token only, normalized
try std.testing.expectError(error.Refused, headOf(a, io, dir, &problem));
try std.testing.expect(std.mem.indexOf(u8, problem, "pooling_mode_mean_tokens") != null);
}
+
+test "a 1_LogitScore folder declares the reranker head, before any pooling, and a malformed one is refused" {
+ var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ const dir = try tmp.dir.realPathFileAlloc(io, ".", a);
+ var problem: []const u8 = "";
+ try tmp.dir.createDirPath(io, "1_LogitScore");
+ try tmp.dir.writeFile(io, .{ .sub_path = "1_LogitScore/config.json", .data = "{\"true_token_id\": 9693, \"false_token_id\": 2152}" });
+ const labels = (try headOf(a, io, dir, &problem)).rerank;
+ try std.testing.expectEqual(api.Rerank{ .yes = 9693, .no = 2152 }, labels);
+ // the reranker's folder also carries no pooling; one that did would still be a reranker by its scores
+ try tmp.dir.createDirPath(io, "1_Pooling");
+ try tmp.dir.writeFile(io, .{ .sub_path = "1_Pooling/config.json", .data = "{\"pooling_mode_lasttoken\": true}" });
+ try std.testing.expect(try headOf(a, io, dir, &problem) == .rerank);
+ const bad = [_][]const u8{
+ "{not json",
+ "[9693, 2152]",
+ "{\"false_token_id\": 2152}",
+ "{\"true_token_id\": 9693}",
+ "{\"true_token_id\": \"9693\", \"false_token_id\": 2152}",
+ "{\"true_token_id\": 9693.5, \"false_token_id\": 2152}",
+ "{\"true_token_id\": -1, \"false_token_id\": 2152}",
+ "{\"true_token_id\": 4294967296, \"false_token_id\": 2152}",
+ "{\"true_token_id\": 2152, \"false_token_id\": 2152}",
+ };
+ for (bad) |text| {
+ problem = "";
+ try tmp.dir.writeFile(io, .{ .sub_path = "1_LogitScore/config.json", .data = text });
+ try std.testing.expectError(error.Refused, headOf(a, io, dir, &problem));
+ try std.testing.expect(std.mem.indexOf(u8, problem, "1_LogitScore/config.json") != null);
+ }
+}
diff --git a/zig/src/families/qwen3/prompt.zig b/zig/src/families/qwen3/prompt.zig
index 2ff8e34887..6bc6fa744f 100644
--- a/zig/src/families/qwen3/prompt.zig
+++ b/zig/src/families/qwen3/prompt.zig
@@ -3,6 +3,7 @@ const std = @import("std");
const lanes = @import("lanes");
const api = @import("engine_api");
const Engine = @import("engine.zig").Engine;
+const score = @import("score.zig");
const be = lanes.backend;
const Stream = lanes.Stream;
@@ -18,6 +19,8 @@ pub const Prompt = struct {
normalize: bool,
/// Each stream's pooled fp32 rows, from its prompt pass to its release.
slots: std.AutoHashMapUnmanaged(*Stream, []f32) = .empty,
+ /// The reranker's head rows, when the checkpoint scores labels (`scoreLabels` is then in `table`).
+ heads: ?score.Heads = null,
table: be.Backend.VTable = .{
.embedding = embedding,
.prefill = prefill,
@@ -39,6 +42,20 @@ pub const Prompt = struct {
var it = p.slots.valueIterator();
while (it.next()) |rows| p.gpa.free(rows.*);
p.slots.deinit(p.gpa);
+ if (p.heads) |*h| h.deinit(p.gpa);
+ }
+
+ /// The score hook: one prompt's last row through the engine, its label logits in f64; no mass.
+ pub fn scoreLabels(ptr: *anyopaque, ids: []const u32, labels: []const u32, logits: []f64, mass: bool) anyerror!f64 {
+ const p = of(ptr);
+ const heads = if (p.heads) |*h| h else return error.NoScoreHead;
+ const e = p.engine;
+ try score.check(ids, labels, logits, mass, e.window, e.m.spec.vocab, heads.count);
+ const row = try p.gpa.alloc(f32, e.m.spec.hidden);
+ defer p.gpa.free(row);
+ try e.run(ids, &.{@intCast(ids.len)}, row);
+ try score.labelLogits(p.gpa, heads, row, labels, logits);
+ return 0;
}
/// The backend over this engine; heads add their hooks to `table` before it serves.
diff --git a/zig/src/families/qwen3/qwen3.zig b/zig/src/families/qwen3/qwen3.zig
index 775e417391..a8825e6829 100644
--- a/zig/src/families/qwen3/qwen3.zig
+++ b/zig/src/families/qwen3/qwen3.zig
@@ -3,6 +3,7 @@
pub const config = @import("config.zig");
pub const model = @import("model.zig");
pub const forward = @import("forward.zig");
+pub const score = @import("score.zig");
pub const engine = @import("engine.zig");
pub const prompt = @import("prompt.zig");
pub const native = @import("native.zig");
@@ -15,6 +16,7 @@ test {
_ = config;
_ = model;
_ = forward;
+ _ = score;
_ = engine;
_ = prompt;
_ = native;
diff --git a/zig/src/families/qwen3/score.zig b/zig/src/families/qwen3/score.zig
new file mode 100644
index 0000000000..e1210ab8d0
--- /dev/null
+++ b/zig/src/families/qwen3/score.zig
@@ -0,0 +1,96 @@
+//! The reranker's label logits: a prompt's final-normed last row dotted in f64 with head rows read on the host.
+const std = @import("std");
+const model = @import("model.zig");
+const Allocator = std.mem.Allocator;
+
+/// Labels one call may ask for: a reranker asks two, a decision a few options.
+pub const labels_most = 16;
+
+pub const Refusal = error{ MassUnsupported, BadPrompt, BadLabels };
+
+/// Head rows read so far, by token id, as fp32 host rows: the 1.24 GB head never goes to the device.
+pub const Heads = struct {
+ rows: std.AutoHashMapUnmanaged(u32, []f32) = .empty,
+ io: std.Io,
+ /// The checkpoint directory and the head's stored name (`Model.head`).
+ dir: []const u8,
+ key: []const u8,
+ hidden: usize,
+ /// Rows the head has: the vocabulary of the checkpoint's config.
+ count: usize,
+
+ pub fn of(m: *const model.Model, io: std.Io, dir: []const u8) Heads {
+ return .{ .io = io, .dir = dir, .key = m.head, .hidden = m.spec.hidden, .count = m.spec.vocab };
+ }
+
+ /// Row `id`, read once from the checkpoint.
+ pub fn row(h: *Heads, gpa: Allocator, id: u32) ![]const f32 {
+ if (h.rows.get(id)) |r| return r;
+ const r = try model.readRows(gpa, h.io, h.dir, h.key, &.{id});
+ errdefer gpa.free(r);
+ if (r.len != h.hidden) return error.BadHeadRow;
+ try h.rows.put(gpa, id, r);
+ return r;
+ }
+
+ pub fn deinit(h: *Heads, gpa: Allocator) void {
+ var it = h.rows.valueIterator();
+ while (it.next()) |r| gpa.free(r.*);
+ h.rows.deinit(gpa);
+ }
+};
+
+/// A call's checks before any GPU work: a bad call fails, it never reads out of bounds.
+pub fn check(prompt: []const u32, labels: []const u32, logits: []const f64, mass: bool, window: usize, vocab: usize, head_rows: usize) Refusal!void {
+ if (mass) return error.MassUnsupported; // only /v1/decisions asks, and a reranker server refuses that route
+ if (prompt.len == 0 or prompt.len > window) return error.BadPrompt;
+ for (prompt) |id| if (id >= vocab) return error.BadPrompt;
+ if (labels.len == 0 or labels.len > labels_most or logits.len != labels.len) return error.BadLabels;
+ for (labels) |id| if (id >= head_rows) return error.BadLabels;
+}
+
+/// Each label's logit from the last row (its BF16 values widened, as Engine.run gives them): sum in f64.
+pub fn labelLogits(gpa: Allocator, heads: *Heads, row: []const f32, labels: []const u32, out: []f64) !void {
+ if (row.len != heads.hidden) return error.BadRow;
+ for (labels, out) |id, *x| {
+ const head = try heads.row(gpa, id);
+ var s: f64 = 0;
+ for (row, head) |v, w| s += @as(f64, v) * @as(f64, w);
+ x.* = s;
+ }
+}
+
+test "head rows are read once each, and a label's logit is the f64 dot of the BF16 row with its head row" {
+ const a = std.testing.allocator;
+ const io = std.testing.io;
+ var tmp = std.testing.tmpDir(.{});
+ defer tmp.cleanup();
+ // lm_head rows: 3 rows of 2 bf16 values, row r = (r + 1, -(r + 1))
+ const head = [_]u16{ 0x3f80, 0xbf80, 0x4000, 0xc000, 0x4040, 0xc040 };
+ try @import("core").checkpoint_fixture.write(a, io, tmp.dir, "model.safetensors", &.{
+ .{ .name = "lm_head.weight", .dtype = "BF16", .shape = &.{ 3, 2 }, .bytes = std.mem.sliceAsBytes(&head) },
+ });
+ const dir = try std.fmt.allocPrint(a, ".zig-cache/tmp/{s}", .{tmp.sub_path});
+ defer a.free(dir);
+ var heads: Heads = .{ .io = io, .dir = dir, .key = "lm_head.weight", .hidden = 2, .count = 3 };
+ defer heads.deinit(a);
+ const row = [_]f32{ 1.5, 0.5 };
+ var out: [2]f64 = undefined;
+ try labelLogits(a, &heads, &row, &.{ 2, 0 }, &out);
+ try std.testing.expectEqualSlices(f64, &.{ 3, 1 }, &out); // 1.5 * 3 - 0.5 * 3, 1.5 * 1 - 0.5 * 1
+ try labelLogits(a, &heads, &row, &.{2}, out[0..1]);
+ try std.testing.expectError(error.BadRow, labelLogits(a, &heads, row[0..1], &.{2}, out[0..1]));
+ try std.testing.expectEqual(@as(u32, 2), heads.rows.count());
+}
+
+test "a bad call is refused before any work" {
+ var out: [2]f64 = undefined;
+ try check(&.{ 1, 2 }, &.{ 3, 4 }, &out, false, 8, 10, 10);
+ try std.testing.expectError(error.MassUnsupported, check(&.{1}, &.{ 3, 4 }, &out, true, 8, 10, 10));
+ try std.testing.expectError(error.BadPrompt, check(&.{}, &.{ 3, 4 }, &out, false, 8, 10, 10));
+ try std.testing.expectError(error.BadPrompt, check(&.{ 1, 2, 3 }, &.{ 3, 4 }, &out, false, 2, 10, 10));
+ try std.testing.expectError(error.BadPrompt, check(&.{10}, &.{ 3, 4 }, &out, false, 8, 10, 10));
+ try std.testing.expectError(error.BadLabels, check(&.{1}, &.{}, out[0..0], false, 8, 10, 10));
+ try std.testing.expectError(error.BadLabels, check(&.{1}, &.{ 3, 10 }, &out, false, 8, 10, 10));
+ try std.testing.expectError(error.BadLabels, check(&.{1}, &.{3}, &out, false, 8, 10, 10));
+}
diff --git a/zig/src/native/hip.zig b/zig/src/native/hip.zig
index 3a74134408..ed5f1d0d81 100644
--- a/zig/src/native/hip.zig
+++ b/zig/src/native/hip.zig
@@ -312,7 +312,7 @@ fn openWith(comptime F: type, a: Allocator, gpa: Allocator, io: std.Io, o: api.O
h.core = lanes.Engine.init(gpa, &h.cfg, loaded.backend, h.clock.clock());
errdefer h.core.deinit();
const what = Serves.of(loaded, r.options.window);
- h.host = api.LaneHost.init(gpa, io, &h.core, .{ .benchmark_ranks = if (r.dev.group) |g| @intCast(g.world) else 1, .lanes = @intCast(loaded.streams), .context_window = @intCast(what.window), .prefill_step = F.prefill_step, .decode_share = o.decode_share, .policy = h.policy_line, .startup = loaded.startup, .structures = what.text_generation, .text_generation = what.text_generation, .embeddings = what.embeddings, .vision = visionOffer(loaded), .reasoning = reasoning(F) });
+ h.host = api.LaneHost.init(gpa, io, &h.core, .{ .benchmark_ranks = if (r.dev.group) |g| @intCast(g.world) else 1, .lanes = @intCast(loaded.streams), .context_window = @intCast(what.window), .prefill_step = F.prefill_step, .decode_share = o.decode_share, .policy = h.policy_line, .startup = loaded.startup, .structures = what.text_generation, .text_generation = what.text_generation, .embeddings = what.embeddings, .decides = what.decides, .rerank = what.rerank, .vision = visionOffer(loaded), .reasoning = reasoning(F) });
h.host.memory = .{ .read = readMemory };
if (@hasField(@TypeOf(loaded), "cache")) {
h.host.cache = loaded.cache;
@@ -328,6 +328,8 @@ fn openWith(comptime F: type, a: Allocator, gpa: Allocator, io: std.Io, o: api.O
const Serves = struct {
text_generation: bool = true,
embeddings: ?api.EmbeddingInfo = null,
+ decides: bool = false,
+ rerank: ?api.Rerank = null,
window: usize,
fn of(loaded: anytype, asked: usize) Serves {
@@ -335,6 +337,8 @@ const Serves = struct {
var s: Serves = .{ .window = asked };
if (@hasField(L, "text_generation")) s.text_generation = loaded.text_generation;
if (@hasField(L, "embeddings")) s.embeddings = loaded.embeddings;
+ if (@hasField(L, "decides")) s.decides = loaded.decides;
+ if (@hasField(L, "rerank")) s.rerank = loaded.rerank;
if (@hasField(L, "window")) if (loaded.window) |w| {
s.window = w;
};
@@ -524,4 +528,10 @@ test "a family's Loaded names what it serves; one without the fields generates t
try std.testing.expect(!e.text_generation and e.embeddings.?.dimensions == 8 and e.window == 2048);
const Unsized = struct { window: ?usize };
try std.testing.expectEqual(@as(usize, 32768), Serves.of(Unsized{ .window = null }, 32768).window);
+ const Rerank = struct { text_generation: bool, decides: bool, rerank: ?api.Rerank, window: ?usize };
+ const r = Serves.of(Rerank{ .text_generation = false, .decides = true, .rerank = .{ .yes = 9693, .no = 2152 }, .window = 32768 }, 40960);
+ try std.testing.expect(!r.text_generation and r.decides and r.embeddings == null and r.window == 32768);
+ try std.testing.expectEqual(@as(u32, 9693), r.rerank.?.yes);
+ try std.testing.expect(p.rerank == null and e.rerank == null);
+ try std.testing.expect(!p.decides and !e.decides);
}
diff --git a/zig/src/server/auth_scope.zig b/zig/src/server/auth_scope.zig
index f7d3cd5aef..bab9a9487f 100644
--- a/zig/src/server/auth_scope.zig
+++ b/zig/src/server/auth_scope.zig
@@ -64,6 +64,7 @@ const inferring = [_][]const u8{
"/responses", "/v1/responses", "/tokenize", "/v1/tokenize",
"/detokenize", "/v1/detokenize", "/v1/audio/speech", "/v1/embeddings",
"/decisions", "/v1/decisions", "/systemone", "/v1/systemone",
+ "/rerank", "/v1/rerank",
};
/// A gated request a key may make: the route as the dispatcher reads it, matched exactly, never by suffix.
@@ -74,7 +75,13 @@ pub fn allows(scopes: Scopes, method: []const u8, raw_path: []const u8) bool {
if ((scopes.inference or scopes.read) and oneOf(route, &listing)) return true;
return scopes.read and oneOf(route, &reading);
}
- return scopes.inference and std.mem.eql(u8, method, "POST") and oneOf(route, &inferring);
+ const put = std.mem.eql(u8, method, "PUT") and putRoute(route);
+ return scopes.inference and (std.mem.eql(u8, method, "POST") or put) and oneOf(route, &inferring);
+}
+
+/// The routes a PUT reaches as it would by POST: SGLang's rerank route takes both.
+pub fn putRoute(route: []const u8) bool {
+ return std.mem.eql(u8, route, "/v1/rerank") or std.mem.eql(u8, route, "/rerank");
}
fn oneOf(route: []const u8, names: []const []const u8) bool {
diff --git a/zig/src/server/rerank.zig b/zig/src/server/rerank.zig
new file mode 100644
index 0000000000..37cb4b0ecd
--- /dev/null
+++ b/zig/src/server/rerank.zig
@@ -0,0 +1,414 @@
+//! POST /v1/rerank in #1895's shape, SGLang's request fields too, each document scored by the engine's score hook.
+const std = @import("std");
+const api = @import("engine_api");
+const json = @import("json");
+const errors = @import("errors.zig");
+const openai = @import("openai.zig");
+const http_body = @import("http_body.zig");
+const reply_text = @import("reply_text.zig");
+const model_text = @import("model_text.zig");
+const Server = @import("server.zig").Server;
+const Conn = @import("http_conn.zig").Conn;
+const Value = json.Value;
+const Cx = errors.Cx;
+const Allocator = std.mem.Allocator;
+
+pub const Failure = errors.Refused || error{Cancelled};
+
+/// The request's fields once checked; `documents` holds each document's text, `given` the item as the request sent it.
+const Ask = struct {
+ query: []const u8,
+ documents: []const []const u8,
+ given: []const Value,
+ instruct: ?[]const u8,
+ top_n: ?usize,
+ /// #1895's cap on a prompt's tokens: a longer prompt's content is cut from its end to fit.
+ max_length: usize,
+ return_documents: bool,
+ return_logits: bool,
+ return_ids: bool,
+};
+
+/// #1895's default cap on a prompt's tokens: the model card's max_length.
+pub const max_length_default = 8192;
+
+/// One document's score and the label logits it came from.
+const Scored = struct { score: f64, yes: f64, no: f64 };
+
+pub fn post(srv: *Server, conn: *Conn, a: Allocator) void {
+ var cx: Cx = .{ .a = a };
+ const raw = http_body.readJson(conn, &cx) catch |err| {
+ if (err == error.OutOfMemory) outOfMemory(&cx);
+ return sendErr(conn, &cx);
+ };
+ const Gone = struct {
+ conn: *Conn,
+ fn check(g: @This()) bool {
+ return g.conn.peerGone();
+ }
+ };
+ const value = reply(srv, &cx, raw, Gone{ .conn = conn }) catch |err| {
+ if (err == error.Cancelled) {
+ conn.close = true;
+ return;
+ }
+ if (err == error.OutOfMemory) outOfMemory(&cx);
+ return sendErr(conn, &cx);
+ };
+ send(conn, a, 200, value);
+}
+
+/// Compact separators and UTF-8 as it is.
+pub fn encode(a: Allocator, v: Value) Allocator.Error![]u8 {
+ return json.stringify(a, v, .{ .compact = true, .ascii = false }) catch return error.OutOfMemory;
+}
+
+fn send(conn: *Conn, a: Allocator, code: u16, v: Value) void {
+ conn.sendJson(code, encode(a, v) catch return);
+}
+
+fn outOfMemory(cx: *Cx) void {
+ cx.kind = .server;
+ cx.message = "out of memory";
+}
+
+/// The server's error envelope, as every route sends it: `{"error": {"message", "type", ...}}`.
+pub fn errorBody(cx: *const Cx) Allocator.Error!Value {
+ return openai.wrapError(cx.a, try openai.errorBody(cx.a, cx, "documents"));
+}
+
+fn sendErr(conn: *Conn, cx: *Cx) void {
+ const body = errorBody(cx) catch return;
+ send(conn, cx.a, cx.status(), body);
+}
+
+/// The ranked documents, or the refusal; `gone` says the client left (checked before each document's pass).
+pub fn reply(srv: *Server, cx: *Cx, body: Value, gone: anytype) Failure!Value {
+ const labels = srv.info.rerank orelse return cx.refuse("this model is not a reranker");
+ const ask = try parse(cx, body);
+ // each prompt is built in a scratch arena and dropped: a small body of many documents keeps no ids alive
+ var scratch = std.heap.ArenaAllocator.init(srv.gpa);
+ defer scratch.deinit();
+ var frame: ?Frame = null;
+ // every document fits the window before the first pass, so a refusal costs no GPU work
+ for (ask.documents, 0..) |doc, i| {
+ if (gone.check()) return error.Cancelled;
+ _ = scratch.reset(.retain_capacity);
+ _ = try ready(srv, cx, scratch.allocator(), ask, &frame, doc, i);
+ }
+ const scored = try cx.a.alloc(Scored, ask.documents.len);
+ var tokens: usize = 0;
+ for (ask.documents, scored, 0..) |doc, *row, i| {
+ if (gone.check()) return error.Cancelled;
+ _ = scratch.reset(.retain_capacity);
+ const ids = try ready(srv, cx, scratch.allocator(), ask, &frame, doc, i);
+ tokens += ids.len;
+ var logits: [2]f64 = undefined;
+ _ = srv.engine.score(ids, &.{ labels.yes, labels.no }, &logits, false) catch |err| switch (err) {
+ error.Unsupported => return cx.refuse("this engine does not score labels"),
+ error.Failed => return if (healthy(srv))
+ cx.fail(.server, "rerank inference failed", .{})
+ else
+ cx.fail(.capacity, "the rerank engine is unavailable", .{}),
+ };
+ // a NaN or an infinity is no score, and would not even be JSON
+ if (!std.math.isFinite(logits[0]) or !std.math.isFinite(logits[1])) return cx.fail(.server, "rerank inference failed", .{});
+ row.* = .{ .score = 1.0 / (1.0 + @exp(logits[1] - logits[0])), .yes = logits[0], .no = logits[1] };
+ }
+ return build(srv, cx, ask, &frame, scored, tokens);
+}
+
+fn healthy(srv: *Server) bool {
+ var st: api.Status = .{};
+ srv.engine.status(&st, &.{});
+ return st.healthy;
+}
+
+/// The request (#1895's fields and SGLang's): its schema whole first ("Bad Request"), then the handler's checks.
+fn parse(cx: *Cx, body: Value) errors.Refused!Ask {
+ if (body != .object) return cx.other("body: input should be a valid dictionary");
+ const query = try content(cx, body.get("query") orelse return cx.other("query: field required"), "query");
+ const docs = body.get("documents") orelse return cx.other("documents: field required");
+ if (docs != .array) return cx.other("documents: input should be a valid list");
+ const parts = try cx.a.alloc(Content, docs.array.len);
+ for (docs.array, parts) |d, *out| out.* = try content(cx, d, "documents");
+ // `instruction` as #1895 names it; SGLang's clients send `instruct`
+ const instruct: ?[]const u8 = switch (body.get("instruction") orelse body.get("instruct") orelse Value.null) {
+ .null => null,
+ .string => |str| if (str.len == 0) null else str,
+ else => return cx.other("instruction: input should be a valid string"),
+ };
+ const top_n: ?usize = switch (body.get("top_n") orelse Value.null) {
+ .null => null,
+ else => |v| try topN(cx, v),
+ };
+ const max_length: usize = switch (body.get("max_length") orelse Value.null) {
+ .null => max_length_default,
+ .int => |d| if (d[0] == '-' or std.mem.eql(u8, d, "0")) return cx.other("max_length: input should be at least 1") else std.fmt.parseInt(usize, d, 10) catch std.math.maxInt(usize),
+ else => return cx.other("max_length: input should be a valid integer"),
+ };
+ const documents = try cx.a.alloc([]const u8, parts.len);
+ const ask: Ask = .{
+ .query = query.text,
+ .documents = documents,
+ .given = docs.array,
+ .instruct = instruct,
+ .top_n = top_n,
+ .max_length = max_length,
+ .return_documents = try flag(cx, body, "return_documents", true),
+ .return_logits = try flag(cx, body, "return_logits", false),
+ .return_ids = try flag(cx, body, "return_prompt_token_ids", false),
+ };
+ // the handler's checks, in its order (serving_rerank.py _validate_request), then this reranker's own
+ if (query.empty) return cx.refuse("Query cannot be empty");
+ if (query.string and reply_text.pyStrip(query.text).len == 0) return cx.refuse("Query cannot be empty or whitespace only");
+ if (parts.len == 0) return cx.refuse("Documents cannot be empty");
+ for (parts) |d| {
+ if (d.empty) return cx.refuse("Each document must be a non-empty string");
+ if (d.string and reply_text.pyStrip(d.text).len == 0) return cx.refuse("Each document cannot be empty or whitespace only");
+ }
+ if (query.media) |kind| return mediaRefused(cx, kind);
+ for (parts) |d| if (d.media) |kind| return mediaRefused(cx, kind);
+ // whitespace text parts: SGLang leaves them to the model; this reranker refuses a prompt with nothing to read
+ if (reply_text.pyStrip(query.text).len == 0) return cx.refuse("Query cannot be empty or whitespace only");
+ for (parts, documents) |d, *out| {
+ if (reply_text.pyStrip(d.text).len == 0) return cx.refuse("Each document cannot be empty or whitespace only");
+ out.* = d.text;
+ }
+ return ask;
+}
+
+/// A query or document as the schema reads it; `media` names a well-formed image or video part (unreadable here).
+const Content = struct { text: []const u8, string: bool, empty: bool, media: ?[]const u8 = null };
+
+fn content(cx: *Cx, v: Value, comptime field: []const u8) errors.Refused!Content {
+ switch (v) {
+ .string => |str| return .{ .text = str, .string = true, .empty = str.len == 0 },
+ // a `{"text": ...}` document, as rerank clients send it
+ .object => {
+ const str = v.strField("text") orelse return cx.other(field ++ ": an object needs its text as a string");
+ return .{ .text = str, .string = true, .empty = str.len == 0 };
+ },
+ .array => |items| {
+ var out: std.ArrayList(u8) = .empty;
+ var media: ?[]const u8 = null;
+ var texts: usize = 0;
+ for (items) |part| {
+ const kind = part.strField("type") orelse return cx.other(field ++ ": each content part needs a type");
+ if (std.mem.eql(u8, kind, "text")) {
+ const t = part.strField("text") orelse return cx.other(field ++ ": a text part needs its text as a string");
+ if (texts > 0) try out.append(cx.a, ' ');
+ try out.appendSlice(cx.a, t);
+ texts += 1;
+ } else if (std.mem.eql(u8, kind, "image_url") or std.mem.eql(u8, kind, "video_url")) {
+ // the part's payload, as the schema requires it: {"": {"url": "..."}}
+ const payload = part.get(kind) orelse return cx.fail(.other, field ++ ": a {s} part needs its {s} object", .{ kind, kind });
+ if (payload.strField("url") == null) return cx.fail(.other, field ++ ": a {s} part needs a url string", .{kind});
+ media = media orelse kind;
+ } else return cx.fail(.other, field ++ ": {s} is not a content part type", .{kind});
+ }
+ return .{ .text = out.items, .string = false, .empty = items.len == 0, .media = media };
+ },
+ else => return cx.other(field ++ ": input should be a string, a {\"text\"} object or a list of content parts"),
+ }
+}
+
+fn mediaRefused(cx: *Cx, kind: []const u8) errors.Refused {
+ return cx.fail(.request, "this reranker reads text: {s} parts are not supported", .{kind});
+}
+
+/// top_n as pydantic's lax int takes it: an integer, an integral float or string, or a bool; at least 1.
+fn topN(cx: *Cx, v: Value) errors.Refused!usize {
+ const bad = "top_n: input should be a valid integer";
+ const digits: []const u8, const negative: bool = switch (v) {
+ .int => |d| if (d[0] == '-') .{ d[1..], true } else .{ d, false },
+ .bool => |b| .{ if (b) "1" else "0", false },
+ .float => |x| blk: {
+ // a float's integer within i64, as pydantic converts it; anything past it or fractional is refused
+ if (!std.math.isFinite(x) or x != @trunc(x) or @abs(x) >= 0x1p63) return cx.other(bad);
+ if (x < 1) return cx.other("Value error, parameter top_n should be larger than 0.");
+ break :blk .{ std.fmt.allocPrint(cx.a, "{d}", .{@as(u64, @intFromFloat(x))}) catch return error.OutOfMemory, false };
+ },
+ .string => |raw| try intText(cx, reply_text.pyStrip(raw), bad),
+ else => return cx.other(bad),
+ };
+ const n = std.fmt.parseInt(usize, digits, 10) catch |err| switch (err) {
+ error.Overflow => std.math.maxInt(usize), // any positive integer: more than the documents returns them all
+ error.InvalidCharacter => return cx.other(bad),
+ };
+ if (negative or n == 0) return cx.other("Value error, parameter top_n should be larger than 0.");
+ return n;
+}
+
+/// A string's integer as pydantic reads it (sign, single underscores, an all-zero fraction): its digits and sign.
+fn intText(cx: *Cx, str: []const u8, comptime bad: []const u8) errors.Refused!struct { []const u8, bool } {
+ var rest = str;
+ var negative = false;
+ if (rest.len > 0 and (rest[0] == '-' or rest[0] == '+')) {
+ negative = rest[0] == '-';
+ rest = rest[1..];
+ }
+ const whole = if (std.mem.indexOfScalar(u8, rest, '.')) |dot| blk: {
+ if (dot + 1 == rest.len) return cx.other(bad); // "2." has no fraction to be zero
+ for (rest[dot + 1 ..]) |c| if (c != '0') return cx.other(bad);
+ break :blk rest[0..dot];
+ } else rest;
+ if (whole.len == 0 or whole[0] == '_' or whole[whole.len - 1] == '_') return cx.other(bad);
+ var out: std.ArrayList(u8) = .empty;
+ for (whole, 0..) |c, i| {
+ if (c == '_') {
+ if (whole[i - 1] == '_') return cx.other(bad);
+ continue;
+ }
+ if (c < '0' or c > '9') return cx.other(bad);
+ try out.append(cx.a, c);
+ }
+ return .{ out.items, negative };
+}
+
+/// An optional boolean as pydantic's lax bool takes it (true/false, 0/1, and its yes/no words); null is refused.
+fn flag(cx: *Cx, body: Value, comptime key: []const u8, default: bool) errors.Refused!bool {
+ const v = body.get(key) orelse return default;
+ const bad = key ++ ": input should be a valid boolean";
+ return switch (v) {
+ .bool => |b| b,
+ .int => |d| if (std.mem.eql(u8, d, "0")) false else if (std.mem.eql(u8, d, "1")) true else cx.other(bad),
+ .float => |x| if (x == 0) false else if (x == 1) true else cx.other(bad),
+ .string => |str| blk: {
+ var buf: [8]u8 = undefined;
+ if (str.len > buf.len) break :blk cx.other(bad);
+ const low = std.ascii.lowerString(&buf, str);
+ for ([_][]const u8{ "0", "off", "f", "false", "n", "no" }) |w| if (std.mem.eql(u8, low, w)) break :blk false;
+ for ([_][]const u8{ "1", "on", "t", "true", "y", "yes" }) |w| if (std.mem.eql(u8, low, w)) break :blk true;
+ break :blk cx.other(bad);
+ },
+ else => cx.other(bad),
+ };
+}
+
+/// The template's tail after the document, from probe documents "0" and "1": a cut keeps it whole (see `ready`).
+const Frame = struct { p0: []const u32, p1: []const u32, suffix: usize, tail: []const u8, fixed: bool };
+
+fn frameOf(srv: *Server, cx: *Cx, ask: Ask) errors.Refused!Frame {
+ const p0 = try prompt(srv, cx, cx.a, ask, "0");
+ const p1 = try prompt(srv, cx, cx.a, ask, "1");
+ const both = @min(p0.ids.len, p1.ids.len);
+ var suffix: usize = 0;
+ while (suffix < both and p0.ids[p0.ids.len - 1 - suffix] == p1.ids[p1.ids.len - 1 - suffix]) suffix += 1;
+ // the whole rendered tail, encoded alone, must be exactly the shared ids: no template byte merged into the document
+ const alone = srv.text.encode(cx.a, p0.tail, false) catch |e| switch (e) {
+ error.OutOfMemory => return error.OutOfMemory,
+ else => return cx.fail(.request, "the chat template failed: {s}", .{"the tokenizer cannot encode this prompt"}),
+ };
+ // a cut needs a tail that starts at an added token (the masked encoder splits only there) and renders alike
+ const fixed = suffix > 0 and suffix < both and std.mem.eql(u8, p0.tail, p1.tail) and
+ srv.text.isAdded(p0.ids[p0.ids.len - suffix]) and std.mem.eql(u32, alone, p0.ids[p0.ids.len - suffix ..]);
+ return .{ .p0 = p0.ids, .p1 = p1.ids, .suffix = suffix, .tail = p0.tail, .fixed = fixed };
+}
+
+/// The length of the ids `x` and `y` begin with alike.
+fn common(x: []const u32, y: []const u32) usize {
+ var n: usize = 0;
+ while (n < @min(x.len, y.len) and x[n] == y[n]) n += 1;
+ return n;
+}
+
+/// Document `i`'s prompt as scored: past max_length its content loses its last tokens, as the card's tokenizer cuts.
+fn ready(srv: *Server, cx: *Cx, a: Allocator, ask: Ask, frame: *?Frame, document: []const u8, i: usize) errors.Refused![]const u32 {
+ const built = try prompt(srv, cx, a, ask, document);
+ var ids = built.ids;
+ const cap = ask.max_length;
+ if (ids.len > cap) {
+ const f = frame.* orelse try frameOf(srv, cx, ask);
+ frame.* = f;
+ // a tail this template renders otherwise for this document, a cut leaving no document token: refused
+ if (!f.fixed or !std.mem.eql(u8, built.tail, f.tail) or !std.mem.endsWith(u32, ids, f.p0[f.p0.len - f.suffix ..]))
+ return cx.fail(.request, "document {d}: the prompt has {d} tokens, past max_length {d}, and this chat template has no fixed tail after the document to cut its content before", .{ i, ids.len, cap });
+ // where this document starts: before the first token either probe's document changes
+ const start = @min(common(ids, f.p0), common(ids, f.p1));
+ if (cap <= start + f.suffix)
+ return cx.fail(.request, "document {d}: max_length {d} leaves no room for the document; the prompt without it takes {d} tokens", .{ i, cap, start + f.suffix });
+ const cut = try a.alloc(u32, cap);
+ @memcpy(cut[0 .. cap - f.suffix], ids[0 .. cap - f.suffix]);
+ @memcpy(cut[cap - f.suffix ..], ids[ids.len - f.suffix ..]);
+ ids = cut;
+ }
+ const window = srv.info.context_window;
+ if (window > 0 and ids.len > window)
+ return cx.fail(.context_length, "document {d}: the prompt has {d} tokens, which does not fit the context length of {d} tokens", .{ i, ids.len, window });
+ return ids;
+}
+
+/// A prompt's ids and the template text after its last request span (the document's).
+const Built = struct { ids: []const u32, tail: []const u8 };
+
+/// The checkpoint template over [system?, query, document] in `a`; the request's text is encoded as plain text.
+fn prompt(srv: *Server, cx: *Cx, a: Allocator, ask: Ask, document: []const u8) errors.Refused!Built {
+ var messages: std.ArrayList(Value) = .empty;
+ if (ask.instruct) |ins| try messages.append(a, try message(a, "system", ins));
+ try messages.append(a, try message(a, "query", ask.query));
+ try messages.append(a, try message(a, "document", document));
+ var problem: []const u8 = "";
+ const rendered = srv.text.render(a, .{ .array = messages.items }, .{ .add_generation_prompt = true, .enable_thinking = false }, &problem) catch |e| switch (e) {
+ error.OutOfMemory => return error.OutOfMemory,
+ error.Template => return cx.fail(.request, "the chat template failed: {s}", .{problem}),
+ };
+ const ids = rendered.encode(srv.text, a, &problem) catch |e| switch (e) {
+ error.OutOfMemory => return error.OutOfMemory,
+ error.Template => return cx.fail(.request, "the chat template failed: {s}", .{"the tokenizer cannot encode this prompt"}),
+ };
+ const spans = rendered.spans;
+ return .{ .ids = ids, .tail = if (spans.len > 0) rendered.text[spans[spans.len - 1][1]..] else rendered.text };
+}
+
+fn message(a: Allocator, role: []const u8, words: []const u8) Allocator.Error!Value {
+ const o = try json.newObject(a);
+ try o.put(a, "role", .{ .string = role });
+ try o.put(a, "content", .{ .string = words });
+ return .{ .object = o };
+}
+
+/// #1895's shape: `results` best first (equal scores in request order, cut to top_n), then `usage`.
+fn build(srv: *Server, cx: *Cx, ask: Ask, frame: *?Frame, scored: []const Scored, tokens: usize) errors.Refused!Value {
+ const order = try cx.a.alloc(usize, scored.len);
+ for (order, 0..) |*o, i| o.* = i;
+ std.sort.insertion(usize, order, scored, struct {
+ fn before(s: []const Scored, x: usize, y: usize) bool {
+ return s[x].score > s[y].score;
+ }
+ }.before);
+ const n = @min(order.len, ask.top_n orelse order.len);
+ const results = try cx.a.alloc(Value, n);
+ for (order[0..n], results) |i, *item| {
+ const o = try json.newObject(cx.a);
+ try o.put(cx.a, "index", try json.intValue(cx.a, i));
+ try o.put(cx.a, "relevance_score", .{ .float = scored[i].score });
+ if (ask.return_documents) try o.put(cx.a, "document", ask.given[i]);
+ // the returned documents' ids built again, cut as scored (the same bytes give the same ids): none kept
+ const ids = if (ask.return_ids) try ready(srv, cx, cx.a, ask, frame, ask.documents[i], i) else &.{};
+ if (try meta(cx.a, ask, scored[i], ids)) |m| try o.put(cx.a, "meta_info", m);
+ item.* = .{ .object = o };
+ }
+ const usage = try json.newObject(cx.a);
+ try usage.put(cx.a, "total_tokens", try json.intValue(cx.a, tokens));
+ const out = try json.newObject(cx.a);
+ try out.put(cx.a, "results", .{ .array = results });
+ try out.put(cx.a, "usage", .{ .object = usage });
+ return .{ .object = out };
+}
+
+/// The opt-in checks' fields (raw label logits, scored ids), or null when the request asked for neither.
+fn meta(a: Allocator, ask: Ask, s: Scored, ids: []const u32) Allocator.Error!?Value {
+ if (!ask.return_logits and !ask.return_ids) return null;
+ const o = try json.newObject(a);
+ if (ask.return_logits) {
+ try o.put(a, "logit_yes", .{ .float = s.yes });
+ try o.put(a, "logit_no", .{ .float = s.no });
+ }
+ if (ask.return_ids) {
+ const list = try a.alloc(Value, ids.len);
+ for (ids, list) |id, *v| v.* = try json.intValue(a, id);
+ try o.put(a, "prompt_token_ids", .{ .array = list });
+ }
+ return .{ .object = o };
+}
diff --git a/zig/src/server/rerank_cut_test.zig b/zig/src/server/rerank_cut_test.zig
new file mode 100644
index 0000000000..d534798de7
--- /dev/null
+++ b/zig/src/server/rerank_cut_test.zig
@@ -0,0 +1,67 @@
+//! /v1/rerank's max_length: a prompt's content cut from its end, the template's tail kept; templates it cannot cut.
+const std = @import("std");
+const json = @import("json");
+const errors = @import("errors.zig");
+const rerank = @import("rerank.zig");
+const fixture = @import("rerank_test.zig");
+const Text = fixture.Text;
+const Scorer = fixture.Scorer;
+const serve = fixture.serve;
+const ask = fixture.ask;
+const results = fixture.results;
+const table = fixture.table;
+const Present = fixture.Present;
+const im_end_id = fixture.im_end_id;
+const t = std.testing;
+
+test "rerank: past max_length a prompt's content is cut from its end, the template's tail kept; 8192 by default" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ // "S:|Q:q|D:dd|A" is 14 tokens; at 13 the document's last byte goes and "|A" stays
+ const out = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"dd\"], \"max_length\": 13, \"return_prompt_token_ids\": true}");
+ try t.expectEqual(@as(usize, 1), s.prompts.items.len);
+ try t.expectEqualStrings("S:|Q:q|D:d#END#|A", s.prompts.items[0]);
+ try t.expectEqual(@as(i64, 13), out.get("usage").?.get("total_tokens").?.int64().?);
+ // the ids the response reports are the ids scored
+ const ids = results(out)[0].get("meta_info").?.get("prompt_token_ids").?.array;
+ try t.expectEqual(@as(usize, 13), ids.len);
+ try t.expectEqual(@as(i64, 'd'), ids[9].int64().?);
+ try t.expectEqual(@as(i64, im_end_id), ids[10].int64().?);
+ // the document is echoed as sent, not as cut
+ try t.expectEqualStrings("dd", results(out)[0].get("document").?.string);
+ // with no max_length, 8192 tokens: a 9,000-byte document keeps its first 8192 - 12 bytes
+ const long = try a.alloc(u8, 9000);
+ @memset(long, 'x');
+ const body = try std.fmt.allocPrint(a, "{{\"query\": \"q\", \"documents\": [\"{s}\"], \"return_documents\": false}}", .{long});
+ const cut = try ask(srv, a, body);
+ try t.expectEqual(@as(i64, rerank.max_length_default), cut.get("usage").?.get("total_tokens").?.int64().?);
+ const scored = s.prompts.items[1];
+ try t.expectEqual(@as(usize, rerank.max_length_default + 4), scored.len); // "#END#" is one token
+ try t.expect(std.mem.endsWith(u8, scored, "x#END#|A"));
+ // a cap at the prompt's own length cuts nothing; a window refusal is a context-length error
+ _ = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"dd\"], \"max_length\": 14}");
+ try t.expectEqualStrings("S:|Q:q|D:dd#END#|A", s.prompts.items[2]);
+ srv.info.context_window = 13;
+ var cx: errors.Cx = .{ .a = a };
+ try t.expectError(error.Refused, rerank.reply(srv, &cx, (try json.parse(a, "{\"query\": \"q\", \"documents\": [\"dd\"], \"max_length\": 14}")).ok, Present{}));
+ const e = (try rerank.errorBody(&cx)).get("error").?;
+ try t.expectEqualStrings("context_length_exceeded", e.get("code").?.string);
+ try t.expectEqualStrings("documents", e.get("param").?.string);
+ try t.expectEqual(@as(usize, 3), s.prompts.items.len);
+ // a tail whose added token is not first, or whose text follows the document, is never cut
+ srv.info.context_window = 0;
+ for ([_]Text{ .{ .loose = true }, .{ .moving = true }, .{ .merging = true } }) |shape| {
+ text = shape;
+ var refused: errors.Cx = .{ .a = a };
+ try t.expectError(error.Refused, rerank.reply(srv, &refused, (try json.parse(a, "{\"query\": \"q\", \"documents\": [\"dd\"], \"max_length\": 13}")).ok, Present{}));
+ const tokens: usize = if (shape.loose) 14 else 15; // the moving tail's '+', the merging tail's 'X'
+ try t.expectEqualStrings(try std.fmt.allocPrint(a, "document 0: the prompt has {d} tokens, past max_length 13, and this chat template has no fixed tail after the document to cut its content before", .{tokens}), refused.message);
+ }
+ try t.expectEqual(@as(usize, 3), s.prompts.items.len);
+}
diff --git a/zig/src/server/rerank_ids_test.zig b/zig/src/server/rerank_ids_test.zig
new file mode 100644
index 0000000000..d64b876155
--- /dev/null
+++ b/zig/src/server/rerank_ids_test.zig
@@ -0,0 +1,121 @@
+//! The route's prompt ids with the checkpoint's own tokenizer and template against the oracle's (transformers) ids.
+
+// Skipped unless TF_QWEN3_RERANK_DIR (the checkpoint) and TF_QWEN3_RERANK_REF (the oracle's JSON) are set.
+const std = @import("std");
+const api = @import("engine_api");
+const json = @import("json");
+const errors = @import("errors.zig");
+const rerank = @import("rerank.zig");
+const HfText = @import("hf_text.zig").HfText;
+const Server = @import("server.zig").Server;
+const t = std.testing;
+
+/// Keeps the ids each score call received; its logits are zeros.
+const Echo = struct {
+ gpa: std.mem.Allocator,
+ seen: std.ArrayList([]u32) = .empty,
+
+ fn engine(e: *Echo) api.Engine {
+ return .{ .ctx = e, .vtable = &.{ .info = info, .submit = submit, .cancel = cancel, .status = status, .memory = memory, .score = score } };
+ }
+ fn info(_: *anyopaque) api.Info {
+ return .{ .text_generation = false, .decides = true, .rerank = .{ .yes = 9693, .no = 2152 } };
+ }
+ fn submit(_: *anyopaque, _: api.Id, _: *const api.Request, _: api.Sink) api.SubmitError!void {}
+ fn cancel(_: *anyopaque, _: api.Id) void {}
+ fn status(_: *anyopaque, out: *api.Status, _: []u32) void {
+ out.* = .{};
+ }
+ fn memory(_: *anyopaque, _: bool) ?api.Memory {
+ return null;
+ }
+ fn score(ctx: *anyopaque, prompt: []const u32, _: []const u32, logits: []f64, _: bool) error{Failed}!f64 {
+ const e: *Echo = @ptrCast(@alignCast(ctx));
+ const ids = e.gpa.dupe(u32, prompt) catch return error.Failed;
+ e.seen.append(e.gpa, ids) catch {
+ e.gpa.free(ids);
+ return error.Failed;
+ };
+ @memset(logits, 0);
+ return 0;
+ }
+ fn deinit(e: *Echo) void {
+ for (e.seen.items) |ids| e.gpa.free(ids);
+ e.seen.deinit(e.gpa);
+ }
+};
+
+fn env(a: std.mem.Allocator, key: []const u8) !?[]u8 {
+ return t.environ.getAlloc(a, key) catch |err| switch (err) {
+ error.EnvironmentVariableMissing => null,
+ else => err,
+ };
+}
+
+fn equalIds(got: []const u32, want: []const json.Value) bool {
+ if (got.len != want.len) return false;
+ for (got, want) |g, w| if (w.int64() != @as(i64, g)) return false;
+ return true;
+}
+
+const Present = struct {
+ pub fn check(_: @This()) bool {
+ return false;
+ }
+};
+
+test "rerank: the checkpoint's template and tokenizer give the oracle's ids, the masked case differing" {
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const dir = try env(a, "TF_QWEN3_RERANK_DIR") orelse return error.SkipZigTest;
+ const ref_path = try env(a, "TF_QWEN3_RERANK_REF") orelse return error.SkipZigTest;
+ var problem: []const u8 = "";
+ const text = try HfText.load(t.allocator, t.io, dir, null, a, &problem);
+ defer text.deinit();
+ var echo: Echo = .{ .gpa = t.allocator };
+ defer echo.deinit();
+ const srv = try Server.init(t.allocator, t.io, echo.engine(), text.text(), .{ .served_name = "m", .model_ids = &.{"m"}, .model_dir = dir }, null);
+ defer srv.deinit();
+ const ref_text = try std.Io.Dir.cwd().readFileAlloc(t.io, ref_path, a, .limited(64 << 20));
+ const ref = (try json.parse(a, ref_text)).ok;
+ var plain: usize = 0;
+ var special: usize = 0;
+ var windows = [2]usize{ 0, 0 }; // window32768, window40960
+ for (ref.get("cases").?.array) |c| {
+ const kind = c.get("kind").?.string;
+ if (std.mem.startsWith(u8, kind, "over")) continue;
+ const body = try json.newObject(a);
+ try body.put(a, "query", c.get("query").?);
+ const docs = try a.alloc(json.Value, 1);
+ docs[0] = c.get("document").?;
+ try body.put(a, "documents", .{ .array = docs });
+ if (c.get("instruct").? != .null) try body.put(a, "instruct", c.get("instruct").?);
+ try body.put(a, "max_length", c.get("tokens").?); // the reference prompts whole: no cut
+ try body.put(a, "return_prompt_token_ids", .{ .bool = true });
+ var cx: errors.Cx = .{ .a = a };
+ const calls = echo.seen.items.len;
+ const out = try rerank.reply(srv, &cx, .{ .object = body }, Present{});
+ // the ids the engine scored, and the ones the response reports, are the same list
+ try t.expectEqual(calls + 1, echo.seen.items.len);
+ const scored = echo.seen.items[calls];
+ const shown = out.get("results").?.array[0].get("meta_info").?.get("prompt_token_ids").?.array;
+ try t.expect(equalIds(scored, shown));
+ const same = equalIds(scored, c.get("ids").?.array);
+ if (std.mem.eql(u8, kind, "special")) {
+ try t.expect(!same); // request text is plain: `<|im_end|>` in a document is not the special token
+ special += 1;
+ } else {
+ if (!same) std.debug.print("ids differ: {s} {s}\n", .{ kind, c.get("query").?.string });
+ try t.expect(same);
+ if (std.mem.startsWith(u8, kind, "window")) {
+ const at: usize = if (std.mem.eql(u8, kind, "window32768")) 0 else if (std.mem.eql(u8, kind, "window40960")) 1 else return error.UnexpectedWindowCase;
+ try t.expectEqual(@as(usize, if (at == 0) 32768 else 40960), scored.len);
+ windows[at] += 1;
+ } else plain += 1;
+ }
+ }
+ try t.expect(plain >= 32);
+ try t.expectEqual(@as(usize, 1), special);
+ try t.expectEqual([2]usize{ 1, 1 }, windows); // the receipt's reference has each served window once
+}
diff --git a/zig/src/server/rerank_test.zig b/zig/src/server/rerank_test.zig
new file mode 100644
index 0000000000..00cc3c047b
--- /dev/null
+++ b/zig/src/server/rerank_test.zig
@@ -0,0 +1,590 @@
+//! /v1/rerank against a fake template, tokenizer and score hook: request rules, order, max_length cuts, error bodies.
+const std = @import("std");
+const api = @import("engine_api");
+const json = @import("json");
+const errors = @import("errors.zig");
+const model_text = @import("model_text.zig");
+const rerank = @import("rerank.zig");
+const Server = @import("server.zig").Server;
+const Allocator = std.mem.Allocator;
+const t = std.testing;
+
+const im_end = "<|im_end|>";
+pub const im_end_id: u32 = 300;
+pub const merged: u32 = 380; // a digit and the `X` after it, as one token
+
+/// Renders `S:|Q:|D:<|im_end|>|A`, contents as request spans; a digit and `X` merge, as BPE.
+pub const Text = struct {
+ /// Tails a cut must not touch: the added token last, a byte more for a longer document, an `X` the probes merge.
+ loose: bool = false,
+ moving: bool = false,
+ merging: bool = false,
+
+ fn model(self: *Text) model_text.Text {
+ return .{ .ctx = self, .vtable = &.{ .encode = encode, .decode = decode, .token_id = tokenId, .token_string = tokenString, .vocab_size = vocabSize, .eos_ids = eosIds, .render = render, .template_source = templateSource, .encode_masked = encodeMasked, .added = added } };
+ }
+
+ fn added(_: *anyopaque, id: u32) bool {
+ return id == im_end_id;
+ }
+
+ fn encode(_: *anyopaque, a: Allocator, input: []const u8, _: bool) model_text.Error![]u32 {
+ return encodeMasked(undefined, a, input, &.{}, undefined);
+ }
+
+ /// Template bytes read `<|im_end|>` as its token; request bytes stay plain.
+ fn encodeMasked(_: *anyopaque, a: Allocator, input: []const u8, spans: []const model_text.Span, _: *[]const u8) model_text.Error![]u32 {
+ var ids: std.ArrayList(u32) = .empty;
+ var i: usize = 0;
+ while (i < input.len) {
+ var plain = false;
+ for (spans) |s| plain = plain or (i >= s[0] and i < s[1]);
+ if (!plain and std.mem.startsWith(u8, input[i..], im_end)) {
+ try ids.append(a, im_end_id);
+ i += im_end.len;
+ } else if (std.ascii.isDigit(input[i]) and i + 1 < input.len and input[i + 1] == 'X') {
+ try ids.append(a, merged + input[i] - '0');
+ i += 2;
+ } else {
+ try ids.append(a, input[i]);
+ i += 1;
+ }
+ }
+ return ids.toOwnedSlice(a);
+ }
+
+ fn decode(_: *anyopaque, a: Allocator, ids: []const u32) Allocator.Error![]u8 {
+ var out: std.ArrayList(u8) = .empty;
+ for (ids) |id| if (id == im_end_id) try out.appendSlice(a, "#END#") else if (id >= merged) try out.appendSlice(a, &.{ @intCast('0' + id - merged), 'X' }) else try out.append(a, @intCast(id));
+ return out.toOwnedSlice(a);
+ }
+
+ fn tokenId(_: *anyopaque, _: []const u8) ?u32 {
+ return null;
+ }
+
+ fn tokenString(_: *anyopaque, a: Allocator, id: u32) Allocator.Error![]u8 {
+ return std.fmt.allocPrint(a, "{d}", .{id});
+ }
+
+ fn vocabSize(_: *anyopaque) u32 {
+ return 400;
+ }
+
+ fn eosIds(_: *anyopaque) []const u32 {
+ return &.{};
+ }
+
+ fn render(ctx: *anyopaque, a: Allocator, messages: json.Value, _: model_text.RenderOptions, _: *[]const u8) model_text.Error!model_text.Rendered {
+ const self: *Text = @ptrCast(@alignCast(ctx));
+ var out: std.ArrayList(u8) = .empty;
+ var last: usize = 0;
+ var spans: std.ArrayList(model_text.Span) = .empty;
+ for ([_][2][]const u8{ .{ "system", "S:" }, .{ "query", "|Q:" }, .{ "document", "|D:" } }) |part| {
+ try out.appendSlice(a, part[1]);
+ for (messages.array) |m| if (std.mem.eql(u8, m.get("role").?.string, part[0])) {
+ const c = m.get("content").?.string;
+ try spans.append(a, .{ out.items.len, out.items.len + c.len });
+ try out.appendSlice(a, c);
+ last = c.len;
+ };
+ }
+ if (self.merging) try out.append(a, 'X');
+ try out.appendSlice(a, if (self.loose) "|A" ++ im_end else im_end ++ "|A");
+ if (self.moving and last > 1) try out.append(a, '+');
+ return .{ .text = out.items, .spans = spans.items };
+ }
+
+ fn templateSource(_: *anyopaque) []const u8 {
+ return "";
+ }
+};
+
+/// A score hook whose yes/no logits come from a table keyed by the document's text; it keeps each prompt.
+pub const Scorer = struct {
+ gpa: Allocator,
+ window: u32 = 0,
+ hook: bool = true,
+ failing: bool = false,
+ healthy: bool = true,
+ table: []const struct { []const u8, f64, f64 } = &.{},
+ prompts: std.ArrayList([]u8) = .empty,
+
+ fn engine(s: *Scorer) api.Engine {
+ return .{ .ctx = s, .vtable = if (s.hook) &scored else &bare };
+ }
+
+ fn of(ctx: *anyopaque) *Scorer {
+ return @ptrCast(@alignCast(ctx));
+ }
+
+ fn info(ctx: *anyopaque) api.Info {
+ return .{ .text_generation = false, .decides = true, .context_window = of(ctx).window };
+ }
+
+ fn submit(_: *anyopaque, _: api.Id, _: *const api.Request, _: api.Sink) api.SubmitError!void {}
+ fn cancel(_: *anyopaque, _: api.Id) void {}
+ fn status(ctx: *anyopaque, out: *api.Status, _: []u32) void {
+ out.* = .{ .healthy = of(ctx).healthy };
+ }
+ fn memory(_: *anyopaque, _: bool) ?api.Memory {
+ return null;
+ }
+
+ fn score(ctx: *anyopaque, prompt: []const u32, labels: []const u32, logits: []f64, mass: bool) error{Failed}!f64 {
+ const s = of(ctx);
+ if (s.failing or mass or labels.len != 2 or labels[0] != 7 or labels[1] != 9) return error.Failed;
+ const text = Text.decode(undefined, s.gpa, prompt) catch return error.Failed;
+ s.prompts.append(s.gpa, text) catch return error.Failed;
+ const at = std.mem.indexOf(u8, text, "|D:").? + 3;
+ const doc = text[at..std.mem.lastIndexOf(u8, text, "#END#").?];
+ for (s.table) |row| if (std.mem.eql(u8, row[0], doc)) {
+ logits[0] = row[1];
+ logits[1] = row[2];
+ return 0;
+ };
+ logits[0] = 0;
+ logits[1] = 0;
+ return 0;
+ }
+
+ pub fn deinit(s: *Scorer) void {
+ for (s.prompts.items) |p| s.gpa.free(p);
+ s.prompts.deinit(s.gpa);
+ }
+
+ const bare: api.Engine.VTable = .{ .info = info, .submit = submit, .cancel = cancel, .status = status, .memory = memory };
+ const scored: api.Engine.VTable = .{ .info = info, .submit = submit, .cancel = cancel, .status = status, .memory = memory, .score = score };
+};
+
+pub const Present = struct {
+ pub fn check(_: @This()) bool {
+ return false;
+ }
+};
+
+pub fn serve(text: *Text, s: *Scorer) !*Server {
+ const srv = try Server.init(t.allocator, t.io, s.engine(), text.model(), .{ .served_name = "m", .model_ids = &.{"m"}, .enable_thinking = false, .use_drafts = false }, null);
+ srv.info.rerank = .{ .yes = 7, .no = 9 };
+ return srv;
+}
+
+fn sigmoid(x: f64) f64 {
+ return 1.0 / (1.0 + @exp(-x));
+}
+
+pub const table = .{ .{ "low", -2.0, 1.0 }, .{ "high", 3.0, -1.0 }, .{ "mid", 0.5, 0.0 }, .{ "tie", 0.5, 0.0 } };
+
+pub fn ask(srv: *Server, a: Allocator, body: []const u8) !json.Value {
+ var cx: errors.Cx = .{ .a = a };
+ return rerank.reply(srv, &cx, (try json.parse(a, body)).ok, Present{});
+}
+
+/// The response's ranked `results`.
+pub fn results(out: json.Value) []json.Value {
+ return out.get("results").?.array;
+}
+
+test "rerank: results come back best first in #1895's shape, scored sigmoid(yes - no), with usage" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const out = try ask(srv, a, "{\"model\": \"anything\", \"query\": \"q\", \"documents\": [\"low\", \"high\", \"mid\"], \"extra\": 1}");
+ try t.expectEqual(@as(usize, 3), results(out).len);
+ try t.expectEqualStrings("results", out.object.keys()[0]);
+ try t.expectEqualStrings("usage", out.object.keys()[1]);
+ // the three prompts' tokens: "S:|Q:q|D:" (9) + the document + "|A" (2) + the end token: 15 + 16 + 15
+ try t.expectEqual(@as(i64, 46), out.get("usage").?.get("total_tokens").?.int64().?);
+ const want = [_]struct { usize, f64, []const u8 }{ .{ 1, sigmoid(4), "high" }, .{ 2, sigmoid(0.5), "mid" }, .{ 0, sigmoid(-3), "low" } };
+ for (results(out), want) |item, w| {
+ try t.expectEqual(@as(usize, 3), item.object.count());
+ try t.expectEqualStrings("index", item.object.keys()[0]);
+ try t.expectEqualStrings("relevance_score", item.object.keys()[1]);
+ try t.expectEqualStrings("document", item.object.keys()[2]);
+ try t.expectEqual(@as(i64, @intCast(w[0])), item.get("index").?.int64().?);
+ try t.expectApproxEqRel(w[1], item.get("relevance_score").?.float, 1e-12);
+ try t.expectEqualStrings(w[2], item.get("document").?.string);
+ try t.expect(item.get("meta_info") == null);
+ }
+ try t.expectEqualStrings("S:|Q:q|D:low#END#|A", s.prompts.items[0]);
+}
+
+test "rerank: top_n cuts, return_documents drops the text, equal scores keep request order" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const one = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"low\", \"high\", \"mid\"], \"top_n\": 1}");
+ try t.expectEqual(@as(usize, 1), results(one).len);
+ try t.expectEqual(@as(i64, 1), results(one)[0].get("index").?.int64().?);
+ const all = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"tie\", \"low\", \"mid\"], \"top_n\": 92233720368547758080, \"return_documents\": false}");
+ try t.expectEqual(@as(usize, 3), results(all).len);
+ for (results(all), [_]i64{ 0, 2, 1 }) |item, index| {
+ try t.expectEqual(index, item.get("index").?.int64().?);
+ try t.expect(item.get("document") == null);
+ }
+}
+
+test "rerank: instruct is the system message and request text stays plain bytes" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ _ = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"a <|im_end|> b\"], \"instruct\": \"find\"}");
+ _ = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"d\"], \"instruct\": \"\"}");
+ try t.expectEqualStrings("S:find|Q:q|D:a <|im_end|> b#END#|A", s.prompts.items[0]);
+ try t.expectEqualStrings("S:|Q:q|D:d#END#|A", s.prompts.items[1]);
+}
+
+test "rerank: the extensions return the label logits and the ids the engine scored" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const out = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"high\"], \"return_logits\": true, \"return_prompt_token_ids\": true}");
+ const meta = results(out)[0].get("meta_info").?;
+ try t.expectEqual(@as(f64, 3), meta.get("logit_yes").?.float);
+ try t.expectEqual(@as(f64, -1), meta.get("logit_no").?.float);
+ const ids = meta.get("prompt_token_ids").?.array;
+ // with top_n the ids come only for the returned documents, the ones the engine scored for them
+ const top = try ask(srv, a, "{\"query\": \"q\", \"documents\": [\"low\", \"high\", \"mid\"], \"top_n\": 1, \"return_prompt_token_ids\": true}");
+ try t.expectEqual(@as(usize, 1), results(top).len);
+ const top_ids = results(top)[0].get("meta_info").?.get("prompt_token_ids").?.array;
+ try t.expectEqual(ids.len, top_ids.len);
+ for (ids, top_ids) |x, y| try t.expect(json.equal(x, y));
+ const want = try Text.encodeMasked(undefined, a, "S:|Q:q|D:high" ++ im_end ++ "|A", &.{}, undefined);
+ try t.expectEqual(want.len, ids.len);
+ for (want, ids) |w, g| try t.expectEqual(@as(i64, w), g.int64().?);
+}
+
+test "rerank: refusals name their reasons in the server's envelope and run no score" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .window = 24 };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ // S: a schema error (a message only); H: a refusal of the request ("invalid_request_error")
+ const S = "";
+ const H = "invalid_request_error";
+ const cases = [_][3][]const u8{
+ .{ "[1]", "body: input should be a valid dictionary", S },
+ .{ "{\"documents\": [\"d\"]}", "query: field required", S },
+ .{ "{\"query\": \"\", \"documents\": [\"d\"]}", "Query cannot be empty", H },
+ .{ "{\"query\": \" \\n\", \"documents\": [\"d\"]}", "Query cannot be empty or whitespace only", H },
+ .{ "{\"query\": \"\\u3000\\u00a0\", \"documents\": [\"d\"]}", "Query cannot be empty or whitespace only", H },
+ .{ "{\"query\": \"q\", \"documents\": [\"\\u001c\\u2028\"]}", "Each document cannot be empty or whitespace only", H },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": -3}", "Value error, parameter top_n should be larger than 0.", S },
+ .{ "{\"query\": [], \"documents\": [\"d\"]}", "Query cannot be empty", H },
+ .{ "{\"query\": [{\"type\": \"image_url\", \"image_url\": {\"url\": \"x\"}}], \"documents\": [\"d\"]}", "this reranker reads text: image_url parts are not supported", H },
+ .{ "{\"query\": [{\"text\": \"q\"}], \"documents\": [\"d\"]}", "query: each content part needs a type", S },
+ .{ "{\"query\": [{\"type\": \"text\", \"text\": 3}], \"documents\": [\"d\"]}", "query: a text part needs its text as a string", S },
+ .{ "{\"query\": [{\"type\": \"bogus\"}], \"documents\": [\"d\"]}", "query: bogus is not a content part type", S },
+ .{ "{\"query\": \"q\", \"documents\": [[{\"type\": \"video_url\"}]]}", "documents: a video_url part needs its video_url object", S },
+ .{ "{\"query\": [{\"type\": \"image_url\", \"image_url\": {}}], \"documents\": [\"d\"]}", "query: a image_url part needs a url string", S },
+ .{ "{\"query\": \"\", \"documents\": \"d\"}", "documents: input should be a valid list", S },
+ .{ "{\"query\": [{\"type\": \"text\", \"text\": \" \"}], \"documents\": [\"d\"]}", "Query cannot be empty or whitespace only", H },
+ .{ "{\"query\": 3, \"documents\": [\"d\"]}", "query: input should be a string, a {\"text\"} object or a list of content parts", S },
+ .{ "{\"query\": \"q\"}", "documents: field required", S },
+ .{ "{\"query\": \"q\", \"documents\": []}", "Documents cannot be empty", H },
+ .{ "{\"query\": \"q\", \"documents\": \"d\"}", "documents: input should be a valid list", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\", \"\"]}", "Each document must be a non-empty string", H },
+ .{ "{\"query\": \"q\", \"documents\": [\"\\t\"]}", "Each document cannot be empty or whitespace only", H },
+ .{ "{\"query\": \"q\", \"documents\": [[{\"type\": \"video_url\", \"video_url\": {\"url\": \"v\"}}]]}", "this reranker reads text: video_url parts are not supported", H },
+ .{ "{\"query\": \"q\", \"documents\": [1]}", "documents: input should be a string, a {\"text\"} object or a list of content parts", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": 0}", "Value error, parameter top_n should be larger than 0.", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"two\"}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": 1.5}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": 1e19}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"1__0\"}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"_1\"}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"2.5\"}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"2.\"}", "top_n: input should be a valid integer", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"top_n\": \"0\"}", "Value error, parameter top_n should be larger than 0.", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"return_documents\": \"maybe\"}", "return_documents: input should be a valid boolean", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"return_documents\": null}", "return_documents: input should be a valid boolean", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"return_logits\": 2}", "return_logits: input should be a valid boolean", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"instruct\": 2}", "instruction: input should be a valid string", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\", \"a document past the window\"]}", "document 1: the prompt has 38 tokens, which does not fit the context length of 24 tokens", H },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"max_length\": 11}", "document 0: max_length 11 leaves no room for the document; the prompt without it takes 12 tokens", H },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"max_length\": 0}", "max_length: input should be at least 1", S },
+ .{ "{\"query\": \"q\", \"documents\": [\"d\"], \"max_length\": \"8\"}", "max_length: input should be a valid integer", S },
+ };
+ for (cases) |case| {
+ var cx: errors.Cx = .{ .a = a };
+ try t.expectError(error.Refused, rerank.reply(srv, &cx, (try json.parse(a, case[0])).ok, Present{}));
+ try t.expectEqualStrings(case[1], cx.message);
+ try t.expectEqual(@as(u16, 400), cx.status());
+ const e = (try rerank.errorBody(&cx)).get("error").?;
+ try t.expectEqualStrings(case[1], e.get("message").?.string);
+ if (case[2].len == 0) try t.expect(e.get("type") == null) else try t.expectEqualStrings(case[2], e.get("type").?.string);
+ }
+ try t.expectEqual(@as(usize, 0), s.prompts.items.len);
+}
+
+test "rerank: no labels, no hook, a failed pass and a stopping engine each fail in their own class" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const body = (try json.parse(a, "{\"query\": \"q\", \"documents\": [\"d\"]}")).ok;
+ const Want = struct { []const u8, u16 };
+ const cases = [_]struct { ?api.Rerank, bool, bool, bool, Want }{
+ .{ null, true, false, true, .{ "this model is not a reranker", 400 } },
+ .{ .{ .yes = 7, .no = 9 }, false, false, true, .{ "this engine does not score labels", 400 } },
+ .{ .{ .yes = 7, .no = 9 }, true, true, true, .{ "rerank inference failed", 500 } },
+ .{ .{ .yes = 7, .no = 9 }, true, true, false, .{ "the rerank engine is unavailable", 503 } },
+ };
+ for (cases) |case| {
+ srv.info.rerank = case[0];
+ s.hook = case[1];
+ srv.engine = s.engine();
+ s.failing = case[2];
+ s.healthy = case[3];
+ var cx: errors.Cx = .{ .a = a };
+ try t.expectError(error.Refused, rerank.reply(srv, &cx, body, Present{}));
+ try t.expectEqualStrings(case[4][0], cx.message);
+ try t.expectEqual(case[4][1], cx.status());
+ const err = try rerank.encode(a, try rerank.errorBody(&cx));
+ // the server's envelope: a 500 carries its message only, a refusal its type too
+ const want = if (case[4][1] == 500)
+ try std.fmt.allocPrint(a, "{{\"error\":{{\"message\":\"{s}\"}}}}", .{case[4][0]})
+ else
+ try std.fmt.allocPrint(a, "{{\"error\":{{\"message\":\"{s}\",\"type\":\"invalid_request_error\"}}}}", .{case[4][0]});
+ try t.expectEqualStrings(want, err);
+ }
+}
+
+test "rerank: a client gone before a document stops before its pass" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const After = struct {
+ scorer: *Scorer,
+ pub fn check(g: @This()) bool {
+ return g.scorer.prompts.items.len >= 1;
+ }
+ };
+ var cx: errors.Cx = .{ .a = a };
+ const body = (try json.parse(a, "{\"query\": \"q\", \"documents\": [\"a\", \"b\", \"c\"]}")).ok;
+ try t.expectError(error.Cancelled, rerank.reply(srv, &cx, body, After{ .scorer = &s }));
+ try t.expectEqual(@as(usize, 1), s.prompts.items.len);
+}
+
+/// The bytes a dispatched request wrote back.
+const Wire = struct {
+ bytes: std.ArrayList(u8) = .empty,
+ fn write(ptr: *anyopaque, bytes: []const u8) error{Closed}!void {
+ const w: *@This() = @ptrCast(@alignCast(ptr));
+ w.bytes.appendSlice(t.allocator, bytes) catch return error.Closed;
+ }
+};
+
+/// POSTs `body` to `path` through the server's dispatcher; the reply's bytes land in `wire`.
+fn dispatch(srv: *Server, path: []const u8, body: []const u8, wire: *Wire) !void {
+ return dispatchAs(srv, "POST", path, body, wire);
+}
+
+/// `dispatch` by `method`.
+fn dispatchAs(srv: *Server, method: []const u8, path: []const u8, body: []const u8, wire: *Wire) !void {
+ const http = @import("http_conn.zig");
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ var c = try http.Conn.init(t.allocator, -1, "fixture");
+ defer c.deinit();
+ c.write_hooks = .{ .ctx = wire, .write = Wire.write };
+ c.method = method;
+ c.path = path;
+ c.version = "HTTP/1.1";
+ c.close = false;
+ c.quiet_log = true;
+ var len: [16]u8 = undefined;
+ var headers = [_]http.Header{.{ .name = "Content-Length", .value = try std.fmt.bufPrint(&len, "{d}", .{body.len}) }};
+ c.headers = &headers;
+ @memcpy(c.buf[0..body.len], body);
+ @memcpy(c.buf[body.len..][0..4], "NEXT");
+ c.end = body.len + 4;
+ srv.dispatch(&c, arena.allocator());
+ try t.expect(c.response_complete);
+ try t.expectEqualStrings("NEXT", c.buf[c.start..c.end]);
+}
+
+test "rerank: a reranker server sends every other engine route to /v1/rerank and serves the route itself" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ for ([_][]const u8{ "/v1/chat/completions", "/v1/completions", "/v1/responses", "/v1/messages", "/v1/audio/speech", "/v1/decisions", "/v1/systemone", "/v1/embeddings" }) |path| {
+ var wire: Wire = .{};
+ defer wire.bytes.deinit(t.allocator);
+ try dispatch(srv, path, "{}", &wire);
+ try t.expect(std.mem.indexOf(u8, wire.bytes.items, "400 Bad Request") != null);
+ try t.expect(std.mem.indexOf(u8, wire.bytes.items, "this model is a reranker: use POST /v1/rerank") != null);
+ }
+ var ok: Wire = .{};
+ defer ok.bytes.deinit(t.allocator);
+ try dispatch(srv, "/rerank", "{\"query\": \"q\", \"documents\": [\"low\", \"high\"]}", &ok);
+ try t.expect(std.mem.indexOf(u8, ok.bytes.items, "200 OK") != null);
+ try t.expect(std.mem.indexOf(u8, ok.bytes.items, "\"index\":1") != null);
+ var bad: Wire = .{};
+ defer bad.bytes.deinit(t.allocator);
+ try dispatch(srv, "/v1/rerank", "{not js", &bad);
+ try t.expect(std.mem.indexOf(u8, bad.bytes.items, "400 Bad Request") != null);
+ try t.expect(std.mem.indexOf(u8, bad.bytes.items, "{\"error\":{\"message\":") != null);
+ try t.expect(std.mem.indexOf(u8, bad.bytes.items, "\"object\"") == null);
+ try t.expectEqual(@as(usize, 2), s.prompts.items.len);
+ try t.expectEqual(@as(u32, 0), srv.generating.load(.seq_cst));
+ const scope = @import("auth_scope.zig");
+ for ([_][]const u8{ "/v1/rerank", "/rerank" }) |path| {
+ try t.expect(scope.allows(.{ .inference = true }, "POST", path));
+ try t.expect(!scope.allows(.{ .read = true }, "POST", path));
+ }
+}
+
+test "rerank: the engine declares a reranker; a 1_LogitScore file in the served folder alone makes none" {
+ var tmp = t.tmpDir(.{});
+ defer tmp.cleanup();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const dir = try std.fmt.allocPrint(a, ".zig-cache/tmp/{s}", .{tmp.sub_path});
+ try std.Io.Dir.cwd().createDirPath(t.io, try std.fs.path.join(a, &.{ dir, "1_LogitScore" }));
+ try tmp.dir.writeFile(t.io, .{ .sub_path = "1_LogitScore/config.json", .data = "{\"true_token_id\": 7, \"false_token_id\": 9}" });
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator };
+ defer s.deinit();
+ const srv = try Server.init(t.allocator, t.io, s.engine(), text.model(), .{ .served_name = "m", .model_ids = &.{"m"}, .model_dir = dir }, null);
+ defer srv.deinit();
+ try t.expectEqual(@as(?api.Rerank, null), srv.info.rerank);
+ const body = (try json.parse(a, "{\"query\": \"q\", \"documents\": [\"d\"]}")).ok;
+ var cx: errors.Cx = .{ .a = a };
+ try t.expectError(error.Refused, rerank.reply(srv, &cx, body, Present{}));
+ try t.expectEqualStrings("this model is not a reranker", cx.message);
+ try t.expectEqual(@as(usize, 0), s.prompts.items.len);
+}
+
+test "rerank: a NaN or an infinite logit is a failed pass, never a 200 with invalid JSON" {
+ var text: Text = .{};
+ const bad = .{ .{ "nan", std.math.nan(f64), 0.0 }, .{ "inf", std.math.inf(f64), 0.0 }, .{ "ninf", 0.0, -std.math.inf(f64) } };
+ var s: Scorer = .{ .gpa = t.allocator, .table = &bad };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ for ([_][]const u8{ "nan", "inf", "ninf" }) |doc| {
+ var cx: errors.Cx = .{ .a = a };
+ const body = try std.fmt.allocPrint(a, "{{\"query\": \"q\", \"documents\": [\"{s}\"], \"return_logits\": true}}", .{doc});
+ try t.expectError(error.Refused, rerank.reply(srv, &cx, (try json.parse(a, body)).ok, Present{}));
+ try t.expectEqualStrings("rerank inference failed", cx.message);
+ try t.expectEqual(@as(u16, 500), cx.status());
+ }
+}
+
+test "rerank: text parts join with a space, as SGLang extracts them, and the document returns as the request sent it" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const out = try ask(srv, a, "{\"query\": [{\"type\": \"text\", \"text\": \"a\"}, {\"type\": \"text\", \"text\": \"b\"}], \"documents\": [[{\"type\": \"text\", \"text\": \"high\"}], \"mid\"]}");
+ try t.expectEqualStrings("S:|Q:a b|D:high#END#|A", s.prompts.items[0]);
+ // the document comes back as the request sent it: here its content-part list
+ try t.expectEqualStrings("high", results(out)[0].get("document").?.array[0].get("text").?.string);
+ try t.expectEqual(@as(i64, 0), results(out)[0].get("index").?.int64().?);
+}
+
+test "rerank: PUT reaches the route as POST does, under the same key scope; other routes keep refusing PUT" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ for ([_][]const u8{ "/v1/rerank", "/rerank" }) |path| {
+ var wire: Wire = .{};
+ defer wire.bytes.deinit(t.allocator);
+ try dispatchAs(srv, "PUT", path, "{\"query\": \"q\", \"documents\": [\"low\", \"high\"]}", &wire);
+ try t.expect(std.mem.indexOf(u8, wire.bytes.items, "200 OK") != null);
+ try t.expect(std.mem.indexOf(u8, wire.bytes.items, "\"index\":1") != null);
+ }
+ var other: Wire = .{};
+ defer other.bytes.deinit(t.allocator);
+ try dispatchAs(srv, "PUT", "/v1/completions", "{}", &other);
+ try t.expect(std.mem.indexOf(u8, other.bytes.items, "501") != null);
+ const scope = @import("auth_scope.zig");
+ try t.expect(scope.allows(.{ .inference = true }, "PUT", "/v1/rerank"));
+ try t.expect(!scope.allows(.{ .read = true }, "PUT", "/v1/rerank"));
+ try t.expect(!scope.allows(.{ .inference = true }, "PUT", "/v1/completions"));
+}
+
+test "rerank: top_n and the flags take what pydantic's lax mode takes, as SGLang's schema does" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const docs = "\"documents\": [\"low\", \"high\", \"mid\"]";
+ for ([_][]const u8{ "\"2\"", "2.0", "\" 2 \"", "\"+2\"", "\"2.0\"", "\"\\u00a02\\u3000\"", "\"0_2\"" }) |n| {
+ const out = try ask(srv, a, try std.fmt.allocPrint(a, "{{\"query\": \"q\", {s}, \"top_n\": {s}}}", .{ docs, n }));
+ try t.expectEqual(@as(usize, 2), results(out).len);
+ }
+ for ([_][]const u8{ "\"false\"", "0", "\"no\"", "\"OFF\"", "0.0" }) |f| {
+ const out = try ask(srv, a, try std.fmt.allocPrint(a, "{{\"query\": \"q\", {s}, \"return_documents\": {s}}}", .{ docs, f }));
+ try t.expect(results(out)[0].get("document") == null);
+ }
+ for ([_][]const u8{ "\"true\"", "1", "\"Yes\"", "\"t\"" }) |f| {
+ const out = try ask(srv, a, try std.fmt.allocPrint(a, "{{\"query\": \"q\", {s}, \"return_documents\": {s}}}", .{ docs, f }));
+ try t.expect(results(out)[0].get("document") != null);
+ }
+}
+
+test "rerank: #1895's request fields: {text} documents and instruction, echoed and used" {
+ var text: Text = .{};
+ var s: Scorer = .{ .gpa = t.allocator, .table = &table };
+ defer s.deinit();
+ const srv = try serve(&text, &s);
+ defer srv.deinit();
+ var arena = std.heap.ArenaAllocator.init(t.allocator);
+ defer arena.deinit();
+ const a = arena.allocator();
+ const out = try ask(srv, a, "{\"query\": \"q\", \"instruction\": \"find\", \"documents\": [{\"text\": \"low\"}, {\"text\": \"high\"}]}");
+ try t.expectEqualStrings("S:find|Q:q|D:low#END#|A", s.prompts.items[0]);
+ try t.expectEqual(@as(i64, 1), results(out)[0].get("index").?.int64().?);
+ try t.expectEqualStrings("high", results(out)[0].get("document").?.get("text").?.string);
+}
diff --git a/zig/src/server/root.zig b/zig/src/server/root.zig
index 794cc76e0d..232948a07d 100644
--- a/zig/src/server/root.zig
+++ b/zig/src/server/root.zig
@@ -116,6 +116,9 @@ test {
_ = @import("decisions_test.zig");
_ = @import("systemone.zig");
_ = @import("systemone_test.zig");
+ _ = @import("rerank_test.zig");
+ _ = @import("rerank_ids_test.zig");
+ _ = @import("rerank_cut_test.zig");
_ = @import("slide_context_test.zig");
// both run over Unix socket pairs
if (comptime @import("builtin").os.tag != .windows) {
diff --git a/zig/src/server/routes.zig b/zig/src/server/routes.zig
index cdf1c90179..34d6779cb9 100644
--- a/zig/src/server/routes.zig
+++ b/zig/src/server/routes.zig
@@ -7,6 +7,7 @@ const json = @import("json");
const http_conn = @import("http_conn.zig");
const http_body = @import("http_body.zig");
const auth = @import("auth.zig");
+const auth_scope = @import("auth_scope.zig");
const openai = @import("openai.zig");
const responses = @import("responses.zig");
const anthropic = @import("anthropic.zig");
@@ -34,9 +35,11 @@ pub fn dispatch(srv: *Server, conn: *Conn, a: Allocator) void {
const path = auth.routePath(conn.path);
const delete_body = std.mem.eql(u8, m, "DELETE") and
(std.mem.eql(u8, path, "/v1/slide/facts") or std.mem.startsWith(u8, path, fact_path));
- if (!std.mem.eql(u8, m, "POST") and !delete_body and !consumeBody(conn, a)) return;
+ // SGLang takes its rerank route by PUT as well as POST: the same handler, admission and key scope
+ const put_rerank = std.mem.eql(u8, m, "PUT") and auth_scope.putRoute(path);
+ if (!std.mem.eql(u8, m, "POST") and !put_rerank and !delete_body and !consumeBody(conn, a)) return;
if (std.mem.eql(u8, m, "GET")) return get(srv, conn, a);
- if (std.mem.eql(u8, m, "POST")) return post(srv, conn, a);
+ if (std.mem.eql(u8, m, "POST") or put_rerank) return post(srv, conn, a);
if (std.mem.eql(u8, m, "DELETE")) {
const route = auth.routePath(conn.path);
if (std.mem.eql(u8, route, "/v1/slide/graph")) return slide.forget(srv, conn, a);
@@ -118,7 +121,8 @@ fn post(srv: *Server, conn: *Conn, a: Allocator) void {
const is_slide = std.mem.eql(u8, route, "/v1/slide/learn");
const imprint_op: ?api.ImprintOp = if (std.mem.eql(u8, route, "/v1/imprint/resume")) .where else if (std.mem.eql(u8, route, "/v1/imprint/load")) .load else null;
const is_chat = std.mem.endsWith(u8, route, "/chat/completions");
- if (!is_embedding and !is_speech and !is_slide and imprint_op == null and !is_anthropic and !is_responses and !is_decisions and !is_systemone and !is_chat and !std.mem.endsWith(u8, route, "/completions")) {
+ const is_rerank = std.mem.eql(u8, route, "/v1/rerank") or std.mem.eql(u8, route, "/rerank");
+ if (!is_rerank and !is_embedding and !is_speech and !is_slide and imprint_op == null and !is_anthropic and !is_responses and !is_decisions and !is_systemone and !is_chat and !std.mem.endsWith(u8, route, "/completions")) {
discardBody(conn, a);
return unknown(conn, a);
}
@@ -142,11 +146,18 @@ fn post(srv: *Server, conn: *Conn, a: Allocator) void {
.rejected => |refusal| return overloaded(conn, is_anthropic, refusal),
};
defer if (lease) |*held| held.release();
- if (srv.info.embeddings != null and (is_speech or is_anthropic or is_responses or is_decisions or is_systemone or is_chat or std.mem.endsWith(u8, route, "/completions"))) {
+ const is_text = is_speech or is_anthropic or is_responses or is_decisions or is_systemone or is_chat or std.mem.endsWith(u8, route, "/completions");
+ if (srv.info.embeddings != null and (is_text or is_rerank)) {
discardBody(conn, a);
conn.sendJson(400, "{\"error\":{\"message\":\"this model produces embeddings; use POST /v1/embeddings\",\"type\":\"invalid_request_error\"}}");
return;
}
+ // a reranker's template reads query and document roles only: every other route would score or write nonsense
+ if (srv.info.rerank != null and (is_text or is_embedding)) {
+ discardBody(conn, a);
+ conn.sendJson(400, "{\"error\":{\"message\":\"this model is a reranker: use POST /v1/rerank\",\"type\":\"invalid_request_error\"}}");
+ return;
+ }
// The wake for every engine-using route: a parked server reloads under the `active` count this begin raised.
if (srv.idle_manager) |m| {
if (idle_unload.wakeReload(m, srv, a)) |wake_problem| {
@@ -165,6 +176,7 @@ fn post(srv: *Server, conn: *Conn, a: Allocator) void {
if (is_embedding) return embeddings.post(srv, conn, a);
if (is_slide) return slide.learn(srv, conn, a);
if (imprint_op) |op| return imprint.post(srv, conn, a, op);
+ if (is_rerank) return @import("rerank.zig").post(srv, conn, a);
if (is_decisions) return decisions.post(srv, conn, a);
if (is_systemone) return systemone.post(srv, conn, a);
if (is_anthropic) return anthropic.post(srv, conn, a);