Repository navigation
Proposal: Clef decision models on the native engine (/v1/systemone and joint-schema scoring for /v1/decisions) #692
Description
The native server keeps /v1/decisions and scores through an engine hook (zig/src/server/decisions.zig), but no family sets one yet. In #241, @outcastofmusic served Cloudflare's Clef-Flash on the Python CUDA engine through its joint schema head, matching the release code's top option on 128 of 128 questions (largest probability difference 0.027); #251 reported 128 of 128 for Clef 27B from a 4.0 bpw EXL3 pack (0.142). The closing comment on #241 said that their prompt encoding and spans, the head's math and the 45-record agreement set would carry over as the route's first scorer once Clef's Qwen backbones are on the Zig engine. I'd like to contribute that port, keeping the credit for #241 and #251 and coordinating with their author.
Clef's backbone is Qwen3.8-27B, and its config.json matches the geometry the qwen27 Metal engine qualifies (hidden 5120, 64 layers, the same attention and linear-attention shapes and layer schedule). The Metal path takes 4-bit affine weights with group size 64, so Clef 27B needs a backbone pack in that format; the joint schema head stays a separate bf16 file. Clef-Flash (Qwen3.5-9B) parses, but the serving loader refuses its geometry today, so it would come later as its own qualification.
Why now: oMLX's 0.7.1.dev1 development release serves Clef and OpenJev through POST /v1/systemone (TypeSafe's System One request format), and llama.cpp has merged the endpoint and Clef support. The intended use is agent routing and review gates.
Proposed slices, each its own PR:
- Checkpoint. qwen27 reads a Clef 27B pack converted from the release (
model.language_model.*,lm_head;model.visual.*skipped) in the qualified 4-bit/group-64 format, and recognizes a decision model byjoint_head.safetensorswithjoint_head_config.json. A decision model starts with drafts off, and the chat routes answer 400 naming the decision route. - Hidden rows. A qwen27 pass that fills a disposable state with the family's own prompt kernels and returns every row's final normed state, with the same kernels and bits as a chat prompt (no separate decision kernels, Serve Clef decision models through their joint schema head on CUDA (/v1/decisions, /v1/systemone) #241's open question 2). Checked so that decision requests, cancellations and failures leave chat replies and resumed conversations unchanged.
- The joint schema head in Zig. The release's
joint_schema_model.py(Apache-2.0): prompt layout, spans and option order, and the head: LayerNorms and projections, two evidence-routing layers, four decoder layers with self-attention over the fields and cross-attention to the prompt rows, and the prior plus gated joint scorer. The release runs the head in bf16; I'd keep its bf16 weights and accumulate in f32, and say so in the receipts. Lexical option vectors average the LM output rows over each option span, read from the quantized head as Clef decision models from quantized packs: LM head rows from the head's own matmul (stacked on #241) #251 did. - Routes. A joint-schema scoring interface that receives all questions and spans of a request together (the current hook scores one question's vocabulary labels), with
/v1/decisionsadapted to it: the label scorer's behavior unchanged, and for head-scored replieslabel_massandprompt_format_versionas null, as in Serve Clef decision models through their joint schema head on CUDA (/v1/decisions, /v1/systemone) #241. If wanted,POST /v1/systemonetakesmodel,stateand named questions of typenoul,choiceandscorewith Clef's answer semantics, plus oMLX-compatible checks: 422 for a malformed body, 400 for invalid criteria, 413 whentruncate: falsekeeps the prompt from fitting.
Separate follow-ups: label scoring for the other families (#127's arithmetic; #572 has the CUDA Nemotron logprobs), Clef-Flash's 9B qualification, OpenJev (calibrated option-letter logits; its weights are CC BY-NC 4.0), Qwen vision on the shared image path of #598, and CUDA on top of #608.
Gate I'd propose:
- On Serve Clef decision models through their joint schema head on CUDA (/v1/decisions, /v1/systemone) #241's 45 records (128 questions, 230 to 14,149 prompt tokens), per checkpoint format: token ids and spans identical to the release code, top-option agreement and the largest absolute probability difference against the release in bf16, with the acceptance tolerance agreed before implementing the head.
- Concurrent decision requests each equal their solo result. On Qwen3.8-27B, the chat checkpoint, the hidden-rows pass leaves chat replies unchanged token for token; on a Clef checkpoint, the chat routes answer 400 naming the decision route.
- Latency and peak memory per prompt length on an M3 Ultra and an M5 Max.
- Zig 0.17.0:
zig build test,test-golden, the lean check,zig fmt --check build.zig zig, and the qwen27 model gates the change touches.
Questions:
- Both routes, or
/v1/decisionsonly? (Serve Clef decision models through their joint schema head on CUDA (/v1/decisions, /v1/systemone) #241's open question 1.)/v1/systemonelets clients use the System One request format, within TensorFold's documented model and input limits. - Should the head live under
zig/src/families/qwen27/or in a shared place, so a CUDA qwen27 (Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608) reuses it? - Is a 4-bit/group-64 backbone pack with the bf16 joint schema head fine as the recommended Clef checkpoint, with its agreement numbers published?
Validation targets: M3 Ultra 256 GB and M5 Max 128 GB, with GB10 in the CUDA follow-up.
Activity
ashhart commentedon Oct 14, 2026
Hello @cwschroeder,
Thank you for the slice plan, and for keeping the credit for #241 and #251.
On main, POST /v1/decisions scores through the engine hook (zig/src/server/decisions.zig:362). An engine with no score function answers "decision scoring is not supported by this engine yet" (decisions.zig:363, zig/src/core/engine_api.zig:369). No family sets that hook. There is no POST /v1/systemone, and no joint_head reader. Qwen3.8-27B on Metal is the 4-bit group-64 checkpoint (README.md:75). Clef is not a model_type the native engine opens.
Both routes, where the head lives, and a 4-bit backbone plus a bf16 head, are on the maintainer's list. The 45-record set, with token ids and spans checked before any probability tolerance is agreed, is the gate that helps a first PR. Hidden rows that leave a chat reply unchanged belong in that same PR.
Thank you again.
ashhart commentedon Oct 15, 2026
Thank you, @cwschroeder, for separating the 4 bit backbone from the bf16 joint head and preserving the credit for #241 and #251.
We'll implement Clef's head as a separate Zig module behind the existing decision routes. Please start with a PR for the checkpoint reader and disposable hidden rows, carrying the 45 record set with exact token ids, spans and option order. We'll check that layout first, including that decision requests leave chat and resumed conversations unchanged, then agree the numerical gate for the head. The backbone and head stay separate formats as proposed.
Posted by Ash's AI Assistant Synthena
cwschroeder commentedon Oct 16, 2026
@ashhart I am taking the checkpoint-reader and disposable-hidden-rows slice on current main
c7137bb3, with the head kept separate from the backbone. The earlier local prototype has only 15 encoding records (35 questions), so it is not the 45-record gate and I will not present it as that receipt.@outcastofmusic could you share the original 45 records from #241/#251, or point me to their generator? I found the unit tests in #241, but not that full agreement corpus. I need its exact token ids, spans and option order for the first Zig PR, with your work credited.
Meanwhile I am validating the standalone bf16 head reader and mapping a disposable qwen27 prompt pass onto the existing prompt kernels. Chat/resume preservation, including cancellation and failure, remains a required test before the slice is ready. I have not set a new probability tolerance.
cwschroeder commentedon Oct 16, 2026
Current Clef reader/disposable-row WIP:
812fc83e, rebased onto upstream46a08c18. The fork-only draft CI receipt now targets that base; all six checks for812fc83epass. The Linux job also executed the four inspector CLI tests successfully.The optional-head reader accepts valid cache symlinks and refuses incomplete or damaged pairs. A CPU inspector checks head geometry against the backbone config and emits hashes without opening backbone weights or a GPU. Its receipt keeps
model_ready=false.Host.hiddenRowsruns capture on the owner thread. The rebase preserves the new drain/Imprint paths and refuses captures after a failed learning put-back until a healing lesson completes. Pending calls are rejected before drain joins an active call. ANE handoff cleanup and the private frame/runner remain in the branch.Local Zig 0.17.0 validation passes: 34 targeted CPU tests, four inspector CLI tests, whole host build 206/206 steps, kernel contract, goldens/fuzz, formatting and zero new lean findings. Mutants bypassing stale refusal or accepting dangling head links fail their tests. Astra reviewed the rebase and changed Qwen frame paths with no material source blocker.
The two synthetic device harnesses compile but have not run. The original 45-record layout corpus, GPU/ANE, real-model chat/resume, concurrent isolation, performance and numerical head gates remain open. Capture takes text token IDs; vision-chat isolation needs its own device test. HTTP scoring is not connected. No upstream first-slice PR yet; #241/#251 credit is retained.
ashhart commentedon Oct 17, 2026
Thank you, @cwschroeder, the pinned reader/disposable-row WIP keeps the optional bf16 head separate and reports model_ready=false.
We checked the supplied follow-up against the release scope. The host inspector and refusal controls are useful, while HTTP scoring, real hidden-row/device behavior and chat/vision isolation remain the next gates. We have not inspected the external fork's implementation here.
Please open the first-slice PR with the original 45-record token/span/option corpus or its generator when it is available, keeping #241/#251 credit. We will review that layout before agreeing the head's numerical tolerance. The 15-record prototype and passing host checks remain separate from the full layout and model gate.
Posted by Ash's AI Assistant Synthena
cwschroeder commentedon Oct 17, 2026
The fork-only Clef WIP is now at
d288181f, based on current main6ee828c2.Astra found a missing memory-admission check in the owner-thread capture path.
4c685d07checks private KV, owned output rows and a conservative fixed-buffer bound before allocation, preserving the server's 8 GiB reserve without evicting chat snapshots. A CPU regression refuses a 196,608-row plan with only 8 GiB available; removing the refusal makes it fail. This is a buffer-budget check, not a measured peak-memory claim.The branch also merges #928's three commits unchanged, with chaog992's authorship. Without that fix, the owner-call lone-driver yield test reaches the missing-inflight unwrap. With it, all 177 engine-API/LaneHost tests pass in Debug. The 36 checkpoint/admission tests pass in Debug and Fast; four CLI tests and all six current-head CI checks pass. The native server and both device harnesses compile only. Astra found no further material source issue in this bounded integration.
The inspector also validates the actual Clef-Flash release head at
8b2e5fd1: 122 tensors, hidden width 4096, withmodel_ready=false. The original 45-record/128-question corpus, device/model/chat/vision isolation and agreed numerical tolerance remain open. HTTP scoring is still disconnected; #241/#251 credit is retained. No upstream first-slice PR or deployment is claimed.ashhart commentedon Oct 17, 2026
Thanks @cwschroeder. The admission check, and the inspector passing on the actual Clef-Flash head, are what we needed. Please open the first slice as a PR now: the inspector, the refusal controls and Astra's admission check, with model_ready=false. Leave #928's commits out of it, since #928 lands on its own. We'll agree the head's numerical tolerance in that PR, against the original 45-record corpus once you have it. HTTP scoring and the device gates come in the next slices, and the #241/#251 credit stays.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 26, 2026
Closing this proposal issue for now. If you'd like to bring the first Clef slice across (the inspector, refusal controls and admission check), open it as a PR and we'll review it with you. Thanks for the detailed write-up.
- Synthema, Ash's AI Assistant.