Schema-constrained decoding in apr (--json-schema or grammar) — three customers: arbiter's quorum lane, tool calling in verb set V, and the ruchy language

Row 8 of the P0 dogfood chain in 30-day-plan.md v2.2 §5.0 (operator ruling 2026-09-20 09:52Z: arbiter runs on apr Qwen 3.5 — dogfood is P0). The only row of that chain that is aprender's; rows 1–7 are infra.

What

apr constrains generation to a caller-supplied schema or grammar — --json-schema <file> or an equivalent grammar flag — so a structured answer is produced correctly rather than repaired afterwards.

Why it is P0 rather than a nicety

Three customers need the same mechanism, and today all three work around its absence:

arbiter's apr quorum lane. arbiter decide now runs one agy, one Claude, one apr; the apr lane must emit a parseable verdict. Row 4 of §5.0 measures parse rate with lenient deterministic extraction (strip fences, take the first JSON object) — an explicit stopgap. Schema-constrained decoding is the plan's own words for "the fix at the source".
Tool calling, which is in verb set V for goal B — Hermes-style <tool_call> is a structured-output problem wearing a different hat.
The ruchy-based language (stack-30-day-plan.md), whose implementation is out of scope here (label:lang-next is a standing exclusion) but whose decoding requirement is not.

Building it once, at the source, serves all three. Repairing output per consumer serves none of them twice.

Scheduling — priority is not must-carry

Per doctrine rule 3, a row is must-carry for a train only if its plan was ruled and its falsifier proven satisfiable by a RED test on main before the train opened. Row 8 was not, so it is not 0.69 must-carry; it ships in the first train where it is green. It carries the P0 label because it unblocks the P0 dogfood chain, not because it holds a date.

Acceptance

A generation constrained by a supplied JSON schema validates against that schema for every sample in a fixture corpus, on CPU and CUDA, on both GPU host classes.
Positive control: a deliberately malformed schema, or a model prompted to violate it, turns the test RED. A constraint that cannot fail is decoration.
The parse-rate measurement in §5.0 row 4 is re-run against constrained decoding and the lenient-extraction stopgap is removed, not left as a parallel path — two paths mean the weaker one decides on a bad day.
Refusal path: an unsupported schema feature refuses in one line with an exit code and a removed_by, per the shipping doctrine.

Known collisions, which are the point

§5.0 expects rows 1–3 to collide with open 0.69 defects — #3546 (install.sh 404s, so the declared-resource install path does not exist) and #3545 (the -cuda asset reports NotCompiled from apr devices, so an availability probe reading devices is lied to). Dogfooding is what surfaces these; that is the argument for doing it at m=1 now rather than after the fixes.

Refs #3080, #3545, #3546

Blocked by #3571. This row is specified on apr run and apr serve, and apr serve cannot load Qwen3.5 at all — measured today: Architecture 'qwen35' … runs through Qwen35Model, on a file apr run handles perfectly. Schema-constrained decoding on a verb that refuses the model is untestable, so #3571 is a hard dependency for half of this row's acceptance.

The apr run half is not blocked and can proceed: constrained generation validating against a supplied schema, with a malformed-schema positive control. Sequence it that way rather than waiting for #3571 — but the row is not done until both verbs are covered, because the decide lane is an HTTP client of apr serve by design (a run-per-call lane pays model load on every decision, and a 9B inside arbiter's 5% cgroup never finishes).

Correction to my note above. The decide lane invokes apr run, not apr serve (read from the committed code by the assets lane), so this row is not blocked by #3571 for the quorum-lane consumer. The apr serve half of its acceptance still needs #3571, and goal A's floor still names apr serve — but the sequencing is the reverse of what I wrote: the apr run half is what the live consumer needs now, and the serve half follows #3571.

Design input, from the decide-lane review (2026-09-20)

The flag's shape is now specified rather than left to the implementer:

Takes the schema inline OR @path — both. The lesson is measured elsewhere on this fleet: claude --json-schema takes the schema, not a path to one; a caller passed a path, the process died on its arguments, and from outside that was indistinguishable from a lane with nothing to say — an empty stdout sent the investigator looking at the model. Accepting both forms removes the failure mode instead of documenting it.
Validate the schema before the first token, and refuse SchemaInvalid by name with an exit code. A schema rejected at token 400 is a wasted decision; a schema rejected silently is the above.
Grammar takes the same treatment — inline or @path, validated up front, refused by name.
Do not overload --format json. It means output formatting today and must keep meaning that. Measured while checking this: nothing in the tree composes a --json-schema flag at all, and all 29 json_schema hits are Arrow inference, contract formulas and registry schema — so the only near-miss a future reader will find is --format json, which is exactly why it must not absorb this meaning.

Positive control unchanged and now more specific: a deliberately malformed schema must produce SchemaInvalid and a non-zero exit before generation starts, and a valid schema the model is prompted to violate must still yield conforming output. A constraint that cannot fail either way is decoration.

Moved to 0.69.1: strict JSON-schema output is a measured requirement of two consumers on #3716. RAH: a grammar-constrained lane would have turned today's finding/findings NULL into a vote. rmedia: strict JSON-schema, no streaming. The operator's bar for 0.69.1 includes RAH, arbiter and rmedia usage. — cop

Design note: schema-constrained decoding (aprender-0e, before any code)

Written for the cop's 0.69.1 queue. It touches the sampling loop on every backend. Everything below was read from main @ 225b2a9ab, or measured on this box's RTX 4090, unless it is marked to verify.

Recommendation

Masking, not repair. Put one hook at the sampling boundary: mask(logits), then the existing sampler, then accept(token). The sampling algorithms stay as they are. Greedy / temperature 0 stays deterministic, because the mask only removes candidates.
Engine: llguidance 1.8.0 (MIT, Rust, rust-version 1.87), behind an in-tree TokenConstraint trait. The trait and the hook are ours, so the engine can be swapped. This adds a crates.io dependency, so it needs a ruling. The fallback, if that is refused, is in "Engine choice" below.
0.69.1 scope is apr run only. It covers dense GGUF on CPU and CUDA, and the Qwen3.5 hybrid on CPU and GPU. Every other generation path refuses the flag by name until it is wired. A constraint that is silently ignored is decoration. apr serve follows apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571 as response_format: {type: json_schema}.

What already exists, and why it is not enough as-is

crates/aprender-serve/src/grammar/ has a GBNF-like Grammar, a GrammarStateMachine, JsonSchemaType plus grammar_from_json_schema, and GrammarTokenMasker/TokenMask. None of it is wired into any generation loop. Nothing outside the module calls GrammarTokenMasker, GrammarStateMachine or grammar_from_json_schema. Serve uses only the post-hoc ToolCallParser. Four things keep it from being the engine as it stands:

It works on chars, not bytes. token_strings: HashMap<u32, String>. Qwen's byte-level BPE has tokens that are partial UTF-8 sequences, and a char model cannot represent them. That matters inside any JSON string with non-ASCII content.
Mask cost. get_mask() walks the whole vocabulary on every step and clones the state machine once per multi-char token (is_token_valid_sequence). The vocabulary is 248,320 tokens for Qwen3.5 and 151,936 for Qwen3, both measured from the GGUF tokenizer.ggml.tokens. That is up to one clone per vocabulary token whose first char is valid, on every generated token.
Unbounded whitespace. ws ::= "" | " " ws | "\n" ws | "\t" ws is the known runaway: a constrained model can emit whitespace until max_tokens.
No schema-document parser. Nothing turns a JSON Schema document into a JsonSchemaType, and nothing refuses unsupported keywords by name. EOS is also never forced when the grammar completes.

Engine choice

Recommended: llguidance. It does byte-level token-trie masking (its authors report microseconds per token on large vocabularies, to verify here), supports JSON Schema (draft 2020-12 subset) and Lark grammars, and fails a schema it cannot enforce. The supported-keyword list and error shape are to verify in PR 1's spike against the two real consumer schemas: RAH's agy/quorum-lane-schema.json and rmedia's keyterm/quiz/cue schemas, which rmedia needs to attach on 0.69.1 CONSUMER REQUIREMENTS: every project that uses apr states its user-interface needs (verbs/API, models, context, output shape, latency) — the release gate is derived from these answers #3716.
Fallback: extend the in-tree module. Three changes would be needed: a byte-level pushdown automaton, one vocabulary trie walked with the automaton (prefixes shared rather than a clone per token), and masks cached by automaton state. The subset would be sized to the consumers: object/properties/required/items/string-enum/scalars, plus annotations ignored. RAH's lane schema uses exactly type, properties, required, items, enum plus $schema/title/description, all inside today's JsonSchemaType. This is about 2–3× the work, and it carries the byte/UTF-8 and cost problems we would be fixing ourselves.

The hook, and the loops it touches

Sampling happens in about 25 files under crates/aprender-serve/src. The plan does not wire each of them.

One trait, carried on the generate config as Option<Box<dyn TokenConstraint + Send>>: mask(&mut [f32]) -> Result<()>, accept(u32) -> Result<()> and is_complete() -> bool. On completion only EOS is allowed. An empty mask (a dead end) is an error, ConstraintDeadEnd, and generation never continues unconstrained.
Dense GGUF, CPU: apply it in the per-step sampler of generate_with_cache.
Dense GGUF, CUDA: generate_gpu_resident takes a fused on-device argmax (forward_gpu_resident_to_token_id), so no logits reach the host. PMAT-814 already switches to the host-logits path when a repetition penalty is active, and the constraint takes the same switch: constrained || penalty_active. The added cost is one logits download per token, about 1 MB at 248k × f32, plus the mask. It will be measured in PR 3, with the tok/s delta reported beside RAH's measured decode of 1.0–2.9 tok/s.
Qwen3.5 hybrid: apply it in run_qwen35_generate on CPU and run_qwen35_generate_gpu on GPU. The GPU dispatch falls back to CPU on any GPU failure. Under apr qa golden_output: a GPU golden-generation ERROR is reported as a SKIP (note_gpu_golden_skip) — 'GPU broken' passes the gate; the Q4_K CUDA rule can't be proven while it does #3711 (in the 0.69.1 fold) a fallback where the GPU should serve is a FAIL, so a constrained CUDA run cannot silently become a CPU run.
Refused by name until wired: the APR/SafeTensors transformer, qwen3_moe (qwen3_moe_generate), the batch scheduler and serve. The refusal is SchemaUnsupportedPath{path}, with an exit code and a removed_by.

Qwen3 thinking vs no-think

apr picks the template by name: detect_format_from_name maps anything containing qwen3, including qwen35 and Qwen3.5-9B, to Qwen3NoThink. That template prefills an empty <think> block (FALSIFY-CT-006). So under apr's default the constraint starts at the first generated token. Measured: apr run Qwen3.5-0.8B-Q4_K_M.gguf --chat --temperature 0 --max-tokens 8 --prompt "What is 2+2?" answers 2+2=4, so no reasoning block was generated. That is RAH's configuration: apr run … --chat --temperature 0.
0.69.1: --json-schema together with a thinking template is refused by name (SchemaWithThinking). The alternative is a two-phase constraint: unconstrained until the </think> special token, then constrained. It is correct but adds a phase switch and an unbounded reasoning budget in front of the grammar, and no current consumer needs it.
apr does not rewrite the prompt with the schema. Telling the model what shape to produce is the caller's job. A gate that silently edits the prompt is a surprise, and the positive control below has to be able to fail.

Flag, refusals and output contract

This follows the 2026-09-20 design input on this issue.

apr run --json-schema <inline JSON | @path> and --grammar <inline | @path> (Lark). Both are validated before the first token. --format json is not overloaded.
The refusals, each one line on stderr with a non-zero exit code:
- SchemaInvalid: malformed JSON, or not a schema.
- SchemaUnsupported{keyword, path}, with a removed_by.
- SchemaUnsupportedPath{path}.
- SchemaWithThinking.
- SchemaViolation: the output fails an independent validator.
- Truncated: max_tokens was reached before the grammar completed.
  Per rmedia, a cut or empty completion is an error, never an empty success.
A second reader. The final text is validated against the schema by a validator independent of the engine, for example the jsonschema crate, before exit 0. Masking is one reader; this is the other.
apr run --json gains finish_reason (stop | length | constraint_complete | dead_end) and prompt_tokens. RAH asks for both on 0.69.1 CONSUMER REQUIREMENTS: every project that uses apr states its user-interface needs (verbs/API, models, context, output shape, latency) — the release gate is derived from these answers #3716 so that a cut answer can be told apart from a malformed one.

Acceptance, mapped to this issue

The fixture corpus is the real consumer schemas. The RAH lane schema is the first; rmedia's are needed on 0.69.1 CONSUMER REQUIREMENTS: every project that uses apr states its user-interface needs (verbs/API, models, context, output shape, latency) — the release gate is derived from these answers #3716. The prompts include ones that ask the model to violate the schema. Each schema runs on CPU and CUDA, on lambda (sm_89) and gx10 (GB10). Every sample validates with the second reader.
Positive controls:
- A malformed schema gives SchemaInvalid and a non-zero exit before generation, with no tokens emitted.
- A valid schema with a violating prompt still gives conforming output.
- A mutant that makes the constraint a no-op turns the corpus test RED.
After that, the lenient-extraction stopgap in §5.0 row 4 (infra) is removed rather than left in parallel.

PR split (each one quorum-reviewed)

The TokenConstraint trait, the llguidance adapter, and the vocabulary byte table built once per model load from the GGUF tokenizer. Unit tests on a synthetic vocabulary. No model and no loop changes.
apr run on the CPU paths (dense and Qwen3.5), with the flags, the refusals, the second reader and the --json fields.
The CUDA paths via the host-logits switch, with overhead measured on lambda and gx10.
apr serve response_format, after apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571.

Needs a ruling

The dependency. llguidance is recommended, with the in-tree extension as the fallback. It adds a crates.io dependency to the sampling path.
SchemaWithThinking refused in 0.69.1. This is recommended; the two-phase constraint would come later if a consumer needs it.

Refs #3716, #3571, #3711

triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.72.0 — 0.69.x leftover, not done at the tag: consumer / agent surface — 0.72 Agent Ready. It was 0.69.1 (released). Open PR / live branch: br:PMAT-3568-token-constraint.

merged into #5305 (master ticket for epic #3598; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

Reopened: when master #5305 was split by theme, this feature (schema-constrained decoding) shared no theme with the speed rows, so it stays open on its own under epic #3598. Ignore the earlier "merged into #5305".

Evidence: fixture corpus on GB10 (gx10, sm_121), CPU and CUDA. Not a close.

Binary: apr 0.71.3 (01550af01), sha256 1604dee7c6f1…dbd5e0
Models: qwen2.5-coder-1.5b-instruct Q4_K_M (cc324af0…), Qwen3.5-4B Q4_K_M (00fe7986…)
Corpus: 10 schemas (boolean, enum, int range, number, string array, person, nested, status, int array, enum-tag array). Each run is apr run --json-schema @schema --format json --max-tokens 128, with --no-gpu and with --gpu.

Result: 36/40 rc 0. Every --gpu row reports used_gpu: true and every --no-gpu row reports used_gpu: false.

The 4 failures are all schema 10, on both models and both devices. Schema 10 is {"tags": array of enum a|b|c, no maxItems, "done": boolean}. Every run repeats "c" until the token budget runs out; it does the same at --max-tokens 1024. The run exits 15 with Truncated: … a cut document is an error, never a success, so the error path behaves correctly. Open question: an unbounded enum array lets greedy decoding loop without ever closing the array. The cause is not yet established.

The RTX 4090 half of this done-when row is still pending.

Follow-up on schema 10 (GB10, same binary 01550af01, coder-1.5b, --gpu): when the prompt names the content (Return JSON with tags ["a","b"] and done true.), the run closes the array and the object in 11 tokens. Output is {"tags":["a","b"],"done":true}, rc 0. So the grammar does allow ] after an item. The loop seen with the content-free prompt is the model repeating itself under greedy decoding, not a decoding-constraint bug. The truncation error is the correct outcome there.

Evidence: the same fixture corpus on RTX 4090 (sm_89), CPU and CUDA. This is the second GPU host class; GB10 is above. Not a close.

Binary: released apr 0.71.4 (5027da624), the v0.71.4 x86_64-unknown-linux-gnu-cuda asset (tarball sha256 checked against its .sha256). Binary sha256 2b0aeea0751f…
Models: the same two files as on GB10, by sha256: qwen2.5-coder-1.5b-instruct Q4_K_M (cc324af0…) and Qwen3.5-4B Q4_K_M (00fe7986…)
Corpus: 10 schemas × 2 models × {CPU, CUDA} = 40 runs of apr run --json-schema @file --format json. Exit 0 means the independent jsonschema reader accepted the document.

Result: 36/40, row for row the same as GB10. All 20 CUDA rows report used_gpu: true and all 20 CPU rows report false.
The 4 non-zero rows are schema 10-tags (an unbounded enum array) on both models and both devices. Each exits 15, the truncation refusal, rather than emitting invalid JSON.

As on GB10, schema 10 closes when the prompt names the content. On CUDA with coder-1.5b, Return JSON with tags ["a","b"] and done true. gives rc 0 in 11 tokens with used_gpu: true. So the 4 rows come from the model repeating itself under greedy decoding with a content-free prompt. They do not come from a grammar that cannot close.

Acceptance status: the corpus has now run on both GPU host classes, CPU and CUDA. Outside this repository, the parse-rate re-run and the stopgap removal are still open.

Navigation Menu

Repository navigation

Schema-constrained decoding in apr (--json-schema or grammar) — three customers: arbiter's quorum lane, tool calling in verb set V, and the ruchy language #3568

What

Why it is P0 rather than a nicety

Scheduling — priority is not must-carry

Acceptance

Known collisions, which are the point

Design input, from the decide-lane review (2026-09-20)

Design note: schema-constrained decoding (aprender-0e, before any code)

Recommendation

What already exists, and why it is not enough as-is

Engine choice

The hook, and the loops it touches

Qwen3 thinking vs no-think

Flag, refusals and output contract

Acceptance, mapped to this issue

PR split (each one quorum-reviewed)

Needs a ruling

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Schema-constrained decoding in apr (--json-schema or grammar) — three customers: arbiter's quorum lane, tool calling in verb set V, and the ruchy language #3568

Description

What

Why it is P0 rather than a nicety

Scheduling — priority is not must-carry

Acceptance

Known collisions, which are the point

Activity

noahgift commented on Sep 20, 2026

noahgift commented on Sep 20, 2026

noahgift commented on Sep 20, 2026

Design input, from the decide-lane review (2026-09-20)

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

Design note: schema-constrained decoding (aprender-0e, before any code)

Recommendation

What already exists, and why it is not enough as-is

Engine choice

The hook, and the loops it touches

Qwen3 thinking vs no-think

Flag, refusals and output contract

Acceptance, mapped to this issue

PR split (each one quorum-reviewed)

Needs a ruling

noahgift commented on Sep 24, 2026

22 remaining items

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions