apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve

P0 on the dogfood critical path. apr serve cannot load a Qwen3.5 model at all — the architecture the last release shipped, and the path goal A's floor names.

Measured 2026-09-20, published 0.68.2 built from crates.io with `--features cuda`

$ apr serve run ~/models/Qwen3.5-0.8B-Q4_K_M.gguf --gpu --port 18742
=== APR Serve ===
Detected format: GGUF
GGUF loaded: 320 tensors, 46 metadata entries
Building quantized inference model...
error: Model load failed: Failed to build quantized model: Format error:
  Architecture 'qwen35' is the Qwen3.5 hybrid (Gated Delta Net, detected tensor 'blk.0.ssm_a'):
  it runs through `Qwen35Model` (re…

The file loads and the architecture is correctly detected; the serve path does not route to Qwen35Model. apr run does — 20/20 cells correct across 0.8B/2B/4B/9B/27B on two GPU hosts, CPU and CUDA. So this is a routing gap in one verb, not a model defect.

Why this is P0 rather than a missing feature

Three things on the critical path assume apr serve works for this model:

Goal A's floor is "≥3 internal consumers on apr serve for 7 consecutive days". Today that floor is unreachable for Qwen3.5 by construction.
The recommended P0 architecture — run the apr engine as a long-lived apr serve unit declared in forjar with its own resource budget, so the model loads once and the lane is an HTTP client. That design is right (a run-per-call lane pays model load on every decision, and arbiter's 5% cgroup would never finish a 9B) and it is not buildable today.
Verb set V (Name the Qwen model 0.69 blesses and commit its parity receipt at the SERVING shape — our gates are all single-stream #3555) includes apr serve single-stream.

Why nothing caught it

#3555 recorded that every gate we own is single-stream apr run — grep -nE 'batch|serve' scripts/check_model_parity.sh and check_model_ladder.sh both return nothing. The model-capability ladder proves apr run; no rung, no parity record and no dogfood row has ever asked apr serve to load a model. The one verb our own consumers are told to use is the one verb nothing exercises.

Acceptance

apr serve run <qwen3.5 gguf> --gpu starts, reports the real CUDA backend, and answers a /v1/chat/completions request correctly, on both GPU host classes, for every size the ladder covers.
The same on --no-gpu.
A ladder rung or parity row that exercises apr serve, not only apr run — otherwise this regresses silently the next time. That is the durable half of the fix.
Positive control: point the rung at a model the serve path genuinely cannot support and require RED.

Consequences to record now

ARB-APR-001 row 2 should declare an apr serve unit, not just the binary — and it cannot be satisfied until this lands.
Until then the apr decide lane has no correct shape: apr run per call pays the model load every decision (~4.5 min of prefill alone at 30k tokens on gx10 at ~9 ms/token), and apr serve refuses the model.
This is a blocking dependency of Schema-constrained decoding in apr (--json-schema or grammar) — three customers: arbiter's quorum lane, tool calling in verb set V, and the ruchy language #3568 (schema-constrained decoding), which is specified on apr run and apr serve.

Refs #3080, #3555, #3558, #3568

Blast radius corrected — one consumer smaller, and two diagnostic leads

Correction (assets lane, from the committed code rather than from my inference): the decide lane invokes apr run, not apr serve:

Engine::Apr => vec![
    lane.runner.display().to_string(),
    "run".to_string(),
    apr_model().display().to_string(),
    prompt.to_string(),
    "--format".to_string(), "json".to_string(),
],

So this issue does not block the quorum lane as built. My earlier framing — "the dogfood architecture rests on a verb that refuses the model" — holds for goal A's floor (which names apr serve), for verb set V, and for any serve-based design, but not for the lane that exists today. I inferred the lane's shape from the recommended architecture instead of reading it.

What it does still block, precisely: model residency. apr run reloads the model per invocation, which is fine for a decision that takes minutes and is why the lane works. Keeping a 9B resident instead of reloading per lane is the reason a serve-based design was proposed at all — so this issue is the prerequisite for that optimisation, not for today's lane. The run-based lane is the design, not a workaround.

Two leads for whoever diagnoses this

apr run on Qwen3.5-0.8B-Q4_K_M answered a schema-constrained prompt in 2.82 s and reported (cached). If run reuses something serve constructs differently, that difference is where the refusal lives — same file, two paths, one refusing.
That run was the default-features crates.io build — no CUDA. So run handles qwen35 on the CPU path at minimum, which places the serve refusal in model construction (Building quantized inference model…), not in anything CUDA-specific. The failing step is the one that does not route to Qwen35Model.

A fleet capability gap this surfaces, worth its own attention

forjar's cargo provider takes no features: field in any manifest on this fleet, so a --features cuda crates.io build is not producible by declaration today. stack-tool-apr on lambda-labs installs the default-features build, which is CPU-only and says so honestly. That matters for the P0 chain's row 3 — a parity receipt for the decide lane's model on the agent's host — because the artifact currently installed there cannot exercise CUDA. Either the receipt is taken against the -cuda release asset, or forjar needs a features capability. That is a forjar decision, not an aprender one, but the receipt cannot be produced until it is made.

Refs #3080, #3558, #3568

Operator ruling 2026-09-20: apr serve is critical for 0.69 — this stays P0 and in this train. It is not deferrable to 0.70 on the grounds that the decide lane uses apr run.

And the corollary, which cuts the other way: waiting for serve before adopting the CLI anywhere is a cop-out. The CLI is proven — 20/20 cells across 0.8B–27B on two GPU hosts, CPU and CUDA — so arbiter and every other consumer should be on apr run now, and #3574's receipt was produced today rather than held for a cell that does not exist yet.

Both halves matter. serve is required because goal A's floor names it, verb set V includes it, and model residency (keeping a 9B resident instead of reloading per invocation) is unreachable without it. But its absence is not a licence for any consumer to stay off the CLI in the meantime.

chore(roadmap): mint PMAT-3571 as a work item so the AD-04 quorum can…

Gap measured past the `<unk>` layer: serve decodes Qwen3.5 through a 0-layer model (aprender-c7, before any code)

The <unk> refusal that stopped layer 1's evidence is fixed by #3678 (PMAT-3609), which is in 0.70 batch-1 (#3707). So I built origin/release/0.70-batch-1 @ 262dfb2f6 with --features cuda (apr 0.69.0 (262dfb2f6)) and asked serve for an answer, on lambda (RTX 4090):

apr serve run Qwen3.5-0.8B-Q4_K_M.gguf --port P {--gpu-layers all | --no-gpu}
curl /v1/chat/completions  temperature 0, max_tokens 1024, 3 messages (user / assistant "Lima" / user "What is my name?"); then the same with "stream": true

route	load	non-streaming	streaming
`--gpu-layers all`	`Model ready: 0 layers` · `gpu-layers: requested=all resolved=0 total=0` · `Pre-uploaded 198 MB weights to GPU` · `CUDA optimized model ready`	HTTP 500 `{"error":"Operation 'generate_gpu_resident_streaming' not supported: Model architecture not supported for GPU-resident path"}`	HTTP 200 with no content: one `{"role":"assistant"}` delta, then the finish chunk
`--no-gpu`	`Model ready: 0 layers` · `gpu-layers: requested=none resolved=0`	HTTP 200, `"content":""`, `finish_reason:"length"`, `completion_tokens:1024` in 6.0 s	HTTP 200, 1024 chunks of `"\n"`, `finish_reason:"length"`

The layer 1 note was right that 0 layers is the correct base. But no serve request handler ever routes to the hybrid forward, which is where the 24 Gated-DeltaNet/attention layers live. git grep -n -i qwen35 origin/main -- crates/aprender-serve/src/api crates/apr-cli/src/commands/serve has exactly one hit: the loader branch from #3608. The base goes to the dense servers unchanged: start_gguf_server_cuda builds an OwnedQuantizedModelCuda over it (198 MB is embeddings + lm_head), and run_cpu_server decodes through it.

The CPU row is worse than a refusal: HTTP 200 for 1024 tokens of embedding → norm → lm_head with no transformer in between. That's the "success that is not an answer" shape. A 0.8B forward cannot produce 1024 tokens in 6 s on this CPU; the speed is the tell.

What the fix will do

Serve routes qwen35 to the same resident Qwen35Session that #3595 introduces for apr chat. It is built once at startup, with F2 once, and it's the only hybrid path. Concretely:

/v1/chat/completions and /v1/completions, streaming and non-streaming, on both routes.
Requests serialized on the session: a single-stream model, the same shape as the dense GPU server.
A prompt that extends the previous request's tokens reuses the state.
A prompt longer than the GGUF's declared context is refused by name, not truncated.
The apr serve ladder rung and positive control from the acceptance list come with it.

Evidence: aprender-c7 scratch m/proof/: serve logs, response bodies, SSE. Build: /mnt/nvme-raid0/targets/c7-3571-proof/release/apr.

Cop (aprender-04): moved to 0.69.1 (apr serve is a required 0.69.1 modality). Field report from Alfredo on main tip: apr serve run Qwen3.5-4B-Q4_K_M.gguf --gpu → Model ready: 0 layers, gpu-layers: requested=all resolved=0 total=0, Pre-uploaded 497 MB, then create_bpe_tokenizer … Unknown token '<unk>' not in vocabulary. That matches c7's 16:14Z measurement. Batch-1 (#3707, armed) removes the <unk> refusal via #3678, and per c7's table that exposes the 0-layer model: HTTP 500 on GPU, and HTTP 200 garbage on CPU. Order for this row: (1) a refusal that no request handler may serve a model whose transformer stack resolved to 0 layers, on any arch, both routes (the smallest safe unit, folded first); (2) the hybrid route on #3595's Qwen35Session.

PMAT-3595: a Qwen3.5 session binds its CUDA context to whatever threa…

PMAT-3571 (1): every apr serve route refuses a zero-layer stack — the…

Seat 3 fill, unit (1): a non-author measured review (cop aprender-04). Lane 3 returned NO-VERDICT twice: gpt-oss r2, flash-high r3

Subject: PMAT-3571-zero-layer-refusal @ 7e6bc1a50, base 52f43da71 (the merge-base). Judged against the fragment's unit (1) and conditions (a) and (b).

VERDICT: PASS. Everything below was re-executed in my own detached worktree and target dir; none of it is taken from the author's receipt.

check	result
`cargo test -p apr-cli --lib --features cuda zero_layer`	rc 0, 11 passed. The case table (`zero_layers_is_refused_by_name_and_any_layer_is_admitted`) and the real-file test (`serve_refuses_the_qwen35_base_at_load`) both ran: the model is present on lambda and the log has 0 `SKIP`
call-site mutant: the check deleted from `build_serve_model`	RED: `serve_refuses_the_qwen35_base_at_load` panicked `a 0-layer stack reached the serve routes`. It failed for the right reason, not at the map step. Restored afterwards
route sweep (condition a), every model constructor in `crates/apr-cli/src/commands/serve/`	each one is followed by the refusal: GGUF `from_mapped`/`create_base_model` → `build_serve_model` (callers: the CPU start `:390`, the CUDA-init-failure CPU rebuild `:743`, wgpu `handlers.rs:937`, and CUDA `with_max_seq_len` takes the refused-or-admitted model); `AprTransformer::from_apr_file` `handlers.rs:1232`; `from_apr` `handlers.rs:1438` and `handlers_include_01.rs:43/213/301`; the Q4K APR pool thread's pre-check `:156`; SafeTensors `convert` `:85` and `convert_sharded` `:286`
a serve entry outside `serve/`	`git grep realizar::cli:: / realizar::api::serve` in apr-cli: none, so `apr serve` has no second loader

Condition (b) (must not block unit 2) is carried in the fragment. qwen35 is refused today, which is correct until unit 2 routes it to Qwen35Session before this check.

Non-blocking notes (not findings against unit 1):

The real-file test prints SKIP and passes where the GGUF is absent, so in CI only the pure case table runs. The call-site mutant is killed on lambda/gx10, not on a runner without the model. A synthetic zero-layer fixture would close that; it can ride unit 2.
The Q4K-APR pre-check does nothing when AprV2Model::load fails. That's acceptable because the pool path then refuses a missing num_layers by name, as the comment says.

PMAT-3571 (1): move zero_layer_refusal_tests to the end of handler_gp…

PMAT-3571: move zero_layer_refusal_tests to the end of handler_gpu_co…

fold c3's batch-1 guard fix (aae778d) into release/0.69.1-batch-1 — z…

PMAT-3571: move zero_layer_refusal_tests to the end of handler_gpu_co…

docs(#3571 unit 2): reconcile the one round with this receipt — it ju…

fold #3791 (+ #3571 unit 1, #3595, #3571 unit 2): APR Q4K GPU serve n…

Confirms the same defect on `/api/chat` (ollama-compat), plus the client-visible symptom it produces

Reproduced independently on main @ 637b31145 (0.69.0, built --features inference,cuda), against Qwen35-4B-Q4_K_M.gguf, RTX 4090, apr serve run --ollama-compat -v ~/Downloads/Qwen3.5-4B-Q4_K_M.gguf --gpu. This adds the ollama-compat route to the existing /v1/chat/completions / /v1/completions evidence above — same root cause, third affected route.

Repro 1 — plain chat, no tools, `/api/chat`

curl -s -o /tmp/plain.json -w "HTTP_STATUS:%{http_code}\n" http://127.0.0.1:11434/api/chat \
  -H 'content-type: application/json' \
  -d '{"model":"qwen3.5","stream":false,"messages":[{"role":"user","content":"2+2=?"}]}'
cat /tmp/plain.json

HTTP_STATUS:500
{"model":"qwen3.5","created_at":"...","message":{"role":"assistant","content":"Operation 'generate_gpu_resident_streaming' not supported: Model architecture not supported for GPU-resident path"},"done":true,"prompt_eval_count":0,"eval_count":0}

Repro 2 — tool-calling request, same route

curl -s -o /tmp/tool.json -w "HTTP_STATUS:%{http_code}\n" http://127.0.0.1:11434/api/chat \
  -H 'content-type: application/json' \
  -d '{
  "model": "qwen3.5", "stream": false,
  "messages": [{"role": "user", "content": "List the files in the current directory using the bash tool."}],
  "tools": [{"type": "function", "function": {"name": "bash",
    "description": "Execute a bash command (bash -c) and return its stdout/stderr.",
    "parameters": {"type": "object",
      "properties": {"command": {"type": "string"}, "description": {"type": "string"}},
      "required": ["command", "description"]}}}]
}'
cat /tmp/tool.json

Same HTTP_STATUS:500, same error text — identical to repro 1, confirming tools are irrelevant to the failure: generation never runs at all.

Control — the CLI path works fine on the identical file

apr run ~/Downloads/Qwen3.5-4B-Q4_K_M.gguf --prompt "2+2=" --max-tokens 8 --gpu

Backend: GPU (CUDA, NVIDIA GeForce RTX 4090, 24106 MB VRAM) [qwen35 hybrid forward, #3090]
Output:
4

apr qa also passes golden-output checks on this file via the same #3090 hybrid path (only the throughput gate fails, 7.1 < 10 tok/s — a perf issue, not a correctness one).

Why this matters beyond another repro

The client-visible failure mode here isn't a clean 500 in every consumer's eyes — it's what happens downstream of it. A tool-calling harness driving apr serve over /api/chat sees this 500 folded into message.content as a plain string (by ollama_handlers.rs::with_upstream_status, working as designed), which is enough to break any client that isn't specifically checking status: it reads as a model that "responded" but then failed tool-argument validation with invalid arguments: missing required property ..., which is a confusing, misleading symptom for anyone not already aware of #3571. Worth adding /api/chat explicitly to the acceptance list alongside /v1/chat/completions / /v1/completions, since it's the route real Ollama-compatible harnesses actually use and has its own re-shaping code path (crates/aprender-serve/src/api/ollama_handlers.rs) that needs the same Qwen35Session routing once unit 2 lands.

fix(gate) #3828: the ladder measured ONE verb while the contract decl…

fold #3715 (Alfredo): the ollama wire's delegation is now ASSERTED, n…

fix(#3874): `/v1/completions` returned 503 for the ENTIRE Qwen3.5 arc…

fix: the three post-fold failures — entity TYPE, a stale doc, and a t…

closed by triage (aprender-dd, #3988 FIXED row, cop OK): fixed by 2de6eabb1, 4e4fe772a, 6f2817ffe, all in v0.69.1. Evidence: serve routes the Qwen3.5 hybrid to Qwen35Session. The sweep's serve cell confirms it

paiml#3571 layer 1: apr serve routes the Qwen3.5 hybrid to Qwen35Mode…

release(0.69.1): batch 1 — 20 receipted rows + 3 combination fixes, o…

Navigation Menu

Repository navigation

apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571

Measured 2026-09-20, published 0.68.2 built from crates.io with `--features cuda`

Why this is P0 rather than a missing feature

Why nothing caught it

Acceptance

Consequences to record now

Blast radius corrected — one consumer smaller, and two diagnostic leads

Two leads for whoever diagnoses this

A fleet capability gap this surfaces, worth its own attention

Gap measured past the `<unk>` layer: serve decodes Qwen3.5 through a 0-layer model (aprender-c7, before any code)

What the fix will do

Seat 3 fill, unit (1): a non-author measured review (cop aprender-04). Lane 3 returned NO-VERDICT twice: gpt-oss r2, flash-high r3

9 remaining items

Confirms the same defect on `/api/chat` (ollama-compat), plus the client-visible symptom it produces

Repro 1 — plain chat, no tools, `/api/chat`

Repro 2 — tool-calling request, same route

Control — the CLI path works fine on the identical file

Why this matters beyond another repro

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

apr serve cannot load Qwen3.5 at all — 'Architecture qwen35 … runs through Qwen35Model'; goal A's floor and the P0 decide-lane design both name apr serve #3571

Description

Measured 2026-09-20, published 0.68.2 built from crates.io with --features cuda

Why this is P0 rather than a missing feature

Why nothing caught it

Acceptance

Consequences to record now

Activity

noahgift commented on Sep 20, 2026

Blast radius corrected — one consumer smaller, and two diagnostic leads

Two leads for whoever diagnoses this

A fleet capability gap this surfaces, worth its own attention

noahgift commented on Sep 20, 2026

noahgift commented on Sep 21, 2026

Gap measured past the <unk> layer: serve decodes Qwen3.5 through a 0-layer model (aprender-c7, before any code)

What the fix will do

noahgift commented on Sep 21, 2026

noahgift commented on Sep 21, 2026

Seat 3 fill, unit (1): a non-author measured review (cop aprender-04). Lane 3 returned NO-VERDICT twice: gpt-oss r2, flash-high r3

9 remaining items

alfredodeza commented on Sep 22, 2026

Confirms the same defect on /api/chat (ollama-compat), plus the client-visible symptom it produces

Repro 1 — plain chat, no tools, /api/chat

Repro 2 — tool-calling request, same route

Control — the CLI path works fine on the identical file

Why this matters beyond another repro

noahgift commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Measured 2026-09-20, published 0.68.2 built from crates.io with `--features cuda`

Gap measured past the `<unk>` layer: serve decodes Qwen3.5 through a 0-layer model (aprender-c7, before any code)

Confirms the same defect on `/api/chat` (ollama-compat), plus the client-visible symptom it produces

Repro 1 — plain chat, no tools, `/api/chat`