Skip to content

EPIC: release train 0.77.0 SERVE-PARITY — apr serve at Qwen3.5 parity with llama.cpp and ollama #6570

Open
0 of 5 issues completed

Description

@noahgift

Plan doc: docs/roadmaps/releases/0.77.0-serve-parity.md (PR #6569).

Work ticket: #6567. Categorization: docs/roadmaps/releases/categorization-2026-10-30.md (reviewed with this epic; 3 rounds, fix-forward after round 3 per the 15:01Z review cap).

Goal gap, stated first. The goal: apr serve serves Qwen3.5 as correctly as llama.cpp and ollama do, and as fast. 0.77.0 scores every cell against that goal.

  • Expected green by 12-04:
    • correctness (S1)
    • API compatibility (S5)
  • Expected green or yellow: the serving shape, i.e. batches form and caches hit (S4).
  • Expected red:
    • c=1 decode speed against llama.cpp (S2a)
    • prefill speed (S3)
    • What was last measured:
      • llama.cpp decodes ~1.55× faster at c=1, and apr is at parity with ollama. Both figures are for qwen2.5-coder-1.5b Q4_K_M, not Qwen3.5. No graded Qwen3.5 three-engine figure exists; SP-00 produces the first one.
      • On Qwen3.5 specifically: our prefill runs at 0.19–0.20× of llama.cpp. Decode GEMV reaches 223–504 GB/s against llama.cpp's 776–894. The GDN recurrence is 17.5× slower.
    • The kernel rows in W2 are each ≤1 day. They narrow the gap but are not expected to close it in one week.
  • Ollama floor (S2b): apr ÷ ollama must not fall below 0.90. On Qwen3.5 it is unmeasured until SP-00.

Operator rulings applied (not relitigated):

  • Dates: 0.76.0 11-27, then weekly. 0.77.0 is due 12-04 and is date-driven: it ships whatever is green; a red cell ships red with its receipt; nothing is waived.
  • NOT_MEASURED counts red.
  • A ticket past 1.5 days is split, or dropped to 0.78, the same day.
  • Caps: ≤5 workers per repo, 1 ticket in progress per worker; capacity 5/day.
  • No new Python. A Python competitor tool runs only under an external-tool ruling like the Unsloth one (pinned digest, a make/forjar target on lambda, JSON output only). None is needed here: llama.cpp and ollama are native binaries.
  • The planning worker files; this document files nothing.
  • The 0.76.0 epic (issue 6532) commits 0.77 to the following, which form the FT-carry lane (§4):

Release: 0.77.0 (milestone #32), due 2026-12-04.

How the cells score. The cells below are a release scorecard, not CI checks.

  • Releases are date-driven, so no cell blocks a merge or a release.
  • Each cell ships GREEN, YELLOW or RED with its receipt, in the colour scheme of the 0.76 epic (issue 6532, ruling 5): exit 0 = GREEN, 3 = YELLOW (ships yellow, counted not-green), 1 or 2 = RED.
  • The falsification tests the rows add are LAB under C324 until promoted under its item 7 (operator yes, three green nights). None goes on a PR or merge-queue path. A cell colour is a scorecard measurement from an on-demand run on the pinned release binary: a red cell ships as red and blocks no merge or release, so scoring a cell on one of these tests gives the test no GATE power. GPU falsifiers go in gpu-lab-nightly.yml.
  • A row that is cut leaves its cell RED (NOT_MEASURED), named in the SP-20 report.

Absorbs. These epics become children of this epic: #5305 (Qwen3.5 speed), #5317 (dogfood lane), #5321 (8 GB Orin), #5325 (GPU refuse-by-name), #5329 (serve request path). Their own children keep their milestones (categorization §2).

Tracked, not reparented: #3111 and #4378 (parent #5323), #6389 (parent #5965).

Carried prerequisites (the carry script moves an item only one release ahead, so at each earlier cut the planner re-milestones it here from backlog; categorization §1 and §6 step 6): #4979, #4449, #4969, #4443 (S1); #4376, #4442, #4483, #4485, #4486, #4313 (S2/S3); #4971, #4274 (S4).

Related: issue 5312 (CUDA concurrent batching, 0.75), issue 5348 (Agent Ready, 0.75), issue 3739 (per-release CRUX dogfood), issue 3719 (apr code on Qwen3.5 CUDA, child of absorbed issue 5325; carried by AGENT-GATEWAY), issue 5323 (kernels; moves to 0.79 ANY-MODEL).

1. Baseline (measured, cited)

Fact Value Source
GPU decode vs ollama, RTX 4090 sm_89, qwen2.5-coder-1.5b Q4_K_M 1.015–1.109× (parity, inside noise) docs/BEATS.md:59, :131
c=1 decode vs llama.cpp, RTX 4090, same model llama.cpp ~1.55× faster (431 vs 277 tok/s) docs/BEATS.md:65, :133
Qwen3.5 prefill vs llama.cpp (llama-bench pp) 0.19–0.20× #4376 title
Qwen3.5 32k prefill 3.2k tok/s #4442 title
Decode GEMV bandwidth 223–504 GB/s vs llama.cpp MMVQ 776–894 #5221 title
GDN delta-rule recurrence 44.7 µs/call vs 2.5 µs (17.5×) #5222 title
c=4 batching under load forms no batch at 200 ms CUDA_BATCH_WINDOW_MS (LAB) #6389
Comparator lane no W1 cell producible in either mode (BLOCKER) #2817
Serve tokenization GGUF prompts tokenized by greedy longest-match, not BPE merges #4979
Chat turn 2 re-prefills from position 0; session extend never engages #4274
Three engines on one GGUF, 4090 exists for qwen2.5-1.5b-instruct Q4_K_M (apr 0.69.1, llama.cpp b10987, ollama 0.34.2), as a producer receipt, never graded as a release verdict. None exists for Qwen3.5 evidence/crux/0.69.1/producer-3962/lambda-gpu-qwen2.5-1.5b-q4km/README.md

CRUX rows in this theme (contracts/crux-<row>-v1.yaml, e.g. contracts/crux-C-04-v1.yaml):

Row Capability Contract status
C-04 ollama /api/chat partial
C-06 continuous batching draft
C-07 paged-attention KV cache draft
C-08 automatic prefix caching draft
C-13 /v1/embeddings partial
  • "Partial" rows have algorithm-level falsification tests (crates/apr-cli/tests/falsification_crux_c_04.rs, …_c_13.rs), wired into CI through ci/explicit-test-commands.d/475-apr-cli-kept-cli-tests.cmd.
  • C-06, C-07 and C-08 have falsification_tests blocks in their contracts (crux-C-06-v1.yaml:67, C-07:77, C-08:69) but no implemented test.

Code state (main, 2026-10-30):

  • The ollama API lives in crates/aprender-serve/src/api/ollama_handlers.rs and crates/apr-cli/src/commands/serve/ollama.rs.
  • /v1/embeddings is routed for apr serve in crates/aprender-serve/src/api/router.rs:190 (Banco has its own route in crates/aprender-orchestrate/src/serve/banco/router.rs).
  • The CUDA batch scheduler, crates/aprender-serve/src/api/cuda_batch_scheduler.rs:
    • reads CUDA_MAX_BATCH at :123 and CUDA_BATCH_WINDOW_MS at :130; the window defaults to 0 ms (:133)
    • logs [PMAT-044] Batch scheduler started: max_batch=…, window=…ms at :392
    • logs [PMAT-044] Batch m=… done in …ms per batch at :537. This is the trace line S4 cites.

2. Exit (done_when)

Measurement:

  • Host and model: lambda-labs RTX 4090; Qwen3.5-4B Q4_K_M; one GGUF file, read by its sha256, for all three engines and for S6 on the Orin.
  • Versions: llama.cpp and ollama pinned, each version recorded in the receipt.
  • Runs: n=5 per cell; ratios as medians. Where a threshold is stated, LCB95 is the lower bound of a one-sided 95% t-interval (4 degrees of freedom) on the 5 per-run ratios, each run of apr paired with an interleaved run of the reference engine.
  • Receipts: under evidence/serve-parity-077/<date>/, one JSON per cell.
  • Exit codes: 0 PASS (GREEN), 1 FAIL, 2 NOT_MEASURED, 3 YELLOW. A crash or OOM is FAIL; NOT_MEASURED counts red.
  • Binary: the apr build is pinned with scripts/apr_bin.sh, and its version+SHA are in every receipt.
  • Batch settings: CUDA_MAX_BATCH and CUDA_BATCH_WINDOW_MS are set explicitly for every serve run and recorded in the receipt, with the :392 start line as proof of what was used.

Exit cells:

3. Cells

Cell Exit cells Host Notes
Q1 Qwen3.5-4B Q4_K_M S1–S5 lambda 4090 primary
Q2 Qwen3.5-9B Q4_K_M S2, S3 lambda 4090 reported only
E1 Qwen3.5-4B Q4_K_M, the §2 GGUF by sha256 (#5321) S6 jetson Orin 8 GB edge

4. Children

New rows are SP-nn, each ≤1 day, filed just in time (categorization §5). Each row lists its depends and its receipt.

W0 Measure first (blocks W2/W3 receipts):

  1. BLOCKER: no W1 cell is producible on the comparator lane in either mode — streaming reports prompt_tokens=0, non-streaming reports 534 against a 512 ± 8 band #2817 comparator lane producible (0.77.0).
  2. SP-00 lambda baseline receipt: cells Q1/Q2 on all three engines, n=5, before any W2 row merges. Due 11-30 (Monday), the first lambda weekday after FT-PARITY's last pass (lambda is shared through 11-25; 11-26/27 are 0.76's final receipts and release).
  3. SP-01 contract contracts/serve-parity-077-v1.yaml, cells S1–S6 with falsification tests; pv validate green.

W1 Correctness (S1):

W2 Speed (S2/S3):

W3 Serving shape and API (S4/S5):

W4 Edge (S6): #5276, #3761.

W5 Report:

  • SP-20: docs/BEATS.md rows and the C-04/06/07/08/13 contract discharge_status values, updated only from receipts; names every cut row.
  • SP-21: open the 0.78 residual-gap ticket for S2/S3 with the measured numbers.

FT-carry lane (committed by issue 6532, ≤6):

Flow lane (cap 10, categorization §3/§3b): #4526, #4652, #5917, #5951, #6269, #6360, #6391, #6408, #6468, #5990.

Cut order (first dropped to 0.78 when capacity runs short):

  1. the flow lane, whole
  2. the Q2 cell
  3. SP-12
  4. S6 (W4)
  5. FT-33c: Flash backward varlen and wiring, with the C1 receipt #6562 and FT-36a: Fused LoRA in the MLP: base + B(Ax) in one pass, hand-written backward #6563 (conditional FT carries; they move to 0.78 with the G1 residual-gap epic)

S1 rows, S5 rows, SP-10, SP-11 and the FT-carry rows other than #6562/#6563 are never cut by this capacity order; the 1.5-day andon still applies to them (split, or drop to 0.78 the same day, and the cell ships red). A cut row is named in the SP-20 report, and its cell counts RED (NOT_MEASURED); Q2 and SP-12 are the exceptions, being reported-only, and the flow lane has no exit cell, so cutting them turns no cell red.

5. Schedule

Window Work Host
11-02 → 11-25 only if categorization §5 allows early filing (open count < 100): W1 code + falsifiers, SP-01, SP-13/14 on CPU, W2 kernel development (non-receipt runs on gx10/yoga) non-lambda
11-27 0.77 rows filed at 0.76 close; W1/W3 code starts non-lambda
11-30 SP-00 baseline lambda 4090
11-30 → 12-02 W2 before/after receipts, SP-10/11/12 live receipts, W4 on Orin lambda, jetson
12-03 final receipts for S1–S6 on what is merged by 12-03 00:00Z; SP-20 lambda
12-04 release (nightly promotion) —

6. Risks

7. Owners

  • ≤5 workers repo-wide, 1 ticket in progress each. The cop assigns at filing time.
  • Suggested split: W0 + W5 + flow lane, one owner; W1 one; W2 one; W3 + W4 one; FT-carry one (the FT-PARITY owner).

Review debt (Haiku, round 3: FAIL; shipped fix-forward under the 15:01Z review cap, not a blocker)

  • (fixed forward 10-30, needs re-review) line 192: Capacity arithmetic does not reconcile, and the cut order cannot restore capacity: after the full cut order, 26 non-flow rows still exceed 25 ticket-days.
  • (fixed forward 10-30, needs re-review) line 44: The risk section treats more than 5 carried prerequisites as the trigger, but the epic's own carried list already names 12 (S1 4, W2 6, S4 2), and the load count excludes them.
  • (open) line 184: The ≤100 open-ticket cap ruling is not applied: the categorization's own arithmetic puts the open count at 206 after the plan, so the 11-02→11-25 early-filing window can never open, and the rows filed on 11-27 breach the cap.
  • (fixed forward 10-30, needs re-review) line 90: LCB95 is named as the threshold statistic for S2a, S2b and S3, but no estimator is given for n=5 medians of a ratio, so these three cells cannot be computed as written.
  • (fixed forward 10-30, needs re-review) line 88: The model is not pinned for S6, and §2 contradicts the S6 host: §2 says one Qwen3.5-4B GGUF is used for all three engines, but S6 runs on the Orin with the size left to issue 5321.
  • (fixed forward 10-30, needs re-review) line 113: The S5 C-13 cosine ≥0.99 is undefined: pooling and normalisation for the embedding of the same GGUF are not stated, so the llama.cpp side can differ from apr for reasons unrelated to correctness.
  • (fixed forward 10-30, needs re-review) line 3: The header cites plan2/cat-final-307b8bc968a6.md as the categorization reviewed with this epic. That file is archived under plan2/old/; the current one is cat-final-a0101943a8f8.md.
  • (open) line 131: NIT: §4 says each row lists its depends and its receipt, but several rows do not (SP-12, SP-13, SP-14, SP-21, issue 3725, issue 5276, issue 3761, and the W1 tickets).
  • (fixed forward 10-30, needs re-review) line 178: NIT: the rule that a cut row's 'cell counts RED' has no cell to count for the flow lane or SP-12, both of which map to no exit cell.
  • (fixed forward 10-30, needs re-review) line 106: NIT: S4 GREEN needs a batch line at each c in {4,8,16}, but YELLOW says only 'batches form', and RED says 'no batch forms', so a run that batches at c=4 and c=8 but not c=16 has no defined colour.
  • (open) line 98: NIT: S1 needs issue 2815, issue 4979, issue 3111, issue 4378 and issue 4374 'closed, each with a falsifier', but the falsifier for each is not named.
  • (fixed forward 10-30, needs re-review) line 137: NIT: 'C-06, C-07 and C-08 have no falsification test' is inaccurate: their draft contracts have falsification_tests blocks (crux-C-06-v1.yaml:67, crux-C-07-v1.yaml:77, crux-C-08-v1.yaml:69). What is missing is an implemented test.
  • (fixed forward 10-30, needs re-review) line 43: NIT: S2b is named 'the beat-ollama threshold', but the existing gate is a median-of-7 no-collapse floor (BEATS.md:131) and this epic uses n=5 with LCB95, so the names describe different gates.
  • (open) line 149: NIT: SP-01's 'pv validate green' could put the new contract's falsifiers on a PR path. pv validate is run in book.yml and in scripts/ and I did not confirm it is off the PR path.
  • (open) line 20: NIT (unverified): the 'colour scheme of the 0.76 epic (issue 6532, ruling 5)' and work ticket issue 6567 cannot be checked here, because gh is unavailable and issue 6567 is not in the 10-30 open-issue snapshot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions