Skip to content

ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list #5254

Closed as not planned
Closed as not planned
ci(lab): nightly fast-verbs CRUX smoke + tok/s on the ARM64 CUDA edge runner, ranked gap list#5254
Labels
P2Medium priority
Milestone
0.73.0

Description

@noahgift
noahgift
opened on Oct 13, 2026

Problem

No aprender workflow targets the ARM64 CUDA edge runner (labels cuda, ARM64). It is online and idle. Its standing role is quick CUDA tests plus a nightly end-to-end run of the fast verbs, and nothing exercises it.

Ask

Add one nightly LAB job on that runner:

  1. CRUX smoke for the fast verbs.
  2. Decode tokens/s for each fast verb.
  3. One ranked "what to improve" list, biggest gap first, posted as a comment on this issue each night.

Constraints

  • LAB only. The job never runs inside a PR, merge-queue or release job, and it never blocks a merge or a release.
  • A red night updates this issue and stays silent. If it is red or unmeasured for 7 days, it stops until its owner brings it back.
  • No report-only, advisory or continue-on-error rows on any PR or release path.

Acceptance

  • The workflow runs on schedule plus workflow_dispatch on the ARM64 CUDA runner only.
  • Each night appends one comment to this issue with the run id, apr version and sha, a CRUX smoke pass/fail per verb, tok/s per verb, and the ranked gap list.
  • A planted regression (a verb forced to fail) shows up at the top of the list.

🤖 Generated with Claude Code

Activity

  1. noahgift
    added a parent issue on Oct 13, 2026
  2. noahgift
    added 3 commits that reference this issue on Oct 13, 2026

    fix(perf-matrix): re-vendor the matrix after adding the edge host

    1b5c442

    fix(edge-nightly): pin apr, compare to ollama through apr qa, clear S…

    d70dbc9

    fix(edge-nightly): no silent truncation of reasons on the gap list

    13d80fe

  3. github-actions commented on Oct 14, 2026

    @github-actions
    github-actionsBot
    on Oct 14, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/38530433463

  4. noahgift
    added a sub-issue on Oct 14, 2026

  5. github-actions commented on Oct 14, 2026

    @github-actions
    github-actionsBot
    on Oct 14, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M · quant Q4_K_M · sha256 aaf42c8b7c3cab2bf3d69c355048d4a0ee9973d48f16c731c0520ee914699223
    apr apr 0.71.2 (494b63aa) (binary sha256 3d4e03660b53) · backend: GPU (CUDA, Orin, 7607 MB VRAM)

    measure value
    apr decode (apr bench --fast) 7.1 tok/s
    ollama parity (apr qa) skipped
    batched prefill refusals in the bench 8
    CRUX smoke verdict RED: 0 GREEN, 304 RED of 304 cells

    Ranked gaps

    1. CRUX serve run: apr correct on 47 of 129; 6 wrong, 76 unanswered (top reason: refused: route POST /v1/completions takes a raw prompt and no reference renderer was given: unrendered text would ask a different question)
    2. CRUX serve stream: apr correct on 64 of 129; 8 wrong, 57 unanswered (top reason: refused: route POST /v1/completions takes a raw prompt and no reference renderer was given: unrendered text would ask a different question)
    3. CRUX chat: apr correct on 21 of 23; 2 wrong, 0 unanswered
    4. CRUX run: apr correct on 17 of 19; 2 wrong, 0 unanswered
    5. comparator ollama refused 304 cells: refused: ollama 0.17.6 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat
    6. ollama comparison: apr qa ollama_parity skipped (Skipped: Dense-loader gate: Gated DeltaNet runs on the CPU (CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091) and the GPU (GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090), but this gate measures through OwnedQuantizedModel::from_mapped, which has no hybrid path)
    7. batched prefill refused 8 times in the bench (one token at a time)

    Run: https://github.com/paiml/aprender/actions/runs/38531159254

  6. noahgift commented on Oct 14, 2026

    @noahgift
    noahgift
    on Oct 14, 2026
    MemberAuthor

    Edge parity rows, measured (runner label jetson, 8 GB unified-memory Orin)

    Model Qwen3.5-2B-Q4_K_M.gguf (sha256 aaf42c8b7c3c…), apr 0.71.2 built --features cuda from #5292 head cebe27dc, ollama 0.17.6 on the same file. Every number below is the engine's own report (apr bench / apr qa / CRUX receipts); nothing is timed by the harness.

    row apr ollama gap / ticket
    GPU decode, apr bench --fast 7.1 tok/s —
    GPU decode, CRUX run, median of 19 cells 4.0 tok/s 26.7 tok/s 0.15× — #5301
    TTFT 897 ms (apr bench) not measured (prompt rate 418 tok/s) ollama TTFT row — #5276
    peak RSS 6857 MB (bench VmHWM) ~2.4 GB (ollama ps SIZE, not the same measure) #5258
    ollama parity gate (apr qa --ollama-from-file) failed, did not measure: apr refused GPU capacity while ollama's import was resident #5302
    CPU decode not measured: this binary has no bench --no-gpu not measured #5276
    apr profile refused: qwen35 is not in the dense profile loader stated refusal

    Template fix confirmed. With #5292's ChatML template built from the GGUF's own tokenizer.chat_template, ollama answers every cell (it refused 304/304 before).

    Verb × device matrix (correct / answered / cells, thinking off)

    verb GPU apr GPU ollama CPU apr CPU ollama
    run 17/19/19 16/19/19 not run not run
    chat 21/23/23 20/23/23 not run not run
    serve run not run not run not run not run
    serve stream not run not run not run not run
    code not run n/a not measured (#3978) n/a

    Every CRUX verdict is RED for one stated reason, "no ground-truth control (neither hf nor vllm answered)": this device has no reference engine. The correct counts above are per engine. This was a reduced run (run + chat, GPU, thinking off), because the full matrix (5 verbs × off/on × gpu/cpu) takes hours on this device. The nightly in #5292 runs all of it, and every not run cell above is a missing measurement there (exit 1), not a skip. Uncovered: serve and code on GPU and everything on CPU, in the next full nightly; apr code on CPU is #3978; CPU decode is #5276.

    A first run this morning was void: host memory was held by an unrelated stray process, apr refused host RAM on every cell, and ollama ran at 1 tok/s. It was killed and the rows above re-measured on a clean box.

    Five-whys: "CRUX 0/304 GREEN" while ollama refused every prompt

    Problem. Nightly run 38531159254 reported CRUX RED, 0 GREEN of 304, and the job was green. ollama refused all 304 cells with "imported the GGUF with no chat template (TEMPLATE {{ .Prompt }})". No gate or alarm saw it.

    # Why Answer Fix
    1 Why did ollama refuse every cell? ollama 0.17.6 imported the Qwen3.5 GGUF with TEMPLATE {{ .Prompt }}, so every chat turn became a raw completion, and the harness correctly refused to judge that. #5292: build a ChatML Go template from the GGUF's own tokenizer.chat_template, re-create the model, and check that the stored template is byte-identical. Otherwise refuse by name.
    2 Why did ollama import it with no template? The GGUF does carry tokenizer.chat_template. ollama 0.17.6 cannot translate that Jinja (the enable_thinking branches), and it silently falls back to {{ .Prompt }}. #5292 (above). The import is checked with ollama show --template, never assumed.
    3 Why was the template never written, for GGUFs that apr exports itself? append_tokenizer_to_metadata returned early on the tokenizer.json path without writing tokenizer.chat_template. Only the APR-metadata fallback and the qwen35 path wrote it. #5292: the tokenizer.json path now writes it (sources in order: the tokenizer, APR metadata, chat_template.jinja, tokenizer_config.json), with a unit test. The nightly now fails when the model carries no template.
    4 Why did no gate see 304 refusals? The CRUX judge maps a comparator that answered nothing to RED by design (no third state), and the nightly only printed the verdict. Nothing asked "did each engine answer anything?". #5292: a fixed verb×device×engine matrix. Any combination with no cells or zero answers is a missing measurement, and the run exits 1.
    5 Why was the nightly built without that check? Its definition of done was "a verdict is printed", not "every row is measured". A RED from a dead comparator looks the same as a RED from wrong answers. #5292: every row has a measured value or a stated, ticketed reason (#5276 CPU bench, #3978 apr code on CPU). An absent or zero tok/s, TTFT or RSS fails the run.

    The same chain for the skipped ollama_parity row

    # Why Answer Fix
    1 Why was ollama_parity skipped? apr qa skips the parity gate for model families its dense loader refuses, and Qwen3.5 is one of them. #5292: --ollama-from-file runs the gate on this file.
    2 Why did it not compare this file anyway? The gate compared against a size-matched qwen2.5-coder:<size> tag, not the file under test. #5292: ollama decodes the same GGUF through a pid-named import that is unloaded and removed on drop.
    3 Why was a skip not a failure? A skipped gate read as "not applicable" in the nightly. #5292: a skipped parity gate is now a missing measurement (exit 1).

    Why the gap rows were invisible until today

    # Why Answer Fix
    1 Why was a 0.15x decode gap and a 6.9 GB RSS never reported? No row measured either on this device: the parity gate skipped Qwen3.5 and then, once it ran, could not co-host both engines. #5292 (rows are mandatory), #5302 (sequential loads)
    2 Why does apr hold 6.9 GB for a 1.2 GB file? The host build keeps an f32 token embedding plus prefill copies in the same unified pool. #5258
    3 Why is decode 4-7 tok/s? Batched prefill is refused for want of free device memory (the pool is full of item 2), and decode on Orin is untuned. #5301
  7. noahgift
    added a commit that references this issue on Oct 14, 2026

    audit(surface): re-point the rows this branch's two qa/profiling edit…

    69fd2fe
  8. noahgift
    added a sub-issue on Oct 14, 2026
  9. noahgift
    removed sub-issues on Oct 14, 2026
  10. noahgift
    added a commit that references this issue on Oct 14, 2026

    fix(export): qwen35 GGUF wrote tokenizer.chat_template twice

    98a3706

  11. noahgift commented on Oct 14, 2026

    @noahgift
    noahgift
    on Oct 14, 2026
    MemberAuthor

    Edge nightly, first full run with the #5292 harness (Orin, Qwen3.5-2B-Q4_K_M sha256 aaf42c8b…, apr 0.71.2 binary sha256 afbf4b654285)

    measure value
    apr decode, GPU (apr bench --fast) 7.1 tok/s
    apr TTFT, GPU 895.7 ms
    apr peak RSS (VmHWM) 6721 MiB
    apr decode, CPU not measured: no bench --no-gpu flag (#5276)
    apr code on CPU not measured (#3978)

    Verb × device (correct / answered / cells):

    verb GPU apr GPU ollama CPU apr CPU ollama
    run 17/19/19 16/19/19 15/19/19 16/19/19
    chat 21/23/23 20/23/23 18/22/23 20/23/23
    serve run 47/53/129 41/53/129 39/51/129 44/53/129
    serve stream 64/72/129 56/72/129 55/71/129 60/72/129
    code 4/4/4 n/a not measured n/a

    The chat-template fix works: ollama now answers every run/chat cell. In the previous nightly it refused all 304.

    Defects this run exposes (none of them is an apr correctness result):

    1. 76–78 serve cells per device are unanswered, on both engines. The route is POST /v1/completions, which takes a raw prompt, and the harness gives no reference renderer, so the cell is refused rather than judged. That is a harness gap: render with the GGUF's own template, or drop the route from the cell set and count it as missing.
    2. The CRUX smoke verdict says 0 GREEN, 304 RED while the table above shows apr correct on 149 GPU cells. The verdict line and the per-cell table disagree; the verdict must be traced before anyone reads it.
    3. ollama_parity could not run: the GPU capacity planner refused because only 1172 MiB of unified memory was free. The likely cause, not yet proven, is that ollama still held the model from the CRUX cells. Unload it (keep_alive: 0) before apr qa, then re-measure.
    4. apr profile refuses qwen35 (dense loader only). Profiling needs the hybrid path (GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090/CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091) or this row stays missing.
    5. Peak RSS of 6.7 GiB for a 1.2 GiB Q4_K_M model is worth its own look. VmHWM covers the whole bench process, so split it by phase before calling it a leak.

    The run exits 1 on the missing rows, as designed.

  12. noahgift
    added this to the 0.73.0 milestone on Oct 14, 2026

  13. 7 remaining items

  14. github-actions commented on Oct 16, 2026

    @github-actions
    github-actionsBot
    on Oct 16, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/38894944048

  15. github-actions commented on Oct 16, 2026

    @github-actions
    github-actionsBot
    on Oct 16, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/38941050008

  16. github-actions commented on Oct 17, 2026

    @github-actions
    github-actionsBot
    on Oct 17, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/38982365249

  17. github-actions commented on Oct 17, 2026

    @github-actions
    github-actionsBot
    on Oct 17, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/39006904903

  18. github-actions commented on Oct 18, 2026

    @github-actions
    github-actionsBot
    on Oct 18, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/39107365210

  19. noahgift
    added
    P2Medium priority
    on Oct 18, 2026

  20. github-actions commented on Oct 21, 2026

    @github-actions
    github-actionsBot
    on Oct 21, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/39565283712

  21. noahgift commented on Oct 22, 2026

    @noahgift
    noahgift
    on Oct 22, 2026
    MemberAuthor

    Tracked in #3999 (triage 2026-10-22).

  22. github-actions commented on Oct 22, 2026

    @github-actions
    github-actionsBot
    on Oct 22, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/39743335295

  23. noahgift commented on Oct 22, 2026

    @noahgift
    noahgift
    on Oct 22, 2026
    MemberAuthor

    tracked in #5348

  24. noahgift
    closed this as not plannedon Oct 22, 2026

  25. github-actions commented on Oct 25, 2026

    @github-actions
    github-actionsBot
    on Oct 25, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/40163902584

  26. github-actions commented on Oct 26, 2026

    @github-actions
    github-actionsBot
    on Oct 26, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/40272245739

  27. github-actions commented on Oct 29, 2026

    @github-actions
    github-actionsBot
    on Oct 29, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/40827106719

  28. github-actions commented on Oct 30, 2026

    @github-actions
    github-actionsBot
    on Oct 30, 2026 – with GitHub Actions

    Edge nightly · model Qwen3.5-2B-Q4_K_M.gguf · not_measured: no gap list (build or tooling failed)

    Run: https://github.com/paiml/aprender/actions/runs/41012146002

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions