Skip to content

parity_block.py: a zero COMPARATOR band median is an unhandled traceback, not a verdict #2735

Closed

Description

@noahgift

Found while verifying PERF-037 (the || true swallow in scripts/parity_host_receipt.sh). Refs #2706. Confirmed present on origin/main and on #2719's rewritten scripts/lib/parity_block.py (head feat/receipt-rule-batch-c), so this is not something #2719 already closes.

The asymmetry

A zero-token band refuses cleanly on one side and crashes on the other.

Zero SUBJECT band — correct refusal:

FAIL  refusing to emit a block this repo's own gate would reject:
        band[8].aggregate_tok_per_sec.subject.aggregate_tok_per_sec: contains a
        non-positive rate -- a throughput of zero or less is not a measurement

Zero COMPARATOR band — traceback:

  File "scripts/lib/parity_block.py", line 143, in _band_from
    ratio = (statistics.median(band["subject"][metric])
             / statistics.median(band["comparator"][metric]))
ZeroDivisionError: division by zero

The same crash occurs at lane level via _lane_from when the lane band's comparator median is zero.

An unhandled traceback is not a verdict. It exits non-zero, so it is not a false green — but it carries no diagnosis, names no band, and reads as a tooling defect rather than as a measurement that must not be published. bench_receipt.py already has the right words for this case and never gets to say them.

This is reachable, not theoretical

gx10's observed state is two zero-token bands. -b 1 in the fairness protocol aborts llama.cpp at load with GGML_ASSERT (rc=134), so the comparator is exactly the side that produces nothing there.

RED-turning mutation

Registered, both polarities, on #2719's version of the file:

W=$(mktemp -d); SHA=$(printf a | sha256sum | cut -d' ' -f1)
mk() { printf '{"runs":[{"tokens_per_sec":%s,"decode_tok_per_sec":%s,"prefill_tok_per_sec":%s,"ttft_p50_ms":20.0,"successful":10,"failed":0,"total_requests":10,"completion_tokens_total":640,"request_details":[{"latency_ms":21.0}]}]}\n' "$2" "$3" "$4" > "$W/$1"; }
for c in 1 4 8 16; do mk "apr-cpu-c$c.json" 100 50 400; mk "llama-cpu-c$c.json" 100 50 400; done
echo "cpu cpu cpu" > "$W/lanes.txt"
run() { python3 scripts/lib/parity_block.py --work "$W" --apr /x --apr-sha "$SHA" \
        --llama /y --llama-sha "$SHA" --llama-build 39173bcac --model m.gguf \
        --install-source crates.io --out - >/dev/null; }

run                                             # control        -> rc=0
mk apr-cpu-c8.json   0 0 400; run               # zero SUBJECT   -> rc=1, named refusal
mk apr-cpu-c8.json 100 50 400
mk llama-cpu-c8.json 0 0 400; run               # zero COMPARATOR-> ZeroDivisionError

The discrimination case matters: the control must stay rc=0 and the subject case must keep its existing named refusal, so a fix cannot be "wrap everything in try/except and return None" — that would silently DROP the band, and _band_from returning None means the band vanishes from lane["bands"]. A dropped band is caught downstream only by _check_bands's declared-vs-seen comparison, which reports the wrong cause.

Suggested shape

Refuse by name, the way the subject side already does: a non-positive comparator median is "not a measurement" and should produce that sentence naming the band and the side, not a stack trace.

Do not fix in-place while #2719 is open — it is already rewriting this file.

Activity

  1. noahgift commented on Sep 5, 2026

    @noahgift
    ContributorAuthor

    PP-066 triage 2026-09-05: in 0.66 scope, no row yet — row I-24 proposed on #2873 (Still live on origin/main: parity_block.py divides by the comparator median at l.571/652/658/680/720/725 with no non-positive guard on that side (the subject side refuses by name). Not fixed by any merged PR.)

    Instance of class #2876 — gate answers empty/degenerate input with a PASS or a crash instead of a named refusal.

    PP-066 triage 2026-09-05 (issue #2873); ledger docs/audits/pp-066-triage-2026-09-05.md.

  2. added this to the 0.68.0 milestone on Sep 10, 2026
  3. added
    automationBot-maintained dashboard, not a defect — skip in triage
    P1High priority
    on Sep 11, 2026
  4. modified the milestones: 0.68.0, 0.69.0, 0.73.0, 0.73.0 on Sep 16, 2026
  5. noahgift commented on Sep 23, 2026

    @noahgift
    ContributorAuthor

    triage 2026-09-23 (aprender-dd, standing triage owner): milestoned to 0.73.0, theme llama.cpp Parity (performance, kernels, parity benches); it moved out of backlog (the 2026-09-17 overflow bucket, not a release).

  6. noahgift commented on Sep 24, 2026

    @noahgift
    ContributorAuthor

    Closed by paiml-implement — fixed by #3006 (merged to main) (triage evidence/triage-1418.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priorityautomationBot-maintained dashboard, not a defect — skip in triage

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions