cbtop reports status PASS and ci_result green over its own grade F when the profiler returns zero bricks #2730

Assimilated from #2728 · child of #2706 (APR-PERF-GATE-001, the receipt rule)

Behaviour

When the BrickProfiler collects no per-brick data, apr cbtop --headless emits a report that says "grade": "F", "brick_scores": [] and "falsification": {"passed": 0, "failed": 1} — and in the same document says "status": "PASS", "ci_result": "green". apr cbtop --ci exits 0 on that report.

This is not a display bug. The CI gate consults the report's own verdict, and that verdict is computed vacuously.

Reproduction

Binary pinned per scripts/apr_bin.sh discipline — built from HEAD d6c6c6f8f, --features cuda, verified apr 0.63.0 (d6c6c6f8f). GPU: RTX 4090.

$ apr cbtop --model-path /home/noah/models/qwen2.5-coder-0.5b-instruct-q4_k_m.gguf \
      --headless --json
rc=0

  "brick_scores": [

  ],
  "pmat_scores": { "brick_score": 0, "grade": "F" },
  "falsification": { "total_points": 1, "passed": 0, "failed": 1, "blocked": 0 },
  "status": "PASS",
  "ci_result": "green"

stderr, same run — the mechanism, stated by the tool itself:

  No per-brick data collected (profiling may need per-brick sync points)
=== Real Brick Scores (from BrickProfiler) ===
  Total: 0.0µs across 1 decoded tokens (0.0µs/decoded_tok)

The gate:

$ apr cbtop --model-path <model> --headless --json --ci --iterations 5 --warmup 2
rc=0          # grade F, failed 1, zero bricks — and the CI gate passes

check_ci_thresholds prints no FAIL line at all, because report.ci_result == "green".

Discrimination case — the tool can fail, so this is specific, not "cbtop never reds":

$ apr cbtop --model-path <model> --headless --json --ci --brick-score 1 --iterations 5 --warmup 2
rc=5
cbtop: FAIL - Brick score 0 < threshold 1

It reds only when the caller supplies an explicit numeric threshold. The report's own verdict never reds.

Root cause — two expressions disagree about the empty set

crates/apr-cli/src/commands/cbtop_measure_batch.rs:344-357

let n_bricks = brick_reports.len().max(1);                              // :344  empty -> 1
let score_sum: f64 = brick_reports.iter().map(|b| b.score as f64).sum(); // empty -> 0.0
let pmat_brick_score = (score_sum / n_bricks as f64) as u32;             // 0/1 = 0 -> grade F

let all_pass = brick_reports.iter().all(|b| b.gap_factor <= 1.0 + 1e-9); // :350  empty -> TRUE
let status    = if all_pass { "PASS" } else { "FAIL" };                  // :353
let ci_result = if all_pass { "green" } else { "red" };                  // :354

let brick_passed = brick_reports.iter().filter(...).count() as u32;      // 0
let brick_failed = (n_bricks as u32).saturating_sub(brick_passed);       // 1 - 0 = 1

.max(1) asserts there is one brick so it can be counted failed; .all() is vacuously true on the same empty vector so everything passed. The identical input is simultaneously "1 brick, 1 failed" and "all bricks passed".

The earlier fix was incomplete. The doc comment on check_ci_thresholds (cbtop_report_tui.rs:1-8) records that it was written precisely because "apr cbtop --ci printed a red report and exited 0 — a gate that could not fail." It fixed that by gating on report.ci_result != "green" — delegating the verdict to the value that is vacuously green. The gate still cannot fail on this input.

Second site, same class: cbtop_get_cpu_memory.rs:103,143-144 (the --simulated path) uses all_pass = brick_scores.iter().all(...) with n_bricks = brick_scores.len() and no .max(1). Empty there yields total_points: 0, passed: 0, failed: 0 with PASS/green — a green report attesting to zero measurements. Both sites need the fix.

RED-turning mutation to register

The verdict is currently unreachable to tests: it is computed inline inside build_and_output_report, which is private and #[cfg(all(feature = "inference", feature = "cuda"))], and carries #[cfg_attr(coverage_nightly, coverage(off))].

1. Extract the verdict into a feature-independent, testable helper, e.g.
   fn report_verdict(bricks: &[BrickScore]) -> (&'static str, &'static str),
   returning ("FAIL", "red") when bricks.is_empty(), and use it at both sites
   (cbtop_measure_batch.rs:353-354 and cbtop_get_cpu_memory.rs:143-144).
2. Add test_empty_bricks_is_fail_not_vacuous_pass asserting report_verdict(&[]) == ("FAIL", "red").
3. Mutation: delete the bricks.is_empty() clause, restoring the bare bricks.iter().all(...). The new test must go RED. Revert; confirm GREEN.
4. Discrimination: report_verdict(&[one brick with gap_factor 0.9]) must still be ("PASS", "green") — the guard must not flag every input.

An end-to-end assertion is also available and does not need a GPU: a zero-brick report must make apr cbtop --headless --ci exit non-zero without any --brick-score / --throughput argument.

Why this belongs to #2706

The receipt rule is that a performance number is evidence only if something can prove how it was measured. This report carries a real, reproducible throughput number (362.85 tok/s) next to "status": "PASS" earned by measuring nothing — zero bricks, zero profiler samples. It is the exact failure mode §5's mutation registry exists to prevent, in the tool the epic uses to produce receipts.