Repository navigation
aprender-verify-ml test_pipeline_generate_small asserts wall-clock (generation_time_ms > 0) — failed v0.69.0's clean-room B2 on a fast gx10 runner; same unchanged test passed on gx10-pool2 (09-20) and intel #3703
Description
Measured 2026-09-21 on the v0.69.0 tag
infra clean-room run 35608872676, clean-room (aprender) on gx10-pool1 (aarch64), GATE B2 FAILED (534 s):
aprender-verify-ml --lib: test result: FAILED. 1144 passed; 1 failed
data::pipeline::tests::test_pipeline_generate_small panicked at crates/aprender-verify-ml/src/data/pipeline.rs:438:9:
assertion failed: stats.generation_time_ms > 0 || stats.total_generated < 10
generation_time_ms = elapsed.as_millis() as u64 (pipeline.rs:236) with count(10): a host that generates 10 programs in under 1 ms gets 0 > 0 || 10 < 10, i.e. false. The code has been unchanged since the 2026-04-06 subtree merge (dd0e464). It passed on gx10-pool2 (nightly 35478882518) and on intel (0.68.2 tag, 35495206021). It's timing-dependent, which is the "NO wall-clock assertion in a required check" class. The tag's clean-room was re-dispatched (run 35610850375, intel-clean-room-14), because a tag's content can't change.
done_when
- the assertion tests what the code promises (the stat is populated and monotone), never elapsed wall time. E.g. drop the timing clause, or record µs and assert
generation_time_usis set, not> 0 - a sweep of every test in the crates clean-room B2 runs (
--libof the published crates) foras_millis() > 0/elapsed ... >/Instant-based assertions, with each hit fixed or listed with its reason.scripts/check_no_timing_in_required.shextended to see this form (it passed while this sat in a required gate) - mutation proof: a test build with an instantaneous generator must still PASS
Activity
noahgift commentedon Sep 21, 2026
Additions from #3702 (closed as a duplicate of this issue, which is the owned one), for the owner:
PipelineStats::throughput()(pipeline.rs:92-96) returns0.0whenevergeneration_time_ms == 0, so a fast host reports 0 programs/s.generation_time_msis a publicu64field, and PipelineStats is built by literal at pipeline.rs:508 (a test), so fixing the stat means either a non-breaking addition or its own row.ml/evaluator.rs:637assert!(result.total_time_ms > 0.0)comes fromelapsed.as_secs_f64() * 1000.0(evaluator.rs:404), which is not truncated;audit.rs:840assert!(path.duration_ns > 0)comes fromas_nanos()(audit.rs:143). Neither reads 0 on a fast host, so pipeline.rs:438 is the only truncating one.assert_eq!(stats.total_generated, programs.len()), which holds by construction (pipeline.rs:231,total_generated: all_programs.len()). No clock is involved.noahgift commentedon Sep 21, 2026
Scope amendment (issue author, cop), 2026-09-21: done_when 2's sweep is satisfied by listing each remaining hit with its reason. The slow-host
elapsed < Nupper-bound class is the named remainder, tracked in #3705, and #3703's ledger may list those hits with reason#3705. The PR for #3703 may therefore sayCloses #3703+Refs #3705.test(qa): assert the duration is PRESERVED, not that elapsed time is …
noahgift commentedon Sep 24, 2026
Fixed on main by 8c7822f (#3772, the 0.69.1 batch; its body carries
Closes #3703but it merged to the release line so the keyword never fired). On origin/main aa7c6ef: crates/aprender-verify-ml/src/data/pipeline.rs:438 no longer assertsgeneration_time_ms > 0(asserts what generate() promises);scripts/check_no_timing_in_required.shsees the shape; remaining hits listed in docs/audits/pmat-3703-timing-sweep.md with reason #3705 per the 2026-09-21 scope amendment; quorum docs/audits/quorum-PMAT-3703.json 3/3 PASS.release(0.69.1): batch 1 — 20 receipted rows + 3 combination fixes, o…