Skip to content

aprender-verify-ml: test_pipeline_generate_small asserts wall-clock ms > 0 — fails on a fast host (stopped the v0.69.0 clean-room on gx10) #3702

Closed as not planned

Description

@noahgift

Found as first responder on the v0.69.0 clean-room run (paiml/infra run 35608872676, job clean-room (aprender), runner gx10-pool1, step "Run clean-room gates"). GATE B2 FAILED (534s): unit tests failed stopped the release autopilot at T-3:

aprender-verify-ml --lib: test result: FAILED. 1144 passed; 1 failed
data::pipeline::tests::test_pipeline_generate_small panicked at crates/aprender-verify-ml/src/data/pipeline.rs:438:9:
assertion failed: stats.generation_time_ms > 0 || stats.total_generated < 10

A wall-clock assertion. generation_time_ms is elapsed.as_millis() as u64 (pipeline.rs:236), and the test asks for count(10). A host fast enough to generate 10 programs in under 1 ms reads 0 ms, so the assertion becomes 0 > 0 || 10 < 10, which is false. It passes on slower runners, which is why PR CI and the merge group were green on the same commit. The code has been unchanged since the 2026-04-06 subtree merge (dd0e464). What is new is a fast gx10 runner executing this gate. A required gate may not assert wall-clock time.

The same truncation has a second symptom: PipelineStats::throughput() returns 0.0 when generation_time_ms == 0, so a fast host reports 0 programs/s.

done_when

  • test_pipeline_generate_small asserts no wall-clock quantity. It asserts what is deterministic: stats.total_generated == programs.len(), which holds by construction.
  • The other timing assertions in the crate are checked for the same truncation: evaluator.rs:637 (as_secs_f64() * 1000.0, not truncated) and audit.rs:840 (as_nanos()). Each is fixed or listed as safe, with the reason.
  • The throughput() zero is recorded. Fixing it changes the public u64 ms field, so it is either done with a non-breaking addition or filed separately.

Refs #3698 (the 0.69.0 release)

Activity

  1. added this to the 0.70.0 milestone on Sep 21, 2026
  2. noahgift commented on Sep 21, 2026

    @noahgift
    MemberAuthor

    Duplicate of #3703, which the cop filed as the owned P0 row (handed to infra-c8). My additional analysis (the throughput() zero, why the evaluator/audit assertions are safe, and a clock-free replacement assertion) is posted there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions