Skip to content

falsification_tests asserts wall-clock speedup/throughput inside a required check — debug build on a contended runner #3359

Closed

Description

@noahgift

crates/aprender-serve/tests/falsification_tests.rs asserts SIMD-vs-scalar speedup and a tok/s floor. Both are wall-clock assertions, and they run inside a required check.

Measured

aprender#3114, run 35089783289, job 104773107023, step "Quick tier: lib + integration tests of the selected crates (BSE-17)":

  • the job printed quick tier crates: apr-cli aprender-serve;
  • that step runs cargo nextest run --profile ci --lib --tests$pkgs — --tests arms every integration target of the selected crates, so this file is dark on main and lights up only when crate selection happens to pick aprender-serve;
  • there is no --release, while the file's own header says "Run with: cargo test --test falsification_tests --release -- --nocapture". A scalar-vs-SIMD speedup assertion compiled in debug compares two unoptimised implementations;
  • readings inside that single job: size=512 scalar=6.657µs simd=16.776µs speedup=0.40x, and elsewhere size=128 speedup=6.41x. That spread is contention plus debug codegen, not a SIMD regression.

git show origin/main:...falsification_tests.rs | sha256sum equals the same file at #3114's head (da20e1137147deef), so this is a main-level defect. #3114 merely selected the crate that arms it.

Why it matters

This is the fifth instance of "no wall-clock assertion in a required check" in this repo — the ci.yml comment eight lines above that very step already cites the rule.

Fix

Keep the hypotheses and keep printing the numbers; the timing claims must not fail a required check, and must still be enforced where they are meaningful. #[ignore] is banned here for flakes. Numerical/behavioural assertions in the same file stay exactly as strict.

Also note falsify_h5_minimum_throughput (">= 1.0 tok/s on TinyLlama-1.1B Q4_0") is a second throughput assertion in the same file.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions