You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
falsification_tests asserts wall-clock speedup/throughput inside a required check — debug build on a contended runner #3359
crates/aprender-serve/tests/falsification_tests.rs asserts SIMD-vs-scalar speedup and a tok/s floor. Both are wall-clock assertions, and they run inside a required check.
Measured
aprender#3114, run 35089783289, job 104773107023, step "Quick tier: lib + integration tests of the selected crates (BSE-17)":
the job printed quick tier crates: apr-cli aprender-serve;
that step runs cargo nextest run --profile ci --lib --tests$pkgs — --tests arms every integration target of the selected crates, so this file is dark on main and lights up only when crate selection happens to pick aprender-serve;
there is no --release, while the file's own header says "Run with: cargo test --test falsification_tests --release -- --nocapture". A scalar-vs-SIMD speedup assertion compiled in debug compares two unoptimised implementations;
readings inside that single job: size=512 scalar=6.657µs simd=16.776µs speedup=0.40x, and elsewhere size=128 speedup=6.41x. That spread is contention plus debug codegen, not a SIMD regression.
git show origin/main:...falsification_tests.rs | sha256sum equals the same file at #3114's head (da20e1137147deef), so this is a main-level defect. #3114 merely selected the crate that arms it.
Why it matters
This is the fifth instance of "no wall-clock assertion in a required check" in this repo — the ci.yml comment eight lines above that very step already cites the rule.
Fix
Keep the hypotheses and keep printing the numbers; the timing claims must not fail a required check, and must still be enforced where they are meaningful. #[ignore] is banned here for flakes. Numerical/behavioural assertions in the same file stay exactly as strict.
Also note falsify_h5_minimum_throughput (">= 1.0 tok/s on TinyLlama-1.1B Q4_0") is a second throughput assertion in the same file.
Activity