feat(ci): gate the coordinated-omission invariant, which nothing checked - #62
Merged
Merged
Conversation
docs/bench/latency.md's headline is that the published p99 was a service time and 13.3x optimistic - 115.7 us service against 1,533.9 us response in the same run. Nothing in CI checked any part of it. The perf gate has twenty checks and not one of them touched a latency figure. What is gated is the structural relationship, not the microseconds. Absolute latency on a shared runner is noise and a tight bound there would be flaky, so the gate asserts what cannot vary by machine: response time is service time plus a wait, a wait cannot be negative, therefore response p99 >= service p99. That holds everywhere. If it ever fails the harness is reading the wrong clock, and every number in latency.md is suspect rather than merely stale. A 100-step synthetic session paced at 1x costs about ten seconds of wall time, which is what a paced run costs by definition - it is supposed to wait. Locally it separates cleanly: 82 us service against 2,964 us response. Two tests pin the JSON the gate reads: the paced block carries response_us and pacing when a speed was given, and must be absent otherwise, since a gate reading a response time off an unpaced run would be comparing a measurement against itself. Validated by breaking it both ways. Inverting the comparison fails with "response >= service: 0 violates v == 1"; renaming the key fails with "value missing from replay output" rather than killing the script, which is the `|| true` fix from the previous gate doing its job.
`mktemp -t basis_pace` works on BSD, where -t takes a bare prefix. GNU
mktemp requires XXXXXX in the template and fails without it, so on the
ubuntu perf-gate runner the assignment errored and `set -euo pipefail`
killed the script before any of the three new checks ran.
The failure was almost invisible in the log: every earlier check printed
"gate ok", nothing printed after them, and the job ended with exit code 1.
It reads like the gate passing and then failing for an unrelated reason.
Template is now "${TMPDIR:-/tmp}/basis_pace.XXXXXX", which both accept.
Same shape as the libc++ / libstdc++ include problem two weeks ago: "it
works here" is a claim about one platform, and the Linux job is the only
thing standing between that claim and a broken build.
DanielWLiu07
deleted the
feat/gate-the-coordinated-omission-invariant
branch
September 1, 2026 01:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/bench/latency.md's headline is that the published p99 was a service time and 13.3x optimistic — 115.7 µs service against 1,533.9 µs response in the same run.Nothing in CI checked any part of it. The perf gate has twenty checks and not one touched a latency figure.
What's gated: the relationship, not the microseconds
Absolute latency on a shared runner is noise, and a tight bound there would be flaky. So the gate asserts what cannot vary by machine:
That holds on every machine that has ever existed. If it ever fails, the harness is reading the wrong clock and every number in
latency.mdis suspect rather than merely stale.A 100-step synthetic session paced at 1x costs ~10s of wall time — which is what a paced run costs by definition, since it's supposed to wait. Locally it separates cleanly:
Two tests pin the JSON the gate reads
The paced block carries
response_usandpacingwhen a speed was given — and must be absent otherwise, since a gate reading a response time off an unpaced run would be comparing a measurement against itself.Validated by breaking it, both ways
GATE FAIL response >= service: 0 violates v == 1GATE FAIL response >= service: value missing from replay outputThat second one is the
|| truefix from PR #57 doing its job — before it, a renamed key killed the script with no diagnostic.228 tests pass.