Repository navigation
feat(ci): ship the one benchmark input that was not committed, and guard the rest - #64
Merged
Merged
Conversation
…ard the rest docs/bench/allocator.md's live row - parse 2.0 allocs/msg at 574 B, books 0.9 - was measured on captures/live-poly.feedlog. /captures/ is gitignored as "large, regenerated". The file has never been in the repo. So that row worked perfectly for anyone with this working tree and could not be reproduced by anyone else. Every command in the doc runs, the numbers come out exactly as printed, and none of it is checkable from a clone. It is the only figure in docs/bench/ in that state, against a README rule that a figure is either measured on a committed capture or labelled as not yet measured. It was neither. The capture is 85 KB gzipped and now sits beside the other six in docs/bench/, with the gunzip-and-replay command in the doc. It reproduces the row exactly: 877 records, parse 2.0/msg 574 B/msg, books 0.9/msg. scripts/check_bench_inputs.py runs in CI and fails if any doc under docs/bench/ names a .feedlog that git does not track. Angle-bracket metavariables are excluded, since `basis xvenue-lead <capture.feedlog>` names an argument rather than a file - the first version of the check flagged exactly that and was wrong to. This class of defect cannot be caught by running the benchmark. Running it is what makes it look fine. The only question that finds it is whether the input ships, which is why this is a script and not a habit. Audited the other allocator figures against fresh runs while here, and they are exact: synthetic 724,298 records at parse 1.0/msg, 56 B/msg, books 0.5/msg.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/bench/allocator.md's live row — parse 2.0 allocs/msg at 574 B, books 0.9 — was measured oncaptures/live-poly.feedlog./captures/is gitignored as "large, regenerated". The file has never been in the repo.So that row worked perfectly for anyone with this working tree and could not be reproduced by anyone else. Every command in the doc runs. The numbers come out exactly as printed. None of it is checkable from a clone.
It's the only figure in
docs/bench/in that state, against a README rule that a figure is either measured on a committed capture or labelled as not yet measured. It was neither.Fix
The capture is 85 KB gzipped and now sits beside the other six, with the gunzip-and-replay command in the doc. Reproduces the row exactly:
Guard
scripts/check_bench_inputs.pyin CI: fails if any doc underdocs/bench/names a.feedlogthat git doesn't track.Angle-bracket metavariables are excluded —
basis xvenue-lead <capture.feedlog>names an argument, not a file. The first version of the check flagged exactly that and was wrong to.This class of defect can't be caught by running the benchmark. Running it is what makes it look fine. The only question that finds it is whether the input ships — which is why it's a script and not a habit.
Also audited
The other allocator figures, re-measured today, are exact: synthetic 724,298 records at parse 1.0/msg, 56 B/msg, books 0.5/msg.
228 tests pass.