Repository navigation
docs(bench): a ratio of timings is not a ratio of counts, and only one reproduces - #65
Merged
Merged
Conversation
…e reproduces The fanout table quotes 45.9x at 16 subscribers. Three runs today measured 39.9x, 43.6x and 43.9x, at 777k, 870k and 875k updates/sec against the tabulated 899k. Nothing regressed - it is a throughput, and a throughput moves with load. The two columns in that table are different kinds of number and the doc did not say so. The sync column is structural: pinned near 20,000 updates/sec at every fan-out size because that is exactly 1 / 50 microseconds, and it lands there on any machine. The conflating column is a measured throughput. Their quotient inherits the instability of the second, so quoting it to three significant figures implies a stability it does not have. This matters beyond one table. An audit of both repos this week found that every figure which survived unchanged was a ratio of counts or bytes - triangles merged, bytes resident, allocations per message - and the voxel engine's are now known to be byte-identical between an arm64 M4 and an x86-64 CI runner. Every figure that turned out wrong was a timing or a ratio of timings. "Gate ratios" was too coarse a rule; the operative distinction is what the ratio is over. The matching engine's 2.1x survives this unchanged, which is the useful control: five runs today measured 2.14 to 2.22x, so the documented figure was already the right shape. It is quoted to two significant figures rather than three, which is the whole difference.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fanout table quotes 45.9x at 16 subscribers. Three runs today measured 39.9x, 43.6x, 43.9x — at 777k, 870k, 875k updates/sec against the tabulated 899k.
Nothing regressed. It's a throughput, and a throughput moves with load.
The two columns are different kinds of number
The doc didn't say so:
Their quotient inherits the instability of the second, so quoting it to three significant figures implies a stability it doesn't have.
Why this matters past one table
The audit this week found every figure that survived unchanged was a ratio of counts or bytes — triangles merged, bytes resident, allocations per message. The voxel engine's are now known to be byte-identical between an arm64 M4 and an x86-64 CI runner.
Every figure that turned out wrong was a timing, or a ratio of timings.
So "gate ratios" was too coarse. The operative distinction is what the ratio is over.
The control
The matching engine's 2.1x survives this unchanged — five runs today measured 2.14 to 2.22x. The documented figure was already the right shape.
It's quoted to two significant figures rather than three. That's the whole difference: significant figures are themselves a claim about stability, and one extra digit is the cheapest way to overstate a result without noticing.