The idea
A 1v1 knockout bracket. Two entries per match, the load goes up until the server breaks, whoever holds more traffic goes through, and the last one standing is the champion of the round.
I like the presentation a lot and I think the metric under it is genuinely missing from the arena today: every profile here measures a server that is coping, and none measures what happens when it stops coping. But I think the bracket and the measurement have to be two separate things, for the reasons below.
The bracket cannot be the measurement
In a match the two servers are not really fighting each other: each one is measured on its own against the load generator, and the numbers are then compared. So a match is a comparison of two independent scalars, which means the whole tournament is a sort over a column that could just be printed as a table.
That would be harmless if it were only theatre, but single elimination over noisy numbers is worse than harmless: overload behaviour is much less repeatable than steady-state throughput, so one bad match knocks out an entry that the table says is second. A champion who is not the top of the list is a bug report, not a story.
So: measure once per entry, like every other profile, then draw the bracket from that number with proper seeding (1 vs 32, 2 vs 31 and so on) so the top seeds meet at the end. The tournament page becomes a view over data you already have, it costs no extra runs, and it can never contradict the ranking.
What "breaking" means, and why the current generators cannot find it
With wrk, gcannon and h2load the client holds a fixed number of connections and only sends the next request after it gets a response. The server is therefore never asked for more than it can do. It does not break, it just gets slower and the client absorbs the delay. Ramping connections mostly ramps queueing.
To have a breaking point you need open-loop load: a fixed arrival rate that does not care whether the server is keeping up. That is a new tool, wrk2, vegeta, oha -q or k6, none of which are in scripts/lib/tools/ today.
The failure condition has to be written down and be the same for everybody. I would take the first of:
- error rate above 0.1% over the step, counting connection resets, timeouts and 5xx
- p99 above a fixed budget, say 1s
- throughput lower than the previous step, which is collapse and not a plateau
- the container dies or gets OOM killed
The result is the last rate that passed, plus which of the four conditions stopped it. How an entry dies is the interesting half: a clean 503 with stable latency is a very different answer from resets, and from a p99 that goes to five seconds while the throughput still looks fine.
Do not measure the machine by accident
At high enough rates the limit stops being the framework: ephemeral ports, somaxconn, file descriptors, conntrack, and the load generator itself all have a ceiling. Two things help:
- run the same ramp against a reference that is known to be faster than everything (pico, or nginx returning a fixed byte) and record where the harness itself tops out. Any entry that reaches that ceiling is reported as "did not break", not as a winner. The board already has the idea of a reference-only profile
- pin the generator to its own cores and publish the kernel limits used for the ramp, the way
/docs/hardware/ publishes the rest
Cost
A ramp is not a 20s run: ten steps of 20s plus warmup and a restart when the server dies is roughly 5 minutes per entry. About 8 hours for the H/1.1 field, more than a day for everything. It does not have to be all of it at once, it can start as a reference profile on the top 16 of one family and grow.
Sketch
- a new profile in
CATALOG, scored=False at first so it cannot move the composite until it is proven stable, exactly like Pipelined
- a ramp driver in
scripts/lib/tools/ around a rate-capable generator, geometric steps (1.25x), restart between steps only when the container died
- new fields in the result JSON:
break_rate, last_ok_rate, break_reason, errors_at_break
- a
/tournament/ page generated from that scalar, seeded, static
- run it once per round, not on every deploy
The part that is actually a duel
If you want a real head to head, it exists but it is a different experiment: both servers running at the same time on the same cores, fighting for the same CPU and memory bandwidth, and you see who degrades better. That is a genuine pairwise measurement and it would be great to watch for the last four.
It also breaks the rule that every entry gets dedicated cores, and the result depends on the opponent, so it is not comparable with anything else on the board. I would run it as an exhibition for the final, clearly marked as not scored, and keep it out of the data the ranking is built from.
The idea
A 1v1 knockout bracket. Two entries per match, the load goes up until the server breaks, whoever holds more traffic goes through, and the last one standing is the champion of the round.
I like the presentation a lot and I think the metric under it is genuinely missing from the arena today: every profile here measures a server that is coping, and none measures what happens when it stops coping. But I think the bracket and the measurement have to be two separate things, for the reasons below.
The bracket cannot be the measurement
In a match the two servers are not really fighting each other: each one is measured on its own against the load generator, and the numbers are then compared. So a match is a comparison of two independent scalars, which means the whole tournament is a sort over a column that could just be printed as a table.
That would be harmless if it were only theatre, but single elimination over noisy numbers is worse than harmless: overload behaviour is much less repeatable than steady-state throughput, so one bad match knocks out an entry that the table says is second. A champion who is not the top of the list is a bug report, not a story.
So: measure once per entry, like every other profile, then draw the bracket from that number with proper seeding (1 vs 32, 2 vs 31 and so on) so the top seeds meet at the end. The tournament page becomes a view over data you already have, it costs no extra runs, and it can never contradict the ranking.
What "breaking" means, and why the current generators cannot find it
With wrk, gcannon and h2load the client holds a fixed number of connections and only sends the next request after it gets a response. The server is therefore never asked for more than it can do. It does not break, it just gets slower and the client absorbs the delay. Ramping connections mostly ramps queueing.
To have a breaking point you need open-loop load: a fixed arrival rate that does not care whether the server is keeping up. That is a new tool,
wrk2,vegeta,oha -qork6, none of which are inscripts/lib/tools/today.The failure condition has to be written down and be the same for everybody. I would take the first of:
The result is the last rate that passed, plus which of the four conditions stopped it. How an entry dies is the interesting half: a clean 503 with stable latency is a very different answer from resets, and from a p99 that goes to five seconds while the throughput still looks fine.
Do not measure the machine by accident
At high enough rates the limit stops being the framework: ephemeral ports,
somaxconn, file descriptors, conntrack, and the load generator itself all have a ceiling. Two things help:/docs/hardware/publishes the restCost
A ramp is not a 20s run: ten steps of 20s plus warmup and a restart when the server dies is roughly 5 minutes per entry. About 8 hours for the H/1.1 field, more than a day for everything. It does not have to be all of it at once, it can start as a reference profile on the top 16 of one family and grow.
Sketch
CATALOG,scored=Falseat first so it cannot move the composite until it is proven stable, exactly like Pipelinedscripts/lib/tools/around a rate-capable generator, geometric steps (1.25x), restart between steps only when the container diedbreak_rate,last_ok_rate,break_reason,errors_at_break/tournament/page generated from that scalar, seeded, staticThe part that is actually a duel
If you want a real head to head, it exists but it is a different experiment: both servers running at the same time on the same cores, fighting for the same CPU and memory bandwidth, and you see who degrades better. That is a genuine pairwise measurement and it would be great to watch for the last four.
It also breaks the rule that every entry gets dedicated cores, and the result depends on the opponent, so it is not comparable with anything else on the board. I would run it as an exhibition for the final, clearly marked as not scored, and keep it out of the data the ranking is built from.