Skip to content

Tournament mode: a 1v1 bracket over a breaking-point profile #1184

Description

@nigrosimone

The idea

A 1v1 knockout bracket. Two entries per match, the load goes up until the server breaks, whoever holds more traffic goes through, and the last one standing is the champion of the round.

I like the presentation a lot and I think the metric under it is genuinely missing from the arena today: every profile here measures a server that is coping, and none measures what happens when it stops coping. But I think the bracket and the measurement have to be two separate things, for the reasons below.

The bracket cannot be the measurement

In a match the two servers are not really fighting each other: each one is measured on its own against the load generator, and the numbers are then compared. So a match is a comparison of two independent scalars, which means the whole tournament is a sort over a column that could just be printed as a table.

That would be harmless if it were only theatre, but single elimination over noisy numbers is worse than harmless: overload behaviour is much less repeatable than steady-state throughput, so one bad match knocks out an entry that the table says is second. A champion who is not the top of the list is a bug report, not a story.

So: measure once per entry, like every other profile, then draw the bracket from that number with proper seeding (1 vs 32, 2 vs 31 and so on) so the top seeds meet at the end. The tournament page becomes a view over data you already have, it costs no extra runs, and it can never contradict the ranking.

What "breaking" means, and why the current generators cannot find it

With wrk, gcannon and h2load the client holds a fixed number of connections and only sends the next request after it gets a response. The server is therefore never asked for more than it can do. It does not break, it just gets slower and the client absorbs the delay. Ramping connections mostly ramps queueing.

To have a breaking point you need open-loop load: a fixed arrival rate that does not care whether the server is keeping up. That is a new tool, wrk2, vegeta, oha -q or k6, none of which are in scripts/lib/tools/ today.

The failure condition has to be written down and be the same for everybody. I would take the first of:

  • error rate above 0.1% over the step, counting connection resets, timeouts and 5xx
  • p99 above a fixed budget, say 1s
  • throughput lower than the previous step, which is collapse and not a plateau
  • the container dies or gets OOM killed

The result is the last rate that passed, plus which of the four conditions stopped it. How an entry dies is the interesting half: a clean 503 with stable latency is a very different answer from resets, and from a p99 that goes to five seconds while the throughput still looks fine.

Do not measure the machine by accident

At high enough rates the limit stops being the framework: ephemeral ports, somaxconn, file descriptors, conntrack, and the load generator itself all have a ceiling. Two things help:

  • run the same ramp against a reference that is known to be faster than everything (pico, or nginx returning a fixed byte) and record where the harness itself tops out. Any entry that reaches that ceiling is reported as "did not break", not as a winner. The board already has the idea of a reference-only profile
  • pin the generator to its own cores and publish the kernel limits used for the ramp, the way /docs/hardware/ publishes the rest

Cost

A ramp is not a 20s run: ten steps of 20s plus warmup and a restart when the server dies is roughly 5 minutes per entry. About 8 hours for the H/1.1 field, more than a day for everything. It does not have to be all of it at once, it can start as a reference profile on the top 16 of one family and grow.

Sketch

  • a new profile in CATALOG, scored=False at first so it cannot move the composite until it is proven stable, exactly like Pipelined
  • a ramp driver in scripts/lib/tools/ around a rate-capable generator, geometric steps (1.25x), restart between steps only when the container died
  • new fields in the result JSON: break_rate, last_ok_rate, break_reason, errors_at_break
  • a /tournament/ page generated from that scalar, seeded, static
  • run it once per round, not on every deploy

The part that is actually a duel

If you want a real head to head, it exists but it is a different experiment: both servers running at the same time on the same cores, fighting for the same CPU and memory bandwidth, and you see who degrades better. That is a genuine pairwise measurement and it would be great to watch for the last four.

It also breaks the rule that every entry gets dedicated cores, and the result depends on the opponent, so it is not comparable with anything else on the board. I would run it as an exhibition for the final, clearly marked as not scored, and keep it out of the data the ranking is built from.

Metadata

Metadata

Assignees

No one assigned

    Labels

    metaHow arena operates in general

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions