diff --git a/README.md b/README.md index 0eef34d..b307ef2 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ what exists is one vertical slice and it is not finished: | Lifting CIR calls into the dialect | works on real compiler output, with or without a hidden result argument; tested end to end from C++ | | Queue ownership analysis: SPSC / MPSC / SPMC / MPMC | prints a verdict; sound for direct calls, UNKNOWN wherever a function's address escapes; thread lifetime and escaping queues not modelled | | Lowering back to CIR | faithful round trip, sret included; an op missing what it needs is a diagnostic | -| Specialised implementations | an unbounded SPSC queue, substituted for a global `wtc::queue` proven SPSC; 3.6x throughput on one producer and one consumer (`benchmarks/`) | +| Specialised implementations | an unbounded SPSC queue, substituted for a global `wtc::queue` proven SPSC; 2.4 to 3x throughput and a much lower tail latency (`benchmarks/RESULTS.md`) | `docs/audit.md` lists the known defects in full, fixed and open. The analysis verdict is now safe to substitute on for the cases it claims; what it does not diff --git a/benchmarks/README.md b/benchmarks/README.md index fac59e4..86880d8 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -14,36 +14,83 @@ for, and the hand-written specialised implementation from `runtime/`. The second one is the ceiling: the compiler cannot beat code a human wrote for this exact case, it can only reach it without the human. -## What is here - -- `queue_bench.cpp`: one producer and one consumer move 5M ints through a - global `wtc::queue`. The consumer checks the count, the sum and the - order, and the run fails if any is wrong. Nothing in it names the - substitute: the shim makes `wtc::runtime::spsc_queue` available. -- `spsc_direct.cpp`: the same workload written by hand against - `runtime/spsc_queue.hpp`, with its allocation-free pop. This is the ceiling. -- `run.sh`: builds `queue_bench.cpp` twice from the same CIR, once as - written and once with `--select-queue-impl` in the pipeline, plus - `spsc_direct`, through the pinned toolchain down to a native binary, and - reports the median of `RUNS` runs of each. Set `LLVM_BUILD` to the pinned - toolchain (default `/opt/llvm`, as in the image) and `WTC_BUILD` to this - project's build tree. - -Only throughput is measured so far, not the tail, and runs are not pinned. - -## Measured - -At the pin (`1ebf89d8c5e6`), 2026-10-18, on an Apple M1 Pro (8 cores, -16 GB), macOS, threads not pinned, five runs each, median: - -| variant | items/s | vs baseline | -| --- | ---: | ---: | -| baseline: `wtc::queue` | 2.07M | 1.0x | -| optimized: substituted by `--select-queue-impl` | 7.44M | 3.6x | -| direct: `spsc_queue` by hand | 43.1M | 20.9x | - -The gap between optimized and direct is almost all one allocation per item. -The substitute has to keep `wtc::queue::try_pop`'s interface, which -returns a `std::shared_ptr`, so every pop builds one; the hand-written -consumer uses `try_pop_optional` and allocates nothing. Closing it means -changing what the dialect's pop returns, not the queue. +## Running it + +Not in CI: shared runners are too noisy for timings to mean anything. Run it +by hand, on a quiet machine, against the pinned toolchain: + +```sh +LLVM_BUILD= WTC_BUILD= benchmarks/run.sh +``` + +It takes about two minutes at the default `RUNS=5` on an M1 Pro and +overwrites [`RESULTS.md`](RESULTS.md): medians, latency percentiles, the +spread over runs, and the commit, toolchain and machine they came from. To +keep a result, commit `RESULTS.md` on its own after the run, so the commit +it names is the one that was measured. `RESULTS` writes the report +elsewhere; `BENCH_CPUS` pins every run to those CPUs with `taskset` on +Linux. + +## How it works + +Each workload in `workloads/` is one source built in up to three variants: + +- **baseline**: `wtc::queue`, lifted and lowered back unchanged, so it + goes through the same pipeline as optimized and differs only in the + substitution; +- **optimized**: the same, with `--select-queue-impl`; +- **direct**: `wtc::runtime::spsc_queue` used by hand, with its + allocation-free pop (`-DWTC_BENCH_DIRECT`). The ceiling. + +`run.sh` holds, for each workload, whether the optimized build must be +substituted, and fails if it was not, or was when it must not be. A number +in the results therefore always measures what its column says. Every +workload also checks what it received, and fails rather than reports if +anything was lost, duplicated or reordered. Variants run interleaved so +that drift over a session affects them alike. + +The queue in each workload is a global touched only by `push` and +`BENCH_POP` at the call sites (`bench.hpp`). Passing it to a helper would be +a use `--select-queue-impl` cannot retarget, and the pass would rightly +refuse to substitute. + +| Workload | What it measures | +| --- | --- | +| `spsc_throughput` | one producer, one consumer, 5M ints, flat out | +| `spsc_latency` | push-to-pop latency at a paced 1M items/s: p50, p99, p99.9, max | +| `spsc_payload` | throughput with a 64-byte record as the element type | +| `spsc_same_thread` | one thread pushes a 200k batch and pops it, 10 times: raw cost, no contention, and a case a bounded ring would deadlock on | +| `mpsc_control` | two producers: must not be substituted; measures the noise floor | + +## Reading the results + +`mpsc_control` is the yardstick. Its two builds run the same queue code; +the only difference is that the optimized one has had the shim's anchor +erased, and with it the unused substitute's code. On the M1 Pro they +measured 6, 13 and 18 percent apart in three sessions, and in all three the +optimized build was the faster, whichever ran first. That is a systematic +difference between two binaries, most likely their layout, rather than +noise, but the conclusion is the same: a difference of that size between +any two columns is not evidence that the queue made it. `RESULTS.md` states +the figure for its own session. + +Tail latency is noisier still. On the M1 Pro, one run in five can be +disturbed by something else on the machine, moving every variant at once, +and even undisturbed runs put the substituted queue's p99 anywhere from +about 2 to 25 µs. The medians are robust to one bad run; `RESULTS.md` also +lists each run's p99, so it can be seen rather than assumed. + +The gap between optimized and direct is mostly one allocation per item. The +substitute keeps `wtc::queue::try_pop`'s interface, which returns a +`std::shared_ptr`, so every pop builds one; the hand-written consumer +uses `try_pop_optional` and allocates nothing. Closing it means changing +what the dialect's pop returns, not the queue. + +## Not measured yet + +- Pinned runs on Linux. On macOS threads cannot be pinned, so latency + includes whatever the scheduler does. +- Latency under bursts or saturation; `spsc_latency` is one steady rate. +- Anything with more than one queue, or with queues that are not globals, + since those are not substituted. +- Memory: the substitute allocates in blocks, the original per item. diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md new file mode 100644 index 0000000..f6b83e5 --- /dev/null +++ b/benchmarks/RESULTS.md @@ -0,0 +1,72 @@ +# Benchmark results + +Generated by `benchmarks/run.sh` on 2026-10-18 15:52 UTC. Rerun it to update this file rather than editing it. How the suite works, and what it does not measure yet, is in `benchmarks/README.md`. + +## Environment + +| | | +| --- | --- | +| Commit | `cf0c29aec0f3` on `bench/suite` | +| LLVM pin | `1ebf89d8c5e6` | +| CPU | Apple M1 Pro, 8 cores | +| Memory | 16 GiB | +| OS | macOS 26.5.2 (arm64) | +| Runs | 5 per variant, interleaved; medians reported | +| Threads | not pinned | + +## Throughput + +Median items per second. The ratios are against baseline, the same program with `wtc::queue`. + +| Workload | Optimized build | baseline | optimized | direct | optimized ÷ baseline | direct ÷ baseline | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| `spsc_throughput` | substituted | 2.65M | 8.02M | 43.58M | 3.02x | 16.44x | +| `spsc_payload` | substituted | 2.65M | 7.41M | 28.82M | 2.80x | 10.88x | +| `spsc_same_thread` | substituted | 3.13M | 7.58M | 38.62M | 2.42x | 12.35x | +| `mpsc_control` | kept | 2.03M | 2.40M | — | 1.18x | — | + +**Noise.** `mpsc_control` is not substituted, so its baseline and optimized builds run the same queue code, and they measured 18% apart. A difference of that size or less between any two columns here is not evidence of anything. + +## Latency + +Time from push to pop, per item: the median over runs of each run's percentile. The worst case is the median of each run's maximum. These workloads are paced (see Workloads), so they are not in the throughput tables. + +| Workload | Variant | p50 | p99 | p99.9 | max | +| --- | --- | ---: | ---: | ---: | ---: | +| `spsc_latency` | baseline | 2.62 µs | 992.04 µs | 1.50 ms | 1.81 ms | +| `spsc_latency` | optimized | 208 ns | 4.38 µs | 133.04 µs | 415.42 µs | +| `spsc_latency` | direct | 166 ns | 1.46 µs | 87.79 µs | 352.83 µs | + +Tail latency varies far more from run to run than throughput does, and one disturbed run moves every variant at once. The p99 of each run: + +| Workload | Variant | run 1 | run 2 | run 3 | run 4 | run 5 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| `spsc_latency` | baseline | 992.04 µs | 1.60 ms | 565.04 µs | 14.94 ms | 621.25 µs | +| `spsc_latency` | optimized | 5.25 µs | 23.04 µs | 3.96 µs | 1.79 µs | 4.38 µs | +| `spsc_latency` | direct | 2.42 µs | 32.46 µs | 917 ns | 708 ns | 1.46 µs | + +## Spread + +Throughput over all runs, slowest to fastest, items per second. + +| Workload | Variant | min | median | max | +| --- | --- | ---: | ---: | ---: | +| `spsc_throughput` | baseline | 2.60M | 2.65M | 2.68M | +| `spsc_throughput` | optimized | 7.89M | 8.02M | 8.07M | +| `spsc_throughput` | direct | 39.12M | 43.58M | 45.28M | +| `spsc_payload` | baseline | 2.40M | 2.65M | 2.73M | +| `spsc_payload` | optimized | 6.89M | 7.41M | 7.47M | +| `spsc_payload` | direct | 26.18M | 28.82M | 31.78M | +| `spsc_same_thread` | baseline | 3.07M | 3.13M | 3.14M | +| `spsc_same_thread` | optimized | 7.17M | 7.58M | 7.70M | +| `spsc_same_thread` | direct | 32.22M | 38.62M | 39.11M | +| `mpsc_control` | baseline | 1.83M | 2.03M | 2.11M | +| `mpsc_control` | optimized | 2.25M | 2.40M | 2.42M | + +## Workloads + +- **`spsc_throughput`**: One producer, one consumer, as fast as they can go: kItems ints through one queue. The consumer spins on an empty queue. Measures throughput under saturation; latency here would only measure how far behind the consumer falls, so none is reported. +- **`spsc_latency`**: One producer, one consumer, at a fixed rate well below either queue's capacity: one item every kIntervalNs. Each item is the time it was pushed, so the consumer can measure how long it waited in the queue. Reports the latency distribution, the number the project's motivation rests on: p50, p99, p99.9 and the worst case. +- **`spsc_payload`**: spsc_throughput with a 64-byte record instead of an int, so each push and pop copies a cache line. Shows whether the substitution's advantage holds when the payload, not the synchronisation, starts to dominate, and that it applies to a user-defined element type at all. +- **`spsc_same_thread`**: One thread is both the producer and the consumer: it pushes a batch, then pops it all, kRounds times. The verdict is still SPSC (one thread pushes, one thread pops; they happen to be the same), and this is the case a bounded ring cannot take: with a batch larger than the ring, a push that waited for room would wait forever. The substitute is unbounded, so it must complete. With no second thread there is no contention at all, so this measures the queues' raw cost per operation. +- **`mpsc_control`**: The control: two producers, one consumer. The verdict is MPSC, so --select-queue-impl must leave the queue alone, and run.sh fails if it does not. Baseline and optimized should then be the same program and measure the same, which bounds how much of the other workloads' difference is noise. There is no direct variant: spsc_queue is not safe with two producers. diff --git a/benchmarks/bench.hpp b/benchmarks/bench.hpp new file mode 100644 index 0000000..629f480 --- /dev/null +++ b/benchmarks/bench.hpp @@ -0,0 +1,81 @@ +#pragma once + +// Shared by every workload in benchmarks/workloads/. +// +// Each workload is one source compiled two ways. Without WTC_BENCH_DIRECT +// it uses wtc::queue, and run.sh builds it as written (baseline) and +// through --select-queue-impl (optimized). With WTC_BENCH_DIRECT it uses +// wtc::runtime::spsc_queue directly and pops without allocating: the +// hand-written ceiling. +// +// The queue is a global, touched only by push and by BENCH_POP at the call +// sites. That is not style: --select-queue-impl substitutes a global only +// when it can retarget every use, and passing the queue to a helper is a +// use it cannot. A workload written with helpers would silently measure +// nothing. +// +// Every workload checks what it received and exits non-zero if anything +// was lost, duplicated or reordered: a fast wrong queue is not a result. +// It then prints one RESULT line, which run.sh collects. + +#include +#include +#include +#include +#include +#include + +#ifdef WTC_BENCH_DIRECT +#include "../runtime/spsc_queue.hpp" +template +using bench_queue = wtc::runtime::spsc_queue; +#define BENCH_POP(q) (q).try_pop_optional() +#else +#include "queue.hpp" +template +using bench_queue = wtc::queue; +#define BENCH_POP(q) (q).try_pop() +#endif + +namespace bench { + +inline std::int64_t now_ns() { + return std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count(); +} + +[[noreturn]] inline void fail(const char *workload, const char *what, long long detail) { + std::fprintf(stderr, "FAIL %s: %s (%lld)\n", workload, what, detail); + std::exit(1); +} + +// The value at quantile q of sorted, by the nearest-rank method. +inline std::int64_t quantile(const std::vector &sorted, double q) { + if (sorted.empty()) + return 0; + std::size_t rank = static_cast(q * static_cast(sorted.size())); + if (rank >= sorted.size()) + rank = sorted.size() - 1; + return sorted[rank]; +} + +// One line, key=value pairs, for run.sh to collect. Latency fields appear +// only for workloads that measure latency. +inline void report(const char *workload, long items, std::int64_t elapsed_ns, + std::vector *latencies = nullptr) { + double seconds = static_cast(elapsed_ns) / 1e9; + std::printf("RESULT workload=%s items=%ld elapsed_ns=%lld items_per_sec=%.0f", + workload, items, static_cast(elapsed_ns), + static_cast(items) / seconds); + if (latencies) { + std::sort(latencies->begin(), latencies->end()); + std::printf(" p50_ns=%lld p99_ns=%lld p999_ns=%lld max_ns=%lld", + static_cast(quantile(*latencies, 0.50)), + static_cast(quantile(*latencies, 0.99)), + static_cast(quantile(*latencies, 0.999)), + static_cast(latencies->back())); + } + std::printf("\n"); +} + +} diff --git a/benchmarks/queue_bench.cpp b/benchmarks/queue_bench.cpp deleted file mode 100644 index 12d437b..0000000 --- a/benchmarks/queue_bench.cpp +++ /dev/null @@ -1,66 +0,0 @@ -// One producer, one consumer, kNumItems ints through a global -// wtc::queue. Built twice from this one source by run.sh: once with the -// queue as written, once after --select-queue-impl has substituted -// wtc::runtime::spsc_queue for it. Nothing here names the substitute; -// the shim makes it available. -// -// The consumer checks what it receives: a substitution that drops, -// duplicates or reorders an item fails the run instead of timing it. - -#include "queue.hpp" -#include "thread.hpp" - -#include -#include - -constexpr long kNumItems = 5'000'000; - -wtc::queue g_queue; - -long g_received = 0; -long g_sum = 0; -long g_out_of_order = 0; - -void producer() { - for (long i = 0; i < kNumItems; ++i) { - int value = static_cast(i); - g_queue.push(value); - } -} - -void consumer() { - long expected = 0; - while (g_received < kNumItems) { - auto value = g_queue.try_pop(); - if (value) { - if (*value != expected) - ++g_out_of_order; - g_sum += *value; - ++expected; - ++g_received; - } - } -} - -int main() { - auto start = std::chrono::steady_clock::now(); - - auto producer_thread = wtc::spawn(producer); - auto consumer_thread = wtc::spawn(consumer); - producer_thread.join(); - consumer_thread.join(); - - auto end = std::chrono::steady_clock::now(); - double seconds = std::chrono::duration(end - start).count(); - - long expected_sum = kNumItems * (kNumItems - 1) / 2; - if (g_received != kNumItems || g_sum != expected_sum || g_out_of_order != 0) { - std::printf("FAIL received=%ld sum=%ld (expected %ld) out_of_order=%ld\n", - g_received, g_sum, expected_sum, g_out_of_order); - return 1; - } - - std::printf("items=%ld elapsed_ms=%.3f items_per_sec=%.0f\n", - kNumItems, seconds * 1000.0, static_cast(kNumItems) / seconds); - return 0; -} diff --git a/benchmarks/report.py b/benchmarks/report.py new file mode 100644 index 0000000..1208e6b --- /dev/null +++ b/benchmarks/report.py @@ -0,0 +1,203 @@ +"""Writes benchmarks/RESULTS.md from what run.sh collected. + + report.py + +raw.txt has one line per run: key=value pairs, as each workload prints them, +plus run= and variant=. The report gives the median of each metric over the +runs, the spread of throughput, and the environment the runs came from. +""" + +import pathlib +import statistics +import sys + +VARIANTS = ["baseline", "optimized", "direct"] + + +def parse_pairs(line): + pairs = {} + for token in line.split(): + if "=" in token: + key, value = token.split("=", 1) + pairs[key] = value + return pairs + + +def read_environment(path): + env = {} + for line in pathlib.Path(path).read_text().splitlines(): + if "=" in line: + key, value = line.split("=", 1) + env[key] = value + return env + + +def describe(source): + """The workload's opening comment, as one paragraph.""" + lines = [] + for line in pathlib.Path(source).read_text().splitlines(): + if not line.startswith("//"): + break + text = line[2:].strip() + if not text: + break + lines.append(text) + return " ".join(lines) + + +def rate(items_per_sec): + if items_per_sec >= 1e6: + return f"{items_per_sec / 1e6:.2f}M" + return f"{items_per_sec / 1e3:.0f}k" + + +def duration(ns): + if ns >= 1e6: + return f"{ns / 1e6:.2f} ms" + if ns >= 1e3: + return f"{ns / 1e3:.2f} µs" + return f"{ns:.0f} ns" + + +def main(raw_path, env_path, substitution_path, workloads_dir, out_path): + runs = {} # (workload, variant) -> [pairs] + order = [] + for line in pathlib.Path(raw_path).read_text().splitlines(): + pairs = parse_pairs(line) + key = (pairs["workload"], pairs["variant"]) + if key not in runs: + runs[key] = [] + order.append(pairs["workload"]) + runs[key].append(pairs) + workloads = list(dict.fromkeys(order)) + + substitution = dict( + line.split() for line in pathlib.Path(substitution_path).read_text().splitlines() if line.strip() + ) + env = read_environment(env_path) + + # A workload that measures latency is paced, so its throughput is its + # pacing rate and says nothing about the queue. + paced = [w for w in workloads if any("p50_ns" in r for v in VARIANTS for r in runs.get((w, v), []))] + unpaced = [w for w in workloads if w not in paced] + + def median(workload, variant, metric): + values = [float(r[metric]) for r in runs.get((workload, variant), []) if metric in r] + return statistics.median(values) if values else None + + out = [] + out.append("# Benchmark results") + out.append("") + out.append( + f"Generated by `benchmarks/run.sh` on {env['date']}. Rerun it to update this " + "file rather than editing it. How the suite works, and what it does not " + "measure yet, is in `benchmarks/README.md`." + ) + out.append("") + out.append("## Environment") + out.append("") + out.append("| | |") + out.append("| --- | --- |") + out.append(f"| Commit | `{env['commit']}` on `{env['branch']}` |") + out.append(f"| LLVM pin | `{env['llvm']}` |") + out.append(f"| CPU | {env['cpu']}, {env['cores']} cores |") + out.append(f"| Memory | {env['memory']} |") + out.append(f"| OS | {env['os']} |") + out.append(f"| Runs | {env['runs']} per variant, interleaved; medians reported |") + out.append(f"| Threads | {env['pinning']} |") + out.append("") + + out.append("## Throughput") + out.append("") + out.append( + "Median items per second. The ratios are against baseline, the same " + "program with `wtc::queue`." + ) + out.append("") + out.append("| Workload | Optimized build | baseline | optimized | direct | optimized ÷ baseline | direct ÷ baseline |") + out.append("| --- | --- | ---: | ---: | ---: | ---: | ---: |") + for w in unpaced: + cells = [median(w, v, "items_per_sec") for v in VARIANTS] + base = cells[0] + shown = [rate(c) if c else "—" for c in cells] + ratio_opt = f"{cells[1] / base:.2f}x" if cells[1] and base else "—" + ratio_dir = f"{cells[2] / base:.2f}x" if cells[2] and base else "—" + out.append(f"| `{w}` | {substitution.get(w, '?')} | {' | '.join(shown)} | {ratio_opt} | {ratio_dir} |") + out.append("") + + # A workload the pass must leave alone builds the same queue code twice; + # how far apart its two builds measure is how much difference this + # machine produces on its own. + for w in unpaced: + if substitution.get(w) != "kept": + continue + base = median(w, "baseline", "items_per_sec") + opt = median(w, "optimized", "items_per_sec") + if base and opt: + apart = abs(opt / base - 1) * 100 + out.append( + f"**Noise.** `{w}` is not substituted, so its baseline and optimized builds " + f"run the same queue code, and they measured {apart:.0f}% apart. A difference " + "of that size or less between any two columns here is not evidence of anything." + ) + out.append("") + + latency_workloads = paced + if latency_workloads: + out.append("## Latency") + out.append("") + out.append( + "Time from push to pop, per item: the median over runs of each run's " + "percentile. The worst case is the median of each run's maximum. These " + "workloads are paced (see Workloads), so they are not in the throughput tables." + ) + out.append("") + out.append("| Workload | Variant | p50 | p99 | p99.9 | max |") + out.append("| --- | --- | ---: | ---: | ---: | ---: |") + for w in latency_workloads: + for v in VARIANTS: + if (w, v) not in runs: + continue + cells = [median(w, v, m) for m in ("p50_ns", "p99_ns", "p999_ns", "max_ns")] + out.append(f"| `{w}` | {v} | " + " | ".join(duration(c) for c in cells) + " |") + out.append("") + out.append( + "Tail latency varies far more from run to run than throughput does, and one " + "disturbed run moves every variant at once. The p99 of each run:" + ) + out.append("") + out.append("| Workload | Variant | " + " | ".join(f"run {i}" for i in range(1, int(env["runs"]) + 1)) + " |") + out.append("| --- | --- | " + " | ".join("---:" for _ in range(int(env["runs"]))) + " |") + for w in latency_workloads: + for v in VARIANTS: + if (w, v) not in runs: + continue + by_run = {r["run"]: float(r["p99_ns"]) for r in runs[(w, v)]} + cells = [duration(by_run[str(i)]) if str(i) in by_run else "—" for i in range(1, int(env["runs"]) + 1)] + out.append(f"| `{w}` | {v} | " + " | ".join(cells) + " |") + out.append("") + + out.append("## Spread") + out.append("") + out.append("Throughput over all runs, slowest to fastest, items per second.") + out.append("") + out.append("| Workload | Variant | min | median | max |") + out.append("| --- | --- | ---: | ---: | ---: |") + for w in unpaced: + for v in VARIANTS: + values = sorted(float(r["items_per_sec"]) for r in runs.get((w, v), [])) + if values: + out.append(f"| `{w}` | {v} | {rate(values[0])} | {rate(statistics.median(values))} | {rate(values[-1])} |") + out.append("") + + out.append("## Workloads") + out.append("") + for w in workloads: + out.append(f"- **`{w}`**: {describe(pathlib.Path(workloads_dir) / (w + '.cpp'))}") + out.append("") + + pathlib.Path(out_path).write_text("\n".join(out)) + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/benchmarks/run.sh b/benchmarks/run.sh index 38c5e42..99e545e 100755 --- a/benchmarks/run.sh +++ b/benchmarks/run.sh @@ -1,13 +1,25 @@ #!/usr/bin/env bash -# Builds the queue benchmark two ways from the same source -- baseline -# (wtc::queue, lifted and lowered back unchanged) and optimized (the -# same pipeline plus --select-queue-impl) -- and spsc_direct, the -# hand-written ceiling. Runs each RUNS times and reports the median. +# The benchmark suite. Not run in CI: shared runners are too noisy for +# timings to mean anything. Run it by hand on a quiet machine. +# +# Every workload in workloads/ is built in up to three variants: +# baseline wtc::queue, lifted and lowered back unchanged +# optimized the same, plus --select-queue-impl +# direct wtc::runtime::spsc_queue by hand, the ceiling +# and each workload states whether the optimized build must be +# substituted. A build that disagrees fails the suite, so a number in the +# results always means what its column says. +# +# Variants are run interleaved, RUNS times each, so drift over the session +# affects them all alike. The medians, the spread and the environment are +# written to RESULTS. # # LLVM_BUILD the pinned toolchain, built or installed; default /opt/llvm, # where the toolchain image puts it # WTC_BUILD this project's build tree; default /build # RUNS runs per variant; default 5 +# RESULTS where to write the report; default benchmarks/RESULTS.md +# BENCH_CPUS Linux only: CPUs to pin each run to with taskset, e.g. 2,3 set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" @@ -15,17 +27,28 @@ repo="$(dirname "$here")" LLVM_BUILD="${LLVM_BUILD:-/opt/llvm}" WTC_BUILD="${WTC_BUILD:-$repo/build}" RUNS="${RUNS:-5}" +RESULTS="${RESULTS:-$here/RESULTS.md}" +BENCH_CPUS="${BENCH_CPUS:-}" out="$WTC_BUILD/benchmarks" mkdir -p "$out" +# name the optimized build must be direct variant +workloads=( + "spsc_throughput substituted yes" + "spsc_latency substituted yes" + "spsc_payload substituted yes" + "spsc_same_thread substituted yes" + "mpsc_control kept no" +) + +die() { echo "run.sh: $*" >&2; exit 1; } + for tool in clang cir-opt mlir-translate llc; do - if [ ! -x "$LLVM_BUILD/bin/$tool" ]; then - echo "no $tool in $LLVM_BUILD/bin; set LLVM_BUILD to the pinned toolchain" >&2 - exit 1 - fi + [ -x "$LLVM_BUILD/bin/$tool" ] || die "no $tool in $LLVM_BUILD/bin; set LLVM_BUILD to the pinned toolchain" done wtc_opt="$WTC_BUILD/bin/wtc-opt" -[ -x "$wtc_opt" ] || { echo "no wtc-opt in $WTC_BUILD/bin; set WTC_BUILD" >&2; exit 1; } +[ -x "$wtc_opt" ] || die "no wtc-opt in $WTC_BUILD/bin; set WTC_BUILD" +command -v python3 > /dev/null || die "python3 is needed to write the report" # A clang built from source does not know where macOS keeps the SDK, and the # final link needs the system's C++ runtime either way. @@ -36,46 +59,106 @@ if [ "$(uname)" = Darwin ]; then link_cxx=(xcrun clang++) fi -emit_cir() { # - "$LLVM_BUILD/bin/clang" -x c++ -std=c++17 -O1 -I "$repo/shim" "${sysroot[@]}" \ - -fclangir -emit-cir "$1" -o "$2" +pin=() +pinning="not pinned" +if [ -n "$BENCH_CPUS" ]; then + if command -v taskset > /dev/null; then + pin=(taskset -c "$BENCH_CPUS") + pinning="pinned to CPUs $BENCH_CPUS with taskset" + else + echo "run.sh: BENCH_CPUS is set but taskset is not available here; runs are not pinned" >&2 + fi +fi + +emit_cir() { # [extra clang flags...] + local src="$1" dst="$2" + shift 2 + "$LLVM_BUILD/bin/clang" -x c++ -std=c++17 -O1 -I "$repo/shim" ${sysroot[@]+"${sysroot[@]}"} "$@" \ + -fclangir -emit-cir "$src" -o "$dst" } # cir-to-llvm leaves CIR types behind in attributes, even for plain clang # output, so mlir-translate has to be told not to choke on them. -link() { # - "$LLVM_BUILD/bin/cir-opt" "$1" --cir-to-llvm -o "$out/$2.llvm.mlir" +link() { # + "$LLVM_BUILD/bin/cir-opt" "$1" --cir-to-llvm -o "$2.llvm.mlir" "$LLVM_BUILD/bin/mlir-translate" --allow-unregistered-dialect --mlir-to-llvmir \ - "$out/$2.llvm.mlir" -o "$out/$2.ll" - "$LLVM_BUILD/bin/llc" -O2 -filetype=obj "$out/$2.ll" -o "$out/$2.o" - "${link_cxx[@]}" "$out/$2.o" -o "$out/$2" -lpthread + "$2.llvm.mlir" -o "$2.ll" + "$LLVM_BUILD/bin/llc" -O2 -filetype=obj "$2.ll" -o "$2.o" + "${link_cxx[@]}" "$2.o" -o "$2" -lpthread } -emit_cir "$here/queue_bench.cpp" "$out/queue_bench.cir" - -"$wtc_opt" "$out/queue_bench.cir" --mlir-print-op-on-diagnostic=false \ - --lift-cir-to-wtc --queue-ownership-analysis --lower-wtc-to-cir \ - --reconcile-unrealized-casts -o "$out/queue_bench_baseline.cir" > /dev/null -link "$out/queue_bench_baseline.cir" queue_bench_baseline - -"$wtc_opt" "$out/queue_bench.cir" --mlir-print-op-on-diagnostic=false \ - --lift-cir-to-wtc --queue-ownership-analysis --select-queue-impl --lower-wtc-to-cir \ - --reconcile-unrealized-casts -o "$out/queue_bench_optimized.cir" > /dev/null -# Linkage keywords differ by target (ELF globals are dso_local), so match -# only the name and the type. -grep -qE 'cir\.global .*@g_queue = #cir\.zero : !rec_wtc3A3Aruntime' "$out/queue_bench_optimized.cir" \ - || { echo "--select-queue-impl did not substitute g_queue" >&2; exit 1; } -link "$out/queue_bench_optimized.cir" queue_bench_optimized - -emit_cir "$here/spsc_direct.cpp" "$out/spsc_direct.cir" -link "$out/spsc_direct.cir" spsc_direct - -for bin in queue_bench_baseline queue_bench_optimized spsc_direct; do - rates=() - for _ in $(seq "$RUNS"); do - line="$("$out/$bin")" - rates+=("$(sed -n 's/.*items_per_sec=\([0-9]*\).*/\1/p' <<<"$line")") +wtc() { # [extra passes...] + local src="$1" dst="$2" remarks="$3" + shift 3 + "$wtc_opt" "$src" --mlir-print-op-on-diagnostic=false \ + --lift-cir-to-wtc --queue-ownership-analysis "$@" --lower-wtc-to-cir \ + --reconcile-unrealized-casts -o "$dst" 2> "$remarks" > /dev/null +} + +variants=() # "workload variant binary" +: > "$out/substitution.txt" + +for entry in "${workloads[@]}"; do + read -r name expect direct <<< "$entry" + src="$here/workloads/$name.cpp" + dir="$out/$name" + mkdir -p "$dir" + echo "== building $name" >&2 + + emit_cir "$src" "$dir/$name.cir" + + wtc "$dir/$name.cir" "$dir/baseline.cir" "$dir/baseline.remarks" + link "$dir/baseline.cir" "$dir/baseline" + variants+=("$name baseline $dir/baseline") + + wtc "$dir/$name.cir" "$dir/optimized.cir" "$dir/optimized.remarks" --select-queue-impl + if grep -q 'remark: substituted' "$dir/optimized.remarks"; then got=substituted; else got=kept; fi + [ "$got" = "$expect" ] || die "$name: the optimized build was $got, but it must be $expect (see $dir/optimized.remarks)" + link "$dir/optimized.cir" "$dir/optimized" + variants+=("$name optimized $dir/optimized") + echo "$name $got" >> "$out/substitution.txt" + + if [ "$direct" = yes ]; then + emit_cir "$src" "$dir/direct.cir" -DWTC_BENCH_DIRECT + link "$dir/direct.cir" "$dir/direct" + variants+=("$name direct $dir/direct") + fi +done + +raw="$out/raw.txt" +: > "$raw" +for run in $(seq "$RUNS"); do + echo "== run $run of $RUNS" >&2 + for entry in "${variants[@]}"; do + read -r name variant binary <<< "$entry" + line="$(${pin[@]+"${pin[@]}"} "$binary")" || die "$name/$variant failed on run $run: its results were wrong" + echo "run=$run variant=$variant ${line#RESULT }" >> "$raw" done - median="$(printf '%s\n' "${rates[@]}" | sort -n | sed -n "$(( (RUNS + 1) / 2 ))p")" - echo "$bin: median $median items/s over $RUNS runs (${rates[*]})" done + +# What the numbers were measured on. +env_file="$out/environment.txt" +{ + echo "date=$(date -u '+%Y-%m-%d %H:%M UTC')" + commit="$(git -C "$repo" rev-parse --short=12 HEAD)" + git -C "$repo" diff --quiet HEAD -- . ':!benchmarks/RESULTS.md' || commit="$commit (with uncommitted changes)" + echo "commit=$commit" + echo "branch=$(git -C "$repo" rev-parse --abbrev-ref HEAD)" + echo "llvm=$(git -C "$repo" rev-parse --short=12 HEAD:llvm)" + if [ "$(uname)" = Darwin ]; then + echo "cpu=$(sysctl -n machdep.cpu.brand_string)" + echo "cores=$(sysctl -n hw.ncpu)" + echo "memory=$(( $(sysctl -n hw.memsize) / 1073741824 )) GiB" + echo "os=macOS $(sw_vers -productVersion) ($(uname -m))" + else + echo "cpu=$(grep -m1 'model name' /proc/cpuinfo | cut -d: -f2- | sed 's/^ *//')" + echo "cores=$(nproc)" + echo "memory=$(( $(grep MemTotal /proc/meminfo | awk '{print $2}') / 1048576 )) GiB" + echo "os=$(uname -sr) ($(uname -m))" + fi + echo "runs=$RUNS" + echo "pinning=$pinning" +} > "$env_file" + +python3 "$here/report.py" "$raw" "$env_file" "$out/substitution.txt" "$here/workloads" "$RESULTS" +echo "== wrote $RESULTS" >&2 diff --git a/benchmarks/spsc_direct.cpp b/benchmarks/spsc_direct.cpp deleted file mode 100644 index 5a5e3d2..0000000 --- a/benchmarks/spsc_direct.cpp +++ /dev/null @@ -1,61 +0,0 @@ -// The same workload as queue_bench.cpp, written by hand against the -// substitute with its allocation-free pop. This is the ceiling: the -// compiler cannot beat it, only reach it without the human. - -#include "../runtime/spsc_queue.hpp" -#include "thread.hpp" - -#include -#include - -constexpr long kNumItems = 5'000'000; - -wtc::runtime::spsc_queue g_queue; - -long g_received = 0; -long g_sum = 0; -long g_out_of_order = 0; - -void producer() { - for (long i = 0; i < kNumItems; ++i) { - int value = static_cast(i); - g_queue.push(value); - } -} - -void consumer() { - long expected = 0; - while (g_received < kNumItems) { - auto value = g_queue.try_pop_optional(); - if (value) { - if (*value != expected) - ++g_out_of_order; - g_sum += *value; - ++expected; - ++g_received; - } - } -} - -int main() { - auto start = std::chrono::steady_clock::now(); - - auto producer_thread = wtc::spawn(producer); - auto consumer_thread = wtc::spawn(consumer); - producer_thread.join(); - consumer_thread.join(); - - auto end = std::chrono::steady_clock::now(); - double seconds = std::chrono::duration(end - start).count(); - - long expected_sum = kNumItems * (kNumItems - 1) / 2; - if (g_received != kNumItems || g_sum != expected_sum || g_out_of_order != 0) { - std::printf("FAIL received=%ld sum=%ld (expected %ld) out_of_order=%ld\n", - g_received, g_sum, expected_sum, g_out_of_order); - return 1; - } - - std::printf("items=%ld elapsed_ms=%.3f items_per_sec=%.0f\n", - kNumItems, seconds * 1000.0, static_cast(kNumItems) / seconds); - return 0; -} diff --git a/benchmarks/workloads/mpsc_control.cpp b/benchmarks/workloads/mpsc_control.cpp new file mode 100644 index 0000000..be1e27b --- /dev/null +++ b/benchmarks/workloads/mpsc_control.cpp @@ -0,0 +1,67 @@ +// The control: two producers, one consumer. The verdict is MPSC, so +// --select-queue-impl must leave the queue alone, and run.sh fails if it +// does not. Baseline and optimized should then be the same program and +// measure the same, which bounds how much of the other workloads' +// difference is noise. There is no direct variant: spsc_queue is not safe +// with two producers. + +#include "../bench.hpp" +#include "thread.hpp" + +namespace { +constexpr long kItemsPerProducer = 2'500'000; +const char *const kName = "mpsc_control"; +} + +bench_queue g_queue; + +long g_received = 0; +long g_sum = 0; + +void producer_even() { + for (long i = 0; i < kItemsPerProducer; ++i) { + long value = 2 * i; + g_queue.push(value); + } +} + +void producer_odd() { + for (long i = 0; i < kItemsPerProducer; ++i) { + long value = 2 * i + 1; + g_queue.push(value); + } +} + +void consumer() { + // Each producer's own items must arrive in its order. + long next_even = 0; + long next_odd = 1; + while (g_received < 2 * kItemsPerProducer) { + auto value = BENCH_POP(g_queue); + if (value) { + long &next = (*value % 2 == 0) ? next_even : next_odd; + if (*value != next) + bench::fail(kName, "a producer's items out of order", *value); + next += 2; + g_sum += *value; + ++g_received; + } + } +} + +int main() { + std::int64_t start = bench::now_ns(); + auto a = wtc::spawn(producer_even); + auto b = wtc::spawn(producer_odd); + auto c = wtc::spawn(consumer); + a.join(); + b.join(); + c.join(); + std::int64_t elapsed = bench::now_ns() - start; + + long n = 2 * kItemsPerProducer; + if (g_sum != n * (n - 1) / 2) + bench::fail(kName, "wrong sum", g_sum); + bench::report(kName, n, elapsed); + return 0; +} diff --git a/benchmarks/workloads/spsc_latency.cpp b/benchmarks/workloads/spsc_latency.cpp new file mode 100644 index 0000000..8458be3 --- /dev/null +++ b/benchmarks/workloads/spsc_latency.cpp @@ -0,0 +1,62 @@ +// One producer, one consumer, at a fixed rate well below either queue's +// capacity: one item every kIntervalNs. Each item is the time it was +// pushed, so the consumer can measure how long it waited in the queue. +// Reports the latency distribution, the number the project's motivation +// rests on: p50, p99, p99.9 and the worst case. +// +// The producer paces itself by spinning on the clock rather than sleeping, +// because a sleep's own wake-up jitter would swamp what is being measured. + +#include "../bench.hpp" +#include "thread.hpp" + +namespace { +constexpr long kItems = 1'000'000; +constexpr std::int64_t kIntervalNs = 1'000; +const char *const kName = "spsc_latency"; +} + +bench_queue g_queue; + +std::vector g_latencies; +long g_received = 0; + +void producer() { + std::int64_t next = bench::now_ns(); + for (long i = 0; i < kItems; ++i) { + while (bench::now_ns() < next) { + } + std::int64_t pushed_at = bench::now_ns(); + g_queue.push(pushed_at); + next += kIntervalNs; + } +} + +void consumer() { + std::int64_t previous = 0; + while (g_received < kItems) { + auto value = BENCH_POP(g_queue); + if (value) { + std::int64_t popped_at = bench::now_ns(); + if (*value < previous) + bench::fail(kName, "out of order", g_received); + previous = *value; + g_latencies[g_received] = popped_at - *value; + ++g_received; + } + } +} + +int main() { + g_latencies.resize(kItems); + + std::int64_t start = bench::now_ns(); + auto c = wtc::spawn(consumer); + auto p = wtc::spawn(producer); + p.join(); + c.join(); + std::int64_t elapsed = bench::now_ns() - start; + + bench::report(kName, kItems, elapsed, &g_latencies); + return 0; +} diff --git a/benchmarks/workloads/spsc_payload.cpp b/benchmarks/workloads/spsc_payload.cpp new file mode 100644 index 0000000..b181eea --- /dev/null +++ b/benchmarks/workloads/spsc_payload.cpp @@ -0,0 +1,61 @@ +// spsc_throughput with a 64-byte record instead of an int, so each push +// and pop copies a cache line. Shows whether the substitution's advantage +// holds when the payload, not the synchronisation, starts to dominate, and +// that it applies to a user-defined element type at all. + +#include "../bench.hpp" +#include "thread.hpp" + +namespace { +constexpr long kItems = 2'000'000; +const char *const kName = "spsc_payload"; +} + +struct Payload { + long seq; + long words[7]; +}; + +bench_queue g_queue; + +long g_received = 0; +long g_checksum = 0; + +void producer() { + for (long i = 0; i < kItems; ++i) { + Payload payload; + payload.seq = i; + for (long w = 0; w < 7; ++w) + payload.words[w] = i + w; + g_queue.push(payload); + } +} + +void consumer() { + while (g_received < kItems) { + auto value = BENCH_POP(g_queue); + if (value) { + if (value->seq != g_received) + bench::fail(kName, "out of order or duplicated", value->seq); + if (value->words[6] != value->seq + 6) + bench::fail(kName, "payload corrupted", value->seq); + g_checksum += value->words[3]; + ++g_received; + } + } +} + +int main() { + std::int64_t start = bench::now_ns(); + auto p = wtc::spawn(producer); + auto c = wtc::spawn(consumer); + p.join(); + c.join(); + std::int64_t elapsed = bench::now_ns() - start; + + long expected = kItems * (kItems - 1) / 2 + 3 * kItems; + if (g_checksum != expected) + bench::fail(kName, "wrong checksum", g_checksum); + bench::report(kName, kItems, elapsed); + return 0; +} diff --git a/benchmarks/workloads/spsc_same_thread.cpp b/benchmarks/workloads/spsc_same_thread.cpp new file mode 100644 index 0000000..3827daf --- /dev/null +++ b/benchmarks/workloads/spsc_same_thread.cpp @@ -0,0 +1,41 @@ +// One thread is both the producer and the consumer: it pushes a batch, +// then pops it all, kRounds times. The verdict is still SPSC (one thread +// pushes, one thread pops; they happen to be the same), and this is the +// case a bounded ring cannot take: with a batch larger than the ring, a +// push that waited for room would wait forever. The substitute is +// unbounded, so it must complete. With no second thread there is no +// contention at all, so this measures the queues' raw cost per operation. + +#include "../bench.hpp" + +namespace { +constexpr long kBatch = 200'000; +constexpr long kRounds = 10; +const char *const kName = "spsc_same_thread"; +} + +bench_queue g_queue; + +int main() { + std::int64_t start = bench::now_ns(); + for (long round = 0; round < kRounds; ++round) { + for (long i = 0; i < kBatch; ++i) { + long value = round * kBatch + i; + g_queue.push(value); + } + for (long i = 0; i < kBatch; ++i) { + auto value = BENCH_POP(g_queue); + if (!value) + bench::fail(kName, "empty before the batch was drained", i); + if (*value != round * kBatch + i) + bench::fail(kName, "out of order or duplicated", *value); + } + if (BENCH_POP(g_queue)) + bench::fail(kName, "item left after the batch", round); + } + std::int64_t elapsed = bench::now_ns() - start; + + // Each item is pushed once and popped once; report operations as pairs. + bench::report(kName, kBatch * kRounds, elapsed); + return 0; +} diff --git a/benchmarks/workloads/spsc_throughput.cpp b/benchmarks/workloads/spsc_throughput.cpp new file mode 100644 index 0000000..d50c734 --- /dev/null +++ b/benchmarks/workloads/spsc_throughput.cpp @@ -0,0 +1,52 @@ +// One producer, one consumer, as fast as they can go: kItems ints through +// one queue. The consumer spins on an empty queue. Measures throughput +// under saturation; latency here would only measure how far behind the +// consumer falls, so none is reported. + +#include "../bench.hpp" +#include "thread.hpp" + +namespace { +constexpr long kItems = 5'000'000; +const char *const kName = "spsc_throughput"; +} + +bench_queue g_queue; + +long g_received = 0; +long g_sum = 0; + +void producer() { + for (long i = 0; i < kItems; ++i) { + int value = static_cast(i); + g_queue.push(value); + } +} + +void consumer() { + long expected = 0; + while (g_received < kItems) { + auto value = BENCH_POP(g_queue); + if (value) { + if (*value != expected) + bench::fail(kName, "out of order or duplicated", *value); + g_sum += *value; + ++expected; + ++g_received; + } + } +} + +int main() { + std::int64_t start = bench::now_ns(); + auto p = wtc::spawn(producer); + auto c = wtc::spawn(consumer); + p.join(); + c.join(); + std::int64_t elapsed = bench::now_ns() - start; + + if (g_sum != kItems * (kItems - 1) / 2) + bench::fail(kName, "wrong sum", g_sum); + bench::report(kName, kItems, elapsed); + return 0; +} diff --git a/runtime/spsc_queue.hpp b/runtime/spsc_queue.hpp index ab915e1..fd0bf6d 100644 --- a/runtime/spsc_queue.hpp +++ b/runtime/spsc_queue.hpp @@ -23,11 +23,14 @@ // once it has drained it and seen the next one; publishing `next` is the // producer's last access to a block, so nothing touches a freed block. // -// Measured, against wtc::queue on one producer and one consumer, at the -// pin on an Apple M1 Pro (benchmarks/README.md): 3.6 times the throughput -// when substituted by the compiler, which keeps try_pop's shared_ptr and so -// allocates once per item; 20.9 times when used by hand through -// try_pop_optional, which does not. +// Measured against wtc::queue by benchmarks/run.sh, results in +// benchmarks/RESULTS.md: about 2.4 to 3 times the throughput when the +// compiler substitutes it, which keeps try_pop's shared_ptr and so +// allocates once per item, and 11 to 16 times when used by hand through +// try_pop_optional, which does not. Median push-to-pop latency falls from +// microseconds to about 200 ns, and the p99 from hundreds of microseconds +// to microseconds or tens of them; substituted, the tail varies from run +// to run far more than it does used by hand. #include #include