From 8ad3163cfa8f1d7038f673bd8401b84f040fef98 Mon Sep 17 00:00:00 2001 From: Daniel W Liu <117604131+LTKers@users.noreply.github.com> Date: Mon, 31 Aug 2026 22:28:56 -0400 Subject: [PATCH] docs: the resume line led with a p99 that was stale and mislabelled PLAN.md's resume line opened with "p99 37 us ingest-to-signal on recorded live sessions". That number was wrong in two independent ways. It had drifted. The same committed captures measure 74 to 78 us today, and docs/bench/latency.md was updated to 77.5 while this line and soak.md's cross-reference were not. More seriously, it is a SERVICE time. latency.md's entire finding is that the service p99 understates what a consumer actually waits by 13.3x, so a resume line leading with an unqualified service p99 contradicts the repo's own headline result. The finding now replaces the figure, and it is the better line: anyone can report a percentile, and almost nobody reports catching their own benchmark flattering itself. Also swapped 2.19M messages/sec for ~2M, since 2.19 was a best-of-five and today's five runs span 1.80 to 1.90 on a busier machine - the ratio against the venue's 269 is what carries the point, not the third digit. soak.md's "p50 0.5 us, p99 37 us" cross-reference gets the same treatment, plus a note that the agreement between the two captures is the durable claim and the absolute figures move with the machine. --- PLAN.md | 29 ++++++++++++++++++++--------- docs/bench/soak.md | 14 ++++++++++---- 2 files changed, 30 insertions(+), 13 deletions(-) diff --git a/PLAN.md b/PLAN.md index 044801c..8238b2c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -253,13 +253,15 @@ measured Kalshi/Polymarket lead until that key exists. > books from four venues over TLS WebSocket into one normalized schema > through a zero-copy hot path, proven at 1-2 heap allocations per message > and at parity with Bloomberg BDE arenas by a counting-allocator -> benchmark, behind a BLPAPI-style subscription interface. **p99 37 us -> ingest-to-signal** on recorded live sessions and **2.19M messages/sec** -> parse-and-apply against a venue producing 269. Measured a **cross-venue -> price lead and replicated it** on an independent capture (two-proportion -> z = 11.8 and 39.3), under a measurement biased against the result. -> Zero message loss across forced disconnects and silent half-open -> connections, all of it reproducible from CI-gated benchmarks. +> benchmark, behind a BLPAPI-style subscription interface. Parses and +> applies **~2M messages/sec** against a venue producing 269. Measured a +> **cross-venue price lead and replicated it** on an independent capture +> (two-proportion z = 11.8 and 39.3 across 204,864 and 642,919 messages), +> under a measurement biased against the result. **Proved its own +> published p99 was a service time and 13.3x optimistic** by replaying at +> the venue's real arrival schedule, and gated the invariant in CI. Zero +> message loss across forced disconnects and silent half-open connections, +> all of it reproducible from CI-gated benchmarks. Notes on the wording, because each choice is load-bearing: @@ -270,5 +272,14 @@ Notes on the wording, because each choice is load-bearing: - "at parity with BDE arenas" not "faster than": it was parity, and claiming otherwise is checkable in thirty seconds by anyone who opens `docs/bench/allocator.md`. -- "against a venue producing 269" is what makes 2.19M mean something. A - throughput number with no denominator invites "so what?". +- "against a venue producing 269" is what makes the throughput mean + something. A number with no denominator invites "so what?". +- No bare ingest-to-signal p99 any more. This line used to lead with + "p99 37 us", which was wrong twice over. It had drifted - the same + committed captures measure 74 to 78 us now - and, more importantly, it + is a SERVICE time. docs/bench/latency.md's whole finding is that the + service p99 understates what a consumer sees by 13.3x, so quoting one + unqualified in a resume line contradicts the repo's own headline. The + finding replaces the figure, which is a better line anyway: anyone can + report a percentile, and almost nobody reports that they caught their + own benchmark flattering itself. diff --git a/docs/bench/soak.md b/docs/bench/soak.md index 2d3d7a6..05e5a82 100644 --- a/docs/bench/soak.md +++ b/docs/bench/soak.md @@ -36,10 +36,16 @@ Three consecutive replays: | 2 | 0.5 us | 0.7 us | 36.2 us | 770k records/sec | | 3 | 0.5 us | 0.7 us | 37.0 us | 771k records/sec | -The percentiles agree with `latency.md`'s 30 minute capture (p50 0.5 us, -p99 37 us) on a session recorded three days later with 1.7x the message -count: the latency numbers are properties of the engine, not of one -lucky recording. +The percentiles agree with `latency.md`'s 30 minute capture on a session +recorded three days later with 1.7x the message count: the latency +numbers are properties of the engine, not of one lucky recording. Both +captures re-measure at p50 0.8 us and p99 74 to 78 us today, against the +p50 0.5 / p99 37 recorded when this was written - the agreement between +the two captures is the durable part, the absolute figures move with the +machine. + +Both are SERVICE times. What a consumer waiting on the feed sees is the +response time, which `latency.md` shows is 13.3x larger at p99. ## What this claims and what it does not