Skip to content

docs: the resume line led with a p99 that was stale and mislabelled - #63

Merged
DanielWLiu07 merged 1 commit into
mainfrom
docs/resume-line-and-stale-p99
Sep 1, 2026
Merged

DanielWLiu07 merged 1 commit into
mainfrom
docs/resume-line-and-stale-p99

Conversation

@DanielWLiu07

Copy link
Copy Markdown
Owner

PLAN.md's resume line opened with "p99 37 us ingest-to-signal on recorded live sessions." That number was wrong in two independent ways.

1. It had drifted

The same committed captures measure 74–78 µs today. docs/bench/latency.md was updated to 77.5; this line and soak.md's cross-reference were not.

2. It's a service time, and the repo's headline says so

latency.md's entire finding is that the service p99 understates what a consumer waits by 13.3x. A resume line leading with an unqualified service p99 contradicts the repo's own strongest result.

The finding now replaces the figure:

Proved its own published p99 was a service time and 13.3x optimistic by replaying at the venue's real arrival schedule, and gated the invariant in CI.

That's the better line anyway. Anyone can report a percentile; almost nobody reports catching their own benchmark flattering itself.

Also

  • 2.19M → ~2M messages/sec. 2.19 was a best-of-five; today's five runs span 1.80–1.90 on a busier machine. The ratio against the venue's 269 carries the point, not the third digit.
  • Added the message counts (204,864 and 642,919) behind the z-scores, since a z with no n invites the question.
  • soak.md's stale cross-reference fixed, with a note that the agreement between the two captures is the durable claim and the absolute figures move with the machine.

PLAN.md's resume line opened with "p99 37 us ingest-to-signal on recorded
live sessions". That number was wrong in two independent ways.

It had drifted. The same committed captures measure 74 to 78 us today,
and docs/bench/latency.md was updated to 77.5 while this line and
soak.md's cross-reference were not.

More seriously, it is a SERVICE time. latency.md's entire finding is that
the service p99 understates what a consumer actually waits by 13.3x, so a
resume line leading with an unqualified service p99 contradicts the
repo's own headline result.

The finding now replaces the figure, and it is the better line: anyone can
report a percentile, and almost nobody reports catching their own
benchmark flattering itself. Also swapped 2.19M messages/sec for ~2M,
since 2.19 was a best-of-five and today's five runs span 1.80 to 1.90 on a
busier machine - the ratio against the venue's 269 is what carries the
point, not the third digit.

soak.md's "p50 0.5 us, p99 37 us" cross-reference gets the same treatment,
plus a note that the agreement between the two captures is the durable
claim and the absolute figures move with the machine.
@DanielWLiu07
DanielWLiu07 merged commit ca65978 into main Sep 1, 2026
9 checks passed
@DanielWLiu07
DanielWLiu07 deleted the docs/resume-line-and-stale-p99 branch September 1, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant