[1/6][load] publish north-star harness capacity run

docs/performance/runs/2026-09-23-north-star-steady-v1-harness-capacity-33c95bf/analysis.md

@@ -0,0 +1,90 @@
+# Run Analysis
+
+Run: [run.json](run.json)
+
+Results: [result.json](result.json)
+
+This document is generated by AI from the run metadata, machine results, and
+retained diagnostics.
+
+## Purpose
+
+Locate the local capacity ceiling of the `kq-bench` harness on the
+`north-star-steady-v1` execution-time distribution (250 ms average, 475 ms p95,
+646 ms p99) by scaling producers, workers, worker concurrency, and ready
+partitions.
+
+## Observations
+
+Capacity probes at 3,000 RPS with 4 workers and 16 ready partitions:
+
+| Concurrency | Active worker RPS | Per worker | Drain | Queue p50 |
+| ---: | ---: | ---: | ---: | ---: |
+| 100 | 529.7 | 132 | 93.3 s | 44.6 s |
+| 250 | 1,136.2 | 284 | 32.8 s | 15.8 s |
+| 500 | 2,006.3 | 502 | 9.9 s | 3.5 s |
+| 1000 | 2,742.1 | 686 | 1.9 s | 501 ms |
+
+Arrival-rate scaling:
+
+| Requested RPS | Producers | Workers x concurrency | Partitions | Enqueue RPS | Worker RPS | Drain | Queue p50 / p95 / p99 | Host busy |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 500 | 4 | 4 x 100 | 4 | 500.0 | 473.5 | 1.7 s | 438 / 982 / 1,295 ms | n/a |
+| 5,000 | 8 | 8 x 1000 | 16 | 4,995 | 4,613 | 1.7 s | 475 / 984 / 1,206 ms | 68% |
+| 8,000 | 10 | 12 x 1000 | 24 | 7,992 | 7,395 | 1.6 s | 475 / 997 / 1,211 ms | 87% |
+| 12,000 | 15 | 18 x 1000 | 32 | 11,984 | 11,114 | 1.6 s | 479 / 1,001 / 1,214 ms | 99% |
+| 16,000 | 20 | 24 x 1000 | 32 | 15,391 | 12,446 | 5.0 s | 700 / 4,087 / 5,233 ms | 100% |
+| 16,000 | 16 | 12 x 2000 | 32 | 15,958 | 12,760 | 5.1 s | 813 / 4,770 / 6,967 ms | 100% |
+| 20,000 | 20 | 15 x 2000 | 40 | 16,925 | 13,463 | 6.1 s | 1,025 / 7,962 / 10,367 ms | 100% |
+
+- 12,000 RPS was the highest sustainable tested rate; 16,000 RPS was the first
+  overloaded rate. Maximum observed worker throughput was 13,463 tasks/s.
+- At 16,000 and 20,000 RPS producers also fell behind the requested rate
+  (15,391 and 16,925 enqueues/s).
+- Steady-state CPU at 12,000 RPS, measured as cumulative process CPU time over
+  a 20 s window: Kafka Docker VM 4.29 cores, producers 2.75, harness Python
+  1.73, workers 0.77, other host processes 1.03.
+- Harness Python resident memory reached about 2.6 GB during the 45 s
+  12,000 RPS point.
+- All 2,815,000 measured tasks across all points completed with zero missing,
+  duplicate, outstanding, or dead-lettered IDs.
+- Measured execution averaged 250 to 252 ms with a 645 to 657 ms p99. Execution
+  maximum rose from about 1,494 ms to 1,805 ms once the host was saturated.
+
+## Interpretation
+
+- The ceiling is host CPU saturation, not a kq-specific limit. Workers used
+  0.77 cores for over 11,000 tasks/s; the harness's own Python accounting and
+  producer-side event emission consumed several cores of the same 12-core host.
+- Each harness producer calls `Enqueue` serially, so every task is an
+  individual `acks=all` produce request. Hypothesis: this drives a large share
+  of producer and Kafka CPU; the lean-probe run measures concurrent `Enqueue`
+  separately.
+- Queue p50 of about 475 ms is nearly independent of arrival rate at
+  concurrency 1000. It reflects worker generation behavior rather than backlog;
+  the queue-latency diagnosis run investigates it.
+- The rising execution maximum at saturated points suggests timer overshoot
+  from CPU contention.
+
+## Suggestions
+
+1. Rework the harness before using it above about 10,000 RPS: concurrent or
+   batched producer enqueues, compact or sampled per-task events, and
+   streaming accounting instead of in-memory rescans every second.
+2. Record per-process CPU and memory in the harness so saturated runs are
+   identified automatically.
+3. Compare against the lean-probe capacity run, which removes harness overhead
+   on the same host.
+
+## Caveats
+
+- Each point ran once on one arm64 laptop with a single Dockerized Kafka broker
+  (RF=1, 12 CPUs and 8 GB assigned to the Docker VM). Load generators, workers,
+  and Kafka shared the host.
+- Points used 20 to 45 s arrival windows; longer runs were not attempted
+  because harness memory grows with event count.
+- Host and process CPU peaks come from 2 s samples of `top`, `docker stats`,
+  and `ps`; `ps` CPU is a decaying average. The 12,000 RPS breakdown uses
+  cumulative CPU time and is more reliable.
+- The harness was invoked from source because the configured package index
+  rejected the `hatchling` build dependency; the harness code was unchanged.