Align every GEAK benchmark on the Hyperloom warm-server protocol

[yueliu14]

GEAK's validation measured throughput as the median of 3 isolated-server replicas, while the Hyperloom orchestrator that rebenches the very same kernel measures one warm round: boot the server, run a full untimed round, report the second pass. Two different statistics over the same kernel is how the two sides ended up disagreeing about whether a win was real.

Why

This is not hypothetical. From GEAK's own KB (e2e_workflow/knowledge/learned/moe-fp8-blockscale-tune-gfx950.md), Qwen3.5-122B-A10B-FP8, TP=2, gfx950/MI355, fp8 block[128,128] MoE at 21.4% of GPU time:

measurement	verdict
milestone interleaved A/B	+1.36%, ranges non-overlapping, byte-exact parity 12/12
Director same-session A/B	+0.16% (1.0016x), ranges overlapping -> validated_no_win

The difference was the baseline moving underneath the comparison: base median rose from 2405.9 (warm-start) to 2451.5 (same-session), so most of the apparent gain was drift, not the kernel. Note that the wrong answer was the more convincing-looking one — non-overlapping ranges plus byte-exact parity. It also shows byte-exact parity is not evidence of a throughput win.

One lifecycle for every throughput number is the fix.

What changes

• warm_server becomes the default lifecycle in bench_e2e.sh, and the default for the search, parity and validation purposes in e2e_workflow.js / run_e2e.py. REPEATS defaults to 1: one timed round is the protocol, not a truncation of it. Carve-outs for REPEATS=0 (shape capture) and PROFILE=1, which produce no throughput number and so have no lifecycle to align.
• isolated_server stays available for boot-to-boot dispersion, opt-in only, with its own validation_measurement_mode / validation_replicas knobs so an operator can make the final arbitration stricter than the search that fed it.
• Delete E2E_REPEATS. It was prompt-only — no role doc or script ever read it.
• Extract the two summary emitters out of bench_e2e.sh into scripts/bench_summarize.py (from-runs / from-replicas). They were duplicated heredocs with no test, and every acceptance decision in the product reads their output; a contract field added to one and not the other produced two bench_summary.json shapes with nothing to catch it. bench_e2e.sh 1053 -> 851 lines.
• Docs and role prompts updated to describe the protocol the code actually runs, instead of the old ">=3 repeats, median + spread".

Verified

Unit: 1171 tests under e2e_workflow/scripts/tests, 168 in interface/test_run_e2e_dispatch. The new emitter is byte-identical to the old heredocs over 12 replayed input shapes.

On hardware — a direct protocol A/B. Same box, model, serving invariant and tuning bundle; adapters/ and bench_replica.sh byte-identical between the two runs, so the lifecycle is the only variable. DeepSeek-V4-Pro, vLLM, TP=8, ISL 8192 / OSL 1024 / conc 64, 192 prompts:

lifecycle	base	final	delta	wall-clock	cold boots
isolated_server x3 (before)	833.298	879.030	+5.488%	~68 min	6
warm_server x1 (this PR's default)	826.936	886.571	+7.212%	30.8 min	2

Same verdict, 2.2x faster, 6 cold boots down to 2. Server boot alone was 466 s and 381 s of the two legs — that is what the extra replicas were re-paying.

The tradeoff, stated up front

The two deltas are 1.72 pp apart, and that gap is the honest cost of the change, so it should not be discovered in review.

One sample per leg means spread = 0.0% by construction — an absence of dispersion evidence, not a quiet box — and the final_min > base_max non-overlap test carries no information at all. The isolated legs measured 2.61% and 2.81% spread, so a low base draw and a high final draw compound to ~1.6 pp on their own, which accounts for the observed 1.72 pp. Read it as sampling, not as a bias between the protocols.

roles/director.md handles this explicitly rather than hiding it: under warm_server the samples share one boot, so spread is narrower than true run-to-run spread; when delta% is within 2x the noise band and only non-overlap carries the win, the Director must say so in notes and prefer validated_no_win. At one sample it must record that the dispersion test was unavailable. bench_summary.json carries measurement_mode and dispersion_basis so any consumer can tell which lifecycle produced a number.

Numbers are comparable only within one lifecycle. The header warning originally claimed warm_server "reads systematically HIGHER"; the hardware run above is a counterexample (warm 826.9 vs isolated median 833.3 — the other way, because RANDOM_RANGE_RATIO=1.0 at ISL 8192 leaves almost no shared prefix to cache). The rule is unchanged; the stated reason is now that the sign and size are not predictable from the config alone.

Out of scope

An earlier revision of this branch also loosened the gsm8k accuracy gate (tol 0.01 -> 0.03, plus an absolute 0.5 floor). That has been reverted here. The argument stands on its own, but it is a gate-policy change rather than a measurement-protocol one, and bundling it would force this PR to be accepted or rejected as a unit. It will come back as its own PR with its own evidence.

🤖 Generated with Claude Code