docs: describe the 0.2.0 engine, and fix the probes that measure it

[emfdavid]

Summary

Two halves, one squashed commit: the architecture page now describes the 0.2.0 engine, and the
benchmark probes that produce its numbers are fixed.

The page was wrong, not merely stale

docs/architecture.md last moved substantively on 2026-09-05; 26 src/ commits landed after
it, including most of what 0.2.0 changed. Eight claims a reader sizing a budget would have
acted on:

• admission described as bounded by the byte budget (it is bounded by read-ahead permits),
  and a budget "sized to ~two blocks" as what admits the next block;
• peak co-residency stated as two blocks unconditionally, which a windowed view is not;
• read-once stated as a flat guarantee, which release-and-re-read deliberately trades;
• the cache log described as needing no compaction, and a stale cache as a whole-directory
  wipe — on a page that 40 lines earlier described per-array staleness correctly.

Both diagrams were further behind than the prose, which is the part worth noting: nobody greps
a mermaid block. The producer/consumer figure drew queue depth as the only backpressure path,
omitting the permit that returns on unpin_block. The handoff figure drew
_drive -> _admit -> try_admit without the read-ahead permit, the pin_if_ready hit check, or
the block boundary that ends an admission run — all three of which now precede it.

Then documents what 0.2.0 added and the page had no home for: the permit protocol, one pool
with many accountable iteration owners, per-block release, the one-writer cache_dir
contract, and readonly_cache.

Vocabulary. The page used one word — "permit" — for both the read-ahead bound and
max_inflight. Unified on "read-ahead permits" throughout docs/, DESIGN.md, scheduler.py
and pool.py. scheduler.py's module docstring said "Two bounded resources" — there are three
since this release.

Paid for by deletion, not growth. Two duplications removed and two rationale blocks moved
to DESIGN.md, where a superseded design belongs per the docs rule in CLAUDE.md. Net effect
on architecture.md: +53 lines and one heading fewer, having absorbed three mechanisms
that previously had no home anywhere.

The probes had two defects and a false premise

• probe_decode.py's decode-thread sweep had stopped measuring anything, and printed a clean
  scaling curve anyway. The decode pool went process-wide in 0.2.0 and the probe's own prewarm
  built it at auto before the sweep began, so every point ran on the same pool. On an 8-vCPU
  box it returned 756 -> 808 -> 921 MB/s across decode_threads=1,2,8: run-to-run noise on
  one 20-thread pool, arranged in the most convincing shape it could have taken. The documented
  diagnosis for a flat sweep is "decode isn't the limit", so the failure pointed away from
  itself. The sweep is gone; the section header now reports the pool's effective thread count.

• The raw-GET ceiling could not be measured on a sharded or a zarr-v2 store: every key 404'd.
  Keys were hand-built from metadata.chunks in zarr-v3's c/i/j/k layout. On a sharded array
  that names the inner chunk while a key addresses the shard (the Sharded zarr-v3 arrays fail to decode: we plan reads on the inner chunk, not the shard #56 distinction); on v2 the
  separator is . with no c/ prefix. It raised rather than returning a wrong number, but it
  meant the insitu-vs-raw comparison was unavailable on every dynamical.org archive (all
  sharded) and on WeatherBench2 ARCO (v2) — the stores the docstring offers as examples. Now
  ChunkGrid.from_metadata plus the array's own encode_chunk_key.

• Every published "% of raw-GET ceiling" figure is computed in mixed units. insitu's MB/s
  counts decoded bytes; a raw GET moves stored bytes. bench/benchmark_plan.md licensed
  dividing one by the other by asserting the synthetic stores are incompressible. They are
  not: zstd level 0 takes ~7.6% off standard_normal f32. The new
  bench/probe_ceiling.py measures the conversion per store rather than assuming it, reports
  "% kept" in one unit, and flags insitu points the read-ahead permit bound capped below their
  max_inflight.

  The conversion it measures is logical-to-stored, not compression — those differ, and
  the distinction is the reason this is stated carefully rather than as "the compression
  ratio". Zarr pads edge chunks to full size (the benchmark geometry's 361 inner rows occupy
  4×91 = 364 stored rows), so ~0.8% of every pass is padding the user never sees. On these
  grids compression and padding nearly cancel and the ratio is 1.081×.

  Validated against exact known answers before being used to correct anything (per
  CLAUDE.md, "validate the instrument before the subject"): a store with no edge padding
  and no codec reports exactly 1.000000; a padded uncompressed one reports exactly
  361/364; and HEAD-derived bytes equal the filesystem's on all four stores. The first
  pass of this validation is what caught the mislabeling — the ratio was not wrong, but
  calling it "compression" was.

⚠️ Known-inflated numbers this PR does not correct

docs/benchmarks.md is untouched here. By the premise fixed above, story 4's 81% / 84% are
each ~6 points high — the honest figures are ~75% / ~78% — as is the 25 Gb/s c6id.16xlarge
scaling table, which used the same method. That is a deliberate deferral, not an oversight:
correcting them properly is a choice between re-labelling them as valid-for-what-they-measured
and re-running, and the compression ratio was measured on a different store
(era5_fat_g16_c64) than story 4's, so dividing through is an inference rather than a
measurement. Tracked as the next step; this PR should not be read as having fixed it.

Likewise not claimed here: the re-run posted in the comments below (≈90% on a NIC-bound
c6id.8xlarge) is not published to docs/ by this PR. Its own caveats are the reason —
best-of-12 insitu medians against best-of-4 raw medians, a raw sweep that runs as a block
rather than interleaved, and no validation of the probe against a known answer (an
--uncompressed store should report exactly 1.000×). The same probe returns 59% on GCS
n2-standard-8, decode-bound at ~450 MB/s.

For reviewers

Behavior is unchanged. The only src/ edits are a module docstring in scheduler.py and
one comment in pool.py; no executable line was touched.

Most valuable second look, in order:

1. The permit prose in docs/architecture.md (the two new bullets under "Prefetch"). It
   describes the general rule — everything live at a block plus the whole of the next. Under
   reread_spill the bound is instead three consecutive blocks (source.py:519-525), which is
   deliberately omitted rather than qualify a bullet twice. Say if that is the wrong call.
2. The handoff diagram's step order, against scheduler.py:508-521: _take_ahead before
   pin_if_ready before _admit, with the block boundary ending the run.
3. The three deletions — whether the material is genuinely recoverable from DESIGN.md and
   docs/tuning.md, or whether the page got less self-contained than you want.

The chunk figures quoted in the new residency prose (259 -> 12, 316 -> 18) are quoted from
0.2.0's own CHANGELOG entry for #66, not re-measured here.

Author attestation

• I have reviewed every change in this PR, I can explain why each one is correct, and I
  have verified the claims made in this description.

Checklist

• uv run ruff check src tests bench examples, uv run ruff format --check src tests bench examples, uv run mypy src bench examples and uv run pytest -q are green
  locally (668 passed, 9 skipped), and uv run --extra docs mkdocs build --strict builds
  with 0 warnings
• Docstrings and API docs for any new or changed public surface
• User-facing behavior documented in docs/*.md
• A bullet added under ## Unreleased in CHANGELOG.md
• No load-bearing invariant is broken
• Touches ChunkPool, the scheduler, or cross-thread readiness → the free-threaded
  (3.13t) run passes — not run locally. The edits to both files are docstring and
  comment only, so there is nothing for it to exercise. CI's test-freethreaded job is
  green on this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_019waAAauPpmW4QX1VaN4vqP

[emfdavid]

Where this leaves off — Tier 3 (docs/benchmarks.md) is not in this PR

This PR closed Tier 1 (false claims) and Tier 2 (missing mechanisms) on docs/architecture.md.
docs/benchmarks.md is untouched and is the remaining work from the 0.2.0 doc sweep. Recording
the plan here so it can be picked up cold.

Context: why benchmarks lagged

docs/benchmarks.md was last substantively updated 2026-09-01 (cb0944c, the run-on-zarr's-loop
refactor). Everything after that — read-ahead permits (#64), per-block release (#66), the pass vs
pool counter split (#81, #84) — landed without it. docs/tuning.md and README.md were updated
through 09-11 and are current; the gap is benchmarks alone.

The one open decision

Story 2 — "decoupling concurrency from residency" (docs/benchmarks.md:89-133). Its tables
report resident flat at 2 x block_chunks (resident = 4 throughout the exp_c sweep). Those
were measured under the pre-#64 admission rule, where admission was bounded only by the byte
budget. The conclusion survives — concurrency and residency are still independent dials — but
the mechanism the prose names is no longer the one in the code, and the story now has a second
half: read-ahead is also decoupled from the consumer's progress.

Two ways to resolve, and it needs a call before the section can be edited:

• (a) Re-word as valid-for-what-it-measured. Cheap, honest, keeps the tables. Frame them as
  the v1-vs-V2 acceptance gate they were, and state the current bound separately.
• (b) Re-run under the current admission rule. Stronger, but it is a GCS/S3 session with a
  control arm and the full setup disclosure the repo requires for a performance claim.

Prior recommendation was (a) plus adding the #64 result as new material — the old tables are
not wrong about what they measured.

The rest of Tier 3, which needs no decision

1. Missing headline result. The Concurrent iterations deadlock at any budget: admission is bounded by the pool, not by the iteration's working set #64 numbers are benchmark-grade and appear nowhere in
   docs/benchmarks.md. From the 0.2.0 CHANGELOG entry, with its setup intact: era5_c8,
   in-region GCS, GCP n2-standard-8, block_chunks=16, batch_size=32, arms alternated three
   times in one session — TTFB 965-10094 ms -> 621-665 ms, throughput 400-562 -> 1082-1152 MB/s.
   Also worth stating: zip(ds.train, ds.val) deadlocked at any budget before this.
2. Batch buffers (docs/benchmarks.md:377-501) — guidance inverted by Batch-buffer counters are pool state under a per-pass label, and lends double-counts under zip #84. The counters are
   the pool's, the epoch line labels that segment (pool), and they run cumulatively across
   every iteration sharing the dataset. The guidance must become "watch allocated stop rising,"
   not "return to zero."
3. Cache stories (Story 3, :135). Should mention that re-reads and evictions now print when
   non-zero, and why a hit rate alone cannot show budget churn — a re-read served from the cache
   counts as a hit, so rising re-reads at a steady hit rate is the signal.

Two smaller things deliberately deferred, not forgotten

• Doc-visible contracts from otherwise non-architectural fixes. An empty split now yields
  nothing rather than raising from inside numpy, and a windowed split that holds chunks but can
  draw nothing is a split-yields-nothing warning with drawable_samples in describe(). Both
  are contract statements worth one line each; neither has a home yet. docs/tuning.md:301
  covers the windowed case from the tuning angle only.
• The reread_spill bound. docs/architecture.md now states the general read-ahead bound
  (live set + next block). Under release-and-re-read it is three consecutive blocks instead
  (src/insitubatch/source.py:519-525). Omitted rather than qualifying the bullet twice; revisit
  if the residency material is ever consolidated into one section.

Consolidation idea, if anyone wants it

Tier 1 required editing three separate sections for one mechanism change — the prefetch
backpressure bullets, "Read-once and sample-once", and the caching-continuum table. That is the
diagnostic for residency being explained in three places. A single "what is held, and what bounds
it" section would consolidate the byte budget, the permit bound, the working-set floor, the
release policy, and ragged-grid padding, and would likely net-shrink the page. Not attempted
here — it is a restructure, not a correction.

[emfdavid]

Added: bench/probe_decode.py (commit df782e6)

Separable from the docs commit — noted here in case this PR gets split.

Heads-up on the title. It currently reads docs:, which now understates the PR: df782e6
is a fix(bench): that changes what the probe measures. If this merges as one squash commit
the title should be re-worded; if it splits, df782e6 carries its own.

What was wrong

1. The decode-thread sweep had stopped measuring anything, and printed a scaling curve
anyway. The decode pool went process-wide in 0.2.0 — built once by the first dataset, later
values ignored with a warning — and the probe's own prewarm built it at auto before the
sweep. Every point ran on the same pool. Verified on an 8-vCPU box:

decode_threads=1 ignored: the decode pool is process-wide and was already built with 20 threads.
decode_threads=2 ignored: ...
decode_threads=8 ignored: ...

1) insitu MB/s vs decode_threads:
   decode_threads=   1:    756.9 MB/s
   decode_threads=   2:    808.1 MB/s
   decode_threads=   8:    921.3 MB/s

Warnings to stderr, above the report; the table below reads as clean monotonic scaling and is
run-to-run noise on one 20-thread pool. The documented reading of a flat sweep is "decode
isn't the limit" — so the miswiring pointed away from itself. This is the
CLAUDE.md instrument rule exactly: it did not raise, it returned a plausible and flattering
number.

2. Raw GET 404'd on sharded and zarr-v2 stores. Keys were hand-built from
metadata.chunks in the v3 c/i/j/k layout. Verified against real arrays:

store	probe built	actually on disk
v3 sharded	c/0/0/0 … c/3/0/0	c/0/0/0, c/1/0/0 — half 404
v2	c/0/0, c/1/0 …	0.0, 1.0 … — all 404

On a sharded array chunks is the inner chunk while a key addresses the shard (#56). So the
insitu-vs-raw comparison — the probe's entire diagnosis — was unavailable on all of
dynamical.org and on WeatherBench2 ARCO, i.e. the stores the module docstring advertises and
the ones the engine has read since #56.

What changed

• Sweep dropped. --decode-threads kept as a single value, applied to the prewarm (the
  first dataset the process builds, and the only one that can set it). The section header now
  prints the pool's effective count via decode_pool_workers(), so a requested-but-ignored
  number can't be quoted. Sweeping means one run per value; the docstring says so.
• Grid from ChunkGrid.from_metadata, key from the array's own encode_chunk_key — the same
  spelling the engine plans reads with. Verified all three layouts now work, with the sharded
  case correctly charging the shard (16,64,64) rather than the inner (8,64,64):

v3 SHARDED     stored_chunk=(16, 64, 64) key0='c/0/0/0'  raw-GET OK ->   565.8 MB/s
v3 plain       stored_chunk=(8, 64, 64)  key0='c/0/0/0'  raw-GET OK ->   402.8 MB/s
v2 (WB2-like)  stored_chunk=(8, 64, 64)  key0='0.0.0'    raw-GET OK ->   384.9 MB/s

• Sections renumbered (1b→1, 1c→1b) now that the sweep is gone.

Two things I did not change

• The MB/s units are correct as designed — I expected this to be the bug and it isn't.
  Raw counts stored bytes, insitu counts decoded bytes, and they agree on the synthetic store
  because standard_normal f32 doesn't compress (bench/benchmark_plan.md:135-143). What was
  missing is that this only holds there: on a real --url, ERA5 compresses ~2–4× and the raw
  number is understated by that ratio, biasing the verdict toward "not decode-limited". Now in
  the probe's docstring, with the instruction to divide first.
• ds.resident_peak is fine and was improved by Per-pass stats report pool-wide counters, so concurrent iterations overwrite each other's numbers #81 — per-pass now rather than pool-wide,
  and recorded in a finally, so the probe's early break still captures it.

The residency wording was stale in the same way as docs/benchmarks.md Story 2: pinned is the
real claim, 2*block_chunks only holds on a non-windowed pass. Measured 6 chunks at
block_chunks=2 --window 2. Corrected in the docstring.

Full check set green: ruff, ruff format, mypy, pytest -q (661 passed / 9 skipped),
mkdocs build --strict at 0 warnings.

[emfdavid]

Setup

┌──────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│          │                                                                                                               │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Box      │ GCP g2-standard-16, 16 vCPU, 62 GB, us-central1-a                                                             │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Store    │ gs://insitubatch-bench-insitubatch/era5_fat_g16.zarr, US-CENTRAL1 (in-region), obstore                        │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Geometry │ 3200 samples, 361×720, sample_chunk=200, inner_chunks=91,180 → 16 outer chunks, 16 tiles/chunk, 13.1 MB tiles │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Size     │ 3327 MB logical / 3077 MB on disk                                                                             │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Method   │ full 3.3 GB pass per point, median of 3, arms interleaved (not blocked), prewarm discarded                    │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Control  │ raw obstore GET, same session, same store                                                                     │
└──────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Results

block_chunks max_inflight bound(tiles) inflight_pk resident MB/s
2 32 64 32 4 1420.3
2 64 64 64 4 1390.9
2 128 64 64 4 1331.1 <-- permit-bound
2 256 64 64 4 1485.2 <-- permit-bound
4 64 128 64 8 1494.4 <- best
4 128 128 128 8 1462.9
8 128 256 128 16 1361.4
8 256 256 256 16 1342.1

raw obstore GET (stored bytes): c=16 1462.8 | c=32 1903.8 | c=64 2346.8 | c=128 1956.7

Best insitu 1494 MB/s decoded · best raw GET 2347 MB/s stored · corrected % kept = 59%.

Two corrections that change the answer

1. The units bias is real and I under-estimated it. standard_normal f32 is not incompressible — zstd gets 8.15% (3327→3077 MB). So the naive ratio flatters insitu: convert raw to decoded-equivalent (×1.0815 → 2538 MB/s) and 63.7% becomes 58.9%. bench/benchmark_plan.md:135-143 asserts ~0% compression, and that premise is wrong. The original 81%/84% almost certainly carries the same ~6-point inflation, since it used the same generator — so the honest AWS figure was likely ~75%, not 81%.

2. Permits do bind at Story 4's config, and cost throughput. At block_chunks=2 (the documented command's default) in-flight caps at 64 tiles regardless of --max-inflight. Raising block_chunks to 4 unlocks 128 and gives the best result — 1494 vs 1391 at the documented setting, ~7%.

Does the claim hold up?

On this hardware, no — 59%, not ~80%. But I'd stop short of calling that a regression, for three reasons:

• Different provider and box. GCS from a g2-standard-16 (16 vCPU, GPU box) vs in-region S3 from a c6id.8xlarge (32 vCPU, compute-optimised). Decode is CPU-bound and this box has half the cores — the denominator also moved: raw GET here hits 2347 MB/s vs 1467 on AWS, a 60% higher ceiling. A bigger ceiling with less CPU to feed it mechanically lowers "% kept."
• insitu's absolute throughput is fine — 1494 MB/s here vs 1187 on the AWS box. It got faster; the ceiling got faster still.
• Raw GET is non-monotonic (peaks at c=64, drops at c=128), so 2347 may itself be near a noisy optimum. Three repeats is thin.

The honest read: this measurement can't confirm or refute the AWS number, and the ~80% claim has a units error in it independent of any of that. What it does establish is that the shape of the argument still holds — decode+gather keeps ~60% of a much higher ceiling on a CPU-halved box — and that two things need fixing before any rerun is quotable.

What I'd fix before an AWS rerun

1. Correct benchmark_plan.md's incompressibility claim and make the probe report the ratio, so "% kept" is computed in one unit. This affects every published "% of raw-GET ceiling" number.
2. Pin --block-chunks ≥ 4 in the Story 4 command, and report inflight_peak so a permit ceiling can't be misread as a network plateau.

[emfdavid]

Story 4 re-run with bench/probe_ceiling.py (commit 44d364c, unmodified)

Re-measured "efficiency vs the raw-GET ceiling" with the new self-checking probe, which (a) measures the compression ratio via HEAD on the exact objects the raw sweep fetches and reports % kept in decoded-equivalent units, and (b) flags insitu points the read-ahead permit bound capped below their max_inflight.

Headline: ≈90% of the ceiling kept on both S3 and S3 Express — on a NIC-bound box with near-incompressible synthetic data. Read the caveats before quoting it.

Setup

hardware	AWS c6id.8xlarge — 32 vCPU, 12.5 Gb/s, us-east-1 / use1-az4
stores	S3 standard s3://insitubatch-bench-808047988126/…; S3 Express s3://insitubatch-bench--use1-az4--x-s3/… (same AZ as the box)
dataset	era5_fat_g16_c64.zarr: t2m, shape (12800, 361, 720) f32, sample_chunk=200, inner chunks 91×180 → 64 outer chunks × 16 stored objects, codecs BytesCodec + ZstdCodec(level=0) on standard_normal
size	13,308 MB logical / 12,307 MB on the wire = 1.0813× (measured)
insitu arm	InSituDataset, obstore store, batch_size=32, shuffle=False, no transforms, heap backing (no cache_dir); sweep block_chunks ∈ {2,4,8} × max_inflight ∈ {32,64,128,256}, interleaved per repeat; one discarded prewarm pass
control	raw obstore GET of the same 1,024 objects, thread pool, concurrency ∈ {16,32,64,128}, no decode/gather
stats	median of 5 per point
versions	Python 3.12.13, insitubatch 0.2.0 (44d364c), zarr 3.3.0, obstore 0.10.1, numpy 2.4.6, numcodecs 0.16.5

Repro

A="--n-samples 12800 --inner 361,720 --sample-chunk 200 --inner-chunks 91,180 --variables t2m"
uv run python bench/make_dataset.py --url s3://$BUCKET/era5_fat_g16_c64.zarr $A
uv run python bench/make_dataset.py --url s3://$XBUCKET/era5_fat_g16_c64.zarr --s3-express $A

uv run python -m bench.probe_ceiling --url s3://$BUCKET/era5_fat_g16_c64.zarr --repeats 5
uv run python -m bench.probe_ceiling --url s3://$XBUCKET/era5_fat_g16_c64.zarr --s3-express --repeats 5

Result

storage	best insitu (decoded MB/s)	raw GET (stored MB/s)	raw, decoded-equiv	naive % (mixed units)	% kept
S3	1471 (bc=8, mi=32)	1512	1635	97.3%	90.0%
S3 Express	1506 (bc=8, mi=32)	1537	1662	98.0%	90.6%

Full insitu grid (decoded MB/s, median of 5)

bc	mi	inflight_pk	resident	S3	Express
2	32	32	4	1159	1274
2	64	64	4	1070	1266
2	128	64 ⚠	4	1030	1290
2	256	64 ⚠	4	1078	1276
4	32	32	8	1392	1437
4	64	64	8	1181	1328
4	128	128	8	1193	1289
4	256	128 ⚠	8	1192	1300
8	32	32	16	1471	1506
8	64	64	16	1231	1443
8	128	128	16	1179	1310
8	256	256	16	1221	1268

⚠ = permit-bound (never reached max_inflight). None of the best points are.

Raw GET, S3: 1168 / 1505 / 1500 / 1512 at c=16/32/64/128. Express: 1535 / 1537 / 1535 / 1535.

Same probe on the 16-chunk era5_fat_g16 (median of 3) gave 72.8% / 83.2%. That run is too short to trust (≈3 s passes, only 2 blocks at bc=8), so larger block_chunks couldn't show its benefit; the 64-chunk numbers supersede it.

What this does and does not show

• Both arms sit at the NIC. Raw GET is flat at ~1.5 GB/s (≈12 Gbit/s) from c=32 on both buckets; the best insitu point moves ~1,390 MB/s of stored bytes. The remaining ~10% is largely a statement about this pipe. S3 vs Express is still a statistical tie at 12.5 Gb/s (Express is steadier run-to-run).
• Decode is nearly free here. zstd level 0 on noise (1.08×) is close to a pass-through. What insitu actually does per pass is GET → zstd decode on the decode pool → adopt the tile by reference (no chunk assembly without transforms) → gather stitches 16 tiles per chunk into each batch. On real ERA5 (2–4× zstd) the decoded-equivalent ceiling rises by the ratio and decode becomes the limit. This is consistent with — not contradicted by — the much lower GCS figure on n2-standard-8, which was decode-bound at ~450 MB/s. The 90% does not transfer to real data or smaller boxes.
• gather is not the limit on this box. It does two memcpys per tile (advanced-index read into a temporary, then masked write). Measured 0.34 ms/tile for a 32-row × 91×180 f32 tile → ~5.4 ms per batch, a single-thread ceiling of ~6 GB/s, ≈4× the NIC. A slice fast path for contiguous rows (shuffle=False / eval) measures 0.19 ms/tile; shuffled rows genuinely need the temporary.
• Throughput falls past max_inflight=32 at every block_chunks, on both buckets, while raw GET does not — so it is on the insitu side (decode/loop contention is the guess, unmeasured). Worth its own issue: the default may be high for fat chunks.

Known biases (all toward insitu)

• Best-of-12 insitu medians vs best-of-4 raw medians.
• The raw sweep runs as a block after the insitu grid, not interleaved — the probe docstring's "interleaved" claim covers only the insitu arms.
• Each raw-GET call builds a fresh obstore client inside the timed window (small at ~8 s passes).
• Not yet validated against a known answer — an --uncompressed store should report exactly 1.000×.

Suggested before publishing to docs/benchmarks.md

1. Interleave the raw sweep with the insitu arms, and add the uncompressed 1.000× check.
2. Re-run on a real compressed store (e.g. WeatherBench2 ERA5) and on the GCS box, so the published % isn't a synthetic-data artifact.
3. The existing 25 Gb/s table (c6id.16xlarge) used the old mixed-unit method; re-measure before comparing against these.

[emfdavid]