Repository navigation
Ceilings: every measured number with its physical bound and the reference - #27
Open
AdvayMonga wants to merge 3 commits into
Open
AdvayMonga wants to merge 3 commits into
AdvayMonga wants to merge 3 commits into
Conversation
…ence - validity/ceiling.py: model spec from the HF config (dense and MoE), decode/prefill floors, largest batch under a TPOT limit, per-objective regime ceilings, pct_of and Amdahl ceiling_gain. Replaces the Gemma-only roofline.py. - validity/peaks.py: HBM bandwidth and bf16 GEMM measured once per GPU+driver (python -m validity peaks, run by vm-setup.sh, fetched by lab.vm setup); spec sheet as the labelled fallback. - lab/calibrate.py: bench/baseline results gain a `calibrated` block (value, % of ceiling, % of reference, shape, fingerprint); profile records carry each kernel's share of the window; kernel records the ncu launches' % of bandwidth peak. A ceiling or reference appears only under a matching config fingerprint, else why it is missing. - lab.validate measures the reference under the regimes once into knowledge/reference/<target>/<regime>.json. - Client rows record prompt_tokens; the worker reports the GPU a measurement ran on; the brief shows the calibrated baseline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Performance reference = the correctness reference minus its determinism switches (batch-invariant vLLM, deterministic SGLang); the stored reference records its launch and env. - Goodput ceiling pays for prefill compute and caps the batch by KV fitting in HBM beside the weights; sessions/s has no bound (it would need turns per session). Each basis states its assumptions. - dtype aliases (bfloat16, float16, ...); unknown dtypes are refused, not a KeyError. - The baseline gets its ceiling (seen base runs keep client rows); profile reads kernels.json at the blob root. - Calibration never raises: a failure is recorded instead of losing the measurement; ncu cells with separators or n/a and unit strings like Gbyte/second are parsed. - Invalid values say why; missing reasons are facts, not commands; seed recorded in the fingerprint, not required. - knowledge/peaks/nvidia-h200.json: measured on Verda (driver 580.126): 4.12 TB/s, 794 TFLOP/s bf16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decided 2026-10-11: a number on its own ("decode at x ms") says nothing to the agent. Every measurement now comes with three denominators, as facts: % of the physical ceiling, % of the reference engine measured the same way, and for parts of a step their share of the whole (the Amdahl denominator).
What
validity/ceiling.py(replaces the Gemma-onlyroofline.py): model spec from the HF config (dense and MoE, with experts-touched per step), decode and prefill floors, the largest batch under a TPOT limit, a ceiling per regime objective,pct_of,ceiling_gain.validity/peaks.py: HBM bandwidth and bf16 GEMM measured once per GPU + driver (python -m validity peaks, run byvm-setup.sh, fetched home bylab.vm setup) intoknowledge/peaks/. Spec sheet as the fallback, labelledsource: spec.lab/calibrate.py:benchandbaselineresults gaincalibratedper regime (value, ceiling, % of ceiling, bound, basis, request shape, reference, % of reference, fingerprint).profilerecords carry each kernel's share of the window;kernelrecords the ncu launches' % of the bandwidth peak.lab.validatemeasures the reference under the regimes once (knowledge/reference/<target>/<regime>.json);--no-reference-regimesskips it.prompt_tokensfrom the server's usage; the worker's measure/profile jobs report the GPU they ran on; the brief shows the baseline with its calibration.Checked on the model card: Qwen3-30B-A3B comes out at 3.0B active / 30.5B total from its config; decode floor at batch 1 on an H200 ≈ 1.3 ms.
Not done: ncu tensor-pipe instruction rates aren't mapped to FLOPs, so kernels get % of bandwidth peak only. Needs one GPU pass: measured peaks on the H200, the reference regimes, and a check that every floor sits below every measurement.
Tests: 248 passed / 9 skipped; engine-side 42 passed; ruff clean.
🤖 Generated with Claude Code