Skip to content

SCORING.md

Latest commit

 

History

History
78 lines (62 loc) · 4.96 KB

SCORING.md

File metadata and controls

78 lines (62 loc) · 4.96 KB
Raw

How the five M5 Ultra predictions get scored

Each prediction is a closed range. It is a HIT if the median of five runs of the pinned configuration below falls inside that range, inclusive, and a MISS otherwise. The score is published whatever it says.

Two dates matter, and neither moves:

  • 2026-09-08 — the five ranges were published, in What actually fits in 512 GB, before the machine existed.
  • 2026-09-09 — the conditions below were published, still before the machine existed. A prediction without pinned conditions is not a prediction; a benchmark number can be moved by a factor of two by choosing the context length alone.

Neither file is edited after publication. A correction is a new dated row, never a changed one.

The rule

  1. Scored value = the median of five repeats of the pinned cell in predictions-m5-ultra.csv. Not the best of five, not the mean.
  2. Outcome is one of:
    • HIT — the median is within [predicted_low, predicted_high], inclusive.
    • MISS_LOW — the median is below predicted_low.
    • MISS_HIGH — the median is above predicted_high.
    • VOID — see below. A void prediction is published as void; it is never quietly replaced.
  3. Error is recorded as the signed percentage distance from the nearest bound, so a miss has a size and not just a verdict: 0 for a HIT, (measured - predicted_low) / predicted_low for a MISS_LOW, (measured - predicted_high) / predicted_high for a MISS_HIGH.
  4. Aggregate is hits / 5, stated in the first line of the scoring post. Five separate ranges, one headline number, no weighting and no partial credit.
  5. VOID applies when, and only when, the named model cannot be run at the named quantization on the named backend at all — no published build exists, or it exists and does not load in 512 GB. Thermal throttling, disappointing numbers, a backend bug and an unflattering result are not void conditions. If a model is voided, the reason and the attempt are published with it.
  6. All five are scored in one pass, within seven days of the machine arriving, before any paid analysis is written from the same runs.

The pinned conditions

Every row is a single cell of the matrix in How we are going to measure it, which also carries the definitions of prefill and decode used here. Prefill is prompt tokens divided by time from request start to first token, cache cold, load and warm-up excluded. Decode is generated tokens divided by time from first token to last.

id Workload Model, as it will be run Context Concurrency Cache Prediction
p1 DeepSeek-R1-class 4-bit, decode DeepSeek-R1-0528, MLX 4-bit 1,024 1 cold 25–30 tok/s
p2 DeepSeek-R1-class 4-bit, prefill DeepSeek-R1-0528, MLX 4-bit 11,000 1 cold 400–750 tok/s
p3 Qwen3-235B-class 4-bit, decode Qwen3-235B-A22B, MLX 4-bit 1,024 1 cold 30–36 tok/s
p4 gpt-oss-120b 8-bit, decode gpt-oss-120b, MLX 8-bit 1,024 1 cold 90–120 tok/s
p5 gpt-oss-120b 8-bit, prefill gpt-oss-120b, MLX 8-bit 1,024 1 cold 2,000–4,000 tok/s

Machine: Mac Studio, M5 Ultra 36-core CPU / 80-core GPU, 512 GB, mains power. The macOS build, the mlx-lm commit, the model revision hash and any iogpu.wired_limit_mb change are recorded per run and published with the raw JSON.

p2 is pinned at 11,000 tokens rather than at a round matrix cell because its baseline was taken at "~11k". Comparing our 8K against someone else's 11K would be the same trick this project exists to stop. The 1K, 8K, 32K and 128K cells are published alongside, unscored.

Three tensions in our own predictions, named before the result

Stating these afterwards would be an excuse. Stating them now makes them checkable.

  1. p4 is more optimistic than our own stated multiple. The post derives decode from a 1.3–1.5× uplift over M3 Ultra, but 90–120 tok/s against a 60 tok/s baseline is 1.5–2.0×. If p4 hits and p1 and p3 miss low, the honest reading is that we got gpt-oss-120b right by accident.
  2. p1's baseline range mixes contexts. 11–18 tok/s spans a short-prompt measurement and an ~11k-prompt one. The prediction range was set against that blend; the scoring is at 1K only, so p1 is being scored against the friendlier half of its own baseline.
  3. p2 and p5 are wide. 400–750 and 2,000–4,000 are 1.9× and 2.0× ranges. A hit inside a range that wide is weak evidence, and it will be reported as such rather than as a hit like the others.

What would void the whole set

One thing: if the 512 GB M5 Ultra does not ship, or does not reach us, the predictions stay unscored and stay published, marked pending, until it does. They are not re-scored on other hardware.