Repository navigation
Benchmark: keep each attempt's draft counts beside the receipt - #1045
ThinkOffApp wants to merge 5 commits into
Conversation
Equal timings do not show that a drafter ran. The server already ends each reply with its speculative counters (verify rounds, drafted and accepted tokens), the same numbers as its "rounds=14 accepted=10/13" log line, but the client dropped them and a managed run's server log is unlinked as it starts. The client now reads them from the finishing chunk and writes RUN_ID.drafts.json beside the receipt: per attempt and in total, with the drafter's repo ID and whether the run passed one. The board's receipt holds exactly its community-v1 fields, so the counts stay local and the receipt is unchanged. The run prints the totals, and says so when a drafter was passed and the server drafted no tokens.
|
Thanks @ThinkOffApp; we've reviewed this draft early. Keeping the counters beside the receipt preserves its schema and separates MTP activity from passing an external drafter. The counters accept full unsigned 64-bit values, but the totals add them unchecked; two valid large counters can overflow and abort an otherwise successful run. Please bound or check the sums and add that regression. Please attach the described raw receipts and controls. We'll carry it onto the 1.1.2 line ourselves, keeping the sample's finish reason and API route beside the new counters, after that fix and our gates. Posted by Ash's AI Assistant Synthena |
…rflow panic The server's counters are full u64 values and the totals added them unchecked, so two valid large counts overflowed: a panic in a safe build and illegal behaviour in a fast one, after the receipt was saved. The sums are now checked; when one would pass 64 bits the totals are null in the counts file, as an attempt without counts is, each attempt keeps its own counts, and the run says so instead of printing totals. The receipt is unchanged. A run test drives two attempts whose counts sit near 2^64 through the fake server and expects a completed run, the null totals and the board's receipt; a unit test covers a sum of exactly 2^64 - 1 and one past it in either the rounds or the drafted tokens. No ratio is computed from the counts, so there is no division to guard.
…r logs, and the host controls docs/receipts/drafter-counts holds sanitized copies of the 18 October runs on a Bosgame M5 (gfx1151): five receipts with their .drafts.json files and console logs, the two attached runs' server logs and manifests, and the run script's log with both binaries' SHA-256. match.py checks every attached sample's rounds, drafted and accepted tokens, completion count and token hash against the server's own "done" line for that request (the README names the lines; the route and finish reason are printed beside the counts from the log), every run's totals against its samples and its printed line, and refuses two controls: one count off by one and the log read one request out of step. host-controls has the overflow fix's host test runs, the negative control that panics with the fix reverted, and the lean and fmt checks. Local paths became ~ and trailing spaces were dropped; the receipts and counts files are byte for byte the raw ones, and raw-SHA256SUMS keeps the raw hashes.
|
Thanks for the early review @ashhart. Two new commits on this branch address both points. Overflow: the totals now use checked sums. If any sum would pass 64 bits, the totals in the counts file are null, as for an attempt without counts. Each attempt keeps its own counts, and the run prints "Draft counts: no totals, as a sum passes 64 bits (k of n attempts reported counts)" instead of aborting. The run still completes and the receipt is unchanged. No ratio is computed from the counts, so there is no division to guard. The changes are in zig/src/server/benchmark_drafts.zig and zig/src/server/benchmark_run_test.zig, plus one sentence in docs/community-benchmarks.md. Regressions: a unit test covers a sum of exactly 2^64 - 1, which still gives a total, and one past it in the rounds, in the drafted tokens, or in all three. A new case in the run test sends two attempts with valid counters near 2^64 through the fake server and expects a completed run, null totals, each attempt's own counts, and a receipt with only the board's fields. With plain additions instead of the checked sums, both tests abort with "panic: integer overflow". Host tests (Apple silicon, CPU only, Zig 0.17.0, zig build test -Dcpu=apple_m1, uncached): 292 of 292 steps, 4315 of 4331 tests passed, 16 skipped. The lean check finds no new problems against main, and zig fmt --check passes. Receipts and controls are in docs/receipts/drafter-counts/, and its README describes each file: the five gfx1151 runs from 18 October (receipts, counts files, console logs, and for the two attached runs the server logs and manifests), the build-and-run log with both binary hashes, and match.py, which checks every attached attempt's counts against the server's own log line for that request. host-controls/ holds the test runs, the negative control, and the lean and fmt output. Local paths were shortened to ~; SHA256SUMS covers every file. One correction to the description: the GPU binary for these runs merged only hip-refuses-drafter (920d134) and this branch (1b4b2d7) onto main 76aafcf. benchmark-rocm-host-identity was added only for the later out-r5b run. |
… branches merged The run offered on ashhart#1001, made after the fixes: main 76aafcf with ashhart#1043 (c949cc9), ashhart#1044 (f2c78ee) and this branch (134a5f4) merged on the box and built for gfx1151. R1 to R5 repeat the 18 October set; R6 repeats R3 with HIP_VISIBLE_DEVICES=0. Every receipt names the CPU, the one GPU and the checkpoint; every run's counts match its printed line, and R5's 20 samples match the server's own log lines, while a count off by one and a log read one request out of step are refused. Both --drafter calls are refused before any weights load. The board's validator refuses each receipt as rocm and accepts it as cuda, as on 18 October.
61e1fb1 to
390b17e
Compare
2 participants
For #1001 (1.1.1). Branch
ThinkOffApp:benchmark-drafter-counts, onmainat76aafcf2. One commit. It does not depend on the other two #1001 PRs, and all three merge cleanly together.What and why
Issue #1001 points out that equal timings do not show that DFlash2 ran: a run has to show that its drafter ran. The server already reports this. Every reply ends with a
speculativeobject (rounds,drafted,accepted, from the engine'sStats). These are the same numbers as the server'sdone ... rounds=14 accepted=10/13log line. The pinned gfx1151 receipt (docs/receipts/gfx1151-qwen-833bbfcb) counted drafted and accepted tokens from that same telemetry. But the benchmark client dropped it, and a managed run's server log is deleted as soon as the server starts. So nothing kept the counts.The board's receipt has exactly the community-v1 fields.
protocol.jsrejects any extra key in a sample, and so does the client's own save check. The counts therefore cannot go in the receipt without a board change. This PR keeps them in a separate local file:speculativefrom each attempt's last chunk. It keeps the counts only if all three are non-negative integers andaccepted <= drafted.RUN_ID.drafts.jsonis written next to it (for--output FILE.json,FILE.drafts.json). It is written the same way as the receipt: mode 0600, to a temp file that is then renamed. It holds the run ID, the receipt's name,drafter_repo_id,drafter_passed(true or false for a managed run, null for an attached one), totals, and one row per receipt sample: fixture, temperature, repeat, status, completion tokens, rounds, drafted, accepted.Draft counts: A of D drafted tokens accepted over R rounds (k of n attempts reported them); saved: PATH, orthe server reported none. When a drafter was passed and the server drafted nothing, it adds:A drafter was passed, but the server drafted no tokens: this run does not show that the drafter ran.Open question for the maintainer: should the board get a field for these counts in a later receipt schema? And should a run with a passed drafter and zero drafted tokens fail (for example with exit 2) rather than only print the note? We left the exit codes unchanged.
Tests
Host tests on an M-series Mac, CPU only, Zig 0.17.0,
zig build test -Dcpu=apple_m1 --summary all:main76aafcf21b4b2d7bFull count on this branch: 4313 of 4329, with the same 16 skips. The server binary runs twice in
zig build test. All three PRs for #1001 also build and pass together on one merged tree.benchmark_sse_test: a last chunk withrounds 14, drafted 13, accepted 10reaches the sample. Nospeculative, accepted past drafted, a missing field, a negative or fractional count, or a non-object all give no counts, and the sample still passes.benchmark_drafts(two tests): the file name next to the receipt; totals over only the attempts that reported counts; the note shows only when a passed drafter drafted nothing; the JSON rows, including null counts for a failed attempt.benchmark_run_test."the draft counts beside the receipt show...": managed runs against a fake server, with a cached DFlash2 for a 27B-shaped config:6 of 8 ... over 6 rounds,drafter_passed: true, no note;--drafter none, 0 drafted:drafter_passed: false, no note;speculative, or accepted past drafted:reported none.In every case the counts file is mode 0600, and the saved receipt still loads under the board's rules with no extra sample keys.
Negative controls. With the SSE read removed, the SSE test and the run test fail. With the zero-drafted note removed, the drafts test and the run test fail. With them back, all tests pass.
GPU receipts
Measured on 18 Oct 2026 on a Bosgame M5 (AMD Ryzen AI MAX+ 395, Radeon 8060S,
gfx1151). The source wasmain76aafcf2with the three #1001 branches merged (5906a3f7,920d1348,1b4b2d7b; merge7b84f909), built withzig build native-hip -Dhip-arch=gfx1151 ... -Doptimize=ReleaseFast, Zig 0.17.0.tensorfold-nativesha256e279e29a54b56dc81ff70985f8c4cffaa0d9aff4a7456a234eb7247667492247. Checkpoints were pulled into the Hugging Face cache at pinned revisions:TensorFold/Qwen3.8-27B-MLX-4bit@70ae7fac...andTensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@f84b054c....z-lab/Qwen3.8-27B-DFlash2is cached at50307d4c.... The machine's own LLM service was stopped for the run.Every run is community-v1 at defaults: 20 measured attempts, all
ok, and every attempt reported counts (reported == attempts == 20).--serialserve--drafter auto--serial. So the counts follow what the engine did, and the timings are not needed as evidence.done ... rounds=R accepted=A/Dlines: 4 warm-ups and 20 measured attempts. The 20 rows of35b-attached.drafts.jsonare exactly those 20 lines, in order (for example106 150/246,89 166/249, ...,189 67/145).cudapass the board'sprotocol.js; the real receipts stayrocmand are not publishable until the board acceptsrocm. The counts files are mode 0600 next to them.Full local receipts: the four receipts, the four
*.drafts.jsonfiles, run logs, the 35B serve log and the manifest, with SHA-256 sums (for example35b.drafts.json9bb8baa0...,35b-serial.drafts.json92fe121f...). They can be attached on request.