Skip to content

dsv41 metal: indexer scores, top-512, candidate blocks and the MoE route - #1712

Closed
gilby wants to merge 1 commit into
ashhart:mainfrom
gilby:dsv41-metal-pr/08-indexer-topk-route
Closed

gilby wants to merge 1 commit into
ashhart:mainfrom
gilby:dsv41-metal-pr/08-indexer-topk-route

Conversation

@gilby

@gilby gilby commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Indexer scores (index.metal), the top-512 (topk512.metal), the candidate blocks (cand.metal) and the MoE route (route.metal) on Metal, each registered with the kernel contract. The route select's two launch records live in the family's dsv41_route_records.zig, so registry.zig is unchanged by this part.

The route select carries planted controls too, on both its backbone and drafter variants: a higher id winning a tie, a NaN key winning, the bias added to the weight, and a write past k. Each compiles away at TF_CTL 0 and is required to turn its gate red.

Main's shared files change only by one-line hooks; this part's records, suites and limits go into the family's own files (dsv41_metal_records.zig, metal_dsv41_main.zig, metal_dsv41_limits.zig).

Part 8 of 18 of the stack for #1676; stacks on dsv41-metal-pr/07-glue-kernels (part 7, #1711), on main with parts 1 to 5 (#1705 to #1709) merged.

Receipt

  • Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold 115c07a on main ad98b60 (native Zig 0.17.0; no MLX or PyTorch); Apple M3 Ultra (60-core GPU), 96 GB, macOS, for the kernel and device tests; checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a: kernels only; the kernel contract checks each launch geometry's bits against the others and against float64.
  • Decode speed before and after, from tools/bench_openai.py: measured on the PR that turns serving on.
  • Prompt speed before and after: the cold first request's ttft_s from tools/bench_cache.py --require-cold on a long prompt, or "n/a" with the reason when the change cannot reach prompt processing: measured on the PR that turns serving on.
  • Tests run, with pass, fail and skip counts: zig build test tf-kernel-contract -Dcpu=apple_m1: exit 0, 621/621 steps succeeded; kernel contract, tf-kernel-contract --only dsv41 (Metal): SUMMARY 56 kernels: 56 pass gates 1-4, 0 fail, 0 known findings; 113 controls: 113 red as planted, 0 missed; 0 limit problems; gates 116592 ms, limits 154 ms; python3 tools/zig/lean_check.py: 5059 files, 424 problems, the same count as main (5027 files, 424). Zig's summary counts only the test binaries it ran in that invocation, so the step count is the comparable figure.
  • Not run, and why: CUDA: no CUDA code changes; Metal on M1, M2, M4 and M5: one M3 Ultra only (the contract's limits check covers the M1 to M5 pipeline limits).

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  • No precision traded for speed. If the bits change, the description says which sums change and why.
  • A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt.
  • Prompt processing is no slower than the last release.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

@gilby
gilby force-pushed the dsv41-metal-pr/08-indexer-topk-route branch from e6e8cab to 8a6d3c9 Compare November 1, 2026 04:45
@gilby gilby mentioned this pull request Nov 1, 2026
6 of 9 tasks
@ashhart

ashhart commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Thanks @gilby. Rebased and clean, with the route's records kept in the family's own file, so registry.zig stays as it is. 49 kernels and 91 controls red. This lands after #1711.

@gilby
gilby force-pushed the dsv41-metal-pr/08-indexer-topk-route branch 2 times, most recently from 68bc6b1 to 2a123e2 Compare November 2, 2026 07:49
@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @gilby. One small change: tf_dsv41_route_select (backbone 384x6, drafter 128x3) has a float64 reference and planted tie and NaN/inf rows, but no planted control, so a broken kernel couldn't turn it red. Please add controls for a higher-id winning a tie, a NaN winning, the bias added to the weight, and a write past k. Everything else checks out: ix_scores/ix_scores_dec, top-512 (NaN as -inf, -0 as +0, ties to the lower index) and tf_cand all have float64 gates with planted controls, and nothing outside the DeepSeek Metal path changes beyond the contract runner's two-line hook.

@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Our Metal kernel contract over the stack (tf-kernel-contract --only dsv41, head #1713, M5 Ultra): 65 kernels pass gates 1-4, 0 fail, 0 known findings; 121 controls, 121 red as planted, 0 missed; 0 limit problems. That clears the device side for #1711 to #1713. They land in order after #1710, with #1712's one ask (route_select controls) first.

@gilby

gilby commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ashhart. Added the four planted controls to tf_dsv41_route_select: a higher id winning a tie, a NaN key winning, the bias added to the weight, and a write past k. They sit behind TF_CTL and compile away at 0. Both variants (backbone 384x6, drafter 128x3) are registered, so the control count at this part is 113 (it was 105). Each fault was planted in the kernel by hand and the gate went red, then restored. The receipt in the body is at this head (115c07ad, on main ad98b602).

index.metal scores FP4 queries against FP4 keys in one stated order with causal and block masks, topk512.metal
keeps each row's top 512, cand.metal picks the candidate blocks as a byte mask, and route.metal takes the top-k of
sqrt(softplus(logit)) + bias for the backbone and the drafter. Each is registered with the contract.
@gilby
gilby force-pushed the dsv41-metal-pr/08-indexer-topk-route branch from 115c07a to 98964c1 Compare November 3, 2026 09:39
@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @gilby. This part landed on main with #1719's merge (cfcbecb), which carries #1712-#1719 as your own commits in order; closing it as included. The same Metal contract, full test and DeepSeek device suite cleared the whole stack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants