Repository navigation
Conversation
|
Thank you, @akol1, for the Metal family, the fp64 reference tests in #826, and an Engram token map built in Zig that matches the release byte for byte (#824). Your split on #635 is the plan. We've opened #847 to coordinate it: one package in What #824 to #827 need before they land:
This is careful work, and we want it in. Thank you again. |
|
The whole-model gate question now has a complete experimental answer, and it changes what "passing" can mean. What was measured. After the sparse-attention accumulation fix (the per-tile compensated pairs, now in the branch), the 8-token oracle dropped from 4.096% to 3.0517% NRMS with greedy agreement intact. Per-token, the error is 0.31%, 0.25%, 0.71% for the first three tokens and climbs to 5.9% by token 7 — the signature of state carried forward, not a per-call kernel error. A 12-layer same-input sweep, the MoE capture (110 comparisons) and the hyperconnection capture then came back cell-for-cell identical between the Metal chain and the reference chain; the route-region diff showed only two routing weights differing by one FP32 ULP with identical expert selections, and a decode of the cache layout confirmed our packed rows hold the same values the reference's quantization produces. What that rules out. With every captured stage bit-exact, no kernel is "wrong" in any way the operator tests can express — the residual 3.05% is the composition itself: the model's own sensitivity to FP32-level differences, accumulating through 40 layers exactly as the 2.90% FP32-vs-FP64 floor predicted. Three further experiments confirmed the direction: hardening the Engram gate's sums to wide-precision pairs, hardening the RMS reduction the same way, and switching the routing-weight sum's association each moved the number up (3.45%, 6.50%, 5.34% on a dirty run) — every deviation from the shipped arithmetic made the match worse, because the shipped kernels and the reference chain were tuned to each other. What this means for the gate. The 1% whole-model limit cannot be met by kernel work: the remaining delta is the model's FP32 sensitivity, the same term the 2.90% floor measures, and it compounds with depth. Three options, in our order of preference:
All three keep a hard correctness floor: greedy agreement 8/8 (measured on every run), the per-operator bit-exactness suite (green), and the residency gate. What we cannot do is tune kernels toward a 1% whole-model number that the model's own FP32 sensitivity forbids — the experiments above are the proof. If the gate-form decision waits, the pieces are complete as they stand: #824's golden test, #825's host tests, #826's per-operator bit-exactness including the new PV pairs, and #827's served-parity legs. The whole-model receipts for everything above are on the branch (oracle-postfix, the 12-layer sweep, the MoE/HC captures, the per-token tables). |
|
@ashhart — the two decisions that gate this series are now consolidated with explicit options on the tracker: the whole-model gate form (three candidates, the evidence above, our pick = the floor-relative bound) and the consolidation sequencing for the four shared files. Linking here so the answer lands next to the measurements: #847 (comment) |
|
Thank you, @akol1, the pinned whole-model result and precision-floor experiment keep the unresolved choices concrete. This stays with the one DeepSeek family, after shared infrastructure. We will also repair the checked verify/partial-keep transaction ownership before enabling real drafted serving. Please answer both questions on #847: which proposed whole-model gate keeps or replaces the 1% NRMS bound, with exact thresholds and long-token greedy controls; and which Engram storage/scales both loaders should read, using pull-time conversion or a pinned prepacked format. We will keep the two answers together on that tracker. The measured 3.0517% and current draft/image/state gaps remain visible until that gate is settled. Posted by Ash's AI Assistant Synthena |
cfd6f8c to
45e8c06
Compare
Require TF_EXPERIMENTAL=deepseek while whole-model gate #827 is pending. Check both checkpoint spellings at the native boundary, the dedicated server and the whole-model factory before device or payload work. Keep the existing family implementation and shared infrastructure, and label the release notes and known issue accordingly.
…-plan test calls math.zig's wide and b64 arrived twice through the two worktree lines; the shard-plan tests now pass the index-ownership argument the loader gained.
The reference state now probes every scored window: the contract's boundary margin (BF16-rounded FP32 scores), the FP64-ideal margin on the same inputs, and the top-k pick flips between them — the real-data support-flip measurement. The poolOf call takes the partials by address; the oracle path had never been compiled since the runner landed, which the probe's first build exposed.
… scratch, verbatim The margins mode prefills a long prompt, then runs a single diagnostic token with the observer hooking each indexer layer, copying the selection scratch (score tiles, heap, positions) out verbatim. The offline decode reads the score tile directly: at 2600 tokens the boundary of the 512-pick selection sits inside an exact tie of the BF16-quantized scores at every writable layer — the support-flip condition the synthetic harness predicted, now on real weights. The candidate-masked layers' tiles are unwritten; their boundaries need the candidate-region decode.
Model.open takes the lane count: every slot owns a full packed-cache State and its 16-byte Engram history slice, and forwardOn/resetSlot run one slot explicitly while slot 0 keeps the single-lane callers unchanged. Admission adds (lanes - 1) cache payloads. tf-dsv41-run lanes interleaves two slots' chunks against their solo runs and requires bit-identical logits, the state-level concurrent == solo check.
The truncated struct let the kernel's 296-byte write smash 152 bytes of stack, so every mode died with a wild jump right after its forward — the four crash reports show the gpu-seconds value landing in the call target. The full layout is the one process_memory's test pins. The two-slot lanes receipt now passes: interleaved chunks against solo runs, zero bit disagreements, zero execution disk reads.
Each verify window runs as its slot's own sub-forward, so a row's bits are its solo bits whatever shares the round; drafts off, greedy draws through a handle ring, and a kept prefix shorter than its window rolls back and re-feeds the kept rows. Facts report the workspace row budget as the exact width; trees, held drafts and grammar masks are refused until their pieces land.
Open builds the model with the requested lanes, the token map beside the checkpoint and the tables from --engram-table-dir, under the one-Mac load limit: 70% of RAM by default, GLM_LOAD_LIMIT_GB to opt past it, refused past the GPU's working set. Greedy only, drafts off, prompt caches and learning refused for now; the engine registers as deepseek_v41/exl3.
The V4.1 lane host reads the original release's Engram tables through the flag; the load refuses past the limit with the opt-in named. Served smoke on the lane backend: up in 115 s under GLM_LOAD_LIMIT_GB=430, a greedy chat completion answered READY with thinking and sane usage.
The builder moves into the family module behind tf.deepseek_v41_token_map.build; the CLI is a thin wrapper. When the checkpoint has no engram-token-map.bin but the original release's tokenizer.json and config.json sit beside the Engram tables, the lane host builds the map at open — the pull no longer needs a Python step. The tokenizer becomes a shared module so tokenizer.zig never lands in two modules of one compilation.
What one Mac needs, the token map the engine builds itself, the serve line with the lane semantics and the honest --no-drafts, the residency gate, and the numerical picture: the per-operator rounding-level tests, the sensitivity floor against the whole-model error at two quantizations, and the measured selection-boundary ties.
…ests The generated nfkd_data.zig now comes from tools/zig/nfkd_tables.py, a peer of tools/zig/unicode_tables.py: it rebuilds the table from unicodedata 16.0.0 and zig fmt, and its --check fails on drift. The table is one file with a generated header, and lean_check.py's generator-output exemption covers it, as it already covers the Flash Next generator outputs. The normalizer chain and the PCG64 draws keep their tests; the Hangul case expected a recomposed syllable, NFKD decomposes it and the release's chain never recomposes, so the expectation and a comment say so. The program was registered only as an installable executable, so its test blocks never compiled under zig build test, which is how a wrong Hangul expectation survived. The program table now takes host_tests, and the test step depends on the program's test artifact.
operator_csa2_fp64.zig (1331 lines) and operator_hc_fp64.zig (1095) sit above the 600-line file budget. Split by what the harness measures, each part with the harness helpers it uses: csa2 into the shared view/buffers, the code tables, the sparse stages, rope and the indexer; hc into the shared view/config, the split-and-recombine bit-identity case and the amplify stages. The reference root imports every part in dependency order, so the tests.deepseek_v41_reference. filter and the suite counts are unchanged. No tolerance, assertion or receipt text moved.
…e line each Folds the remaining comment groups into single lines of at most 120 columns, keeping the technical content. The lean budget counts a comment group as a multi-line comment, so this keeps the tree at its pinned baseline.
…ment `lanes` sat in the six-or-seven argument list, so the eight-argument form with CHUNK and CONTEXT was rejected before the parser could read it, and the usage text never listed the mode at all. Remove it from that list, document the mode's chunk and context arguments, and cover the accepted and rejected arities with a host test now that the runner registers its tests.
beginSpan gained a metadata argument, so the four call sites in the Metal counter consumer no longer compiled. zig build test does not build that consumer, so nothing caught it. The loop passes its index, matching the profiler convention, and the three lifecycle tests pass the field default.
1.1.2 replaced the nested text_config with one flat Config and renamed its fields, so every reader in the Metal line moves to the new names: rms_norm_eps to eps, rope_theta and the scaled rope settings to rope.*, and the layer-id, vocab and width fields to their short names. The engram table directory is a fixed-capacity byte list now, so the table plan takes its items and treats an empty list as absent. metal_targets registers the operator qualification steps beside the other families' so the device, weights and reference suites build and run again.
604122a to
636030f
Compare
|
One design question on the shared tier, small enough that I would rather ask than guess. Our branch does not build Why. This is not new: the same failure is on #1051, and it predates 1.1.2 on our fork. It is also not in What I tried, and why I backed it out. Rooting the step at The question. Which of these do you want for backend-specific loaders that need to appear in a
I am not proposing a branch before you pick, and I am not asking you to fix it: it is our file |
|
Thank you, @akol1, for the Metal lane adapter and its slot diagnostics. In the integration described on #825, Metal serves through main's shared lane machinery with drafts on, so the drafts-off backend won't land as written (
|
|
Zig 0.17.0, M5 Ultra / 256 GB, cold cache, tree = the head plus the changes named below.
Result, cold: Converting the imports instead costs the other roots. Its root imports Scope: the same relative-import pattern appears in ten families at this head — flashnext 53, glm 39, nemotron 38, gemma4 27, glm_dsa 19, step5 16, deepseek_v41 14, lfm2_moe 12, qwen3_5 7, qwen_image 6, 231 in total. On Untested alternative, for what it is worth: |
… picks Two things in the reference checks needed fixing: - operator_csa2_fp64.zig printed its Metal-vs-FP64 comparison but never asserted it, so any regression in pool reductions passed silently. Turn each case into a PoolDelta carrying the Metal NRMS, the FP64 reference NRMS and the Metal-vs-reference NRMS, run six cases (typical, wide dynamic range, cancellation, gate skew, balanced cancellation, and the 1e7 cancellation case), and assert each stays under the contract floor and that Metal is not materially worse than the FP64 reference. - operator_csa2_index_fp64.zig derived its overlap count from its own reference mask, so 512 copies of one valid key would have counted as full coverage. Sort the reference picks, build the chosen set from that, and count repeats and kernel-only picks over the kernel's own output; assert zero out-of-range picks, zero repeated picks, zero kernel-only picks, and that any weight flip sits within two quanta of the boundary.
…re its own tests
test-dsv41-shared did not build: the family is a module rooted at its own
directory, so its relative `../../core/*` imports reached outside the module
path, and where they resolved they gave the same file a second identity
next to the one `@import("core")` already created (names.zig used both).
Reach core by name. zig/dsv41_core_family.zig is that name's surface: it
re-exports exactly the ten core modules the family uses, and both build
lanes point their `core` at it. The family's fourteen relative up-imports
become `@import("core").<m>`, matching what upstream already does in
names.zig and elsewhere. The lanes thread that module through weights() and
checks(), and dsv41_shared.zig gives it `metal` so gpu_counters resolves.
The cost is that a test block reached through a named module never runs, so
core's files cannot be path-owned and module-owned in the same binary. Give
core one place to run instead: test-core, rooted at zig/src/core/root.zig,
with the six core test files that were only reached transitively now named
there. Every test the Metal lane used to collect by path still runs --
gpu_counters, expert_groups, safetensors and the rest in test-core,
exl3_projection_gpu_test in test-exl3-gpu -- and csa2's check moves from its
own file to kv_test.zig so it stays in the same binary.
Register the dsv4_vision CUDA family (lane engine, rank-1 worker, DSpark drafts, distributed prefix snapshots and HTTP vision) without family code in zig/src/server: the templateless-chat bridge crosses native/cuda_chat.zig on std primitives only, engines.zig probes it behind @hasDecl, and hf_text takes one neutral chat parameter. The release gate opens deepseek_v4 only under TF_EXPERIMENTAL=deepseek and the checkpoint's own vision_n_layers fingerprint; deepseek_v41 stays behind ashhart#827. Lane admission gains the generic max_lanes capability, threaded through page planning, host placement and admit. The prompt encoder moves to the family (serve/template.zig with golden parity) and dsv4_prompts goes with it; build shares one json module between the server root and the family; tf-dsv4-verify gates to linux with its POSIX link. Adds the TP2 recipe, cluster contract tests, refusal matrix, images guard and CLI format note. Host suites and lean (442/442, 0 new) green; hardware qualification on two Sparks pending.
Route DeepSeek-V4.1 checkpoints without Jinja through the existing native prompt renderer. Preserve checkpoint templates and explicit overrides, thinking choices and text-only image refusal. Request bytes, tool names and serialized arguments remain masked by the added-token encoder. Return HTTP 400 for prompt-template refusals before engine submission. Carried onto main: server/hf_text.zig:341 selects the family fallback; families/deepseek_v41/serve/template.zig:105 tracks request ranges at output rather than reconstructing provenance from marker spellings. Existing experimental admission and whole-model release gate #827 remain. Host regressions cover 178 committed encoding.py text goldens, control spellings, the actual OpenAI prompt path and missing-template HTTP status. Addresses #1208; no checkpoint or device qualification is implied. Reported-by: flashosophy (#1208)
Piece (d) of the #690 split, rebased onto the 1.1.0 release (
76aafcf27) and stacked on the family and test pieces: the lane backend on the 1.1.0 lane core — one stream per slot, windows of rows — so prompt caching, concurrency and drafts come from the core. facts() reports exact_width = the plan's rows, drafts 0 (greedy only, as GLM-5.3-Flash landed), max_streams = lanes; drafts, grammar masks and tree requests return error.Unsupported until their pieces land.after.disk_read == before.disk_readover that full struct.--engram-table-dir, and the token map built at open when missing (the pull needs no Python step; the builder is the first piece).docs/deepseek-v41.md): what one Mac needs, the serve line, the lane semantics, the numerical picture.Rebase notes: the branch now rides 1.1.0's rewritten shared infrastructure —
paced_read's admission gauge replaces the olderwaitForRoomform, the tokenizer module is owned once by the engines registry, and the families table inmetal.zigis the merged one. A comment-only tidy commit brings the reference, family, native and Metal comments onto one line each within the 120-column rule.The usual checks, measured on the 2.8bpw pack served on one Mac: drafted == "draft": false (speculative.drafted 0 with drafts requested, content identical to --no-drafts), resumed == fresh (a conversation arriving across two calls equals the one-shot full-context reply), concurrent == solo (two lanes, Paris/Paris and 1024/1024). The lane receipt shows 0 bit disagreements between slots and their solo runs on both packs; each receipt carries
evidence_origin: current_invocation_not_historical_qualificationand a residency gate over the full rusage counters.Suites at this head:
test-deepseek-v4179/79 (41 steps),test-deepseek-v41-reference26/26,test-deepseek-v41-weights13/13,zig build native -Dcpu=apple_m154/54 steps,zig build testexit 0, 307/307 steps (4440/4455 cases passed, 15 pre-existing platform skips, fresh cache);tools/zig/lean_new.py501 problems against 503 on the 1.1.0 base, 0 new;zig fmt --checkclean; the built token map is byte-identical to the release's. The runner documentslanes MODEL TABLES IDS_JSON OUT_DIR [CHUNK] [CONTEXT]and a host test pins the accepted arities: six to eight arguments parse, five or nine are usage errors.Receipts live on the fork beside the branch; the tree carries the tests that regenerate them.
Status on the 1.1.2 line. This head is unchanged and still rebased onto
76aafcf27(96 commits behindd1018b318), so it shows as conflicting; it is not retargeted here. The shared lane host and token map already landed in 1.1.1; the DeepSeek-specific lane host, guide and mHC evidence in this PR stay the piece that waits on the serving gate tracked here. Refreshed family base for review:pr-dsv41-bc-family-11at4f69fea10. No action requested.