Repository navigation
Conversation
Owner
|
Thanks @laudney. The engine side holds: sigmoid(yes - no) from the label logits, an f64 dot of the final-normed BF16 row with the stored lm_head rows, no new kernel, checks before any GPU work, stable ranking, other routes refused by name on a reranker, and other families unchanged. Four asks, two of them our decisions:
Smaller points: rerank is detected from a file in the model folder rather than declared by the engine; the label rows are read on the first request; every server now answers |
Owner
|
Thanks @laudney. An independent review on current main (ef9fa70): Applying it:
Nothing was built or run here. The thread has no reply from the author since our 04:03 comment. The only change is The four asks1. The response shape (your decision: #1895's): met. Commit 30 (
The request now takes #1895's fields beside SGLang's:
Tests pin the key order, the count of 3 keys per result, 2. The FP32 oracle (your decision): not met.
3. The model card's formatter: not met.
4. Docs: not met.
New in commit 30, for the lead
Unchanged since our earlier review (small points still open)
What holds
Device cells (after changes 2-4)
|
Contributor
Author
|
Thanks @ashhart. All four asks and the smaller points are addressed in five new commits on top. Nothing was force-pushed.
Fidelity against transformers float32 (same pinned checkpoint and Transformers 5.22.1, 32 cases from 16 to 8,199 tokens, served end to end on an AMD Radeon AI PRO R9700):
Restated bound. #1895's 1e-5 in score is not reachable with BF16 activations: transformers' own BF16 run sits 0.0108 from FP32 on these cases. So against FP32, the oracle now gates at transformers' own BF16 distance, rounded up: |yes − no| ≤ 0.125 and |score| ≤ 0.0125. A ranking may differ from FP32's only for a pair FP32 puts within 0.25 (twice the diff gate). Each swap is printed, and a swap beyond that fails. TensorFold passes. The BF16 reference keeps its gates (worst 0.108 / 0.0082) for the two window cases (32,768 and 40,960 tokens), which the CPU FP32 run does not cover. Speed. This PR keeps the dot2 GEMM tile. Nearly all of the 6.3x is the seven BF16 projections, which run at about 19 TFLOPS on dot2. The follow-up is a gfx12 WMMA dense tile: a separate PR for #1977. Its probe reaches 39 to 58 TFLOPS on these shapes, and its gate is a per-projection FP64 comparison against dot2. |
laudney
force-pushed
the
pr-qwen3-reranker-hip
branch
from
November 3, 2026 09:55
ef3daab to
2d18324
Compare
Contributor
Author
|
Thanks @ashhart. Your 08:27 review and my 08:27 reply crossed: the review read commit 30 only. Since then the PR is restacked on the rebased #1978 (a56bda4), as you asked. That was a force-push, and the old tip ef3daab is kept. It is 18 commits, all noreply, with CHANGELOG-only conflicts in the restack.
Fidelity against transformers float32 (pinned checkpoint, Transformers 5.22.1; 34 cases from 16 to 40,960 tokens; served end to end on an AMD Radeon AI PRO R9700):
Restated bound. The 1e-5 bar in score from #1895 is not reachable with BF16 activations: transformers' own BF16 run sits 0.0108 from FP32. Against FP32 the oracle therefore gates at transformers' own BF16 distance, rounded up: |yes − no| ≤ 0.125 and |score| ≤ 0.0125. Rankings must equal FP32's except for a pair FP32 puts within 0.25 (twice the diff gate, the most two documents within it can move). Each such swap is printed, and a swap past 0.25 fails. On these inputs identical rankings are not reachable for BF16 either: transformers BF16 itself swaps one pair. The two pairs are synthetic river-trade paragraphs whose FP32 scores differ by 0.0017 and 0.0023. Device cells (gfx1201, R9700, Qwen3-Reranker-8B at the pinned revision):
Speed. The follow-up gfx12 WMMA dense tile is in progress (separate PR, #1977). On this card it runs Qwen3-8B's projections at 45 to 67 TFLOPS at 276 rows and up to 99 at 2,048, against dot2's 22 to 30. |
Owner
|
Thanks @laudney. The re-review finds all three asks met. The open points are settled #1895's way: On the bound: accepted. #1895's 1e-5 in score isn't reachable with BF16 activations, since transformers' own BF16 run sits 0.0108 from FP32. So the bar is that distance, rounded up, and this sits inside it on every summary number except the score median (0.00023 against transformers BF16's 0.00014). The ranking difference (two near-tie pairs in another order, FP32 gaps of 0.0077 and 0.0119) is the same kind as transformers BF16's own one. Please put the bound in the PR body and the recipe, so a reader sees what "matches" means here. This lands right after #1978, which it's stacked on. |
laudney
added a commit
to laudney/TensorFold
that referenced
this pull request
Nov 3, 2026
The review asked the embedding gate to follow the rule accepted for the reranker in ashhart#1981: transformers' own BF16 distance from fp32, rounded up, not twice it. The oracle and the GPU test now take the BF16 bar's worst cosine distance at each width, round it up to two significant digits with exact power-of-ten steps (never below it), and cap it at 0.001, so the limits are 1.6e-4, 1.7e-4 and 1.9e-4 at widths 4096, 1024 and 32; --bar-factor goes, and the recipe states the new limits beside the posted distances, which already meet them.
Contributor
Author
This was referenced Nov 3, 2026
5 tasks done
A checkpoint published in BF16 widened to fp32 by quant.read takes twice its memory and needs an fp32 kernel; Qwen3-Embedding-8B would not fit a 32 GiB card that way. quant.readStored reads like read but keeps a BF16 or FP16 dense tensor as stored, dense.Matrix carries the weight's dtype for the kernels, and dense.reference reads each element by its dtype. Callers of quant.read keep fp32 weights and their bits; the host tests cover both reads, views and the halves.
The repository's lean checker allows only single-line comments of at most 120 columns, and the stored-dtype change gave the dense format's module comment a second line. The comment now says the same in one line, so the checker adds no problem over main.
A BF16 checkpoint kept as stored needs a prompt GEMM that reads BF16 weights; HIP had only the fp32 one-row dense kernel. A BF16 weight decoder plugs into the existing dot2 GEMM tile with unit scales and no bias path, which the tile now leaves out for such a decoder; the MLX tiles compile to the same code. project dispatches a dense weight by its dtype, and the dense embedding gather reads BF16 tables. GPU tests check it against float64 on gfx1201.
The HIP qwen3 forward runs one shape of Qwen3 dense model, so its config reader takes the geometry and constants from config.json, defaults head_dim to 128 as Qwen3Config does, and refuses by name the features the forward does not run (projection biases, sliding window, RoPE scaling, head_dim above 256, a non-SiLU MLP, uneven GQA groups), with host tests on the 8B embedding and reranker configs.
Qwen3's attention has no output gate, so nothing turned the causal kernel's fp32 (heads, len, d) rows into o_proj's row-major input; tf_heads_rows does that transpose with one rounding to the output kind, reached through Ops.headsRows, with an exact kernel test, a kernel contract (copy and round, served Qwen3-8B shapes) and its compiled-usage record on all four RDNA chips.
The embedding and reranker checkpoints share one Qwen3 dense body, so the qwen3 family now uploads it as stored (BF16 projections, fp32 norms, an optional model. prefix), reads lm_head rows on the host for the reranker, and runs lastRows: inputs batched layer by layer, each attending only to itself from position 0, final-normed last rows out. It adds the module, host tests and an env-gated GPU check of both checkpoints against transformers.
Qwen3-Embedding-8B is trained for any output width from 32 to 4096, and the embedding route only took widths from a declared list, so a request for 384 or 2049 would be refused. The lane's embedding Info gains mrl_min: with it set, every width from that one to the full dimension crops; the list rule and the refusal without either stay. Tests cover both bounds, widths between them and the old rules.
Qwen3-Reranker-4B ties its output head to the embedding and stores no lm_head.weight, so readRows of the head failed for it; the model now records the stored name of its head rows (lm_head.weight, or the embedding as the checkpoint names it when tie_word_embeddings is set) and the GPU check reads that name, with host tests for both layouts.
The embedding route and lane contract had no device backend, and the HIP registry no qwen3 family. A prompt-only engine sizes its scratch to the memory budget, shrinking the window to fit, and a lane backend runs each batch's whole prompt in one step and returns the pooled last rows, unit-normed when modules.json lists Normalize; the head is read from the sentence-transformers folder. The registry gains qwen3, a family's Loaded can say it serves embeddings in its own window, and one-GPU families refuse --tp before any group forms.
The embedding gate compares the HIP engine with the model card's Transformers example, so the reference must come from known sources. The oracle pins the checkpoint revision, the SHA256 of every file in the snapshot, the Transformers 5.22.1 package tree and its Qwen3 files, checks them before any model code runs, writes them into each reference, and the server check refuses a reference made otherwise. It also checks distance, norms, batch and token-ID equality.
Loading Qwen3-Embedding-8B took about 400 s, warm cache or not. The loader released the checkpoint after every layer, and that release also drops each whole shard from the page cache, so every later layer's upload faulted its pages back in from disk. The loader now keeps the mapping until the checkpoint closes at the end of the load; the same server loads in 4 s and serves the same vectors.
Every kernel must pass the contract's gates before it lands, and the BF16 dense GEMM had only its unit test. It now runs gates 1 to 4 for fp32 and BF16 output at Qwen3-8B's projection shapes, on BF16 builds, and gate 5 records its compiled usage on gfx1100, gfx1151 and gfx1201. A record can mark a chip not served, as gfx1030's FP16 build keeps only the kernel's trap body; the audits skip such a chip for a launch, never for a planted control.
The embedding gate's limit comes from the BF16 bar's distance to the fp32 target, so a bar of other inputs or a non-finite vector could loosen it or make it pass on NaN. The oracle checker and the GPU test now refuse references that differ in dtype, texts, token ids or pins, or hold a ragged, zero or non-finite vector, and fail on any non-finite distance before comparing.
The unit test and the kernel contract bounded the BF16 dense GEMM's error with different rounding chains, and the contract's ignored the fold every 128 inputs, allowing about 28 times more error at K 12288. Both now take one chain from the launcher: a group's dot2s, then one add a group. Every contract shape still passes within it.
The GPU gate checked only the revision and the Transformers tree of its references, so a record without its checkpoint, weight or library hashes still passed where the oracle's checker refuses it. Both references must now carry the complete record, the same in each, and a refused reference fails without printing a diff.
Two references could carry the same wrong pin maps and still pass the GPU gate, which never looked at the checkpoint it loads. The test now hashes every pinned checkpoint and weight file in the directory it loads and requires every shard of the index to be pinned, before any GPU work; a host test refuses a wrong hash and an unpinned shard.
Probing SGLang's Qwen3-Embedding-8B server with the same requests as this one showed two gaps: whitespace-only text was embedded, and one input longer than the context was refused only as an over-full batch. Text that is empty or only Unicode whitespace, as Python's strip sees it, is now refused with its index in a list, and an input past the context is refused with its token count before the batch's own limit applies.
The BF16 dense GEMM runs only where the build has BF16 activations; on an FP16 build (RDNA2) its kernel keeps only a trap body, and the qwen3 engine failed late as a scratch count error. It now refuses before the checkpoint loads, naming the GPU it was built for.
The new qwen3 HIP family needs a recipe like the other native models: what it serves and refuses, how to build and serve it on gfx1201 with ROCm 10.1's hipcc, the request form the model card uses, and the pinned oracle and GPU gate with their results on a Radeon AI PRO R9700.
Review of ashhart#1978 asked for the request refusals this branch adds to every embedding family to be written down. A new Embeddings section in docs/api.md lists the whitespace-only text refusal, the over-long input that is named with its token count and the context_length_exceeded code before the batch limit, the batch limit itself, and how dimensions crops to an MRL width, and the CHANGELOG gains the Qwen3-Embedding-8B entry that points to it.
Review of ashhart#1978 asked for the pinned fidelity set to cover 2K to 32K inputs and MRL widths 32, 1024 and 4096. The oracle adds five long inputs, each the longest whole-line prefix of a pinned file within 2K to 32K tokens, pins every input's text and token IDs by SHA256, runs the long ones alone at full length, and checks a server in requests that fit its window, at full width and at each MRL width against the cropped fp32 vector; the GPU test covers the same nine inputs, and the recipe records the measured distances.
Merging main put hip_records.zig past lean_check's 600-line limit, so the tf_heads_rows record joins the BF16 dense GEMM record in hip_records_qwen3.zig, whose tests are now collected from the contract module's test block; the record values and chip order are unchanged. The oracle's new comments and one docstring become single lines, so lean_check reports the same problems as main.
The review asked the embedding gate to follow the rule accepted for the reranker in ashhart#1981: transformers' own BF16 distance from fp32, rounded up, not twice it. The oracle and the GPU test now take the BF16 bar's worst cosine distance at each width, round it up to two significant digits with exact power-of-ten steps (never below it), and cap it at 0.001, so the limits are 1.6e-4, 1.7e-4 and 1.9e-4 at widths 4096, 1024 and 32; --bar-factor goes, and the recipe states the new limits beside the posted distances, which already meet them.
The engine thread took the whole label-score queue and ran every call even after a drain set closing, so long rerank prompts could run past the drain's deadline; answer now reads closing under the host lock before each call, finishes the one already started and fails the rest without running them, with a test that closes the host mid-queue.
A Qwen3 reranker scores by its yes and no logits, and the checkpoint names those tokens in 1_LogitScore/config.json, so the server now reads them at startup into Server.rerank, treats a missing file as a plain model and refuses a malformed one by name rather than silently hiding the route, with a test-rerank-host step for the reranker tests.
Clients of the SGLang reranker service call /v1/rerank, so the server now serves it for a checkpoint with reranker labels: each document is rendered with the checkpoint template (request text masked as plain), checked against the window, scored through the engine's score hook as sigmoid(yes - no) and returned best first in SGLang's compact body and error envelope, with opt-in raw logits and prompt ids for receipts; a reranker server refuses every other engine route in routes.zig, which covers chat and the other text routes before any body is read.
Native reranker scores need an independent reference, so this tool renders the checkpoint template as SGLang does, records transformers' ids and the yes-minus-no logit from the final-normed last row in f64 (plus the BF16-logit score), and its compare mode drives a server's /v1/rerank with the raw-logit and id extensions to gate ids, logits, scores, rankings, the masked special case and the window boundary.
The reranker's scores are only comparable to SGLang's when the route feeds the engine the same prompt ids, so this env-gated test loads the checkpoint's own tokenizer and template, sends every oracle case through /v1/rerank and requires the ids the engine scored to equal transformers' (both served windows included) and the masked special-token case to differ.
A Qwen3 reranker reads only its yes and no logits, so its score hook needs a couple of head rows, not the 1.24 GB head on the device; the new score module checks a call before any GPU work (no mass, prompt inside the window and vocabulary, at most 16 labels inside the head) and turns the widened last row into f64 label logits against head rows it reads once from the checkpoint and keeps.
A reference is only worth its sources, so the reranker oracle now pins both reranker snapshots (revision and every non-weight file's sha256) and the Transformers 5.22.1 package tree, checks them before any model code runs (also as a sources command that loads nothing), hashes the weight shards before loading, refuses an incomplete load or an untied head, and writes its receipt whole or not at all.
A Qwen3-Reranker checkpoint declares its yes and no tokens in 1_LogitScore, so the qwen3 HIP family now takes that folder as the reranker head: the engine's prompt backend answers the lane host's score hook with one prompt's last row and f64 label logits from head rows on the host, the family reports decides and no embeddings, and the HIP registry passes decides into the served capabilities, so /v1/rerank serves the checkpoint and the text routes refuse it.
A client of SGLang's reranker could still hit refusals or error types TensorFold does not share, so the route now validates the whole body the way SGLang's schema does (pydantic's lax ints and bools, text content parts joined as SGLang extracts them, malformed parts refused) before the handler's own checks, labels schema errors "Bad Request" and handler refusals "BadRequestError", and takes PUT as SGLang does.
test-rerank-host carries an env-gated check of real prompt ids against transformers, but Zig cached the step's run, so setting TF_QWEN3_RERANK_DIR and _REF replayed an earlier skip; the run step now has side effects, as the checkpoint loader tests do, so the ids check runs whenever its variables are set.
The maintainer chose one rerank shape for every family, ashhart#1895's, so the route now returns results (index, relevance_score, the document as sent) best first with usage.total_tokens, takes ashhart#1895's request fields (instruction, {text} documents, max_length enforced as a limit, since this server does not truncate) beside SGLang's, and keeps the raw logits and ids extensions behind their opt-in flags.
The review asked that the engine, not a file in the served folder, make a server a reranker, as embedding engines declare .embeddings, and that the label rows not wait for the first request: Info gains rerank (the yes and no tokens), the qwen3 family parses and checks 1_LogitScore and reads both lm_head rows at load, and the server drops its own reader, so no other engine can answer /v1/rerank.
The review asked for the route in docs/api.md and the CHANGELOG, and for the shared lane-host change to be named: api.md gains a Rerank section (fields, prompt, score, response, refusals, the key scope), and the CHANGELOG entry states that the score queue now stops at a drain's deadline, which also reaches /v1/decisions on Flash Next.
The review asked for ashhart#1895's FP32 transformers reference and for the prompt checked against the model card's own formatter: the oracle takes --dtype float32 --device cpu (8B FP32 does not fit the GPU), records dtype, device and torch in its file, and a formatter command compares every served prompt's ids with the card's process_inputs, with the default instruction and custom ones sent both ways.
The review asked to restate the bound from what FP32 shows: against an FP32 reference, compare now gates |yes - no| at 0.125 and the score at 0.0125 (transformers' own BF16 run sits 0.110 and 0.0108 from FP32), lists each pair ranked in another order than FP32's and fails one only beyond twice the diff gate, and prints the median, p99 and worst; a BF16 reference keeps its gates.
The review asked for one max_length meaning with ashhart#1895's plan and a settled error envelope: max_length now defaults to 8192 and a longer prompt loses its content's last tokens, the template's tail kept, as the model card's tokenizer cuts it (refused where no document token would stay, or where the template's tail is not fixed), the window refusal is a context-length error, and errors use the envelope every route sends; the oracle checks four cuts against the card's ids.
The route's max_length now cuts a prompt's content as the model card's tokenizer does, with 8192 by default, and its errors use the server's own envelope: docs/api.md states both and the window error's context_length_exceeded code, and the CHANGELOG entry names the cut.
On the rebased ashhart#1978 base zig/build/cuda.zig reached 601 lines, past the lean checker's 600, with this PR's three lines for the rerank host step; the step moves unchanged (filter, side effects, name and description) to zig/build/rerank_host.zig, called with one line, as proposal_vocab_host.zig does.
The review asked for the bound in the recipe, so a reader sees what "matches" means: the new recipe names the files, the build and serve lines, the request, the oracle's FP32 and BF16 references and formatter check, the restated gates (transformers BF16's own distance from FP32, rounded up, and the ranking rule) and today's results.
laudney
force-pushed
the
pr-qwen3-reranker-hip
branch
from
November 3, 2026 18:32
90715b4 to
158489e
Compare
Contributor
Author
|
Restacked on #1978's rebased head
On the restacked stack (R9700): |
Contributor
Author
|
Reranker GPU test on this PR's restacked head TF_QWEN3_RERANK_DIR=MODEL_DIRECTORY TF_QWEN3_RERANK_REF=reference.json \
zig build qwen3-hip-gpu-test -Dhip=true -Dhip-arch=gfx1201 -Dhipcc=/opt/rocm/core/bin/hipcc --summary allThe skipped test is the embedding check (no |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1977. Stacked on #1978 (the Qwen3 dense HIP core); until #1978 lands, this diff also shows its commits.
The native HIP server serves Qwen3-Reranker-8B as published, through
POST /v1/rerankin #1895's rerank shape.Qwen/Qwen3-Reranker-8Bon an AMD Radeon AI PRO R9700 (gfx1201)POST/PUT /v1/rerank(and/rerank)max_lengthcuts--context 0; one token past it iscontext_length_exceeded, the next request still scoresWhat "matches" means. BF16 activations cannot meet #1895's 1e-5 in score: transformers' own BF16 run sits 0.0108 in
score and 0.110 in yes − no from float32 on these cases. The oracle's gate against float32 is that distance rounded up:
|yes − no| ≤ 0.125 and |score| ≤ 0.0125 on every case. Rankings must equal float32's, except that a pair float32 puts
within 0.25 in yes − no (twice the gate) may come in the other order; each such pair is printed, and one beyond 0.25
fails. The recipe (
docs/recipes/qwen3-reranker-hip.md) states the same.Why
Serving the checkpoint as Qwen published it, with no conversion, makes TensorFold usable where SGLang serves it today:
benchmarks compare the same bytes on both engines, and rerank clients reach it with the schema they already send.
What this changes
1_LogitScore/config.jsonis a reranker. The family reads its yes and no tokens andboth
lm_headrows at load (a malformed file or row stops startup) and declaresInfo.rerank; only an engine thatdeclares it answers the route. The prompt-only backend from Qwen3-Embedding-8B (BF16) on native HIP: BF16 dense GEMM and a qwen3 embedding engine #1978 answers the lane host's
scorehook: one prompt'sfinal-normed last row, dotted in f64 with the two stored rows (not the 1.24 GB head on the device).
zig/src/server/rerank.zig: Proposal: POST /v1/rerank, and Qwen3-Reranker-0.6B scoring on Metal #1895's request (query,documentsas strings,{text}objects or textparts,
instruction,top_n,return_documents,max_length) and SGLang's fields too (instruct, PUT); eachdocument rendered with the checkpoint's own template, request text encoded as plain text; past
max_length(default8,192) the content loses its last tokens, the template's tail kept, as the card's tokenizer cuts; scored as
sigmoid(yes − no);results(index,relevance_score,documentas sent) best first, ties in input order, andusage.total_tokens. Errors use the server's envelope; refusals are by name. Opt-inreturn_logitsandreturn_prompt_token_idsaddmeta_infofor checks. A reranker server refuses every other model route by name, andevery other server answers the route with HTTP 400.
running. This also reaches
/v1/decisionson Flash Next on CUDA.tools/rerank/qwen3_reranker_oracle.py: pinned sources (revision, every non-weight file, theTransformers package tree, shard hashes before load), float32 (CPU) and BF16 references,
comparewith the gatesabove, and
formatter.docs/api.md(Rerank), the recipe, and the CHANGELOG.Checks
zig build test -Dhip=true: 518/518 steps;test-rerank-host33/34 (1 skip without the checkpoint).compareagainst both references at both windows,formatter, and the device cells above, on the R9700 with ROCm 10.1.