Repository navigation
Unlimited-OCR-2 on CUDA: the family's steps on NVIDIA GPUs, the router's long-context speedup (stacked on #2074) - #2077
BobClawblaw wants to merge 39 commits into
Conversation
|
Thanks @BobClawblaw. Porting all 28 steps with a five-group contract, and planting 33 controls that all go red, is thorough. The router change also keeps its picks bit for bit. This waits on #2074, so we'll review it fully once #2074 lands. Before then:
|
|
Thanks. On 3: #2074 adds no findings. On Linux, its merge base ( On 1 and 2: we'll rebase #2074 and this onto main, then post the CUDA hashes for the five pages and the 45-page document next to your Metal ones, plus the reference's top-two margins at each first differing id. |
|
Thanks @BobClawblaw, that's a real find. |
|
Rebased both onto main ( 1. CUDA against Metal. I ran both sides on the rebased head with the same page PNGs: Metal on an M5 Ultra, CUDA on an RTX 5090 (sm_120, Linux). Each engine repeats its own bytes exactly, but CUDA does not equal Metal:
The differences start at box coordinates one or a few grid points apart, and paper-3 is the 2. Margins against the torch reference. This is the release's own
paper-10 and report-2 equal the reference exactly. The x86 summary on this build is the same as the 4080's: text without boxes identical in 5 of 6, box coordinates moved 13 of 284, by at most 2. 3. As above: no new findings. On Linux the rebased head reads 425, the same as main. Checks on the rebased head: |
0632249 to
029369d
Compare
029369d to
e1e345a
Compare
|
Thanks @BobClawblaw, and for reporting the Metal comparison straight. Two of seven equal, and every other difference is a near-tie in the reference, all nine of them an exact tie or one bf16 step: that's the clearest way to show it. Whether a second backend must match Metal bit for bit was our call, and the answer is no here: each engine exact against itself, and agreement with the publisher's reference within near-ties, is the bar for this family. Forcing Metal's summation order on CUDA would give up cuBLASLt for differences that are exact ties or one bf16 step. Two small things. The CHANGELOG has only #2074's Metal line; please add one for CUDA, saying it may part from Metal at near-ties. And gate 5's usage should be at nvcc 13.0, which the release fatbins use; if you can't run 13.0, say so and we'll generate those records on a GB10. Once #2074 lands and this is rebased, we'll run |
|
Thanks @ashhart. Both are in. The head is
On this head, Linux: GB10 (sm_121, Linux, nvcc 13.0), this code. We had one free, so we ran it ahead of yours:
So each GPU is exact against itself, but the bytes depend on the GPU model as well as on the backend. Not done: I have not driven the reference along the GB10's replies to show its departures are near-ties, as I did for the 5090. This stays a draft until #2074 lands; then I'll rebase it. |
|
Thanks @BobClawblaw, both are in: the CUDA CHANGELOG line says what users need to know, and measuring gate 5 |
afd186e to
dfb2246
Compare
|
1. The reference driven along the GB10's replies. Same tool and reference as the 5090 table (the release's own
paper-10 equals the reference. paper-3's step 145 is the same One correction to the description. The GB10 run's report-1 and report-2 PNGs are not the ones the 5090 and Metal runs read (the GB10 recipe renders them in Liberation Sans, the others in macOS's Arial), and its paper-9 differs too. So the GB10-against-5090 comparison holds only for paper-1, paper-3 and paper-10. paper-10 is equal there; paper-1 and paper-3 differ. The report-1 title box I quoted, and the hashes for both documents, compare different inputs. 2. THIRD_PARTY_NOTICES. Yes, the family follows the release's code, not only the paper and the weights. That code is |
SAM ViT-B, its neck and CLIP-L to the projector's rows, for the 1024 global view and 640 tiles (position tables resized as torch does, bit-identical). tf-uocr-vision gates each stage against float32 on a reference dump's pixels.
…oE) and greedy generation tf-uocr-decode runs a page end to end against a reference dump: the prompt's ids, the spliced embeddings, each layer's prefill rows against float32, the first logits, and the greedy reply.
…matvecs for one token 49 -> 142 tok/s on a one-page reply; the two-page reply now equals torch bf16's over all 337 tokens.
Prompt rows always take the GEMM (an expert's lone row in a chunk took the matvec before), so a row's bits do not depend on the chunking; tf-uocr-decode --chunk ROWS checks the first logits and the reply.
… template in the family folder) One request at a time on the lane host, greedy under the page guard. A page takes gundam tiles when they fit its share of the image tokens (the publisher's server default), else the base view.
…ernel groups, their records)
…ld-native uocr.cu ports uocr/vision.metal's and uocr/decoder.metal's steps op for op, rounded at the same points; cuBLASLt (cuda/vit.zig's Gemms) runs the projections. The CUDA host (cuda_weights, cuda_vision, cuda_decoder, cuda_serve) mirrors the Metal one: one device buffer for the bf16 weights, matvecs and on-GPU expert picks for a decoded token, the greedy pick under the page guard on the GPU. native/uocr_cuda.zig opens it beside the registry families. The host files stay shared: preprocess.With takes the picture decoder (core's on CUDA, a file belongs to one module), and the page guard moves to guard.zig so the CUDA module does not reach the Metal decoder. On an RTX 4080 SUPER (Windows 11): the six OCR cases' text matches the publisher's torch reference (one near-tie on a page alone, `$ [5] $` vs `[5]`, which the Mac's run takes the other way), 236-242 tok/s on a page, first token 0.35-0.46 s, and a 45-page document (40,685 prompt tokens) runs to its last page in a 98,304-token window.
… then merged A decoded token's router read every block's summary key with one warp, then took the top 16 in 16 rounds: its time grew with the context (140 tok/s at 41k tokens against 238 at 1k on an RTX 4080 SUPER). Each block's score is now a thread's (the same sequential sum), each warp keeps its own top 16 by shuffles, and warp 0 merges the eight lists in the one order (value, then the lower block), so the reads and their biases are bit for bit what they were. RTX 4080 SUPER, 1,500 decoded tokens: 224 tok/s after a 40,685-token prompt (was 140), 235 after 13,565 (was 192), 242 after 909; the 45-page document's 43,027-token reply 222 tok/s (was 118), byte-identical to before.
…ee headers of at most 600 lines
…, their records and control headers)
…m_80, 86, 89, 120 and 121; 13.0 pending)
…indow_add, rel_pos, concat, clip_embed, gather, scatter, embed)
… MoE gate and combine, greedy pick)
…on as SAM windows, SAM global and CLIP; tf_uocr_attend), 5 controls
…eir 13.3 usage checked; the lean rule's 600 lines kept
…d, so its errdefers release the host, context and driver
… stream, as the Metal engine does
…e-SM fatbins (13.2 still pending) tf-cuda-test kernel-contract-limits on nvcc 13.0's fatbins for sm_80, 86, 89, 120 and 121: every uocr launch PASS on every SM. 19 of the 28 compile as under 13.3, the other nine differ by a few registers. The records keep later_pending, so 13.2 stays pending and 13.3 keeps its own set.
dfb2246 to
e0b153e
Compare
|
Thanks @BobClawblaw, that's the table we hoped for: every departure on the GB10 is a tie or one bf16 step, so it |
What this changes
Unlimited-OCR-2 (#2074's
unlimited-ocrfamily) gains a CUDA backend:tensorfold-native serve --visionon an NVIDIA GPU serves page images with the release's BF16 weights as published, one stream, greedy under the publisher's page guard, as the Metal engine does. Before this, the family served on Metal only and CUDA refusedunlimited-ocrcheckpoints.6f2582554, on mainac51580b2). This adds:e68d29be8the CUDA port;4cee6f066the router's long-context speedup;f2d0f1769the lean rule;53a54f74dto9a9e409e1the kernel contract: the scaffold, the five groups and the launch records;e2c332c68a fix to the CUDA open: its refusals after the host is made (no driver, a checkpoint it cannot load) returned null past its errdefers, leaking the host (42,592 bytes) and, on the load path, the context and driver. They now returnerror.Refusedasnative/gemma4.zig's do; a checkpoint that fails to load no longer reports a leak;5ef4ac949a refusal of a named--parallelabove one ("Unlimited-OCR-2 serves one stream: serve it without --parallel"), as Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's Metal engine refuses it, with a test. Before it, CUDA served one stream whatever--parallelasked.9c5d5a1cathe launch records' nvcc 13.0 usage, measured;e0b153ec1the CHANGELOG line.atc overlap --jsonon this head: incomplete (pullFilesLimit). It lists 576 open pull requests sharing files, almost all throughCHANGELOG.md, the contract registry andzig/build/cuda.zig: 8 with textual conflicts (lfm3: serve the MLX text decoder on Metal #1514, Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835, glm cuda: TF_GLM_OVERLAP_PIECES defaults to 8 under solo on the wire #2116, fix(drama): report speech admission units without raising context #2143, fix(server): omit undiscoverable reasoning defaults #2184, fix(metal): publish Nemotron fallback capacity without #2195, lanes: one fixed code for every wait and refusal (#2135) #2196, notices: name the model release and library code these files follow #2197), 2 clean, the rest not checked. None touchesfamilies/uocrexcept Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074, which this is stacked on.How it is built:
zig/kernels/cuda/uocr.cu(withuocr_common.cuh,uocr_vision.cuhanduocr_decoder.cuh) holds the Metal family's 28 own steps, each rounded at the same points as its Metal kernel:The projections run on cuBLASLt through
cuda/vit.zig'sGemms(the bias in its epilogue, or fp32 sums and one rounding), as MOSS-Image's CUDA path does. They sum in cuBLASLt's order, not Metal's, so the replies are not bit-equal to the Metal engine's: they part at near-ties (Exactness, below).families/uocr/cuda_*.zigandnative/uocr_cuda.zigmirror the Metal host: the weights in one device buffer, the encoder, the decoder and the served model, opened fromnative/cuda.zigbeside the registry families.The family's host files are shared rather than copied. A file may belong to only one module, so two things move:
preprocess.With(picture)takes the picture decoder: core/picture through its relative path on Metal, thecoremodule's on CUDA;guard.zig(generate.zigre-exports it), so the CUDA module never reaches the Metal decoder.The CUDA module is rooted at
zig/src/uocr_cuda.zigfor the same reason.The router change: a decoded token's router scored every 32-key block of the context with one warp, then took the top 16 in 16 rounds, so decode slowed as the context grew. It is now a 256-thread block. Each block's score is one thread's sum, in the same order as before. Each warp takes its own top 16 by shuffles, and warp 0 merges the eight lists in the same order: value first, then the lower block. The reads, their biases and the replies are bit for bit what they were.
Checks on this head
e0b153ec1: the branch rebased onto Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's6f2582554, which is Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074 rebased onto mainac51580b2(62 commits pastb331731e6) plus the THIRD_PARTY_NOTICES entry for the release's Python files. OnlyCHANGELOG.mdneeded resolving: both lines kept.python3 tools/zig/lean_check.pyon Linux: 425 problems on main (ac51580b2), 425 on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's head and 425 on this head (and 425 each on the earlier baseb331731e6, on Linux and macOS). New findings: 0;zig/build/cuda.zigandregistry.zigstay at 600 lines. (On Windows the checker reads files as cp1252 and counts 15 more on every tree.)e0b153ec1:zig fmt --check build.zig zigclean;zig build native install -Dsm=120builds;zig build test-cuda-host -Dsm=120 -Dnvcc=...:Build Summary: 309/309 steps succeeded; 4791/4812 tests passed (21 skipped), no leak reports; the built server refusesserve --vision --parallel 2;tf-cuda-test uocr-contract:SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed.ocr-matchback to back:e0b153ec1against the pre-rebasee1e345a25(the same served code asafd186eac). All seven replies are byte-identical (the sha256 in the table below), with the same token counts; decode 275.5-306.4 tok/s against 275.2-306.5, first token within 0.03 s except report-1, the run's first request (0.63 s against 0.36 s).zig build test-cuda-host -Dsm=121 -Dnvcc=...:Build Summary: 307/307 steps succeeded; 4748/4771 tests passed (23 skipped), the records' new test among them;zig fmt --checkclean;tf-cuda-test uocr-contract:SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed.e1e345a25, which differs from this head only in the launch records, their host test and the CHANGELOG.zig build test-cuda-host -Dsm=120 -Dnvcc=...:Build Summary: 307/307 steps succeeded; 1183/1184 tests passed (1 skipped): zig re-ran the test binaries this head changes and took the rest from its cache of029369d16's full run (4748/4769, 21 skipped). The new refusal test fails with the check disabled. The built server refusesserve --vision --parallel 2with that message. In the full run three passing steps print to stderr and so show afailed command:line (Dory RoPE's word table, a gfx1030 N/A line, the libtfgrammar warning); none is this change. No leak reports.zig build native install -Dsm=120 -Dnvcc=...builds.tf-cuda-test uocr-contracton029369d16(the same kernels and contract files as this head):SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed.zig fmt --check build.zig zig,zig build test,test-golden,test-kernel-contract,test-server-security test-pull-security(24 passed, 0 failed),check-release-paths,fuzz-cortheon,dev-toolsandzig build native install: all pass on this head and on Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's head. No leak reports.06322491fon Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's10fe5c459), Windows 11, RTX 4080 SUPER:test-cuda-host -Dsm=89285/285 steps;tf-cuda-test uocr-contractgave the same SUMMARY on sm_89.Receipt
e1e345a25(this head less the 13.0 records and the CHANGELOG line, which change no served code), except where marked as the RTX 4080 SUPER's earlier base.baidu/Unlimited-OCR-2at revision0480966938291445591f69f08a13b041c073a96a.ocr-matchplan on the same page PNGs: the recipe's five pages alone, the five as one document, and the release's 15-page paper three times over as one 45-page request.Against the Metal engine (M5 Ultra, Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's code on this head): equal on the two report pages, different on the other five cases. Each engine repeats its own bytes exactly.
58c9ad432da973e458c9ad432da973e4366f37903b4eec02366f37903b4eec029211a96c01c0dd849960608a870adef444da612e99c36206357d7be06ce47b968d377476d6ea03c666767fb282460f00f7fc6f672d7c67d1a5acde0bcdabd6d13904790bb7f0ef286e4269decbf659acAgainst the release's own
infer(modeling_unlimitedocr.py) on the CPU in torch bf16 (x86, torch 2.13.0+cpu, transformers 4.52.1), greedy, gundam mode: the text without layout boxes is identical in 5 of 6 cases; box coordinates moved in 13 of 284, by at most 2 on the 0-999 grid (the RTX 4080 SUPER on the earlier base: 13 of 284, by at most 5). report-2 and paper-10 are identical.Every place the CUDA reply leaves the reference is a near-tie in the reference. Driving the reference along the CUDA reply and recording its logits after the page guard wherever its own pick (
argmax, the lowest id on a tie) differs: 9 such steps over the four cases, and at each the CUDA token is the reference's second choice, either an exact tie (4) or one bf16 step below (0.125 or 0.25, 5). The sixth case's text difference (paper-3,$ [5, 6, …] $against[5, 6, …]) is one of them: 24.50 against 24.375.GB10 (sm_121), the same code. Its recipe renders report-1 and report-2 in Liberation Sans (the other runs used macOS's Arial) and its paper-9 differs too, so only paper-1, paper-3 and paper-10 compare with the RTX 5090 and Metal on the same images: paper-10 equals the 5090, paper-1 and paper-3 do not, and none equals Metal. GB10 sha256: report-1
597333ed92039bdf, report-22b39eebc29d23b82, paper-116439a803c144e9f, paper-309b699e1a4d6f621, paper-108d377476d6ea03c6, the document2071ded6b6e9af5b(3,764 tokens), 45 pages3678e6a64800e404(42,994). So the bytes depend on the GPU model as well as the backend. Against the x86 reference, on the GB10's own pages, report-1 and paper-10 are identical. Driving the reference along the GB10's replies, as above, finds 11 steps where its own pick differs, and at each the GB10's token is its second choice: an exact tie (6) or one bf16 step below (0.25 or 0.125, 5).173172251247135134506501275274[797796MMP100101119120409407The same
inferon the GB10's own CPU (aarch64, the recipe's reference) writes paper-3's$ [5, 6, …] $as the GB10 does: the reference's own pick at that near-tie depends on the CPU.A repeated request gives the same bytes, and two requests at once give their solo replies (one stream: they queue). There are no drafts, so there is no
"draft": falsecomparison.decode_tps;tools/bench_openai.pynot run):tools/bench_cache.pywas not run: the family has no prompt cache.Kernel contract, only when a GPU kernel is new or changed
tf_uocr_*kernels inuocr.cu. None replaces a CUDA kernel. The command istf-cuda-test uocr-contract, with five groups that mirror Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074's Metal groups (zig/tests/cuda/contract_uocr_*.zig):nan_to_num), silu_mul;sqrtfcorrectly rounded;expfwithin 2 ulps andlogfwithin 1;zig/kernels/cuda/contract_uocr_*.cuh(each a copy of the production kernel with one break), all red as planted:SUMMARY uocr: 35 kernels, 35 pass gates 1-4, 0 fail; 33 controls: 33 red as planted, 0 missed. There are no known findings. The tightest passes:uocr_cuda_*_records.zig, amongcuda_pending. Their nvcc 13.0 usage comes fromtf-cuda-test kernel-contract-limitson nvcc 13.0's-Dsm=80,86,89,120,121fatbins: every launch PASS on every SM. 13.3's is inmeasurements/uocr_133.zig, from the same tool on 13.3's fatbins, also PASS everywhere; 19 of the 28 compile the same under both, the other nine differ by a few registers. 13.2 stays pending (later_pending). The registry's host test checks 13.0's and 13.3's usage against all five SMs.kernel-contract-limitsrun stopped atdsv41_x3ld's sm_121a-only kernel (SymbolNotInImage): this build has no sm_121a image. That is not this change.Checklist
"draft": false(no drafts), a resumed prompt equals a fresh one (no prompt cache), and concurrent equals solo.