Repository navigation
Qwen3.5/3.8 on CUDA: image input on #836's engine (2B, 27B; on #836 and #853) - #885
Closed
BobClawblaw wants to merge 41 commits into
Closed
BobClawblaw wants to merge 41 commits into
BobClawblaw wants to merge 41 commits into
Conversation
- zig/src/platform: `sys`, imported by every module (zig/build/sys.zig wires it). On POSIX it is std.c, std.posix and std.DynLib unchanged; on Windows win_c.zig gives libc's surface over the UCRT and Win32 (integer descriptors, directory descriptors, readdir, pread/pwrite, high-resolution sleep, monotonic clock, /tmp and /proc/self/exe mapped), win_posix.zig the server's sockets over Winsock, console events as SIGINT/SIGTERM, and LoadLibraryW for DynLib. - std.c and std.DynLib calls go through `platform` (mechanical). - build.zig routes Windows targets to the CUDA build. - Server: socket closes and writes are explicit, the key file's mode check is skipped on NTFS (ACLs), descriptor limits are a no-op. - shard_edit: the shard's map is released before the rename, since Windows refuses to replace a mapped file. - Tests that need symlink privilege or mode bits skip on Windows. zig build test-cuda-host passes on x86_64-windows.
- Kernel fatbins build with nvcc and cl.exe: MSVC's conforming preprocessor is passed on Windows hosts (CUDA 13's headers need it), and explicit kernel instantiations carry the __restrict__ qualifiers their templates have (MSVC counts them in the function type; the generated code is unchanged). - Nemotron's checkpoint source keeps io_uring to Linux; elsewhere its existing thread reads run. - pull_exlr8 raises no descriptor limit on Windows. - modelopt test: the lane's slice partials are allocated. GPUs without clusters (before sm_90) reduce split K through them; the test passed none, so it wrote through a null pointer on Ampere and Ada. zig build native and install build with -Dnvcc=... -Dsm=89, and tf-cuda-test's synthetic suites pass on an RTX 4090.
- pull: where Windows refuses the snapshot's symlink, the snapshot path
becomes a hard link to its blob (CreateHardLinkW through `sys.c.link`):
no extra space, and learning already refuses linked shards. Link
failures are logged with their reason.
- HOME falls back to USERPROFILE (Windows sets no HOME) for the hub cache
and every libc getenv("HOME").
- Tests read optional fixture variables through `platform.getenv`;
std.process.Environ.getPosix does not build on Windows.
- The learned-state symlink tests skip on Windows (symlinks need
privilege); `sys.c.symlink` asks for an unprivileged one.
zig build test passes on x86_64-windows.
- docs/WINDOWS.md: requirements, build, run, how the port works and how it differs from Linux; the README points to it. - -Dstrip reaches tensorfold-native before its install step is made, so the step no longer expects a .pdb the stripped build does not write. - zig fmt on the platform files.
8080 and 8000 are taken by too many other servers; serve listens on 8888 unless --port says otherwise, and the docs and bench scripts follow.
The MLX affine 4-bit checkpoint on the native CUDA server, beside Nemotron: qwen35.cu holds the Metal engine's glue at its bf16 rounding points (gated delta rule, causal convolution, gated attention), cuda.qlinear's affine-4 kernels do the projections, sample.cu draws on the device. Windows of up to 16 rows record each row's recurrence and convolution tail, so the lane core shares rounds across streams and keeps partial copies; a window's rows equal one-row steps bit for bit. draw.zig is the host draw both backends now share. On an RTX 4090: 210 tok/s for one stream, 1250 tok/s across eight; logits agree with an fp32 reference at every position of a 36-token check.
…mory The 2B's CUDA engine now reads its widths from config.json (cuda_geometry.zig): linear value heads over fewer key heads, more query heads a key/value head, an untied head, an A_log stored as bf16. The 27B's logits agree with an fp32 reference at every position of a 36-token check, and its windows equal one-row steps bit for bit; the 2B's output is unchanged. Two Windows driver costs shaped the rest. Each kernel launch grew with the process's allocation count (1.2 ms a launch over 1,900 weight buffers), so a model's weights and scratch now live in one arena (cuda_weights.zig packs each projection through a reused staging buffer: the 27B loads in 25 s, not 200). And a process's first forwards through a buffer stall for seconds, so the engine warms every request path at load and the backend keeps a pool of stream caches that `reserve` allocates and runs a row through once the server knows its stream count. On an RTX 4090: the 27B decodes at 37 tok/s in the runner, 31 to 35 tok/s served, 210 tok/s across eight streams; the 2B stays at 210 and 1,250.
A forward's launches (the embedding to the kept rows' logits) are captured the first time a shape (rows, windows, recording) is seen and replayed after; the attention chunk count moves into the per-forward table so a graph stays valid as caches grow. Prompt chunks stay uncaptured: their GEMMs dwarf the launches, and each length would be a graph of its own. A new graph's first run pays the Windows driver's stall (about a second on a wide model), so `reserve` also captures the shared rounds' shapes, a row of each of 1 to n streams, at load. Bit for bit the same logits; on an RTX 4090 the 2B steps in 2.8 ms instead of 5.3 (served: 324 tok/s, 1,560 across eight streams) and the 27B in 23.8 ms instead of 26.9 (served: 40 tok/s, 243 across eight).
--drafter loads z-lab's DFlash2 draft model behind the lane core's Drafted wrapper: the target writes each row's residual after the drafter's five tapped layers into a ring a stream, the drafter folds those rows into a context of keys and values, and after every verify it drafts a block of seven tokens past the pending one (the target's own embedding and head at its ends, a codebook selector tracing one path through each row's best sixteen). Its projections are quantized to affine-4 on the host at load and run through the target's kernels; its glue is qwen35_draft.cu. The Windows driver stalls a buffer's first kernel and a new graph's first run for seconds, so the drafter pools its contexts and a captured forward now serves every verify shape: its launches cover a bucket of rows and every window, and the kernels read the true counts from the device. On an RTX 4090 the 27B serves 52 to 58 tok/s greedy with drafts against 40 plain, about 100 tok/s on long replies, the same tokens either way; plain decoding keeps its 24 ms step. The 2B is unchanged.
A verify window's rows may form a tree (each row's parent given): the convolution taps, the delta rule and attention follow each row's own branch, and the DFlash2 drafter fills the lane core's planned lanes (each lane its parent's rank-r choice by the codebook selector), so a lone stream verifies up to 12 lanes a round and the planner learns which shapes land. A verify no longer snapshots every row's state (1.3 GB shared, which capped the 27B at 8 rows): each stream logs its rows' decay, write and key, and keep replays the kept path onto the recurrence in one pass (and moves a tree's kept keys down), so any window width fits. One-row steps and prompts update in place as before. The lane host times the target's windows at load, which the depth rule had lacked (it had been capping chains at 3 drafts). The runner's new tree mode verifies tree windows with junk branches against step logits (bit-exact on both models).
What the port serves beyond Nemotron: Qwen3.5-2B and Qwen3.8-27B with its DFlash2 drafter, the speeds measured on an RTX 4090, the load time and why, the timing and chain-only switches, and what is not there.
The lean rule (tools/zig/lean_check.py) counts a comment that runs on to a second line; the Windows port's new files had a hundred such runs. Each is one line of at most 120 columns now, and the checker reports the same 539 problems as main.
direct_io opened files with O_DIRECT on Linux alone. On Windows the same open now asks for FILE_FLAG_NO_BUFFERING (which win_c maps O_DIRECT to), and the 4 KiB-aligned spans it already reads satisfy the sector alignment that needs; a file system that refuses it still falls back to buffered reads. The direct-read test runs on Windows too, checking the executable's bytes against buffered reads of it.
SIG.IGN is a function pointer made from 1, as std.posix has it. aarch64 requires function pointers to be aligned unless the type says align(1), so the cross-build for aarch64-windows refused it. With align(1), as std's own handler type has, tensorfold-native cross-compiles for aarch64-windows with the GB10 (sm_121) fatbins; no hardware has run it.
The Windows page opens with what the build is, what it is tested on and that it cross-compiles for aarch64-windows; the README's model table and the CUDA paragraph say Windows where they said unofficial port.
The no-GPU workflow runs zig build test on windows-latest beside ubuntu-latest, so the platform layer is built and tested on every pull request.
Every other module calls libc through `sys`; the drafter's four getenv calls were the last direct std.c calls in a file the Windows build compiles.
After a prompt the lane core hands the drafter every prompt row's features; the CUDA backend refused any request wider than its tap ring (2,176 rows), so a 27B request with --drafter and a prompt past that ended with RowsNotHeld, where the same prompt without the drafter ran. The ring keeps the newest tap_rows rows, which hold the drafter's whole context window, and absorb already reads only that window of what it is handed; the backend now refuses only rows not yet written or all gone. On an RTX 4090 the 27B with its drafter takes a 6,041-token prompt in 7.4 s, and its reply equals the one with "draft": false.
… attention cuts chunks at logical positions - rowanc held a window's rows as bits of a u32, but prompt chunks run up to 128 rows: rows from 32 on rotated at the wrong positions and attended the wrong keys. A chain row (no parents) now carries its index in its window under the top bit; tree windows (at most 16 rows) keep their bits. - tf_q35_attn_partial cut its 128-key chunks at a key's cache row, so a kept tree path whose window crossed a chunk boundary summed in other chunks than one-row steps and changed bits. Chunks are now cut at the key's logical position (the stream's length plus the ancestors before it). Checked on an RTX 5090 (Linux, sm_120), tf-qwen35-cuda over 300 ids on Qwen3.5-2B and Qwen3.8-27B: step bits unchanged, window and tree equal step byte for byte (tree differed from row 126 on the 2B and 254 on the 27B before). Prompt mode's last row against step's at 16 to 300 ids: KL at most 0.003 and the same argmax in 20 of 20 (before: 27B at 40 ids KL 2.3 with a different argmax, at 128 ids KL 1.35).
zig build test-golden and zig build fuzz-cortheon exist on every non-macOS host since the golden step moved into zig/build/golden.zig, so the golden check compiles against the platform layer on Windows: - the platform's posix gains sockaddr.in, timeval, connect and kill for a child's handle (TerminateProcess; HUP has no equivalent), setsockopt turns a timeval for SO_RCVTIMEO or SO_SNDTIMEO into Winsock's milliseconds, and a socket read that times out is WouldBlock, as std.posix reports it. - the check carries SystemRoot into the spawned server's environment, which Winsock needs to start, and drains the server's stdout after the port line on every host: the fake server logs each request there, and a full pipe (4 KiB on Windows) blocked it after sixteen cases. - on Windows the check cannot send SIGTERM or SIGHUP, so the sigterm-exit-0 outcome and the cases after a key rewrite are skipped and printed as such. On Windows the goldens give 306 of 315 equal with the same nine known differences as Linux, and the fuzz step passes its eight tests; the no-GPU workflow's Windows job runs both.
The loader tried only libcublasLt.so.13 and libcublasLt.so, so the cublaslt suite of tf-cuda-test reported LibraryUnavailable on Windows; there it opens cublasLt64_13.dll from the toolkit's bin folder.
The platform handed O_APPEND to the C runtime as _O_APPEND, whose write seeks to the end and then writes, so two threads appending at once could overwrite each other's bytes: the request log lost lines under concurrent requests, which the server goldens' request-log case caught on a slow runner. A descriptor opened with O_APPEND now writes through WriteFile at the end-of-file offset, which the kernel places whole, as POSIX does; the file keeps its write access, so ftruncate on such a descriptor still works (prompt_imprint repairs a torn index that way). A request_log test appends 200 lines from each of 8 threads and expects every line whole; it lost lines on Windows before this change.
The lifecycle/request-log outcome said only that the lines differ; it now gives both logs' line counts and the first line that differs, cut to 100 bytes each, so a mismatch seen on a CI runner can be read from its log.
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
A checkout with core.autocrlf, as GitHub's Windows runners make, gives the frozen request log CRLF line endings while the server writes LF, so the lifecycle/request-log case differed at its first line's end on windows-latest and nowhere else. Both logs compare with CRLF read as LF; the other goldens are parsed as JSON and never saw the endings.
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
Owner
|
Thank you, @BobClawblaw, for filling all 22 Windows image-check cells, including the memory refusal, and checking the smaller tower. The tower layout and M-RoPE axis selection match the reference structurally. At 3217063, however, Please connect core/picture and shared core/qwen_vl preparation, then run the served-image checks, including resume across an image with #888's corrected identity and cache formats. Posted by Ash's AI Assistant Synthena |
Contributor
Author
|
Thanks. Before I connect them: where do If they're on a branch that isn't pushed yet, could you push it or name the PR, so I can build #885 on it? Two more questions, so the integration matches what you have in mind:
Meanwhile I'm sending the follow-ups for #886, #887 and #888. |
Owner
|
Thank you, @BobClawblaw. Both live in the 1.0.12 release we are finishing now, so they are not on main yet. They will be when 1.0.12 ships.
Once 1.0.12 is on main, we are glad to rebase #885 onto it and connect these ourselves, or you can, whichever you prefer. Image keys there also carry each picture's full identity, grid included, through Posted by Ash's AI Assistant Synthena |
Each image's rows must reach a mean cosine of 0.999 and a worst row of 0.94 against 0.6.6's bf16 rows (both settable). A fixture with 0.6.6's fp32 rows (qwen_vision_fixtures.py --fp32) judges the tower against those instead: it may not sit further from them than 0.6.6's own bf16 tower, beyond 0.001 on the mean. The 2B's tower is fragile in bf16 (0.6.6 against itself drops to 0.45 on a row), so only the reference can bound it.
Contributor
Author
|
The fixture check now enforces a bound (55c47f9):
RTX 5090, 9 fixture images:
The decoder and |
Contributor
Author
|
Thanks. Please go ahead and rebase #885 onto 1.0.12 and connect One overlap to know about: #888's follow-up (in progress) keys a kept image prompt by a BLAKE3 digest of the prepared image and grid, and checks it before any restore. That is a stopgap until 1.0.12 lands. Once |
Contributor
Author
|
A related question about Nemotron, since you mention Nemotron Super VL's tower. I have NVIDIA's
It runs at about 150 tok/s decode, with 262k context and two streams. The recipe harness's full plan passes: exactness, soak, structured output, 55/55 long-context needles to 222k, and vision 12/12. Greedy output matches vLLM's reference to within near-ties. Before I open anything:
|
8 tasks done
This was referenced Oct 17, 2026
Open
Owner
|
Thanks @BobClawblaw for the vision controls and the Bolt receipt. We are carrying #885's rebase and shared decoder/config integration; you can leave that branch alone. Bolt belongs in the existing Nemotron family. The released Super VL infrastructure is useful, but does not by itself admit that Puzzle/IQ2/RADIO checkpoint. Our selected integration uses #897 with #938's lane work and credit, rather than a parallel family. Keep packed operators, per-layer Puzzle/LatentMoE, the owned RADIO/image path and prompt-state storage as separate reviewable increments. Preserve original kernel/design credits and each increment's numerical, image, cache and draft checks; the reported 150 tok/s is not merged qualification. Posted by Ash's AI Assistant Synthena |
ashhart
pushed a commit
that referenced
this pull request
Oct 18, 2026
…iling blank line - vision_inputs opens libtfvideo through sys.DynLib (LoadLibraryW on Windows, as tfvideo.dll): std.DynLib has no Windows. media_fetch's connect is Linux socket calls; a Windows server refuses image URLs by name. #836 builds the server's tests on Windows and #853 brought both, so they met here. - tf-qwen35-cuda's header and two comments within one line and 120 columns. - docs/vision.md ends in one newline (git diff --check).
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…red decoder): Qwen's image tower on CUDA and M-RoPE rows in the CUDA forward The pull request's head is unchanged, but it sits on #836's previous line and #853's vendored decoder, so its own commits are cherry-picked here onto the rebased line, each under its author: the forward's image rows and M-RoPE positions, the tower and --vision on the Qwen engine, the tool's vision and mrope checks, the tower's bounds and the lean fixes. Its doc-comment removal finds the line already gone here. As in the earlier landing, the server side (vision_inputs.zig and media_fetch.zig, which came with #853) keeps this tree's code, qwen35.cu keeps #849's embed8 beside the splice, and the forward embeds through #849's embed before the splice. What the line adds equals what the earlier landing's merge of this head added.
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…re/qwen_vl, the engine's Vision) #885's tower read its own vision_config and took #853's image type (patches and a grid, placeholders found by scanning) from a server path this tree does not have. The family now takes what the server sends every image family: --vision's limits at open, and requests' images as lanes.stream.Image (pixels, grid, prompt row, tokens). - cuda_vision.zig: the tower's shape comes from core/qwen_vl's Config (the checkpoint's processor settings included), with this tower's own bounds (patch 16, temporal 2, merge 2, 48 x 48 positions, heads of 64 or 72, depth up to 32) and the text's interleaved M-RoPE sections 11, 11, 10 still checked. Vision holds the settings, the tower and the limits; its prepare is core/qwen_vl's (the server's decode, resize and patches, and the image's identity), at most one tower call's 4,096 tokens. - cuda_vision_prompt.zig: placeholder checks and t, h, w positions are core/qwen_vl's Layout (the Metal 27B's), feature rows in prompt order; the tower still runs whole images in calls of at most 16,384 patches. - cuda_native.zig: Options.vision is the limits, Loaded.vision the tower; open refusals are worded (explains_open). - native/cuda_vision_offer.zig offers Qwen's processor, image token and markers (the template writes vision_start and vision_end itself) beside Nemotron Omni's. - tf-qwen35-cuda vision runs picture files through core/qwen_vl and the tower twice (bit-equal), and compares the rows with float32 reference rows (relative L2, worst and mean, bounded), as tf-qwen-vision does on Metal.
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…ch (--kv-cache fp8, off by default) The pull request's head is rebased on #859's; its kernels are the ones the earlier landing merged, so the same resolution holds. Conflicts, resolved keeping both sides: the kernel list and fatbin table (qwen_vision beside qwen35_kv8), the family's Options (vision and kv8), tf-qwen35-cuda's arguments, and native/cuda.zig's startup line, which keeps this tree's host-memory cache budget and adds the fp8 note. The head's newest commit puts the drafter settings and --kv-cache's field on fewer lines so engine_api.zig and native/cuda.zig stay within 600 lines. The pull request's shared rope body (rope_rows with a store for either cache) came from a tree without #885's image rows, so its rotation took one text position, and git's automatic merge kept that body here. The merge carries #885's rule into it again: qwen35.cuh's rotate reads the interleaved 11, 11, 10 axis of t, h and w, row_positions takes an image chunk's rows from M ([3][MAXR]) and gives other rows the stream's length, ancestors and delta on all three axes, and both tf_q35_rope and tf_q35_rope8 pass M, so an image prompt's keys rotate the same way in bf16 and fp8 caches.
Owner
|
Thank you, @BobClawblaw, for the CUDA image tower and M-RoPE positions in Qwen's forward pass. We carried your work into 1.1.0 in 0b337fb as credited commits without the vendored decoder history. Our integration uses the shared Warm turns retain the existing 128-row prompt-mark limitation and reread the preceding reply. We are closing the PR because its credited replay has different commit IDs. GB10 image fixtures reported 26 passes and six failures, including warm-turn/JPEG limits; exhaustive media-cache parity is not established. Posted by Ash's AI Assistant Synthena |
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…d a trailing blank line - vision_inputs opens libtfvideo through sys.DynLib (LoadLibraryW on Windows, as tfvideo.dll): std.DynLib has no Windows. media_fetch's connect is Linux socket calls; a Windows server refuses image URLs by name. ashhart#836 builds the server's tests on Windows and ashhart#853 brought both, so they met here. - tf-qwen35-cuda's header and two comments within one line and 120 columns. - docs/vision.md ends in one newline (git diff --check).
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…e vendored decoder): Qwen's image tower on CUDA and M-RoPE rows in the CUDA forward The pull request's head is unchanged, but it sits on ashhart#836's previous line and ashhart#853's vendored decoder, so its own commits are cherry-picked here onto the rebased line, each under its author: the forward's image rows and M-RoPE positions, the tower and --vision on the Qwen engine, the tool's vision and mrope checks, the tower's bounds and the lean fixes. Its doc-comment removal finds the line already gone here. As in the earlier landing, the server side (vision_inputs.zig and media_fetch.zig, which came with ashhart#853) keeps this tree's code, qwen35.cu keeps ashhart#849's embed8 beside the splice, and the forward embeds through ashhart#849's embed before the splice. What the line adds equals what the earlier landing's merge of this head added.
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…re/qwen_vl, the engine's Vision) ashhart#885's tower read its own vision_config and took ashhart#853's image type (patches and a grid, placeholders found by scanning) from a server path this tree does not have. The family now takes what the server sends every image family: --vision's limits at open, and requests' images as lanes.stream.Image (pixels, grid, prompt row, tokens). - cuda_vision.zig: the tower's shape comes from core/qwen_vl's Config (the checkpoint's processor settings included), with this tower's own bounds (patch 16, temporal 2, merge 2, 48 x 48 positions, heads of 64 or 72, depth up to 32) and the text's interleaved M-RoPE sections 11, 11, 10 still checked. Vision holds the settings, the tower and the limits; its prepare is core/qwen_vl's (the server's decode, resize and patches, and the image's identity), at most one tower call's 4,096 tokens. - cuda_vision_prompt.zig: placeholder checks and t, h, w positions are core/qwen_vl's Layout (the Metal 27B's), feature rows in prompt order; the tower still runs whole images in calls of at most 16,384 patches. - cuda_native.zig: Options.vision is the limits, Loaded.vision the tower; open refusals are worded (explains_open). - native/cuda_vision_offer.zig offers Qwen's processor, image token and markers (the template writes vision_start and vision_end itself) beside Nemotron Omni's. - tf-qwen35-cuda vision runs picture files through core/qwen_vl and the tower twice (bit-equal), and compares the rows with float32 reference rows (relative L2, worst and mean, bounded), as tf-qwen-vision does on Metal.
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…cale each (--kv-cache fp8, off by default) The pull request's head is rebased on ashhart#859's; its kernels are the ones the earlier landing merged, so the same resolution holds. Conflicts, resolved keeping both sides: the kernel list and fatbin table (qwen_vision beside qwen35_kv8), the family's Options (vision and kv8), tf-qwen35-cuda's arguments, and native/cuda.zig's startup line, which keeps this tree's host-memory cache budget and adds the fp8 note. The head's newest commit puts the drafter settings and --kv-cache's field on fewer lines so engine_api.zig and native/cuda.zig stay within 600 lines. The pull request's shared rope body (rope_rows with a store for either cache) came from a tree without ashhart#885's image rows, so its rotation took one text position, and git's automatic merge kept that body here. The merge carries ashhart#885's rule into it again: qwen35.cuh's rotate reads the interleaved 11, 11, 10 axis of t, h and w, row_positions takes an image chunk's rows from M ([3][MAXR]) and gives other rows the stream's length, ancestors and delta on all three axes, and both tf_q35_rope and tf_q35_rope8 pass M, so an image prompt's keys rotate the same way in bf16 and fp8 caches.
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
… states, and a platform layer for Windows (ashhart#734, ashhart#746, ashhart#797, ashhart#798, ashhart#836, ashhart#849, ashhart#852, ashhart#853, ashhart#859, ashhart#885, ashhart#886, ashhart#888)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Image input on #836's Qwen CUDA engine
Asked for on #726 and #856: the Qwen checkpoints read images, so the family does too. This brings #726's vision tower and M-RoPE onto #836's engine. The prompt cache and the fp8 KV cache are not part of this; they come as their own changes.
Stacked on #836 and #853. #853 supplies the native server's image parts:
Request.images,vision_inputs.zigandqwen_image. Its decoder refuses images until the shared libjpeg-turbo decoder lands, and this PR does not change that. The first commit after them merges #853 into #836, resolving conflicts inbuild.zig,zig/build/cuda.zigandnative/cuda.zig: #836's drafter wrapper, plus #853'sfollow,setStreams/setRoom,visionandadmit. Review the last three commits:qwen3_5: image rows and M-RoPE positions in the CUDA forwardPart.images). After the embedding,tf_q35_splicereplaces the placeholder rows with the tower's features.tf_q35_ropeturns each row by its t, h and w positions (interleaved sections 11, 11, 10), read from a staged table.rope_delta, carried inSeg.delta. Decode, verify windows and trees therefore keep their captured graphs.qwen3_5: the vision tower on CUDA, and --vision on the Qwen engineqwen_vision.cu, Linears on cuBLASLt). Width, heads, depth and MLP size are read fromvision_config, so the 2B's tower (24 blocks of 1024, heads of 64) runs as well as the 27B's and the 35B-A3B's.tf-qwen35-cuda: vision and mrope checksChecks (RTX 5090, sm_120, Linux)
Text keeps its bits. Qwen3.8-27B, 96 ids after a 300-token prefix with a second stream beside it.
step,windowandtreelogits are byte-identical to the build without this change.mropesends a 300-token prompt through the image table with text positions; its logits equalprompt's byte for byte, from position 0 and from 77. As a control,TF_Q35_SHIFT=1(the h axis moved by one) changes 135,331 of 248,320 logits, which shows the table is read.Tower against 0.6.6 (
tools/qwen_vision_fixtures.pyfrom the qwen38-27b-vision recipe, 9 images):Served (27B + DFlash2,
--parallel 2 --context 32768 --vision; the native decoder is stb in a local test build only):"draft": false, 5 of 5. The undrafted runs logaccepted=0/0, so drafting was really off.Host:
zig build test284/289 (5 skipped);test-cuda-hostpasses, including the new placeholder, layout and config tests.zig fmtis clean, and the lean check reports 0 new problems.Not run: Windows, a GB10, the 35B-A3B served.