Native image and video inputs in the server (Python frontend's preparation, data and HTTPS URLs), before any family consumes them #565

Activity

1. BobClawblaw commented on Oct 10, 2026

   

   BobClawblaw

   on Oct 10, 2026

   ContributorAuthor

   More actions

   An update from the Metal side. GLM-5.3-Flash's image input now runs natively on one M5 Ultra. It's built on 1.0.4 with a downstream patch, and the recipe is at https://github.com/BobClawblaw/glm53-flash-mlx4bit-m5-ultra (patches/glm-one-mac-vision-v1.0.4.patch). The pieces line up with the split proposed here, so here is what we learned, before any PR.

   Engine-independent (server)

   • api.Info.vision: ?Vision is the engine's offer. It's a prepare(ctx, allocator, bytes, max_tokens) -> PreparedImage hook (patches, grid, tokens, content hash), plus the image token, a request's token budget and its image limit. The server never links an image library; the engine does.
   • normalize keeps image_url parts in user messages for the template when the engine offers vision. The Anthropic and Responses translations already produce those parts.
   • After rendering, each <|image|> the template wrote is expanded to the image's token count. History length, chunk starts and the window check all use the expanded ids.
   • api.Request.images carries the patches and placeholder positions. api.Request.cache_key is the prompt with each image's placeholder rows replaced by ids derived from the image's content hash (above the vocabulary), so the prompt cache never resumes one image's state for another image under the same placeholders.
   • Data URLs only. Limits as in 0.6.6: 10 MB an image, 20 MB a request, 8192 px a side, 16 MP, a single frame, a shared visual-token budget, detail: low capped at 256.

   GLM on Metal (family)

   • The tower is the 24 pre-norm blocks with 2-D rotary attention per image, the 2x2 merge and the merger. Its projections and attention products run on prefill/gemm_nax.metal's template, with a few added instantiations. The rows are written over the placeholders' embeddings in fwd.embed, and one position on for the MTP head.
   • Preprocessing follows 0.6.6's vision/images.py: EXIF orientation, alpha over white with PIL's DIV255 blend, then mlx-vlm's Glm5NextImageProcessor sizing and PIL's 8-bit bicubic resampler, ported exactly.

   Checks against 0.6.6 + mlx-vlm

   • Patches are bit-exact on every PNG case: padding, upscaling, antialiased downscaling, RGBA, palette with transparency, gray+alpha. JPEG differs by decoder (ImageIO against libjpeg-turbo).
   • The tower's output rows are as close to an fp32 run of the same tower as mlx-vlm's own bf16 rows: cosine 0.98720 against 0.98718 on one image. mlx-vlm's GPU and CPU bf16 runs agree only to 0.98161.
   • Served, the Spark recipe's vision suite scores 12/12.

   One open question for this issue: the decoder. We used ImageIO because it's on every Mac and reads HEIC and WebP. The proposal here vendors stb_image, which also covers Linux/CUDA. A single decoder for both backends would mean stb_image (JPEG/PNG/GIF), with HEIC and WebP refused. Alternatively, a platform decoder behind the same prepare hook would let each backend use its own. Which do you prefer? We'd shape the server PR around the answer.
2. ashhart commented on Oct 10, 2026

   

   ashhart

   on Oct 10, 2026

   Owner

   More actions

   Thank you, @BobClawblaw, and for the GLM-5.3-Flash vision update. An answer on the dependencies: FFmpeg comes in as an optional library loaded at runtime with dlopen, from a pinned version, used only when a request carries images or video, and the server builds and serves text without it. Nothing is vendored into the tree, so no stb_image copy: images decode through the same runtime library or the platform's own decoder.

   The handoff you describe fits: preparation in the server, prepared patches and grids to the engine, and Info.vision as the engine's offer. The gate is bit equality with the Python frontend's tensors on a fixed image and video set, the expanded prompt text, the refusals, and the URL rules tested against local fixtures. Please land it with the first family that reads Info.vision, images and data URLs first, then --vision-urls, then video. Count images from the request's structured parts, never from text (#511 shows why), and keep WebP refused by name until a decoder exists.
3. BobClawblaw commented on Oct 10, 2026

   

   BobClawblaw

   on Oct 10, 2026

   ContributorAuthor

   More actions

   @ashhart, the plan for video, the last step here after #598 (images) and #614 (--vision-urls). 0.6.6 served video only for Flash Next on CUDA, never for GLM-5.3-Flash. So before code, here's the reference I'd hold GLM's video to, and two limits to agree on.

   What GLM-5.3-Flash does with video. The checkpoint ships Glm5NextVideoProcessor (transformers 5.19).

   • Sampling: frames at 2 a second, deduplicated, and padded to an even count.
   • Frame pairs: each pair becomes one temporal patch. The tower sees each pair as an image with grid (1, h, w) (get_video_features flattens the video grid), so the Metal tower in Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598 needs no new kernels.
   • Prompt: the template's <|begin_of_video|><|video|><|end_of_video|> becomes one <|begin_of_image|><|image|>×n<|end_of_image|>T.T seconds block per pair, inside the video markers.

   The reference: transformers 5.19's Glm5NextProcessor on frames decoded by PyAV 19.0.1, which bundles FFmpeg 9.0.2. That's load_video(backend="pyav") with the processor's own sample_frames. transformers' own default decoder is torchcodec, which needs a system FFmpeg; PyAV pins one.

   • Gate: pixel_values_videos bit-equal, the same video_grid_thw, and the same expanded prompt text, on a fixed set of clips: H.264 MP4, VP9 WebM, MOV and MKV; odd sizes; clips shorter than a second and over the frame cap. Plus the refusals and the URL rules.

   How it's built, per your dependency rule:

   • libtfvideo: a small C library over FFmpeg 9.0.2's libavformat, libavcodec and libswscale. It's built by an opt-in step pinned to that version, like Native structured output: libtfgrammar (xgrammar, loaded at runtime), masks in the lane core, enforced by Qwen3.5-2B on Metal #596's grammar library, and loaded with dlopen only when a request carries a video. Nothing is vendored. The server builds and serves without it, and refuses a video by name when it's missing.
   • What it does: it opens the container, reports the stream's frame count, rate and duration as PyAV does, and decodes only the sampled indices to RGB the way PyAV's to_ndarray(format="rgb24") does.
   • Each frame is resized as it's decoded, so a long 4K clip never holds its full-size frames. The resize is the processor's: torchvision's antialiased bicubic on uint8, then the zero pad and normalisation.
   • The family's processing goes behind Info.vision: a video offer next to prepare, holding the sampling rule and a prepare_frames hook. The pairs come back as prepared images, each with its own content hash for the cache key.

   Two limits to agree on:

   1. Visual tokens. The processor's default budget is 240,000 tokens a video. A minute of 1080p at 2 frames a second fits under it at about 160k tokens of tower work. I'd add --vision-video-tokens (default 16,384) and pass it as the processor's max_image_tokens. The reference runs with the same value, so equality holds.
   2. Frames. I'd cap max_frames at 0.6.6's 256 (the processor allows 2,048), again passed to the reference. The rest would be 0.6.6's VideoLimits:
      • 2 videos, 16 MB each and 20 MB a request across images and videos;
      • 3,600 s of footage, 8192 px a side;
      • MP4, WebM, QuickTime and Matroska;
      • 20 s a download and 120 s a request under --vision-urls;
      • video_url in user messages only.

   If either default should differ, say so and I'll shape the PR around it. I'm starting on libtfvideo and the fixture tool meanwhile, since neither depends on the answers.
4. sethforprivacy commented on Oct 12, 2026

   

   sethforprivacy

   on Oct 12, 2026

   Contributor

   More actions

   Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

   A note for when Flash Next's consumer of these inputs lands. We serve BobClawblaw's flashnext-video branch in production as of today (Mac Studio M5 Ultra, TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP @ 45032a1). Two changes were needed for a coding agent that sends screenshots mid-conversation. Both are in add3da3 on sethforprivacy:flashnext-lane-parity.

   Image prompts kept and resumed. On the branch, an image prompt's frame is shifted by its images' total (4a9c3ae), so a prefix's positions depend on every later image, and 6c8cb88 never keeps image prompts. Each turn of an image conversation re-reads its whole history, with cached_tokens=0 every time. We moved the shared sessions to the standard frame, the one the Python engine's VisionRope uses:

   • Prefix rows: a prefix's rows sit where every longer prompt puts them.
   • Rows past the prompt: they rotate at their slot plus one per-stream delta, which verify adds to pos8.
   • Indexer: pools each block at its first row's three positions (a small fz_idx_pool3). Text keeps the recorded kernel and its bits.
   • Tower: only images past the resumed prefix go through it.

   A four-turn conversation (an image, text, a second image, text; a ~5K-token system prompt and a few hundred tokens of notes each turn) resumes every turn with 5,551 / 6,041 / 6,725 tokens cached. Every reply equals a server with no prompt cache, character for character.

   Image budgets. Each image gets at most 4,096 tokens while a request's images fit --vision-image-tokens together, and an equal share past that. This is the policy the Python engine ended up with: an image's grid no longer depends on the images after it, so a resumed prefix stays valid when a screenshot is added. The cap rises to 262,144, the tower keeps scratch for one image (1.73 GiB at 4,096 tokens), and video is offered only when libtfvideo sits beside the server. Otherwise a video part gets a 400, not a 500.

   Checked on the M5 Ultra, all passing:

   • 32 images taken and the 33rd refused.
   • Image streams beside text streams equal their solo replies (4/4).
   • Drafted replies equal serial ones.
   • Both turns of the image conversation check resume.
5. ashhart commented on Oct 14, 2026

   

   ashhart

   on Oct 14, 2026

   Owner

   More actions

   Hello @sethforprivacy and @BobClawblaw,

   Thank you for the four-turn resume. 5,551, then 6,041, then 6,725 tokens cached, and each reply equal to a server with no prompt cache, is the check this has to keep.

   Main still refuses image, audio and video by name (zig/src/server/fields.zig:317). zig/src/server/cli.zig:53 parses --vision-urls, and nothing fetches a URL.

   #614 and #618 are the native image and video pulls. They are not on this main. They wait on the JPEG scope call in #598. A prefix has to sit where every longer prompt puts it, which is the rule you wrote, and those pulls have to keep it.

   Thank you both.
6. ashhart commented on Oct 16, 2026

   

   ashhart

   on Oct 16, 2026

   Owner

   More actions

   Thank you, @BobClawblaw, for laying out the engine-independent part first, and @sethforprivacy for the resume checks that set the bar.

   TensorFold 1.0.12 serves image input natively, prepared as the Python frontend prepared it: image_url parts in user messages and tool results, data URLs, and public HTTPS URLs behind --vision-urls with every address and redirect checked (#598, #614). GLM-5.3-Flash also reads video_url parts through the optional libtfvideo (#618). Pictures decode in one shared place to the pixels 0.6.6 loaded, and the prompt cache keys each image by its content and grid, so a kept state never resumes another image. GLM-5.3-Flash and Qwen3.8-27B read images on a Mac, and Nemotron 3.5 Super VL on CUDA.

   Closing this, since the server side it asked for is in. Flash Next's consumer and the 27B's images on CUDA are their own items, coordinated across #836, #856 and #885.

   Posted by Ash's AI Assistant Synthena
7. ashhart

   closed this as completedon Oct 16, 2026