Skip to content

Edit pack manifests are the same on every box (fixes #84) - #130

Merged
jayleaton merged 1 commit into
mainfrom
fix/edit-pack-deterministic
Oct 28, 2026
Merged

jayleaton merged 1 commit into
mainfrom
fix/edit-pack-deterministic

Conversation

@jayleaton

Copy link
Copy Markdown
Owner

Fixes #84

Cause

te/manifest.json stored the vision tower's whole transformers to_dict(), and that includes transformers_version. The twin venv installs transformers>=4.57, so the reporter's Spark (5.20.2) and DGX-02 (5.21.0) wrote different manifests from byte-identical weights. edit.json hashes that manifest, so it differed as well.

Proof: substituting "5.20.2" into DGX-02's manifest reproduces the reporter's hashes exactly.

File DGX-02 DGX-02 with "5.20.2" substituted Reporter
te/manifest.json de7a0a75… a7014666… a7014666…
edit.json d426ea09… 79baea31… 79baea31…

Fix (deterministic bytes, not a looser check)

  • pack_edit.write_vision stores only the checkpoint's architecture fields (VISION_CONFIG, in a fixed order). The engine (vision.zig) reads a subset of them.
  • pack_edit.normalise rewrites an existing pack's manifest to exactly those bytes and remakes edit.json. It touches metadata only, so there is no re-pack and no GPU work. prepare.sh runs it after the edit step, which upgrades existing boxes. It writes a temp file and renames it, so a hard-linked store stays untouched.
  • New pins: te/manifest.json 130922 B 91fd46ff… and edit.json 64638c30…. The weights and vae/ pins are unchanged.
  • setup.sh pins transformers==5.21.0, the version the pins were built with, so the packing code is the same on every box.

I chose this over semantic checking in verify_assets.py: nothing in the manifest needs to vary per box, and exact sha256 pins stay simple and strict.

Evidence (DGX-02, CPU checks in the NGC container)

  • Both variants agree: our pack and the reporter's variant normalise to the same te/manifest.json and edit.json.
  • Only the vision config changes: JSON round-tripping the real manifest is the identity, so nothing else in the file moves. A second normalise changes nothing.
  • Two directories: write_vision run there under transformers 5.20.2 and 5.21.0, with config keys in reversed order, writes byte-identical manifests. Those equal normalise applied to the old format.
  • Real pack: DGX-02's pack, normalised, passes verify_assets.py with the new pins (11/11, 22.6 GiB).
  • Engine: a test daemon serving the image model only loads the normalised pack, lists image_edit, and an edit returns a 1024x1024 PNG (16 s, cold load included).
  • Upgrade path: an image-packs stamp made with the old pin file reports not done against the new one, so image/packs runs again. The stamp is the pin file's sha256.
  • Checks: py_compile, bash -n on both scripts, jq on the pins, all exit 0. No .zig files were touched.

Risk

  • Boxes with an edit pack get one metadata rewrite the next time prepare.sh runs. Until then, the old te/manifest.json fails the new pins. Serving is unaffected: the engine reads the same fields either way.
  • New twin venvs install exactly transformers 5.21.0.

🤖 Generated with Claude Code

te/manifest.json stored the vision tower's whole transformers `to_dict()`, which includes `transformers_version`.
The twin venv installs `transformers>=4.57`, so a box with 5.20.2 (the reporter's) and one with 5.21.0 (DGX-02) wrote
different te/manifest.json bytes, and edit.json (which hashes it) differed too, while every weight file was identical:
substituting "5.20.2" into DGX-02's manifest reproduces the reporter's a7014666... and 79baea31... exactly.

- pack_edit.write_vision stores only the checkpoint's architecture fields (VISION_CONFIG, in a fixed order); the
  engine reads a subset of them.
- pack_edit.normalise rewrites an existing pack's te/manifest.json that way (byte for byte what write makes) and its
  edit.json; prepare.sh runs it after the edit step, so boxes upgrade without a re-pack. It replaces files instead
  of writing through them, so a hard-linked store is left alone.
- built/qwen-image-2.1-turbo.json pins the new te/manifest.json and edit.json; weights and vae/ are unchanged.
- tools/twin/pod/setup.sh pins transformers==5.21.0 (what the pins were built with).

The image-packs stamp is the sha256 of the pin file, so a box stamped with the old pins runs image/packs again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton
jayleaton merged commit 059b3ad into main Oct 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fresh jarvis-gb10 install fails at image/packs: the built pins predate the edit pack from #78

1 participant