Repository navigation
Edit pack manifests are the same on every box (fixes #84) - #130
Merged
Merged
Conversation
te/manifest.json stored the vision tower's whole transformers `to_dict()`, which includes `transformers_version`. The twin venv installs `transformers>=4.57`, so a box with 5.20.2 (the reporter's) and one with 5.21.0 (DGX-02) wrote different te/manifest.json bytes, and edit.json (which hashes it) differed too, while every weight file was identical: substituting "5.20.2" into DGX-02's manifest reproduces the reporter's a7014666... and 79baea31... exactly. - pack_edit.write_vision stores only the checkpoint's architecture fields (VISION_CONFIG, in a fixed order); the engine reads a subset of them. - pack_edit.normalise rewrites an existing pack's te/manifest.json that way (byte for byte what write makes) and its edit.json; prepare.sh runs it after the edit step, so boxes upgrade without a re-pack. It replaces files instead of writing through them, so a hard-linked store is left alone. - built/qwen-image-2.1-turbo.json pins the new te/manifest.json and edit.json; weights and vae/ are unchanged. - tools/twin/pod/setup.sh pins transformers==5.21.0 (what the pins were built with). The image-packs stamp is the sha256 of the pin file, so a box stamped with the old pins runs image/packs again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #84
Cause
te/manifest.jsonstored the vision tower's whole transformersto_dict(), and that includestransformers_version. The twin venv installstransformers>=4.57, so the reporter's Spark (5.20.2) and DGX-02 (5.21.0) wrote different manifests from byte-identical weights.edit.jsonhashes that manifest, so it differed as well.Proof: substituting
"5.20.2"into DGX-02's manifest reproduces the reporter's hashes exactly.te/manifest.jsonde7a0a75…a7014666…a7014666…edit.jsond426ea09…79baea31…79baea31…Fix (deterministic bytes, not a looser check)
pack_edit.write_visionstores only the checkpoint's architecture fields (VISION_CONFIG, in a fixed order). The engine (vision.zig) reads a subset of them.pack_edit.normaliserewrites an existing pack's manifest to exactly those bytes and remakesedit.json. It touches metadata only, so there is no re-pack and no GPU work.prepare.shruns it after the edit step, which upgrades existing boxes. It writes a temp file and renames it, so a hard-linked store stays untouched.te/manifest.json130922 B91fd46ff…andedit.json64638c30…. The weights andvae/pins are unchanged.setup.shpinstransformers==5.21.0, the version the pins were built with, so the packing code is the same on every box.I chose this over semantic checking in
verify_assets.py: nothing in the manifest needs to vary per box, and exact sha256 pins stay simple and strict.Evidence (DGX-02, CPU checks in the NGC container)
te/manifest.jsonandedit.json.normalisechanges nothing.write_visionrun there under transformers 5.20.2 and 5.21.0, with config keys in reversed order, writes byte-identical manifests. Those equalnormaliseapplied to the old format.verify_assets.pywith the new pins (11/11, 22.6 GiB).image_edit, and an edit returns a 1024x1024 PNG (16 s, cold load included).image-packsstamp made with the old pin file reports not done against the new one, soimage/packsruns again. The stamp is the pin file's sha256.py_compile,bash -non both scripts,jqon the pins, all exit 0. No.zigfiles were touched.Risk
prepare.shruns. Until then, the oldte/manifest.jsonfails the new pins. Serving is unaffected: the engine reads the same fields either way.🤖 Generated with Claude Code