Repository navigation
Conversation
The default ASR model was loaded by name with no revision, so the Hub repo's 2026-10-21 multilingual release silently changed every transcript (now capitalized and punctuated) and broke the GPU tests (#155). The default also lived only in the library, so a checkpoint's config did not show which ASR it ran. Compose with --enable-audio now always writes asr_model_id and a new asr_model_revision; the default model is pinned to its English-only revision 947f59a, and --asr-revision sets it for a custom model (left unpinned with a warning otherwise). Serving has no fallback: an audio checkpoint without asr_model_id fails at startup with the config fix. This is breaking for checkpoints composed before 0.2.0, hence the version bump. For airgapped customers (#158), --bundle-asr copies the ASR into the checkpoint's asr/ folder, sets asr_bundled_path, and checks at compose time that it loads with no network. Serving loads a bundled ASR from disk, or fetches only asr/ when served by Hub repo id. --asr-model may be a local directory so the bundle can be built offline. test_asr_ctc_default_gpu.py no longer asserts transcript text, which is model quality and not ours to own; it now loads the pinned revision. Fixes #155 Fixes #158 Signed-off-by: aviv ron <rona@il.ibm.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
/gpu-test-multi |
Two stubs in test_audio_processor.py still had get_transcriber's old signature, so the processor's new revision= argument made them raise TypeError: 13 tests failed on the vllm30 GPU leg (14 on vLLM < 0.29, where test_uncached_path_reports_updates_not_applied also runs). The stubs ignore their arguments, so they now take **kwargs and do not break on the next signature change either. Signed-off-by: aviv ron <rona@il.ibm.com>
✅ GPU tests passed —
|
aviv1ron1
force-pushed
the
feature/release-test-tier
branch
from
November 3, 2026 22:55
a56ebca to
f5abeb9
Compare
✅ GPU tests passed —
|
Collaborator
Author
|
/gpu-test-multi |
bundle_asr_model wrote into an existing <output>/asr/ with dirs_exist_ok and never removed what was there. Re-composing into the same --output with a different ASR model left the first model's shards, tokenizer and feature-extractor files beside the new ones; the offline validation could still pass, and serving then loaded a mix of two models. The destination is now removed first, and a local --asr-model that is the destination itself is refused rather than deleted. Signed-off-by: aviv ron <rona@il.ibm.com>
The copytree ignore for a local --asr-model only filtered files, so every subdirectory was copied whole: a git-lfs clone brought its .git/ object store (about another 1 GB), and onnx/ or nested .bin weights landed in the bundle although the patterns exist to keep them out. The Hub path filters nested files through allow_patterns, so the two sources gave different bundles. Local files are now matched by their relative path with huggingface_hub's own filter_repo_objects, and hidden directories are not walked. Signed-off-by: aviv ron <rona@il.ibm.com>
The ASR bundle was downloaded and validated only after save_pretrained had written a config.json with asr_bundled_path set. A failed download, or a model with no safetensors weights, raised with the weights and that config already on disk: the output looked complete and failed only at serve time. The bundle now runs first, so a failure stops the compose before the config exists. It is also built in a staging directory and swapped in only once it validates, so a failed re-compose leaves a previous good bundle in place instead of an empty or partial asr/. Signed-off-by: aviv ron <rona@il.ibm.com>
resolve_asr_source fetched a Hub checkpoint's bundle with the path stripped of slashes but joined it to the root unstripped, so "/asr" downloaded asr/* and then looked in the filesystem's /asr. Nothing stopped a value such as "../../x", from a config that may come off the Hub, pointing the ASR at any local directory. The path is now normalized once, used for both the fetch pattern and the load path, and refused when absolute, "." or reaching outside with "..". Signed-off-by: aviv ron <rona@il.ibm.com>
A local directory as --asr-model was accepted with plain --enable-audio and written verbatim into asr_model_id, so the checkpoint pointed at a path that exists on the build machine only: it passed startup and then failed on the first audio request once moved or uploaded. With --bundle-asr the same full path, e.g. a home directory, went into the published config and model card. A local directory is now refused unless --bundle-asr is set, and is then recorded by its directory name only; the bundle itself is built from --asr-model. The ASR model is resolved before the model is built, so a refused value fails the compose in seconds. Signed-off-by: aviv ron <rona@il.ibm.com>
Collaborator
Author
|
/gpu-test-multi |
1 similar comment
Collaborator
Author
|
/gpu-test-multi |
aviv1ron1
marked this pull request as ready for review
November 5, 2026 09:27
aviv1ron1
requested review from
antonpibm,
freunda and
yairallouche
as code owners
November 5, 2026 09:27
A new `release` pytest marker, excluded by default (addopts and every suite's -m now carry "not release"), run with the new `release` GPU suite: -m "release and not deep" (/gpu-test-release). For tests that check real external artifacts, and for the non-representative cases of the expensive GPU matrices; PR runs keep one case per matrix. Release only: test_asr_ctc_default_gpu.py (the real default ASR), TestRealHubMetadata, test_quantization.py, and test_granite4_fullsize's GPU class. Split by pytest.param marks: test_granite4_mini.py (4.0-micro on PRs), test_multi_adapter_equivalence.py (4.1-3b on PRs, 4.2-3b in release), test_audio_serving_smoke.py (4.0-micro on PRs, 4.1-3b in release). The two vLLM audio serving tests compose with --bundle-asr and the 30 KB local CTC model (tests/shared/tiny_asr.py), so no PR run downloads the real ASR, and they also cover serving a bundled ASR. test_quantization.py is gated on GRANITE_SWITCH_E2E_MODELS=1, which no GPU pod sets, so it never ran in CI; tests/conftest.py now sets it when -m selects release. test_adapter_routing_audio_enabled.py stays on PRs but composes 13 empty built-in slots on 4.0-micro instead of three real adapter libraries on 4.1-3b: routing reads only the control-token table. Signed-off-by: aviv ron <rona@il.ibm.com>
PR runs no longer download any real model. Every test that needs a Granite base or adapter library is parametrized over two sources (tests/shared/model_sources.py): a fake on every PR run, and the real model marked `release`. The fakes are captured, not invented. tests/shared/fixtures/ holds verbatim copies of each base's config, generation config, tokenizer config, special-tokens map and chat template, and each adapter's adapter_config.json and io.yaml (`--capture` refreshes them). fake_bases.py adds tiny random weights in the real dtype and a byte-level tokenizer whose special tokens, and the words tests use as probe labels, keep their real ids; fake_adapter_libraries.py adds synthetic weights at each adapter's real rank and target modules. The real compose CLI accepts them, and the fake tokenizers match the real ones on special-token ids, added vocab and template render. Converted: test_compose_e2e (which composed 4.1-3b despite naming 4.0-micro), test_save_load_compose, test_multi_audio_compose_e2e, test_probe_compose, test_upstream_files, test_lora_substitute_probe (its 4.2 case, which always skipped, now runs), test_audio_marker _output_row, test_adapter_filtering, test_adapter_loader, and the GPU tests test_audio_serving_smoke, test_audio_uncached_processor, test_adapter_routing_audio_enabled, test_multi_adapter_equivalence, test_probe_chunked_prefill and test_tp_integration. Probe tests that skipped on a cold HF cache now always run. Release only, with no fake counterpart: test_generation_equivalence and test_skinning_equivalence (numerics vs upstream on real weights) and the BPE split property in test_chat_template. The fixtures are excluded from the whitespace hooks: a fixed-up final newline changes what a captured chat template renders. Signed-off-by: aviv ron <rona@il.ibm.com>
The placeholder model.safetensors.index.json had an empty weight_map. vLLM's loader trusts an index when one exists and loads only the files it names, so it found no weights and test_multi_adapter_equivalence's upstream run on the fake base failed with "Cannot find any model weights" (all 8 fake cases errored on vllm26). The index now maps every tensor to the single shard, as a real one-shard repo index does. Signed-off-by: aviv ron <rona@il.ibm.com>
test_multi_adapter_equivalence's `distinct` check failed on the fake base for both flavors: every adapter was indistinguishable from base (JSD 0.000000), so its passthrough/slot/batch checks had been passing vacuously. The fake base was effectively embedding -> norm -> head. At body std 0.02 the 12x embedding_multiplier swamped the decoder layers, whose output residual_multiplier 0.22 shrinks further: zeroing every attention or MLP output left the logits unchanged, so no adapter could reach them at any strength. And the output was uniform (entropy 11.516 of 11.516 nats) under logits_scaling 10. Now body and embedding std 1.0, and a final-norm gain of 4: dropping attention or the MLP moves the output by JSD 0.98 / 0.99, entropy is ~4 nats. The real Granite multipliers are unchanged. With that, the synthetic adapters clear `distinct` at their real-model calibration (LoRA 0.29-0.89, SR 0.85-0.99, gate 0.01) and passthrough still equals upstream exactly, measured on CPU through the worker's own compose. Signed-off-by: aviv ron <rona@il.ibm.com>
…to 3 The suite compares three models per variant: upstream, the adapter-free switch and the zero-LoRA switch. Each of its five tests booted a fresh pair of engines to score one prompt, so a variant cost ten boots, five of them the identical upstream model; at ~9+9+5 min it was the largest single file in a GPU run. A module fixture in _granite4_mini_tests.py now builds the three checkpoints once and boots each model once, scoring every prompt its tests need. The outer test runs a variant's three classes in one pytest process (it ran one per class) so the fixture is shared. Checkpoints, seeds and prompts are the ones the per-test runners built. _vllm_logprobs_worker.py and run_vllm_logprobs_many score several prompts in one engine, with one generate call per prompt: batching could move logprobs by itself, and these gates sit at ~1e-5, so each prompt is still scored exactly as it was alone. Prefix caching stays off in every engine, so an earlier prompt cannot affect a later one. run_vllm_logprobs is now a one-prompt wrapper; other callers unchanged. Signed-off-by: aviv ron <rona@il.ibm.com>
…yers on request 8c411e0 raised the fake bases' random-weight scales so adapters reach the output, which fixed test_multi_adapter_equivalence's `distinct` check. It also grew intermediate hidden states from rms ~0.24 to ~340, and the exact-score tests failed on the fake cases: test_probe_chunked _prefill checks probe scores to +/- 0.3 against an fp32 reference, and at scores near 2000 bf16 rounds by ~16 (3 failures on vllm26). On a 2-layer base with the real Granite multipliers no single set of scales gives both live layers and small activations, so build_fake_base takes a WeightPreset. REALISTIC, the default, is bit-identical to the fake base every exact-score test passed on before 8c411e0. LIVE_LAYERS (body 1.0, embed 1.0, final norm 4) is used only by test_multi_adapter_equivalence's fake case, where `distinct` still clears its 0.01 gate (LoRA >= 0.29, SR >= 0.85). model_sources caches each preset in its own directory, keeping the model name as the leaf so adapter discovery still resolves the base flavour. Signed-off-by: aviv ron <rona@il.ibm.com>
aviv1ron1
force-pushed
the
feature/release-test-tier
branch
from
November 5, 2026 10:02
5b67886 to
4a47cec
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR runs no longer download any real model. Every test that needs a Granite base or an adapter library runs twice: on a fake on every PR run, and on the real model in a new
releasetest tier (pytest -m "release and not deep",/gpu-test-release), run before releases.The release tier also takes the non-representative cases of the expensive GPU matrices. On the current CI legs this is expected to move about 1h30m out of every full PR run down from 4h roughly to 2.5h
Reviewing: 123 of the 155 files are verbatim copies of published Hub files under
tests/shared/fixtures/, and can be skipped. The code is 32 files.How the fakes work
The fakes are captured, not invented:
tests/shared/fixtures/bases/holds verbatim copies of each base's config, generation config, tokenizer config, special-tokens map and chat template, forgranite-4.0-micro,4.1-3band4.2-3b.tests/shared/fixtures/adapter_libraries/holds each adapter'sadapter_config.jsonandio.yaml, from the four granitelib repos.--capturerefreshes both.At test time:
tests/shared/fake_bases.pyadds tiny random weights, in the real dtype and the geometry existing vLLM tiny-model tests use (head_dim 32). It also adds a byte-level tokenizer whose special tokens, and the words tests use as probe labels, keep their real ids.tests/shared/fake_adapter_libraries.pyadds synthetic weights at each adapter's real rank and target modules.The composer therefore sees the real chat template, token ids, configs and repo layout. The real compose CLI accepts the fakes, and the fake tokenizers match the real ones exactly on all three bases: special-token ids, added vocab, template render, bos/eos/pad.
The fakes leave out real weights and the BPE vocabulary: ordinary words tokenize to one token per byte. Tests that depend on either are release-only (below).
A test takes its model from
tests/shared/model_sources.pyand is parametrized overmodel_sources.sources(). See the "Release test tier" section ofdocs/CICD.md.Converted: fake on PRs, real in release
test_compose_e2e,test_save_load_compose,test_multi_audio_compose_e2e,test_probe_compose,test_upstream_files,test_lora_substitute_probe,test_audio_marker_output_row.test_adapter_filteringandtest_adapter_loaderused the real config only to pick the architecture, so they now use the fake alone.test_audio_serving_smoke,test_audio_uncached_processor,test_adapter_routing_audio_enabled,test_multi_adapter_equivalence,test_probe_chunked_prefill,test_tp_integration.Release only, with no fake counterpart
tests/integration/test_asr_ctc_default_gpu.pytests/shared/tiny_asr.py)tests/vllm/test_generation_equivalence.py,tests/composer/test_skinning_equivalence.pytest_granite4_mini.pytest_chat_template.py::TestInvocationFirstCharDropPropertytest_selective_download.py::TestRealHubMetadatatests/hf/test_quantization.pytests/vllm/test_granite4_fullsize.py(GPU class)Breadth matrices: PR runs keep one case; release adds the rest.
test_granite4_mini.py:4.0-microon PRs;4.0-1band4.0-350min release.test_multi_adapter_equivalence.py: a fake 4.1-3b on PRs; the real 4.1-3b and 4.2-3b in release.test_audio_serving_smoke.py: a fake 4.0-micro on PRs; the real 4.0-micro and 4.1-3b in release.Fixes found along the way
test_compose_e2e.pycomposed the wrong base: it never passed--base-model, so it downloaded the CLI default 4.1-3b while its constants claimed 4.0-micro.test_lora_substitute_probe, which always skipped;test_probe_composeandtest_multi_audio_compose_e2e, which skipped whenever the CI HF cache was cold.test_quantization.pynever ran in CI: all 16 tests were skipped on vllm31 and vllm32, behindGRANITE_SWITCH_E2E_MODELS=1, which no pod sets.tests/conftest.pynow sets it when-mselectsrelease.pytest.skipon a fetch failure, so in the release tier a broken Hub fetch is a failure, not a skip.Other changes
addopts, theaudiosuite andci.yamlexcluderelease. Thereleasesuite runs-m "release and not deep"..pre-commit-config.yaml: the whitespace hooks skiptests/shared/fixtures/, because a final newline added to a captured chat template changes what it renders.