Skip to content

Feature/release test tier - #167

Open
aviv1ron1 wants to merge 13 commits into
mainfrom
feature/release-test-tier
Open

aviv1ron1 wants to merge 13 commits into
mainfrom
feature/release-test-tier

Conversation

@aviv1ron1

Copy link
Copy Markdown
Collaborator

Summary

PR runs no longer download any real model. Every test that needs a Granite base or an adapter library runs twice: on a fake on every PR run, and on the real model in a new release test tier (pytest -m "release and not deep", /gpu-test-release), run before releases.

The release tier also takes the non-representative cases of the expensive GPU matrices. On the current CI legs this is expected to move about 1h30m out of every full PR run down from 4h roughly to 2.5h

Reviewing: 123 of the 155 files are verbatim copies of published Hub files under tests/shared/fixtures/, and can be skipped. The code is 32 files.

How the fakes work

The fakes are captured, not invented:

  • tests/shared/fixtures/bases/ holds verbatim copies of each base's config, generation config, tokenizer config, special-tokens map and chat template, for granite-4.0-micro, 4.1-3b and 4.2-3b.
  • tests/shared/fixtures/adapter_libraries/ holds each adapter's adapter_config.json and io.yaml, from the four granitelib repos.
  • --capture refreshes both.

At test time:

  • tests/shared/fake_bases.py adds tiny random weights, in the real dtype and the geometry existing vLLM tiny-model tests use (head_dim 32). It also adds a byte-level tokenizer whose special tokens, and the words tests use as probe labels, keep their real ids.
  • tests/shared/fake_adapter_libraries.py adds synthetic weights at each adapter's real rank and target modules.

The composer therefore sees the real chat template, token ids, configs and repo layout. The real compose CLI accepts the fakes, and the fake tokenizers match the real ones exactly on all three bases: special-token ids, added vocab, template render, bos/eos/pad.

The fakes leave out real weights and the BPE vocabulary: ordinary words tokenize to one token per byte. Tests that depend on either are release-only (below).

A test takes its model from tests/shared/model_sources.py and is parametrized over model_sources.sources(). See the "Release test tier" section of docs/CICD.md.

Converted: fake on PRs, real in release

  • Composer: test_compose_e2e, test_save_load_compose, test_multi_audio_compose_e2e, test_probe_compose, test_upstream_files, test_lora_substitute_probe, test_audio_marker_output_row.
  • test_adapter_filtering and test_adapter_loader used the real config only to pick the architecture, so they now use the fake alone.
  • GPU: test_audio_serving_smoke, test_audio_uncached_processor, test_adapter_routing_audio_enabled, test_multi_adapter_equivalence, test_probe_chunked_prefill, test_tp_integration.

Release only, with no fake counterpart

Test Why a fake can't stand in
tests/integration/test_asr_ctc_default_gpu.py the real default ASR at its pinned revision; every other audio test uses a 30 KB local ASR (tests/shared/tiny_asr.py)
tests/vllm/test_generation_equivalence.py, tests/composer/test_skinning_equivalence.py compares our model against upstream on real weights, with thresholds tuned on them; the same check on random weights runs on PRs in test_granite4_mini.py
test_chat_template.py::TestInvocationFirstCharDropProperty depends on how the real BPE vocabulary splits strings
test_selective_download.py::TestRealHubMetadata the live Hub repo layout
tests/hf/test_quantization.py the quantization libraries on a real checkpoint
tests/vllm/test_granite4_fullsize.py (GPU class) the mini file covers the same equivalence at small size

Breadth matrices: PR runs keep one case; release adds the rest.

  • test_granite4_mini.py: 4.0-micro on PRs; 4.0-1b and 4.0-350m in release.
  • test_multi_adapter_equivalence.py: a fake 4.1-3b on PRs; the real 4.1-3b and 4.2-3b in release.
  • test_audio_serving_smoke.py: a fake 4.0-micro on PRs; the real 4.0-micro and 4.1-3b in release.

Fixes found along the way

  • test_compose_e2e.py composed the wrong base: it never passed --base-model, so it downloaded the CLI default 4.1-3b while its constants claimed 4.0-micro.
  • Tests that skipped silently now always run:
    • the 4.2 case in test_lora_substitute_probe, which always skipped;
    • the probe tests in test_probe_compose and test_multi_audio_compose_e2e, which skipped whenever the CI HF cache was cold.
  • test_quantization.py never ran in CI: all 16 tests were skipped on vllm31 and vllm32, behind GRANITE_SWITCH_E2E_MODELS=1, which no pod sets. tests/conftest.py now sets it when -m selects release.
  • Real-tokenizer checks fail loudly: they no longer pytest.skip on a fetch failure, so in the release tier a broken Hub fetch is a failure, not a skip.

Other changes

  • Marker filters: addopts, the audio suite and ci.yaml exclude release. The release suite runs -m "release and not deep".
  • .pre-commit-config.yaml: the whitespace hooks skip tests/shared/fixtures/, because a final newline added to a captured chat template changes what it renders.

The default ASR model was loaded by name with no revision, so the Hub
repo's 2026-10-21 multilingual release silently changed every
transcript (now capitalized and punctuated) and broke the GPU tests
(#155). The default also lived only in the library, so a checkpoint's
config did not show which ASR it ran.

Compose with --enable-audio now always writes asr_model_id and a new
asr_model_revision; the default model is pinned to its English-only
revision 947f59a, and --asr-revision sets it for a custom model (left
unpinned with a warning otherwise). Serving has no fallback: an audio
checkpoint without asr_model_id fails at startup with the config fix.
This is breaking for checkpoints composed before 0.2.0, hence the
version bump.

For airgapped customers (#158), --bundle-asr copies the ASR into the
checkpoint's asr/ folder, sets asr_bundled_path, and checks at compose
time that it loads with no network. Serving loads a bundled ASR from
disk, or fetches only asr/ when served by Hub repo id. --asr-model may
be a local directory so the bundle can be built offline.

test_asr_ctc_default_gpu.py no longer asserts transcript text, which is
model quality and not ours to own; it now loads the pinned revision.

Fixes #155
Fixes #158

Signed-off-by: aviv ron <rona@il.ibm.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@aviv1ron1

Copy link
Copy Markdown
Collaborator Author

/gpu-test-multi

Two stubs in test_audio_processor.py still had get_transcriber's old
signature, so the processor's new revision= argument made them raise
TypeError: 13 tests failed on the vllm30 GPU leg (14 on vLLM < 0.29,
where test_uncached_path_reports_updates_not_applied also runs). The
stubs ignore their arguments, so they now take **kwargs and do not
break on the next signature change either.

Signed-off-by: aviv ron <rona@il.ibm.com>
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

✅ GPU tests passed — vllm26-multi

2949 passed, 51 skipped, 140 deselected, 21 warnings in 6230.80s (1:43:50)

Commit: 5b678864f610f882e91c012c9557418f514a8bc4
Full run & artifact log

Last 40 log lines
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

tests/composer/test_compose_e2e.py:117
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:117: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:132
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:132: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:138
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:138: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:249
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:249: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:232
  /tmp/granite-switch/tests/composer/test_upstream_files.py:232: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
== 2949 passed, 51 skipped, 140 deselected, 21 warnings in 6230.80s (1:43:50) ==
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1104 16:10:30.398417406 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
===== ALL GPU TESTS PASSED =====
[16:10:43] <job> Succeeded
[16:10:46] verified: found success sentinel in pod log
[16:10:46] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@aviv1ron1
aviv1ron1 force-pushed the feature/release-test-tier branch from a56ebca to f5abeb9 Compare November 3, 2026 22:55
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

✅ GPU tests passed — vllm30-multi

2948 passed, 52 skipped, 140 deselected, 24 warnings in 6614.24s (1:50:14)

Commit: 5b678864f610f882e91c012c9557418f514a8bc4
Full run & artifact log

Last 40 log lines
tests/composer/test_compose_e2e.py:117
  /tmp/granite-switch/tests/composer/test_compose_e2e.py:117: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:132
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:132: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:138
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:138: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_multi_audio_compose_e2e.py:249
  /tmp/granite-switch/tests/composer/test_multi_audio_compose_e2e.py:249: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("multi_audio_compose_e2e")

tests/composer/test_upstream_files.py:232
  /tmp/granite-switch/tests/composer/test_upstream_files.py:232: PytestUnknownMarkWarning: Unknown pytest.mark.xdist_group - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.xdist_group("upstream_build_e2e")

tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[44100-16000]
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/torch/backends/mkldnn/__init__.py:96: UserWarning: TF32 acceleration on top of oneDNN is available for Intel GPUs. The current Torch version does not have Intel GPU Support. (Triggered internally at /__w/pytorch/pytorch/aten/src/ATen/Context.cpp:265.)
    torch._C._set_onednn_allow_tf32(_allow_tf32)

tests/vllm/test_probe_capture_safety.py::test_sr_pipeline_stack_partitions_post_attention_metadata
  /tmp/granite-switch/.venv/lib/python3.12/site-packages/flashinfer/cute_dsl/availability.py:135: DeprecationWarning: Arch from arch.py is deprecated. Use Arch from enums.py instead.
    from cutlass.base_dsl.arch import Arch

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
== 2948 passed, 52 skipped, 140 deselected, 24 warnings in 6614.24s (1:50:14) ==
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
[rank0]:[W1104 16:16:58.945866707 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
===== ALL GPU TESTS PASSED =====
[16:17:21] <job> Succeeded
[16:17:24] verified: found success sentinel in pod log
[16:17:24] cleanup: deleting <job> (exit=0)
<job> "<job>" deleted

Verdict: PASSED

@aviv1ron1

Copy link
Copy Markdown
Collaborator Author

/gpu-test-multi

bundle_asr_model wrote into an existing <output>/asr/ with
dirs_exist_ok and never removed what was there. Re-composing into the
same --output with a different ASR model left the first model's shards,
tokenizer and feature-extractor files beside the new ones; the offline
validation could still pass, and serving then loaded a mix of two
models. The destination is now removed first, and a local --asr-model
that is the destination itself is refused rather than deleted.

Signed-off-by: aviv ron <rona@il.ibm.com>
The copytree ignore for a local --asr-model only filtered files, so
every subdirectory was copied whole: a git-lfs clone brought its .git/
object store (about another 1 GB), and onnx/ or nested .bin weights
landed in the bundle although the patterns exist to keep them out. The
Hub path filters nested files through allow_patterns, so the two
sources gave different bundles. Local files are now matched by their
relative path with huggingface_hub's own filter_repo_objects, and
hidden directories are not walked.

Signed-off-by: aviv ron <rona@il.ibm.com>
The ASR bundle was downloaded and validated only after save_pretrained
had written a config.json with asr_bundled_path set. A failed download,
or a model with no safetensors weights, raised with the weights and that
config already on disk: the output looked complete and failed only at
serve time. The bundle now runs first, so a failure stops the compose
before the config exists. It is also built in a staging directory and
swapped in only once it validates, so a failed re-compose leaves a
previous good bundle in place instead of an empty or partial asr/.

Signed-off-by: aviv ron <rona@il.ibm.com>
resolve_asr_source fetched a Hub checkpoint's bundle with the path
stripped of slashes but joined it to the root unstripped, so "/asr"
downloaded asr/* and then looked in the filesystem's /asr. Nothing
stopped a value such as "../../x", from a config that may come off the
Hub, pointing the ASR at any local directory. The path is now
normalized once, used for both the fetch pattern and the load path,
and refused when absolute, "." or reaching outside with "..".

Signed-off-by: aviv ron <rona@il.ibm.com>
A local directory as --asr-model was accepted with plain --enable-audio
and written verbatim into asr_model_id, so the checkpoint pointed at a
path that exists on the build machine only: it passed startup and then
failed on the first audio request once moved or uploaded. With
--bundle-asr the same full path, e.g. a home directory, went into the
published config and model card.

A local directory is now refused unless --bundle-asr is set, and is
then recorded by its directory name only; the bundle itself is built
from --asr-model. The ASR model is resolved before the model is built,
so a refused value fails the compose in seconds.

Signed-off-by: aviv ron <rona@il.ibm.com>
@aviv1ron1

Copy link
Copy Markdown
Collaborator Author

/gpu-test-multi

1 similar comment
@aviv1ron1

Copy link
Copy Markdown
Collaborator Author

/gpu-test-multi

@aviv1ron1
aviv1ron1 marked this pull request as ready for review November 5, 2026 09:27
A new `release` pytest marker, excluded by default (addopts and every
suite's -m now carry "not release"), run with the new `release` GPU
suite: -m "release and not deep" (/gpu-test-release). For tests that
check real external artifacts, and for the non-representative cases
of the expensive GPU matrices; PR runs keep one case per matrix.

Release only: test_asr_ctc_default_gpu.py (the real default ASR),
TestRealHubMetadata, test_quantization.py, and test_granite4_fullsize's
GPU class. Split by pytest.param marks: test_granite4_mini.py
(4.0-micro on PRs), test_multi_adapter_equivalence.py (4.1-3b on PRs,
4.2-3b in release), test_audio_serving_smoke.py (4.0-micro on PRs,
4.1-3b in release).

The two vLLM audio serving tests compose with --bundle-asr and the
30 KB local CTC model (tests/shared/tiny_asr.py), so no PR run
downloads the real ASR, and they also cover serving a bundled ASR.

test_quantization.py is gated on GRANITE_SWITCH_E2E_MODELS=1, which no
GPU pod sets, so it never ran in CI; tests/conftest.py now sets it when
-m selects release.

test_adapter_routing_audio_enabled.py stays on PRs but composes 13
empty built-in slots on 4.0-micro instead of three real adapter
libraries on 4.1-3b: routing reads only the control-token table.

Signed-off-by: aviv ron <rona@il.ibm.com>
PR runs no longer download any real model. Every test that needs a
Granite base or adapter library is parametrized over two sources
(tests/shared/model_sources.py): a fake on every PR run, and the real
model marked `release`.

The fakes are captured, not invented. tests/shared/fixtures/ holds
verbatim copies of each base's config, generation config, tokenizer
config, special-tokens map and chat template, and each adapter's
adapter_config.json and io.yaml (`--capture` refreshes them).
fake_bases.py adds tiny random weights in the real dtype and a
byte-level tokenizer whose special tokens, and the words tests use as
probe labels, keep their real ids; fake_adapter_libraries.py adds
synthetic weights at each adapter's real rank and target modules. The
real compose CLI accepts them, and the fake tokenizers match the real
ones on special-token ids, added vocab and template render.

Converted: test_compose_e2e (which composed 4.1-3b despite naming
4.0-micro), test_save_load_compose, test_multi_audio_compose_e2e,
test_probe_compose, test_upstream_files, test_lora_substitute_probe
(its 4.2 case, which always skipped, now runs), test_audio_marker
_output_row, test_adapter_filtering, test_adapter_loader, and the GPU
tests test_audio_serving_smoke, test_audio_uncached_processor,
test_adapter_routing_audio_enabled, test_multi_adapter_equivalence,
test_probe_chunked_prefill and test_tp_integration. Probe tests that
skipped on a cold HF cache now always run.

Release only, with no fake counterpart: test_generation_equivalence
and test_skinning_equivalence (numerics vs upstream on real weights)
and the BPE split property in test_chat_template.

The fixtures are excluded from the whitespace hooks: a fixed-up final
newline changes what a captured chat template renders.

Signed-off-by: aviv ron <rona@il.ibm.com>
The placeholder model.safetensors.index.json had an empty weight_map.
vLLM's loader trusts an index when one exists and loads only the files
it names, so it found no weights and test_multi_adapter_equivalence's
upstream run on the fake base failed with "Cannot find any model
weights" (all 8 fake cases errored on vllm26). The index now maps every
tensor to the single shard, as a real one-shard repo index does.

Signed-off-by: aviv ron <rona@il.ibm.com>
test_multi_adapter_equivalence's `distinct` check failed on the fake
base for both flavors: every adapter was indistinguishable from base
(JSD 0.000000), so its passthrough/slot/batch checks had been passing
vacuously. The fake base was effectively embedding -> norm -> head.

At body std 0.02 the 12x embedding_multiplier swamped the decoder
layers, whose output residual_multiplier 0.22 shrinks further: zeroing
every attention or MLP output left the logits unchanged, so no adapter
could reach them at any strength. And the output was uniform (entropy
11.516 of 11.516 nats) under logits_scaling 10.

Now body and embedding std 1.0, and a final-norm gain of 4: dropping
attention or the MLP moves the output by JSD 0.98 / 0.99, entropy is
~4 nats. The real Granite multipliers are unchanged. With that, the
synthetic adapters clear `distinct` at their real-model calibration
(LoRA 0.29-0.89, SR 0.85-0.99, gate 0.01) and passthrough still equals
upstream exactly, measured on CPU through the worker's own compose.

Signed-off-by: aviv ron <rona@il.ibm.com>
…to 3

The suite compares three models per variant: upstream, the adapter-free
switch and the zero-LoRA switch. Each of its five tests booted a fresh
pair of engines to score one prompt, so a variant cost ten boots, five
of them the identical upstream model; at ~9+9+5 min it was the largest
single file in a GPU run.

A module fixture in _granite4_mini_tests.py now builds the three
checkpoints once and boots each model once, scoring every prompt its
tests need. The outer test runs a variant's three classes in one pytest
process (it ran one per class) so the fixture is shared. Checkpoints,
seeds and prompts are the ones the per-test runners built.

_vllm_logprobs_worker.py and run_vllm_logprobs_many score several
prompts in one engine, with one generate call per prompt: batching
could move logprobs by itself, and these gates sit at ~1e-5, so each
prompt is still scored exactly as it was alone. Prefix caching stays
off in every engine, so an earlier prompt cannot affect a later one.
run_vllm_logprobs is now a one-prompt wrapper; other callers unchanged.

Signed-off-by: aviv ron <rona@il.ibm.com>
…yers on request

8c411e0 raised the fake bases' random-weight scales so adapters reach
the output, which fixed test_multi_adapter_equivalence's `distinct`
check. It also grew intermediate hidden states from rms ~0.24 to ~340,
and the exact-score tests failed on the fake cases: test_probe_chunked
_prefill checks probe scores to +/- 0.3 against an fp32 reference, and
at scores near 2000 bf16 rounds by ~16 (3 failures on vllm26).

On a 2-layer base with the real Granite multipliers no single set of
scales gives both live layers and small activations, so build_fake_base
takes a WeightPreset. REALISTIC, the default, is bit-identical to the
fake base every exact-score test passed on before 8c411e0. LIVE_LAYERS
(body 1.0, embed 1.0, final norm 4) is used only by
test_multi_adapter_equivalence's fake case, where `distinct` still
clears its 0.01 gate (LoRA >= 0.29, SR >= 0.85). model_sources caches
each preset in its own directory, keeping the model name as the leaf so
adapter discovery still resolves the base flavour.

Signed-off-by: aviv ron <rona@il.ibm.com>
@aviv1ron1
aviv1ron1 force-pushed the feature/release-test-tier branch from 5b67886 to 4a47cec Compare November 5, 2026 10:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants