Feature/release test tier

.github/workflows/ci.yaml

@@ -73,7 +73,7 @@ jobs:
           # are heavy enough to exhaust the GitHub Actions CPU-minutes quota and
           # get killed mid-run. All of that coverage runs on the GPU cluster.
           uv run pytest tests/unit/ \
-            -m "not requires_model and not gpu and not slow and not deep" \
+            -m "not requires_model and not gpu and not slow and not deep and not release" \
             -v -s --tb=short -x \
             --cov=granite_switch --cov-report=xml
 

.github/workflows/gpu-tests.yaml

@@ -60,11 +60,11 @@ on:
       # family is added HERE and nowhere else -- this entry plus its arm in the
       # Resolve test scope step below.
       suite:
-        description: "Which tests to run: full (all five suites), short (vllm + integration), dev (one fast GPU file), audio (the audio/ASR marker), multi (all five on 2 GPUs)."
+        description: "Which tests to run: full (all five suites), short (vllm + integration), dev (one fast GPU file), audio (the audio/ASR marker), multi (all five on 2 GPUs), release (the pre-release tier: real external models)."
         required: false
         default: full
         type: choice
-        options: [full, short, dev, audio, multi]
+        options: [full, short, dev, audio, multi, release]
       # RESULT CACHE OPT-OUT. Default off, i.e. cache on: the expensive thing here is
       # a GPU run, and re-running one for a commit+scope that was already tested buys
       # nothing in the common case (a reviewer re-reading a PR, a second maintainer
@@ -225,19 +225,33 @@ jobs:
             # so a path list would be long and would drift silently every time a
             # marked test is added. A marker cannot drift.
             #
-            # "and not deep" is not optional. pytest's -m is single-valued and the
-            # command line REPLACES pyproject's addopts `-m "not deep"` rather than
-            # adding to it, so a bare `-m audio` would quietly start running the
-            # expensive deep audio tests that every other scope excludes.
+            # "and not deep and not release" is not optional. pytest's -m is
+            # single-valued and the command line REPLACES pyproject's addopts
+            # `-m "not deep and not release"` rather than adding to it, so a bare
+            # `-m audio` would quietly start running the expensive deep and
+            # release-tier audio tests that every other scope excludes.
             #
             # The quotes survive sed -> YAML -> helm -> the pod's shell; the
             # neighbouring `python -c "..."` lines in the job template rely on the
             # same thing. The one forbidden character is #, the sed delimiter.
             audio)
-              PATHS='-m "audio and not deep" tests/'
+              PATHS='-m "audio and not deep and not release" tests/'
               SUFFIX="audio"
               LEG="$LABEL-audio"
               ;;
+            # The pre-release tier, by MARKER for the same reason as `audio`: tests
+            # against real external artifacts (the default ASR, Hub repo layouts,
+            # quantization libraries) plus the non-representative cases of the
+            # breadth matrices. Every other scope excludes them. Run before tagging
+            # a release, and on PRs that bump uv.lock or a model pin. See
+            # docs/CICD.md, "Release test tier".
+            release)
+              # "and not deep": -m replaces addopts, so the deep code-theory
+              # suite would otherwise ride along on a release run.
+              PATHS='-m "release and not deep" tests/'
+              SUFFIX="release"
+              LEG="$LABEL-release"
+              ;;
             # Everything `full` runs, but on TWO GPUs. Not a different test
             # selection -- a different SHAPE of pod.
             #

.pre-commit-config.yaml

@@ -31,10 +31,13 @@ repos:
       - id: check-case-conflict
       - id: mixed-line-ending
         args: [--fix=lf]
+      # tests/shared/fixtures/ holds byte-for-byte copies of published Hub files
+      # (chat templates, configs): a fixed-up final newline changes what a
+      # template renders, so the captures must stay verbatim.
       - id: end-of-file-fixer
-        exclude: \.ipynb$
+        exclude: (\.ipynb$|^tests/shared/fixtures/)
       - id: trailing-whitespace
-        exclude: \.ipynb$
+        exclude: (\.ipynb$|^tests/shared/fixtures/)
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.9.0

CHANGELOG.md

@@ -9,6 +9,34 @@ This project uses [semantic versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Breaking
+
+- Audio checkpoints must name their ASR model. Serving no longer falls back to a
+  built-in default when `asr_model_id` is null; it fails at startup with the fix.
+  Checkpoints composed before this release need `asr_model_id` and
+  `asr_model_revision` added to `config.json`, or a re-compose (see
+  docs/AUDIO.md, *Migrating a checkpoint composed before 0.2.0*).
+- Checkpoints composed with this release need this release to serve: older
+  versions ignore the new ASR fields and load the ASR unpinned from the Hub.
+- `DEFAULT_ASR_MODEL_ID` moved from `granite_switch.vllm.audio` to
+  `granite_switch.config`. `ASRTranscriber` and `get_transcriber` now require
+  `model_id`.
+
+### Added
+
+- `--bundle-asr` composes the ASR model's weights into the checkpoint (`asr/`)
+  for airgapped serving with no Hub access; `asr_bundled_path` in the config.
+- `--asr-revision` and the `asr_model_revision` config field.
+
+### Fixed
+
+- The default ASR model is pinned to its English-only Hub revision (`947f59a`).
+  It loaded unpinned, so the Hub repo's 2026-10-21 multilingual release silently
+  changed every transcript (now capitalized and punctuated) and broke the GPU
+  tests.
+- GPU ASR tests no longer assert transcript text, which is model quality, not
+  something these tests own.
+
 ## [0.1.0] — MVP
 
 Initial MVP release. API is not yet stable.

docs/AUDIO.md

@@ -50,13 +50,29 @@ This adds the `<|audio|>` marker token to the tokenizer and writes the audio
 settings into `config.json` so the checkpoint is self-describing:
 
 ```json
-{ "asr_enabled": true, "asr_model_id": null, "asr_device": "cuda" }
+{ "asr_enabled": true,
+  "asr_model_id": "ibm-granite/granite-speech-5.0-470m-turboctc",
+  "asr_model_revision": "947f59af40db9791170a0628cf0f3f4812d720f1",
+  "asr_bundled_path": null,
+  "asr_device": "cuda" }
 ```
 
-- `asr_model_id` — HF id of the speech-to-text model (default:
-  `ibm-granite/granite-speech-5.0-470m-turboctc`, a 470M English conformer CTC
-  encoder). Override with `--asr-model <hf-id>`, e.g. `openai/whisper-small` for
-  multilingual.
+- `asr_model_id` — HF id of the speech-to-text model. Compose always writes it:
+  `ibm-granite/granite-speech-5.0-470m-turboctc` (a 470M English conformer CTC
+  encoder) unless `--asr-model <hf-id>` is given, e.g. `openai/whisper-small` for
+  multilingual. Serving has **no built-in default**: an audio checkpoint without
+  `asr_model_id` is refused at startup, so the config on the Hub always shows
+  which ASR a checkpoint runs.
+- `asr_model_revision` — Hub revision (commit, tag or branch) the ASR model loads
+  at. The default model is pinned to `947f59a`, its English-only release; the
+  repo's `main` moved on 2026-10-21 to a multilingual release that capitalizes and
+  punctuates, which silently changed every unpinned transcript. Set it with
+  `--asr-revision`. A custom `--asr-model` without one composes with a warning and
+  loads the repo's current `main` at serve time. A `revision` in
+  `asr_pipeline_kwargs` overrides it.
+- `asr_bundled_path` — set by `--bundle-asr` (see *Airgapped deployment* below).
+  When set, the ASR loads from that directory inside the checkpoint and
+  `asr_model_id`/`asr_model_revision` only record where the copy came from.
 - `asr_device` — `cuda` (default): the default encoder is small and its speed
   comes from running on GPU. Set `--asr-device cpu` to leave vLLM's whole GPU
   memory budget to the KV cache — transcription is then several times slower
@@ -85,6 +101,62 @@ options configure the cascade; they do not enable it. Passing one without
 deliberate: one explicit flag decides, rather than the decision being inferred
 from which options happen to be set.
 
+### Airgapped deployment
+
+A checkpoint normally downloads its ASR model from the Hub on first use. For an
+environment with no internet access, compose with `--bundle-asr` (it implies
+`--enable-audio`):
+
+```bash
+python -m granite_switch.composer.compose_granite_switch \
+  --base-model ibm-granite/granite-4.1-3b \
+  --built-in-adapters core \
+  --bundle-asr \
+  --output ./granite-switch-audio-bundled
+```
+
+This copies the ASR model, at its pinned revision, into `<output>/asr/` (config,
+safetensors weights, tokenizer, feature extractor, README and license; about
+1 GB for the default model) and sets `"asr_bundled_path": "asr"`. Compose then
+loads the bundle with `local_files_only=True`, so a bundle that would need the
+network fails at compose time, not at serve time. Only safetensors models can be
+bundled.
+
+`--asr-model` may also be a local directory, so the bundle can be built inside the
+airgapped network from a copy of the ASR repo brought in by hand. A local
+directory needs `--bundle-asr` (a plain `--enable-audio` checkpoint would point at
+a path that exists on the build machine only), and the config records it by its
+directory name, not its full path.
+
+Serve it with the Hub switched off:
+
+```bash
+HF_HUB_OFFLINE=1 python -m vllm.entrypoints.openai.api_server \
+  --model ./granite-switch-audio-bundled
+```
+
+A bundled checkpoint can also be uploaded to the Hub and served by repo id: only
+its `asr/` folder is fetched for the ASR, through the normal HF cache, so a
+pre-populated cache serves offline too. vLLM does not mistake `asr/*.safetensors`
+for LLM weights: it only reads weight files at the checkpoint's top level.
+
+### Migrating a checkpoint composed before 0.2.0
+
+Older audio checkpoints have `"asr_model_id": null` and relied on a default
+inside the library, which no longer exists. Serving one fails at startup with an
+error naming the fix. Either re-compose, or add these two fields to its
+`config.json`:
+
+```json
+"asr_model_id": "ibm-granite/granite-speech-5.0-470m-turboctc",
+"asr_model_revision": "947f59af40db9791170a0628cf0f3f4812d720f1"
+```
+
+Checkpoints composed with 0.2.0 need granite-switch 0.2.0 or newer to serve
+correctly. An older library ignores `asr_model_revision` and `asr_bundled_path`
+without an error: it loads the ASR unpinned from the Hub, and a bundled
+checkpoint still goes online.
+
 ### Tuning the ASR model
 
 Two optional config fields let a checkpoint carry ASR tuning so no code change is
@@ -319,8 +391,8 @@ fallback.
   `language` override is inert. For other languages use `--asr-model` with a
   multilingual generative model and set the language via `asr_generate_kwargs` (or
   per request via `mm_processor_kwargs`; see *Tuning the ASR model* above).
-- **Transcripts from the CTC default are lowercase and unpunctuated**
-  (`what is the capital of israel`). They are spliced into the prompt as ordinary
+- **Transcripts from the CTC default (at its pinned revision) are lowercase and
+  unpunctuated** (`what is the capital of israel`). They are spliced into the prompt as ordinary
   text, so the LLM reads them that way. A generative backend such as Whisper
   restores case and punctuation.
 - **HF `pipeline` backends only.** Any `automatic-speech-recognition` pipeline

docs/CICD.md

@@ -80,7 +80,7 @@ Two workflows run automatically when a PR is opened against `main` or a commit i
 
 **Jobs:**
 1. `pre-commit` — runs `uvx pre-commit run --all-files`, executing the **exact same hooks** as the local pre-commit stage, pinned to the same versions in `.pre-commit-config.yaml`. This is the single source of truth: CI and local pre-commit can never drift, and any hook added or bumped in the config applies in CI automatically with no workflow edits. It covers ruff-format + ruff, `check-toml` / `check-yaml`, `validate-links` (broken local links / stale labels / broken first-party imports), `check-headers` (SPDX), the hygiene hooks (`end-of-file-fixer`, `trailing-whitespace`, `mixed-line-ending`, `check-merge-conflict`, `check-added-large-files`, `check-case-conflict`), `nbstripout`, and `uv-lock`. All are **blocking**: the tree is clean (see [Ruff rollout](#ruff-rollout-format-first-then-enforce)), so any new violation fails the build. Auto-fixing hooks (ruff, `end-of-file-fixer`, `check-headers`, `uv-lock`) modify files when something is wrong; pre-commit then reports failure and prints the diff (`--show-diff-on-failure`), telling the contributor to run pre-commit locally and re-commit. The DCO hooks (`add-signoff`, `check-dco`) run at the `prepare-commit-msg` / `commit-msg` stages, which `pre-commit run --all-files` does **not** execute — they are covered by `dco.yaml` instead. Hook environments are cached (`~/.cache/pre-commit`, keyed on the config file) so they don't reinstall every run.
-2. `test-cpu` (needs `pre-commit`) — runs `tests/unit/` on Python 3.11 and 3.12 in parallel, filtered with `-m "not requires_model and not gpu and not slow and not deep"`. This is the **CI-safe set**: within `tests/unit/`, tests that download/compose real models (`requires_model`), need CUDA (`gpu`), hit the network or run long (`slow`), or are the expensive code-theory suite (`deep`) are still **excluded** by the marker filter. `tests/composer/` and `tests/hf/` are **not run in CI at all**: even filtered, they are heavy enough to exhaust the GitHub Actions CPU-minutes quota and get killed mid-run. All coverage beyond the unit suite — composer, HF, vLLM, and integration — runs on the GPU cluster instead (see [Manual: GPU Tests](#manual-gpu-tests))
+2. `test-cpu` (needs `pre-commit`) — runs `tests/unit/` on Python 3.11 and 3.12 in parallel, filtered with `-m "not requires_model and not gpu and not slow and not deep and not release"`. This is the **CI-safe set**: within `tests/unit/`, tests that download/compose real models (`requires_model`), need CUDA (`gpu`), hit the network or run long (`slow`), or are the expensive code-theory suite (`deep`) are still **excluded** by the marker filter. `tests/composer/` and `tests/hf/` are **not run in CI at all**: even filtered, they are heavy enough to exhaust the GitHub Actions CPU-minutes quota and get killed mid-run. All coverage beyond the unit suite — composer, HF, vLLM, and integration — runs on the GPU cluster instead (see [Manual: GPU Tests](#manual-gpu-tests))
 
 Coverage is uploaded to [Codecov](https://codecov.io) after each test run (see [Coverage](#coverage) below).
 
@@ -183,14 +183,87 @@ While in pre-1.0, increment `MINOR` for new features and `PATCH` for bug fixes.
    - Bump `version` in `pyproject.toml`
    - Add a section to `CHANGELOG.md`
 
-2. **Merge to `main`**
+2. **Run the release tier** — comment `/gpu-test-release` on the release PR and
+   get it green. It is the only place real external models are tested (see
+   [Release test tier](#release-test-tier)).
 
-3. **Create a GitHub Release** (`Releases` → `Draft a new release`):
+3. **Merge to `main`**
+
+4. **Create a GitHub Release** (`Releases` → `Draft a new release`):
    - Tag: `vX.Y.Z` (created from `main`)
    - Title: `vX.Y.Z`
    - Body: paste the `CHANGELOG.md` section for this release
 
-4. **Click Publish release** — `publish.yaml` triggers automatically
+5. **Click Publish release** — `publish.yaml` triggers automatically
+
+### Release test tier
+
+**PR runs download no real model.** Every test that needs a Granite base or an
+adapter library runs twice: once on a fake on every PR run, and once on the real
+model in the release tier, marked `release`. Release tests are excluded from every
+other scope (`addopts` and each `-m` filter carry `not release`) and run with the
+`release` GPU suite (`/gpu-test-release`).
+
+**The fakes are captured, not invented.** `tests/shared/fixtures/` holds verbatim
+copies of the real repos' small files: each base's config, generation config,
+tokenizer config, special-tokens map and chat template, and each adapter's
+`adapter_config.json` and `io.yaml`. At test time,
+`tests/shared/fake_bases.py` adds tiny random weights (in the real dtype) and a
+tokenizer whose special tokens, and the words tests use as probe labels, keep
+their real ids; `tests/shared/fake_adapter_libraries.py` adds synthetic weights at
+each adapter's real rank and target modules. So the composer sees the real
+template, token ids, configs and repo layout, and the real compose CLI accepts
+the fakes. What the fakes leave out is weights and the BPE vocabulary: ordinary
+words tokenize to one token per byte.
+
+**Writing a test that needs a model:** take it from `tests/shared/model_sources.py`
+rather than naming a Hub id, and parametrize over `model_sources.sources()`:
+
+```python
+@pytest.mark.parametrize("source", model_sources.sources())
+def test_something(source):
+    base = model_sources.base_model(source, "granite-4.1-3b")
+    library = model_sources.adapter_library(
+        source, "ibm-granite/granite-lib-rag-r1.0", "granite-4.1-3b"
+    )
+```
+
+**Refreshing the captures:** `python -m tests.shared.fake_bases --capture` and
+`python -m tests.shared.fake_adapter_libraries --capture`. A failing real case in
+the release tier, whose fake case passes, usually means a capture has fallen
+behind its Hub repo.
+
+**Release only, with no fake counterpart**, because the fake cannot stand in:
+
+- `tests/integration/test_asr_ctc_default_gpu.py`: the real default ASR at its
+  pinned revision. Every other audio test uses a tiny local ASR
+  (`tests/shared/tiny_asr.py`).
+- `tests/vllm/test_generation_equivalence.py` and
+  `tests/composer/test_skinning_equivalence.py`: our model against upstream on the
+  real weights, with thresholds calibrated on them. On random weights the same
+  equivalence runs on PRs in `tests/vllm/test_granite4_mini.py`.
+- `tests/composer/test_chat_template.py::TestInvocationFirstCharDropProperty`:
+  how the real BPE vocabulary splits invocation strings.
+- `tests/composer/test_selective_download.py::TestRealHubMetadata`: the published
+  adapter repos' layout, against the live Hub.
+- `tests/hf/test_quantization.py`: the quantization libraries on a real
+  checkpoint. Its `GRANITE_SWITCH_E2E_MODELS=1` gate is opened by
+  `tests/conftest.py` whenever `-m` selects `release`.
+
+**The rest of a breadth matrix** is release-only too: PR runs keep one
+representative case.
+
+- `tests/vllm/test_granite4_mini.py`: vLLM classes on `4.0-micro`; `4.0-1b` and
+  `4.0-350m` in release. The CPU weight-transfer classes run every variant.
+- `tests/vllm/test_granite4_fullsize.py`: the GPU class is release-only.
+- `tests/vllm/test_multi_adapter_equivalence.py`: a fake `granite-4.1-3b` on PRs;
+  the real `4.1-3b` and `4.2-3b` (the ChatML template) in release.
+- `tests/integration/test_audio_serving_smoke.py`: a fake `granite-4.0-micro` on
+  PRs; the real `4.0-micro` and `4.1-3b` in release.
+
+Run the release tier before every release, and on any PR that bumps `uv.lock`,
+touches a model pin (e.g. `DEFAULT_ASR_MODEL_REVISION`) or a capture, or changes
+code that depends on real weights or the BPE vocabulary.
 
 ### What `publish.yaml` does
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "granite-switch"
-version = "0.1.1"
+version = "0.2.0"
 description = "Granite Switch: Composable model building"
 readme = "README.md"
 license = "Apache-2.0"
@@ -47,14 +47,15 @@ register_granite_switch = "granite_switch.vllm:register"
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
-addopts = ["--ignore=tests/_legacy", "-m", "not deep"]
+addopts = ["--ignore=tests/_legacy", "-m", "not deep and not release"]
 markers = [
     "gpu: requires CUDA GPU",
     "vllm: requires vLLM installed",
     "slow: takes > 30s",
     "deep: expensive code-theory tests (m=8 / 256-dim); run with: pytest -m deep",
     "requires_model: needs a real model checkpoint",
     "audio: exercises the audio/ASR path; run with: pytest -m audio",
+    "release: pre-release tier, real external models (e.g. the default ASR) checked against our code; run with: pytest -m release",
 ]
 
 [dependency-groups]