feat: resolve oci:// model references via llmman serve

[ericcurtin]

Summary

Adds an oci:// scheme so a model published as a CNCF ModelPack OCI artifact can be used anywhere a HuggingFace repo id can:

aphrodite run oci://ghcr.io/org/model:tag

Model distribution is increasingly moving to OCI registries -- the same registries, credentials, mirroring and air-gap tooling a deployment already uses for container images.

How it works

ModelConfig.maybe_pull_model_tokenizer_for_runai is already the one rewrite hook that runs before anything else touches model=/tokenizer=, so the oci:// branch goes there. The two schemes are disjoint, so it dispatches and returns early -- the object-storage path is reached only when no oci:// reference is present, and is otherwise bit-for-bit unchanged.

Acquisition is delegated to a running llmman serve rather than hand-rolled: llmman already implements the ModelPack media types, registry auth, resumable blob download and a content-addressed store.

New aphrodite/transformers_utils/llmman.py is the client, stdlib-only (urllib), no new dependency:

• GET /api/version probes reachability and identity; a server answering without a version field is reported as "not an llmman daemon", worth distinguishing from nothing listening.
• POST /api/pull streams NDJSON so a multi-gigabyte fetch is not silent. An error arrives in-band at HTTP 200, and a stream that ends without success is also a failure -- both are errors, not a completed pull.
• llmman resolve --no-pull reports where the bytes landed. The daemon deliberately exposes no local path, so the CLI is the documented interface; --no-pull guarantees it only reports on what /api/pull already fetched, keeping the daemon the only thing that touches the network.
• LLMMAN_HOST is honoured with llmman's own parsing, including rewriting a wildcard bind (0.0.0.0, [::]) to loopback.

aphrodite/transformers_utils/oci_utils.py handles the scheme, deliberately mirroring the shape of the neighbouring runai_utils.py (including the str() cast so pathlib.Path inputs work).

A pull needs both the daemon reachable and the binary on PATH (or APHRODITE_LLMMAN_BIN); each missing piece has its own actionable error. Neither is required unless an oci:// reference is actually used.

Design notes

• Explicit scheme, no sniffing. A bare registry/name:tag is indistinguishable from a HuggingFace repo id (org/model); guessing would silently hijack existing --model org/model deployments.
• model_weights keeps the reference the user typed, matching what the runai path does with its URL.
• One pull for a shared reference. A --tokenizer naming the same image reuses the resolved directory -- a ModelPack image is pulled whole, so the tokenizer is already there.

Testing

Two new files under tests/transformers_utils/. test_llmman.py (8 cases) runs against a real HTTP server on a loopback port, not mocks, so the NDJSON streaming contract is genuinely exercised: /api/version accepted, a non-llmman server rejected, nothing-listening reported actionably; pull success with forwarded byte progress and the exact request body asserted; in-band error at HTTP 200; a stream ending without success; non-OK status; a non-JSON diagnostic tolerated.

test_oci_utils.py: scheme detection incl. case-insensitivity and pathlib.Path; that a HF repo id, a local path, and s3:// / gs:// / az:// are not claimed (the regression that matters most); strip_oci_scheme round-trips; the resolve contract plus eight malformed-output cases; every LLMMAN_HOST form incl. wildcard-to-loopback; binary default/override and the missing-binary error; empty reference rejected without touching the daemon; and the scheme stripped before hand-off with progress wired.

Verified honestly:

• 23 logic assertions executed standalone against a live loopback daemon (scheme handling incl. every non-claimed shape, delegation with the bare reference, endpoint parsing incl. wildcard rewriting, /api/version, pull success + progress + request body, in-band error, stream without success, resolve contract) -- all pass
• ruff format and ruff check clean on all touched files
• The pytest files were not executed in-tree: importing aphrodite.config.model needs torch and the full stack, unavailable here. Flagging rather than implying coverage I do not have.
• No end-to-end run against a live llmman serve backed by a real registry.

Disclosure: written with AI assistance.