Repository navigation
Add native per-request LoRA adapters to the DeepSeek-V4.1 CUDA engine - #47
Draft
GoatInAHat wants to merge 9 commits into
Draft
GoatInAHat wants to merge 9 commits into
GoatInAHat wants to merge 9 commits into
Conversation
…a rank A folder of PEFT files (and an optional tensorfold_adapter.json that must agree with them) is checked against the checkpoint's sizes. It may edit attention wo_b and the shared expert's w2 of a backbone layer, at any rank and scale per module. Every other module, MTP blocks included, and anything that is not a plain LoRA pair is refused by name instead of dropped. The registry is fixed at start and gives each adapter an identity key that follows the weights' bytes, the module set and every module's scale. Placement cuts A by columns and keeps B whole, which is what both modules need under tensor parallelism; the packed shared expert's A is rewritten once to read the rotated hidden state the expert kernels leave for its down projection. tools/dsv41/adapter_check.py runs the same checks offline and prints the memory a rank would hold.
--adapter NAME=FOLDER (repeatable) and --adapter-strength NAME=NUMBER are checked before any weight is downloaded: the CUDA backend, a family that serves adapters, --parallel 2 or more, and names that do not take --name or an --alias. Each adapter is listed in /v1/models as a model id of its own with the base model as parent. A request picks one by its model id or its "adapter" field; an unknown or contradictory choice is a 400 before a stream opens, never the base model's answer. A server started without --adapter, and every other family, behave exactly as before. The choice rides on the request (PreparedRequest, Scheduler.submit, Stream) as an opaque reference and reaches the engine as a generate() argument that is sent only for an adapter.
…nd ranks The delta scale * B @ (A @ x) is added to the fp32 partial sum of attention wo_b and of the shared expert's w2, before the cross-rank reduction the layer already makes. A concurrent round's input buffer gets a sixth row holding each row's adapter slot, read at replay by two Triton kernels, so one captured graph serves any mixture of adapters and padded rows copy their original's slot. Prompt chunks run one slot for all their rows. The kernels take one program a row and tile with fp32 sums in a fixed order and one tile shape for every table, so a row's bits depend on its own input and its adapter's factors only; a row of slot 0 is not read or written, and a layer no adapter edits launches nothing. The packed shared expert's w2 reads the rotated, scaled fp16 hidden state still in the expert scratch (factors folded at load); an expert outside the pack reads its bf16 hidden state. The drafter always runs the base model. A kept prompt and a spilled manifest carry the identity key of the adapter that made their rows, and a prompt continues only a prompt of the same key. The spill's prefix hashes start with the key (the base model's are unchanged), and a manifest naming something that is not a hex digest is dropped at start. Ranks: engine start gathers a digest of the base config and every adapter's slot, name and key, and refuses to serve if two ranks differ. An admission names the adapter's slot and key after the grammar, a follower checks that it holds the same adapter, and every step's agreement check includes the slots of the live streams, so a rank that disagrees fails the step on every rank before a collective. Without --adapter nothing here runs: every new argument defaults to the stock path, the round input buffer keeps five rows, and the ops between ranks and the spill's manifests and hashes are what they were.
docs/adapters.md: what an adapter may edit and what is refused, the folder format and manifest, tensor-parallel placement and the shared-expert form, graphs and rows, cache identity, memory and the limits. README option row and the API page's adapter selection.
tools/dsv41/adapter_mix.py rotates a list of model ids (the base model's and the adapters') over a burst of concurrent streams and checks each stream's token hash against its own solo run, and each solo run against "draft": false. Standard library only; its tests run it against a stand-in server.
What start refuses (one stream at a time, the torch path, the exact-matmul test mode), that it loads, warms and returns the registry, how a name resolves to a reference, and what generate does with an adapter with and without a concurrent scheduler.
… traceback A manifest whose files, files["adapter_model.safetensors"], base or provenance is not an object, a safetensors header that is not an object (or whose dtype is not a string, or whose shape or offsets hold Infinity), an rank_pattern or alpha_pattern of the wrong type, and a lora_alpha or alpha_pattern value that is not a finite number now raise AdapterError, so the server and tools/dsv41/adapter_check.py give the reason. A non-object "files" was skipped without checking the weights' hash; it is refused now. adapter_config.json with alora_invocation_tokens set (PEFT's activated LoRA, which edits only after its invocation tokens) is refused instead of being served as an always-on LoRA. The null PEFT writes for a plain LoRA is accepted.
…er must match across ranks docs/api.md and docs/adapters.md no longer state that a request without an adapter runs "bit for bit as without the flag" and that a row's bits never depend on its neighbours as results. Both follow from the kernels' construction (a slot-0 row is not read or written; one program per row, fixed sum order). The CPU tests check them under Triton's interpreter; the compiled kernels, graph replay, prompt chunks and a larger rank beside an adapter, and a server with the flag against one without, are the GPU tests' and the running-server checks' to show. The start-up check that every rank holds the same adapters now says "files, names, strengths and order" and is a function the CPU tests call: a swapped --adapter order was already refused through the slot digest, without saying why. docs/adapters.md states the rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adds native per-request LoRA selection to the DeepSeek-V4.1 CUDA engine.
--adapter NAME=FOLDERregisters an adapter model ID, with optional--adapter-strength; requests select it by model ID or theadapterfield. Base and adapter requests share the existing lane rounds, CUDA graphs, tensor-parallel base weights and KV pool.Supported writers are attention
wo_band shared-expertw2, with per-module rank and scaling. Unsupported writers and MTP edits are rejected at startup. Adapter factors are immutable for the lifetime of the engine. Prompt reuse and disk spill are scoped by adapter identity; tensor-parallel ranks check the registry and per-request selection.Draft: hardware correctness and performance are not established. This does not change checkpoint bytes or provide an abliterated checkpoint. Running a quantized base plus serialized factors is not proof of equivalence to independently requantizing another checkpoint.
The attention delta is added in fp32 before the existing cross-rank reduction. The packed shared-expert path reuses the rounded fp16 hidden scratch produced by the existing expert kernel and folds the adapter's A factor to that representation. Agreement with the intended activation-domain reference remains a hardware validation requirement. The stock drafter remains unchanged.
Without configured adapters there are no factor tables or adapter kernel launches. The startup rank-agreement packet is extended. The engine-source identity changes, so existing disk spills are invalidated by the existing mechanism; preserve them before a first start if needed. Dynamic adapter loading and
--parallel 1are not supported.See
docs/adapters.md. The target repository has issues disabled, so this draft also provides the design discussion for the arithmetic extension requested by CONTRIBUTING.md.Type
type: fix: a bug fix, or a fix after review findingstype: feature: new code (a feature, tool or add-on)type: docs: docs onlytype: results: run results or a close-outtype: research: a study or failure research, with no fixtype: spec: a plan or spec before the workReceipt
a9526f4c1b579bab51ca76a920c9d2f4e804b02b; PR headb3e6ef794f0c23f6e3c1b8982af400838e213515. No GPU or checkpoint was used.bench_concurrent.py --alone --serial, draft-on versus draft-off token hashes): not run.tools/dsv41/adapter_mix.pyprovides a mixed-model comparison against each model's solo and draft-off run.MultiDecoderwithoutslack0. The fixture is byte-identical to base, the memory-admission methods are unchanged, and all 35 fail identically on pristine basea9526f4c.test_cuda_memory_plan::test_the_drafter_loads_after_the_target). That test also fails on pristine base under the same low host-memory conditions: it requires about 1.78 GiB of staging while the host reports 1.15–1.27 GiB. These failures were not waived or changed.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Hardware proof pending.