Skip to content

feat(models): install and remove weight variants per node - #353

Open
Lorchie wants to merge 1 commit into
devfrom
feat/weight-variants
Open

Lorchie wants to merge 1 commit into
devfrom
feat/weight-variants

Conversation

@Lorchie

@Lorchie Lorchie commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

A model node that publishes the same weights in several variants (quantizations, precisions) can declare weight_variants next to hf_repo. Each variant is downloaded and deleted on its own from the Extensions drawer, and the params_schema select that picks one labels the variants missing from disk.

Manifest

"params_schema": [
  { "id": "quant", "type": "select", "default": "Q5_K_M",
    "options": [{ "value": "Q4_K_M" }, { "value": "Q5_K_M" }] }
],
"weight_variants": {
  "param": "quant",
  "default": "Q5_K_M",
  "options": [
    { "id": "Q4_K_M", "size_gb": 2.4, "include_prefixes": ["dit/model_Q4_K_M.gguf"], "checks": ["dit/model_Q4_K_M.gguf"] },
    { "id": "Q5_K_M", "include_prefixes": ["dit/model_Q5_K_M.gguf"], "checks": ["dit/model_Q5_K_M.gguf"] }
  ]
}

Validation is mirrored in electron/main/model-sources.ts and api/services/model_sources.py: safe relative paths, no overlap between two variants, download_check outside every variant, no combination with model_sources, and the selecting param must exist (node or extension params_schema) and offer every variant id.

Behaviour

  • Every install downloads the shared files first (all variants excluded), then the requested variant — the default one when none is asked for. Files already complete on disk are skipped, so adding a second variant only fetches its own weights.
  • New IPC: model:installedWeightVariants (returns null when the install state cannot be read, so the renderer never mistakes it for "none installed") and model:deleteWeightVariant, which unloads the model first and removes only that variant's files, .part included.
  • A node counts as installed once its download_check and at least one variant are present.
  • Generation refuses a variant that is not installed (GeneratorRegistry.assert_weight_variant_installed, called at the top of _run_generation, so workflows and the agent are both covered). Picking a missing variant in a node opens the extension drawer.

Tests

npm run test:py (115 tests, OK) and npm run test:node (37 + 142, 0 fail) pass; npx eslint . is clean. tsc -b --noEmit reports the 29 errors already present on dev, none in the files touched here.

Not covered: a real Hugging Face download of a variant has not been exercised end to end — the variant files were faked on disk for the UI, deletion and generation-guard runs.

A model node that publishes the same weights in several variants
(quantizations, precisions) can declare `weight_variants` next to
`hf_repo`. Each variant is downloaded and deleted on its own from the
Extensions drawer, and the params_schema select that picks one labels the
variants that are missing from disk.

- manifest validation mirrored in electron/main/model-sources.ts and
  api/services/model_sources.py: safe relative paths, no overlap between
  two variants, download_check outside every variant, no combination with
  model_sources, and the selecting param must exist and offer every id.
- every install downloads the shared files first (all variants excluded),
  then the requested variant, or the default one; files already complete
  on disk are skipped, so a second variant only fetches its own weights.
- new IPC: model:installedWeightVariants (null when unreadable) and
  model:deleteWeightVariant, which unloads the model before removal and
  touches only the files of that variant.
- generation refuses a variant that is not installed
  (GeneratorRegistry.assert_weight_variant_installed) and picking one in a
  node opens the extension drawer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant