Skip to content

feat(musubi): Ideogram 4 LoRA 支持(前后端) - #425

Merged
wochenlong merged 6 commits into
devfrom
feat/musubi-ideogram4
Oct 10, 2026
Merged

wochenlong merged 6 commits into
devfrom
feat/musubi-ideogram4

Conversation

@wochenlong

Copy link
Copy Markdown
Owner

实现 #424:Musubi 引擎支持 Ideogram 4 LoRA(前后端)。

做了什么

后端

  • 新增 mikazuki/engines/musubi/families.py:把模型族(Krea 2 / Ideogram 4)的脚本名、LoRA 网络模块、参数约束、采样与缓存默认值集中登记;adapter / launcher / preflight / run / process 全部改为按族分发,不再写死 krea2。
  • manifest.TRAIN_TYPES 增加 ideogram4-lora → ideogram4,CAPABILITIES 同步;/api/engines/musubi/status 增加 train_types。
  • 族专属约束:基底固定 FP8(拒绝/剔除 fp8_base/fp8_scaled)、turbo_dit 仅 Krea 2、weighting_scheme 只能 none、blocks_to_swap ≤ 33、非对称 CFG 需显式开启 use_unconditional_dit_for_lora_sampling;vae_dtype/text_cache_dtype 作为缓存阶段参数传给缓存脚本、不写入训练 TOML。
  • model_assets:ideogram4-lora 资源清单(conditional/unconditional DiT、Qwen3-VL-8B 文本编码器、Flux2 VAE、tokenizer 目录)与两处上游补丁:
    1. rotary 修复:我们 pin 的 2026-08-13 快照中,load_ideogram4_text_encoder() 在 to_empty() 后没有重建 rotary embedding,inv_freq 未初始化会让文本缓存变成垃圾值(上游 v0.3.6 / PR #1139 修复)。这里不升级 pin,改为幂等源码补丁 + 写入前语法校验。
    2. tokenizer 本地化:把 tokenizer 指向本地目录(存在时),离线/国内环境不再依赖 Hub。
  • 配置导入:识别 ideogram4-lora(network_module + 标记字段 + 路径规则),跨页导入会提示跳转到 /lora/ideogram4.html。

前端

  • 新 schema mikazuki/schema/ideogram4-lora.ts:FP8 组件路径、ideogram4_shift 时间步采样、caption 结构校验开关、text_cache_dtype 磁盘提示、blocks_to_swap 上限 33、SageAttention 不可用说明、三个官方采样 preset。
  • 模型表 / i18n / 路由(/lora/ideogram4.html)/ 引擎标签增加 Ideogram 4;MusubiGatePage 不再写死 Krea 2,TrainingPage 对全部 musubi schema 走预检。
  • 跨页携带保护:vae / turbo_dit / unconditional_dit / fp8_* 不再在 Krea 2 ↔ Ideogram 4 之间串台。

文档

  • 新增 docs/ideogram4-musubi.md:许可(非商用、仅发 FP8 量化权重、整合包不预置)、组件清单、缓存磁盘占用(bf16 约 13.6 MB/图)、参数要点、上游补丁说明与已知限制。
  • CHANGELOG / dev-progress / README 引擎表同步。

验证

  • python -m pytest tests/test_musubi_ideogram4.py tests/test_musubi_backend.py tests/test_musubi_routes.py tests/test_engine_registry.py tests/test_model_assets.py tests/test_config_import.py tests/test_musubi_install.py → 全绿(新增 28 项)。
  • npm run typecheck、npm run lint(0 error)、npm test → 61 文件 / 363 测试全绿(新增 schema、模块表与 Gate 页用例)。
  • 全量 pytest 在本机有两个与本次改动无关的既有问题,已在干净 origin/dev 检出上复现确认:4 个 symlink 用例因 Windows 权限失败、test_diffsynth_engine.py::test_missing_shard_fails_before_submission 单条耗时约 104s(不是挂死)。

还没做(需要 GPU 真机)

  • 三阶段真机冒烟(cache latents → cache TE → train)并保存 LoRA
  • 训练中采样预览 + 缓存复用二次启动不发散(验证 rotary 补丁生效)
  • 显存实测(9.3B FP8 DiT + Qwen3-VL-8B)与 blocks_to_swap 档位建议
  • 权限与合规确认:不预置权重、界面已标注非商用许可

上游为 experimental(社区反馈过 TE 预缓存 OOM / resume 变慢),真机验收完成前不建议把这条能力写进发布说明。

- families.py: per-family spec (scripts, network module, constraints,
  sampler/cache defaults) so adapter/launcher/preflight/run stay generic
- manifest: register ideogram4-lora -> ideogram4; capabilities updated
- adapter: family-aware constraints (FP8-only base, no turbo/weighting_scheme,
  blocks_to_swap<=33, asymmetric-CFG opt-in) and cache-only field filtering
- launcher/process/run: family scripts, cache-stage extra flags, per-family
  tokenizer patching on launch
- model_assets: ideogram4-lora assets (FP8 components + tokenizer) and the
  upstream rotary-embedding patch (pre-v0.3.6 snapshots)
- config_import: detect ideogram4-lora configs and route them to their page
- new ideogram4-lora schema (FP8 components, ideogram4_shift sampling,
  caption validation, cache dtype/disk hints, sampler presets)
- module table/i18n/router/catalog for model Ideogram 4 on musubi
- MusubiGatePage takes the selected schema instead of hardcoding Krea 2;
  TrainingPage preflights every musubi schema
- carry-over guard: vae/turbo/unconditional-DiT/fp8 flags no longer cross
  Krea 2 and Ideogram 4 pages
@wochenlong

Copy link
Copy Markdown
Owner Author

验证补充:全量 pytest 已完整跑完(包含 test_diffsynth_engine.py):

5 failed, 1572 passed, 32 skipped, 77 subtests passed in 2856.42s (0:47:36)

5 个失败与前述一致(4 个 symlink 用例因 Windows 权限、1 个 README CLI 文档用例),均已在干净 origin/dev 检出上复现确认与本 PR 无关。此前日志里的线程级 access-violation dump 不影响结果,套件照常跑完。本机跑满全量约需 48 分钟(单个 diffsynth 用例就要 104 秒),复跑建议加 --ignore=tests/test_diffsynth_engine.py。

- assets: add ModelScope mirrors for every Ideogram 4 component — the download
  dialog defaults to ModelScope, and a missing mirror failed the whole batch
- tokenizer patch now follows (and reverts from) the local directory, and writes
  the installed source atomically
- vae_dtype stays in the train TOML (it also drives the sampling VAE load);
  only text_cache_dtype is cache-only, and dry-run returns the filtered values
- an unregistered family variant fails loudly instead of silently running Krea 2
- drop ideogram4_timestep_mu/std (ignored upstream) and annotate the inert
  preview CFG/steps fields
- carry-over denies are scoped to model-family pages, so kohya keeps sharing an
  external VAE and its fp8 switches
- tests: registry mapping, asset sources, patch compile guard, tokenizer
  re-point, dispatch guard, kohya carry-over
@wochenlong
wochenlong merged commit 2bdccd1 into dev Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant