Add mergekit-check-compat pre-merge compatibility checker

[nloughl]

Summary

Adds mergekit-check-compat, a new CLI that validates model compatibility before a merge runs — loading only configs and tokenizers, no weights, no GPU required.

mergekit-check-compat config.yml

• Tensor shape params (hidden_size, num_hidden_layers, num_attention_heads, num_key_value_heads, intermediate_size) → ERROR if any differ; merge will fail at the weight level
• model_type mismatch → WARNING; shapes may be compatible but architecture semantics may differ (activation functions, attention variants, norm placement)
• RoPE theta / rope_scaling divergence → WARNING; positional encoding mismatch degrades coherence, especially at longer sequence lengths
• Vocabulary size mismatch → WARNING with advice to use tokenizer_source: union or resize embeddings; notes FIM token drop if applicable
• FIM token presence in one model but not others → WARNING
• Chat template / instruct-model mixing → INFO

Verdict line exits with code 1 on any ERROR, 0 otherwise.

Multimodal config support

Multimodal wrapper configs (e.g. Mistral3ForConditionalGeneration, LLaVA-style) store architecture params under a text_config sub-object rather than at the top level. The checker transparently unwraps text_config when top-level arch params are absent, so models like Mistral Medium 3.5 report real values instead of N/A. Also handles rope_theta embedded inside rope_scaling dicts (Ministral3 style).

Example output

Model Compatibility Report
============================================================
                     Llama-2-7b-hf     CodeLlama-7b-hf   
hidden_size          4096              4096              ✓
num_hidden_layers    32                32                ✓
num_attention_heads  32                32                ✓
num_key_value_heads  32                32                ✓
intermediate_size    11008             11008             ✓
model_type           llama             llama             ✓
rope_theta           10000.0           1000000.0         ⚠
rope_scaling         none              none              ✓
vocab_size           32000             32016             ⚠
FIM tokens           No                Yes               ⚠
chat_template        No                No                ✓

Issues
----------------------------------------
[WARNING] RoPE theta mismatch (Llama-2-7b-hf=10000.0,
          CodeLlama-7b-hf=1000000.0). Positional encodings are
          incompatible; merged model will likely have degraded coherence,
          especially for longer outputs.

[WARNING] Vocabulary size mismatch (Llama-2-7b-hf=32000,
          CodeLlama-7b-hf=32016). MergeKit will truncate to the base
          model's vocab. FIM tokens (<PRE>, <MID>, <SUF>, <EOT>) in
          CodeLlama-7b-hf will be dropped. Consider: resize_tok_vocab.py
          or set `tokenizer_source: union` in config.

[WARNING] CodeLlama-7b-hf contains FIM tokens (<PRE>, <MID>, <SUF>,
          <EOT>) but other models do not. FIM tokens may appear in natural
          language completions after merging.

Verdict: MERGE POSSIBLE WITH WARNINGS

Test plan

• mergekit-check-compat config.yml runs without downloading weights
• Architecture shape mismatches produce ERROR and exit code 1
• model_type-only mismatch (same shapes) produces WARNING, not ERROR
• RoPE, vocab, FIM, and chat-template checks fire correctly
• Multimodal wrapper configs (Mistral3, LLaVA) populate all rows without N/A

---

Note

Low Risk
New read-only diagnostic script and console entry point; no changes to merge execution or model loading paths used during actual merges.

Overview
Adds mergekit-check-compat, a CLI that reads a merge YAML config and validates referenced models before any weight merge—only configs and tokenizers are loaded.

It prints a side-by-side compatibility table and severity-tagged issues for tensor shape params (ERROR on mismatch), model_type, RoPE theta / rope_scaling, vocab size (with awareness of tokenizer_source: union), FIM tokens, and chat templates. A final verdict exits with code 1 only when ERROR-level issues are present.

Multimodal configs are handled by unwrapping text_config when top-level arch fields are missing, and rope_theta inside rope_scaling is read when needed. The command is registered in pyproject.toml as mergekit-check-compat.

^{Reviewed by Cursor Bugbot for commit aae614d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[nloughl]

I have read the CLA Document and I hereby sign the CLA

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit aae614d. Configure here.}

[cursor]

Missing model_type partial mismatch

Medium Severity

model_type compatibility only flags cases where two or more distinct non-null types appear. When one model reports a type and another is missing (None/N/A), no WARNING is added, unlike tensor shape checks. The table may still show a warning marker without a matching Issues entry.

 

^{Reviewed by Cursor Bugbot for commit aae614d. Configure here.}