Models to port to MLX-VLM

[Blaizzy]

• MiniCPM-Llama3-V-2_5
• Florence 2
• Phi-3-vision
• Bunny
• Dolphi-vision-72b
• Llava Next
• Qwen2-VL
• Qwen2.5-VL
• Pixtral
• Llama-3.2
• Llava Interleave
• Idefics 3
• OmniParser
• Llava onevision
• internlm-xcomposer2d5-7b
• InternVL
• CogVLM2
• Copali
• MoonDream2
• Yi-VL
• CuMo
• Kosmos-2.5
• Molmo
• Ovis Gemma
• Aria
• NVIDIA NVLM
• GOT
• InternVL 2.5

Instructions:

1. Select the model and comment below with your selection
2. Create a Draft PR titled: "Add support for X"
3. Read Contribution guide
4. Check existing models
5. Tag @Blaizzy for code reviews and questions.

If the model you want is not listed, please suggest it and I will add it.

[Blaizzy]

Next release of Llava-Next

TODO:
update text config defaults to avoid errors with Llava-v1.6-vicuna:

class TextConfig:
    model_type: str
    hidden_size: int = 4096
    num_hidden_layers: int = 32
    intermediate_size: int = 11008
    num_attention_heads: int = 32
    rms_norm_eps: float = 1e-05
    vocab_size: int = 32064
    num_key_value_heads: int = 32
    rope_theta: float = 1000000
    rope_traditional: bool = False
    rope_scaling: Optional[Dict[str, Union[float, str]]] = None

[BoltzmannEntropy]

Thanks for the great repo. This should also be on the list: https://github.com/THUDM/CogVLM2
I am now just reading the code, and trying to free some time for the conversion routine.

[jrp2014]

https://llava-vl.github.io/blog/2024-08-05-llava-onevision/

[Blaizzy]

Hey @BoltzmannEntropy and @jrp2014,

Thanks for the suggestions!

I have added them to the backlog

[jrp2014]

MiniCPM-V v2.6

[jrp2014]

MiniCPM-V v2.6

[s-smits]

Do you have a link to Florence-2?

[ChristianWeyer]

Is the above list the ultimate and up-to-date list of supported models @Blaizzy? Thanks for your hard work!

[Blaizzy]

Hey @ChristianWeyer
Its mostly up-to-date, just missing qwen2-vl

[Blaizzy]

@s-smits here you go:

https://huggingface.co/microsoft/Florence-2-large/blob/main/modeling_florence2.py

[ChristianWeyer]

[x] Phi-3-vision

Thanks!
I guess Phi-3-vision includes 3.5?

[Blaizzy]

Yes, they have the same arch so there are no changes needed :)

[pulkitjindal88]

Hey @Blaizzy, thanks for this great framework. Is there any priority for InternVL? I can see it is present in your list. Just wanted to know if it planned in your near term. Want to make the model run on my macbook and mlx-vlm looks to be the best way for that.