Multimodal Representation Collapse: LLaVA/CLIP and EBR Experiments

This repository collects two investigations motivated by A Closer Look at Multimodal Representation Collapse:

1. a checkpoint-wise comparison of visual representations learned by LLaVA-1.5-7B and standalone CLIP on VQAv2; and
2. a three-seed reproduction of the paper's avMNIST Figure 9 experiment using Explicit Basis Reallocation (EBR) with ordinary Kaiming initialization.

The repository is an independent research artifact. It is not the official implementation of the paper, and the paper's authors did not release the EBR training code used for their reported result.

Research questions

• Does multimodal fine-tuning leave the LLaVA vision tower with less linearly accessible VQA information than a standalone CLIP baseline?
• Under a documented reconstruction of the avMNIST protocol, does EBR reduce semantic cross-entropy without materially reducing validation accuracy?
• Do rank, similarity, clustering, and persistent-homology measurements support a simple geometric explanation of the observed optimization result?

Headline results

LLaVA versus CLIP linear probes

Frozen CLS features were extracted from matched training checkpoints. A fresh linear probe was trained on one fixed VQAv2-derived split with 1,593 training and 395 validation examples.

Step	CLIP accuracy	LLaVA accuracy	CLIP - LLaVA
500	19.75%	15.95%	+3.80 pp
1,000	22.28%	16.46%	+5.82 pp
1,500	21.77%	15.95%	+5.82 pp
2,000	21.27%	16.71%	+4.56 pp
2,500	23.80%	17.47%	+6.33 pp

These runs use a single probe seed (seed=0). They show a consistent gap in this setup, but they are preliminary evidence rather than an uncertainty-aware estimate of a population effect. See the LLaVA/CLIP experiment notes for the exact comparison and limitations.

Kaiming-initialized, non-exclusive EBR

The primary Figure 9 window is epochs 181-200. Values below are means across seeds 17, 29, and 43.

Metric	Vanilla	EBR	Difference
Semantic cross-entropy	0.08876	0.03366	-0.05510
Clean validation accuracy	97.56%	97.11%	-0.44 pp

The semantic-loss reduction was positive for every seed. During modality blocks, the model optimized Lsem - Lmd; semantic learning was therefore not switched off. This is the non-exclusive run. The separate exclusive-block variant is not included here.

The 1,200-epoch continuation remained finite but became oscillatory for some seeds. Rank and topology results were mixed: EBR improved image-to-fusion and audio-to-fusion CKA at epoch 200, but did not uniformly increase rank or preserve normalized audio topology. The repository reports those negative and ambiguous findings alongside the successful optimization result.

Repository map

Path	Contents
paper/	Reference paper, attribution, and external links
llava_clip/	LLaVA/CLIP training and probe code, configs, metrics, and plots
ebr_kaiming_nonexclusive/	avMNIST protocol, reproduction code, reports, tables, and figures

Reproducing the analyses

Each experiment has its own environment and instructions:

cd llava_clip
python -m pip install -r requirements.txt
python analysis/timeseries_linear_probe.py --help

cd ebr_kaiming_nonexclusive
python -m pip install -r requirements.txt
python validate_kaiming_init.py --device cpu
python run_kaiming_pipeline.py --device cuda

The complete training runs require external datasets, pretrained base models, and substantial GPU time. Their locations are configured through command-line arguments or the environment variables documented in each experiment README.

Artifact policy

This Git repository contains source code, small configuration files, aggregate metrics, reports, and publication-quality figures. It intentionally excludes:

• COCO/VQAv2 and avMNIST source data;
• Hugging Face model snapshots and tokenizer caches;
• trained checkpoints, optimizer states, and extracted feature tensors;
• Conda environments, Python caches, logs, and temporary analysis caches; and
• local filesystem manifests containing machine-specific paths.

These exclusions keep the artifact reviewable and avoid redistributing datasets or model weights under terms that belong to their original providers.

Important limitations

• The LLaVA/CLIP probe comparison currently has one probe seed.
• The LLaVA and CLIP objectives are not identical, so the probe gap should not be read as a controlled causal estimate of multimodal collapse.
• The EBR implementation reconstructs undisclosed choices including batch size, phase order, noise scaling, gradient clipping, and initialization.
• Three EBR seeds are insufficient for high-powered inferential statistics; the topology and phase-conditioned tests are descriptive.
• A successful loss-curve reproduction does not imply that the undisclosed original 1,200-epoch trajectory was exactly reproduced.

Citation and attribution

The motivating paper is:

Abhra Chaudhuri, Anjan Dutta, Tu Bui, and Serban Georgescu. "A Closer Look at Multimodal Representation Collapse." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 2025. arXiv:2505.22483.

See paper/README.md for links and a BibTeX entry. Model, dataset, and upstream-project names remain the property of their respective authors. This repository does not apply a blanket license to third-party material; consult the upstream terms before reuse.