Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

Abstract: Contemporary studies have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models. Building a robust, unified understanding of these phenomena requires disassembling a model within the scope of its training. While the intractable scale of pretraining corpora limits a bottom-up investigation in this direction, simplistic assumptions of the data generation process limit the expressivity and fail to explain complex patterns. In this work, we use probabilistic context-free grammars (PCFGs) to generate synthetic corpora that are faithful and computationally efficient proxies for web-scale text corpora. We investigate the emergence of three mechanistic phenomena: induction heads, function vectors, and the Hydra effect, under our designed data generation process, as well as in the checkpoints of real-world language models. Our findings suggest that hierarchical structures in the data generation process serve as the X-factor in explaining the emergence of these phenomena. We provide the theoretical underpinnings of the role played by hierarchy in the training dynamics of language models. In a nutshell, our work is the first of its kind to provide a unified explanation behind the emergence of seemingly unrelated mechanistic phenomena in LLMs, augmented with efficient synthetic tooling for future interpretability research.

This repository contains the code and experimental pipelines for the paper “Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale”. Specifically, for reproducing Figures 2, 3, 4, 5(a).

UKP Lab | TU Darmstadt

Prerequisites

This project uses uv as the Python package manager. Install dependencies and setup the virtual environment:

uv sync --frozen

Run N-gram pipeline

uv run experiment --config-path=configs/ngram.yml

Run PCFG pipeline

uv run experiment --config-path=configs/pcfg.yml

Run OLMo-1b evaluation

To reproduce all OLMo evaluation results run:

Step 1: Download prerequisites

WANDB_API_KEY=your_api_key uv run python olmo_evaluation/prerequisites/download_wandb_log.py
uv run python olmo_evaluation/prerequisites/download_checkpoints.py
uv run python olmo_evaluation/prerequisites/download_paloma.py

Step 2: Run evaluation

uv run olmo_evaluation/multi_gpu_entry.py 

Cite

If you found our data or code helpful, please cite our paper:

@article{rohweder2026hierarchical,
  title={Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale},
  author={Rohweder, Jonas and Dutta, Subhabrata and Gurevych, Iryna},
  journal={arXiv preprint arXiv:2603.06592},
  year={2026}
}

Disclaimer

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.