Beyond OPD: CSOPD → VGHOPD

Cheap methods for surpassing teacher performance in on-policy distillation

Standard on-policy distillation (OPD) is bounded by the teacher: once the student matches π*, there is no gradient left to improve. ExOPD breaks part of this ceiling by amplifying the implicit token reward (λ > 1), but still treats every student trajectory equally and never asks whether the rollout actually solved the task.

This repo started with one idea, hit a conceptual wall, and evolved into another.

The starting question

Can we surpass the teacher without extra rollouts or a task reward?

ExOPD already shows that reward extrapolation works — but it applies a uniform λ at every token. The first hypothesis in this repo was CSOPD (Confidence-Scaled OPD): weight extrapolation by teacher entropy, on the theory that low-entropy tokens are "confident" and safe to amplify, while high-entropy tokens are noisy and should be dampened.

A_t^CSOPD = (log π_θ - log π*) + (λ·w_t - 1)(log π_ref - log π*)

where  w_t = exp(-α · H_t)   (soft)
   or  w_t = 1[H_t ≤ τ]      (hard)

CSOPD strictly generalizes OPD (λ=1) and ExOPD (w_t=1). See CSOPD_PROPOSAL.md.

The doubt

The key question became:

Does teacher entropy actually correlate with "how much I should trust the teacher"?

I'm not convinced.

In reasoning models, some of the most important tokens are precisely the ones where entropy is high. When a model is considering multiple valid approaches to a proof or multiple code implementations, entropy rises. That doesn't necessarily mean the signal is noisy — it may mean the model is exploring genuinely useful alternatives.

Down-weighting high-entropy tokens risks suppressing exactly the reasoning steps where the student needs to learn something the teacher hasn't fully committed to. Entropy measures spread in the teacher's next-token distribution, not correctness of the full trajectory.

VGHOPD: outcome gating instead of entropy gating

That led to VGHOPD (Verifier-Gated Hindsight OPD): keep dense token-level teacher supervision from G-OPD / CSOPD, but gate the whole trajectory by whether it actually succeeded.

Ã_t = g(y) · A_t^CSOPD

where  g(y) = 1[verifier(x, y) = correct]   (filter gate)
   or  g(y) = ±1                              (advantage gate)
   or  g(y) = smooth sigmoid on outcome       (soft gate)

The verifier is cheap and deterministic — for math, extract the last \boxed{} and compare to gold (beyondopd/math_utils.py). No extra LLM rollouts, no critic, no PPO loop.

Signal	CSOPD (ω_t)	VGHOPD (g(y))
Granularity	Per-token	Per-trajectory
Source	Teacher entropy	Task verifier
Fixes	Uncertain teacher tokens	Wrong trajectories the teacher still likes

The two compose: VGHOPD selects which rollouts to learn from; CSOPD shapes how aggressively to extrapolate within successes. See VGHOPD_PROPOSAL.md.

Recommended default: vghopd_csopd_math — filter gate + CSOPD-soft, λ=1.25, α=1.0.

FA-VGHOPD: learning from failures

FA-VGHOPD (Failure-Aware VGHOPD) keeps VGHOPD’s outcome signal but does not discard failed rollouts. On a failure it:

Salvages verified prefix tokens (CSOPD on the good prefix),
Localizes the first bad step (prefix_scan by default),
Applies negative supervision on the bad continuation (−log(1 − π_θ)),
Resamples teacher repairs from the prefix,
Optionally trains a preference margin (repair > failed).

Headline preset: failure_aware_vghopd_math (M7). Compare against vghopd_csopd_math (M6) on the same G-OPD data contract. See FAVGHOPD_PROPOSAL.md and docs/REPLICATION.md.

What's implemented

Component	Status
CSOPD core (soft / hard / uniform → OPD / ExOPD)	`beyondopd/csopd.py`
VGHOPD gates (filter, advantage, soft) + math verifier	`beyondopd/vghopd.py`
FA-VGHOPD (prefix / neg / repair / preference)	`beyondopd/failure_aware_vghopd.py`
ExOPD replication loop (G-OPD data, rollout IS, eval)	`beyondopd/train_loop.py`
Named experiments + ablations	`beyondopd/experiments.py`
Math eval (AIME24/25, HMMT Feb/Nov)	`beyondopd/eval_suite.py`
Modal / single-GPU training	`modal_train.py`

All training goes through train_loop.run_training (scale profile, LR warmup, rollout correction, VGHOPD / FA-VGHOPD when enabled). The CLI in train_csopd.py is a thin wrapper — there is no separate legacy loop and no path that skips verifier gating when an experiment enables it.

Primary-matrix runs use G-OPD parquet with gold labels (required for VGHOPD and FA-VGHOPD). Optional YAML without --experiment can still load DeepMath-103K over Hugging Face (no gold) for CSOPD-only smoke; that path is outside the frozen comparison spec.

Experiment contract (frozen primary matrix): docs/EXPERIMENT_CONTRACT.md. Step-by-step matrix commands: docs/REPLICATION.md.

Code-domain FA-VGHOPD and automated analysis jobs (B1–B5) are not wired yet.

Repository structure

beyondopd/
├── csopd.py           # CSOPD / ExOPD / OPD advantages + CSoPDTrainer
├── vghopd.py          # Verifier, trajectory gates, VGHOPDConfig
├── failure_aware_vghopd.py  # FA-VGHOPD losses, masks, repair indexing
├── train_loop.py      # Unified training (G-OPD data + VGHOPD + FA)
├── train_csopd.py     # TrainConfig, CLI → train_loop.run_training
├── experiments.py     # Named presets (opd, exopd, csopd, vghopd, …)
├── math_utils.py      # \\boxed{} extract + grade (train + eval)
├── gopd_data.py       # Keven16/G-OPD-Training-Data parquet loader
├── eval_suite.py      # Math benchmark suite
├── rollout_correction.py
└── analysis.py        # Entropy breakdown utilities (CSOPD diagnostics)

CSOPD_PROPOSAL.md      # Original entropy-weighting proposal
VGHOPD_PROPOSAL.md     # Verifier-gated follow-up
FAVGHOPD_PROPOSAL.md   # Failure-aware extension
docs/EXPERIMENT_CONTRACT.md  # Canonical comparison spec (frozen)
docs/REPLICATION.md          # Quick start & commands
tests/                 # test_csopd.py, test_vghopd.py, test_replication.py

Quick start

uv sync --extra dev   # pytest, ruff, wandb (for tests / optional logging)

# List all experiment presets
uv run python -m beyondopd.train --list-experiments

# Smoke train (1 step)
uv run python scripts/run_replication.py --experiment vghopd_csopd_math --max-steps 1
uv run python scripts/run_replication.py --experiment csopd_math --max-steps 1
uv run python scripts/run_replication.py --experiment exopd_math --max-steps 1

# Run tests
uv run python -m pytest tests/ -v

Core experiments (primary matrix)

See docs/EXPERIMENT_CONTRACT.md for the full specification.

Experiment	Method
`opd_math`	OPD (λ=1)
`exopd_math`	ExOPD (λ=1.25)
`csopd_math`	CSOPD-soft
`vghopd_math`	VGHOPD filter + ExOPD
`vghopd_csopd_math`	VGHOPD + CSOPD (recommended)
`failure_aware_vghopd_math`	FA-VGHOPD
`rlvr_lite_math`	RLVR-lite ablation

# Print full-matrix commands (default profile: full)
uv run python scripts/compare_methods.py

# Run all 7 methods at once (one GPU each; default GPUs 0–6)
./scripts/run_matrix_parallel.sh --run

# Smoke all headline methods (1 step each; sequential on one GPU)
./scripts/smoke_all_methods.sh --run

# Smoke all 7 methods in parallel (GPUs 0–6)
./scripts/smoke_all_methods_parallel.sh --run

# 8-GPU DDP: sequential matrix (one experiment at a time, all GPUs each)
./scripts/run_distributed_matrix.sh --run --total-steps 300
./scripts/run_distributed_train.sh opd_math --scale-profile full --total-steps 300

# 8-GPU DDP smoke (1 step per method; rank-0 data broadcast)
./scripts/smoke_distributed_all.sh --run

# CI smoke (CPU, mocked training)
uv run pytest tests/test_smoke_matrix.py -m smoke

# Print smoke commands only
uv run python scripts/compare_methods.py --smoke

# Run one full-profile method
uv run python scripts/run_replication.py --experiment vghopd_csopd_math --scale-profile full

# Headline FA vs VGHOPD+CSOPD
uv run python scripts/run_replication.py --experiment failure_aware_vghopd_math --scale-profile full
uv run python scripts/run_replication.py --experiment vghopd_csopd_math --scale-profile full

Run on Modal

uv sync --extra modal
modal setup
modal secret create huggingface HF_TOKEN=<hf_token> WANDB_API_KEY=<wandb_key>

modal run modal_train.py --experiment vghopd_csopd_math --max-steps 1
modal run modal_train.py --config configs/replication/vghopd_csopd_math_full.yaml

Scale profiles: pilot (short 300-step runs) vs full (primary comparison). See modal_train.py for --lam, --teacher, --run-name, etc.

Training data

Replication experiments use G-OPD parquet (Keven16/G-OPD-Training-Data, DeepMath level ≥ 6) with gold answers for verifier methods (VGHOPD, FA-VGHOPD).

uv run python scripts/download_replication_assets.py

Evaluation

Math benchmarks: aime24, aime25, hmmt25_feb, hmmt25_nov. Grading uses the same \boxed{} path as training.

uv run python -m beyondopd.train --experiment exopd_math --eval-only --eval-max-problems 5

Set eval_num_samples: 32 in YAML for pass@32 (paper default).

Pilot ladder (pilot profile, 300 steps): ./scripts/run_pilot_ladder.sh --dry-run

Eval JSON + table: training checkpoints use lm-eval + vLLM by default (uv sync --extra eval); standalone: scripts/run_lmeval.py or scripts/eval_checkpoint.py; aggregate → scripts/aggregate_eval_results.py → results/eval_matrix.csv. Slow HF loop: --eval-backend hf.

Pilot ladder: ./scripts/run_pilot_ladder.sh · Analysis B1–B5: scripts/run_analysis.py · Results template: results/EXPERIMENT_MATRIX.md

Weights & Biases

export WANDB_API_KEY=<key>
uv sync --extra dev
uv run python -m beyondopd.train --experiment vghopd_csopd_math --use-wandb --max-steps 1

Logged metrics (see docs/REPLICATION.md § Metrics, implemented in beyondopd/observability.py):

All methods (with gold): train/verifier_reward, train/verifier_reward_ema, train/verifier_frac_success
CSOPD modes: train/entropy_mean, train/weights_mean
VGHOPD: train/verifier_mean_gate
FA-VGHOPD only: train/failure_aware_* component losses

At rollout_batch_size=1, raw verifier_reward is binary — use verifier_reward_ema for trends (not a crash).

Minimal API usage

from beyondopd import CSoPDConfig, CSoPDTrainer

# CSOPD (entropy-weighted ExOPD)
trainer = CSoPDTrainer(CSoPDConfig(lam=1.25, mode="soft", alpha=1.0))

loss, stats = trainer.step(
    log_prob_student=student_lp,
    log_prob_teacher=teacher_lp,
    log_prob_ref=ref_lp,
    teacher_logits=teacher_logits,
    mask=response_mask,
    trajectory_gate=gate_tensor,  # (B,) from VGHOPD — optional
)

from beyondopd.vghopd import compute_trajectory_gate, VGHOPDConfig

cfg = VGHOPDConfig(enabled=True, gate_mode="filter")
reward, gate = compute_trajectory_gate(response_text, gold_answer, domain="math", config=cfg, step=step)

Special cases via CSoPDConfig:

CSoPDConfig(lam=1.0, mode="uniform")   # standard OPD
CSoPDConfig(lam=1.25, mode="uniform")  # ExOPD
CSoPDConfig(lam=1.25, mode="soft")     # CSOPD

Qwen3-4B uses enable_thinking=False on the chat template (ExOPD Non-Thinking setup). Always format prompts through format_chat_prompt.

Comparison with prior work

Method	Surpass teacher	Extra rollouts	Task reward	Key signal
OPD	✗	✗	✗	Teacher KL
ExOPD	✓ (limited)	✗	✗	Uniform λ extrapolation
CSOPD	✓ (hypothesis)	✗	✗	Teacher entropy
VGHOPD	✓	✗	✓ (gate only)	Outcome + teacher dense adv.
FA-VGHOPD	✓ (hypothesis)	✗	✓ (gate + failure signal)	Prefix / neg / repair / preference
MOPD-peer	✓	✓ (N≈8)	✓	Peer conditioning
RLVR / GRPO	✓	varies	✓	Sparse outcome reward

CSOPD and VGHOPD are complementary, not competing — and VGHOPD does not require trusting entropy as a proxy for signal quality.

Citation

@misc{beyondopd2026,
  title     = {Beyond OPD: CSOPD and Verifier-Gated Hindsight Distillation},
  year      = {2026},
  note      = {Technical proposals. Based on G-OPD / ExOPD (Yang et al. 2026).}
}

Prior work:

@article{yang2026learning,
  title   = {Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation},
  author  = {Yang, Wenkai and Liu, Weijie and Xie, Ruobing and Yang, Kai and Yang, Saiyong and Lin, Yankai},
  journal = {arXiv preprint arXiv:2602.12125},
  year    = {2026}
}

@article{yu2026mopd,
  title   = {Multi-Rollout On-Policy Distillation via Peer Successes and Failures},
  author  = {Yu, Weichen and Li, Xiaomin and Zhao, Yizhou and others},
  journal = {arXiv preprint arXiv:2605.12652},
  year    = {2026}
}

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
.github/workflows		.github/workflows
beyondopd		beyondopd
configs		configs
docs		docs
lm_eval_tasks/beyondopd		lm_eval_tasks/beyondopd
paper		paper
results		results
scripts		scripts
tests		tests
.DS_Store		.DS_Store
.gitignore		.gitignore
.python-version		.python-version
CSOPD_PROPOSAL.md		CSOPD_PROPOSAL.md
FAVGHOPD_PROPOSAL.md		FAVGHOPD_PROPOSAL.md
LICENSE		LICENSE
README.md		README.md
TASK.md		TASK.md
VGHOPD_PROPOSAL.md		VGHOPD_PROPOSAL.md
modal_train.py		modal_train.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock
verify_math.py		verify_math.py

Navigation Menu

Repository files navigation

Beyond OPD: CSOPD → VGHOPD

The starting question

The doubt

VGHOPD: outcome gating instead of entropy gating

FA-VGHOPD: learning from failures

What's implemented

Repository structure

Quick start

Core experiments (primary matrix)

Run on Modal

Training data

Evaluation

Weights & Biases

Minimal API usage

Comparison with prior work

Citation

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Beyond OPD: CSOPD → VGHOPD

The starting question

The doubt

VGHOPD: outcome gating instead of entropy gating

FA-VGHOPD: learning from failures

What's implemented

Repository structure

Quick start

Core experiments (primary matrix)

Run on Modal

Training data

Evaluation

Weights & Biases

Minimal API usage

Comparison with prior work

Citation

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages