Released checkpoint architecture differs from paper Table V (hidden 2560 vs 1536) — is this the same as Table VI?

[AlbertHuyb]

Thanks for the great work and for open-sourcing the checkpoints! Inspecting LDA-robocasa.pt directly via torch.load:

	Paper Table V (LDA-1B)	Released LDA-robocasa.pt
Hidden Size	1536	2560
Layers	16	16
Heads	32	32
Action Chunk	16	16
MMDiT trunk (action_model.model.blocks)	~0.45 B (est.)	1.77 B
action_model total trainable	~1 B (est.)	2.38 B
Qwen3-VL (frozen at pretrain)	~4.83 B	4.83 B
state_dict total (bf16)	—	~7.2 B / 14 GB

Released ckpt matches Table V on depth, heads, and action chunk, but uses hidden_size = 2560 instead of 1536 — ~2.4× larger trainable params than Table V's LDA-1B.

Sim eval result

Running batch_eval_args.sh with default settings (24 RoboCasa-GR1 tasks, 50 trials/task, MAX_EPISODE_STEPS=720, N_ACTION_STEPS=12 — paper says 51 trials, essentially the same protocol):

• Released ckpt avg SR: 49.92 %
• Paper Table VI LDA-1B: 55.4 %

Most per-task SRs are within ±5 pp of paper, with one big outlier: PnPBottleToCabinetClose (paper 76 → ours 54).

So the released ckpt is wider than Table V but ~5.5 pp lower than Table VI on the same task suite — the opposite of what scaling would predict.

Questions

1. Is Wayer2/LDA-robocasa.pt the exact checkpoint behind paper Table VI's LDA-1B = 55.4 %, or a later scale-up (1536 → 2560)?
2. Either way, any insight on the 5.5 pp gap? We've considered RoboCasa / robosuite version drift, asset issues (TrayToTieredshelf 0.14 / PlacematToTieredshelf 0.12 look like MuJoCo penetration), and eval hyperparams (MAX_EPISODE_STEPS / N_ACTION_STEPS) not specified in the paper.

Happy to share more diagnostic info (full per-task SR table, state_dict stats) if useful. Thanks!

[KaiLiu18]

Thanks a lot for your interest in LDA!

1. The hidden size reported in Table V of the paper refers to the hidden size of MMDiT, whose value is calculated as:

attention_head_dim * num_attention_heads

In the DiTConfig defined in MMDiT_ActionHeader.py
, we record the attention_head_dim and num_attention_heads for different DiT variants.

The setting used in our training is DiT-L, where attention_head_dim = 48 and num_attention_heads = 32. Therefore, the hidden size of DiT is:

48 * 32 = 1536

By contrast, the hidden_size under action_model in LDA_pretrain.yaml refers to the hidden size of the state encoder and action decoder, rather than the hidden size of MMDiT.

2. For the RoboCasa evaluation gap, we have also observed that some RoboCasa tasks can have relatively high variance across evaluation seeds and environment initializations. PnPBottleToCabinetClose is a good example: in some initializations the bottle may fall over at the beginning of the episode, in which case the policy typically fails directly. This can make the success rate of individual tasks fluctuate noticeably.

Our evaluation setting for the released checkpoint follows the repo evaluation script, with 50 episodes per task, MAX_EPISODE_STEPS=720, and N_ACTION_STEPS=12. Using the released checkpoint, we obtained the following numbers:

Task	Success rate
PnPBottleToCabinetClose	0.62
PnPCanToDrawerClose	0.74
PnPCupToDrawerClose	0.50
PnPMilkToMicrowaveClose	0.48
PnPPotatoToMicrowaveClose	0.18
PnPWineToCabinetClose	0.48
PosttrainPnPNovelFromCuttingboardToBasketSplitA	0.68
PosttrainPnPNovelFromCuttingboardToCardboardboxSplitA	0.56
PosttrainPnPNovelFromCuttingboardToPanSplitA	0.68
PosttrainPnPNovelFromCuttingboardToPotSplitA	0.66
PosttrainPnPNovelFromCuttingboardToTieredbasketSplitA	0.46
PosttrainPnPNovelFromPlacematToBasketSplitA	0.68
PosttrainPnPNovelFromPlacematToBowlSplitA	0.72
PosttrainPnPNovelFromPlacematToPlateSplitA	0.54
PosttrainPnPNovelFromPlacematToTieredshelfSplitA	0.36
PosttrainPnPNovelFromPlateToBowlSplitA	0.50
PosttrainPnPNovelFromPlateToCardboardboxSplitA	0.64
PosttrainPnPNovelFromPlateToPanSplitA	0.54
PosttrainPnPNovelFromPlateToPlateSplitA	0.68
PosttrainPnPNovelFromTrayToCardboardboxSplitA	0.48
PosttrainPnPNovelFromTrayToPlateSplitA	0.46
PosttrainPnPNovelFromTrayToPotSplitA	0.56
PosttrainPnPNovelFromTrayToTieredbasketSplitA	0.58
PosttrainPnPNovelFromTrayToTieredshelfSplitA	0.36
Average	0.5475

[AlbertHuyb]

@KaiLiu18

Thanks for your response.

I'm trying to reproduce the RoboCasa fine-tuning results. The README recommends:

bash scripts/run_scripts/Robocasa/run_lerobot_datasets_LDA.sh

which uses lda/config/training/lda_robocasa.yaml as the base config. However, I noticed significant discrepancies between this recommended recipe and the auto-saved config.yaml bundled with LDA-robocasa.pt (saved by save_accessed_config in train_LDA.py):

Parameter	Recommended script / yaml	LDA-robocasa.pt config.yaml
framework.name	QwenGR00T (from lda_robocasa.yaml)	QwenMMDiT
action_dim	29	138
obs_horizon	1	2
repeated_diffusion_steps	1 (script) / 4 (yaml default)	4
future_obs_index	5	16
freeze_modules	qwen_vl_interface, action_model.vision_encoder	action_model.vision_encoder (VLM unfrozen)
pretrained_checkpoint	null (from scratch)	LDA-pretrain checkpoint
max_train_steps	100k (yaml) / 400k (script)	300k
data_mix	fourier_gr1_unified_1000	fourier_gr1_unified_1000_two_history

These are quite fundamental differences (different framework class, action space, VLM training strategy, etc.). The recommended script would produce a structurally different model from the released checkpoint.

What I tried

I fine-tuned from LDA-pretrain.pt using parameters aligned with the ckpt config (QwenMMDiT, action_dim=138, obs_horizon=2, VLM unfrozen), but with repeated_diffusion_steps=1 and future_obs_index=5 (following the yaml defaults). At 35k steps (action_dit_loss converged to ~0.007), I got 0% SR on RoboCasa eval, while the released checkpoint achieves ~50%.

Questions

1. Could you share the actual training script/config used to produce LDA-robocasa.pt?
2. Is repeated_diffusion_steps=4 critical for convergence? With rds=1 at 35k steps, the effective diffusion training samples are ~34x fewer than the released model (35k×1 vs 300k×4).
3. The data_mix key fourier_gr1_unified_1000_two_history doesn't exist in mixtures.py — is it equivalent to fourier_gr1_unified_1000 (which already uses the _twohistory DataConfig)?

Any guidance on the correct reproduction recipe would be greatly appreciated. Thanks!

[KaiLiu18]

@AlbertHuyb

1. The specific training configuration for LDA-robocasa.pt can be found in the config.yaml file on Hugging Face. We have also updated the training script in the latest GitHub commit so that it is aligned with the training configuration on Hugging Face.

2. We recommend training with repeated_diffusion_steps=4. This is effectively equivalent to increasing the batch size of the DiT component, while avoiding excessive GPU memory usage.

3. We have updated fourier_gr1_unified_1000_two_history in data_mix. You can now directly run this script for training.

[AlbertHuyb]

@KaiLiu18 Thanks for the updated script and confirming repeated_diffusion_steps=4!

I've aligned my training config exactly with the auto-saved config.yaml from LDA-robocasa.pt — verified by diff, only path differences remain. Here's my setup:

• QwenMMDiT, DiT-L, action_dim=138, obs_horizon=2
• repeated_diffusion_steps=4, future_obs_index=16
• data_mix=fourier_gr1_unified_1000_two_history
• freeze_modules=action_model.vision_encoder (VLM unfrozen)
• pretrained_checkpoint=LDA-pretrain.pt
• per_device_batch_size=10, 8 GPUs
• max_train_steps=300000, num_warmup_steps=15000
• Base yaml: LDA_pretrain.yaml (all params overridden via CLI to match your config)

Training loss converges well — action_dit_loss ~0.007, dynamics_loss ~0.02 at 40k steps.

However, RoboCasa eval at 40k steps gives 0% SR across 6 tested tasks

For comparison, an action-only baseline (QwenGR00T, action_dim=29, same VLM pretrain weights, no vision encoder or multi-task) achieves ~17% avg SR at 35k steps with similar effective training volume.

Checkpoint	Steps	Avg SR (6 tasks)
Official LDA-robocasa.pt	300k	~67% (verified on our eval setup)
Action-only (QwenGR00T)	35k	~17%
Our QwenMMDiT finetune	40k	0%

The official checkpoint works perfectly in our eval pipeline (75% on PnPCanToDrawer), so the eval setup itself is validated.

Questions:

1. At roughly what step count during the 300k training did RoboCasa SR start becoming non-trivial (e.g., >5%)? Is 0% at 40k steps expected?
2. Did you observe a sudden phase transition in SR, or was it a gradual climb from early in training?
3. Any other suggestions for what might cause this gap despite matching configs?

[KaiLiu18]

@AlbertHuyb

Thank you for your feedback.

In our training, we used 32 GPUs. Under this setting, the RoboCasa SR already reached around 18% at 10k steps and around 30% at 25k steps.

So 0% SR at 40k steps is not what we observed. One possible reason is that your effective global batch size could be small. You can set gradient_accumulation_steps=4 under your current setup to increase the effective batch size. This should likely improve the training stability and final performance.

[AlbertHuyb]

@KaiLiu18 Thanks for the quick response — this is very helpful!

The batch size difference likely explains our issue. We're using 8 GPUs × per_device_batch_size=10 = 80 effective batch, while your setup is 32 GPUs × 10 = 320. That's a 4× gap.

32 GPUs × 300k steps for a 1000-trajectory RoboCasa finetune — seems quite heavy for a sim finetune task. Is this the standard recipe?

We'll restart with gradient_accumulation_steps=4.

[ziyangwang1992]

@KaiLiu18 Thanks for your advices, it's important.
and there're some different params between github and pretrain.pt:

1. cross_attention_dim=2048 but 2560 in pt
2. future_obs_index=5 but 16 in pt
3. freeze_module_list='qwen_vl_interface,action_model.vision_encoder' but action_model.vision_encoder in pt

Regarding future_obs_index, this value should be in DataConfig.future_observation_indices, but different embodiment have inconsistent fps, such as 15Hz, 20Hz, 30Hz, etc. How should this issue be handled during pre-training? Could it be that the model learns on its own, based on the EmbodimentTag, that some embodiments need to act faster, and some slower?

[KaiLiu18]

@ziyangwang1992 Thanks for pointing this out.

1. For cross_attention_dim, in QwenMMDiT.py, this value is aligned with the hidden dimension of the VLM. For Qwen3-VL, the hidden size is 2560, so the checkpoint uses cross_attention_dim=2560.

2. For future_obs_index and freeze_module_list, the RoboCasa fine-tuning script run_lerobot_datasets_LDA.sh explicitly overrides these fields. Therefore, the values in the actual checkpoint may differ from the default values in the config file.

3. Regarding inconsistent FPS across embodiments, during pre-training we first normalize the sampling frequency of different datasets to roughly the 10–20 Hz range. In addition, we use embodiment-specific action encoders, so the model can condition on different embodiments and capture differences in execution speed caused by hardware or control frequency.

[ziyangwang1992]

@KaiLiu18 Thanks for your kindly reply.

1. For future_obs_index and freeze_module_list, actually, I'm looking to understand the pretraining settings., not RoboCasa fine-tuning.
2. And, could the stats.json in pretrain stage be released, i think it's important for inference.
3. Since the preprocessing pipeline is not open, could you confirm if the model processes rotation as Euler angles?

[ziyangwang1992]

@KaiLiu18 Thanks for the quick response — this is very helpful!

The batch size difference likely explains our issue. We're using 8 GPUs × per_device_batch_size=10 = 80 effective batch, while your setup is 32 GPUs × 10 = 320. That's a 4× gap.

32 GPUs × 300k steps for a 1000-trajectory RoboCasa finetune — seems quite heavy for a sim finetune task. Is this the standard recipe?

We'll restart with gradient_accumulation_steps=4.

@AlbertHuyb hi，have you solved this problem