Reproducing GenieSim manipulation benchmark scores

[Idan-BenAmi]

Hi, I'm trying to reproduce the GenieSim manipulation benchmark results and would like to make sure our evaluation protocol matches the reported numbers. Could you clarify a few details?

1. Are the reported π0.5 / π0 / GR00T manipulation results evaluated using a single checkpoint across all 10 manipulation tasks, or is each task fine-tuned separately?
2. How many episodes are used per task to compute the reported success rates?
3. Are results averaged over multiple --benchmark.seed values?
4. Are scene randomization and benchmark-instance randomization enabled during the official evaluation?

Thank you.
Idan

[Anonymous-694]

@Idan-BenAmi Hi, here is the detail:

1. the reported results are evaluated using a single checkpoint for each task suite (instruction, manipulation, etc),
2. episode number is by default as is in the the repo
3. currently it's not evaluated over multiple seed, seed is also by default.
4. official evaluation setting is exactly the same as what you can see in this repo, no further configuration is needed

[Idan-BenAmi]

Thanks @Anonymous-694

[aopolin-lv]

Hello, I also have a question about the evaluation results. Which specific evaluation indicator is given in the readme now? Is it e2e or statistics.average?

[Anonymous-694]

@aopolin-lv Hi, in the readme leaderboard it is statistics.average not e2e.

[aopolin-lv]

@aopolin-lv Hi, in the readme leaderboard it is statistics.average not e2e.

Thanks!