Issues with Scene Rendering and Gripper Replacement in RLinf/GenieSim

[CharlesWiranto]

I am currently integrating RLinf with GenieSim and have encountered two issues that I need assistance with.

1. Discrepancy Between Mujoco and Isaac Sim Scene Rendering

I have successfully built the RLinf Docker container for GenieSim. However, I am unsure if the final scene is loading correctly. When I run the simulation, the scene appears correctly in the Mujoco viewer (showing the expected rack), but the same rack is not visible in the Isaac Sim viewport. Is it expected?

2. Replacing the Default Gripper with Omnipicker

I would like to replace the current default gripper in the RLinf environment with the Omnipicker gripper. Additionally, I need guidance on how to control this new gripper and how to add other objects (e.g., tables, bottles) to the scene.
My specific questions are:

• What is the recommended procedure to swap the default gripper for the Omnipicker in the RLinf/GenieSim asset pipeline?
• Once the Omnipicker is loaded, how can I control it? Can I use Pico to control the arm and omnipicker?
• Is there a documented workflow or example for adding custom static and interactive objects to a benchmark task configuration (similar to the extra_objects, fix_objects, and task_related_objects mentioned in the documentation)?

[Chenghao-Yin]

Hello!

1. Isaac Sim Scene Issue
   Please make sure background/common/rlinf_place_workpiece is in your local assets folder. If it's missing, you may need to download the latest GenieSim assets. If the assets are downloaded correctly, please check the Isaac Sim logs to see if there are any errors during asset loading.

2. Replacing the Gripper
   The key step here is to generate an MJCF file for the Omnipicker robot, which you can generate from either the URDF or USDA files.

• Control: Please refer to the teleoperation code in the GenieSim project.
• RL Training: If you want to train an RL model using Pico, you will need to write a PicoInterventionWrapper on the RLinf side.

3. Adding Custom Objects
   To add custom objects to the benchmark task configuration, you can generate the simulation scene using geniesim_generator, which fully supports adding custom objects.