What is molmoact2-so101-sim?

This project connects two pieces of robotics software that were not built to work together. MolmoAct2 is an open AI model from Ai2 that can look at a scene and decide how a robot arm should move to grasp an object. SO-101 is a specific low cost robot arm, and there is a free computer simulator of it that lets you test robot arm behavior without owning real hardware. The problem the author solved is that the AI model expects the arm's joints described one way, while the simulator describes them another way, and there was no code linking the two, no way to collect training examples, and no way to score how well it worked. This repository is that missing connector, plus a training recipe. Using this project, the author took the AI model from being able to grasp a cube zero percent of the time to being able to grasp it successfully ninety three percent of the time, in five days of work on a single high end graphics card, without any human physically demonstrating the task by hand. Instead, a scripted program generated example movements automatically. Along the way, the author tested several variations and shares which changes actually helped, such as simplifying the gripper to a simple open or closed state, and which changes that seemed reasonable actually made results worse, such as adding small relative movement adjustments. The repository includes the connector code, the scripts used to generate training examples, the evaluation and scoring tools with recorded videos of the arm's attempts, and a troubleshooting guide covering thirteen specific problems the author ran into. A trained model and its training data are shared publicly so others can reuse them without repeating the training process. The project also explains how someone could adapt the same approach to teach the arm a different task besides picking up a cube. This is research level robotics work aimed at people already comfortable with AI models and robot simulation, requiring a capable graphics card with substantial memory for the training and evaluation steps. It is shared under the Apache 2.0 license, which allows free use including commercial use.