How to get your robot to almost play with its toys

In the last blog, we set up a few auto-resetting environments. Here we trained a policy to perform with high accuracy on the setup (98%). In doing so, we A) understood what is good and bad about such evals, B) the practical approach in how they can help with training a policy, and C) why running enough rollouts is critical for extracting insights. This is a much more technical blog.

Key ideas

We used auto-evals to set up a well-performing demo, and to score it. We only achieved 98% (not 99%), which for demo purposes works really well, but is not good enough to make a highly polished post. Here's the rougher version. We hope you enjoy some of the hidden gems we found.

• Consistent evals were harder than expected. Wi-Fi speeds, limited rollouts, and slight environmental changes introduced noise. With 50 rollouts I expected to have a good signal, but found that results could really change by the time we got to 100 eval rollouts.
• Intervention-data quality matters for performance. The SO-101 arm is weak. When collecting intervention data and switching between a policy and teleoperation the arm would bounce a bit. This harmed intervention-data quality. Removing the first few data points during intervention data helped.
• Automated evals are cool—and might be useful. Auto-evals made scoring incredibly easy (we ran them for >20 hours). I would let them run while I worked on something else, paying no attention to them. Data collection also becomes easier because the environment does not need to be reset. Setting up multiple training runs, and scoring multiple policies at once, I felt a bottleneck in having just a single setup. More to come.

What we did

Here’s the practical recipe we used: how we collected the data, trained MolmoAct, and evaluated each checkpoint.

• Data: We first collected an hour of purely going and continuously doing the task. We collected an hour of normal rollouts (~250 picks and places), 30 minutes of rollouts where we approached the ball from different directions, and an hour of intervention data across 3 iterations of retraining. In collecting data, we also noticed that at the start of the intervention, the arm slightly sagged, so dropping the first 200 ms prevented that drop from entering the dataset. This was interesting.
• Training: We tuned only the Action Expert with batch size 64 and learning rate 5e-5. Early experiments with MolmoAct outperformed tuning π0.5, so we switched over. Intervention data collected with π0.5 was helpful.
• Evaluation: Our evals were totally automated (in reality, they were 95% automated. The ball would sometimes roll out of the environment and couldn't be recovered. Oops, our next iteration of evals will be better). Next, we considered a rollout a failure if it didn't succeed within a minute. However, sometimes Wi-Fi lag existed. To account for it, we would count only policy inference time + rollout time. Second, when comparing models, we ended up consistently swapping models to minimize any effects from environmental drift like bumping the setup. In the end, we could train multiple experiments in parallel, and then compare them all at once. This was very convenient.

Results

We tracked model performance at as training progressed, and how model performance drops when we drop partitions of data from training (ie. the value of data). The results were straightforward. We saw that (A) dropping the first 0.2 s of intervention data was helpful and (B) intervention data played the most significant role in policy performance.

• Evals ran for about 20 hours. We wish we had collected more evaluations. Scaling this now becomes a question of how many arms we have. We may simply need to run on more arms to get more granular results.
• While we report success, we found throughput on successful rollouts to yield a more consistent signal for policy performance. We have seen similar findings in industry.
• An interesting observation: when using graph-optimized inference, there was lots of jitter around step 500, but step 2,000 became very smooth.

Postmortem

We’ve reached 98% and, alongside our broader work, plan to keep pushing toward 99%. Once we get there, we’ll open-source the full setup. In the meantime, if you’d like to try something similar, we’d be happy to discuss our approach and share pointers. In the meantime, here’s what we wish we had done differently.

• Setup consistency: Performance dropped for later experiments. It was likely because jerky motions caused servos to slightly slip. Next time, we would recalibrate repeatedly and collect data across multiple calibrations so that calibration would not affect performance.

  The setup could also be better. While 3D-printable, this setup is not 100% reproducible due to the custom ball. Similarly, the cup would interfere with the camera in some spots, making the task harder. This will improve in the next iteration.

• Limited experiments: There were many experiments that would be easy to kick off, but we never got to. For example, (A) running LoRA on the action expert, (B) making the prompt more granular, and (C) experimenting with multiple cameras. This was ultimately due to a GPU + hardware constraint we faced initially.

  Similarly, I wish I had just collected more data off the bat and trained a longer run for a better baseline. The MolmoAct repo suggests training for 50k steps, yet we noticed that performance flattened around 2,000 and intervention data was more effective at improving performance. The full run would have been a better baseline.

• RL: When model performance reached > 95%, collecting intervention data became much harder. 98% is not good enough. We expect RL to fix this.
• Auto-eval: The auto-eval let us run evals in the background for 20 hours. However, it still needed occasional resets, especially with weaker policies. This will be improved for future runs.

A big thanks to VESSL AI for providing GPUs for these experiments. This solved a huge bottleneck for us. If there is interest from the hardware side, we would love to partner with a company to scale the eval fleet.