BluPe316 Industrial Benchmark

Robotics Ecosystem

We believe robotic learning is starting to mature as an ecosystem. Companies are beginning to gain an advantage by focusing on individual components rather than building everything themselves. The core areas we see are hardware, models, data, and applications.

We believe much of the value for deployment comes from deep knowledge of an application. However people closest to these applications often have limited visibility into the state of the art in robot learning, and where its fits in with classical robotics.

Our goal is to bridge the two by:

1. Present the performance of current robotics systems on those applications.
2. Characterize the economics of the specific applications
3. Make a baseline deployment procedure that easy to follow.

We hope this will make it easy to understand which applications are possible to deploy today, and which applications are close.

Challenges With an Industrial Benchmark

Robotics benchmarks currently evaluate entire tasks end to end. This differs from production, where learned methods are typically combined with classical robotics. Although more representative, hybrid systems increase the amount of bias that a benchmark itself introduces.

Reproducibility is another challenge as with all real world robotics benchmarks. Small differences in hardware and environment can significantly affect results.

We hope that working directly with downstream applications will force our benchmarks to be reflective of production level implementation, and as reproducible as possible.