Benchmark Setup
Tasks
We set up five tasks where we believe learned motion may be the best fit for industrial applications, aiming to cover edge cases that come up unexpectedly when designing classical robotic systems. These include unstructured bin picking with diverse objects; pin insertion and washer placement while allowing slight motion in the base; diverse object handling for warehouses; and wire manipulation. Please find more details below.
Generaliz-
ability
A big challenge in robotics is the lack of generalizability. This means that policies may work in one environment but fail when moved into a factory. To address this, we visually modify images before sending them to models, using simulated backgrounds that represent a variety of factory settings. This enables us to evaluate models’ ability to generalize across scenes.
Unstructured Bin Picking
Pick parts from an unstructured pile and place them on a known fixture. Changes in part geometry currently require custom grippers, suction tooling, or part-presentation hardware.
We expect learned policies to handle picking objects up and classical manipulation to place them onto fixtures. Even with a high failure rate, retries are typically not costly.
Pin Insertion (After Base Movement)
We heard that a frequent failure case occurs when objects move slightly away from their fixed position due to disturbances or vibrations.
We plan to set up a task where a target hole varies in position and a rigid pin must be inserted. Classical computer vision can fail when insertion needs to be highly precise.
Washer on Peg (After Base Movement)
As with the previous task, we test the robot’s ability to adjust to a changed environment.
A washer must be placed onto a peg whose position is not fixed. Although the image shows the peg in one position, we will move it between trials. When motion needs to be precise, classical computer vision can fail.
Diverse Object Handling
Pick and pack a wide variety of objects into shipping boxes. Shape, stiffness, surface finish, weight distribution, and safe grasp points change from item to item.
With traditional classical approaches, this level of generalization becomes difficult.
Wire Manipulation
Wires are small and deformable, making them hard to position exactly. Slight mis-grips can create more variance than classical approaches can tolerate.
The task is to insert the tip of a wire into an opening whose position can change.