LIBERO spatial · Pi0.5 · controlled visual intervention

Measuring background impact on performance

Task-irrelevant backgrounds can substantially affect robot performance and cause failures.
Offline comparisons and attention maps can reveal which backgrounds are harmful and why, and give a signal on where a policy can be improved.

1 · Intervention

Same task, different backgrounds, different outcomes

Task 1: pick up the black bowl next to the ramekin and place it on the plate. Both transformed rollouts start from the same clean simulator state.

Baseline · normal background The clean rollout completes the task in 115 steps.
Success · food processing line The transformed rollout completes the task in 111 steps.
Failure · stocked warehouse The transformed rollout reaches the 220-step cap without completing the task.
1

Hold the task, camera, robot, and simulator state fixed.

2

Preserve the robot and task-relevant objects; replace everything else with the selected background.

3

Predict and execute 50 actions, then re-observe the actual state.

2 · Result

What backgrounds make the model fail?

We ran this experiment across all 10 LIBERO-Spatial tasks. The policy succeeded on 9 of 10 clean task rollouts. Across those same clean-success tasks, replacing the non-task background caused 51 failures across 90 matched rollouts, reducing observed success to 43%.

LIBERO-Spatial tasks (baseline)
90%

9 of 10 clean task rollouts succeeded. Please note, Pi0.5 executes 50 actions per inference, increased from the published 10, to reduce the impact of repeated model-inference drift on the comparison.

With replaced backgrounds
43%

39 of 90 matched transformed rollouts succeeded.

Most impactful background
22%

india textile workers, heavy factory machine, food processing line: 2 of 9 tasks succeeded when these backgrounds were used.

Least impactful background
78%

wet lab modern: 7 of 9 tasks succeeded with this background.

Ten tested replacement backgrounds ordered by induced failure rateindia textile workers background photograph
Failure rate per tested background. Each tile reports how many of the nine clean-success tasks became failures. Red indicates high impact, amber medium impact, and green low impact; backgrounds are ordered from greatest to least impact.

Why do they fail?

We analyze a failed example to see whether we can gain insight into what caused the failure. In this example, attention on the bowl's edge drops under the replaced background, and the rollout ultimately misses the bowl. Other attention differences are sporadic.

Failure · stocked warehouse The replaced-background rollout reaches the 220-step cap after missing the bowl.
Baseline, augmented, and difference attention maps at step 0 for the agent and wrist cameras
Baseline versus replaced-background attention at step 0. Red indicates more attention than baseline; blue indicates less. We can see that the edge of the bowl lost attention from the model, and the arm missed the grab.
3 · Offline evals

Can we see this without running more rollouts?

We tested whether the outcome of an expensive closed-loop rollout (simulated in this case) can be estimated from model predictions alone, without executing a new trajectory. We find a high correlation between Acc@.1 measured without rollouts and success rates from full closed-loop rollouts, when averaged across backgrounds.

How the offline comparison works

Start from saved states along a successful baseline rollout. At each frozen state, query Pi0.5 twice: once with the baseline images and once from the same state with the backgrounds replaced. Compare the two predicted 50-step action sequences without executing either sequence in the simulator.

What Acc@.1 measures

Acc@.1 is the fraction of the 50 × 7 predicted action values whose clean and replaced-background predictions differ by no more than 0.1 normalized action units. Higher Acc@.1 means the background replacement changed fewer predicted actions.

Acc@.1
Acc@.1 versus induced failure rate by backgroundTen background replacements. Lower accuracy is associated with more induced failures.0%25%50%75%100%india textile workers: Acc@.1 0.405, 78% induced failure rateheavy factory machine: Acc@.1 0.403, 78% induced failure ratefood processing line: Acc@.1 0.401, 78% induced failure ratewet lab bright: Acc@.1 0.404, 67% induced failure ratefactory conveyor: Acc@.1 0.412, 56% induced failure rateindia textile machinery: Acc@.1 0.411, 56% induced failure rateempty warehouse racks: Acc@.1 0.410, 44% induced failure ratewarehouse workers: Acc@.1 0.417, 44% induced failure ratestocked warehouse: Acc@.1 0.416, 44% induced failure ratewet lab modern: Acc@.1 0.426, 22% induced failure rate0.3990.428Spearman ρ = -0.885Acc@.1 (higher is better)As with the Dyna2 blog, Acc@.1 is the fraction of predicted action values whoseclean and replaced-background predictions differ by no more than 0.1 normalized action units.Induced failure rate in closed-loop rollouts (%)
4 · Real-world evals

Why this matters for real-world deployments

Real-world rollouts are expensive to collect and often do not cover the full range of scenarios a deployment will face. By augmenting the visual cues in existing rollouts, we can expose the model to a higher degree of environmental complexity without physically changing the environment or running more rollouts. This extracts meaningful signals about where a policy may fail and what data would be most beneficial for improving the model.