BeTTER: Diagnose the Illusion of Embodied Reasoning in Vision-Language-Action Models¶
Conference: ECCV 2026
Paper: Official ECCV page
Full Text: Paper PDF
Code: https://github.com/BeingBeyond/BeTTER
Area: Robotics / Embodied Intelligence
Keywords: embodied reasoning, vision-language-action models, causal interventions, compositional generalization, state tracking
TL;DR¶
BeTTER separates motor competence from semantic, compositional and state reasoning through task interventions designed to keep familiar motions executable: three representative VLAs achieve only 5%, 15% and 0% success on unseen subgoal compositions, while physical robot tests also reveal behavioral inertia and phase conflicts.
Background & Motivation¶
A robot completing an instruction does not necessarily understand it. If the red mug always occupies the same location in training and evaluation, mapping the word "red" to a fixed grasp trajectory can yield high success. If packing always starts with a burger and then proceeds to fries, a policy can memorize the order without interpreting which items the current instruction requests. Near-saturated scores on benchmarks such as LIBERO therefore cannot, on their own, establish reliable semantic connections between vision, language and action.
Simply moving objects does not automatically solve this evaluation problem: failures could reflect grasp precision, collisions or unfamiliar motions rather than reasoning. LIBERO-Plus and LIBERO-PRO already expose fragility under distribution shifts. BeTTER focuses more specifically on changing task semantics, compositional relationships and progress cues while minimizing low-level execution confounders. Its question is not merely whether the robot still succeeds, but which correlations its original success depended on.
The paper builds composable tasks and interventions, then uses simulator states and physical robot controls to interpret behavior. Core idea: actively break spurious correlations between language, position, object attributes and task phase within the support of familiar motor skills, diagnosing embodied reasoning through failure modes rather than a single success rate.
Method¶
Overall Architecture¶
BeTTER is a diagnostic benchmark and experimental protocol, not a new VLA architecture. It takes interaction templates, object descriptions and a small demonstration set, constructs executable scenes, amplifies trajectories, and then applies targeted evaluation interventions while recording behavior and physical states. The current instantiation contains 10 base manipulation tasks and 60 diagnostic variations, spanning physical requirements from loose containment to precision insertion.
The important connection is between environment construction and diagnostic questions. Replaceable assets allow researchers to change target identity rather than just pixels; controlled variations test whether a particular shortcut is being exploited rather than merely making a task harder. Privileged states support annotation and analysis, which does not imply that evaluated policies receive those ground-truth states during execution.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Task templates and<br/>object descriptions"] --> B["Template-based<br/>Scene Construction"]
B --> C["Demonstrations and<br/>Trajectory Amplification"]
C --> D["Controlled Reasoning<br/>Interventions"]
D --> E["Privileged-state<br/>Diagnosis"]
E --> F["Failure modes and<br/>physical validation"]
Key Designs¶
1. Template-based Scene Construction: vary task logic and object assets separately
The process starts with high-level interaction templates for manipulation, placement, containment and sequential assembly. A VLM instantiates these templates into concrete objects, goals and physical constraints. Object descriptions then guide retrieval of compatible 3D assets from repositories such as Objaverse, followed by simulation normalization, including centering and collision-bound adjustment. Because contact-rich tasks are particularly sensitive to geometric errors, lightweight human verification also refines placement and physical compatibility instead of treating every automatically generated scene as valid.
The purpose is not simply to increase asset count, but to provide an actionable interface for semantic counterfactuals. A fruit-packing task can retain its container and motor requirements while changing object identity near the target. The difference between a red apple and a red block thus becomes an evaluation variable rather than being ruled out by a fixed asset collection. Physical checks are essential: if the replacement makes grasping infeasible, failure cannot cleanly be attributed to semantic confusion.
2. Demonstrations and Trajectory Amplification: reduce manual variation costs while retaining motor controls
A small set of canonical demonstrations is collected through teleoperation for each base task. Procedural amplification methods such as MimicGen retarget demonstrations to different object poses, layouts and instances, providing physically valid manipulation trajectories without manually recording every variant. The ability to generate trajectories for variations must not be confused with exposing every test combination during training. Each diagnostic still needs its designated held-out instruction-scene pairings, primitive compositions or layouts.
In the coffee diagnostic, in particular, objects, layouts and motor primitives have all appeared in training; novelty lies in how they are paired. Physical experiments further constrain targets to configurations covered by learned motor primitives. The authors call this kinematic isolation: the aim is to make failures informative about target selection or task-progress estimation rather than an inability to perform a new movement. This control reduces confounding; it does not prove that every perceptual or contact-related error has been eliminated.
3. Controlled Reasoning Interventions: break position, semantics, action-order and progress shortcuts
Spatial layout interventions rearrange objects or their relationships to distractors and recombine spatial terms with attribute terms. In Preparing Morning Coffee, "the top mug" and "the red mug" can refer to different targets under a counterfactual layout. Following the trajectory associated with "red" in training exposes a lexical-to-kinematic shortcut. Primitive recomposition instead trains on \(A\to B\) and \(A\to C\) and evaluates \(B\to C\), where the letters denote packing a burger, fries and a drink. The novel sequence requires no new grasp skill, but it requires abandoning the most frequent opening action as an obligatory first step.
Adversarial object perturbation replaces targets or distractors with visually similar but semantically different objects, such as exchanging a red apple for a red block at the same location. Temporal extrapolation extends horizons through chained subgoals or repeated actions to test persistent state tracking. A related progress diagnostic directly changes the initial state: training always starts with two burgers, one is packed, and fries come next; evaluation starts with one burger while retaining the instruction to pack the burger first. Immediately packing fries indicates that "one burger remaining" has become a phase cue, without checking whether a burger actually entered the destination container. These interventions expose semantic feature collapse, behavioral inertia and causal state confusion, which should not all be reduced to one undifferentiated generalization failure.
4. Privileged-state Diagnosis: identify the wrong object or skipped step behind a failure
At each timestep, the simulator synchronously records depth maps, 2D and 3D bounding boxes, and object segmentation masks. These support task-oriented VQA annotations and scene descriptions, allowing analysis of object grounding, spatial errors and task progress beyond a terminal failure flag. In Tidying Office Desk, valid targets include crumpled napkins and cans, while functional objects such as computer mice and intact mugs are distractors. Evaluating only novel layouts further weakens fixed-position priors.
This task uses Distractor Grasp Rate, or DGR: the proportion of grasp attempts directed at invalid objects, excluding null or aborted actions from the denominator. It therefore measures how often an attempted grasp targets the wrong object, not episode failure, and does not penalize every instance of inaction. The cache provides this verbal definition; this note does not invent an equation absent from the source. DGR should be interpreted alongside refusal behavior and task completion, since a frequently stalled policy might otherwise appear safer.
A Worked Example¶
Consider the paper's physical long-horizon task, "Put all the fruits into the fruit basket." A canonical trajectory packs the lemon and then the mangosteen. At evaluation, the lemon is already in the basket and the mangosteen remains on the table, while the robot starts from its initial pose. The appropriate response is to pack only the mangosteen. Instead, visual evidence that the first step is complete conflicts with a robot pose associated with the beginning, producing freezing or skipped actions.
A second intervention leaves the lemon on the table but removes the mangosteen. After packing the lemon, the robot still performs an air grasp at the mangosteen's expected location. Completing the first step has triggered a remembered successor rather than a fresh check for the next target. These are qualitative diagnostics reported in the paper, not success-rate estimates accompanied by trial counts or confidence intervals.
Loss & Training¶
The paper proposes no new loss function. Its argument is that behavioral cloning rewards cues that predict demonstrated actions without directly requiring causal task structure. Static layouts and fixed subgoal orders can consequently become reinforced shortcuts. The authors use InternVL-3.5 capacity, joint-training data and input-resolution ablations to examine representational bottlenecks, followed by pre-training recipe comparisons on LIBERO and CALVIN to test whether high success masks these differences.
Physical experiments use SO101 and \(\pi_{0.5}\) with short and long task horizons under two layouts, giving four canonical configurations. Each supplies 25 demonstrations, for 100 teleoperated trajectories in total, followed by 10,000 fine-tuning steps. The training figure uses blue tablecloths for short tasks and green tablecloths for long tasks, a contextual correlation worth considering when interpreting the protocol. The cached main text does not provide enough hyperparameters to reproduce all simulation training; the physical setup must not be generalized into a uniform training recipe for every model.
Key Experimental Results¶
Main Results¶
The following numbers come from the paper's Table 2. Success rates are percentages, while changes are percentage points relative to the mean success of the two seen sequences, not relative percentage decreases.
| Model | Seen A→B | Seen A→C | Unseen B→C | Change from seen mean |
|---|---|---|---|---|
| \(\pi_{0.5}\) | 60.0 | 45.0 | 5.0 | -47.5 |
| GR00T-N1.6 | 75.0 | 40.0 | 15.0 | -42.5 |
| Being-H0.5 | 65.0 | 40.0 | 0.0 | -52.5 |
Target-selection results in Table 1 further localize language shortcuts: GR00T-N1.6 achieves 100.0% for both top and bottom spatial instructions, but only 5.0% for both red and blue attribute instructions; each random baseline is 50.0%. This suggests a failure to consistently combine language conditioning with visual verification, not complete disregard for language. Target-selection accuracy must not be conflated with full-sequence success in the table above.
Ablation Study¶
Table 3 progressively applies VLA deployment constraints to static embodied question answering with InternVL-3.5. EmbSp. measures embodied spatial understanding, RefSp. fine-grained reference grounding, and EgoPlan sequential planning. These are the paper's aggregate scores, higher is better, not robot execution success rates.
| Configuration | EmbSp. | RefSp. | EgoPlan |
|---|---|---|---|
| 8B Base | 74.96 | 27.50 | 40.80 |
| 4B Base | 72.08 | 23.00 | 36.83 |
| 2B Base | 60.19 | 6.00 | 33.38 |
| 2B + VLM + VLA Data | 57.47 | 29.50 | 31.95 |
| 2B + VLM + VLA Data, single 224×224 input | 52.42 | 7.50 | 31.95 |
Reducing capacity from 8B to 2B lowers RefSp. from 27.50 to 6.00. Mixed-data training restores it to 29.50, but a single low-resolution input pushes it back to 7.50. Meanwhile, EgoPlan changes from 33.38 to 31.95 rather than recovering alongside grounding. This supports the narrower claim that local perceptual alignment is easier to recover than sequential planning, not that adding VLM data generally repairs reasoning.
Pre-training recipe ablations from Table 4 provide a complementary comparison. CALVIN ABC→D tests environmental transfer; five-task completion and average completed length are both higher-is-better metrics.
| Pre-training recipe | LIBERO average success % | CALVIN five-task completion % | CALVIN average completed length |
|---|---|---|---|
| No pre-training | 97.4 | 58.7 | 3.67 |
| VLA (OXE) | 96.3 | 62.7 | 3.86 |
| VLA + VLM | 97.2 | 71.1 | 4.09 |
Key Findings¶
- Static benchmarks can compress representational differences: no pre-training still yields 97.4% on LIBERO, whereas joint pre-training raises CALVIN average length from 3.86 to 4.09 and five-task completion from 62.7% to 71.1%. These results do not imply that pre-training is worthless.
- Physical tests reproduce memorized action chains and phase conflicts, but Tables 5 and 6 are behavioral classification matrices, not quantitative success tables with repeated-trial counts. Short-horizon target-absence cases may also violate the task premise; the authors explicitly avoid treating them as standalone evidence of reasoning failure.
- All three VLAs suffer large drops on unseen compositions, arguing against a single model's isolated grasping defect. The experiments nevertheless do not represent every robot, task or training paradigm.
Highlights & Insights¶
- The diagnostic unit is the correlation being broken. Recombining familiar objects, actions and instructions makes it easier to investigate what a policy relies on than simply adding noise.
- The protocol separates a state that looks intermediate from a subgoal that has actually been completed. Pre-placement tests can transfer to assembly or cleaning to check whether a policy can resume from a valid intermediate state.
- Static question-answering ablations complement robot interventions. The former examine representational damage under deployment constraints and the latter reveal action-level fragility, but they do not provide a one-to-one causal proof.
Limitations & Future Work¶
- Coverage remains limited: the current benchmark has 10 base tasks, 60 variations and three simulation-evaluated models, while physical validation covers only \(\pi_{0.5}\) and SO101. More complex bimanual coordination, mobile manipulation and longer historical dependencies remain untested here.
- The causal analysis is stronger than an ordinary leaderboard but weaker than complete attribution. Capacity, resolution and pre-training ablations support the explanation without proving that a particular representational bottleneck uniquely causes each control failure. The paper also does not demonstrate a systematic repair using a new training objective.
- Statistical and reproduction details are limited. The cached main text does not fully specify repetitions, confidence intervals or all simulation hyperparameters. The correlation between tablecloth color and task horizon in physical training also merits crossed randomization.
- Kinematic isolation reduces rather than eliminates execution confounding. Future protocols could add reachability and contact-success controls and jointly report target selection, refusal, valid grasps and full-task completion, preventing one metric from masking other failures.
Related Work & Insights¶
- Compared with LIBERO-Plus / LIBERO-PRO: these already test visual and structural shifts. BeTTER emphasizes replaceable semantic assets, primitive recomposition and task-progress interventions, extending diagnostic questions rather than first discovering VLA distribution-shift fragility.
- Compared with MimicGen: MimicGen amplifies trajectories from a small demonstration set. BeTTER uses that capability to support controlled scenes and motor-competence comparisons; its main contribution is the diagnostic protocol rather than a trajectory-generation algorithm.
- Implications for VLM–VLA joint training: preserving grounding does not imply preserving planning. Explicit task state, pre-action target verification and intermediate-state recovery training are worthwhile directions, but they are not methods validated by this paper.
Rating¶
- Novelty: 4/5. An extensible diagnostic protocol combining semantic counterfactuals, composition tests and state interventions, with clear value on top of prior robustness evaluation.
- Experimental Thoroughness: 4/5. Multiple models, architectural and data-recipe ablations, and physical validation are included; statistical details and real-world coverage remain limited.
- Writing Quality: 4/5. Failure modes map clearly to concrete tasks, although claims about root causes require interpretation proportional to the evidence.
- Value: 5/5. A directly reusable evaluation approach for deciding what high robot success rates actually establish.