Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain¶

Conference: ECCV2026
Paper: ECCV Paper
Project: https://robo-bench.github.io
Area: Multimodal VLM / Robotics & Embodied AI
Keywords: embodied cognition, long-horizon planning, affordance prediction, failure analysis, world-simulator judge

TL;DR¶

RoboBench evaluates 18 models using 6092 questions across five embodied cognitive dimensions and a DAG-guided MLLM world-simulator judge, showing that strong general question answering and verbal planning do not guarantee implicit-intent understanding, spatial grounding, or real execution failure diagnosis.

Background & Motivation¶

Putting an object into a drawer requires more than moving a robot's end effector to a coordinate. The robot must understand what the user wants stored, recognize whether the drawer is closed, open it before insertion, and reconsider its next step after a failed grasp. Dual-system embodied architectures assign this slower reasoning and decision-making to System 2, while System 1 handles high-frequency control. The paper calls the former the embodied brain and asks whether multimodal large language models (MLLMs) possess the cognition needed for that role; it does not introduce a new control policy. Measuring only overall manipulation success conflates perception, planning, and control errors, leaving researchers uncertain about which capability needs improvement.

Existing embodied question-answering benchmarks make cognitive abilities measurable, but often study environment understanding, next-step prediction, or failure explanation separately. Human egocentric video also differs from robot data: arm configuration, camera viewpoint, and available skills can change which answer is valid. Text similarity is particularly problematic for planning, because plans with different wording or action orders may be equally executable. Conversely, a plan can contain the reference answer's keywords while omitting a prerequisite such as opening the drawer. RoboBench therefore expands capability coverage, realistic scene complexity, and planning evaluation together, rather than merely increasing the number of questions.

Existing large-scale robot videos and MLLM-assisted structured annotation followed by human review make this design feasible. The data span embodiments, object attributes, multi-view occlusion, and memory-dependent navigation, reducing reliance on commonsense completion alone. Nevertheless, the benchmark primarily evaluates responses to offline inputs and must be distinguished from closed-loop robot deployment. Core Idea: separate intent understanding, scene perception, planning, spatial guidance, and failure diagnosis along the manipulation pipeline, then evaluate open-ended plans through action dependencies and object-state changes rather than surface language similarity.

Method¶

Overall Architecture¶

RoboBench is a dataset and evaluation protocol, not a new trainable network. Inputs vary by task and include scene images, robot video observations, instructions, answer options, or the current subtask state. The evaluated model produces a multiple-choice answer, a structured action sequence, a next action, or a subtask-completion judgment. The evaluator then applies accuracy, action-component scoring, or world-simulation scoring according to the task, rather than sending every output to the same free-form judge. The five dimensions are Instruction Comprehension, Perception Reasoning, Generalized Planning, Affordance Prediction, and Failure Analysis, covering 14 capabilities and 25 tasks. They represent cognitive roles within a manipulation pipeline, but the dataset does not require one model to complete every stage on the same real robot trajectory.

Table 2 on page 8 distinguishes 4336 unique items from 6092 questions; the question count should not be interpreted as the number of independent scenes. It lists 1895 multiple-choice questions for perception, affordance, and failure analysis; 1973, 842, and 1192 questions for Q1, Q2, and Q3; and 190 navigation questions separately. Q1 has an average of 6.75 steps, with 1403 unique task instructions and 1462 unique answers. These statistics establish the presence of multi-step dependencies, but the average alone does not imply that every example is long or difficult. The protocol can be understood through four connected designs: capability decomposition and paired questions, real data and human quality control, DAG-guided world simulation, and task-specific scoring with downstream validation.

Key Designs¶

1. Capability decomposition and paired questions: separate intent inference, visual understanding, and action grounding

Instruction comprehension uses explicit-implicit pairs that preserve the image and target answer while changing how the user expresses the goal. Explicit instructions specify the action and object; an MLLM generates implicit instructions from the scene and explicit goal, followed by human checks for naturalness and inferability. This compares planning with a known goal against inferring that goal from an indirect request under closely matched conditions, rather than comparing unrelated task sets. Perception reasoning is divided into robotic-centric, object-centric, scene-centric, and task-centric abilities, covering configuration and viewpoint, static and functional attributes, spatiotemporal relations and causality, and instruction-relevant objects. Task-centric questions use human-labeled target boxes to connect long instructions to visible objects, limiting opportunities to guess the referent from language alone. These perception questions are ultimately standardized as multiple choice, so they measure recognition among candidates rather than unconstrained localization accuracy.

Generalized planning covers embodiments, objects, views, and tasks, including multi-view occlusion and navigation based on spatial memory from video. Affordance describes how a subgoal becomes spatial interaction guidance: static affordance identifies a contact point, dynamic affordance specifies a motion trajectory, and navigation affordance identifies a robot base position. The authors sample representative frames from the planning pool, annotate these cues, and ask models to choose among scene-grounded candidates rather than output continuous control commands. Failure analysis further separates low-level execution errors from high-level planning errors, avoiding the conflation of gripper misalignment with an omitted action. This decomposition localizes weaknesses, but choosing the correct contact point does not demonstrate precise trajectory generation or reliable execution by a low-level controller.

2. Real data and human quality control: retain the constraints of robotic manipulation

The planning pool combines public robot videos and in-house collection; Gemini first produces task summaries, timestamped steps, and metadata for objects, actions, scenes, and embodiments. Human annotators refine these outputs and map actions to shared skill templates such as pick_up(object) and move_to(object, target). This intermediate representation turns natural-language plans into inspectable combinations of skills, objects, and parameters, with manipulation and navigation skills organized separately. Perception annotation depends on the task: robot type and viewpoint use metadata, functional attributes and distractors use model assistance, and spatial relations and keypoint changes use human annotation. Video step segmentation supports temporal grounding, while target boxes support reference comprehension; both must remain aligned with visual evidence. Model-assisted annotation therefore does not mean that the benchmark is an automatically generated question set without human verification.

Low-level failures come from real VLA rollouts in RoboMIND, with experts labeling position deviations, trajectory errors, gripper failures, and incomplete actions. Because real high-level planning failures are scarce, the authors follow a REFLECT-style protocol that injects wrong objects, missing steps, incorrect order, or wrong conditions into correct plans, then checks identifiability and task relevance manually. These different failure sources are important conditions when interpreting difficulty gaps, not incidental preprocessing details. Quality control first filters image quality, task validity, and embodiment mismatches, then checks linguistic clarity, answerability, and correctness. Every MLLM-assisted annotation receives at least one human cross-pass; residual disagreement on the validation subset is reported below 4%, which is not a full-dataset label error rate. Items answered correctly by all evaluated models are removed, while universally failed items receive manual review, making the resulting difficulty partly dependent on the filtering model set.

3. DAG-guided world simulation: evaluate long plans through prerequisites and state changes

Q1 asks the model to predict a complete action sequence from the first frame; the evaluator has that image, a reference action list, and a human-annotated directed acyclic graph (DAG). DAG nodes represent atomic actions and edges impose precedence, allowing valid reorderings that preserve dependencies rather than enforcing one exact sequence. The evaluated model generates the plan; the DAG and reference answer are evaluation resources and should not be mistaken for answers supplied to the model at inference time. NodeCorrectness uses one-to-one matching to check agreement on skill, object, and parameters between predicted and reference actions, scoring coverage of reference nodes. This checks whether necessary actions appear, but a correct node set alone does not establish that the proposed order is executable. The second component, TaskCompletion, therefore examines whether the plan achieves critical object-state changes instead of counting textual matches again.

The judge first identifies objects, spatial relationships, and physical constraints in the initial image to form a world state, such as a closed drawer and an apple on the table. It extracts critical state milestones from the reference actions and DAG, establishing their dependencies, permitted concurrency, and causal relationships. It then simulates predicted actions sequentially, checking preconditions against the current world, updating state, and recording achieved milestones. A state must not merely become true temporarily; it must remain valid until its last dependent action consumes it to count as an achieved and protected milestone. For example, the drawer must remain open until insertion finishes; an earlier opening action cannot excuse closing it before the object is placed inside. World simulation here means MLLM reasoning over symbolic states and visual constraints, not execution in a physics engine or verification through generated video dynamics.

The long-horizon score combines node correctness and task completion, with Q1 results displayed on a percentage scale in the tables. Text extraction on pages 9-10 damages the node, completion, and final aggregation equations, removing some operators or scale information. This note therefore preserves the identifiable definitions and procedure without presenting a guessed reconstruction as the authors' exact formula; implementation requires checking the original equations. Although more constrained than free-form scoring, the judge can still misread the initial scene or mistake a verbally plausible action for a physically feasible one.

4. Task-specific scoring and downstream validation: distinguish planning, classification, and control

Q2 predicts the next action from the current observation, and the judge separately evaluates its skill, object, and parameters rather than a complete plan. Skill matching is binary, while object and parameter scores are 0, 0.5, or 1 according to visual reasonableness. The identifiable scoring definition on page 10 is:

\[ \mathrm{NextStep}=\frac{s_{\mathrm{skill}}+s_{\mathrm{obj}}+s_{\mathrm{param}}}{3},\qquad s_{\mathrm{skill}}\in\{0,1\},\quad s_{\mathrm{obj}},s_{\mathrm{param}}\in\{0,0.5,1\}. \]

Q3 determines whether a subtask has already been completed and uses binary accuracy against the annotated state. Perception, affordance, and failure analysis also use accuracy, but on different multiple-choice question sets; their difficulty should not be directly compared with partially credited Q1 or Q2 scores. The main benchmark obtains responses from existing models and introduces neither a shared new loss function nor a new model trained on these questions. Separately, following VLM4VLA, the authors convert several open-source VLMs into VLA policies through minimal fine-tuning and evaluate manipulation performance on CALVIN and LIBERO-10. Only this additional experiment involves policy training, testing whether cognitive scores correlate with downstream capability rather than treating question-answering scores as execution success rates. Judge stability is checked separately: NodeCorrectness achieves ICC(2,k)=0.948 across 200 stratified Q1 items, four vendor judges, and three prompt variants. This supports stability of the tested node-scoring protocol, but does not establish unbiased scoring for every task, all state simulations, or every model.

Key Experimental Results¶

Main Results¶

The following representative results come from Table 3 on page 11, Table 4 on page 12, and Table 6 on page 13, preserving the original Avg values. Perception, affordance, and failure analysis use accuracy (%); generalized planning uses Q1 scores on a percentage scale, not robot execution success rates.

Model	Perception Reasoning Avg	Generalized Planning Q1 Avg	Affordance Avg	Failure Analysis Avg
Human Evaluation	74.30	54.50	82.63	63.99
GPT-5.4-text-only	27.12	73.74	25.61	22.28
Claude-Sonnet-4.6	54.09	79.38	43.87	47.83
Gemini-3.1-Pro	67.32	70.71	85.70	52.95
Qwen3-VL-8B	41.49	56.71	21.24	39.04
RoboBrain-2.5-4B	44.04	31.85	48.09	45.12

Gemini-3.1-Pro averages 85.70% on affordance prediction, but scores only 25.17% on real execution failure diagnosis versus 80.74% on planning failure diagnosis in Table 6. RoboBrain-2.5-4B reaches 43.71% on execution failure diagnosis, exceeding the other listed models, without correspondingly stronger overall Q1 planning. The text-only baseline's generalized Q1 average of 73.74 exceeds GPT-5.4 at 70.91 and Gemini-3.1-Pro at 70.71. The paper's claim that text-only performance is consistently lower on planning therefore does not hold throughout Table 4; language priors and evaluation effects warrant further controls. Human Evaluation should be treated as the paper's human reference rather than a strict ceiling in every column; the available main text does not provide enough detail to fully reconstruct that protocol.

Ablation Study¶

As this is a benchmark paper, the following is a paired-condition analysis from Table 4 on page 12, not a network-component ablation. Explicit and implicit instructions preserve the image and target answer; values are Q1 scores on a percentage scale, with differences calculated from the original table.

Model	Explicit Instruction	Implicit Instruction	Explicit Minus Implicit (points)
Claude-Sonnet-4.6	79.94	61.38	18.56
Gemini-3.1-Pro	73.25	59.90	13.35
Qwen3-VL-8B	59.46	30.80	28.66
MiMo-Embodied-7B	66.87	37.30	29.57

Claude-Sonnet-4.6 performs strongly with an explicit goal but loses 18.56 points when the request becomes indirect, showing that goal-conditioned planning is not equivalent to intent inference. The difference cannot simply be attributed to a perception module, because the intervention changes instruction phrasing and requires joint interpretation of language and scene context.

Key Findings¶

Pearson correlation with human judgments rises from 0.73 for general LLM scoring to 0.83 for world-simulation scoring in Figure 7 on page 15, but correlation does not imply exact agreement on every example.
The best robot-view accuracy is 53.62% and the best temporal-grounding accuracy is 52.07% in Table 3, showing that object knowledge does not ensure reliable reference-frame or temporal reasoning.
Planning case analysis attributes 45%, 24%, 25%, and 6% of errors to execution-related, identification, commonsense, and format failures respectively, on page 13 and in Figure 6; these are planning-case categories, not real robot accident rates.
CALVIN performance correlates with object-centric and scene-centric perception at 0.884 and 0.833; LIBERO-10 correlates with static and dynamic affordance at 0.617 and 0.517, respectively, on page 14 and in Figure 8.

Highlights & Insights¶

Converting a reference plan into dependency and state constraints accommodates valid action reorderings. This better represents planning correctness than rewarding similarity to one reference text.
Paired instructions isolate indirect requests as a practical interaction challenge distinct from planning with a known goal. They support diagnosis of multimodal intent understanding beyond overall leaderboard rankings.
Grouping contact points, trajectories, and base positions under affordance clarifies the interface between high-level subgoals and low-level control. Candidate selection nevertheless tests only part of that interface.
Different downstream benchmarks correlate with different cognitive dimensions, suggesting that backbone selection should consider the target task's capability profile. Neither model size nor one average score is sufficient by itself.

Limitations & Future Work¶

The world-simulator judge lacks a real physical rollout guarantee for each sample; human DAGs reduce structural ambiguity but cannot eliminate mistaken judgments about collision, contact, or reachability.
High-level failures are injected while low-level failures come from real execution, so their difficulty gap reflects both cognitive demands and data provenance rather than a pure capability difference.
Removing universally solved questions makes the evaluation distribution depend on the filtering model set; reporting stability on an independent held-out set would help, as a suggestion from this reading.
Strong text-only Q1 performance indicates that language priors can support some planning scores; counterfactual scenes or swapped visual states could isolate genuine visual dependence, as a suggestion from this reading.
Downstream correlations do not establish causal improvement: data, scale, and fine-tuning may affect both VLM scores and VLA success, while the main text assigns full implementation details to supplementary material not included in this cache.
Exact Q1 equations are damaged in extraction, and the prose claim about text-only planning conflicts with Table 4; this note preserves tabulated values without altering results or inventing missing formulas.

vs RoboVQA: Both study long-horizon robotic cognition; RoboBench adds affordance and failure analysis and replaces BLEU-only planning evaluation with action dependencies and state simulation, according to its Table 1.
vs BEAR / ECBench: The paper treats these as neighboring broad-coverage embodied benchmarks but emphasizes its five cognitive dimensions and open-ended planning evaluation. Broader coverage does not replace separate audits of each task's labels and difficulty.
vs REFLECT: RoboBench borrows its high-level failure construction approach to include error recognition in the benchmark; it does not thereby introduce a new online self-correcting controller.
Research direction: With controller and training budget fixed, strengthen only spatiotemporal perception or affordance and test whether corresponding downstream performance improves. This follows from the correlation analysis but is not an effect already established by the paper.

Rating¶

Novelty: 4/5. Five-dimensional cognition coverage and DAG-constrained world-simulation scoring are clear contributions, rather than a new model architecture.
Experimental Thoroughness: 4/5. The study includes 18 models, text-only and human references, judge consistency, and downstream correlations, but limited closed-loop validation and causal ablations.
Writing Quality: 3/5. The task organization is clear, although some broad claims do not fully match the tables; exact formula reproduction is additionally limited by local text extraction.
Value: 4/5. Useful for diagnosing embodied-model weaknesses and informing backbone selection, but insufficient as standalone evidence of real robot deployment reliability.