LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Vision-Language Reasoning (vlm_reasoning)
Keywords: multi-step spatial reasoning, LEGO assembly, sequence planning, benchmark, image generation
TL;DR¶
LEGO-Puzzles progressively evaluates elementary spatial questions, assembly planning with distractors, and multi-turn intermediate-state generation, revealing substantial gaps across 29 multimodal models: GPT-5 trails humans by 21.8 percentage points on the Lite subset, while its planning accuracy falls from 90% at one step to 0% at eight steps.
Background & Motivation¶
Recognizing left-right relationships in an image does not establish that a model can guide an assembly process. Practical spatial tasks require understanding the current structure, imagining the effect of installing a part, and using the resulting state as the starting point for subsequent decisions. Existing spatial benchmarks cover object attributes, relative positions, and viewpoint changes, but many questions concern a single static scene or one hypothetical action. Success on those questions does not establish whether a model can maintain consistency across intermediate states or reject candidate steps that look similar but contain assembly errors.
LEGO assembly provides a controlled yet concrete test environment: open-source projects often include part lists and step-by-step instructions, supplying state sequences designed by people. Rendering tools can fix the camera while changing part orientations and installation positions, producing incorrect answers with similar appearances but different spatial relationships. Unlike planning reduced to language-based commonsense ordering, this setup requires attention to specific parts, connections, and state differences; compared with physical robot experiments, it also makes inputs easier to control and answers easier to check.
The paper therefore first separates elementary assembly skills, combines them into planning tasks with adjustable horizons, and finally removes candidate images so that models must generate intermediate states themselves. Its purpose is not to introduce a stronger solver, but to locate failures across output formats: identifying parts, selecting valid states, or preserving assembly constraints during generation. Core Idea: use a shared family of LEGO assembly states to connect elementary spatial judgments, distractor-based step selection and ordering, and multi-turn visual generation, while separately checking local decisions and complete trajectories.
Method¶
Overall Architecture¶
LEGO-Puzzles contains two evaluation sets, Elementary and Planning, both built around rendered images of LEGO parts and assembly states. Elementary diagnoses basic capabilities through 11 question types; Planning supplies an initial and a target state and asks models either to select and order intermediate images or to generate those states over multiple turns. The former tests whether a model can recognize a correct process, while the latter tests whether it can depict a spatially valid process; their outputs and scores should not be conflated.
On the data side, the authors collect open-source LEGO projects with assembly instructions, render and organize states and parts, generate questions using task templates, and perform human review. On the evaluation side, models are not fine-tuned: the main Elementary comparison is zero-shot, and Planning further evaluates models that perform relatively well on Elementary. This is a benchmark and diagnostic study, with no dedicated trainable network, training loss, or training stage that supplies ground-truth intermediate states to the generation models. Its principal designs are controlled data construction, hierarchical capability diagnosis, distractor-based sequence planning, and multi-solution generation evaluation.
Key Designs¶
1. Controlled data construction: establish traceable state changes from real assembly instructions
The authors select projects of moderate final size: very large structures make the visual changes from individual additions too small, while very small structures lack sufficient spatial complexity and sequence length. Projects span multiple object categories; Studio renders their assembly instructions, with the camera fixed across consecutive assembly steps to avoid confusing viewpoint changes with structural changes. Part type, quantity, color, and position can be edited; rotation and multiview tasks deliberately vary orientation or viewpoint and use POV-Ray-style rendering and lighting adjustments. This control allows incorrect candidates to target specific spatial attributes instead of merely using unrelated images as distractors.
PDF-Extract then extracts structured information such as individual parts and assembled objects, after which image names are standardized and manually designed templates specify each input image's role. For example, an installation-position question provides the current structure, the part to install, and the resulting structure, requiring the model to infer the connection point from the state difference rather than merely recognize the part's shape. Quality control includes image-similarity deduplication, manual checks of cameras and part attributes, and verification of questions, image-reference order, and answers by three trained annotators. Samples with unresolved disagreements are revised or removed, making answer clarity part of the construction process.
2. Hierarchical capability diagnosis: distinguish static recognition from state reasoning
Elementary is derived from 407 LEGO building instructions, with 100 questions per task and 1,100 questions across 11 tasks; some passages describe the total as "over 1,100," while this note follows the explicit per-task configuration. The first level comprises Height, Adjacency, Rotation, and Multiview, testing true 3D height relationships, adjacency, rotation angles, and appearance after a viewpoint change. Height is not image-plane vertical position: a part that appears higher in a perspective projection need not be taller in 3D, which probes reliance on a 2D shortcut. The second level comprises Rotation Status, Position, Next-Step, and Dependency, asking whether a part needs rotation, where it should be installed, what the next state looks like, and which parts account for a state transition.
The third level comprises Backwards, Ordering, and Outlier: identifying a valid intermediate state from a target, ordering supplied states, and detecting a state that does not belong to an assembly sequence. These tasks do more than increase the image count: they require judgments constrained by multiple states and by compatibility with the eventual target. All tasks report accuracy, but chance levels differ: binary Adjacency has a 50% random baseline, whereas Ordering four images has a baseline of approximately 4.2%. The authors also provide critical accuracy thresholds for exceeding chance at a significance level of 0.05, so the same raw accuracy across tasks does not imply the same degree of competence.
3. Distractor-based sequence planning: select valid states before ordering them
Plan-k-Step-VQA combines ordering, incorrect-state detection, and backward reasoning. Given an initial state \(x_0\) and a target state \(x_{k+1}\), a model receives \(2k\) candidate images: exactly \(k\) are correct intermediate states, and the other \(k\) are constructed from erroneous or out-of-sequence states. The model must select the correct images and output their assembly order; each setting of \(k\) from 1 to 8 contains 20 test cases. Here, \(k\) denotes the number of intermediate states to recover and should not be equated directly with the total number of physical assembly actions from start to finish.
This design turns ordering into a state-selection problem closer to practical use, where a model cannot assume that every supplied step image is valid. Three metrics separate failure modes: exact match requires the entire sequence to match the ground truth; set match requires precisely the correct set of steps without checking order; overlap ratio measures the fraction of predicted steps that belong to the ground-truth set, also ignoring order. Under the prescribed output format of \(k\) distinct candidates, overlap ratio is the number of correctly selected steps divided by \(k\), aggregated across cases. If exact match falls while set match remains high, ordering is the main issue; if both fall together, the model is already failing to select the valid set of steps consistently.
4. Multi-solution generation evaluation: permit alternative valid routes but require trajectory consistency
Plan-k-Step-Generation removes the candidate answers and asks the model to generate the first intermediate state from the initial and target images, then produce subsequent states conditioned on its own generated history. The evaluation uses \(k=2,3,4,5\); later turns see model-generated states rather than ground-truth corrections, so a misplaced part can affect the entire trajectory. To avoid penalizing alternative valid assembly orders, the authors first expand multiple topological paths under physical and assembly constraints, such as requiring lower parts to be installed before upper parts. This expansion constructs reference candidates for the evaluator; it does not give the tested model an additional planning tool.
For scoring, each generated image is matched using mean squared error (MSE) against all valid candidate images at that step, after which the matched states are checked for membership in a complete legal path. A legal full trajectory receives 1 and an invalid one receives 0; planning success is not an average of per-step image-similarity scores. The protocol passage does not specify an independent image-matching threshold or sufficient alignment details, so it should not be rewritten as strict pixel equality, nor should visual similarity be treated as verified geometric equivalence. To investigate whether failures begin at the first step, the authors also convert five Elementary tasks into single-turn image generation and score appearance similarity and instruction following separately through human evaluation.
Loss & Training¶
The paper introduces no training loss or benchmark-specific fine-tuning. The 29-model Elementary comparison is zero-shot; multiple-choice answers are extracted with rules first and with ChatGPT assistance when rule-based extraction fails, and Ordering uses the same approach for sequence extraction. These auxiliary operations parse tested models' outputs rather than solve the LEGO questions for them. The single-turn generation analysis uses 20 questions per task, totaling 100 questions, with five experts assigning separate appearance and instruction-following scores on a 0-3 scale. These human ratings are neither percentages nor the binary trajectory-success measure used for multi-turn generation.
Key Experimental Results¶
Main Results¶
The table below selects overall accuracies (%) from the paper's Tables 1 and 2. Elementary is the zero-shot evaluation with 100 questions in each of 11 tasks; Lite samples 20 per task for 220 questions and uses 30 human experts for the human comparison. Only values within the same column concern the same evaluation set; the human Lite score should not be subtracted directly from a model's full-set score.
| Model / Baseline | Elementary Overall | Lite Overall |
|---|---|---|
| GPT-5 | 72.0 | 71.8 |
| GPT-o3 | 69.1 | 70.9 |
| Gemini-2.5-Pro | 68.8 | 65.9 |
| Qwen3-VL-32B-Instruct | 57.0 | 60.5 |
| InternVL3-78B | 47.7 | 48.2 |
| Human experts | Not reported | 93.6 |
| Random guessing | 27.5 | Not reported |
The prose in Section 3.2 swaps the 68.8 and 72.0 scores of GPT-5 and Gemini-2.5-Pro; this note follows Table 1 and explicitly preserves the source discrepancy. The human-GPT-5 gap on Lite is 21.8 percentage points in overall average accuracy, not evidence that every task has a gap of at least 20 percentage points.
Ablation Study¶
There is no network-module removal ablation; the following summarizes the planning-horizon analysis in Section 4.1 and Figure 3. The VQA protocol has 20 cases per horizon, and the metric is full-sequence exact match (%); only values or ranges stated explicitly in the paper's prose are included, without estimating decimals from plotted curves.
| Model / Group | Horizon Condition | Full-Sequence Accuracy | Diagnostic Meaning |
|---|---|---|---|
| GPT-5 | \(k=1\) | 90 | Many short sequences remain solvable |
| GPT-5 | \(k=8\) | 0 | Complete success vanishes at long horizons |
| Qwen3-VL-32B-Instruct | \(k>4\) | 0 | Complete-path recovery fails earlier |
| InternVL3-78B | \(k>4\) | 0 | Also fails on longer sequences |
| All four tested models | \(k>3\) | All below 50 | Includes Gemini-2.5-Pro |
| Five human experts | \(k=1\) to \(k=8\) | 100 | Perfect scores at every horizon |
A separate single-turn generation diagnostic in Table 4 reports mean appearance / instruction-following scores of 2.52 / 1.99 for Nano Banana Pro and 2.25 / 1.88 for GPT-5 on 100 questions, both on a 0-3 scale. For multi-turn generation, Section 4.2 reports that all three tested models already fail completely at \(k=3\); this result is not directly comparable to those human-rating averages. That section also states both "50 cases" and four horizons with 10 cases each, an inconsistency that this note does not silently resolve into a definite total.
Key Findings¶
- GPT-5's exact-match and set-match curves nearly coincide, supporting failure at step selection rather than only an inability to order otherwise correct candidates.
- Some proprietary models retain an overlap ratio of approximately 65% at \(k=7/8\) even when complete success reaches zero; complete success and partial identification must be interpreted separately.
- On Height, 22/29 models fall below the critical threshold for exceeding chance. The authors observe that models often answer using 2D projected height rather than true 3D height.
Highlights & Insights¶
- Shared assembly scenarios support both local questions and long-horizon planning, allowing score degradation to be connected to specific capabilities. Set match is particularly useful for separating selection errors from ordering errors.
- Multi-solution topological evaluation recognizes that valid assembly orders need not be unique, instead of accepting only one instruction sequence. It still checks whether the entire trajectory is valid, rather than whether individual images merely look plausible.
- Separating appearance from instruction following exposes the gap between visual resemblance and correct assembly. This diagnostic approach can transfer to visual-assistant evaluations requiring precise spatial edits.
Limitations & Future Work¶
- The authors primarily document model limitations: unstable elementary 3D judgments, multi-step state integration, and spatially precise generation; the paper does not separately develop a systematic benchmark-limitations section.
- This note's assessment: rendered LEGO scenes lack contact errors, changing occlusion, and execution feedback found in real manipulation, so their results do not directly establish real robot success rates.
- This note's assessment: each VQA planning horizon contains only 20 cases, and human baselines use experts rather than general users; the results are diagnostic, but fine-grained rankings need larger samples and uncertainty reporting.
- This note's assessment: MSE matching may depend on rendering style and image alignment; future evaluation should incorporate structural or part-level verification and report agreement with human judgments.
- The source contains the score and sample-count inconsistencies noted above, and the supplied full text does not contain the appendix scoring guidelines referenced by the paper; reproduction should resolve these issues rather than infer implementation details from missing material.
Related Work & Insights¶
- vs CLEVR / Super-CLEVR: these benchmarks provide controlled compositional visual reasoning scenes; LEGO-Puzzles emphasizes consecutive assembly states of the same object and planning horizon rather than static-scene questions alone.
- vs 3DSRBench: the latter evaluates 3D spatial judgments using natural images. Table 3 of this paper reports model-performance correlations of 0.93 and 0.98 for corresponding height and adjacency tasks, but these correlations do not establish transfer of multi-step planning to real-world execution.
- Implications for future methods: explicitly maintaining part states and validating candidate legality before ordering are directions suggested by the failure analysis; this paper does not train or validate such a solution.
Rating¶
- Novelty: 4/5. LEGO itself is not a new setting, but the combination of hierarchical diagnosis, distractor-based planning, and generation evaluation has a clear purpose.
- Experimental Thoroughness: 4/5. The study covers 29 models, humans, and two generation analyses, although long-horizon samples are limited and some statistical descriptions need clarification.
- Writing Quality: 3/5. Task definitions are generally clear, but conflicting prose/table scores and generation sample counts complicate precise citation.
- Value: 4/5. Useful for identifying specific spatial-reasoning failures, but not a complete substitute for evaluating real-world assembly competence.