ZAP: Zero-Shot Assembly Planning with Large Language Models¶

Conference: ECCV 2026
Paper: ECCV paper page
Code: https://github.com/ZJU-PLP/ZAP
Area: Robotics & Embodied AI / Assembly Planning
Keywords: zero-shot assembly, multi-view perception, structured part dossiers, digital-twin verification, backtracking search

TL;DR¶

ZAP first infers a structured "implicit assembly manual" from multi-view part images, then has a vision-language model (VLM) propose the next assembly action and uses digital-twin verification with backtracking search to correct mistakes, reaching an 88.67% planning success rate on 150 ASAP-G assembly tasks — a number that measures the simulation-verified pass rate, not a real-robot success rate at the same scale.

Background & Motivation¶

Robotic assembly must not only recognize which parts lie on the table; it must also understand how the parts connect and which one to attach first so that later operations are not blocked. Classical geometric planners can search collision-free orders given accurate CAD models, but model acquisition and real perception error limit deployment; reinforcement learning (RL) and learning from demonstration can learn the actions themselves, yet they typically demand large amounts of task-specific interaction. When new parts and new connection topologies appear, previously learned experience does not necessarily transfer.

Vision-language models offer a different prior: shown a base, a support rod, and a wheel axle, a human can infer their rough assembly relations from functional common sense, and a VLM may share a similar ability. Systems such as Manual2Skill, however, still take a human-written manual as their entry point; given only the parts and a goal instruction, the system must first infer the "manual" by itself. Directly asking a VLM to output the complete order from images in one shot is also unreliable, because connections that look plausible may collide, and a part that fits right now may seal off the access needed by later assembly steps.

ZAP therefore places the problem at the intersection of perception, semantic priors, and geometric verification, rather than claiming that language reasoning can replace physical constraints. Core idea: use multi-view vision to produce assembly-oriented structured part dossiers, let the VLM narrow the candidate order space, and then use stepwise digital-twin feasibility verification with backtracking to turn "looks like it fits" into "fits under the verifier's constraints".

Method¶

Overall Architecture¶

The input is a set of loose parts placed in a controlled preparation region with sufficient spacing between them, together with an assembly goal given in language. Three extrinsically calibrated RGB-D cameras observe from approximately front, side, and top viewpoints, and automatic instance segmentation with cropping yields a three-view image set for each part. Segmentation here is only preprocessing to obtain part-level inputs; what the paper actually studies is robot assembly sequence planning, not a segmentation algorithm.

The method consists of three key designs in sequence: multi-view structured parsing, where the VLM-Parser produces part dossiers and a relational summary; physics-semantic heuristic proposal, where the VLM-Planner judges the next step from the current subassembly and the remaining parts; and digital-twin verification and backtracking, organized by the Iterative Decision Aggregator (IDA), which checks candidate connections, updates the state, and retracts earlier decisions when a dead end is reached. The final output is a complete order that starts from a base part and adds the remaining parts one at a time, which is then handed to the robot execution system.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Three-view part images<br/>and assembly goal"] --> B["Multi-view structured parsing"]
    B --> C["Physics-semantic heuristic proposal"]
    C --> D["Digital-twin verification and backtracking"]
    D -->|verified, update state and continue| C
    D -->|dead end, backtrack to another branch| C
    D -->|all parts assembled| E["Verified assembly sequence<br/>and robot execution"]

It is important to separate "the planner's inputs" from "the whole system's dependencies": precise object meshes are not used as the VLM's primary planning input, but the digital twin still uses meshes to instantiate the simulation and to check collisions and connection alignment. ZAP therefore reduces the planning stage's direct dependence on human manuals and explicit geometric priors, not to a fully geometry-free end-to-end assembly pipeline. Depth maps are mainly reserved for grasp-pose recovery at execution time, so this paper should not be read as feeding dense depth directly into the VLM for full 3D dynamics reasoning.

Key Designs¶

1. Multi-view structured parsing: compressing images into assembly-ready part dossiers

A part's name alone cannot determine how it connects, while the full image mixes in many texture details that do not affect assembly. The VLM-Parser therefore adopts an assembly-oriented sparse structural encoding: exploiting the orthogonal-projection prior of manufactured parts, it infers the shape envelope from the front, side, and top views and localizes mating features at the six primary axial extremities, instead of asking the model to freely imagine the entire 3D shape from a single image. The three views constrain one another — for example, a region that looks like a solid cylinder in one view may reveal a central hole in another view, changing whether it can mate with an axle.

Each structured part dossier carries three groups of fields. sem_sum stores auxiliary attributes such as name, color, and material, plus the part's functional role relative to the global goal; phys_prop stores geometric quantities obtained from cross-view consistency and calibrated tools, such as axis-aligned bounding volumes and 3D centroids; interfaces stores mating features such as concentric holes and axial plugs together with their axial positions, tying the qualitative features seen in the images to metric dimensions checked programmatically. The first group relies on the VLM's semantic understanding, while the other two are constrained by deterministic geometric tools, forming a representation in which a semantic stream and a geometric stream cooperate. The "implicit manual" is the functional and relational description inferred from observation, and the structured dossier is its machine-readable version — it is not a pre-provided correct assembly order.

2. Physics-semantic heuristic proposal: weigh both whether the part stands now and whether later parts can still get in

The planner does not write out all steps at once; it runs a short "mental rehearsal" on the current subassembly. It reads the part dossiers and keeps direct visual evidence as a supplement, proposing candidate mobile parts from those not yet assembled. Chain-of-thought (CoT) reasoning externalizes the physical rationale for each candidate, which is then evaluated heuristically along three aspects: structural stability concerns the visually estimated center of mass relative to the support base; functional affordance concerns whether the part fits the overall assembly goal; assembly precedence concerns whether the current move would harm the geometric accessibility of later parts. The paper gives no explicit weights or full numeric implementation for these scoring items, so no fixed weighted-sum formula should be invented on this basis.

The value of this step is not replacing collision checking but shrinking the combinatorial space that expensive verification has to face. For example, a base should usually provide support first, and an outer cover, even when it can be installed now, may block the entry for internal parts; such functional and ordering common sense turns "try every remaining part once" into prioritizing a few plausible candidates. Images and dossiers are also not an either-or choice: dossiers make functional, interface, and scale constraints easy to retrieve, while the raw images supply fine geometry that a textual summary may miss. The modality ablation later specifically compares the two degraded modes of keeping dossiers without images and keeping images without dossiers.

3. Digital-twin verification and backtracking: let semantic proposals be wrong without baking local errors into later state

IDA places the heuristic proposals into a recursive "propose–verify–update" loop. Every high-confidence candidate must pass the digital twin's physical feasibility verification: meshes are used for discrete collision detection, joint alignment, and connection-consistency checks, and only connections satisfying kinematic and topological constraints enter the assembly sequence. After verification passes, the system adds the mobile part to the current subassembly and records both the sequence and the state history; if no remaining part can pass verification, it reverts to the previous stable subassembly state and re-evaluates other branches. The search ends either by covering all parts or by returning failure when no further backtracking is possible.

The state machine in the paper's Algorithm 1 is clear: start from a selected base part, and maintain the current subassembly, the remaining-part set, the accepted action sequence, and a history log. For K parts in total, completing the assembly takes K−1 transitions of "add one mobile part to the subassembly". Validity requires both a collision-free motion path at every step and a terminal state containing all initial parts; therefore a sequence whose final shape looks close to the target but whose intermediate insertions are infeasible still does not count as success. IDA here stands for Iterative Decision Aggregator — it should not be equated with the classical IDA search with explicit cost bounds and provable optimality merely because the label "IDA" appears in the figure.

A Worked Example¶

Take the toy car on a pallet from Figure 2 of the paper. The system first observes the car body, the wheel axles, the roll bar, and the assembly pallet. The parser records the body as a main structure with a seat cavity, the axle as a rolling component with a cylindrical axle mount, the bent roll bar as a support providing structural protection, and the pallet as a reference base with locating features. These descriptions answer "what it is and where it connects" — they are not yet a fixed order.

The planner then proposes the next part on the current base, and the state is updated only after the verifier confirms that its insertion and connection are feasible. The completion process shown in the original figure follows the order "base with body → add wheels → add frame". If a step, while locally valid, leaves the remaining parts unable to enter, IDA retreats along the history to earlier states and tries other branches; what is described here is the algorithm's mechanism for handling dead ends, not how many backtracks actually occurred in this example — the paper provides no full search log for this execution trajectory.

Loss & Training¶

ZAP defines no new training loss and is not fine-tuned on ASAP-G. Its zero-shot protocol has three boundaries: no ASAP-G samples are used for prompt tuning or model alignment; the agent receives no ground-truth assembly graphs or target connectivity matrices; and all test samples use a frozen backbone, a fixed prompt template, and a fixed sampling temperature. The paper only states that the Gemini and GPT model families are used at temperature 0.1, without specifying concrete model versions or per-module combinations precise enough for reproduction, and low-temperature sampling does not make the calls strictly deterministic under all conditions.

ASAP-G is the evaluation set, not ZAP's training set. It selects 150 assemblies from ASAP by stratified sampling over topology and category, each with 3–30 parts; the simple, medium, and hard tiers are in a 7:5:3 ratio, corresponding to 3–5, 6–10, and 11–30 parts. Multi-view images are generated by local rendering, and part semantic annotations are first drafted by a VLM and then checked by researchers, with about 6.5% of part annotations requiring correction, typically miscounts of symmetric or repeated structures. The meshes also pass watertightness checks and initial inter-part collision cleanup. What humans revise is the benchmark annotation, which should not be miswritten as a human supplying the assembly answer in every robot planning run; at the same time, the paper does not explain in sufficient detail the boundary between the human-revised dossiers used for evaluation and the online parsing results.

Key Experimental Results¶

Main Results¶

The primary metric is the Planning Success Rate (PSR): the fraction of generated sequences that pass the digital-twin feasibility verification and produce the target assembly topology. When multiple legal orders exist, any order satisfying the feasibility and connectivity requirements counts as success. The numbers below come from Table 1 of the paper; evaluation covers all 150 ASAP-G assemblies, all units are percentages, and incremental comparisons use percentage points.

Method	Simple: 3–5 parts	Medium: 6–10 parts	Hard: 11–30 parts	Overall PSR
ASAP (original table merges Heuristics / Learning)	90.00	86.00	76.67	86.00
Random Permutation + physical verification	88.57	84.00	76.67	84.67
ZAP	90.00	88.00	86.67	88.67

ZAP's overall advantage over ASAP is 2.67 percentage points, and 10.00 percentage points on the hard tier, while the simple tier is exactly tied. By category, the toy class rises from ASAP's 76.19% to ZAP's 95.24%, a gain of 19.05 percentage points; however, furniture drops from 87.14% to 84.29%, and others drop from 92.86% to 89.29%. The advantage is therefore not an across-the-board lead but is concentrated in hard assemblies and non-standard geometric combinations like toys.

This comparison is also not a replacement experiment under identical inputs: the ASAP baseline can access full geometric priors, and ASAP-G is drawn from the original ASAP corpus; ZAP's VLM does not directly read ground-truth connectivity graphs or exact meshes as primary planning inputs, but its verifier still uses meshes. Manual2Skill does not appear in this quantitative main table, so the paper cannot be said to numerically prove superiority over it under the same experimental protocol.

The real-robot part uses a UR5e arm, a Robotiq-85 gripper, a custom 3D-printed assembly fixture, and three RealSense D435 cameras, matching the MuJoCo 3.3.4 digital twin. Each of a desk lamp, a stool, and a toy car gets 10 independent trials, with success rates of 100%, 90%, and 80% respectively — i.e., 10/10, 9/10, and 8/10. The authors explicitly position these as a pilot study of end-to-end feasibility, not a real-robot evaluation covering the 150 assemblies.

Ablation Study¶

The table below merges the overall results of Table 2 with the difficulty-tier results of Table 3; overall deltas use Table 2's full system as the reference, in percentage points. The version without IDA directly accepts the top-ranked proposal — a greedy open-loop policy that loses both the recursive propose–verify loop and backtracking, not just an isolated backtracking switch.

Config	Overall PSR (Table 2)	Overall Δ	Simple (Table 3)	Medium (Table 3)	Hard (Table 3)
Full ZAP	88.67	—	90.00	88.00	86.67
w/o IDA: greedy policy	74.00	−14.67	81.42	68.00	63.33
w/o image input, dossiers kept	86.00	−2.67	88.57	86.00	80.00
w/o structured dossiers, images kept	86.67	−2.00	88.57	84.00	83.33

The paper's numbers contain a summary inconsistency; the original table values are kept above without silent correction. Weighting Table 3 by the paper's 7:5:3 difficulty ratio, the w/o-IDA overall value is about 73.33%, not Table 2's 74.00%; w/o dossiers gives about 86.00%, not 86.67%. Both gaps are about 0.67 percentage points and cannot be explained by two-decimal rounding alone. The full system and the w/o-image version do align under the same computation; the paper does not explain whether different run batches or statistical accounting exist, and the exact summary values should keep this open question.

Key Findings¶

The clearest gain comes from closed-loop search. Removing IDA drops the reported overall PSR by 14.67 percentage points; the hard tier falls from 86.67% to 63.33%, a drop of 23.34 percentage points, and the simple tier falls by 8.58 percentage points. The longer the assembly sequence, the more it needs early errors to be discovered and retracted, rather than treating a single VLM judgment as an immutable decision.
Structured dossiers cannot fully replace images. Removing images costs 6.67 percentage points on the hard tier; removing dossiers costs 4.00 percentage points on the medium tier. The former suggests language summaries lose fine-grained geometry, the latter that reasoning from pixels alone easily misses functional and ordering constraints.
The verifier itself provides a strong floor. Random permutation with physical verification already reaches 84.67% overall PSR, 4.00 percentage points below ZAP. The authors state that both use the same verification budget, but the paper reports no budget numbers, search counts, or latency, so computational-efficiency gains cannot be asserted from success rates alone.

Highlights & Insights¶

Separating "understanding the parts" from "proving the action feasible". The VLM forms useful candidate rankings from function and appearance, while deterministic tools reject connections that do not hold. This fits the competence boundary of foundation models better than demanding they directly output trustworthy robot actions.
The implicit manual is a constrained intermediate representation. It is easier to search over than raw pixels yet is not a human-provided answer; the six-axis interfaces and cross-view geometric constraints tie semantic inference to the structure assembly actually needs, rather than producing just a passage of natural language.
Backtracking matters more than one-shot long-chain reasoning. In the experiments, the degradation from removing closed-loop search far exceeds that of removing either input modality alone, showing that reliable assembly depends not only on getting the first thought right but also on whether the system can recognize wrong states and recover from them.

Limitations & Future Work¶

Still depends on an available digital twin. Meshes not being handed to the VLM does not mean the verifier does not need them; for completely unknown real parts that are hard to model, obtaining a sufficiently accurate verification environment remains a deployment prerequisite.
Perception conditions are relatively controlled. The initial parts need sufficient spacing, and the three viewpoints are calibrated. Self-occlusion, small interfaces, and symmetric structures still cause misparsing, and the occlusion-avoiding initial-segmentation setup should not be generalized to arbitrarily cluttered pile scenes.
Mechanical realism is still limited. The authors note that connection points in the benchmark are not always distinct and many parts lack fine fastening mechanisms and friction models. Passing geometric verification does not mean stable execution under tight tolerances, real contact forces, and flexible deformation; force-feedback control and contact-rich manipulation are the follow-up directions the authors propose.
Reproduction and statistical reporting have gaps. Specific VLM versions, the heuristic scoring implementation, the search budget, and run costs are not sufficiently given; part of the summary values across the two ablation tables are inconsistent, and the main experiment provides no confidence intervals. Each real-robot object has only 10 trials, whose percentages cannot prove broad statistical robustness.
The denominator of the failure analysis is unclear. The paper reports that 12.5% of complex cases misidentify symmetric interfaces and that backtracking recovers 60% of the cascading errors among them, but it does not clearly list the corresponding sample sizes. These numbers describe the observed failure types and cannot be merged with the 30 real-robot trials to derive an overall recovery rate.

vs ASAP: ASAP plans from exact geometric priors, while ZAP uses visual parsing and functional common sense to propose candidates and then bounds them with geometric verification. The difference lies mainly in candidate generation and knowledge sources, not in one side using physical constraints while the other uses none.
vs Manual2Skill: Manual2Skill converts existing manuals into executable skills, while ZAP targets structural inference when no pre-written manual exists. The former solves "how to follow the manual"; the latter first solves "how to infer the manual from the parts". The paper provides no quantitative head-to-head under a unified input protocol.
vs direct VLM greedy planning: Greedy methods accept only the current top choice and have no explicit mechanism for states where the current action is feasible but no path remains afterward. ZAP puts physical feedback and historical state back into the search loop, so errors need not propagate all the way to final execution.
Reusable insight: In embodied tasks that already have a reliable checker, one can let the model generate compact semantic candidates, let the checker decide whether to commit a state, and keep a retractable history. This is the design lesson this paper supports, not a validated new research method.

Rating¶

Novelty: ⭐⭐⭐⭐ (4/5). Combining multi-view implicit manuals, semantic candidate generation, and verification with backtracking forms a complete problem setting for manual-free assembly, but the core remains a systematic integration of existing models and search tools.
Experimental Thoroughness: ⭐⭐⭐ (3/5). There are 150 tasks, difficulty and modality ablations, and real-robot trials, but the inconsistent ablation summaries, missing reproduction details, and small-scale real-robot statistics limit the strength of the conclusions.
Writing Quality: ⭐⭐⭐ (3/5). Module responsibilities and method motivations are fairly clear, yet the title-version discrepancy, the algorithm naming, and some numeric accounting need further clarification from the authors.
Value: ⭐⭐⭐⭐ (4/5). It clearly shows how to constrain foundation-model planning with a physical checker, and suits readers interested in zero-shot assembly, embodied reasoning, and model–tool collaboration.