Skip to content

IterCAD: An Iterative Multimodal Agent for Visually-Grounded CAD Generation and Editing

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KnowledgeXLab/IterCAD
Area: Multimodal VLM
Keywords: Multimodal Agent, Parametric CAD Generation, Closed-Loop Self-Correction, Reinforcement Learning, Survivor Bias Elimination

TL;DR

IterCAD reframes parametric CAD synthesis and editing as a multi-turn closed-loop interaction between an MLLM agent and an executable CAD sandbox, combining progressive cold-start SFT, geometry-aware RL (GSPO) with Geometry-Viable Prefix Masking (GVPM), and a survivor-bias-free CD-TR metric to achieve superior code executability and geometric precision across complex industrial drawing and editing tasks.

Background & Motivation

Computer-Aided Design (CAD) forms the foundational bedrock of modern mechanical engineering and industrial manufacturing, yet manually constructing high-quality, fully parametric 3D models remains an intensely labor-consuming task. Recent advances in Multimodal Large Language Models (MLLMs), combined with programmatic CAD frameworks such as CadQuery and build123d, have catalyzed a promising paradigm: automatically synthesizing executable CAD code directly from natural language specifications, images, and multi-view engineering drawings. Although pioneering programmatic methods such as CAD-Coder and CADrille demonstrate the immense flexibility of programmatic geometry representation, existing automated CAD pipelines predominantly adhere to an "open-loop, one-shot" generative regime. This paradigm produces complete code scripts in a single forward pass without intermediate verification, directly clashing with the standard human engineering reality characterized by iterative "generate–verify–refine" cycles.

This structural disconnect leads to profound vulnerabilities during real-world CAD synthesis and evaluation. When previous multi-turn agentic frameworks attempted to introduce iterative correction loops, their supervisory feedback remained critically shallow and unanchored: compiler diagnostics merely register syntactic errors while staying oblivious to distorted geometric semantics, and point-cloud metrics (such as Chamfer Distance) yield only unstructured global shape errors. Crucially, global spatial discrepancies fail to localize failures or map them to specific 2D sketch dimensions, feature extrusions, or topological parameters, causing iterative revisions to degenerate into an unguided, blind trial-and-error search. Compounding this challenge, widely used public CAD benchmarks are heavily biased toward simplistic "sketch-and-extrude" primitives, completely omitting critical finishing operations such as fillets, chamfers, shells, and through-cuts. Most critically, conventional CAD evaluation protocols suffer from severe "survivor bias" by computing geometric fidelity metrics exclusively over successfully executed programs; non-executable scripts (\(CD \to \infty\)) are conveniently omitted from averages, resulting in a fundamentally warped assessment of an agent's true robustness and recovery capabilities.

To resolve these interconnected bottlenecks across interaction mechanisms and diagnostic benchmarks, the authors establish a drawing-driven interactive CAD agent paradigm grounded in physical engineering anchors. They introduce IterCAD, an agent framework governed by a "Look and Loop" philosophy: "Look" leverages standardized multi-view engineering drawings with explicit dimensional constraints as persistent spatial references to isolate defects and guarantee metric fidelity, while "Loop" integrates compiler diagnostics, geometric execution outputs, and rendered orthographic visual feedback into an autonomous multi-turn repair loop. Core idea: reframe parametric CAD modeling as a closed-loop multi-turn interaction between a multimodal agent and an executable sandbox, combining progressive cold-start SFT with geometry-aware RL enhanced by Geometry-Viable Prefix Masking (GVPM) to achieve robust self-correction anchored on dimensioned engineering drawings.

Method

Overall Architecture

IterCAD unifies CAD modeling and editing into a discrete multi-turn dialogue game between a multimodal agent policy \(\pi_\theta\) and an executable CAD execution sandbox. The system encompasses three unified design settings: 1) Drawing-to-Code, which reverse-engineers CadQuery code conditioned on a dimensioned multi-view engineering drawing \(V_{\text{ref}}\); 2) Text-to-Code, which synthesizes geometry based on natural language engineering requirements \(T_{\text{ref}}\); and 3) Interactive Editing, which applies targeted incremental modifications to an imperfect source script \(C_{\text{src}}\).

At each turn \(t \le T\) (with a maximum budget of \(T=5\) turns), the agent observes the cumulative interaction history \(O_t = \{S, X_{\text{ref}}, (C_1, F_1), \ldots, (C_{t-1}, F_{t-1})\}\), where \(S\) denotes the system prompt and \(X_{\text{ref}}\) represents the task reference context. The agent must strictly produce a structured action \(a_t\) consisting of two components: a reasoning trace wrapped in <thinking> tags performing multimodal defect diagnosis and planning, followed by an execution block containing either a fenced CadQuery code variant \(C_t\) or an explicit <DONE> termination token. The sandbox executes \(C_t\) via the OpenCASCADE (OCCT) geometric engine, catching runtime exceptions and programmatically projecting successfully generated 3D solids into standard orthographic views with explicit linear and radial dimensional annotations as visual feedback \(F_t\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Design Specification<br/>Dimensioned Drawings / Text Specs / Source Code"] --> B["Industrial Advanced Feature Data Synthesis<br/>Pbase ∪ Padv feature space & SolidWorks COM drawing automation"]
    B --> C["Progressive Cold-Start SFT<br/>DS1 expert trajectories + DS2 on-policy error correction"]
    C --> D["Geometry-Aware Policy Optimization GSPO<br/>Group sequence-level policy optimization with relative advantage"]
    D --> E["Geometry-Viable Prefix Masking GVPM<br/>Execution cascade & geometry stall truncation with one-sided clamping"]
    E --> F["Closed-Loop Sandbox Feedback Self-Correction<br/>OCCT orthographic visual feedback & compiler diagnostic logging"]
    F -->|Turn t < T & not converged| C
    F -->|Done token / final code| G["Survivor-Bias-Free Evaluation<br/>CD-TR curve & AUC-TR metric under engineering tolerances"]

Key Designs

1. Industrial Advanced Feature Data Synthesis: Expanding Operation Spaces and SolidWorks COM Automation

Standard CAD datasets are severely dominated by basic 2D planar profiles followed by single extrusions, neglecting essential finishing features and topological operations indispensable in realistic mechanical fabrication. To bridge this gap, IterCAD formally constructs an expanded parametric operation space \(\mathcal{P} = \mathcal{P}_{\text{base}} \cup \mathcal{P}_{\text{adv}}\). Here, \(\mathcal{P}_{\text{base}}\) encompasses foundational sketch entities and basic extrusions, while \(\mathcal{P}_{\text{adv}}\) integrates professional manufacturing features including fillets, chamfers, shells, through-cuts (cutThruAll), and pattern arrays. Complex solid geometries are synthesized by randomly sampling an operation subset \(\mathcal{P}_{\text{sub}} \subset \mathcal{P}\) as strict execution constraints for an expert LLM, followed by headless sandbox verification to strictly purge non-manifold boundaries or self-intersecting topologies.

Crucially, to establish reliable visual supervision rather than abstract rendered images, the framework builds an automated drafting pipeline via the SolidWorks Component Object Model (COM) interface. This pipeline programmatically translates valid 3D STEP solids into standard-compliant multi-view engineering drawings \(V_{\text{ref}}\). These drawings integrate standard first-angle/third-angle orthographic projections alongside explicit dimensional annotations (linear extents, hole center offsets, and arc radii) extracted directly from the underlying solid topology, establishing rigid metric coordinate anchors for subsequent multimodal reasoning.

2. Progressive Cold-Start SFT: Expert Demonstrations and On-Policy Refinement Curriculum

Directly deploying raw base LLMs in an interactive CAD sandbox results in frequent format breakdowns, unsupported API invocations, and inability to parse compiler stack traces. The authors introduce a two-stage progressive supervised fine-tuning curriculum. The first stage trains on an expert trajectory corpus \(\mathcal{D}_{S1}\) (20K samples) generated by Qwen3-VL-235B-A22B-Instruct, enforcing three strict filtering criteria: format compliance (mandating strict alternation between <thinking> and executable code), logic coherence (prompting an LLM to prune trajectories whose internal reasoning contradicts executed code or sandbox feedback), and geometric correctness (requiring final scripts to execute successfully with Chamfer Distance \(CD < 10^{-5}\)).

The second stage explicitly teaches iterative debugging by collecting an on-policy refinement trajectory corpus \(\mathcal{D}_{S2}\) (8K samples). By executing the intermediate policy \(\pi_\theta\) (trained on \(\mathcal{D}_{S1}\)) within the sandbox, typical runtime exceptions, invalid parameter assignments, and geometric deviations are recorded. A superior teacher model then synthesizes localized corrective actions conditioned on the error trace and rendered views. The unified SFT loss is optimized via standard autoregressive cross-entropy:

\[\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(x_i, y_i) \sim (\mathcal{D}_{S1} \cup \mathcal{D}_{S2})} \left[ \log \pi_\theta(y_i \mid x_i) \right]\]

3. Geometry-Aware Policy Optimization GSPO: Group Sequence Optimization and Composite Reward Design

To transition from pure behavioral cloning to autonomous exploration, IterCAD adopts Group Sequence Policy Optimization (GSPO). Given design specification \(x\), the model samples a group of \(G=8\) trajectory candidates \(\{y_g\}_{g=1}^G\) from the reference policy \(\pi_{\theta_{\text{old}}}\). The sequence-level relative advantage \(\hat{A}_g\) is standardized across the sampled group:

\[\hat{A}_g = \frac{r_g - \text{mean}(\{r_{g'}\}_{g'=1}^G)}{\text{std}(\{r_{g'}\}_{g'=1}^G)}\]

Optimization maximizes a clipped surrogate objective with asymmetric clipping thresholds \(\epsilon_l < \epsilon_h\) and a length-normalized importance ratio \(\rho_g(\theta)\), preventing lengthy multi-turn rollouts from skewing gradient updates:

\[J(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \frac{1}{G} \sum_{g=1}^G \min\left( \rho_g(\theta)\hat{A}_g, \; \text{clip}(\rho_g(\theta), 1-\epsilon_l, 1+\epsilon_h)\hat{A}_g \right) \right]\]

The sequence-level scalar reward combines geometric fidelity, format validity, and multi-turn progress: \(R(y) = R_{\text{CD}}(y) + \lambda_f R_{\text{fmt}}(y) + \lambda_p R_{\text{prog}}(y)\) (with \(\lambda_f=0.5, \lambda_p=0.2\)). While \(R_{\text{fmt}} \in \{0, 1\}\) penalizes formatting violations, the progress bonus \(R_{\text{prog}}\) rewards trajectories that span at least two turns, start with a valid geometry, and exhibit monotonically decreasing Chamfer Distances converging below threshold \(\eta\). The geometric reward \(R_{\text{CD}}\) smoothly scales point-cloud distance via a piecewise linear mapping:

\[R_{\text{CD}}(y) = \begin{cases} 1, & CD \le \delta_{\min} \\ \frac{\delta_{\max} - CD}{\delta_{\max} - \delta_{\min}}, & \delta_{\min} < CD < \delta_{\max} \\ 0, & CD \ge \delta_{\max} \text{ or Execution Fail} \end{cases}\]

4. Geometry-Viable Prefix Masking GVPM: Resolving Credit Assignment in Multi-Turn Interactive CAD

A chronic failure mode in interactive code generation RL is the credit assignment dilemma across turns: once an agent introduces a fatal bug or hits an unrecoverable geometric plateau, all subsequent turns inherit an irrecoverable context. Standard sequence-level RL propagates the negative advantage \(\hat{A}_g\) uniformly across all tokens, wrongly penalizing early valid decisions that established correct foundational geometry.

Geometry-Viable Prefix Masking (GVPM) resolves this by dynamically identifying a viable prefix cut-off boundary \(f = \min(f_{\text{exec}}, f_{\text{stall}})\). Specifically, \(f_{\text{exec}}\) flags an execution cascade whenever \(K=2\) consecutive runtime errors occur, while \(f_{\text{stall}}\) detects a geometry stall when the Chamfer Distance across \(K=2\) valid turns ceases to strictly decrease while remaining above quality gate \(\eta\). For tokens beyond turn \(f\), loss masking is enforced via \(m_u = \mathbb{I}(\tau(u) < f)\), and masked trajectories receive a one-sided advantage clamp:

\[\hat{A}_g \leftarrow \max(\hat{A}_g, 0)\]

This one-sided clamping prevents productive foundational operations in early turns from being suppressed by downstream degradation, encouraging persistent multi-turn exploration.

Loss & Training

The entire agent is fine-tuned from the open-source multimodal backbone Qwen3.5-4B using the Swift framework across 8 NVIDIA A100 GPUs. The cold-start SFT stage optimizes \(\mathcal{D}_{S1} \cup \mathcal{D}_{S2}\) (28K trajectories) over 3 epochs with a learning rate of \(1 \times 10^{-5}\). The subsequent GSPO reinforcement learning stage sets group size \(G=8\), learning rate \(1 \times 10^{-6}\), maximum interaction budget \(T=5\) turns, and prefix trigger window \(K=2\). RL focuses on 2K challenging Drawing-to-Code instances, providing maximal gradient signal for closed-loop multi-view visual self-correction.

Key Experimental Results

Main Results

Evaluation spans the newly established IterCAD-Bench (1,000 drawing-to-code tasks in IterCAD-Draw and 200 local editing tasks in IterCAD-Edit), alongside public text-to-CAD benchmarks Text2CAD (8,046 samples) and CADPrompt (200 expert samples). Metrics include Invalid Ratio (IR %, lower is better), Mean/Median Chamfer Distance (Mean CD / Med. CD, scaled by \(10^3\), lower is better), Average Interaction Turns (Avg. Turn, lower is better), and Area Under the CD-TR Curve (AUC-TR, higher is better), which explicitly penalizes failed executions with zero recall to eliminate survivor bias.

The table below summarizes performance on the IterCAD-Draw benchmark under Direct Inference and Agentic Workflow:

Model Workflow Setting IR (%) ↓ AUC-TR ↑ Mean CD (\(\times 10^3\)) ↓ Med. CD (\(\times 10^3\)) ↓ Avg. Turn ↓
Proprietary Models
GPT-5 Direct Inference 28.10 0.41 12.97 0.13 1.00
GPT-5 Agentic Workflow 5.30 0.51 8.68 0.46 3.35
Gemini-2.5-flash-lite Direct Inference 60.00 0.24 13.51 0.08 1.00
Gemini-2.5-flash-lite Agentic Workflow 38.90 0.28 16.09 0.82 4.49
Gemini-3-flash-preview Direct Inference 50.60 0.37 6.47 0.06 1.00
Gemini-3-flash-preview Agentic Workflow 12.60 0.55 7.03 0.10 4.31
Open-Source Baselines
GLM-4.6v Direct Inference 27.50 0.36 14.88 1.90 1.00
GLM-4.6v Agentic Workflow 8.50 0.46 9.83 1.26 4.07
InternVL3.5-30B-A3B Direct Inference 89.60 0.02 25.36 18.33 1.00
InternVL3.5-30B-A3B Agentic Workflow 80.50 0.04 27.98 20.45 4.80
Qwen3.5-35B-A3B Direct Inference 68.80 0.24 7.39 0.06 1.00
Qwen3.5-35B-A3B Agentic Workflow 20.60 0.42 11.40 0.30 3.97
Qwen3.5-4B (Backbone) Direct Inference 95.30 0.03 9.22 0.07 1.00
Qwen3.5-4B (Backbone) Agentic Workflow 64.60 0.24 10.48 0.07 4.56
IterCAD (4B, Ours) Direct Inference 6.50 0.54 9.75 0.30 1.00
IterCAD (4B, Ours) Agentic Workflow 0.30 0.61 5.09 0.35 2.97

On the instruction-guided editing track (IterCAD-Edit), IterCAD achieves 1.00% IR, 0.54 AUC-TR, and a swift 2.34 average turns, outperforming its 4B backbone by 62.00% IR and 2.15 turns, closely matching frontier proprietary leader GPT-5 (0.50% IR, 0.79 AUC-TR, 2.31 turns). On external zero-shot text-to-CAD benchmark CADPrompt, IterCAD achieves 2.00% IR, 10.45 Mean CD, and 2.42 Med. CD, surpassing previous state-of-the-art CAD-Judge (4.50% IR, 152.40 Mean CD) by more than an order of magnitude.

Ablation Study

The table below isolates the contributions of each training component on the IterCAD-Draw benchmark:

Variant \(\mathcal{D}_{S1}\) \(\mathcal{D}_{S2}\) GSPO GVPM IR (%) ↓ AUC-TR ↑ Mean CD (\(\times 10^3\)) ↓ Avg. Turn ↓ Note
Qwen3.5-4B (Base) – – – – 64.60 0.24 10.48 4.56 Zero-shot base model; high failure rate
IterCAD-SFT1 ✓ – – – 7.50 0.52 10.55 2.23 Expert demonstration SFT; establishes syntax
IterCAD-SFT2 ✓ ✓ – – 0.80 0.52 12.44 3.08 On-policy refinement SFT; drives IR below 1%
IterCAD-GSPO ✓ ✓ ✓ – 1.30 0.58 8.00 2.51 Standard GSPO RL; improves CD but terminates prematurely
IterCAD-Full ✓ ✓ ✓ ✓ 0.30 0.61 5.09 2.97 Full framework; resolves credit assignment, best CD & AUC-TR

Key Findings

  • GVPM eliminates premature termination in RL: As shown in ablation dynamics (paper Fig. 8), optimizing with GSPO alone leads the model to quickly collapse into a single-pass shortcut (average turns dropping to 2.51) to avoid punishment from subsequent turn failures. GVPM shields early productive prefixes, stabilizing interaction length at ~3 turns and allowing the model to perform deeper geometric self-correction, dropping Mean CD from 8.00 to 5.09.
  • Visual feedback is an indispensable physical anchor: The visual-observation ablation (paper Tab. 7) reveals that withholding rendered multi-view orthographic images during inference or RL rollouts degrades Mean CD from 5.09 to 6.62 and 8.42, respectively. Textual stack traces alone are blind to spatial distortions, confirming that rendered dimensional projections provide critical spatial calibration signals.
  • Error modes concentrate in complex contours and Boolean unions: Thorough failure analysis across all 500 Hard-subset cases (with a 17.2% overall failure rate under \(CD > 0.01\)) reveals that manufacturing finishing operations are not the failure drivers: chamfers achieve a 0% failure rate, and shells/fillets exhibit low failure rates of 4.2% and 5.3%. Instead, ~30% of failures stem from contour approximation errors in intricate multi-segment sketches (lineTo and threePointArc), and ~37% stem from non-manifold topology crashes during multi-body Boolean union operations.

Highlights & Insights

  • Chamfer Distance Tolerance-Recall (CD-TR) establishes a survivor-bias-free evaluation standard: By integrating recall across logarithmic tolerance thresholds \(\tau \in [10^{-1}, 10^{-5}]\) and penalizing non-executable codes as zero recall across all thresholds, the resulting AUC-TR metric faithfully unifies programmatic reliability and geometric precision without distorting benchmark results.
  • GVPM provides a principled credit-assignment mechanism for multi-turn code agents: Combining runtime cascade and geometric stall detection with one-sided advantage clamping offers a broadly reusable reinforcement learning pattern for multi-turn code repair agents, preventing productive foundational work from being penalized by unrecoverable downstream errors.
  • Seamless bridge between industrial drafting and multimodal reasoning: Leveraging the SolidWorks COM interface to synthesize multi-view drawings with ground-truth dimensional callouts produces a realistic training and evaluation pipeline that reflects genuine mechanical engineering workflows.

Limitations & Future Work

  • Single-part scope and proprietary drawing dependency: The current framework is restricted to single-part parametric modeling within CadQuery and specific drafting standards, not yet supporting complex multi-component mechanical assemblies or non-standard hand-drawn blueprints.
  • Absence of functional engineering semantics: Geometric point-cloud metrics like Chamfer Distance evaluate spatial boundary overlap but fail to capture higher-level mechanical functions, such as shaft-hole tolerances, keyway clearances, or standard thread specifications.
  • Trade-off between parametric structure and geometric convergence: Reinforcement learning aggressively rewards minimal Chamfer Distance, which can occasionally encourage the agent to produce hard-coded absolute coordinates rather than maintainable, hierarchical feature trees favored by human engineers.
  • vs. CAD-Coder / CADrille: CAD-Coder and CADrille pioneer text-to-code generation via CoT and reinforcement learning but operate in an open-loop or coarse-reward regime; IterCAD integrates multi-view drawing parsing and multi-turn sandbox feedback, achieving closed-loop self-repair.
  • vs. CAD-Judge / PR-CAD / CAD-Editor: While CAD-Judge explores morphological verification, it relies primarily on compiler signals and text edits; IterCAD integrates visual orthographic feedback, addresses multi-turn credit assignment via GVPM, and introduces the survivor-bias-free AUC-TR evaluation metric.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering closed-loop CAD agent with multi-view drawing anchors, GVPM RL credit assignment, and survivor-bias-free CD-TR curve]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across IterCAD-Bench, Text2CAD, and CADPrompt, backed by exhaustive ablations and failure mode analyses]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear prose narrative, and exceptional architectural diagrams and case studies]
  • Value: ⭐⭐⭐⭐⭐ [High practical relevance to automated manufacturing and reverse engineering with fully open-sourced implementation]