DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: CAD generation, rational Bezier surfaces, differentiable tessellation, flow matching, geometric continuity

TL;DR

DreamCAD uses differentiable parametric surfaces with shared boundaries to learn from point supervision on large-scale meshes, then supports image, point-cloud, and image-mediated text conditioning through coarse-to-fine flow matching, reducing ABC point-to-CAD CD from Cadrille's 2.98 to 0.93 (both displayed as raw CD multiplied by \(10^3\)), while leaving complete CAD topology to an additional recovery stage.

Background & Motivation

A CAD model must do more than resemble an object: it needs surfaces that can be evaluated precisely, edited, and connected, together with topological relationships among faces, edges, and vertices. Triangle meshes are convenient for rendering but do not automatically specify which regions form a single face or which curves define trimming boundaries. Methods such as DeepCAD and Text2CAD represent modeling histories as sketch-and-extrude sequences, producing structures with explicit operational meaning. However, this supervision requires design histories, limits the available training data, and may not express complex free-form surfaces through a small set of sketch-and-extrude operations.

Another family predicts BRep UV grids or topology graphs directly, reducing dependence on specific modeling procedures but still requiring costly face-level or topological annotations. Generating adjacent faces independently can also leave seams, forcing post-processing to address both geometric fitting and topological validity. Many public 3D objects, by contrast, provide only meshes, whose surfaces already contain rich geometric supervision. The missing ingredient is not more 3D shapes, but an intermediate representation that learns from meshes while retaining CAD-style parametric surfaces.

DreamCAD therefore separates learning generalizable geometry from recovering complete CAD topology and addresses the former first. It converts quads on a voxel shell into connected rational Bezier patches, allowing sampled surface points to receive supervision directly from meshes. For text conditioning, geometry-aware CAD descriptions and an image generator provide visual grounding instead of mapping language directly to every surface parameter. Core Idea: use structurally continuous, differentiable parametric surfaces to connect mesh data with CAD geometry learning, then perform multimodal generation in that geometric space without forcing discrete topology prediction into the same training objective.

Method

Overall Architecture

During training, DreamCAD constructs sparse voxel features from meshes and trains a variational autoencoder (VAE) that outputs parametric surfaces. Two conditional flow models then learn to generate coarse voxel structure and per-active-voxel structured latents (SLAT), respectively, before a pretrained surface decoder produces geometry. Images and point clouds provide conditions directly; text first passes through an image model fine-tuned on CAD data and then follows the image-to-CAD path. Solid arrows below show generation-time data flow, while dashed arrows indicate training data or supervision; ground-truth meshes are not additional inputs required at generation time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Text["Text"] --> Visual["Metadata-Based Visual Bridging"]
    Captions["Four Views and CAD Metadata"] -.->|Training| Visual
    Visual -->|Generated Image| Coarse["Coarse-to-Fine Structured Generation"]
    Inputs["Image or Point Cloud"] --> Coarse
    Mesh["Training Meshes"] -.->|Voxel Encoding and Latent Learning| Coarse
    Coarse -->|Active Voxels and SLAT| Surface["Shared-Boundary Surface Decoding"]
    Mesh -.->|Surface Point Supervision| Surface
    Surface --> Output["Continuous Parametric Surfaces"]

The output is a collection of surfaces defined by control points and weights, not an inherently complete STEP solid with semantic face partitions, trimming relationships, and full topology. The paper additionally studies topology recovery with a language model, but that feasibility experiment is not a mandatory step of the geometry-generation backbone above.

Key Designs

1. Metadata-Based Visual Bridging: ground language constraints in observable geometry

CAD descriptions often specify hole counts, tooth counts, and relative dimensions, which are more precise constraints than ordinary object-category names. Predicting 3D structure directly from text requires semantic interpretation, spatial layout, and surface-parameter generation at once; the authors report slow convergence and inadequate prompt fidelity. CADCap-1M addresses this by collecting models from ABC, Automate, CADParser, Fusion360, ModelNet, and 3D-Future and providing over 1M CAD descriptions. Each model is rendered from four orthographic views, and GPT-5 receives available model names, hole counts, and length-width-height ratios. Model names can be extracted from STEP files, while hole counts are computed with an existing method rather than guessed by the captioning model from images alone. This information constrains a generic description such as a disk to a geometric object with a particular hole layout, part identity, and dimensional proportions.

The authors then fine-tune Stable Diffusion 3.5-2B on these captions so that its generated images approach the training distribution of the image-to-CAD model. At inference time, an image is generated first and passed to the pretrained image-to-CAD model, rather than using a separate end-to-end text-to-BRep decoder. This reuses the image model's visual prior and supplies the 3D model with more explicit spatial evidence. The image remains an information bottleneck: if the requested hole count is lost at this stage, the geometry model may not recover the original constraint. The four-view captioning procedure in CADCap-1M differs from the VAE's 150-view feature extraction and should not be treated as the same input requirement. The former constructs text data, while the latter builds latent representations of training shapes; users do not need to provide 150 images.

2. Coarse-to-Fine Structured Generation: establish occupancy before local geometry

The shape VAE first voxelizes each training mesh at resolution \(32^3\) and retains features only for active voxels. The authors render 150 RGB and normal views, extract DINOv2 features, project them to voxel centers, and average them across views. Features also include per-view normals, voxel-center coordinates, and signed distance values; the paper reports a final feature dimension of 2502. These signals expose local appearance, surface orientation, and distance to the actual surface rather than merely a binary occupancy flag. A sparse Transformer encoder compresses them into position-associated latent codes with a per-voxel latent dimension of 8. This establishes a learnable geometric representation: coarse positions organize space, while local latents describe details that the voxel shell alone cannot capture.

During conditional generation, the first Flow Transformer produces a low-resolution voxel latent structure, which a pretrained lightweight voxel VAE decoder upsamples into a voxel grid. The second Flow Transformer generates SLAT features for active voxels from that grid and the input condition. Both use flow matching, learning continuous trajectories from prior samples to target representations rather than regressing all control points in one step. Images are encoded with pretrained DINOv2, whereas point clouds use PointNet++ trained jointly with the flow models. Finally, the pretrained parametric surface decoder receives voxel positions and SLAT features and produces evaluable surfaces. This also explains error propagation: if holes or thin parts are misplaced at the voxel stage, local surface refinement cannot be assumed to repair the global structure.

3. Shared-Boundary Surface Decoding: enforce seam agreement through structure

Each bicubic rational Bezier patch is specified by a \(4\times4\) control-point grid and 16 corresponding weights. Bernstein basis functions evaluate the surface over a two-dimensional parameter domain; control points set positions, while weights adjust their influence on surface shape. If patches are predicted independently, low sampled-point distances can still coexist with cracks or overlaps between adjacent surfaces. DreamCAD first applies flood-fill to the sparse voxels, removing internal quads and retaining the exterior surface. Each exterior quad initializes control points through bilinear interpolation of its four corners, with all weights initially set to 1. Adjacent patches share boundary control points instead of leaving two nearly coincident but independent boundaries for a loss function to align.

The decoder predicts a displacement and a weight update for each control point, using bounded displacements and positive weights to reduce early training degeneracy. The following retains only the local update explicitly readable in Section 4.1: \(c\) is an initial control point, and \(d\) and \(\widehat w\) are predictions.

\[ c \leftarrow c + \tanh(d), \qquad w = \log(1+\exp(\widehat w)). \]

For shared boundaries, the authors average displacement and weight updates from all incident patches so that boundary positions and weights agree. With rational surfaces, weights must be coordinated alongside positions: coincident control points alone do not ensure identical boundary curves. Each patch is then sampled on a uniform parameter-domain grid, its cells are triangulated, and shared boundaries between adjacent patches are merged. Surface evaluation is differentiable with respect to control points and weights, so geometric losses on sampled points can update these parameters. \(C^0\) continuity establishes positional agreement at seams, not tangent smoothness, global freedom from self-intersections, or complete CAD topological validity. The paper additionally uses \(G^1\) tangent-continuity and Laplacian smoothing regularizers to reduce rough surfaces and spikes caused by optimizing point distances alone.

A Worked Example

The cylindrical hub with a recessed center and eight surrounding circular holes in Figure 5 illustrates the text-generation path. This walkthrough explains a paper example, not a new experiment, and does not imply that every sample exactly reproduces eight holes. The text first passes through the fine-tuned image model to obtain an image condition containing the cylindrical outline and hole layout. The coarse flow model generates voxel structure, and the fine flow model supplies local geometric latents for active voxels. The surface decoder initializes shared-boundary patches on the voxel shell, then adjusts control-point displacements and weights to approximate the cylinder, recess, and hole walls. A complete CAD file still requires a separate topology-recovery procedure; continuous surfaces alone do not establish recovered design intent.

Loss & Training

The VAE objective includes Chamfer distance between sampled surface points and target points, \(G^1\) regularization, mesh Laplacian smoothing, and latent KL regularization. Equations (1), (4), and (5) have extraction damage in the full-text cache, so missing operators or tensor dimensions are not reconstructed as exact author formulas. The text explicitly gives \(\lambda_{cd}=10^2\), \(\lambda_{g1}=5\times10^{-3}\), and \(\lambda_{lp}=1\); it does not list the KL weight at that location. Both generation stages use the clearly readable flow-matching objective below, where \(x_0\) is a prior sample, \(x_1\) the target representation, and \(x_t\) an intermediate state.

\[ \mathcal{L}_{FM}=\mathbb{E}_{x_0,x_1,t}\left[\left\|v_\theta(x_t,t)-(x_1-x_0)\right\|_2^2\right]. \]

The VAE encoder and decoder each have 8 Transformer layers and train for 700k steps with batch size 32, taking approximately 3 weeks according to the authors. Optimization uses AdamW with learning rate \(5\times10^{-5}\) and weight decay \(10^{-4}\); without the hardware configuration, this duration is not a general training-cost estimate. During training, sigmoid scheduling increases CD samples from 16K to 100K, while tessellation resolution increases from \(4\times4\) to \(16\times16\). Each flow model trains for 500k steps with 10% condition dropout; inference uses 50 steps and a classifier-free guidance scale of 7.5. The text-to-image model uses LoRA with rank and alpha both set to 4, training for 300k steps. Point clouds are normalized to \([-0.5,0.5]\), and surface normals are included in 50% of training batches; image training randomly selects one of four orthographic views per epoch. The authors report approximately 15 seconds for image- and point-to-CAD and 30 seconds for text-to-CAD, representing their experimental setup rather than a standardized hardware benchmark.

Key Experimental Results

Main Results

Table 1 lists 1,306,409 meshes from 10 public datasets for VAE training; this total should not be equated directly with the caption count of CADCap-1M. The main evaluation uses 15K samples each from ABC and Objaverse; Objaverse is filtered using CAD-related caption keywords and is therefore a selected out-of-distribution test. Geometric metrics use 8192 uniformly sampled points from shapes normalized into a unit cube centered at the origin. GPT-5 preference evaluation uses 5k samples, while the user study involves 14 CAD experts and 1k samples; evaluators select the output best matching the input from multiview renderings. User therefore measures selection share within a candidate set, not CAD validity or independently assessed correctness of every output.

The following excerpts Table 2 on page 11; CD is displayed as raw distance multiplied by \(10^3\) and is lower-is-better, while User is a higher-is-better percentage.

Task	Model	ABC CD	Objaverse CD	ABC User (%)	Objaverse User (%)
Point-to-CAD	CAD-Recode	3.73	7.92	N/A	N/A
Point-to-CAD	Cadrille	2.98	6.28	N/A	N/A
Point-to-CAD	DreamCAD	0.93	1.25	N/A	N/A
Image-to-CAD	BRepDiff	20.69	57.51	17.63	16.63
Image-to-CAD	DreamCAD	4.12	20.16	77.03	82.92
Text-to-CAD	NURBGen	50.84	73.54	4.44	7.28
Text-to-CAD	BRepDiff	54.12	74.32	3.20	6.41
Text-to-CAD	DreamCAD	20.32	34.61	85.40	83.48

Table 2 reports ABC point-to-CAD F1 of 92.12 for DreamCAD and 78.86 for Cadrille, both displayed after multiplication by \(10^2\); normal consistency is 0.94 and 0.80, respectively. The text-conditioned BRepDiff baseline receives outputs from the same fine-tuned image model as DreamCAD, so the difference is not entirely attributable to the text-to-image front end. However, training data scale is not matched in the main comparison, preventing attribution of all gains to the surface representation; the paper refers to additional DeepCAD experiments in supplementary material not supplied for this task.

Ablation Study

Table 3a on page 12 trains the VAE from scratch on 300K samples and evaluates 15K ABC test shapes, studying reconstruction regularization rather than conditional generation. CD and Lap retain the table's display scale of raw values multiplied by \(10^3\); HD retains its original scale, and all three are lower-is-better.

Regularization	CD	Lap	HD
No regularization	0.0210	0.0073	0.020
G1 only	0.0230	0.0064	0.022
Laplacian only	0.0225	0.0022	0.022
G1 + Laplacian	0.0259	0.0020	0.024

Joint regularization lowers Lap from 0.0073 to 0.0020 but raises CD from 0.0210 to 0.0259, showing that smoother surfaces do not simply follow from minimizing point distance. The resolution study in Table 3b on page 12 does not retrain four VAEs: it separately optimizes 1000 training meshes for 2000 epochs at learning rate \(10^{-4}\). Patch counts below are averages, and CD retains the table's scale of raw values multiplied by \(10^3\); absolute values should not be compared directly with Table 3a or the main tasks.

Voxel Grid Side Length	Mean Patch Count	CD
24	1434.32	0.0231
32	2546.18	0.011
48	7720.48	0.0109
64	10179.34	0.0105

Increasing resolution from 32 to 64 nearly quadruples the mean patch count while reducing CD only from 0.011 to 0.0105, supporting coarse voxel localization followed by parametric detail modeling. Table 4b on page 14 further shows that perturbing coarse voxels with noise of standard deviation \(10^{-2}\) increases ABC point-to-CAD CD from 0.93 to 4.51, both displayed as raw values multiplied by \(10^3\). This supports the importance of accurate coarse structure, but perturbation is not equivalent to removing the coarse stage and retraining, nor a direct comparison against every alternative architecture.

Key Findings

• ABC user preference is 77.03% for image-to-CAD and 85.40% for text-to-CAD; their candidate baseline sets differ, so these percentages do not directly compare task difficulty.
• Caption correctness is rated at 95.8% by users and 98.31% by GPT-5 on page 13; these are sample-based judgments and do not guarantee correct hole counts in every caption.
• In the experiments accompanying Figure 7, metadata-augmented captions receive 80.3% user preference on 1000 samples, while fine-tuned images receive 75.6% preference in a comparison of 500 images.
• The topology-recovery feasibility study fine-tunes Qwen3-4B on 50K samples and tests 600 samples, 200 per task, reporting 99.2% valid CAD and \(\mathrm{CD}=0.17\times10^{-3}\) in Section 5.4 on pages 14-15.
• Numerical caveat: the prose claims a 62% ABC text-to-CAD CD reduction over NURBGen, but Table 2's 50.84 to 20.32 implies approximately 60.0%; this note retains the table values without treating the prose percentage as verified.

Highlights & Insights

• The key contribution changes the supervision interface: samples need neither design histories nor complete BReps, provided their mesh surfaces can be sampled to constrain parametric patches. This brings existing 3D data into CAD geometry learning.
• Seam continuity follows from shared variables rather than hoping Chamfer distance will align adjacent patches. The same principle can benefit other geometric representations assembled from local parametric pieces.
• Coarse voxels and fine surfaces serve different roles: spatial organization versus shape recovery through editable parameters. The resolution study suggests that denser discrete grids are not always the most economical route to detail.
• Reader insight: CAD generation with hole-count or dimensional constraints could add explicit checks between intermediate images and final geometry. This is a possible extension, not a closed-loop mechanism implemented in the paper.

Limitations & Future Work

• The authors explicitly acknowledge that the main model lacks full CAD topology. The additional language model's 600-sample experiment establishes feasibility, not production-ready solids for all open-vocabulary shapes.
• \(C^0\) continuity does not replace global self-intersection checks, semantic face merging, or manufacturability validation; \(G^1\) and Laplacian terms are soft regularizers, not certificates of smoothness at every boundary.
• Training data scale differs from the baselines, and Objaverse is filtered by CAD keywords. Results support generalization under that setup rather than arbitrary natural shapes or an isolated architectural contribution.
• GPT-5 is used both to create CADCap-1M and for part of its automated evaluation, creating possible shared-model preferences despite additional expert assessment; there is no evidence here of programmatic geometric verification of every caption.
• The supplied cache contains the main paper but not supplementary material, and some formulas are damaged in extraction; this note does not invent the missing KL weight, unspecified F1 threshold, or detailed topology-validity criteria.

Related Work & Insights

• Compared with DeepCAD / Text2CAD: these methods emphasize modeling-operation sequences, whereas DreamCAD prioritizes scalable parametric geometry learning; its geometric output does not directly recover the original design history.
• Compared with BRepGen / BRepDiff: UV or BRep generation relies on corresponding annotations and conversion procedures, while DreamCAD directly supervises shared-boundary patches with points but postpones complete topology.
• Compared with NURBGen: NURBGen uses symbolic NURBS sequences, whereas DreamCAD first obtains visually aligned geometry and then explores topology recovery using a related sequence representation; the approaches are complementary at this downstream stage.
• Implication for structured 3D generation: SLAT supplies spatial organization and analytic surfaces provide editable CAD parameters; the differentiable intermediate representation determines which data without specialized annotations can contribute to training.

Rating

These are reader assessments on a 5-point scale, not experimental metrics reported by the paper.

• Novelty: 4/5. The integration of shared-boundary surfaces, point supervision, and multimodal generation is valuable primarily for its representation and training interface.
• Experimental Thoroughness: 4/5. Three conditioning modalities, out-of-distribution evaluation, and several ablations are covered, but data-scale differences and limited topology evaluation constrain the conclusions.
• Writing Quality: 4/5. The main argument clearly distinguishes geometry from topology, though some percentage claims conflict with table values and require checking.
• Value: 4/5. It offers a route from large mesh collections to editable CAD geometry, while important steps remain before reliable generation of complete engineering solids.