ShellMaker: Language-Guided Exterior Completion under Structural Constraints¶

Conference: ECCV 2026
arXiv: 2606.31680
Paper: Project Page
Code: None yet (the paper promises open-source)
Area: 3D Vision
Keywords: Building Exterior Completion, Language-Guided Generation, Structural Constraint Preservation, PBR Materials, Parametric Roofs

TL;DR¶

ShellMaker proposes a language-guided building exterior completion framework: given a building scaffold (wall + door/window opening layouts output by indoor generators/CityGML/CAD) and a style prompt, it produces a complete PBR exterior facade mesh through four modules: parametric roof generation, LLM part-aware prompt refinement, compatibility-aware joint wall-roof texture retrieval, and geometry-aware assembly. While strictly preserving footprints and opening layouts, it matches any architectural style, achieving a footprint IoU of 0.992 and an opening center error of only 0.091.

Background & Motivation¶

Recently, indoor scene generation systems (such as Holodeck, SceneWeaver, and Artiscene) have become capable of automatically synthesizing structured indoor spaces with wall layouts and door/window opening semantics. However, these pipelines stop at the building shell level—no facades, no roofs, and no stylized exterior details. The generated buildings are thus "semi-finished products" that cannot be directly used for digital twins, architectural visualization, or urban scene modeling.

Directly applying existing controllable 3D generation models (like Trellis-2, SpaceControl, and SAT-Skylines) to exterior completion presents a fundamental contradiction: these models pursue "perceptual realism" but do not guarantee "layout fidelity." The generated exterior facades cannot strictly align with the wall footprints and door/window opening locations in the indoor layout because large-scale paired datasets of "structured building layout—complete textured exterior facade" simply do not exist, and the models have never learned this constraint. This is the core challenge: perceptual quality and layout constraints cannot be simultaneously satisfied.

The goal of this paper is to define building exterior completion as a language-guided generation task under hard structural constraints. The footprint topology, wall boundaries, and door/window opening positions provided by the scaffold are "immutable constraints," while the style prompts provide "mutable semantics." The core idea is to replace end-to-end generation models with a staged modular pipeline—the structure phase handles constraint satisfaction, the stylization phase handles semantic alignment, and the assembly phase handles geometric integration. The responsibilities of each phase are separated, which not only avoids cross-phase error propagation but also allows each module to use the most suitable tool (parametric geometry, LLM, retrieval, or pre-trained generative models) rather than relying on a single all-powerful model.

Method¶

Overall Architecture¶

The core problem solved by ShellMaker is: given a structured building scaffold \(S\) and a text style prompt \(P\), generate a complete PBR exterior facade mesh \(E = \Phi(S, P)\) that is both faithful to the geometric constraints of the scaffold and semantically aligned with the specified architectural style. The framework breaks this generation process into three sequential stages (Figure): the structure generation stage derives a fixed footprint from the scaffold and parametrically generates the roof geometry; the stylization stage refines the user prompt into part-level specifications, synthesizes style-consistent door and window assets, and retrieves compatible wall-roof texture pairs; the assembly stage combines all parts into the final mesh through geometry-aware boolean carving, semantically aligned asset placement, and scale-consistent UV unwrapping.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Building Scaffold<br/>+ Style Prompt"] --> B["Unified Scaffold Parsing<br/>and Parametric Roof Generation"]
    B --> C["LLM-Driven Part-Aware<br/>Prompt Refinement & Text-to-3D Synthesis"]
    C --> D["Compatibility-Aware<br/>Joint Wall-Roof Texture Retrieval"]
    D --> E["Geometry-Aware Assembly:<br/>Shape-Fitting Carving & UV Parameterization"]
    E --> F["Output: Complete PBR<br/>Building Exterior Facade Mesh"]

Key Designs¶

1. Unified Scaffold Parsing and Parametric Roof Generation: Standardizing Three Heterogeneous Inputs into a Single Pipeline

Indoor generators output scene graphs, CityGML outputs semantically labeled surfaces, and BIM/CAD outputs geometric primitives—the representation formats of these three inputs are entirely different. The first key design of ShellMaker is to define a unified scaffold representation layer: walls are represented by 2D endpoints + outward normal vectors, floors and ceilings by polygons, and door/window openings record both world coordinates and wall local coordinates. Each of the three inputs has a dedicated parser (the indoor generator parser parses scene graph semantics, the CityGML parser merges coplanar wall segments and projects openings to the wall local coordinate system, and the CAD parser performs subdivision + normal inference using ifcopenshell), but they ultimately produce the same standardized scaffold, hiding the input format differences from all downstream stages.

Roof generation begins with footprint decomposition. Complex polygonal footprints are first split into simple sub-regions: either using straight skeleton decomposition (calculating the medial axis \(\rightarrow\) partitioning ribbon regions \(\rightarrow\) merging with convexity tolerance, suitable for shape-preserving roofs) or greedy rectangle decomposition (recursively extracting the largest axis-aligned rectangle \(\rightarrow\) subtracting it \(\rightarrow\) repeating for the remaining area, suitable for diverse composite roofs). Subsequently, five standard roof types (flat, gabled, pyramidal, hipped, half-hipped) are applied globally or on sub-regions, with each roof controlled by four continuous parameters: slope angle, eave overhang, ridge length ratio, and half-hip clipping ratio. The key insight of this design is: the roof is the critical bridge connecting 'fixed footprint constraints' and 'style variability'—parametric generation allows the roof shape to change continuously with the style while naturally guaranteeing that it does not exceed the footprint boundary.

2. LLM-Driven Part-Aware Prompt Refinement and Two-Stage Text-to-3D Synthesis: Turning a Single Sentence into Five Useful Prompts

User-input style prompts like "a Victorian bakery" indicate the overall style but offer almost no actionable geometric description for generating specific doors, windows, or roof decorations. Directly using this sentence as a text-to-3D prompt would result in assets with disconnected styles. ShellMaker's approach is to use an instruction-tuned LLM (GPT-4.1) to automatically expand a single sentence into five categories of structured prompts—wall material, roof material, entrance door, window, and roof decoration. The system instructions for each category explicitly require describing "a single isolated architectural component without surrounding walls or environmental background, with style details strictly fitting the target architectural style." This step translates "coarse-grained style semantics" into "actionable geometric descriptions at the part level," which is crucial for style consistency.

Each refined prompt undergoes a two-stage text-to-3D pipeline: Nano Banana first generates a reference image, and Trellis-2 reconstructs the reference image into a textured mesh with PBR material parameters (diffuse/normal/roughness). This "image generation + 3D reconstruction" decomposition leverages the strength of image generation models in style fidelity while allowing 3D reconstruction to focus solely on geometry—without requiring fine-tuning or supervision for each architectural style. Generated assets are cached and reused based on <style, category, part-name> triplets. Openings of the same type on the same building share a single mesh to ensure visual neatness.

3. Compatibility-Aware Joint Wall-Roof Texture Retrieval: Not Just "Finding Similar Ones", but "Finding Compatible Pairs"

For large architectural surfaces (walls, roofs), using generative textures often leads to non-tileable results and misaligned PBR channels. Therefore, ShellMaker opts for retrieval instead. However, simple independent retrieval ignores a common-sense rule: in real buildings, wall and roof materials exhibit strong co-occurrence patterns (brick walls with terracotta tile roofs, concrete with metal roofs, limestone with slate roofs). Independent retrieval might yield contradictory combinations (e.g., red brick walls with thatched roofs).

ShellMaker's solution is to pre-compute a compatibility matrix \(C\) offline, with dimensions \(N_w \times N_r\) (\(498\) wall textures \(\times 81\) roof textures). Each entry integrates four signals: \(C_{\text{mat}}\) (material co-occurrence prior, giving high scores to classic combinations like brick-terracotta or concrete-metal), \(C_{\text{color}}\) (CIELAB color difference + saturation difference, measuring color harmony), \(C_{\text{freq}}\) (cosine similarity of the 2D FFT radial power spectrum, measuring texture scale matching), and \(C_{\text{clip}}\) (CLIP embedding similarity, high-level semantic signal), weighted by \(\lambda = (0.45, 0.25, 0.15, 0.15)\).

At runtime, CLIP similarities are calculated between the wall/roof prompt and the texture database to select the top-\(K\) (\(K=32\)) textures, forming \(K^2\) candidate pairs. Then, single-side similarities and pre-computed compatibility scores are combined: \(s_{ij} = \alpha \cdot s_w(i) + \beta \cdot s_r(j) + \gamma \cdot C_{ij}\), with weights \((\alpha, \beta, \gamma) = (0.40, 0.30, 0.30)\). The top-\(M\) (\(M=8\)) pairs are retained, and temperature-controlled softmax is used to sample the final texture pair: a lower temperature \(\tau\) is more conservative (choosing the most compatible), while a higher temperature is more diverse. The elegance of this design lies in the offline pre-computation of compatibility; at runtime, it only requires lightweight table lookups and weighting—ensuring near-zero retrieval latency.

4. Geometry-Aware Assembly: Carving Walls, Inserting Assets, and Unwrapping UVs—All Without Breaking Constraints

The assembly stage is the final gate for "enforcing constraints." For each door/window asset to be embedded, shape-fitting carving is first performed: removing the outer 5% area of the asset to exclude decorative protrusions (such as pediments or pilasters that inflate bounding boxes), and flattening the back to ensure it does not protrude into the interior. If the projected contour is rectangular enough (\(\text{area} \ge 95\%\) of the bounding box), rectangular carving is used directly; otherwise, a binned contour envelope is applied—partitioning the X-axis into 64 bins, taking the upper and lower Y extrema of each bin to form a closed polygon, simplifying it, and dilating it outward by 0.01 to ensure full coverage. This polygon serves as the cutter for wall boolean subtraction.

UV unwrapping uses world-space planar projection: each wall/roof is projected along its principal normal direction. Roof slopes use ground projection to unify the UV origin, and vertical eave faces use the same horizontal coordinate with vertical offsets to eliminate eave-slope seams. The most critical component is the automatic UV scale factor: for each texture map, its 2D FFT, radial average power spectrum, and weighted average frequency \(f_{\text{dominant}}\) are calculated from its luminance map. The median frequency across the texture database is taken as a reference—coarse textures (like large bricks) have larger scale factors, repeating more frequently and making individual blocks smaller, while fine textures do the opposite. This step ensures that different materials exhibit consistent perceived texture density on the building surfaces, preventing scale inconsistencies like "fine brick wall texture paired with coarse concrete texture."

A Complete Example¶

Take "a Victorian bakery" as a complete example to walk through the pipeline: (1) In the structure stage, parsing the bakery scaffold output by the indoor generator (rectangular footprint, two large windows on the front facade + centered door) selects the global mode, straight skeleton decomposition, and a hipped roof due to the simple footprint, with a slope angle of 35 degrees and an eave overhang of 0.3m; (2) In the stylization stage, the LLM refines "Victorian bakery" into five sets of prompts—wall: "aged red brick with decorative string courses", roof: "dark grey natural slate", door: "Victorian double door with fanlight transom and carved pediment", window: "tall arched sash window with stone keystone and sill brackets", chimney: "ornate brick chimney with corbelled cap"; Nano Banana generates reference images for each part, and Trellis-2 reconstructs them into PBR-textured meshes; (3) For texture retrieval, red brick and dark grey slate score highest in the offline compatibility matrix (due to material co-occurrence + harmonious complementary warm-cool colors). After CLIP similarity confirmation at runtime, this combination is sampled with a temperature of 0.12; (4) In the assembly stage, window areas undergo binned contour envelope carving (as arched window projections are non-rectangular), doors use rectangular carving, downspouts are automatically generated under the eaves, the chimney is embedded into the roof and clipped to keep only the part above the roof surface, and all wall and roof surfaces receive FFT-calibrated UV unwrapping. The final output is a complete PBR exterior facade mesh with door/window positions matching the scaffold and the roof slope and materials echoing the Victorian style.

Loss & Training¶

ShellMaker is a modular pipeline and does not involve end-to-end training of neural networks; thus, there are no traditional loss functions. The "optimization" in the pipeline is reflected in the offline pre-computed compatibility matrix and the runtime texture pair scoring and sampling mechanisms. The compatibility matrix \(C_{ij}\) is synthesized by weighting four manually defined signals, with the weights \(\lambda = (0.45, 0.25, 0.15, 0.15)\) set manually based on the importance of each signal. Hyperparameters for runtime retrieval include: \(K=32\) (shortlist length), \(M=8\) (number of candidate pairs retained), sampling temperature \(\tau = 0.12\) (conservative, ensuring texture pair harmony), and pair scoring weights \((\alpha, \beta, \gamma) = (0.40, 0.30, 0.30)\). All hyperparameters are fixed and do not vary with inputs, and the paper does not report sensitivity analyses for these hyperparameters.

Key Experimental Results¶

Main Results¶

Method	CLIP-Sim ↑	UNI3D ↑	Footprint IoU ↑	Opening IoU ↑	Opening Center L2 ↓
Trellis-2	0.318	0.401	0.492	0.431	0.914
SpaceControl	0.274	0.258	0.812	0.508	0.523
SAT-Skylines	0.281	0.265	0.761	0.548	0.463
ShellMaker (Ours)	0.315	0.424	0.992	0.883	0.091

ShellMaker establishes a massive lead in structural consistency: the footprint IoU of 0.992 is near-perfect (the pipeline never modifies the wall geometry), and the opening center error is a mere 0.091. In terms of style metrics, the CLIP-Sim is on par with the strongest baseline, Trellis-2 (0.315 vs. 0.318), while the UNI3D score significantly outperforms all baselines (0.424 vs. a maximum of 0.401). This indicates that multiview-independent 3D style evaluation is more favorable to ShellMaker—as its part-level generation naturally yields more consistent cross-view style performance than end-to-end image-conditioned methods. Note that all methods share the same image conditioning (scaffold depth/semantic map rendering \(\rightarrow\) Nano Banana \(\rightarrow\) generated image \(\rightarrow\) fed to respective 3D models), so the performance gap arises entirely from the pipeline design itself.

Ablation Study¶

Configuration	CLIP-Sim ↑	UNI3D ↑	Footprint IoU ↑	Opening IoU ↑	Opening Center L2 ↓
ShellMaker (Ours)	0.315	0.424	0.992	0.883	0.091
w/o Joint Texture Retrieval	0.285	0.357	0.992	0.889	0.083
w/o LLM Prompt Refinement	0.293	0.364	0.992	0.922	0.095
w/o Both	0.280	0.305	0.992	0.881	0.084

Removing joint texture retrieval drops CLIP-Sim from 0.315 to 0.285 (-9.5%) and UNI3D from 0.424 to 0.357 (-15.8%), indicating that material compatibility across surfaces contributes heavily to the overall perception of style. Removing LLM prompt refinement leads to a slightly smaller decline in CLIP-Sim (-7.0%), but UNI3D still drops significantly (-14.2%), demonstrating that directly using coarse-grained prompts for text-to-3D leads to cross-view style inconsistencies. Removing both collapses both style metrics (-11.1% / -28.1%), validating the complementary nature of texture retrieval and prompt refinement. Structural metrics remain stable across all ablations (footprint IoU is constant at 0.992, as the pipeline never alters wall geometry), and minor fluctuations in opening IoU stem from differences in the window shapes generated under different prompt configurations (e.g., arched vs. rectangular windows).

Perceptual Evaluation¶

ShellMaker vs	Overall Win Rate ↑	Style Win Rate ↑	Structure Win Rate ↑
Trellis-2	71.2%	48.3%	84.1%
SpaceControl	80.1%	71.2%	92.8%
SAT-Skylines	87.4%	68.8%	91.2%

Evaluation on 100 scaffold-prompt pairs using GPT-4.1 as the 2AFC referee. The style win rate against Trellis-2 is only 48.3% (nearly a tie), but the structure win rate is 84.1%—this is the most direct manifestation of the "perceptual quality vs. layout fidelity" contradiction: images generated by Trellis-2 are indeed visually appealing, but their geometry completely fails to align with the scaffold. Against SpaceControl and SAT-Skylines, the structure win rates reach 92.8% and 91.2% because their geometry injection mechanisms still cannot guarantee hard constraint satisfaction.

Key Findings¶

Structural constraint preservation is achieved through "staged isolation" rather than "constraint injection": The fundamental reason ShellMaker's footprint IoU remains constant at 0.992 is not some constraint injection mechanism, but rather the complete decoupling of the structural and stylization stages—wall geometry is determined during the structural stage and never modified, while the stylization stage only generates parts to be inserted and materials to be applied. This design principle is far more robust than any conditional generation hacks.
Joint texture retrieval is the largest single contributor to style scores: Removing it drops CLIP-Sim by 9.5%, which is more than the 7.0% drop caused by removing LLM refinement. This indicates that "whether the walls and roof look compatible" contributes more to the overall style than "how good an individual part looks".
CAD input is the biggest challenge: In cross-format generalization experiments, the footprint IoU for CAD sources drops to 0.889 (compared to 0.992 for indoor generators and 0.991 for CityGML) because geometry noise and topological inconsistencies in CAD files make scaffold parsing more difficult.

Highlights & Insights¶

The design philosophy of "hard constraint decoupling": Leaving constraint satisfaction to deterministic parametric geometry and boolean operations, and semantic alignment to LLMs and retrieval, completely decouples the two. This approach is much cleaner and more reliable than trying to force-inject constraints into the latent space of generative models. This design philosophy can be transferred to any scenario requiring structured constraints in a generative pipeline (e.g., maintaining room dimensions in furniture layout, or body measurements in clothing design).
FFT-driven automatic UV scaling: Using the frequency-domain features (dominant frequency) of a texture to normalize perceived texture densities of different materials is a low-cost but highly effective trick. This concept essentially targets "perceptual consistency" instead of "geometric consistency" and can be transferred to any scenario involving multi-material mapping (e.g., terrain textures, product rendering).
LLMs for "semantic decomposition" rather than "end-to-end generation": Instead of expecting an LLM to generate 3D models directly, LLMs are used to translate coarse style descriptions into part-level text-to-3D prompts—this resides well within the current capabilities of LLMs and is highly stable. The approach of using LLMs as "translators" rather than "generators" can be widely applied.

Limitations & Future Work¶

Door and window generation quality is bottlenecked by the upstream model: Occasionally, degenerate parts (deformed window frames, bad meshes) are produced because the pipeline lacks geometric validation or retry mechanisms for Trellis-2's output. Adding a simple geometric quality check (such as manifold check or face-count threshold filtering) could serve as a fallback.
No per-floor facade variation: Currently, all windows share the same style, which fails to simulate the variation found in real-world buildings, such as large storefront windows on the ground floor and smaller residential windows on upper floors. Resolving this requires introducing "floor context" in the LLM refinement stage to route different floors to different prompt branches.
Texture retrieval limits style diversity: Although the texture database of 498 walls + 81 roofs is curated, it remains a closed set, rendering it unable to handle unconventional style demands like "pink exterior walls with a rainbow roof." The paper itself notes that future work should introduce generative texture synthesis to break the database limitations.
Lack of collision detection: High-density opening regions can lead to intersecting parts (e.g., two adjacent window decorative frames clipping into each other) because the assembly stage lacks geometric collision inspection.

vs Trellis-2 / Gen-purpose text-to-3D: These methods offer stronger generation quality and style diversity but completely ignore predefined geometric layouts, allowing the generated meshes to manifest arbitrary shapes at arbitrary locations. The value of this paper is not on "generating more beautiful objects" but rather on "generating them correctly"—strictly preserving input wall topologies and opening semantics, which generic methods cannot do.
vs SpaceControl / SAT-Skylines (Controllable 3D Generation): These methods feed depth/normal/geometric priors into the latent space to "guide" the generation toward the target structure. However, this guidance is soft and approximate, falling short of guaranteeing hard constraints. The staged decoupling scheme in this paper provides a more fundamental solution—it avoids betting on the generative model's ability to learn constraints, instead keeping the generative model entirely away from constraint-related matters.
vs Procedural Urban Modeling (CityEngine, etc.): Traditional procedural methods guarantee geometric constraints but require manually written grammar rules and cannot automatically adapt their appearance based on natural language style prompts. The innovation of this paper lies in using LLMs and retrieval to graft "language-driven style control" onto procedural pipelines that originally required manual parameter tuning.

Rating¶

Novelty: ⭐⭐⭐⭐ Formulates building exterior completion as a language-guided generation task under hard constraints for the first time. The "structure-style decoupling + three-stage pipeline" design paradigm is clean and convincing. However, because each sub-module (parametric roofs, LLM prompting, texture retrieval) relies on mature technologies, the main contribution lies in their combination rather than individual technical components.
Experimental Thoroughness: ⭐⭐⭐⭐ Evaluation covers 200 scaffolds, 500+ textures, 60 prompts, 3 baselines, a full set of ablations, cross-format generalization tests, and 2AFC perceptual evaluation, making it highly comprehensive. One star is deducted because "perceptual evaluation" uses GPT-4.1 instead of human subjects, and no hyperparameter sensitivity analysis is reported.
Writing Quality: ⭐⭐⭐⭐ Clear structure, abundant illustrations, and high reproducibility, with supplementary materials covering texture library samples, runtime analysis, complete prompt sets, and failure cases. A minor drawback: the ablation experiments are embedded in the main results table rather than presented independently, making them easy for readers to overlook.
Value: ⭐⭐⭐⭐ Bridges the critical "exterior wall" gap in the indoor scene generation \(\rightarrow\) complete building asset pipeline. Furthermore, its format-agnostic scaffold abstraction extends its applicability far beyond the initial indoor generator scenarios, offering direct utility for downstream applications like digital twins, architectural visualization, and urban modeling.