WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion

Conference: ECCV 2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: multi-room scene generation, mesh-conditioned synthesis, multi-view consistency, projective texture accumulation, Gaussian Splatting

TL;DR

WorldMesh first builds a multi-room mesh carrying building structure and reconstructed objects, then feeds progressively accumulated wall textures and depth as conditioning to image generation, and finally reconstructs the scene with mesh-depth-regularized 3D Gaussian Splatting (3DGS), achieving a 96.2% average preference over baselines in a 31-participant perceptual study; its core is a persistent geometric constraint rather than merely generating a video that looks coherent.

Background & Motivation

Image and video generation models deliver realistic materials, lighting, and interior furnishing, but turning that capability into a freely navigable multi-room 3D scene is not straightforward. A single image can plausibly fill in unseen regions, yet during sequential generation the same piece of furniture may be reinterpreted from another viewpoint; this surfaces as duplicated objects, geometric drift, or inconsistent walls when rotating closely around furniture, passing through doorways, or revisiting old rooms. Looking coherent along a fixed trajectory is not the same as every viewpoint corresponding to one world.

Existing methods mitigate this with frame-by-frame outpainting, panoramas, camera-conditioned video, or coarse layouts. The first families tend to accumulate depth and appearance errors; bounding-box layouts can fix approximate object positions but cannot decide how a bed edge or chair back should look from a different angle. On the other hand, asking a language model to enumerate furniture and coordinates directly often yields monotonous or under-specified furnishings. The paper therefore lets the language model handle the architectural layout, lets the image model propose rich object configurations, and then reconstructs those objects into a persistent mesh instead of letting every viewpoint imagine them independently.

Core idea: write building and object geometry into a mesh that can be rendered repeatedly, and write already-generated wall appearance back onto the same mesh, so that subsequent image generation and the final 3DGS optimization are all constrained by this shared spatial state.

Method

Overall Architecture

The input is natural language describing style and environment; the output is a multi-room 3DGS scene renderable from novel viewpoints. Geometry-first mesh construction first yields walls, floors, ceilings, openings, and reconstructed objects; mesh-anchored appearance generation then synthesizes images in camera order, projecting wall colors back onto the mesh step by step. Validated images together with mesh depths drive a geometry-regularized 3DGS reconstruction.

The mesh here is not a one-off conditioning image but an intermediate representation shared across viewpoints and rooms. Wall textures accumulate during the image generation loop rather than being produced as a complete texture map in advance; the final output is not the coarse mesh itself but a Gaussian representation constrained by it. The pipeline still includes interactive segmentation, so "text input" should not be read as the experiments having achieved a fully hands-off system.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    T["Text description"] --> M["Geometry-first mesh construction<br/>Building layout and object reconstruction"]
    M --> A["Mesh-anchored appearance generation<br/>Hybrid conditioning and nearby-view reference"]
    H["SAM3 interactive point prompts"] -.-> M
    A --> V["Depth-edge structural validation"]
    V -->|Failed: new seed| A
    V -->|Passed: images and cameras| G["Geometry-regularized 3DGS reconstruction"]
    A -->|Wall texture back-projection| M
    M -->|Initial point cloud and supervising depth| G
    G --> O["Navigable multi-room scene"]

Key Designs

1. Geometry-first mesh construction: determine space jointly by building constraints and furnishings from images

The language model Claude Opus 4.6 converts the description into a JSON floor plan containing wall thickness, ceiling height, room floor polygons, and doors or passageways. Candidate layouts are checked against coarse spatial rules, such as no windows between rooms; failing candidates are resampled. Walls are then built from floor-polygon insets, extruded vertically, with shared walls at half thickness, matching edges detected within a 0.01 m tolerance, and door or window openings carved by boolean subtraction and propagated to adjacent rooms sharing the wall. Thin slabs complete floors and ceilings, fixing room connectivity first.

Furniture is not dictated item by item by the language model. The system renders structural depth from an initial camera at the wall midpoint of the room's shorter side and passes it to Flux2-Klein, together with the room type, global style, and descriptions of visible openings. The architectural context matters: given only depth holes, the image model may misread a doorway as a TV. The generated image provides both the furnishing layout and appearance; SAM3 then segments individual objects, and SAM-3D-Objects reconstructs each object's mesh and its transform into the camera coordinate system.

In practice, segmentation uses a few interactive point prompts per object; the main text only notes that the appendix discusses a text-prompt-based automatic alternative. After independent per-object reconstruction, oriented bounding boxes are aligned to the gravity direction and objects are projected onto the nearest support surface below (or above for lamps and the like). This corrects some floating and pose issues but is not a full solver for collision, load-bearing, or physical feasibility. The final mesh keeps object textures and geometry from reconstruction, while building-surface textures are left for the later loop to accumulate.

The camera set is shared between object instantiation and final appearance generation: two eye-level bootstrap views facing each other cover all four walls, extended by roughly 16 perimeter and 8 overhead cameras; redundant perimeter views within 10° of a bootstrap direction are pruned, and cameras are nudged away from placed objects. Initial furnishing images use only the first coverage camera, while the full camera set is used for final reconstruction.

2. Mesh-anchored appearance generation: hand geometry, accumulated textures, and a style reference to the image model

Even with mesh conditioning on every image, independent sampling can still shift wall colors, lighting, and materials. Each room therefore starts with two opposing bootstrap views to establish broad appearance, after which the remaining cameras are selected greedily by rotational similarity to any already-generated camera. Similarity is the absolute inner product of unit quaternions; when generating the current image, the rotationally closest previously generated image serves as the style reference. This rule prioritizes pose continuity and should not be read as explicitly optimizing pixel correspondences across all images.

The conditioning image is not a naive stack of RGB and depth channels: object regions carry their own textures, structural regions with accumulated wall textures use alpha-blended colors, and all other pixels carry grayscale depth. Each generated image is then projected back onto visible wall surfaces through the known camera to update the mesh texture for later viewpoints and other rooms. In this way, the shared mesh stores structure as well as appearance memory; bootstrap views first cover large wall areas, and later views fill in the rest. Occlusions and projection seams remain, and fine appearance is supplied by the image model rather than assuming the back-projected texture is already perfect.

To filter structurally violating images, the system estimates depth for the generated image with Depth Pro, extracts Canny edges from it and from mesh-rendered depth, and computes how much of the mesh's structural edges are covered by the generated image's depth edges. The following is an interpretive restatement assembled from the paper's set definitions, not a character-for-character recovery of the corrupted original:

\[ R_{\mathrm{edge}}=\frac{|E_{\mathrm{mesh}}\cap\operatorname{dilate}(E_{\mathrm{generated}},\delta)|}{|E_{\mathrm{mesh}}|}. \]

The tolerance is 10 px, and Canny uses hysteresis thresholds of 0.1/0.3 of the normalized depth range with a 3×3 Sobel aperture. Images exceeding the acceptance threshold pass; others are regenerated with a new random seed. Using recall only is a deliberate choice: it penalizes missing building edges while allowing the model to add plausible object edges absent from the mesh. It is therefore not a bidirectional geometric-consistency guarantee, and the main text does not report the acceptance threshold value.

3. Geometry-regularized 3DGS reconstruction: keep attractive training views from being fit into a wrong space

Multi-view generation is still not the final navigable scene. The system initializes Gaussians from a point cloud obtained by back-projecting mesh depths, then optimizes 3DGS with the validated synthesized images, camera poses, and mesh depths. The image term combines L1 and DSSIM, while the depth term requires the rendered depth of the Gaussians to approach the mesh depth; the former learns detail and appearance, the latter stops the Gaussians from drifting off building surfaces to fit sparse views.

The loss equation in the cache is missing extracted weights and operators, so this note does not reconstruct the authors' exact formula. The main text explicitly gives λs=0.2 and λd=0.7 and defines DSSIM as 1−SSIM. For the mechanism, the more important distinction is between two geometric roles: initialization decides where the Gaussians start, and depth regularization keeps constraining them throughout optimization. The ablation shows that image fitting alone cannot substitute for the persistent constraint—2D appearance can remain decent while 3D reprojection error degrades sharply.

Loss & Training

Rather than training a new end-to-end multi-room diffusion model, this work composes off-the-shelf generation, segmentation, object reconstruction, and depth models, and optimizes a 3DGS per scene. Furnishing images use a 90° field of view while all other images use 60°; after object placement, roughly 24 images of 1376×768 are generated per room. The camera counts above describe the generation and pruning scheme and should not be mechanically summed into a fixed final view count.

By default, the local Flux2-Klein generates furnishing images and the first appearance image, while subsequent images call the Nano Banana Pro API, which the authors found most faithful to the mesh conditions. The main text also states that the pipeline runs fully locally with Flux2-klein (9B) and that the layout LLM is swappable, but those comparisons are in the supplementary material; the default experiment numbers cannot be attributed to the fully local variant.

Local hardware is a single RTX A5000 with 24 GB VRAM; this excludes the external API's server-side compute. The main text does not expand on per-scene time, API cost, 3DGS optimization iterations, or navigability evaluation details, deferring them to the supplementary material. The cache provided for this task ends at the references, so unreported cost figures cannot be reconstructed from it.

Key Experimental Results

Main Results

Image quality is measured with CLIP-IQA+ and CLIP Aesthetic on frames rendered along trajectories of the final 3DGS; these do not directly measure 3D consistency. The perceptual study involves 31 participants rating overall quality, cross-view object consistency, and room-structure consistency on a 1–5 scale, plus randomized pairwise comparisons. Test trajectories include close-ups and rotations around objects.

Method	CLIP-IQA+ ↑	Aesthetic ↑	Overall quality ↑	3D objects ↑	3D structure ↑
WonderWorld	0.4872	4.7795	1.94	2.19	1.84
FlexWorld	0.4097	4.9453	2.05	2.00	2.34
LayerPano3D	0.3949	4.7312	2.27	2.21	2.44
WorldExplorer	0.4053	5.5476	2.58	2.47	2.76
SpatialGen	0.4648	5.0633	2.61	3.00	3.00
DreamScene360	0.3565	4.7270	3.19	3.10	3.37
WorldMesh	0.5114	5.5799	4.48	4.40	4.35

Data from Table 1 of the paper. The average pairwise preference is 96.2%; 100.0% against SpatialGen and WonderWorld, 96.8% against WorldExplorer, and 93.5% against the remaining three. This is preference within that perceptual study, not a success rate for generation or navigation completion.

Baseline inputs are not identical: SpatialGen receives the layout, object locations, and initial image generated by this method; video baselines use the same text; WonderWorld receives one image generated from the same text. Since most baselines target single rooms, the qualitative comparison shows one room within a multi-room result, which cannot be read as all methods compared under an equally scaled multi-room setting.

Ablation Study

Table 2(a) evaluates cross-view depth reprojection and final 3DGS alignment to the mesh over five multi-room scenes. MAE-norm and AbsRel are reprojection errors; M-MAE measures mesh-depth alignment. The table below selects these three structural metrics plus IQA+, and does not treat fit to the method's own mesh as external ground-truth geometric accuracy.

Config	MAE-norm ↓	AbsRel ↓	M-MAE ↓	IQA+ ↑
Full model, NB Pro	0.1615	0.0761	0.0383	0.5529
Depth + objects, no wall texture	0.2229	0.0997	0.1167	0.3870
Depth + wall texture + object boxes	0.2722	0.1988	0.3335	0.3735
Depth only	0.1970	0.0888	0.2777	0.3818
w/o edge-recall validation	0.1660	0.0784	0.0478	0.5237
w/o mesh point-cloud init	0.1706	0.0803	0.0479	0.5204
w/o mesh depth loss	0.3009	0.1641	0.1904	0.5312
w/o mesh init and depth loss	1.8804	0.6730	2.2549	0.4393

The image evaluation set of Table 2 differs from that of Table 1, so the full model's IQA+ of 0.5529 versus 0.5114 is not a numerical conflict. The perceptual ablation in Table 2(b) covers only one six-room Gothic Revival scene and should not be read as a participant average over five scenes.

Key Findings

• Persistent depth constraints matter more than initialization alone. Removing the depth loss raises MAE-norm from 0.1615 to 0.3009, removing initialization to 0.1706, and removing both collapses it to 1.8804. The 3D structure degrades far more than IQA+ does, confirming that looking good in 2D and being correct in 3D can separate.
• Object meshes cannot be cheaply replaced by boxes. Depth + wall texture + object boxes yields an M-MAE of 0.3335 versus the full model's 0.0383; bounding boxes constrain only coarse positions, not fine object surfaces.
• Different error metrics do not always agree. Depth only attains a reprojection MAE-norm of 0.1970, better than depth + objects at 0.2229, yet its mesh alignment is worse. A single ablation number should not be equated with an overall consistency conclusion.

Highlights & Insights

• The mesh connects building rules with visual generation priors: language handles walls and connectivity, images propose furnishings, and object reconstruction pins those furnishings down. It avoids demanding that a single generative model solve layout, detail, and global memory simultaneously.
• Wall texture back-projection is an appearance state that can be read repeatedly, not just an implicit reference to the previous image. The next viewpoint can draw color evidence from the same 3D surface, which particularly suits cross-room and revisit scenarios.
• Filtering images by depth-edge recall rather than raw depth differences reduces the impact of monocular scale errors, while explicitly tolerating extra detail. It is an interpretable compromise between structural adherence and generative freedom.

Limitations & Future Work

• Admitted by the authors: only single-story layouts are currently supported and multi-level buildings are not handled directly; SAM-3D-Objects can leave incomplete back faces or missing detail in occluded regions.
• The experimental pipeline uses interactive segmentation, and default later appearance generation depends on an API. The quality and cost of a fully automated, fully local variant cannot be inferred directly from the main tables.
• Support-surface projection and edge recall are heuristic constraints; they do not guarantee that all added objects are cross-view consistent, physically plausible, or robot-traversable. The main text mentions a navigability analysis in the supplementary material, which the current cache does not contain.
• The five-scene automatic ablation and single-scene perceptual ablation support only a limited test range; "arbitrarily large" and linear scaling with room count still require the unreported scale and cost evidence.
• Promising directions: automating object segmentation, completing occluded surfaces, and adding uncertainty-aware structural acceptance; quality changes after removing the manual step and after swapping the API should be measured separately.

Related Work & Insights

• vs WorldExplorer / FlexWorld: video priors excel at coherent frames along a trajectory; WorldMesh uses persistent object surfaces and building structure to constrain novel-view interpretation, focusing on stability once the camera leaves predefined paths.
• vs SpatialGen / ControlRoom3D: coarse layouts constrain where objects may appear; this work goes further by conditioning on reconstructed object surfaces and accumulated wall textures, at the cost of extra reconstruction, segmentation, and texturing steps.
• vs DreamScene360 / LayerPano3D: panoramas give broad central-perspective coverage but struggle with depth and occlusion under close-up movement; this work fixes space before generating appearance rather than inferring 3D from a panorama.
• Resources: The project page with code and data is listed on the paper's first page; its online availability has not been independently verified.

Rating

• Novelty: 4/5. Chains an explicit building-and-object mesh, online texture accumulation, and geometry-regularized reconstruction into a clear multi-room pipeline.
• Experimental Thoroughness: 4/5. Covers diverse baselines, structural ablations, and a perceptual study, though scale, cost, and navigation details in the main text are limited.
• Writing Quality: 4/5. The division between geometry and appearance is clear; some implementation details live in the supplementary material, and corrupted equations in the cache require caution.
• Value: 4/5. Offers a practical route to persistent spatial representations for multi-room generation, while fully automatic and physically credible environment generation remains further off.