SceneHI: High-Resolution 3D-Consistent Scene Texturing with Controllable Illumination

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Thanos-DB/SceneHI
Area: 3D Vision
Keywords: scene texturing, high-resolution texture, 3D consistency, shadow baking, diffusion model

TL;DR

Addressing view inconsistency, limited resolution, and missing physical illumination in complex multi-object scene texturing, SceneHI introduces a training-free framework driven by analytical pixel-to-texel mapping, high-resolution latent textures (HRLTs), patch-based refinement, and geometry-consistent shadow baking, achieving photorealistic, multi-view consistent, and relightable textures.

Background & Motivation

Photorealistic, high-fidelity textures are fundamental to immersive experiences in video games, extended reality (XR), and digital twins. However, manual creation of high-quality textures is prohibitively labor-intensive, while the high cost of acquiring detailed 3D assets has prevented the assembly of massive, high-quality 3D scene datasets required for end-to-end training of generative texturing models. Although 2D diffusion foundation models demonstrate extraordinary generative priors in image synthesis, existing 3D texturing efforts have predominantly focused on isolated single objects. When extended to full indoor scenes composed of multiple interconnected meshes, severe inter-object occlusions, disparate geometric scales, cross-object stylistic discrepancies, and viewpoint-dependent lighting issues escalate dramatically.

Existing scene-level texturing methods suffer from severe architectural limitations. Optimization-based frameworks such as SceneTex rely on iterative multi-view back-propagation, which is exceedingly slow (taking over 20 hours per scene) and produces blurry, low-frequency artifacts in occluded surfaces due to weak cross-view gradient constraints. Panoramic projection approaches like RoomTex introduce pronounced spherical distortions and lack native per-object UV maps, rendering them incompatible with standard graphics engines and DCC tools like Blender. Meanwhile, iterative inpainting pipelines such as RoomPainter lack global spatial awareness across disjointed viewpoints, inevitably producing visible boundary seams, texture repetition, and Janus-face artifacts. Crucially, existing generative methods completely overlook a core requirement of production-grade lightweight rendering pipelines (e.g., mobile XR and Web3D): pre-computed, physically plausible baked illumination. As a result, prior generated scenes either exhibit arbitrary, chaotic shading artifacts or appear flat and ungrounded.

This work addresses these challenges by fully harnessing 2D diffusion priors through an exact analytical perspective mapping between screen pixels and UV texels, establishing a deterministic bidirectional render/inverse-render loop across views without model fine-tuning. This architecture completely prevents seam artifacts caused by sequential inpainting, while modularly decoupling global base texture generation, localized patch-based super-resolution, and geometry-consistent shadow baking. Core idea: by establishing an exact analytical pixel-to-texel mapping and High-Resolution Latent Textures (HRLTs) in UV space, SceneHI enforces 3D consistency across views via joint multi-view denoising with normal-based inverse rendering and mode-based forward rendering, while modularly integrating High-Definition Texture Refinement (HDTR) and controllable Geometry Consistent Shadow (GCS) baking.

Method

Overall Architecture

SceneHI operates as a modular, three-stage generative pipeline. First, given an untextured 3D scene mesh alongside global scene prompts and object-specific prompts, the system performs global base texture synthesis via synchronized multi-view denoising, covering the full environment and handling occlusions without optimization. Second, an optional High-Definition Texture Refinement (HDTR) pass applies zoomed-in patch-level diffusion with cosine-decayed low-resolution guidance to significantly enhance texel density on designated high-priority objects. Third, a Geometry Consistent Shadow (GCS) stage renders differentiable shadow maps from arbitrary virtual light placements and diffuses them into physically plausible, soft contact shadows baked directly into the final per-object UV atlases. The end-to-end data flow is depicted below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Untextured 3D Scene Meshes<br/>+ Global & Local Text Prompts"] --> B["Analytical Pixel-to-Texel Mapping & HRLT Init<br/>Homogeneous depth interpolation + variance-preserving scaling"]
    B --> C["Multi-View Parallel Denoising & Sync Loop<br/>Joint screen-space diffusion + normal inverse rendering + mode rendering"]
    C --> D["Per-Object Occlusion Inpainting Pass<br/>Freeze global base + localized viewpoint denoising for occluded surfaces"]
    D --> E["High-Definition Texture Refinement (HDTR)<br/>Zoomed-in overlapping patch diffusion + cosine guidance decay"]
    E --> F["Geometry Consistent Shadow Baking (GCS)<br/>Differentiable shadow priors + diffusion harmonization & UV baking"]
    F --> G["Final Production-Ready High-Res UV Texture Atlases"]

Key Designs

1. Analytical pixel-to-texel mapping and deterministic latent synchronization: preserving i.i.d. noise without resampling drift

Conventional point-sampling or rasterized re-projection methods inevitably suffer from sampling gaps, moiré patterns, and aliasing, while spatial blur filtering disrupts the independent and identically distributed (i.i.d.) noise assumption inherent to diffusion models. SceneHI exploits an exact geometric invariance of perspective projection: for any planar triangle projected onto screen space, the reciprocal of homogeneous depth \(1/w\) and the perspective-corrected texture coordinates \(u/w, v/w\) vary strictly linearly across screen space. By linearly interpolating \((u, v)\) and \(w\) at triangle vertices during rasterization and applying the quotient rule, screen-space gradients of \(u/w\) and \(v/w\) are derived analytically without higher-order truncation errors.

To establish pixel footprints in texture space, the algorithm evaluates \((u/w)/(1/w)\) and \((v/w)/(1/w)\) at the four corners of each screen-space latent pixel, forming a quadrilateral (quad) footprint \(\Omega_{ij}\) in UV space. Because standard diffusion initializes from standard Gaussian noise, directly averaging texels within a large quad back into pixel space causes the latent variance to collapse toward zero due to the central limit effect. To prevent this statistical breakdown, SceneHI introduces a variance-preserving initialization scheme:

\[\mathcal{R}_{\text{init}}(\Omega_{ij}) = \sqrt{|\Omega_{ij}|} \cdot \left( \frac{1}{|\Omega_{ij}|} \sum_{(u,v) \in \Omega_{ij}} t(u,v) \right) = \frac{1}{\sqrt{|\Omega_{ij}|}} \sum_{(u,v) \in \Omega_{ij}} t(u,v)\]

Multiplying by \(\sqrt{|\Omega_{ij}|}\) strictly maintains unit variance in screen space, preserving the Gaussian distribution required for stable diffusion sampling.

2. Multi-view parallel denoising with normal-based and mode-based aggregation: eliminating seams and low-pass blur

To bypass the seam artifacts and directional drift common in autoregressive inpainting, SceneHI denoises \(M\) camera viewpoints concurrently at every diffusion timestep. Individual screen-space predictions inevitably exhibit minor latent discrepancies across views. The framework uses the inverse rendering operator \(\mathcal{R}^{-1}\) to project screen-space latents back into the shared High-Resolution Latent Texture (HRLT). When multiple camera frustums overlap onto the same texel, naive arithmetic averaging acts as a destructive low-pass filter, suppressing high-frequency details into blurry washes.

To maintain texture sharpness, SceneHI selects latent contributions during inverse rendering based on camera-aligned surface normals: among all viewpoints observing texel \((u, v)\), the algorithm selects the latent value from the view whose surface normal has the largest component along the camera viewing direction:

\[n^* = \arg\max_{n \in \{1,\dots,N\}} \tilde{n}^{(n)}(i,j)\]

Conversely, when forward-rendering \(\mathcal{R}\) from HRLT back to screen space for the subsequent denoising timestep, the framework employs mode-based selection (majority voting) across the quad footprint \(\Omega_{ij}\):

\[\mathcal{R}(\Omega_{ij}) = \arg\max_{z \in \mathcal{V}} \sum_{(u,v) \in \Omega_{ij}} \mathbb{I}[t(u,v) = z]\]

Combining surface normal selection in inverse rendering with mode-based selection in forward rendering completely suppresses oblique perspective stretching and prevents multi-view blurring, enforcing sharp, 3D-consistent features.

3. High-Definition Texture Refinement (HDTR): balancing global stylistic coherence with local texel density

Due to GPU memory constraints and diffusion model input limits (typically \(128 \times 128\) in latent space for \(1024 \times 1024\) image resolution), global scene views fail to allocate sufficient texel density to intricate surfaces. Rather than re-running the entire scene at excessive computational expense, the HDTR stage allows users to select high-priority objects (such as detailed rugs or ornate artwork) for localized patch-based refinement. The algorithm zooms the camera frustum into target surfaces and performs patch-level denoising with spatial overlaps, mapping fine-grained details directly into higher-resolution per-object UV atlases.

To prevent local patches from drifting away from the established global color palette and stylistic context, HDTR injects low-resolution guidance latents \(z_{\text{guid}}\) derived from the base texture stage. The blending ratio follows a cosine decay schedule over diffusion timesteps \(\tau \in [1, K]\):

\[\hat{z}_\tau = \gamma_\tau z_{\text{guid}} + (1 - \gamma_\tau) z_\tau, \quad \text{where } \gamma_\tau = \left( \frac{1 + \cos(\pi \tau / K)}{2} \right)^\alpha\]

In the early diffusion phase (\(\tau\) large), guidance dominates to lock in macro geometry and coarse appearance; as denoising advances, \(\gamma_\tau\) smoothly diminishes, empowering the model to generate rich, micro-surface details without global deviation.

4. Geometry Consistent Shadows (GCS): transforming differentiable ray shadows into photorealistic baked illumination

In real-world graphics pipelines, baked shadows are essential for performant real-time rendering. However, raw analytically rendered shadow maps feature sharp, pixelated, and unnatural edges that fail to capture ambient occlusion, soft contact shadows, and realistic light diffusion. SceneHI leverages raw shadow maps generated via differentiable shadow rendering as geometric structural priors, refining them into soft, photorealistic illumination using a shadow-aware generative diffusion pass.

Given arbitrary user-defined light source coordinates, analytically rendered shadow maps \(s\) are blended with the unshadowed textured views \(v\) via intensity parameter \(w \in [0, 1]\):

\[s_v = s \cdot w + v \cdot (1 - w)\]

The blended views receive injected noise and undergo a short-step diffusion pass guided by ControlNet geometry conditioning. This transforms hard step-function shadow boundaries into smooth, physically plausible penumbrae while naturally blending ambient light into underlying material albedos. A final inverse rendering pass \(\mathcal{R}^{-1}\) bakes the illumination directly into the persistent UV maps, producing production-ready textured assets compatible with any standard graphics engine.

Key Experimental Results

Main Results

Evaluation was conducted on the standard 3D-FRONT benchmark across 10 complex multi-object indoor rooms, each tested with 2 distinct prompts across 25 unseen camera trajectories. Baseline comparisons include optimization-based SceneTex, panoramic pipeline RoomTex, and single-object baseline Text2Tex (evaluated in holistic mode Text2Tex-H and per-object prompt mode Text2Tex-C). Quantitative metrics comprise Aesthetic Score (AS), Inception Score (IS), CLIP Score (CS), and total wall-clock generation time (Minutes).

Method	AS ↑	IS ↑	CS ↑	Minutes ↓
Text2Tex-H [ICCV 2023]	4.10	2.06	23.4	8
Text2Tex-C [ICCV 2023]	4.22	2.15	23.7	75
SceneTex [CVPR 2024]	4.66	2.07	27.4	1320
RoomTex [ECCV 2024]	4.67	2.07	25.3	1200
SceneHI (Full Model, Ours)	5.04	2.24	28.8	240

To evaluate aspects where 2D metrics fall short—such as spatial 3D consistency, fine micro-details, and realistic shading—a double-blind user study was conducted with 20 participants rating rendered fly-through videos on a 1–5 scale (mean ± std):

Metric	Text2Tex-H	Text2Tex-C	SceneTex	RoomTex	Ours (SceneHI)
Multi-View Consistency ↑	2.72 ± 0.64	2.94 ± 0.58	3.31 ± 0.51	3.82 ± 0.42	4.45 ± 0.35
Fine Detail ↑	2.15 ± 0.68	2.65 ± 0.62	3.12 ± 0.55	3.34 ± 0.48	4.62 ± 0.38
Realistic Shadows ↑	1.45 ± 0.52	1.68 ± 0.58	2.34 ± 0.62	2.62 ± 0.66	4.68 ± 0.42

Ablation Study

The independent contributions of the Geometry Consistent Shadow (GCS) stage and the High-Definition Texture Refinement (HDTR) stage were quantified under identical base sampling parameters:

Config	AS ↑	IS ↑	CS ↑	Note
Full Model	5.04	2.24	28.8	Full pipeline including base texture, HDTR, and GCS
w/o Shadows	4.88	2.15	27.9	Omits GCS; textures lack baked geometric shadows
w/o Refinement	4.90	2.19	28.2	Omits HDTR; uses only base multi-view resolution

Key Findings

• Impact of Baked Illumination: Removing the GCS module lowers AS from 5.04 to 4.88 and CS from 28.8 to 27.9. Even when text prompts contain no lighting-related keywords, human evaluators and aesthetic scoring models heavily penalize unshadowed scenes, proving that physically grounded lighting is essential for perceived spatial realism.
• Substantial Generation Speedup: By replacing iterative per-scene optimization and heuristic inpainting with deterministic analytical synchronization, SceneHI reduces generation runtime by over 80% (from 20–22 hours in RoomTex and SceneTex down to 4 hours on a single NVIDIA RTX A6000 GPU).
• Long-Range Texture Consistency: In extreme occlusion scenarios (e.g., undersides of tables, sofa backs, and dense furniture clusters), normal-prioritized inverse rendering and mode selection prevent texture blurring and seam tearing, maintaining crisp, aligned textures across opposite 180-degree views.

Highlights & Insights

• Analytical Perspective Mapping over Resampling Heuristics: Exploiting the strict screen-space linearity of homogeneous coordinates \((u/w, v/w, 1/w)\) eliminates numerical drift and holes without requiring expensive optimization or post-hoc heuristic hole filling.
• Variance-Preserving Latent Scaling: Introducing the \(\sqrt{|\Omega_{ij}|}\) scaling factor during Gaussian latent initialization resolves the mathematical degradation where multi-texel averaging causes latent variance to collapse to zero.
• Synergistic Normal Selection and Mode Aggregation: Utilizing surface normal angles to filter out oblique viewing distortions during inverse projection, paired with mode-based voting during forward projection, builds a discrete high-frequency filter that preserves sharp surface textures.
• Controllable Generative Lighting Integration: Bridging classical computer graphics shadow mapping with diffusion priors allows synthetic hard shadows to be softened into realistic ambient contact shadows, offering a modular blueprint for production-ready asset creation.

Limitations & Future Work

• Computational Overhead: While 80% faster than prior optimization-based pipelines, 4 hours per scene remains too slow for interactive or real-time authoring workflows. Exploring few-step rectified flow distillation could substantially reduce inference latency.
• Illumination Complexity: GCS currently models direct illumination from point/spot sources. It does not explicitly model complex indirect global illumination (such as diffuse color bleeding) or separate specular view-dependent BRDF components from diffuse albedo.
• Sensitivity to Mesh Topology: The analytical mapping relies on consistent surface normals. Meshes with severe non-manifold artifacts or inverted face normals require adaptive gradient clamping to prevent quad distortion.

Related Work & Insights

• vs SceneTex: SceneTex optimizes texture maps via multi-view iterative backpropagation, resulting in slow runtimes (>22 hours) and blurry occluded surfaces; SceneHI utilizes deterministic analytical forward/inverse projection loops, covering occluded areas rapidly without backpropagation.
• vs RoomTex: RoomTex relies on panoramic projection that introduces severe spherical distortions and lacks native UV map support; SceneHI operates directly in per-object UV spaces, integrating seamlessly into downstream graphics engines like Blender and Unreal Engine.
• vs RoomPainter: RoomPainter uses attention-guided multi-view inpainting without isolating individual objects, often leaving gaps between tightly clustered furniture; SceneHI decouples holistic scene views and per-object localized refinement passes, ensuring complete geometric coverage.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers exact analytical perspective mapping and controllable generative shadow baking in a training-free 3D scene texturing pipeline]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation on 3D-FRONT, detailed ablations, and a 20-participant double-blind video study]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous mathematical formulation, clear structural exposition, and compelling visual demonstrations]
• Value: ⭐⭐⭐⭐⭐ [Bridges generative AI and industrial 3D graphics pipelines by generating high-resolution, UV-mapped, shadow-baked scene assets]