title: >- [Paper Note] DualDiff3D: Dual Structure-Appearance Diffusion Priors for Reliability-Enhanced 3D Gaussian Splatting description: >- [ECCV 2026][3D Vision][3DGS] Resolves structure-appearance conflicts in sparse-view 3D Gaussian Splatting via dual diffusion branches and a reliability-enhanced Render-Refine-Optimize loop. tags: - ECCV 2026 - 3D Vision - 3D Gaussian Splatting - Diffusion Priors - Novel View Synthesis date: 2026-09-19 content_hash: 513f2cab973daef7
DualDiff3D: Dual Structure-Appearance Diffusion Priors for Reliability-Enhanced 3D Gaussian Splatting¶
Conference: ECCV 2026
Paper: ECCV Official Page
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Diffusion Priors, Structure-Appearance Decoupling, Render-Refine-Optimize Loop, Novel View Synthesis
TL;DR¶
To eliminate blurring caused by conflicting structure and appearance when conditioning image diffusion on sparse reference views, DualDiff decouples them into dual diffusion branches coupled by Structure-Appearance Attention (SAA), while DualDiff3D embeds this into a reliability-enhanced Render-Refine-Optimize (RRO) loop to robustly enhance sparse-view 3DGS reconstruction.
Background & Motivation¶
3D Gaussian Splatting (3DGS) has emerged as a groundbreaking paradigm in 3D scene reconstruction and novel view synthesis (NVS), combining explicit geometric representation with efficient tile-based rasterization to deliver photorealistic rendering at real-time frame rates. However, under sparse-view input regimes, the optimization of 3DGS is severely under-constrained. Gaussian primitives in unobserved or poorly observed regions frequently suffer from severe overfitting, floaters, and anisotropic stretching, causing novel views rendered from these defective representations to exhibit pronounced blur, geometric tearing, and hole artifacts.
To overcome the lack of constraints in sparse scenarios, recent works have sought to introduce 2D or video generative diffusion priors to refine artifact-ridden novel view renderings and incorporate them back into 3DGS optimization. While video diffusion models (e.g., 3DGS-Enhancer) maintain cross-view continuity, their exorbitant computational cost and high inference latency render them unsuitable for iterative 3DGS pipelines. Consequently, single-step image diffusion approaches (such as DIFIX3D+) have been adopted for fast refinement. To condition on reference viewpoints, these models typically concatenate rendered novel views and reference views along the spatial or batch dimension within a single U-Net. However, this unified processing fundamentally ignores the intrinsic dichotomy between novel and reference perspectives: the rendered novel view provides correct viewpoint geometry and perspective cues (structure), whereas the reference view provides high-fidelity, uncorrupted textures from a different camera pose (appearance). Forcing a single network to resolve both leads to severe informational conflicts; as angular distance between views increases, cross-view attention collapses into severe blur, resulting in the counter-intuitive observation where reference-guided refinement yields worse quality than reference-free generation.
The key insight of this paper is that structural perspective extraction and appearance texture transfer must be explicitly decoupled across distinct diffusion pathways and integrated via directed attention. Core idea: develop a dual-branch diffusion pipeline (DualDiff) that disentangles structural geometry from reference appearance via Structure-Appearance Attention (SAA), embedded within a reliability-enhanced Render-Refine-Optimize loop (DualDiff3D) to iteratively and robustly reconstruct high-fidelity 3DGS scenes.
Method¶
Overall Architecture¶
DualDiff3D achieves high-fidelity 3DGS reconstruction under sparse-view settings through two synergistic systems: a single-step dual-branch refinement network (DualDiff) and a reliability-enhanced Render-Refine-Optimize (RRO) optimization framework. Given an initial defective 3DGS trained on sparse inputs, the pipeline progressively samples smooth camera trajectories, renders flawed novel views, enhances them with DualDiff using neighboring reference views, evaluates their reliability via a composite confidence mask, and feeds only geometrically coherent regions back into 3DGS optimization while guarding against degradation via validation rollbacks.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Initial Sparse 3DGS Representation"] --> B["Progressive Sampling and Filtering (PSF)<br/>Pose interpolation & jitter filtering"]
B --> C["Render Degraded Novel Views"]
C --> D["Dual-Branch Refinement (DualDiff)<br/>Structure branch + Appearance branch"]
D --> E["Confidence-Driven Weighting (CDW)<br/>Variance / discrepancy / reprojection mask"]
E --> F["Validation & Rollback 3DGS Optimization<br/>Monitored supervision on reliable regions"]
F -->|Iterative RRO convergence| G["High-Fidelity Reconstructed 3DGS"]
Key Designs¶
1. Dual-Branch Structure-Appearance Decoupled Architecture: Eliminating Cross-View Feature Interference To resolve the structural blurring caused by concatenating disparate viewpoints into a single backbone, DualDiff employs two parallel denoising U-Net branches instantiated from the single-step SD-Turbo architecture. The structure branch takes the latent code of the low-quality rendered novel view as input under a low-noise timestep (\(t=200\)), strictly preserving the spatial composition, viewing perspective, and coarse geometry of the target novel view. Concurrently, the appearance branch processes the closest ground-truth reference image to extract crisp, artifact-free textures and semantic details. To prevent high-frequency detail loss inherent to VAE downsampling, zero-initialized convolutional skip connections are introduced between corresponding encoder and decoder stages of the structure branch, while the VAE decoder is fine-tuned with LoRA.
2. Structure-Appearance Attention (SAA) Module: Directed Reference Feature Injection To inject appearance cues into the structural backbone without corrupting novel view geometry, the standard self-attention layer in the structure branch U-Net is replaced with the SAA module. Denoting the hidden states from the appearance branch self-attention as \(\mathbf{H}_{\text{app}} \in \mathbb{R}^{n \times d}\) and those from the structure branch as \(\mathbf{H}_{\text{str}} \in \mathbb{R}^{m \times d}\), they are concatenated along the sequence length dimension into a joint representation \(\mathbf{H}_{\text{mix}} = [\mathbf{H}_{\text{str}} ; \mathbf{H}_{\text{app}}] \in \mathbb{R}^{(m+n) \times d}\). The queries \(\mathbf{Q}\) are projected solely from the structure states to preserve target view focus, while keys \(\mathbf{K}\) and values \(\mathbf{V}\) are derived from the combined representation:
This asymmetric attention allows each novel view location to selectively query and warp relevant textures from the reference view. Remarkably, this decoupled architectural design acts as a "free lunch": even when both branches directly load pre-trained DIFIX weights without any retraining (inference-only), SAA produces an immediate +0.7 dB PSNR gain over DIFIX.
3. Progressive Sampling and Filtering (PSF) Strategy: Controlled Viewspace Expansion Unconstrained random viewpoint sampling across 3D space frequently places virtual cameras in unobserved frustums, inducing catastrophic hallucination. PSF addresses this by interpolating and mildly extrapolating between valid training camera poses with controlled perturbations. Each candidate viewpoint is screened at both pixel and image levels using geometric and appearance criteria, admitting only well-conditioned novel viewpoints into the training pool and pairing them with their nearest ground-truth reference views for DualDiff refinement.
4. Confidence-Driven Weighting (CDW) Process and Rollback Mechanism: Robustifying Optimization Directly backpropagating diffusion-refined outputs into explicit 3D Gaussians inevitably injects subtle geometric inaccuracies that destabilize training. CDW constructs a composite per-pixel confidence mask \(M(u) \in [0, 1]\) via three complementary uncertainty indicators: - Refine Variance Confidence \(C_{\text{var}}(u) = \exp(-\alpha \cdot \sigma^2(u))\): Measures stochastic generative uncertainty by computing per-pixel variance across \(K\) stochastic refinements; - Pixel Discrepancy Confidence \(C_{\text{diff}}(u) = \exp(-\beta \cdot d(u))\): Penalizes large absolute differences \(d(u) = \|I_{\text{refine}}(u) - I_{\text{low}}(u)\|\) against the initial rendering to suppress spurious hallucinations; - Reprojection Confidence \(C_{\text{proj}}(u) = \exp(-\gamma \cdot e_{\text{proj}}(u))\): Uses the 3DGS rendered depth map \(D_{\text{novel}}\) to reproject pixels into overlapping reference views \(j\), penalizing photometric reprojection error \(e_{\text{proj}}(u)\) that violates epipolar consistency.
The composite mask \(M(u) = \lambda_1 C_{\text{var}}(u) + \lambda_2 C_{\text{diff}}(u) + \lambda_3 C_{\text{proj}}(u)\) adaptively scales the optimization loss:
Additionally, the Gaussian scene parameters are cached at every refinement checkpoint (every \(2\text{k}\) iterations). During the subsequent validation window, if performance on a held-out set drops across 3 consecutive checks, the system rolls back to the cached state and discards the corrupted novel view batch, guaranteeing strictly monotonic 3D representation improvement.
Loss & Training¶
The DualDiff refinement model is trained end-to-end on DL3DV-10K triplets using a linear combination of \(L_2\) pixel reconstruction loss \(\mathcal{L}_{\text{Recon}}\), perceptual LPIPS loss \(\mathcal{L}_{\text{LPIPS}}\), and a VGG-16 Gram-matrix style loss \(\mathcal{L}_{\text{Gram}}\):
with \(\lambda_0 = 0.1\). The model is optimized using Adam (\(\text{lr} = 2 \times 10^{-5}\), batch size 8) for \(10\text{k}\) iterations. In the 3DGS RRO loop, hyper-parameters are calibrated to \(K=3\), \(\lambda_1=0.2\), \(\lambda_2=0.3\), and \(\lambda_3=0.5\).
Key Experimental Results¶
Main Results¶
DualDiff3D demonstrates clear superiority across all sparse-view settings (3, 6, 9, and 24 views) on both the large-scale DL3DV dataset and forward-facing LLFF scenes (from Table 2 of the original paper):
| Dataset / View Setting | Metric | Vanilla 3DGS | DIFIX3D | DualDiff3D (Ours) | Gain vs. DIFIX3D |
|---|---|---|---|---|---|
| DL3DV (3-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 10.70 / 0.2608 / 0.6001 | 11.36 / 0.3314 / 0.5759 | 12.56 / 0.3731 / 0.5473 | +1.20 dB / +0.0417 / -0.0286 |
| DL3DV (6-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 12.72 / 0.3298 / 0.5419 | 13.35 / 0.3892 / 0.5084 | 14.11 / 0.4122 / 0.4837 | +0.76 dB / +0.0230 / -0.0247 |
| DL3DV (9-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 14.19 / 0.3886 / 0.4950 | 14.96 / 0.4464 / 0.4617 | 15.60 / 0.4697 / 0.4436 | +0.64 dB / +0.0233 / -0.0181 |
| DL3DV (24-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 19.42 / 0.6209 / 0.3212 | 20.17 / 0.6469 / 0.2971 | 20.49 / 0.6576 / 0.2745 | +0.32 dB / +0.0107 / -0.0226 |
| LLFF (3-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 14.11 / 0.3819 / 0.4666 | 15.21 / 0.4645 / 0.4119 | 16.34 / 0.4981 / 0.3835 | +1.13 dB / +0.0336 / -0.0284 |
| LLFF (6-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 18.69 / 0.5687 / 0.3237 | 19.85 / 0.6159 / 0.2821 | 20.35 / 0.6709 / 0.2611 | +0.50 dB / +0.0550 / -0.0210 |
| LLFF (9-View) | PSNR ↑ / SSIM ↑ / LPIPS ↓ | 20.50 / 0.6508 / 0.2726 | 21.53 / 0.6807 / 0.2410 | 21.89 / 0.7106 / 0.2154 | +0.36 dB / +0.0299 / -0.0256 |
Furthermore, on the challenging Mip-NeRF360 benchmark (Table 4), under the 3-view configuration, DualDiff3D reaches 15.70 dB PSNR / 0.373 SSIM / 0.554 LPIPS, surpassing vanilla 3DGS (13.06 dB), FSGS (14.17 dB), DIFIX3D (14.32 dB), and GenFusion (15.29 dB).
Ablation Study¶
On the LLFF 3-view benchmark, component-wise ablation verifies the necessity of each architectural innovation (Table 3 in the original paper):
| Variant | Refine Backbone | PSF Strategy | CDW Weighting | PSNR ↑ (dB) | SSIM ↑ | LPIPS ↓ | FID ↓ | Note |
|---|---|---|---|---|---|---|---|---|
| (a) Baseline RRO | DIFIX | ✗ | ✗ | 15.21 | 0.4645 | 0.4119 | 115.23 | Naive RRO integration |
| (a) DualDiff Backbone | DualDiff | ✗ | ✗ | 15.43 | 0.4649 | 0.4031 | 107.50 | Decoupled refinement backbone |
| (b) + PSF | DualDiff | ✓ | ✗ | 15.71 | 0.4733 | 0.3910 | 105.02 | Smooth pose trajectory guidance |
| (c) + CDW | DualDiff | ✗ | ✓ | 16.04 | 0.4877 | 0.3881 | 103.71 | Pixel-level uncertainty filtering |
| (d) Full DualDiff3D | DualDiff | ✓ | ✓ | 16.34 | 0.4981 | 0.3835 | 101.25 | Optimal synergy across all modules |
In Table 5, decomposing the CDW terms demonstrates that the reprojection term \(C_{\text{proj}}\) is the single most critical confidence component, elevating PSNR from 19.97 dB (w/o CDW) to 20.26 dB when used alone. Combining all three confidence terms yields the peak score of 20.35 dB.
Key Findings¶
- Decoupling prevents catastrophic negative transfer: On LLFF, single-network DIFIX drops from 19.01 dB (w/o reference) down to 18.79 dB (w/ reference), proving that unified feature concatenation creates cross-view interference. In contrast, DualDiff achieves 19.49 dB in an inference-only setting, confirming that decoupling structure and appearance solves view conflict.
- Geometric consistency prevents floater artifacts: The reprojection confidence \(C_{\text{proj}}\) contributes the largest performance gain among CDW components (+0.29 dB PSNR), highlighting that multi-view epipolar constraints are critical to suppress out-of-distribution hallucinations.
- Fast per-step inference: By separating the reference view into an independent pathway rather than expanding the batch dimension, DualDiff reduces single-frame refinement time to 600 ms (compared to 950 ms for DIFIX, Table 6), though total 3DGS training time is slightly higher (10m40s vs. 8m52s) due to \(K=3\) stochastic sampling.
Highlights & Insights¶
- Dual-Pathway Disentanglement: Reframing novel view refinement into distinct structural extraction and appearance conditioning neatly resolves the classic dilemma of view disparity versus texture preservation in 2D diffusion priors.
- Zero-Shot "Free Lunch": By simply rewiring the attention layers into SAA, existing pre-trained diffusion weights can be leveraged without further fine-tuning to achieve immediate quantitative and perceptual gains.
- Defensive Multi-Cue Closed Loop: Combining progressive viewpoint perturbation, three-pronged uncertainty masking, and automatic validation rollback creates an exceptionally stable framework for integrating generative models into explicit 3D representations.
Limitations & Future Work¶
- Increased Memory Consumption: Running dual U-Net architectures raises peak GPU VRAM usage during refinement to 12.5 GB (a ~26% increase over DIFIX3D's 9.9 GB), limiting scalability on edge devices.
- Early-Stage Reprojection Sensitivity: In extreme few-view regimes, initial depth renderings can be inaccurate, which may lead to false-negative rejections in \(C_{\text{proj}}\) and slow down early optimization.
- Future Directions: Exploring parameter sharing or cross-view LoRA adapters to lower VRAM footprint, and incorporating pre-trained metric monocular depth estimators to provide reliable geometric grounding during early RRO iterations.
Related Work & Insights¶
- vs. DIFIX3D+: DIFIX processes both views in a single U-Net, leading to severe blur when camera distance grows; DualDiff cleanly decouples them with SAA, improving both visual sharpness and inference latency.
- vs. 3DGS-Enhancer: 3DGS-Enhancer relies on computationally heavy video diffusion models that generate temporal sequences; DualDiff3D uses single-step image diffusion priors, offering a vastly more practical pipeline for online optimization.
- vs. ControlNet / IP-Adapter: Adapts the spatial conditioning concept of ControlNet and the cross-attention style conditioning of IP-Adapter specifically to the 3D domain by casting structure as Query and mixed features as Key/Value.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Decouples structure and appearance explicitly in diffusion-based view refinement, addressing a major blind spot in multi-view conditioning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across DL3DV, LLFF, and Mip-NeRF360 with thorough component ablations and runtime benchmarking.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, clean mathematical formulation, and intuitive visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides a robust, generalizable closed-loop blueprint for integrating 2D diffusion models into 3DGS reconstruction.