title: >- [Paper Note] CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance description: >- [ECCV 2026][3D Vision][3D Gaussian Splatting] Proposes CoIn, a bidirectional framework bridging 2D diffusion generation and explicit 3DGS via Reference-Adaptive GS (Ref-GS), Consistency Loss Guidance (CLG) on intermediate warped features, and a Texture-Enhancing Discriminator (TE-D) for robust multi-view object removal and insertion with arbitrary masks. tags: - ECCV 2026 - 3D Vision - 3D Gaussian Splatting - 3D Scene Inpainting - Diffusion Guidance date: 2026-09-19 content_hash: 77fa25abf49e5039

CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance¶

Conference: ECCV 2026
Paper: ECCV Official Poster
PDF: ECCV Open Access
Area: 3D Vision
Keywords: 3D Scene Inpainting, 3D Gaussian Splatting, Diffusion Guidance, Multi-View Consistency, Object Insertion & Removal

TL;DR¶

CoIn establishes a bidirectional collaborative framework bridging 2D latent diffusion generation and 3D Gaussian Splatting (3DGS). By utilizing Reference-Adaptive GS (Ref-GS) to establish coarse geometry, Consistency Loss Guidance (CLG) via GS-based reference feature warping to enforce cross-view coherence during 2D diffusion denoising, and a patch-based Texture-Enhancing Discriminator (TE-D) to restore high-frequency fidelity, CoIn supports arbitrary-shaped masks (including loose bounding boxes) across both object removal and object insertion tasks.

Background & Motivation¶

Reconstructing incomplete or corrupted 3D scenes caused by occlusions, sensor noise, or limited viewpoints is a core problem in 3D vision, virtual reality, and spatial computing. Early NeRF-based inpainting approaches suffered from high computational cost, prolonged optimization schedules, and the difficulty of directly manipulating implicit volumes. With the emergence of 3D Gaussian Splatting (3DGS), contemporary methods (such as IMFine, 3DGIC, and AuraFusion) predominantly embrace a "3D-first" pipeline: they first build a 3DGS scene from captured images, leverage multi-view 2D segmentation masks (e.g., from SAM2) to isolate and prune target Gaussians directly in 3D space, and then fill the missing geometry and appearance via spatial smoothing or reference view reprojection. However, this strategy is fragile in practice. Across dense camera perspectives, even slight boundary misalignments in 2D segmentation masks cause severe spatial inconsistencies, leading to the erroneous pruning of intact background structures. Furthermore, the prune-then-inpaint paradigm is fundamentally locked to object removal and cannot accommodate object insertion or coarse inputs such as bounding-box masks.

Conversely, "2D-first" pipelines attempt to perform 2D inpainting across multiple views using powerful generative models (such as Stable Diffusion Inpainting) before enforcing consistency through post-hoc 3D priors like optical flow or coarse surface meshes. Nonetheless, due to the inherent stochasticity of multi-step diffusion sampling, independently inpainted views inevitably suffer from noticeable visual discrepancies and structural jitter. Feeding these inconsistent 2D observations directly into 3DGS optimization leads to floating artifacts, double vision, and catastrophic blurriness. In addition, existing consistency-tuned multi-view diffusion models frequently demand extensive dataset-specific retraining, severely restricting generalizability across arbitrary mask geometries.

CoIn addresses these challenges by replacing one-way pipelines with a tightly correlated bidirectional information flow between 2D generative diffusion and explicit 3DGS representations. The core idea is to initiate inpainting in 2D to naturally support arbitrary masks and insertion tasks, construct a coarse 3D geometry via Reference-Adaptive GS (Ref-GS), inject 3D geometry priors into the 2D diffusion sampling process via intermediate-pose GS feature warping (CLG) without additional network retraining, and finally employ a patch-level Texture-Enhancing Discriminator (TE-D) to decouple geometric smoothness from high-frequency photometric realism.

Method¶

Overall Architecture¶

The input to CoIn comprises a multi-view image sequence $\{I_n\}_{n=1}^N$ along with corresponding target masks $\{M_n\}_{n=1}^N$ (which can be either fine segmentation masks or loose bounding boxes). The entire pipeline executes across four collaborative stages: First, an initial 2D inpainting model produces unconstrained completions, lifting restrictions on mask shapes and enabling novel object synthesis. Second, in a 2D $\to$ 3D coarse stage, Reference-Adaptive GS with Feature Attention (Ref-GS) optimizes a coarse 3DGS scene biased toward a designated reference view and extracts a clean point cloud. Third, in a 3D $\to$ 2D guidance stage, Consistency Loss Guidance (CLG) projects both reference and intermediate diffusion latents onto an interpolated virtual camera pose to guide the diffusion sampling via energy minimization. Finally, in a 2D $\to$ 3D refinement stage, a local Texture-Enhancing Discriminator (TE-D) works adversarially with 3DGS to infuse sharp high-frequency textures into the geometrically aligned scene.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multi-view Images & Masks<br/>{I_n, M_n}"] --> Init["Initial 2D Inpainting<br/>Generate Initial Reconstructions {Î_n}"]
    Init --> RefGS["Reference-Adaptive GS & Feature Attention<br/>Adaptive View Weight W_n + Anchor Unidirectional Attention"]
    RefGS --> Cloud["Reconstruct Coarse 3D Scene & Point Cloud P_k"]
    Cloud --> CLG["Consistency Loss Guidance (CLG)<br/>Intermediate View Feature Warping + 3D Energy Minimization"]
    CLG --> Consist["Consistent 2D Inpainted Views {Ĩ_n}"]
    Consist --> TED["Texture-Enhancing Discriminator (TE-D)<br/>Patch-level Adversarial Training + Photometric Fine-tuning"]
    TED --> Out["High-Fidelity Consistent 3DGS Scene 𝒢"]

Key Designs¶

1. Reference-Adaptive GS with Feature Attention (Ref-GS): Anchoring Coarse 3D Geometry to a Reference View Directly training a vanilla 3DGS model on independently inpainted initial 2D images $\{\hat{I}_n\}_{n=1}^N$ leads to conflicting gradient signals and severe view-dependent blur. To circumvent this, CoIn builds its coarse scene upon the lightweight anchor-based Scaffold-GS architecture. Pruning is confined solely to the initial sparse COLMAP points whose projections fall inside the masks to remove residual target objects. A representative view $\hat{I}_k$ is designated as the primary reference image. To suppress gradient conflicts from diverging viewpoints, dynamic loss weights $W_n$ are applied to the photometric rendering loss: $$W_n = \begin{cases} \lambda_r, & \text{if } n = k \\ \exp\left(-\lambda \left(\mathcal{L}_R^{(n,t)} - \mathcal{L}_R^{(n,t-1)}\right)\right), & \text{if } n \neq k \end{cases}$$ where $\lambda_r$ denotes the reference view boost factor and the exponential term penalizes viewpoints that exhibit unstable or diverging photometric errors $\mathcal{L}_R$ relative to the preceding iteration. In parallel, unidirectional cross-attention is enforced across Scaffold-GS anchor features across all training views: $$f_{in}^A \leftarrow \text{Attn}\left(Q = f_{in}^A, K = f_{side}^A, V = f_{side}^A\right)$$ which smoothly propagates anchor features $f_{side}^A$ from the surrounding intact geometry into the interior anchor representations $f_{in}^A$. Coupled with monocular depth supervision $D_k$ derived from the reference view, this yields a geometrically stable reference point cloud $P_k$.

2. Consistency Loss Guidance (CLG): Energy-Based 2D Diffusion Control via GS Feature Warping To eliminate stochastic appearance drift in 2D diffusion models without task-specific retraining, CoIn introduces an energy-based guidance mechanism inspired by FreeDoM. At diffusion denoising timestep $t$, the intermediate latent $z_{(n,t)}$ is decoded to image space as $X_{(n,t)} = \text{Dec}(z_{(n,t)})$. Utilizing the reference point cloud $P_k$, the features of reference view $\hat{I}_k$ and current latent image $X_{(n,t)}$ are warped to an interpolated intermediate camera pose $\phi$. The virtual viewpoint $\phi$ interpolates translation linearly and rotation via spherical linear interpolation (Slerp) with $\alpha = 0.5$, substantially mitigating extreme parallax distortions. In this intermediate view, high-frequency structural features extracted via FeatUp-enhanced DINO (dino16) and RGB appearance features are evaluated: $$\mathcal{L}_{3D} = -\cos\left(f_\phi^{hr}(\hat{I}_k), f_\phi^{hr}(X_{(n,t)})\right) + \|f_\phi^{rgb}(\hat{I}_k) - f_\phi^{rgb}(X_{(n,t)})\|_2^2$$ To handle visual disocclusions and non-overlapping perspective regions, a point cloud support mask $m_n^P$ indicates valid 3D correspondence zones. For non-supported areas, the current 3DGS rendering $R_n$ acts as a fallback prior, formulating the final energy objective: $$\mathcal{E}(X_{(n,t)}) = m_n^P \cdot \mathcal{L}_{3D} + (1 - m_n^P) \|X_{(n,t)} - R_n\|_1$$ Minimizing $\mathcal{E}(X_{(n,t)})$ at each diffusion reverse step aligns all view syntheses $\{\tilde{I}_n\}_{n=1}^N$ into strict geometric and appearance consensus with zero architectural weight changes.

3. Texture-Enhancing Discriminator (TE-D): Adversarial Synthesis of High-Frequency Surface Details While CLG guarantees multi-view consistency, geometric regularization and feature warping inevitably introduce slight over-smoothing, leaving the resulting 3DGS renderings lacking sharp high-frequency textural micro-details. To restore realism, CoIn integrates a localized patch discriminator (PatchGAN) during fine-stage 3DGS optimization. Operating on $64 \times 64$ patches extracted probabilistically around the target region, TE-D treats patches from the initial 2D inpainting outputs $\hat{I}_n$ (which contain rich texture distributions despite lacking 3D consistency) as real, and patches rendered from the current 3DGS scene $R_n$ as fake. Simultaneously, the 3DGS model continues to minimize photometric rendering loss and depth loss supervised by the geometrically consistent CLG images $\tilde{I}_n$: $$\mathcal{L} = \mathcal{L}_R(R_n, \tilde{I}_n) + \lambda_D \mathcal{L}_D + \lambda_{gen}\mathcal{L}_{gen}(R_n, \hat{I}_n)$$ This design explicitly decouples global geometric structure from localized high-frequency texture synthesis: large-scale alignment is governed by $\tilde{I}_n$, while micro-textures are restored via adversarial distribution matching against $\hat{I}_n$.

Loss & Training¶

The overall training pipeline is organized into a two-phase optimization schedule. In the coarse Ref-GS phase, Scaffold-GS is trained using weighted photometric loss and reference depth constraints while pruning masked COLMAP points. In the CLG phase, Stable Diffusion 2.0 operates with frozen weights on $512 \times 512$ crops, while DINO/FeatUp features are computed on $256 \times 256$ resolution for GPU memory efficiency. In the fine TE-D phase, the 3DGS generator is first fine-tuned on consistent CLG images $\tilde{I}_n$ for 10k iterations, followed by 10k iterations of joint adversarial training with TE-D. Executed entirely on a single NVIDIA RTX 4090 GPU (24GB), the complete workflow finishes within approximately 2.5 hours per scene.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on two standard benchmarks: SPIn-NeRF (10 scenes, 60 object-present and 40 object-absent images, evaluating novel-view synthesis at object-absent viewpoints) and IMFine (10 representative scenes spanning camera coverage of 90°, 120°, and 180°). Metrics evaluate perceptual similarity (LPIPS), structural quality (PSNR), and distributional realism (FID) across full images and masked regions (denoted by prefix m-).

Table 1: Quantitative evaluation on the SPIn-NeRF dataset (Original Paper Table 1)

Representation	Method	Mask Type	m-LPIPS (↓)	LPIPS (↓)	m-FID (↓)	FID (↓)
NeRF	SPIn-NeRF	Seg mask	0.053	0.31	153.4	49.6
NeRF	MVIP-NeRF	Seg mask	0.050	0.31	173.4	50.5
NeRF	MALD-NeRF	Seg mask	0.031	0.30	113.5	44.7
3DGS	Gaussian Grouping	Seg mask	0.037	0.26	132.5	44.9
3DGS	3DGIC	Seg mask	0.028	0.26	96.3	36.4
3DGS	GScream	Seg mask	0.032	0.26	86.1	31.2
3DGS	Ours (CoIn)	Seg mask	0.032	0.23	80.4	28.9
3DGS	3DGIC	BB mask	0.047	0.38	168.0	104.1
3DGS	GScream	BB mask	0.043	0.29	104.6	34.2
3DGS	Ours (CoIn)	BB mask	0.033	0.24	95.1	26.6

Table 2: Quantitative results on the IMFine dataset (Original Paper Table 2)

Method	LPIPS (↓)	PSNR (↑)	FID (↓)
3DGIC	0.3386	20.29	200.99
GScream	0.2000	22.68	112.91
IMFine	0.1747	23.59	57.53
Ours (CoIn)	0.1685	23.88	53.37

Table 3: Quantitative evaluation of object insertion (Original Paper Table 3)

Method	CLIPdir (↑)	Consistency (%)	Visual (%)	Align with T.P. (%)	w/o Artifacts (%)
Gaussian Editor	0.0222	20.825	23.625	13.900	17.000
Infusion	0.1589	31.950	31.950	26.975	30.550
Ours (CoIn)	0.1628	47.225	44.425	59.125	45.400

Ablation Study¶

The impact of each architectural design is evaluated on the SPIn-NeRF benchmark, isolating the contributions of Ref-GS (with adaptive weighting and feature attention decoupled), CLG, and TE-D.

Table 4: Ablation study on the SPIn-NeRF dataset (Original Paper Table 4)

Configuration	m-LPIPS (↓)	LPIPS (↓)	m-FID (↓)	FID (↓)	Description
w/o Ref-GS	0.0359	0.2534	87.05	30.47	Removes reference-adaptive coarse reconstruction
w/o Adaptive Weight	0.0356	0.2571	82.07	30.76	Omits view-dependent dynamic loss weighting
w/o Feature Attention	0.0355	0.2559	88.15	31.78	Removes unidirectional anchor feature attention
w/o CLG	0.0428	0.3057	97.04	36.47	Omits consistency loss guidance (causes jitter artifacts)
w/o TE-D	0.0535	0.2902	132.55	39.31	Removes patch discriminator (severe over-smoothing)
Full model (Ours)	0.0317	0.2349	80.44	28.92	Full integrated pipeline achieves best performance

Key Findings¶

High Invariance to Mask Morphology: When switching from fine segmentation to bounding-box masks in Table 1, 3DGIC's FID deteriorates drastically from 36.4 to 104.1 (due to over-pruning of intact scene geometry), whereas CoIn maintains steady scores of 26.6 (full FID) and 95.1 (m-FID). This verifies that the 2D-first guidance pipeline fundamentally circumvents the mask-sensitivity bottleneck inherent to 3D pruning approaches.
TE-D Counteracts Guidance Over-Smoothing: Disabling TE-D results in the steepest degradation in masked distributional realism, sending m-FID from 80.44 up to 132.55 (an increase of over 64%). This demonstrates that local patch-level adversarial matching is indispensable for recovering fine, realistic textures without compromising multi-view geometry.
CLG Secures 3D Geometric Stability: Without CLG guidance, LPIPS degrades severely to 0.3057. Qualitative depth inspections further confirm that optimizing 3DGS on unguided multi-view diffusion outputs yields severe floating artifacts and broken surface geometry.

Highlights & Insights¶

Interpolated Intermediate Pose Warping: In CLG, warping features from both the reference camera and the current camera to an intermediate interpolated virtual viewpoint $\phi$ significantly suppresses projection stretch and occlusive distortion, facilitating robust semantic alignment via DINO and RGB representations.
Decoupled Geometry and Texture Adversarial Optimization: Utilizing unguided 2D inpainting outputs as "real" references in a localized PatchGAN framework neatly solves the persistent dilemma in 3D inpainting between rigid multi-view geometric smoothness and high-frequency textural detail.
Unified Object Removal and Insertion: By operating in a generative 2D-first manner rather than relying on explicit 3D Gaussian subtraction, CoIn naturally accommodates both object removal and prompt-guided object insertion under a singular architectural formulation.

Limitations & Future Work¶

Reliance on a Single Reference View: The framework currently selects one primary reference viewpoint $\hat{I}_k$. In full 360° unbounded scenes with severe self-occlusions, regions opposite the reference view receive attenuated guidance, necessitating extension toward multi-reference clustering.
Computational Overhead of the Multi-Stage Pipeline: Sequential execution across initial diffusion, coarse 3DGS training, diffusion guidance, and joint adversarial refinement takes ~2.5 hours per scene, leaving room for acceleration toward real-time interactive editing.
Potential Geometry Drift under Extreme Surface Complexities: Monocular depth estimation and coarse point cloud projection can struggle on intricate, non-Lambertian surfaces, occasionally impacting the fidelity of intermediate feature warping.

vs 3DGIC / IMFine: 3DGIC and IMFine utilize a 3D-first strategy requiring accurate multi-view masks to prune Gaussians, causing excessive scene removal under loose bounding boxes and failing on insertion tasks. CoIn utilizes a 2D-first bidirectional pipeline robust to coarse masks and capable of diverse edits.
vs GScream: GScream relies on a single 2D reference without explicit cross-view guidance during diffusion, leading to noticeable inconsistencies across wide baselines. CoIn enforces explicit 3D consistency guidance across all views via intermediate-view feature warping.
vs Infusion / Gaussian Editor: Prior diffusion-guided 3D editors often exhibit saturated colors or distorted geometries during object insertion. CoIn achieves significantly higher text alignment (59.1% preference) and multi-view consistency (47.2% preference) through its decoupled patch adversarial formulation.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Elegantly bridges 2D generative diffusion and 3DGS via intermediate-pose feature warping and decoupled patch adversarial refinement)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive evaluations across SPIn-NeRF and IMFine, mask ablation from segmentation to bounding box, depth map verification, and user study)
Writing Quality: ⭐⭐⭐⭐⭐ (Well-structured paper with lucid motivation, clear formulations, and high-quality figures)
Value: ⭐⭐⭐⭐☆ (Resolves the long-standing mask-brittleness issue in 3DGS inpainting, paving the way for practical 3D scene editing)