GaINeR: Geometry-Aware Implicit Neural Representation for Image Editing¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Image Generation
Keywords: Implicit Neural Representations / Image Editing / Gaussian Embeddings / Physics Simulation / 2D-to-3D Lifting
TL;DR¶
GaINeR incorporates learnable Gaussian distributions into the Implicit Neural Representation (INR) framework, aggregating neighborhood features via radius-limited Gaussian-weighted KNN to condition an MLP decoder, delivering breakthrough 2D image reconstruction fidelity alongside robust, artifact-free physical deformation and single-image 3D lifting.
Background & Motivation¶
Implicit Neural Representations (INRs) formulate images as continuous coordinate-based functions parameterized by neural networks, bypassing discrete pixel grids while providing resolution independence, compact memory footprints, and smooth subpixel interpolation. Advanced architectures featuring high-frequency activations—such as SIREN, WIRE, and FINER—have significantly improved the capacity of INRs to capture fine spatial details. However, conventional INRs map global coordinates directly to visual attributes without explicit geometric structures, which severely impedes intuitive local manipulations and makes direct coupling with particle-based physical simulation engines virtually impossible.
Conversely, explicit Gaussian primitives, popularized by 3D Gaussian Splatting (3DGS) and extended to 2D domains by GaussianImage and MiRaGe, offer direct geometric manipulability and straightforward physics integration. Yet, these explicit methods fundamentally rely on discrete rasterization where each Gaussian primitive directly stores opacity and color. When physics engines subject the image to severe geometric deformations—such as bending, stretching, or cutting—discrete Gaussians detach, producing visible gaps, spiky boundaries, and transparency artifacts; shrinking Gaussian primitives alleviates this only at the cost of manual hyperparameter tuning and substantial memory overhead, without ever achieving true spatial continuity.
This paper breaks the conventional assumption that explicit primitives must serve as standalone renderers, introducing a deep synergy between explicit geometric control and continuous neural decoding: Gaussians act purely as geometric feature providers, while a coordinate-based neural decoder synthesizes continuous, coherent colors. Core idea: introduce GaINeR (Geometry-Aware Implicit Network Representation), which couples a continuous coordinate MLP decoder with a spatial distribution of trainable Gaussian components; by interpolating features via radius-limited, Gaussian distance-weighted KNN, it achieves ultra-high-fidelity continuous image representation while supporting smooth, artifact-free geometric editing and single-image 3D lifting.
Method¶
Overall Architecture¶
The GaINeR framework consists of an optimized set of learnable Gaussian embeddings, a radius-limited local neighborhood aggregator, and a coordinate decoding MLP. Given an input image, the model optimizes Gaussian primitives characterized by spatial means, covariance matrices, and feature vectors. For any continuous spatial query coordinate, the framework retrieves the \(K\) nearest Gaussians within a localized radius, computes a normalized Gaussian distance-weighted embedding, and decodes the feature through a lightweight MLP into target RGB values (or volumetric density for 3D lifting).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Continuous Query Coordinate x ∈ ℝ²"] --> B["Gaussian Geometric Embeddings & Multi-Resolution Hashgrid<br/>Optimized Gaussian distribution G = {N(μ_i, Σ_i, e_i)}"]
B --> C["Local Geometry-Aware Aggregation<br/>Radius-limited KNN search & Gaussian-weighted interpolation"]
C --> D["Continuous Coordinate Decoding & Deformation Mapping<br/>MLP decoder predicts RGB (or 3D radiance c, σ)"]
D --> E["Continuous Reconstruction / Arbitrary Upscaling / Physical Edits / 3D Novel Views"]
Key Designs¶
1. Gaussian Geometric Embeddings & Multi-Resolution Hashgrid: Decoupling Geometry from Color Decoding Traditional INRs feed raw global coordinates directly into MLPs, lacking localized geometric handles. GaINeR introduces a set of \(N\) learnable Gaussian components \(\mathcal{G} = \{ (\mathcal{N}_i(\boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i), \mathbf{e}_i) \}_{i=1}^N\), where \(\boldsymbol{\mu}_i \in \mathbb{R}^2\) represents the centroid position, \(\boldsymbol{\Sigma}_i \in \mathbb{R}^{2\times 2}\) is the spatial covariance matrix, and \(\mathbf{e}_i \in \mathbb{R}^d\) is a learnable feature vector. To equip the embeddings with rich multi-scale spatial details, the feature vectors during training are derived from a multi-resolution hashgrid: \(\mathbf{e}_i = \mathcal{H}(\boldsymbol{\mu}_i)\). At inference time, the hashgrid encoder is discarded entirely, leaving only the optimized Gaussian parameter set \(\mathcal{G}\). This design isolates geometric manipulation to explicit Gaussian primitives while freeing them from storing rigid, discrete color attributes.
2. Local Geometry-Aware Aggregation: Radius-Limited KNN and Kernel Weighting To construct a continuous, smooth feature field from discrete Gaussian primitives, GaINeR queries any continuous coordinate \(\mathbf{x} \in \mathbb{R}^2\) by locating its \(K\) nearest Gaussian centers within a bounding radius \(r\). The feature embedding is computed by normalizing the neighbor features using their corresponding Gaussian probability densities: $$ \mathbf{e}{\mathrm{KNN}}(\mathbf{x}, \mathcal{G}) = \frac{\sum}(\mathbf{x}, \mathcal{G})} \mathcal{Ni(\mathbf{x}; \boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i) \, \mathbf{e}_i}{\sum $$ This weighted interpolation explicitly incorporates Euclidean proximity and anisotropic covariance shapes. The resulting feature field is continuously differentiable, eliminating the spatial discontinuities and seam boundaries typical of discrete graphic primitives.}(\mathbf{x}, \mathcal{G})} \mathcal{N}_i(\mathbf{x}; \boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i)
3. Continuous Coordinate Decoding & Deformation Mapping: Spatial Equivariance for Artifact-Free Edits The aggregated feature \(\mathbf{e}_{\mathrm{KNN}}(\mathbf{x})\) is mapped to RGB pixel values by a coordinate MLP decoder: \(\mathbf{c} = \mathrm{MLP}_\theta(\mathbf{e}_{\mathrm{KNN}}(\mathbf{x}))\). Because the decoder is conditioned solely on relative local embeddings rather than absolute coordinates, the system exhibits spatial equivariance. When Gaussian centers are shifted, rotated, stretched, or bent (\(\boldsymbol{\mu}_i \mapsto \boldsymbol{\mu}_i'\)), the local interpolated feature transitions smoothly, enabling the decoder to synthesize coherent textures without tears or hole artifacts. When integrated with physical simulation engines (e.g., Taichi Elements using the Material Point Method), Gaussian centroids act directly as material particles under continuum mechanics.
4. Single-Image 2D-to-3D Lifting and Volume Rendering: Dimension-Agnostic Neural Fields GaINeR's formulation is dimension-agnostic. By combining monocular metric depth estimation (Depth-Pro) and instance segmentation (SAM), the 2D Gaussian centroids are projected into 3D Euclidean space. The neighborhood query shifts from 2D Euclidean KNN to 3D Mahalanobis distance, and the MLP decoder extends its output to predict both color \(\mathbf{c}\) and volume density \(\sigma\): \(f_\theta(\mathbf{x}) = (\mathbf{c}, \sigma)\). Standard NeRF-style numerical quadrature performs volume rendering along camera rays \(\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}\): $$ \hat{\mathbf{C}}(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t))\,dt, \quad T(t) = \exp\left(-\int_{t_n}^t \sigma(\mathbf{r}(s))\,ds\right) $$ Because the neural decoder naturally interpolates sparse geometries, GaINeR avoids the reprojection holes common in explicit point cloud methods, enabling novel view synthesis and 3D-aware scene edits from a single image.
Loss & Training¶
The framework jointly optimizes the Gaussian parameters \(\{(\boldsymbol{\Sigma}_i, \mathbf{e}_i)\}_{i=1}^N\), the multi-resolution hashgrid encoder \(\mathcal{H}\), and the MLP decoder weights \(\theta\) using a Smooth L1 reconstruction loss: $$ \mathcal{L}{\mathrm{rec}} = \frac{1}{M}\sum_j\right) $$ where residual }^M \mathrm{SmoothL1}\left(f_\theta(\mathbf{x}_j) - \mathbf{c\(\mathbf{r} = f_\theta(\mathbf{x}_j) - \mathbf{c}_j\), and threshold \(\beta\) smoothly transitions between \(\ell_1\) and \(\ell_2\) penalties. For RGBA images, training incorporates alpha-based sampling to discard transparent background regions, focusing computation on visible pixels. A dedicated auxiliary mask model is trained concurrently to reproduce the alpha channel under subsequent animations and transformations.
Key Experimental Results¶
Main Results¶
On standard 2D image reconstruction benchmarks—Kodak (24 natural images) and the DIV2K validation set (100 high-resolution images, \(2\times\) bicubic downscaled)—GaINeR is benchmarked against continuous INRs (WIRE, SIREN, I-NGP, NeuRBF) and Gaussian-based models (3DGS, GaussianImage, MiRaGe).
| Method | Paradigm | Kodak PSNR (dB) ↑ | Kodak MS-SSIM ↑ | DIV2K PSNR (dB) ↑ | DIV2K MS-SSIM ↑ |
|---|---|---|---|---|---|
| WIRE (CVPR 2023) | Continuous INR (Wavelet) | 41.47 | 0.9939 | 35.64 | 0.9511 |
| SIREN (NeurIPS 2020) | Continuous INR (Periodic Sine) | 40.83 | 0.9960 | 36.02 | 0.9568 |
| I-NGP (SIGGRAPH 2022) | Hybrid INR (Hashgrid) | 43.88 | 0.9976 | 37.06 | 0.9894 |
| NeuRBF (2023) | Continuous INR (RBF) | 43.78 | 0.9984 | 37.06 | 0.9901 |
| 3DGS (SIGGRAPH 2023) | Explicit Gaussian Primitives | 44.09 | 0.9991 | 39.12 | 0.9980 |
| GaussianImage (ECCV 2024) | Explicit 2D Gaussians | 38.93 | 0.9984 | 41.48 | 0.9981 |
| MiRaGe (ICML 2025) | Explicit 2D Gaussians Editable | 59.52 | 0.9999 | 54.54 | 0.9998 |
| GaINeR (Ours) | Gaussian-Guided INR | 77.09 | 0.9999 | 62.20 | 0.9999 |
Ablation Study¶
To ensure controlled evaluation across Gaussian architectures, models were benchmarked under identical setups: 500k Gaussian primitives initialized without densification or pruning, trained for 30k iterations.
| Config / Model | Gaussian Count | Kodak PSNR (dB) ↑ | Kodak MS-SSIM ↑ | DIV2K PSNR (dB) ↑ | DIV2K MS-SSIM ↑ | Note |
|---|---|---|---|---|---|---|
| GaussianImage | 500k Fixed | 30.46 | 0.9635 | 28.58 | 0.9565 | Direct primitive rasterization |
| MiRaGe | 500k Fixed | 62.49 | 0.9999 | 58.45 | 0.9999 | Explicit 2D Gaussian blending |
| GaINeR (Full Model) | 500k Fixed | 77.09 | 0.9999 | 62.20 | 0.9999 | Gaussian embeddings + MLP decoding |
In single-image Novel View Synthesis (NVS) across 10°–15° baselines, GaINeR outperforms an Explicit Point Cloud (EPC) baseline: - Hotdog scene (NeRF Synthetic): GT depth gives 20.58 dB for GaINeR (SSIM 0.874, LPIPS 0.186) vs. 8.57 dB for EPC; Depth-Pro depth achieves 19.50 dB for GaINeR. - Scan 114 (DTU): GT depth yields 18.09 dB for GaINeR vs. 13.51 dB for EPC; Depth-Pro depth achieves 17.74 dB for GaINeR. - In DIV2K \(\times 8\) super-resolution, GaINeR scores 23.22 dB PSNR, outperforming 3DGS (17.33 dB) and MiRaGe (16.61 dB).
Key Findings¶
- Decoupling geometry from rendering yields unprecedented reconstruction accuracy: GaINeR exceeds the prior state-of-the-art MiRaGe by 17.57 dB on Kodak and 7.66 dB on DIV2K. Pixel error maps confirm near-zero absolute deviations across the entire image domain.
- Robustness in dynamic continuum mechanics: Under Material Point Method (MPM) simulations of granular sand and fluid mechanics, discrete splatting in MiRaGe tears apart into spiky primitives and transparent gaps. GaINeR smoothly interpolates visual features across sparse particle distributions, achieving superior perceptual scores (MUSIQ 62.62 vs. 61.03, CLIPIQA 0.530 vs. 0.528).
- Inherent continuous super-resolution: At extreme upscaling factors (\(\times 8\), \(\times 10\), \(\times 12\)), discrete Gaussian methods deteriorate into noticeable splat footprints and blurred patches. In contrast, GaINeR synthesizes continuous sharp details directly at arbitrary query resolutions without retraining.
Highlights & Insights¶
- Dual advantages of explicit geometry and implicit neural fields: Using explicit Gaussians as dynamic spatial anchors and neural MLPs as continuous color decoders resolves the longstanding trade-off between editability and reconstruction fidelity.
- Inference decoupling via hashgrid eviction: Incorporating multi-resolution hashgrids during training provides rapid convergence and fine-grained guidance, while discarding the hash encoder during inference leaves a clean, fast Gaussian representation for downstream edits.
- Unified 2D-to-3D representation: By generalizing Euclidean KNN to 3D Mahalanobis distance and predicting volume density, GaINeR offers a mathematically elegant bridge between 2D continuous images and 3D radiance fields.
Limitations & Future Work¶
- Reliance on monocular depth accuracy: 3D lifting quality is upper-bounded by off-the-shelf estimators like Depth-Pro. Depth inaccuracies and occlusions in complex scenes lead to geometric shear and texture stretching.
- Computational overhead of dynamic KNN search: Performing radius-limited nearest-neighbor lookups on millions of Gaussians or high-resolution query batches can become an inference bottleneck without heavily optimized GPU spatial index structures.
- Absence of generative infilling for extreme disocclusions: When large-scale topological tearing or extreme viewing angle changes occur, the continuous decoder interpolates smooth values but cannot synthesize unobserved semantic content (such as hidden object backsides).
Related Work & Insights¶
- vs Classical INRs (SIREN, WIRE, FINER): Classical INRs operate on global coordinate mappings without explicit local primitives, preventing direct spatial and physical deformation; GaINeR introduces localized Gaussian primitives as explicit geometric handles.
- vs 2D Gaussian Splatting (GaussianImage, MiRaGe): Existing 2D Gaussian renderers blend colors directly from discrete primitives, generating severe tearing under non-rigid physical deformations; GaINeR uses Gaussians purely as feature carriers, letting an MLP decode continuous color.
- vs 3DGS & NeRF: GaINeR bridges single-view 2D continuous representation and 3D volumetric fields, eliminating the reprojection holes typical of explicit point clouds.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering fusion of explicit Gaussian spatial handles and continuous implicit neural decoding.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across ultra-high-precision reconstruction, physics simulations, 2D-to-3D lifting, and arbitrary upscaling.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear conceptual motivation, precise mathematical formulations, and compelling qualitative visualizations.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational framework bridging neural rendering, interactive image editing, and physics-based animation.