IGL-Nav: Incremental 3D Gaussian Localization for Image-goal Navigation

Conference: ICCV 2025 arXiv: 2508.00823 Code: Project Page Area: Autonomous Driving Keywords: image-goal navigation, 3D Gaussian Splatting, visual navigation, coarse-to-fine localization, embodied AI

TL;DR

This paper proposes IGL-Nav, a system that builds a renderable scene memory via incremental 3D Gaussian representations and efficiently solves the image-goal navigation problem through a coarse-to-fine localization strategy, while supporting a free-view setting with arbitrary camera viewpoints.

Background & Motivation

Image-goal Navigation requires an agent to navigate to a target location given a single goal image. Existing approaches fall into two main categories:

End-to-end RL methods: directly map observations to actions, but suffer from low sample efficiency and catastrophic forgetting.

Modular methods: construct topological graphs or BEV maps as explicit memory, but struggle to preserve low-level visual features (e.g., texture, color) for image-goal tasks.

RNR-Map introduces a renderable NeRF representation, but due to NeRF's implicit nature and high computational cost, it is constrained to a 2D BEV map, losing critical 3D structural information and requiring the goal image to be captured horizontally. GaussNav also leverages 3DGS but requires complete exploration of the entire environment before optimizing the 3DGS, precluding online usage.

Core Motivation: A 3D-aware memory representation is needed that can be built incrementally, supports real-time rendering, and addresses the 6-DoF search space of goal localization.

Method

Overall Architecture

IGL-Nav consists of three core modules: 1. Incremental Scene Representation: builds 3DGS online via feed-forward monocular prediction. 2. Coarse Localization: formulates goal localization as matching in a 5D discrete space, equivalent to efficient 3D convolution. 3. Fine Localization: solves for the precise goal pose via differentiable rendering and matching-constraint optimization.

The navigation pipeline operates in two stages: an exploration phase driven by coarse localization, followed by a goal-reaching phase driven by fine localization.

Key Designs

1. Incremental 3DGS Scene Representation:

   • At each timestep, an RGB-D observation is fed into a UNet encoder \(\mathcal{E}\) to extract dense scene embeddings \(\boldsymbol{E}'_t\).
   • A Gaussian head \(\mathcal{H}\) (CNN + linear layers) regresses 3DGS parameters: position residuals \(\Delta\boldsymbol{C}_{2D}\), depth residuals \(\Delta\boldsymbol{D}\), opacity \(\alpha\), covariance \(\boldsymbol{\Sigma}\), and spherical harmonics coefficients \(\boldsymbol{c}\).
   • 3D positions are obtained via back-projection using camera intrinsics and pose: \(\boldsymbol{\mu} = \text{Proj}^{-1}(\boldsymbol{C}_{2D}+\Delta\boldsymbol{C}_{2D}, \boldsymbol{D}+\Delta\boldsymbol{D} | \boldsymbol{M}, \boldsymbol{T}_t)\).
   • The scene representation is updated incrementally via union: \(\boldsymbol{G}_t = \boldsymbol{G}_{t-1} \cup (\boldsymbol{\mu}_t, \alpha_t, \boldsymbol{\Sigma}_t, \boldsymbol{c}_t)\).
   • Design Motivation: Avoids the offline optimization of conventional 3DGS, enabling real-time reconstruction from streaming video input.

2. Coarse Goal Localization (5D Search → 3D Convolution):

   • Observing that the camera's top edge is nearly always parallel to the ground during photography, the camera pose is parameterized in a 5D spherical space \((x,y,z,\theta,\phi)\).
   • The 3D space is voxelized, and the sphere is discretized into \(N\) vertices via level-\(\gamma\) icosahedral subdivision.
   • The goal embedding, rotated according to each discrete orientation, is voxelized into a 3D convolution kernel \(\boldsymbol{K} \in \mathbb{R}^{L\times L\times L\times C_{in}\times C_{out}}\).
   • The matching problem is reduced to an efficient 3D convolution: \(\text{argmax}_{x,y,z,k}\; \mathcal{C}(f_1(\mathcal{V}(\boldsymbol{E}_t)), f_2(\boldsymbol{K}))[x][y][z][k]\).
   • Pillar-based voxelization is further applied for speed-up.
   • Design Motivation: Naïve enumeration over all voxels requires \(V\times N\) comparisons; the convolution-equivalent formulation achieves substantial acceleration.

3. Fine Goal Localization (Differentiable Rendering Optimization):

   • Rendering-based stopper: The 3DGS scene is rendered at the current viewpoint using the goal camera intrinsics, and LoFTR is used to match the rendered image against the goal image. Fine localization is triggered when the number of matching pairs exceeds a threshold \(\tau\).
   • Matching-constraint optimization: At each iteration, the image corresponding to the current pose is rendered, LoFTR obtains matching point pairs \((\boldsymbol{x}_g, \boldsymbol{x})\), and these are back-projected to 3D space as \((\boldsymbol{X}_g, \boldsymbol{X})\).
   • Optimization loss: \(\mathcal{L} = \frac{1}{Q}\sum_{i=0}^{Q-1}|\boldsymbol{X}_g^i - \boldsymbol{X}^i|_2\).
   • The method focuses exclusively on the 3D distances of high-quality matching points, mitigating imperfect rendering detail from the incremental 3DGS.
   • Design Motivation: Global photometric loss performs poorly on incremental 3DGS; focusing on high-confidence matched regions yields greater robustness.

Loss & Training

• Scene Representation Training: Uses offline RGB-D video streams; \(K\) frames are randomly sampled to predict 3DGS, which is then rendered at other viewpoints. The loss is a weighted sum of L2 and LPIPS.
• Coarse Localization Training: Focal Loss supervises the 3D convolution activation map, with an auxiliary cross-entropy loss supervising the neighborhood of the goal pose.
• Navigation Policy: Combines frontier-based exploration with the coarse localization activation map, using the Fast Marching Method (FMM) for path planning.

Key Experimental Results

Main Results (Tables)

Standard Image-goal Navigation (Gibson dataset):

Method	Straight-Overall SR/SPL	Curved-Overall SR/SPL
DDPPO	29.0/26.8	15.7/12.9
NRNS	45.7/37.7	20.3/8.8
OVRL	44.9/30.0	45.6/28.0
RNR-Map	68.2/43.9	65.7/40.8
FeudalNav	67.5/55.5	60.2/39.1
IGL-Nav	76.8/64.1	73.5/62.4

Free-view Image-goal Navigation (Zero-shot Transfer):

Method	Narrow FOV Overall SR/SPL	Wide FOV Overall SR/SPL
DDPPO	10.3/6.9	15.5/11.6
OVRL	17.1/11.3	21.7/13.5
OVRL+SLING	21.1/15.3	27.7/17.3
IGL-Nav (zero-shot)	43.1/35.9	47.4/39.4
IGL-Nav (supervised)	57.0/48.2	63.3/55.0

Ablation Study (Tables)

Effect of Icosahedral Subdivision Level on Coarse Localization:

Subdivision Level γ	Narrow FOV SR/SPL	Wide FOV SR/SPL
1	19.7/12.0	24.9/16.8
2	41.3/34.4	48.9/42.1
3	57.0/48.2	63.3/55.0

Ablation of Fine Localization Stopper:

Stopper	Narrow FOV SR/SPL	Wide FOV SR/SPL
No stopper	45.7/32.9	46.2/37.6
SLING	49.0/40.7	52.4/45.0
Rendering-based stopper	57.0/48.2	63.3/55.0

Depth Source Availability:

Depth Source	Narrow FOV SR/SPL	Wide FOV SR/SPL
Predicted depth	53.8/44.7	61.0/51.7
Ground-truth depth	57.0/48.2	63.3/55.0

Key Findings

• IGL-Nav substantially outperforms the state of the art on standard benchmarks (~8% SR gain, ~9% SPL gain).
• Zero-shot transfer to the free-view setting surpasses the supervised performance of competing methods.
• The rendering-based stopper is better suited for cross-camera scenarios than the feature-matching-based SLING stopper.
• The use of predicted depth incurs only ~3% performance degradation, demonstrating robustness for real-world deployment.
• The system has been successfully deployed on a physical robot platform, supporting smartphone photos as navigation goals.

Highlights & Insights

1. Goal Search as 3D Convolution: The paper cleverly leverages voxelization and icosahedral discretization to reduce a high-dimensional search problem to a standard 3D convolution, achieving substantial efficiency gains.
2. Incremental Feed-forward 3DGS: This represents the first feed-forward 3DGS reconstruction model for monocular RGB-D sequences, enabling online real-time construction.
3. Matching-Constraint Optimization: By focusing exclusively on the 3D distances of high-confidence matched points, the method elegantly addresses the insufficient rendering quality of incremental 3DGS.
4. Free-view Setting: The paper is the first to propose the more practical free-view image-goal navigation task, accommodating arbitrary cameras and arbitrary poses.

Limitations & Future Work

• The voxelization and spherical discretization in coarse localization introduce quantization errors; finer discretization incurs higher computational cost.
• Incremental 3DGS lacks an optimization step, so rendering quality is inferior to offline reconstruction, and fine localization depends on the quality of LoFTR matching.
• Evaluation is primarily conducted in the Habitat simulator; real-robot deployment scenarios remain limited.
• Performance under dynamic scenes is not discussed.

Related Work & Insights

• The combination of 3DGS and navigation holds substantial promise; this work demonstrates that incremental 3DGS is viable for real-time navigation tasks.
• The idea of discretizing a high-dimensional search problem and reformulating it as a convolution is generalizable to other 6-DoF localization tasks.
• The coarse-to-fine hierarchical strategy achieves a favorable balance between efficiency and accuracy.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Creatively incorporates 3DGS into navigation; the 5D search → 3D convolution equivalence is highly inventive.
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers both standard and free-view settings with thorough ablations and real-robot deployment, though more real-world scenarios would strengthen the work.
• Writing Quality: ⭐⭐⭐⭐ Logic is clear, method descriptions are detailed, and figures are highly informative.
• Value: ⭐⭐⭐⭐⭐ Introduces a new task setting (free-view), significantly surpasses the state of the art, and demonstrates practical deployment value.

Related Papers

• [ICCV 2025] GaussRender: Learning 3D Occupancy with Gaussian Rendering
• [ICCV 2025] CCL-LGS: Contrastive Codebook Learning for 3D Language Gaussian Splatting
• [ICCV 2025] LookOut: Real-World Humanoid Egocentric Navigation
• [ICCV 2025] 3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation
• [ICCV 2025] DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmetic