Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting

Conference: ECCV2026
Paper: ECCV Paper
PDF: Full Paper
Project: Habitat-GS
Code: https://github.com/zju3dv/habitat-gs
Area: 3D Vision
Keywords: Gaussian Splatting, embodied navigation, dynamic avatars, visual-navigation decoupling, cross-domain generalization

TL;DR

Habitat-GS extends Habitat-Sim with zero-copy Gaussian rendering, offline-prepared drivable avatars, and dynamic capsule obstacles while retaining its navigation interfaces; under a fixed budget, mixed mesh/GS training raises GS-test success from 61.30% to 79.60%, although real-world transfer evidence remains limited to open-loop video evaluation.

Background & Motivation

A navigation simulator must do more than specify where a robot can move: it must produce visual observations suitable for training perception-driven policies. Mesh rasterization in platforms such as Habitat is efficient, but producing detailed textures, complete geometry, and high-quality human assets from real environments requires scanning and manual cleanup. When training images lack realistic materials, detail, and view-dependent effects, a robot may learn simulated geometric regularities without handling real camera observations reliably. Static rooms also provide no opportunity to learn to yield to walking people, so improving background rendering alone leaves an important task requirement unmet.

3D Gaussian Splatting, or 3DGS, offers a route to high-fidelity images from reconstructed assets, but it is not a ready-made navigation world model. A Gaussian collection has no explicit collision surface, its fast rasterizers mainly use CUDA, and Habitat's sensor pipeline is built on OpenGL. Even a photorealistic person inserted into a room is ineffective as a navigation obstacle if the planner still considers that location traversable. The paper therefore addresses the integration of Gaussian appearance, human motion, and navigation constraints into one trainable simulation loop, rather than improving reconstruction in isolation.

The authors retain Habitat's NavMesh and task ecosystem, assigning appearance to Gaussians and traversability to conventional navigation geometry. This avoids rebuilding a complete physics engine around Gaussians, while setting a clear scope: obstacle avoidance for navigation, not contact mechanics or object manipulation. Core Idea: decouple the visual and navigation representations of scenes and humans, then bind them through synchronized poses and capsule obstacles so that realistic avatars are both visible entities and effective obstacles.

Method

Overall Architecture

Inputs comprise standard PLY scene Gaussians and a NavMesh, plus each avatar's canonical Gaussians, per-frame joint transformations, and proxy capsules. Asset and motion precomputation precede parallel scene rendering, avatar deformation, and dynamic blocking; composited RGB-D observations then enter a Habitat-Lab policy. The policy selects move-forward, turn-left/right, or stop actions, while additional queries expose avatar blocking and nearest-avatar distance for rewards and metrics. The four designs below follow Sections 3.1–3.4 and Figure 2; rendering and navigation must use the same simulation time rather than play back independently.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scene and avatar assets"] --> B["Asset and Motion<br/>Precomputation"]
    B --> C["Zero-Copy<br/>Scene Rendering"]
    B --> D["Drivable<br/>Gaussian Avatars"]
    B --> E["Dynamic<br/>Capsule Blocking"]
    C --> F["Depth-composited RGB-D"]
    D --> F
    F --> G["Policy action and environment step"]
    E --> G
    G -->|Advance simulation time| D
    G -->|Update occupancy and queries| E

Key Designs

1. Asset and Motion Precomputation: remove expensive avatar preparation from the simulation loop

Scene preparation produces two assets with different responsibilities: a PLY file specifies rendering Gaussians, and a NavMesh specifies traversable regions. Gaussians may come from self-reconstruction, public datasets, or generative pipelines, but importing a visual asset does not automatically provide usable navigation geometry. For each avatar, a trained Gaussian avatar model exports canonical positions, spherical-harmonic coefficients, opacities, scales, rotations, and linear-blend-skinning weights. These attributes encode the original appearance and how individual Gaussians follow skeletal transformations, avoiding a fresh avatar-reconstruction network evaluation every frame. Experiments use six AnimatableGaussians identities, with three for training and three held out for testing.

To obtain executable motion, the authors sample waypoints along NavMesh shortest paths and run GAMMA offline to synthesize walking trajectories. SMPL-X forward kinematics converts these motions into per-frame joint transformation matrices, while world-space capsules are also precomputed for each frame. Runtime interpolation at the current simulation time supplies a consistent motion state to both rendering and blocking. This saves online neural inference; it does not allow an avatar to decide to stop, turn, or yield in response to the robot. Trajectories are pre-synthesized, and avatars act as high-priority moving obstacles, an assumption that matters when interpreting the safety results.

2. Zero-Copy Scene Rendering: feed CUDA outputs directly into Habitat sensors

The scene Gaussians produce color and depth through CUDA tile-based rasterization, whereas Habitat exposes observations through OpenGL textures. Copying data to the CPU and back to the GPU every frame would undermine the sustained observation throughput required for reinforcement learning, regardless of image quality. The system therefore pre-registers OpenGL textures and uses CUDA–OpenGL interoperability for mapping, rendering, and unmapping while keeping data on the GPU. The main paper establishes this zero-copy design, but places the detailed Map–Render–Unmap implementation in supplementary material that is not included in the available cache. Buffer layouts and synchronization details not present in that source are consequently not reconstructed here.

A scene can contain Gaussians, traditional meshes, and multiple avatars to preserve compatibility with conventional assets. The compositor compares channel depths and retains the fragment closer to the camera; the rule in Section 3.2 can be written as:

\[ (C,D)= \begin{cases} (C_g,D_g), & D_g<D_m,\\ (C_m,D_m), & \text{otherwise}. \end{cases} \]

Subscripts \(g\) and \(m\) denote the Gaussian and mesh channels, while \(C\) and \(D\) denote color and depth. Deformed Gaussians from all active avatars enter a single avatar CUDA rendering pass and are composited using the same kind of depth comparison. Avatars therefore participate in foreground/background occlusion with the scene and conventional meshes, rather than simply being overlaid on the image. This rule addresses visibility across rendering channels; it does not establish a physical contact model for Gaussian surfaces.

3. Drivable Gaussian Avatars: update realistic appearance with the same skeletal motion

Avatar rendering cannot merely translate a static Gaussian collection: different body regions must deform with the SMPL-X joint pose. At runtime, a lightweight CUDA Linear Blend Skinning (LBS) kernel reads interpolated joint matrices and deforms canonical Gaussians using stored skinning weights. This converts the offline appearance asset into the current renderable person while preserving clothing and hair detail represented by the Gaussians. Compared with running a complete avatar neural network online, the design concentrates per-frame work on skinning and rasterization. Figure 4 illustrates appearance differences, but better-looking avatars do not mean that this paper independently solves avatar reconstruction or motion generation.

This module and scene rendering have distinct responsibilities: the scene channel handles the relatively static environment, while the avatar channel updates and jointly renders active humans. Outputs remain RGB and depth with the same format as Habitat's mesh renderer, so existing PointNav policies need no Gaussian-specific input interface. This compatibility allows experiments to hold the policy architecture fixed while varying asset domains and avatar-training conditions instead of introducing a new navigation network at the same time. However, scene and avatar assets come from different pipelines, and successful visual compositing alone does not establish physically consistent lighting, contact, or all dynamic details.

4. Dynamic Capsule Blocking: make visible people occupy traversable space

Gaussians lack a reliable closed collision surface, so treating color rendering as collision geometry would leave navigation without an explicit boundary. The authors construct proxy capsules from SMPL-X bones, store their positions offline for each frame, and interpolate them at the same time used for avatar rendering. At each navigation step, capsules from all active avatars enter the extended path-planning logic as dynamic obstacles. When a candidate movement intersects a capsule, the simulator clips that movement to prevent passage through the proxy body. This is a navigation-level movement constraint, not an impulse-based collision solver for robot–human contact.

Habitat-Lab can also query the nearest capsule distance and whether a candidate step is blocked, supporting proximity penalties, collision penalties, and associated metrics. Blocked movement must be distinguished from a policy that has learned safe avoidance: clipping can prevent penetration even while a policy repeatedly proposes colliding actions. The paper therefore still reports nonzero collision-step fractions; its anti-penetration mechanism should not be interpreted as a collision-free policy. The main text does not provide complete reward weights or the exact PSI aggregation formula, so this note retains only the interfaces and metric meanings explicitly described.

A Worked Example

Consider the paper's dynamic PointNav setting with three avatars: a robot must cross a room to reach its goal while people follow offline walking trajectories. At a given simulation time, the system retrieves the current skeletal pose and capsule locations, then updates avatar appearance and navigation occupancy. The robot receives RGB-D observations containing scene–avatar occlusions, and its policy chooses whether to move forward or turn. If the forward step intersects an avatar capsule, NavMesh blocking clips the motion and exposes the blocking flag and nearest-avatar distance. At the next time step, people continue along their prescribed trajectories and the policy decides again from new observations; they do not replan because the robot is in their way. The example shows that training signals depend on consistency between visible people and actual movement constraints, not merely on inserting people into rendered images.

Loss & Training

This is a simulation-system and navigation-training paper, not a proposal for a new Gaussian reconstruction loss. PointNav comparisons use DD-PPO, a ResNet visual encoder, and a GRU policy head with \(256\times256\) RGB and depth inputs. Each static experiment trains for \(5\times10^7\) steps, changing only the mesh/GS composition of 100 training scenes. Dynamic experiments first use the same static pretraining budget and then fine-tune for \(5\times10^6\) additional steps on 20 GS scenes, comparing training with and without avatars. The additional budget is 10% of static pretraining; it should not be described as the total cost of learning avatar avoidance from scratch. Although the authors explain mixed-training gains as mesh scenes first establishing basic competence, the experimental configuration specifies scene ratios, not an explicit staged curriculum that switches domains.

Key Experimental Results

Main Results

The GS collection combines InteriorGS and additional real-world reconstructions at a 4:1 ratio, yielding 120 scenes split into 100 training and 20 test scenes. The mesh collection uses HM3D with the same 100/20 split; the two test domains contain different scene collections rather than paired representations of identical rooms. The comparison therefore measures the effect of scene-domain composition and cannot attribute every gain to the renderer alone. SR is the fraction of episodes reaching the goal threshold, SPL combines success and path efficiency, and DTG is the final Euclidean distance to the goal.

Table 1: training-domain comparison, selected from original Table 3(a). Every configuration uses \(5\times10^7\) training steps; SR/SPL are percentages and DTG is in meters.

Training configuration	Mesh SR ↑	Mesh SPL ↑	Mesh DTG ↓	GS SR ↑	GS SPL ↑	GS DTG ↓
A: 100 Mesh	59.00	51.23	5.537	61.30	52.09	4.982
B: 100 GS	53.00	43.13	6.439	70.70	58.49	3.550
C: 80 M + 20 G	60.20	52.06	6.004	73.40	64.32	3.757
D: 50 M + 50 G	61.80	51.34	5.938	78.10	67.42	3.008
E: 20 M + 80 G	59.60	51.01	5.901	79.60	68.38	2.698

Relative to A, E improves GS-test SR by 18.30 percentage points and SPL by 16.29 points, while Mesh SR is 59.60% versus 59.00%. However, E's Mesh DTG is 5.901 meters versus A's better 5.537 meters, so the result is not a cost-free improvement on every metric. GS-only B performs worse on the Mesh test, and the training curves support mixed-domain training over replacing all assets under this fixed budget.

Original Table 4 further evaluates real videos in an open loop: five self-recorded scenes contain 38 episodes, and five InCrowd-VI scenes contain 25 episodes. APA±1 measures agreement between policy actions and the human's next motion with a ±1-step tolerance, not goal-reaching success after executing actions on a robot. On self-recorded videos, A scores 55.6% and E scores 61.2%; on InCrowd-VI, A scores 20.4% and D scores 29.6%. This supports better action prediction under real observations but cannot replace closed-loop deployment tests in which policy errors accumulate.

Ablation Study

The following is a task-level comparison of training with or without avatars, not a module ablation disabling zero-copy rendering, LBS, or capsule blocking individually. After static pretraining, the groups receive GS fine-tuning without or with avatars; evaluation uses 20 dynamic mesh scenes and 20 dynamic GS scenes. CR is the fraction of collision steps, and PSI measures the average degree of intrusion into a 1.0-meter personal-space radius around avatars; lower is better for both. The main paper does not show PSI's precise normalization or aggregation across avatars, so its values should not be interpreted as meters or percentages.

Table 2: avatar-training comparison from original Table 5; SR/SPL/CR are percentages, and PSI is reported in the source's scale.

Test domain	Fine-tuning configuration	SR ↑	SPL ↑	CR ↓	PSI ↓
Mesh scene + avatars	GS, no avatars	54.80	46.98	2.521	0.075
Mesh scene + avatars	GS + GS avatars	58.00	46.80	2.342	0.068
GS scene + avatars	GS, no avatars	81.80	71.35	6.713	0.092
GS scene + avatars	GS + GS avatars	80.00	65.72	4.746	0.077

GS dynamic-test CR falls from 6.713% to 4.746%, a decrease of 1.967 percentage points, while PSI falls from 0.092 to 0.077. The trade-off is lower SPL, from 71.35% to 65.72%, and lower SR, from 81.80% to 80.00%, revealing tension between safety and direct goal-reaching efficiency. Mesh dynamic-test CR and PSI also decrease, but this comparison cannot independently establish that Gaussian appearance is better than training with mesh avatars, because the baseline has no avatars at all.

Table 3: throughput analysis from original Table 6(b); one RTX 4090, \(256\times256\) resolution, approximately 2M scene Gaussians, and varying avatar counts.

Avatars	FPS ↑	Frame time (ms) ↓	GPU memory (GB) ↓
0	94.16	10.6	3.917
1	75.14	13.3	4.197
2	57.70	17.3	4.457
5	37.74	26.5	5.513
10	24.67	40.5	7.497

Key Findings

• Realism has multiple sources of evidence: original Table 2(a) reports 88 participants, 825 valid batches, and 7425 pairwise comparisons, with GS preferred in 61.9% and mesh in 38.1%. This remains a perceptual study of selected assets, not a renderer-only controlled experiment.
• A more realistic training domain is not automatically better: mixed configurations outperform pure GS on the GS test under equal step budgets, highlighting both visual richness and learning efficiency. The best configuration also differs across test domains.
• Real-time operation depends on scene and avatar load: approximately 2M Gaussians with two avatars yields 57.70 FPS, while ten avatars yields 24.67 FPS. The former is not a measured guarantee for arbitrary avatar counts or arbitrary datacenter GPUs.

Highlights & Insights

• Visual-navigation decoupling defines the system's capabilities, not just its rendering implementation. It brings Gaussian assets into a mature navigation ecosystem while leaving contact physics to a separate representation.
• Driving appearance and capsules at the same time addresses embodied training more directly than image quality alone. A robot must both see a person and experience different action consequences because that person is present.
• Offline precomputation turns high-quality avatar models into assets with low online cost. The limitation is equally explicit: their behavior is not a policy that responds to the robot in real time.
• Mixed-domain results caution against treating high-fidelity assets as a universal replacement. A more transferable lesson is to fix the policy and compute budget before studying the combination of asset diversity and convergence speed.

Limitations & Future Work

• The authors explicitly limit physical interaction to navigation-level avoidance: capsule blocking does not compute contact forces or impulses and cannot directly support grasping, pushing, pulling, or physical human–robot interaction. Manipulation would require deeper physics-engine coupling.
• Avatars follow pre-synthesized trajectories as high-priority dynamic obstacles rather than interactive partners. Responding to robot actions would require online inference or another runtime behavior-control mechanism.
• Mesh and GS tests are not paired representations of the same physical environments, so representation, scene content, and asset quality may all affect outcomes. Stronger causal analysis would use paired scenes and controlled acquisition conditions.
• Real-world validation is open-loop video evaluation, without physical-robot closed-loop success, collision rates, or recovery after failures. Calling this validated safe navigation in the real world would exceed the evidence.
• Avatar-training comparisons omit a mesh-avatar-training group and do not separate appearance quality from exposure to dynamic obstacles. Such controls are needed to isolate the independent value of Gaussian avatars.
• The available main paper leaves reward coefficients, the exact PSI definition, and low-level interoperability details underspecified. Reproduction requires supplementary material or code rather than inferring complete implementation details from this note.

Related Work & Insights

• Versus Habitat / Habitat 3.0: Habitat-GS reuses their NavMesh, task interfaces, and training ecosystem while upgrading scene and avatar visual assets. Its contribution is integration within that ecosystem, not a reinvention of PointNav or social-navigation tasks.
• Versus 3DGS / NeRF: these methods provide novel-view-synthesis representations, whereas Habitat-GS addresses sensor interfaces, occlusion composition, and navigation occupancy. High-quality views are only one component of a trainable environment.
• Versus AnimatableGaussians / GAMMA: the former supplies avatar appearance and the latter supplies scene-aware motion; Habitat-GS exports them offline and connects them to online navigation. Neither the avatar model nor the motion generator is newly proposed here.
• Research direction: with scenes, trajectories, and training budgets fixed, vary avatar rendering and behavioral responsiveness separately to distinguish visual realism from interactive realism. This is a follow-up experiment suggested by the paper's scope, not an established result.

Rating

• Novelty: 4/5. Integrating Gaussian rendering, drivable avatars, and dynamic navigation blocking has clear value, while the underlying representations and motion models largely build on prior work.
• Experimental Thoroughness: 4/5. The study covers realism, cross-domain navigation, avatar safety, and performance, but lacks paired-asset controls, avatar-representation ablations, and real closed-loop experiments.
• Writing Quality: 4/5. The data flow and navigation scope are clear, although some implementation and metric details rely on supplementary material.
• Value: 4/5. Useful infrastructure for high-fidelity navigation training, but not a general-purpose simulator for robotic contact interaction.