Efficient Camera Pose Augmentation for View Generalization in Robotic Policy Learning¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/SanMumumu/GenSplat
Area: Robotics & Embodied AI / 3D Vision
Keywords: view generalization, 3D Gaussian Splatting, 3D-prior distillation, camera pose augmentation, imitation learning

TL;DR¶

GenSplat reconstructs 3D Gaussian scenes from sparse, uncalibrated robot observations in a forward pass, stabilizes their geometry through 3D-prior distillation, and renders new-view demonstrations with unchanged action labels, increasing average \(\pi_0\) success from 48.9% to 62.2% under large camera perturbations at 0.14 seconds per synthesized frame.

Background & Motivation¶

Robot imitation learning connects spatial cues in images with expert actions, but this mapping often implicitly depends on the camera positions used during collection. The same gripper-object relationship can produce different pixel locations, occlusions, and foreshortening from another viewpoint, which a fixed-view policy may not interpret correctly. Consequently, even modest changes in camera extrinsics can substantially reduce deployment success while the task, objects, and actions remain unchanged. Collecting more demonstrations from the same camera positions does not fill the missing observation directions in the training data. Moving cameras and repeating teleoperation incurs additional human effort, making viewpoint coverage a data bottleneck distinct from task coverage.

Novel view synthesis appears to offer a way to reuse action labels, provided that the synthesized image still depicts the same physical state. Diffusion-based methods can generate visually plausible but geometrically incorrect images, including scale drift, inconsistent parallax, or altered occlusion relationships. Such errors may be tolerable in general image generation but break the correspondence between images and correct actions in tasks such as ring insertion or stacking. 3D Gaussian Splatting (3DGS) renders multiple views from a shared scene and is therefore a suitable representation for preserving this correspondence. However, conventional per-scene optimization often depends on dense, calibrated observations, whereas robot datasets usually contain only a few fixed cameras and may lack reliable extrinsics.

The paper asks whether existing sparse observations can quickly yield sufficiently reliable 3D structure specifically for expanding the camera pose distribution used in policy training. Applying feed-forward 3DGS directly is insufficient: with only image reconstruction errors, a model can fit current views using floating Gaussians or fragmented surfaces that fail when rendered elsewhere. The authors therefore use point maps and relative camera poses from a pretrained visual geometry model as structural supervision, while retaining RGB supervision for texture and boundaries. The goal is not to require additional 3D reasoning inside the policy, but to reduce confusion between viewpoint changes and action semantics through training data. Core Idea: combine permutation-equivariant reconstruction with 3D-prior distillation to obtain renderable robot scenes, turning each expert trajectory into demonstrations with unchanged actions but different viewpoints.

Method¶

Overall Architecture¶

The input is an expert demonstration containing synchronized multi-camera images, an action sequence, and a language instruction; reconstruction operates at each time step. GenSplat extracts multi-view features, predicts camera-to-global transformations and dense point maps in camera coordinates, and regresses renderable Gaussian attributes. During reconstruction training, 3D-prior distillation supervises point maps and relative poses; during policy-data preparation, target camera poses are sampled and rendered from the reconstructed scenes. The new images retain their original actions, time indices, and language instructions, producing augmented demonstrations that are combined with real demonstrations for policy training. Deployment runs the trained \(\pi_0\) or Diffusion Policy (DP), without inserting the teacher or GenSplat into every control step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Synchronized multi-view<br/>demonstrations"] --> Equivariant["Permutation-equivariant<br/>aggregation"]
    Equivariant --> Gaussian["Point-map-driven<br/>Gaussian prediction"]
    Prior["3D-prior distillation"] -.->|Training supervision only| Gaussian
    Gaussian --> Augment["Trajectory-consistent<br/>pose augmentation"]
    Augment --> Policy["Policy training on real<br/>and augmented demonstrations"]
    Observation["Novel-view observations<br/>at deployment"] --> Execution["Trained policy outputs actions"]
    Policy -.->|Trained parameters| Execution

Here, pose-free means that externally calibrated camera poses are not required as inputs; it does not mean that rendering operates without a camera model. Camera geometry is estimated by the network, and target cameras are perturbed around the estimated scene, so estimation errors can still propagate through rendering into policy data. The distillation branch constrains reconstruction training rather than adding another sequential image-generation stage during augmentation.

Key Designs¶

1. Permutation-equivariant aggregation: avoid dependence on the reference camera

Each image is encoded into patch tokens by DINOv2, followed by alternating within-frame self-attention and global self-attention across images. Within-frame attention preserves image-specific structure, while global attention lets cameras jointly explain the same scene instead of producing unrelated geometry. Symmetric treatment of the input views is intended to prevent an arbitrarily chosen reference view from dominating coordinate organization and introducing order-dependent artifacts. Permutation equivariance does not mean that all outputs remain unchanged: exchanging input cameras should also exchange their corresponding per-view predictions. Nor does it mean that the policy is already invariant to camera motion; that property still needs to be established through novel-view training and evaluation.

The aggregated features feed two lightweight Transformer decoders: one predicts camera rotations and translations, and the other predicts per-pixel 3D points and confidence. Point maps use local camera coordinates, while the predicted poses place these local structures into a shared coordinate frame. This separation allows multi-view alignment and dense local geometry to receive distinct supervision instead of requiring color losses to infer every geometric quantity. The implementation uses 36 alternating attention blocks and two 5-layer Transformer decoders containing only self-attention. Relevant geometry networks are initialized from \(\pi^3\), reference [57]; this visual geometry model is distinct from the downstream control policy \(\pi_0\).

2. Point-map-driven Gaussian prediction: separate geometry from appearance

Gaussian centers are not freely guessed by an attribute head; they are obtained by transforming local 3D coordinates from the point maps into the global frame. The explicit relationship given on page 5 is:

\[ \boldsymbol{\mu}_g = \boldsymbol{R}_i\boldsymbol{P}_i + \boldsymbol{T}_i. \]

Here, \(\boldsymbol{P}_i\) denotes local point coordinates for camera \(i\), and \(\boldsymbol{R}_i\) and \(\boldsymbol{T}_i\) specify its local-to-global transformation. This constraint anchors renderable elements to supervised 3D point maps, reducing the freedom to fit colors while allowing geometry to drift. Compared with regressing only scalar depth, a point map supplies a 3D position for each pixel and can directly receive structural information from a visual geometry teacher. This does not make point maps inherently physically correct; it gives the proposed prior supervision a concrete representation to constrain.

A Dense Prediction Transformer (DPT) attribute head regresses opacity, quaternion rotation, anisotropic scales, and spherical harmonic coefficients. It combines deep multi-view features with shallow high-resolution RGB features so that both geometric context and fine appearance inform the predictions. Centers determine placement, while the remaining attributes determine local shape, transparency, and view-dependent color; together they enter differentiable rendering. Training also aligns rendered depth with the z component of the point map over valid pixels, preventing the Gaussian representation and its geometric source from fitting different surfaces. The method therefore builds a shared spatial representation before producing images, rather than generating an attractive RGB image and recovering structure afterward.

3. 3D-prior distillation: constrain sparse reconstruction with points and relative poses

With RGB supervision alone, sparse inputs leave large regions unobserved, allowing floating Gaussians to fit known views while producing tears and incorrect occlusions after camera movement. A pretrained visual geometry model supplies pseudo-ground-truth point maps and camera relationships, constraining reconstruction through both appearance and structure. The prose defines the point-map loss as the mean squared Euclidean distance between predicted and teacher points, supervising per-pixel 3D positions rather than only scalar depth. The camera loss uses an SO(3) geodesic distance for relative rotation between view pairs and a Huber loss for relative translation. These relative constraints do not require external absolute tracking data and encourage camera predictions to remain mutually consistent, reducing drift and degeneracy.

The 3D-prior term is a weighted combination of the point-map and camera losses, optimized jointly with RGB reconstruction and depth alignment. Teacher priors do not replace real-image supervision: the paper notes that pseudo-depth and pseudo-normal maps can be oversmoothed, leaving RGB losses to recover boundaries and high-frequency details. Distillation consequently acts as structural regularization rather than demanding that the student reproduce every teacher detail. This also explains the joint evaluation of reconstruction quality and manipulation success: structural artifacts matter to the central task when they corrupt action supervision. Equations (2) through (8) contain extraction damage such as missing operators in the supplied text; this note follows the readable prose and does not reconstruct exact norms, constants, or loss weights.

4. Trajectory-consistent pose augmentation: change the camera, not the expert behavior

After reconstructing each frame, the method defines a rotation pivot at the midpoint of the shortest segment connecting the optical axes of the two external cameras. The target camera rotates about a designated world X or Y axis within \([-30^\circ,30^\circ]\), then moves forward by 0.5 to 6 cm along its own +Z direction. Forward movement mitigates the risk that a large rotation moves the viewpoint outside valid Gaussian coverage; it does not add translation to the robot action. In the main setting, each original demonstration receives exactly one sampled rotation-translation pair, producing one augmented trajectory with the same perturbation parameters throughout. The network still reconstructs the dynamic scene frame by frame; a shared camera perturbation does not freeze the entire robot motion into one static scene.

The augmented demonstration retains the original action labels, temporal alignment, and language instruction, and is combined with real trajectories for imitation learning. The supervision remains meaningful because actions and object relationships in the scene are unchanged; only the observation viewpoint differs. If rendering moves an object or corrupts occlusion geometry, this label reuse becomes unreliable, making reconstruction fidelity a requirement rather than a cosmetic feature. The paper does not equate new viewpoints with new actions, object arrangements, or tasks; additional diversity primarily lies in camera pose. Sampling more views within this range mainly densifies the training distribution and does not automatically extend its generalization boundary.

A Worked Example¶

Consider a Stack Block demonstration with synchronized observations and actions as the robot approaches, grasps, and stacks a block. At each time step, aggregated external-view features jointly explain the same block, and point maps plus camera transformations place it in a shared Gaussian scene. A valid camera perturbation is selected for the trajectory, and the manipulation is rendered frame by frame from this virtual camera. The block's image location may change because of perspective, but the label remains the expert action performed at the corresponding time. The policy therefore receives supervision for the same action from both real and virtual viewpoints without another teleoperation session. The main setting turns 100 original demonstrations into 100 augmented demonstrations; the quantity study additionally uses 200, 300, or 400 augmented demonstrations. These trajectories are not independently collected expert behaviors, so their benefit must be established through physical robot evaluation rather than dataset size alone.

Loss & Training¶

GenSplat training combines RGB reconstruction, depth alignment, and 3D-prior distillation; the RGB term includes L1 and LPIPS perceptual constraints. The reconstruction model is pretrained on DROID, sampling each episode at 1 frame per second to obtain 271k training images and reduce the gap between general scenes and robot manipulation. Training lasts 30k iterations with a batch size of 16 and an exponential moving average of model weights for stable evaluation. The learning rate uses cosine annealing with 1k warmup steps and a peak of \(2\times10^{-4}\); geometry-initialized parameters receive 0.1 times the base learning rate. FlashAttention, gradient checkpointing, and bfloat16 reduce memory requirements. Downstream \(\pi_0\) and DP policies predict future action sequences from visual observations and proprioception, using both real and augmented expert demonstrations. The paper describes the policy objective abstractly as behavioral cloning, which does not imply that both policy families implement the same explicit likelihood formulation. Detailed policy settings are deferred to Appendix A.1, but the supplied cache ends with the references and does not include that appendix, so optimizers, policy training steps, and action horizons are not supplied here.

Key Experimental Results¶

Main Results¶

The platform comprises a 7-DoF Franka Research 3, a Robotiq 2F-85 gripper, and 3 RGB cameras: 2 external cameras and 1 wrist-mounted camera. The authors evaluate 6 real-world manipulation tasks with 100 expert demonstrations each and variations in object positions and colors. Small perturbations use 3 to 6 degrees of rotation and 0.5 to 2 cm of translation; Medium uses 8 to 15 degrees and 2 to 4 cm; Large uses 18 to 30 degrees and 4 to 6 cm. The following excerpt from Table 1, page 12, uses \(\pi_0\) throughout, with 30 episodes per task under Large perturbations; success rates are percentages and synthesis times are measured on the same computing device.

Augmentation method	Average success over 6 tasks ↑	Insert Ring	Place Can	Synthesis time (seconds/frame) ↓
Unaugmented Baseline	48.9	33.3	26.7	Not applicable
VISTA	37.8	36.7	6.7	2.81
SEVA	57.7	43.3	30.0	50.00
InstantSplat	55.0	40.0	33.3	85.20
NoPoSplat	36.7	33.3	10.0	0.05
AnySplat	56.1	43.3	30.0	0.16
GenSplat	62.2	46.7	40.0	0.14

In Table 1, GenSplat improves over the unaugmented baseline by 13.3 percentage points and over AnySplat by 6.1 percentage points; neither difference is a relative percentage gain. The prose on page 10 calls the \(\pi_0\) gains of +6.7%, +8.9%, and +13.3% relative improvements, but the large-perturbation figure matches the absolute percentage-point difference in Table 1, so this note does not repeat that relative-gain wording. The same page reports DP improving from 27.78% to 43.33% under large perturbations, approximately a 56.0% relative gain, which must be distinguished from the \(\pi_0\) comparison. The 0.14 seconds/frame measurement concerns data synthesis, not the robot control period or policy inference latency.

Ablation Study¶

The following table corresponds to Table 2, page 13, examining the architecture and 3D prior in rendering and policy performance. Check and cross symbols are damaged in the text extraction, so configurations are identified from the accompanying prose; the policy conditions are 30 degrees with 6 cm and 60 degrees with 12 cm.

Config	PSNR ↑	LPIPS ↓ (values as reported)	Large success (%) ↑	Extreme success (%) ↑
VGGT baseline, no 3D prior	22.35	6.01	66.0	30.0
Permutation-equivariant architecture, no 3D prior	25.87	5.03	70.0	40.0
Permutation-equivariant architecture + 3D prior	26.53	3.72	74.0	46.0

Adding the 3D prior to the permutation-equivariant architecture increases PSNR from 25.87 to 26.53 and Extreme success from 40.0% to 46.0%. The scale of LPIPS values 6.01, 5.03, and 3.72 is unspecified; they must not be silently rewritten as 0.0601, 0.0503, and 0.0372 or directly compared with commonly normalized values in other papers. Table 2 reports 74.0% Large success rather than Table 1's 62.2%; the cache does not sufficiently establish how the task aggregation and policy evaluation correspond between these tables, so their results are kept separate.

Table 3, page 14, fixes 100 real demonstrations and examines view quantity for DP on Stack Block and Stack Cups, with 50 episodes per task. Only the Large-perturbation columns are reproduced below; S denotes real demonstrations, G denotes generated demonstrations, and success rates are percentages.

Training demonstrations	Stack Block ↑	Stack Cups ↑
100S	30.0	26.0
100S+100G	48.0	44.0
100S+200G	58.0	50.0
100S+300G	64.0	56.0
100S+400G	62.0	54.0

Key Findings¶

Synthetic data can be harmful: VISTA and NoPoSplat perform below the unaugmented baseline on average, showing that quantity does not compensate for geometric errors.
Distillation benefits extend beyond image metrics: Table 2 reports better reconstruction and a 6.0-percentage-point gain under Extreme perturbations, though this does not isolate every gain to a single geometric cause.
Augmentation quantity saturates: 300G gives the best Large results on both tasks in Table 3, whereas 400G lowers each by 2.0 percentage points, contradicting an unconditional more-is-better interpretation.
Figure 8 uses 3 independent trials with 30 episodes per task in each trial, unlike Table 3's 50-episode protocol; their uncertainty statistics cannot be interchanged.

Highlights & Insights¶

The paper treats 3D reconstruction as a training-data tool rather than a required control-network input. This supports different policy architectures while keeping reconstruction computation in data preparation.
Point maps provide a direct target for geometric distillation, while RGB supervision compensates for teacher oversmoothing. Their complementary roles address action supervision more directly than visual realism alone.
Camera perturbations remain consistent over a trajectory while action labels and timing are preserved. This avoids treating independent framewise viewpoint jumps as plausible camera motion.
The authors interpret view-count saturation as increased pose density within a fixed boundary. A transferable lesson is to fill coverage gaps before adding more same-range samples, rather than treating the observed saturation as a proven universal law.

Limitations & Future Work¶

The authors identify dynamic wrist-camera integration as future work for fine-grained, occlusion-aware geometry; the presence of a wrist camera in the hardware does not establish that this reconstruction problem is solved.
Per-frame reconstruction and a shared Gaussian scene do not guarantee absolute metric, occlusion, or temporal correctness. Teacher errors, unseen surfaces, and views outside reliable coverage can still corrupt augmented samples.
Evidence mainly comes from 6 tasks on one robot platform, supporting view generalization but not automatically cross-robot, cross-action-space, or arbitrary object-change generalization.
Table 1 lacks confidence intervals for its averages, and Table 2 leaves evaluation correspondence and LPIPS scale underspecified; small differences should not all be treated as statistically established advantages.
Geometry-confidence filtering and occlusion-aware pose sampling could allocate augmentation to reliable regions with insufficient training coverage. This is a reader-proposed extension, not a component validated in the paper.

Compared with AnySplat and NoPoSplat: all use feed-forward Gaussian representations, while this paper emphasizes permutation-equivariant processing, point-map supervision, and 3D-prior distillation for sparse robot inputs. Table 1 shows that fast rendering alone does not ensure policy gains.
Compared with VISTA and SEVA: these represent generative view augmentation, whereas GenSplat controls viewpoints through a shared 3D scene. The experiments establish differences in the tested setting, not a claim that every diffusion method fails on every robot dataset.
Compared with \(\pi^3\) and VGGT: these provide visual geometry capabilities or architectural baselines; GenSplat applies related representations to Gaussian rendering and augmentation that preserves action labels, rather than evaluating reconstruction alone.
Compared with policy-side spatial modeling: the method primarily broadens the camera distribution without changing the core \(\pi_0\) and DP control architectures. Whether augmentation complements policy-side geometric constraints is a future research question, not an established result here.

Rating¶

Novelty: 4/5. Combines permutation-equivariant feed-forward Gaussians, point-map distillation, and robot view augmentation, with contributions centered on task-specific integration and geometric supervision.
Experimental Thoroughness: 4/5. Covers 6 real tasks, two policy families, method comparisons, and augmentation-count ablations, but some evaluation details remain incomplete.
Writing Quality: 3/5. The main mechanism is clear, but relative-gain wording, LPIPS scale, and correspondence between tables need greater precision.
Value: 4/5. Avoids repeated teleoperation for new viewpoints and adds no reconstruction stage at policy deployment, making it relevant to researchers studying real-robot data efficiency.