4Diff: 3D-Aware Diffusion Model for Third-to-First Viewpoint Translation

Conference: ECCV 2024
Code: https://klauscc.github.io/4diff
Area: 3D Vision
Keywords: Viewpoint Translation, Diffusion Models, Third-to-First Person Translation, 3D Geometric Prior, Cross-View Image Generation

TL;DR

This paper proposes 4Diff, a transformer-based diffusion model integrating 3D geometric priors. By incorporating egocentric point cloud rasterization and 3D-aware rotary cross-attention mechanisms, it translates exocentric (third-person) images into egocentric (first-person) images, achieving state-of-the-art performance on the Ego-Exo4D dataset and demonstrating strong generalization capabilities to novel environments.

Background & Motivation

Background: Viewpoint translation is an important task in computer vision, particularly the translation from exocentric (exo) viewpoints to egocentric (ego) viewpoints, which has broad applications in augmented reality, imitation learning for robotics, and egocentric video generation. Existing methods typically rely on conditional image-to-image generation, employing GANs or simple conditional diffusion models to perform the translation.

Limitations of Prior Work: Exo-to-ego translation involves extreme viewpoint changes, shifting from an external observer's perspective to the actor's first-person perspective, resulting in massive geometric discrepancies. Conventional 2D conditional generative methods lack 3D geometric reasoning, making them unable to handle such spatial transformations accurately. Consequently, they tend to generate blurry and inconsistent images, particularly under large viewpoint changes and complex spatial layouts.

Key Challenge: Exo-to-ego translation is fundamentally a 3D problem—it requires reconstructing the 3D structure of the scene to infer the perspective from another position and orientation. However, existing methods operate primarily in the 2D image space, lacking explicit 3D geometric reasoning capabilities.

Goal: (1) How to effectively integrate 3D geometric priors into diffusion models to enhance spatial transformation modeling? (2) How to ensure geometric consistency with the source view while maintaining the photorealism of the generated images?

Key Insight: The authors leverage depth estimation of the exo image and known camera parameters to "lift" the 2D exo image into a 3D point cloud, which is then reprojected from the perspective of the ego camera. This process provides an explicit 3D geometric guidance. Although the rasterized point cloud is imperfect (containing occlusions and holes), it serves as a strong conditioning signal for the diffusion model.

Core Idea: Use the point cloud rasterized layout from the ego viewpoint (converted from the exo image) as the conditional input to a diffusion model, and further integrate 3D spatial information during the denoising process via a 3D-aware rotary cross-attention mechanism.

Method

Overall Architecture

The pipeline of 4Diff takes an exocentric (exo) image and corresponding camera parameters (including intrinsic and extrinsic parameters for both exo and ego cameras) as input, and outputs the generated egocentric (ego) image. The process consists of two primary parts: (1) Geometric Preprocessing: lifting the exo image into a 3D point cloud via depth estimation, followed by rasterization from the ego camera perspective to obtain an ego layout map; (2) Conditional Diffusion Generation: conditioning on the ego layout map and exo image features to generate high-quality images in the ego perspective using a transformer-based diffusion model.

Key Designs

1. Egocentric Point Cloud Rasterization:

   • Function: Translates the exo image into a coarse layout map in the ego perspective, providing explicit 3D geometric guidance.
   • Mechanism: First, a pre-trained depth estimation model (such as DPT or MiDaS) is used to estimate pixel-wise depth from the exo image. Leveraging the intrinsic and extrinsic parameters of the exo camera, each pixel is back-projected into 3D space to form a colored point cloud. Then, the 3D point cloud is projected onto the ego image plane using the ego camera parameters, obtaining an ego-view layout map via rasterization. While this layout map contains holes (due to occluded regions unseen from the exo perspective) and inaccuracies (due to depth estimation errors), it provides a coarse spatial layout and content distribution of the scene from the ego perspective.
   • Design Motivation: Purely 2D conditional generation lacks geometric guidance, forcing the model to "imagine" the ego layout out of thin air. Providing a meaningful, albeit imperfect, initial layout via point cloud rasterization significantly reduces the complexity of the generation task, allowing the diffusion model to focus on filling in holes and refining details.

2. 3D-Aware Rotary Cross-Attention:

   • Function: Integrates 3D spatial information and semantic features of the exo viewpoint during the diffusion denoising process.
   • Mechanism: Within each denoiser block of the diffusion transformer, a specialized cross-attention mechanism is designed, where the queries originate from the denoising features of the ego viewpoint, and the keys/values come from the encoded features of the exo image. The key innovation is using a 3D-aware Rotary Position Embedding (RoPE) to encode spatial relationships: for each position on the ego side and each position on the exo side, a rotation factor is computed based on their relative orientation in 3D space. This allows the attention mechanism to "know" the relative relationship of the two positions in 3D space. This encoding scheme enables the attention to naturally focus on geometrically corresponding areas.
   • Design Motivation: Simple cross-attention lacks 3D spatial awareness—it only registers the relative positions of two tokens in a sequence rather than their physical relationship in 3D space. Using a 3D-aware rotary position embedding allows the model to leverage camera geometry to establish spatial correspondences between exo and ego positions, enabling cross-attention to capture cross-view semantic correspondences more accurately.

3. Diffusion Image Transformer (DiT) Backbone:

   • Function: Serves as the core generative model to generate high-quality ego-view images conditioned on the ego layout.
   • Mechanism: Using DiT (Diffusion Image Transformer) as the backbone, the ego point cloud rasterized layout is encoded and fed into the transformer as conditional embeddings (analogous to the conditioning injection in ControlNet). The denoising process is performed in the latent space (based on a pre-trained VAE encoder-decoder). Each transformer block contains self-attention, 3D-aware rotary cross-attention, and a feed-forward network. Timestep embeddings are injected into each layer via adaptive layer normalization.
   • Design Motivation: The DiT architecture offers superior global modeling capabilities and scalability compared to U-Net. The global attention mechanism of transformers is inherently suited for viewpoint translation tasks that require long-range dependencies, while also facilitating the integration of the 3D-aware rotary cross-attention module.

Loss & Training

Training utilizes the standard diffusion model objective: given a ground truth ego-view image \(x_0\), Gaussian noise is added to obtain \(x_t\), and the model is trained to predict the noise \(\epsilon\) using a simple MSE loss:

\[\mathcal{L} = \| \epsilon - \epsilon_\theta(x_t, t, c) \|^2\]

where \(c\) represents the conditioning signals (including the ego layout map and exo image features). During inference, a DDIM or DPM-Solver sampler is used to accelerate generation. Training is conducted on the Ego-Exo4D dataset using the AdamW optimizer.

Key Experimental Results

Main Results

Evaluations are conducted on the Ego-Exo4D multi-view dataset using FID, LPIPS, SSIM, and PSNR as evaluation metrics.

Dataset	Metric	4Diff	Pix2Pix	InstructPix2Pix	Prev. SOTA	Gain
Ego-Exo4D	FID ↓	Best	Poor	Moderate	Second Best	Significant
Ego-Exo4D	LPIPS ↓	Best	Poor	Moderate	Second Best	Clear
Ego-Exo4D	SSIM ↑	Best	Lower	Moderate	Second Best	Stable
Ego-Exo4D(Novel Env)	FID ↓	Still Superior	Significant Degradation	Degradation	Degradation	Strong Gen.

4Diff achieves SOTA performance across all metrics and demonstrates more robust generalization capabilities in unseen novel environments compared to other methods.

Ablation Study

Configuration	FID ↓	Description
Diffusion model only (no geometric prior)	High	Lacks 3D guidance, inaccurate generation
+ Ego Point Cloud Rasterization	Significantly decreased	Point cloud layout provides strong geometric conditioning
+ 3D-Aware Rotary Cross-Attention	Further decreased	3D-aware attention enhances spatial consistency
Replace 3D-aware version with standard cross-attention	Slightly increased	3D positional encoding makes a practical contribution
Different depth estimation models	Minimal impact	The method is somewhat robust to depth accuracy

Key Findings

• Point cloud rasterization is the largest contributor to performance improvement, providing indispensable 3D geometric guidance.
• The 3D-aware rotary cross-attention is particularly effective in complex scenes involving multiple people or objects.
• 4Diff generalizes well to unseen novel environments, demonstrating that 3D geometric priors help the model learn scene-agnostic perspective translation patterns.
• Even when depth estimation is not fully accurate, point cloud rasterization still provides useful spatial layout cues.

Highlights & Insights

• Ingenious Integration of Geometry and Generation: Combining deterministic 3D geometric transformations (point cloud rasterization) as a condition for the stochastic generative process (diffusion models) excels by leveraging the strengths of both: geometric accuracy and photorealistic generation.
• 3D Extension of RoPE: Generalizing RoPE (originally designed for 1D/2D sequential positional encoding) to 3D spatial relationship encoding represents an elegant and effective technical contribution.
• Strong Generalization Capability: For viewpoint translation methods, generalization to novel environments is critical, which 4Diff accomplishes outstandingly due to its 3D geometric prior.
• Research from FAIR: The team contains renowned scholars such as Kristen Grauman and Lorenzo Torresani, with the research positioned on Ego-Exo4D, an important large-scale dataset.

Limitations & Future Work

• Dependency on Monocular Depth Estimation Quality: Although the method exhibits a level of robustness, severely erroneous depth estimates will degrade the final generation quality.
• Inevitable Holes in Rasterized Point Clouds: These require the diffusion model to hallucinate content in these unseen regions, which can lead to inconsistencies.
• Single-Frame Translation: Currently, the model only processes single-frame translations, leaving video temporal consistency unutilized.
• Inference Speed Constraints: The generation speed is limited by the sampling steps of diffusion models, posing a challenge for real-time applications.
• Dataset Limitations: Although Ego-Exo4D is highly diverse, the performance under extreme lighting conditions and in outdoor scenes remains to be verified.

Related Work & Insights

• vs Pix2Pix/InstructPix2Pix: These generic image-to-image translation methods lack 3D geometric understanding and fail to effectively handle large viewpoint changes. 4Diff addresses this deficiency through explicit 3D point cloud transformation.
• vs Novel View Synthesis (NVS): NVS methods like NeRF require dense multi-view inputs, whereas 4Diff only requires a single exo image. While NVS emphasizes geometric accuracy, 4Diff's diffusion model is better suited for hallucinating invisible regions.
• vs Ego-Exo Transfer Methods: Previous first-person-to-third-person transfer works primarily focused on feature alignment for action recognition, whereas 4Diff directly accomplishes viewpoint translation at the pixel level.

Rating

• Novelty: ⭐⭐⭐⭐ The combination of 3D point cloud rasterization and 3D-aware rotary cross-attention is an effective innovation, and the 3D RoPE extension is a noteworthy technical contribution.
• Experimental Thoroughness: ⭐⭐⭐⭐ Exhaustive comparative, ablation, and generalization experiments were conducted on the large-scale Ego-Exo4D dataset.
• Writing Quality: ⭐⭐⭐⭐ From a top-tier lab, with a clear problem definition and well-structured methodological description.
• Value: ⭐⭐⭐⭐ Significantly advances the fields of egocentric vision and viewpoint translation, offering a highly generalizable paradigm for 3D-aware diffusion.

Related Papers

• [ECCV 2024] ShapeFusion: A 3D Diffusion Model for Localized Shape Editing
• [ECCV 2024] ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance
• [ECCV 2024] MALD-NeRF: Taming Latent Diffusion Model for Neural Radiance Field Inpainting
• [ECCV 2024] Transferable 3D Adversarial Shape Completion using Diffusion Models
• [ECCV 2024] MVDiffusion++: A Dense High-Resolution Multi-View Diffusion Model for Single or Sparse-View 3D Object Reconstruction