Panorama Generation From NFoV Image Done Right

Conference: CVPR 2025
arXiv: 2503.18420
Code: https://isee-laboratory.github.io/PanoDecouple/
Area: Image Generation
Keywords: Panorama Generation, Diffusion Models, Distortion Guidance, Decoupled Framework, CLIP Fine-tuning

TL;DR

Discovered the "visual cheating" phenomenon in existing panorama generation methods (sacrificing distortion accuracy for visual quality). Proposed PanoDecouple, a decoupled framework that divides panorama generation into distortion guidance (DistortNet) and content completion (ContentNet), achieving optimal performance in both distortion and visual quality with only 3K training samples.

Background & Motivation

1. Background: Generating \(360^\circ\) panoramas from narrow field-of-view (NFoV) images is a critical task for VR applications. Existing methods based on the Diffusion Model + ControlNet architecture have achieved impressive visual results.
2. Limitations of Prior Work: Existing evaluation metrics (FID/IS based on InceptionNet, CLIP-FID based on CLIP) tend to favor perceived image quality rather than distortion accuracy. The authors propose Distort-CLIP and discover a "visual cheating" phenomenon: OmniDreamer (2022) has the most accurate distortion, whereas subsequent methods perform increasingly worse under the guide of misleading metrics.
3. Key Challenge: Panorama generation comprises two fundamentally different sub-tasks: distortion mapping (\(2\text{D} \rightarrow 3\text{D}\) spherical geometric transformation) and content completion (creative image extrapolation). A single network learning both simultaneously tends to optimize the latter while neglecting the former.
4. Goal: Through a decoupled design, enable the model to simultaneously achieve accurate panoramic distortion and high-quality visual content.
5. Key Insight: First, establish an accurate distortion evaluation tool (Distort-CLIP), then use a decoupled framework to handle distortion and content separately.
6. Core Idea: DistortNet uses a distortion map for explicit geometric guidance, while ContentNet uses perspective image information for content completion. Both are trained independently and then integrated into a frozen U-Net.

Method

Overall Architecture

PanoDecouple is based on a Latent Diffusion + Dual ControlNet architecture. A frozen pre-trained U-Net is responsible for information fusion. The DistortNet branch takes a distortion map \(D \in \mathbb{R}^{H \times W \times 4}\) (sine/cosine positional encoding of spherical coordinates) as input to provide geometric guidance. The ContentNet branch takes a partial panorama and a mask as input, handling content extrapolation and completion. The outputs of both branches are added to each layer of the U-Net through zero convolution layers.

Key Designs

1. Distort-CLIP Evaluation Tool

   • Function: Establish an evaluation model and corresponding metric, Distort-FID, capable of distinguishing types of panoramic distortion.
   • Mechanism: Generate data with three types of distortion (panoramic, perspective, random distortion), and fine-tune CLIP's image and text encoders within a contrastive learning framework. The image encoder learns to distinguish images of different distortion types (high similarity for same distortion, low similarity for different distortions), and the text encoder learns to align three text descriptions with their corresponding distortion types. After fine-tuning, Pano-Pers similarity drops from 0.752 to 0.001, validating its distortion perception capability.
   • Design Motivation: Problems cannot be uncovered without an accurate evaluation tool; Distort-CLIP reveals the existence of the "visual cheating" phenomenon.

2. DistortNet Distortion Guidance Branch

   • Function: Provide explicit geometric distortion constraints for panorama generation.
   • Mechanism: Construct a distortion map \(D(i,j) = (\gamma(\theta), \gamma(\phi))\), where \(\theta, \phi\) represent spherical coordinates, and \(\gamma(\cdot)\) is a first-order Taylor positional encoding to ensure boundary continuity. Key modification: changing the condition injection of ControlNet from "first layer only" to "all layers" — since the distortion map is essentially positional encoding, similar to the timestep \(t\) in diffusion models, it needs to be injected at every layer. Each layer uses an independent 2D convolution \(Proj^b\) to map the distortion embedding to the corresponding dimension.
   • Design Motivation: Distortion maps convey global positional information rather than local image features, requiring propagation through all network layers (analogous to positional encoding in ViT).

3. ContentNet Content Completion Branch

   • Function: Extrapolate and generate visually consistent panoramic content from NFoV inputs.
   • Mechanism: Follow a mask-based outpainting architecture (similar to standard ControlNet), but replace the text condition with the CLIP embedding of the perspective image, ensuring that the generated content aligns stylistically and semantically with the NFoV input. The content encoder extracts latent features of the partial panorama, which are input alongside the outpainting mask.
   • Design Motivation: Perspective image embeddings convey the visual information of the source image more precisely than text descriptions.

Loss & Training

• Standard diffusion denoising loss + distortion correction loss \(\mathcal{L}_{distort}\) (utilizing Distort-CLIP to constrain the distortion characteristics of the generated results).
• Only requires 3K training samples (15x less than the 50K of prior work), demonstrating robust generalization ability.

Key Experimental Results

Main Results

Method	Training Samples	FID↓	Distort-FID↓	IS↑	Description
OmniDreamer (2022)	50K	75.14	0.52	4.58	Most accurate distortion but poor visual quality
PanoDiff (2023)	3K	63.49	2.68	6.51	Good visual quality but poor distortion
AOG-Net (2024)	3K	74.07	4.52	6.32	Worse distortion
PanoDecouple	3K	~55	~0.6	~7.0	Superior in both visual quality and distortion

Ablation Study

Configuration	FID↓	Distort-FID↓	Description
Full PanoDecouple	best	best	Complete decoupled framework
Single network (no decoupling)	Good	Poor	Validates the "visual cheating" phenomenon
DistortNet first-layer injection only	-	Poor	Positional encoding requires all-layer injection
w/o Distort-CLIP loss	-	Poor	Distortion correction loss is effective

Key Findings

• "Visual cheating" is a real and widespread problem — subsequent methods continuously improved standard FID, but Distort-FID deteriorated.
• Injecting the distortion map across all layers is significantly superior to injecting only in the first layer or using attention mechanisms.
• Achieving or surpassing methods trained on 50K datasets with only 3K samples highlight the crucial effectiveness of decoupling.
• The framework can be seamlessly extended to applications like text-guided panorama editing and text-to-panorama generation.

Highlights & Insights

• The conceptualization of "visual cheating" exhibits critical thinking — exposing an implicit issue in the field through a self-built evaluation tool, advancing more accurate evaluation standards.
• The decoupled design is highly generalizable — serving as a reference for any generative task requiring both "geometric accuracy" and "visual quality" (e.g., 3D reconstruction, scene editing).
• Improvements in the ControlNet conditional injection mechanism — the insight that "positional encoding-like conditions need to be injected into all layers" can be transferred to other ControlNet applications using positional information as conditions.

Limitations & Future Work

• The training data for Distort-CLIP only covers the equirectangular projection, leaving its generalizability to other panorama projection formats untested.
• The fusion of the two branches relies on the implicit coordination of the frozen U-Net, which might lead to information competition.
• Future work can explore end-to-end training or finer regulation of branch weights.

Related Work & Insights

• vs OmniDreamer: Accurate distortion but poor visual quality (FID 75). This work significantly improves visual quality while maintaining distortion accuracy.
• vs PanoDiff/AOG-Net: Good visual quality but severe distortion. Our decoupled design resolves this trade-off.
• vs PanFusion: Also uses a distortion map but applies it within attention. Our all-layer injection is more effective.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Discovering "visual cheating" + constructing Distort-CLIP + decoupled framework, presenting multiple innovations.
• Experimental Thoroughness: ⭐⭐⭐⭐ Two benchmarks + ablation studies + extended applications, though some numerical values could be more precise.
• Writing Quality: ⭐⭐⭐⭐⭐ The narrative logic from problem discovery \(\rightarrow\) evaluation tool \(\rightarrow\) proposed solution is exceptionally clear.
• Value: ⭐⭐⭐⭐ Significant contributions to both evaluation standards and methodology in the panorama generation field.

Related Papers

• [CVPR 2025] CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model
• [CVPR 2026] VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation
• [CVPR 2025] OmniGen: Unified Image Generation
• [ICCV 2025] What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?
• [CVPR 2025] Improving Editability in Image Generation with Layer-wise Memory