360Anything: Geometry-Free Lifting of Images and Videos to 360°¶

Conference: ECCV2026
arXiv: 2601.16192
Code: None
Area: 3D Vision
Keywords: Panorama Generation, Diffusion Transformer, Geometry-Free, Circular Latent Encoding, Perspective-to-Panoramic Lifting

TL;DR¶

360Anything proposes a geometry-free diffusion Transformer framework that treats perspective inputs and panoramic targets uniformly as token sequences. By concatenating these sequences, the model autonomously learns the geometric correspondences between them without requiring any camera metadata. It lifts arbitrary-view images/videos into gravity-aligned, seamless 360° panoramas, completely outperforming prior state-of-the-art methods that rely on ground-truth camera parameters in both image and video tasks.

Background & Motivation¶

Background: "Lifting" perspective perspective images/videos into 360° panoramas is a critical technology for generating immersive 3D worlds, with broad prospects in AR/VR, robotics, and gaming. Existing methods (such as CubeDiff, Argus, Imagine360, etc.) mostly follow the same paradigm: they first project perspective inputs onto an Equirectangular Projection (ERP) space via explicit geometric projection to obtain pixel-aligned conditioning signals, and then perform diffusion generation in that space.

Key Challenge: Explicit geometric projection requires precise camera metadata—Field of View (FoV), pitch, roll, etc. For "in-the-wild" images/videos casually captured on the internet, this information is virtually unavailable. Although external tools (e.g., MegaSaM, GeoCalib) can be used for estimation, their predictions are prone to noise and drift in complex scenes (such as large-range motion, illumination variations, and occlusions), resulting in alignment failures and severely degraded generation quality. This implies that the performance upper bound of existing methods is locked by the accuracy of external camera estimators—they treat "geometric alignment" as a prerequisite rather than a task that the model can learn.

Key Insight & Core Idea: The authors argue that explicit geometric projection is not a prerequisite for panoramic generation. Given sufficient model capacity and training data, a general-purpose architecture can implicitly learn the geometric mapping between perspective and ERP formats from data. The core idea of 360Anything is to treat perspective inputs and panoramic targets uniformly as token sequences. By simply concatenating them along the sequence dimension and feeding them into a diffusion Transformer (DiT), the global self-attention mechanism is left to establish the spatial correspondence between them autonomously—the model must "learn on its own to place perspective contents at the correct positions on the panoramic canvas."

Method¶

Overall Architecture¶

360Anything is built on a pretrained latent diffusion Transformer (DiT). The overall pipeline consists of three stages.

Stage 1: Tokenization and Conditioning. The perspective input \(X_{\text{pers}}\) is encoded into a latent token sequence \(x_{\text{pers}}\) using a standard VAE. The target panorama (which is the ground-truth panorama with added noise during training, and random noise during inference) is encoded into a panoramic token sequence \(y_{\text{equi}}^t\) via Circular Latent Encoding. Then, the two sequences are concatenated along the sequence dimension: \(\text{Concat}([x_{\text{pers}}, y_{\text{equi}}^t])\).

Stage 2: Global DiT Inference. The concatenated token sequence is fed into the DiT. The model establishes relationships among all tokens through global self-attention, learning the mapping from perspective tokens to panoramic tokens, while iteratively denoising using the Flow Matching objective.

Stage 3: Panoramic Decoding. The denoised panoramic tokens are reconstructed into a 360° panorama in the pixel space via Circular Latent Decoding.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Perspective Input<br/>Image / Video"] --> B["VAE Encode"]
    B --> C["Perspective Token Sequence<br/>x_pers"]
    D["Target Panoramic Token Sequence<br/>Inference: Random Noise<br/>Training: Noised"] --> E["Sequence-dimension Concatenation<br/>Concat([x_pers, y_equi_t])"]
    C --> E
    E --> F["DiT<br/>Global Self-Attention<br/>+ Flow Matching"]
    F --> G["Denoised Panoramic Tokens"]
    G --> H["Circular Latent Decoding<br/>Circular VAE Decode"]
    H --> I["Output 360° Panorama<br/>1024×2048 / 512×1024×81 frames"]

Key Designs¶

1. Sequence Concatenation for Conditioning: Allowing DiT to Self-Learn Geometric Correspondence

Existing methods project perspective inputs into the ERP space to obtain pixel-aligned conditioning signals, which strictly requires known camera metadata. 360Anything discards this geometric prior: it directly encodes perspective images into a token sequence \(x_{\text{pers}} = \mathcal{E}(X_{\text{pers}})\) and concatenates it with the noisy panoramic tokens \(y_{\text{equi}}^t\) along the sequence dimension before feeding them into the DiT. The model establishes correspondences between the two types of tokens using global self-attention—it must infer the position, FoV, and orientation of the perspective view on the panoramic canvas to correctly generate the surrounding environment. The key advantages of this design are: (1) it completely eliminates the dependency on camera metadata, making it applicable to arbitrary in-the-wild inputs; (2) the number of panoramic tokens is about 8 times that of perspective tokens, so sequence concatenation only increases the input sequence length by about 12.5% (\(1.125\times\)), incurring minimal computational overhead; (3) the model naturally acquires camera parameter estimation capabilities through data-driven learning (as shown in zero-shot FoV and pose estimation experiments), achieving "generation as inference."

2. Circular Latent Encoding: Eliminating Panoramic Seam Artifacts from the Source

Since the left and right boundaries of an ERP panorama are contiguous, a common artifact in generation is the appearance of visible "seams" at the boundary of the generated results. Prior methods (such as Argus) use inference-time tricks (e.g., rotational denoising, blended decoding) to "mask" the seams. 360Anything is the first to identify the root cause of these seam artifacts: modern diffusion models operate in the latent space of a convolutional VAE, where the VAE encoder uses zero-padding at image boundaries. This introduces discontinuity at the boundaries of the panorama—even if the pixel-level boundaries are seamless, their latent representations already contain a fracture. The proposed solution is simple yet elegant. Before encoding the panorama with the VAE, columns of pixels of width \(w'\) (set to \(W/8\) in experiments) are cropped from both the left and right sides and padded to the opposite sides of the image. This boundary-extended image is then encoded:

\[y_{\text{equi}}^{\text{pad}} = \mathcal{E}(\text{Concat}([Y_{\text{equi}}[-w':], Y_{\text{equi}}, Y_{\text{equi}}[:w']]))\]

After encoding, the latents corresponding to the padded regions are discarded, keeping the sequence length fed into the DiT unchanged. This "wrapping" operation ensures container continuity of latent representations at the boundaries—even when shifting the encoded latents by \(180^\circ\), no discontinuity occurs. Consequently, it eliminates the root cause of seams during the training phase, without introducing any inference-time overhead or quality loss.

3. Canonical Coordinate Training: Enforcing Gravity-Aligned Upright Panorama Outputs

Since the model does not receive explicit camera poses, it inherently does not know in which coordinate system it should output the panorama. Prior works (e.g., CubeDiff, ViewPoint) assumed that the input perspective view is always located at the center of the ERP space, which means the model has to learn different spherical distortion patterns depending on the actual input pose, placing a heavy burden on it. 360Anything employs a "canonical coordinate constraint": training the model to always output a gravity-aligned upright panorama, regardless of the camera pose of the input perspective view. This forces the model to infer the camera pose of the input perspective view and "place" it at the correct position on the upright panoramic canvas. Implementing this constraint requires a two-stage preprocessing on the training data: (1) running rig-based COLMAP on the raw 360° videos to estimate camera poses per frame and rotating each frame to remove inter-frame rotation (i.e., video stabilization); (2) using GeoCalib to estimate the global gravity direction of the stabilized videos and rotating the videos to align the gravity direction with the vertical axis. Through this canonicalization, the model is consistently trained on consistent, upright data, and the generated videos are naturally gravity-aligned without requiring any special treatment during inference.

Loss & Training¶

A Flow Matching training objective is adopted, where the denoiser \(\mathcal{G}_{\boldsymbol{\theta}}\) learns the mapping from a standard normal distribution to the panoramic data distribution:

\[\min_{\boldsymbol{\theta}} \mathbb{E}_{t\sim p(t), Y_{\text{equi}}\sim p_{\text{data}}, \boldsymbol{\epsilon}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})} \|(\boldsymbol{\epsilon}-Y_{\text{equi}}) - \mathcal{G}_{\boldsymbol{\theta}}(Y_{\text{equi}}^t, t, \boldsymbol{c})\|^2\]

The image model is fine-tuned based on FLUX.1-dev (12B parameters) using the Adam optimizer with lr=5e-5 and batch size=512 for 50k steps, outputting a resolution of 1024×2048. The video model is fine-tuned based on Wan2.1-14B with lr=1e-5 and batch size=64 for 20k steps (10k steps on a coarse set at 256×512 + 10k steps on a high-quality set at 512×1024), generating 81 frames. During training, text and perspective condition embeddings are randomly dropped with a 10% probability to enable Classifier-Free Guidance (CFG). For data augmentation, the FoV of the perspective view is uniformly sampled within [30°, 120°], with pitch in [-60°, 60°], and roll in [-15°, 15°] during training to simulate diverse input conditions; horizontal rolling augmentation is applied to the panoramas. For video data, simulated + real-world camera trajectories (an 8:2 mix) are used to crop the perspective rendering views, enhancing generalization to complex motions.

Key Experimental Results¶

Main Results¶

Image Panorama Generation (Laval Indoor / SUN360):

Dataset	Metric	360Anything	CubeDiff (Prev. SOTA)	Gain
Laval Indoor	FID ↓	8.0	9.5	-1.5
Laval Indoor	KID (×10²) ↓	0.22	0.32	-0.10
Laval Indoor	FAED ↓	9.8	18.4	-8.6 (-47%)
SUN360	FID ↓	22.4	25.5	-3.1
SUN360	KID (×10²) ↓	1.27	1.33	-0.06
SUN360	FAED ↓	3.8	7.6	-3.8 (-50%)
SUN360	CLIP-Score ↑	28.07	25.00	+3.07

FAED is the only metric evaluated directly on panoramas. 360Anything reduces the error by nearly 50%, indicating a qualitative leap in overall geometry and quality of the generated panoramas.

Video Panorama Generation (Argus 101 test videos):

Trajectory Type	Metric	360Anything	ViewPoint	Gain
Real	PSNR ↑	25.75	23.25	+2.50
Real	LPIPS ↓	0.0468	0.1364	-0.0896
Real	FVD ↓	483.4	844.3	-360.9
Real	Imaging Quality ↑	0.5515	0.5293	+0.0222
Simulated	PSNR ↑	23.64	22.77	+0.87
Simulated	LPIPS ↓	0.0846	0.1326	-0.0480
Simulated	FVD ↓	432.9	957.8	-524.9
Simulated	Aesthetic Quality ↑	0.5394	0.5045	+0.0349

It comprehensively outperforms prior methods across all metrics (including baselines using ground-truth camera parameters as inputs). The significant reduction in FVD indicates that the generated panoramic videos exhibit more natural spherical distortion and better temporal consistency.

Ablation Study¶

Comparison of Seam Elimination Techniques:

Configuration	Image DS ↓	Video DS ↓	Description
Vanilla (No treatment)	9.92	35.52	Significant seam artifacts visible under direct generation
Blended Decoding (Argus)	5.29	19.84	Seams are blurred during inference, but gray line artifacts remain
Circular Latent Encoding (Ours)	3.87	13.28	Root cause eliminated during the training phase, zero inference overhead

Robustness across Conditioning Views (Average FID Degradation):

Method	Avg. Degradation (FID ↑)	Description
w/o Camera Aug.	+6.48	Without camera augmentation, non-standard views degrade severely
CC w/ GT Camera	+1.17	Channel concatenation + ground-truth camera parameters, robust
360Anything (Ours)	+0.98	No camera parameters required, robustness is comparable or slightly superior

Ablation of Video Canonical Coordinates (FVD / Imaging Quality):

Is Canonical	Real FVD ↓	Real Imag. ↑	Simulated FVD ↓	Simulated Imag. ↑
No	559.5	0.4689	527.0	0.4601
Yes (Ours)	470.8	0.5437	449.8	0.5387

Canonical coordinate training significantly improves visual quality (FVD drops by ~15%, Imaging Quality increases by ~7.5 points), despite slightly lower PSNR/LPIPS (non-canonical methods directly place perspective views at the center of the ERP, making reconstruction easier).

Key Findings¶

Sequence Concatenation vs. Channel Concatenation: With equivalent model sizes, sequence concatenation outperforms the channel concatenation variant across various VBench metrics (0.5515 vs 0.5403 in Imaging Quality), while introducing less than a 20% increase in inference wall-clock time (55 min vs 46 min @ A100). Since panoramic tokens are about 8 times the perspective tokens, sequence concatenation only expands the input sequence length by 1.125 times.
Camera Augmentation is Surprisingly Effective: Randomly sampling FoV and poses during training not only preserved performance on standard views (90°, 0°, 0°) but also improved all metrics. Diverse inputs force the model to deeply understand the perspective-to-panoramic geometric correspondence, preventing overfitting to a static mapping.
Zero-Shot Camera Calibration Capability: 360Anything outperforms several supervised baselines in FoV estimation (mean error of 4.93°) and pose estimation (0.87° Roll / 2.56° Pitch on MegaDepth), performing only slightly worse than task-specific methods like MoGe and GeoCalib. This demonstrates that the sequence concatenation mechanism indeed learns accurate geometric correspondences, showcasing implicit geometric reasoning capabilities inside the generation network.

Highlights & Insights¶

Replacing geometric projection with sequence concatenation shifts "explicit hard alignment" to "implicit soft learning." This fundamentally unsubscribes panoramic generation from camera metadata dependency, allowing the model to continuously scale up with data volume.
Identifying and resolving the root cause of panoramic seam artifacts (VAE zero-padding) and offering a simple circular padding solution during training. While prior works could only mask seams using inference tricks, the "root-cause-targeting" approach of this paper can be generalized to other boundary-continuity tasks (such as 360° video, environment maps, etc.).
Canonical coordinate constraints force gravity-aligned upright panoramic outputs even when camera information is absent. This not only enhances visual quality but also enables directly applying downstream 3D reconstruction pipelines to the output without post-processing alignment.
Unexpected zero-shot camera calibration capabilities: The fact that a generative network internally learns accurate geometric correspondences serves as another solid proof of the "generation as inference" paradigm, hinting at the potential of generative models as general-purpose geometric priors.

Limitations & Future Work¶

Dependency on base model capacities: 360Anything is fine-tuned from FLUX/Wan2.1, thereby inheriting their training biases (such as the appearance of tripods or human hands at the bottom of the panorama, which are common in YouTube 360° videos); complex physical scenes (e.g., fluids, hair) remain difficult to handle.
Resolution and temporal length limitations: The current video model only supports 81 frames at a 512×1024 resolution. Since the pixel density of panoramas is around 8 times that of perspective views, enlarging the context window is a necessary direction for future detail enhancement.
Panorama upsampling is still an open problem: Utilizing off-the-shelf perspective video upsamplers on panoramas reintroduces ERP boundary seams and structural distortions, emphasizing an urgent need for panorama-specific upsampling technologies.
Temporal expansion directions: This framework can be integrated with causal DiTs (such as CausVid, SelfForcing) to scale up to long video generation; since panoramas naturally possess a "working memory" trait (as the entire scene is generated simultaneously), they can be further developed toward long-term episodic memory.

vs CubeDiff / ViewPoint: Both methods replace ERP representations with Cubemap and Viewpoint Map respectively to mitigate distortions, but they assume that inputs are centered in the panorama (front view). For non-centered inputs, they must rotate the panoramic frame, resulting in distortion. The sequence concatenation + canonical coordinate training of 360Anything thoroughly bypasses this constraint.
vs Argus: Argus projects the perspective view onto the ERP and applies channel concatenation as a condition, which depends on external camera estimators and uses inference-time blended decoding to mask seams. 360Anything's sequence concatenation eliminates camera dependency, and its circular encoding eradicates seams during the training phase.
vs General DiTs (FLUX/Wan2.1): 360Anything showcases an ingenious application of the DiT architecture to the field of panoramic generation—by simply modifying how tokens are organized (from channel concatenation to sequence concatenation), it learns completely new geometric mappings. This strategy can be transferred to other cross-view or cross-modal generation tasks.
Insights for 3D Reconstruction: Panoramic videos generated by 360Anything can be directly utilized for 3DGS reconstruction (achieving sub-pixel re-projection errors and 100% frame registration rates), paving a new paradigm for end-to-end pipelines from narrow-view monocular videos to 3D scenes.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Replacing geometric projection with sequence concatenation represents a paradigm shift in panoramic generation. It successfully identifies and resolves the root cause of panoramic seam artifacts caused by VAE zero-padding.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations on both image (2 datasets, 5 metrics) and video (2 trajectory categories, 6 metrics) tasks, complemented by zero-shot camera calibration and 3D reconstruction validations. Ablation studies cover all major design choices.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is clear, problem definitions are precise, and technical descriptions flow logically. The narrative arc tracking the seam issue from discovery to solution is complete and highly convincing.
Value: ⭐⭐⭐⭐⭐ Panoramic generation is freed from the requirement of camera metadata availability, directly boosting downstream applications like AR/VR, world models, and 3D reconstruction.