AnyView: Synthesizing Any Novel View in Dynamic Scenes¶
Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: dynamic novel view synthesis, video diffusion model, Plücker rays, multi-view consistency, cross-domain generalization
TL;DR¶
AnyView feeds a monocular video and source/target camera rays into a pretrained video diffusion model, learning implicit spatiotemporal relationships from cross-domain multi-view data without depth reprojection or test-time optimization, and achieves 12.13 dB average PSNR and 1757 FVD on zero-shot extreme-view AnyViewBench tests without guaranteeing that hallucinated occluded content is correct.
Background & Motivation¶
Dynamic novel view synthesis is not simply rotating an image: people, vehicles, and robot arms move as the camera changes position, and the target camera may reveal regions never observed in the input. Traditional dynamic reconstruction can use depth, point tracks, or motion bases to recover a scene. Recent generative approaches often estimate depth, project a point cloud into the target camera, and use a video model to fill the holes. This works well for nearby viewpoints, where most textures can be transferred directly. Across larger viewpoint changes, however, depth errors corrupt the projection and the extensive unknown regions are no longer small holes to repair.
This motivates the authors' critique of common evaluation settings. Results often emphasize small rotations and highly overlapping fields of view, so optimizing pixel metrics does not necessarily demonstrate robustness to extreme camera displacement. Conversely, a purely generative model has rich appearance and motion priors but does not inherently know how two specified camera trajectories observe the same dynamic event. Training camera control on a small amount of synthetic multi-view data can also leave such implicit methods specialized to familiar scenes rather than transferable to real driving, robotics, and human activities.
AnyView avoids committing to a potentially erroneous reprojected image. It conditions on the cameras themselves, learns cross-view relationships from synchronized multi-view videos, and introduces AnyViewBench to test limited-overlap settings. Its objective is plausible completion consistent with observations and stable over time, not unique recovery of the real world from insufficient monocular evidence. Core Idea: use a unified per-pixel ray representation to tell a video foundation model where to look, then use cross-domain multi-view supervision to teach how the same event should appear from another viewpoint, learning extreme dynamic view synthesis as conditional video generation.
Method¶
Overall Architecture¶
The inputs are a monocular RGB video and per-frame source and target camera poses and intrinsics. The output depicts the same time interval along the target camera trajectory. Both cameras may move, and their videos need not start from the same position. Some content overlap must nevertheless exist across the full interval, although that overlap need not occur at the same instant.
The model builds on the approximately 2B-parameter Cosmos-Predict2-2B-Video2World. Ray-space camera conditioning first converts camera information into latents aligned with the video, followed by joint two-view denoising. Domain-balanced training provides learning signals only during training; inference does not access the training datasets, depth maps, or reconstructed point clouds.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Source and target cameras<br/>Per-frame poses and intrinsics"] --> B["Ray-space camera conditioning"]
B --> C["Joint two-view denoising"]
V["Input video and<br/>noisy target latents"] --> C
D["Domain-balanced training"] -.->|Training only: paired-video supervision| C
C --> O["Decode target-view video"]
The implicit representation here is not a separately exported NeRF or 4D Gaussian field. It is the diffusion model's generative capability organized through camera conditioning. Because the model outputs video directly, generating a plausible new view and recovering a verifiable, unique scene geometry remain different claims.
Key Designs¶
1. Ray-space camera conditioning: associate every pixel with its viewing ray
Providing camera matrices directly leaves the model to learn how matrices, pixel coordinates, and fields of view relate. AnyView instead constructs a Plücker representation for every pixel: a ray direction and a moment vector describing its spatial position form a 6-channel map. In the paper's notation, with direction \(r\) and camera center \(o\), the moment is \(m=r\times o\). The direction specifies where the ray points, while the moment supplies its spatial location. This therefore encodes both intrinsics and extrinsics rather than merely a camera rotation angle.
All cameras are first expressed in the reference frame of the target camera's initial frame, so the target trajectory starts at the identity pose. This removes irrelevant variation caused by the absolute coordinate origin. Ray maps can describe non-pinhole cameras because the network receives actual spatial rays for individual pixels rather than being tied to one intrinsic-matrix format. This is representational compatibility, not evidence that the paper evaluated every type of non-pinhole lens.
2. Joint two-view denoising: fuse channels within a view and interact across views in the sequence
A video tokenizer compresses the input by a factor of 4 in time and 8 along each spatial dimension, with \(d=16\) RGB latent channels. Ray directions and moments are each treated as a 3-channel video, encoded separately, and concatenated, giving \(2d\) camera latent channels per view. Matching the spatiotemporal compression grid lets RGB and ray conditions describe the same local region in space and time.
Within each view, RGB and camera latents are concatenated along channels; the two views are then concatenated along the sequence dimension. The Transformer consequently receives \(2thw\) tokens with \(3d\) channels each, where \(t,h,w\) are the temporal and spatial latent dimensions. The source view contains clean video information, whereas the target RGB latents are noisy. Both views carry camera conditions, with rotary positional embeddings and separate view embeddings distinguishing temporal, spatial, and view identity.
This ordering serves a specific purpose: content and viewing rays for the same pixel region are tightly coupled, but equally indexed tokens from different cameras are not overlaid as if they were the same pixel. The model can seek correspondences across the entire sequence, for example using a vehicle revealed later in the source video to constrain an earlier target frame instead of relying only on current-frame depth projection. This describes the spatiotemporal interaction enabled by the architecture, not an explicit matching, optical-flow, or object-tracking module.
The denoising network repeatedly updates the target video in latent space before decoding it into RGB frames. There is no per-scene fine-tuning and no monocular depth error imposed as a fixed geometric condition. The trade-off is that geometry becomes a learned soft constraint, which cannot rule out all plausible but incorrect completions.
3. Domain-balanced training: move camera control beyond a small collection of synthetic scenes
A simple architecture does not imply modest training requirements. Without explicit reprojection supplying pixel correspondences, the model must learn appearance, motion, and cross-view relationships from data. The authors integrate 12 datasets into one training pipeline, grouped into Robotics, Driving, 3D, and Other, with weighted sampling assigning 25% of each batch to every group. Their Kubric-5D variant further diversifies camera trajectories. Internet-scale monocular appearance and motion knowledge primarily comes from Cosmos pretraining, while multi-view training adds camera controllability.
Physical scales also differ across domains. Mixing raw translations from tabletop robot motions and road-scale camera movements would produce very different ray-moment distributions. The implementation selects a translation normalization constant per dataset to keep Plücker values within \([-1,1]\), clipping where necessary. This standardizes camera conditions rather than estimating scale from the input video. It also means deployment on new data still requires correct calibration and scale conventions.
A Worked Example¶
Consider the driving scenario described in the paper: the input is a front-camera video and the target is a synchronized front-left video. A vehicle should appear earlier in the target view but only enters the front camera's field of view in later input frames. Current-frame depth reprojection has no pixels for that vehicle to transfer.
AnyView converts both camera trajectories into ray latents, uses the entire front-camera video as a clean condition, and jointly denoises the front-left sequence. Appearance and motion cues from later frames can help explain earlier target frames, while entirely unobserved backgrounds are completed using pretrained priors. Figure 6 shows a related phenomenon, but this is whole-video regeneration and should not be mistaken for online future prediction using only current and past frames.
Loss & Training¶
The main text specifies L2 supervision on target latents and iterative denoising at inference, but that passage does not fully specify the noise parameterization, loss weights, or sampler. A standard diffusion objective is therefore not reconstructed here and attributed to the authors. The source video and cameras are conditions, paired target videos provide training supervision, and language conditioning is disabled.
Training runs for 40,000 steps on 64 NVIDIA H200 GPUs with a global batch size of 512. The longest image dimension is 384 for the first 30,000 steps, then increases to 576. The learning rate follows a cosine schedule from \(5\times10^{-5}\) to \(1\times10^{-5}\). This continues training from pretrained Cosmos weights; individual test videos do not incur this optimization budget.
Key Experimental Results¶
Main Results¶
AnyViewBench includes 8 in-distribution test entries and 5 zero-shot entries spanning driving, robotics, and human activities. Approximately 10% of in-distribution data is reserved for validation, and official test subsets contain at most 64 episodes per dataset. Selected DROID stations and Ego-Exo4D institutions and activities are held out for zero-shot testing. Thus zero-shot includes new datasets as well as station or activity transfer.
PSNR measures pixel error and SSIM measures structural similarity, with higher values preferred. LPIPS uses VGG features to measure perceptual distance, with lower values preferred. FVD compares feature distributions of generated and real videos, also preferring lower values, but cannot independently establish that a particular occluded object was reconstructed correctly. The paper uses TF-GAN/I3D and only 20–64 samples per dataset; the authors explicitly associate the high FVD values with these small sample sizes.
The following extracts average rows from the paper's Table 3, with each row averaging across datasets within the same split. In-distribution datasets belong to AnyView's training distribution but may be zero-shot for some baselines. Baselines do not fully support every camera trajectory, so the authors adapt them and use DepthAnythingV2 with dataset-specific maximum-depth tuning for methods that require depth. This is therefore not an architecture-only experiment controlling training data and camera interfaces.
| AnyViewBench Split | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
|---|---|---|---|---|---|
| In-distribution | GCD | 13.95 | 0.400 | 0.623 | 4126 |
| In-distribution | GEN3C | 12.44 | 0.333 | 0.617 | 3100 |
| In-distribution | AnyView | 17.78 | 0.533 | 0.399 | 1239 |
| Zero-shot | TrajCrafter | 11.10 | 0.279 | 0.628 | 3004 |
| Zero-shot | GEN3C | 10.34 | 0.257 | 0.667 | 2090 |
| Zero-shot | AnyView | 12.13 | 0.350 | 0.591 | 1757 |
The extreme-view advantage does not imply universal superiority in narrow-view settings. In Table 2, AnyView scores 13.47 dB PSNR on DyCheck iPhone, below TrajCrafter's 14.24 and CogNVS with test-time optimization at 16.94. On ParDom-4D gradual, AnyView reaches 26.29 against GCD's 24.75. The table reports approximately 10 seconds for AnyView on a new video versus approximately 1 hour for CogNVS including test-time optimization. Because complete standardized hardware conditions are not given there, these numbers are not a directly transferable equal-hardware speedup ratio.
Ablation Study¶
The following extracts Table 5, whose three configurations use the same underlying architecture. GCD data only means training on Kubric-4D and ParallelDomain-4D alone; from scratch removes Cosmos pretrained weights. These experiments test data coverage and pretraining, not independent ablations of ray representations or token concatenation.
| AnyViewBench Split | Config | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| In-distribution | From scratch | 16.01 | 0.456 | 0.500 |
| In-distribution | GCD data only | 14.54 | 0.384 | 0.593 |
| In-distribution | Full model | 17.78 | 0.533 | 0.399 |
| Zero-shot | From scratch | 11.37 | 0.267 | 0.641 |
| Zero-shot | GCD data only | 11.00 | 0.286 | 0.682 |
| Zero-shot | Full model | 12.13 | 0.350 | 0.591 |
Key Findings¶
- Zero-shot average PSNR improves by 1.13 dB over GCD data only and by 0.76 dB over training from scratch. Both multi-domain data and pretraining contribute on these aggregate metrics.
- Improvements are not universal across datasets: on in-distribution DROID, training from scratch reaches 15.36 dB versus the full model's 14.47; on LBM, the corresponding values are 19.07 and 17.94. The paper's statement that scratch training consistently degrades performance should be read as an overall trend, not as overriding these exceptions.
- Geometry evaluation uses MapAnything to estimate extrinsics from generated and real videos and then compares them. In Table 4, in-distribution rotation error is 0.654° versus CogNVS's 2.490°; zero-shot errors are 1.781° and 2.759°, respectively. This supports better camera consistency but remains indirect evidence dependent on an external estimator.
Highlights & Insights¶
- Channel fusion within views and sequence fusion across views distinguish two relationships: pixels should be bound to rays, while different cameras should learn correspondences. This gives the conditioning a clearer geometric interpretation than stacking every signal indiscriminately as additional images.
- The paper moves evaluation from texture repair under small displacement to completion of dynamic events under large displacement. Synchronized target videos in AnyViewBench make this harder generative task quantitatively testable rather than relying entirely on subjective demonstrations.
- Figure 8 offers intriguing examples of generating unseen vehicles from headlight reflections or adding oncoming traffic during a left turn. These are qualitative anecdotes that may reflect useful cue exploitation or merely plausible priors; they do not establish causal reasoning ability.
Limitations & Future Work¶
- The authors explicitly acknowledge the need for known input camera poses, the absence of hard geometric guarantees, and plausible but incorrect generation under severe occlusion. Any does not imply factual recovery without overlap, calibration, or restrictions on the scene.
- This note's assessment: architecture, pretraining, and data scale change together. The two ablations do not isolate the contributions of Plücker conditions, view embeddings, and cross-view fusion; equal-data-budget comparisons of conditioning encodings would help.
- This note's assessment: FVD based on only 20–64 episodes per dataset lacks large-sample stability, while pose metrics depend on MapAnything. Larger test sets, confidence intervals, and cross-view correspondence metrics focused on moving objects would strengthen the conclusions.
- Soft geometric constraints or uncertainty outputs could preserve implicit completion while indicating which regions follow observations and which rely mainly on priors. These are future research suggestions, not implemented capabilities of this paper.
Related Work & Insights¶
- vs GCD: Both follow an implicit generative approach. AnyView uses a stronger Cosmos backbone, per-pixel ray conditions, and a broader data mixture. Table 5 shows that placing the new architecture on the old pair of synthetic datasets is insufficient to recover the same cross-domain generalization.
- vs GEN3C / TrajCrafter / CogNVS: These methods rely on depth reprojection or inpainting, with CogNVS additionally using test-time optimization. They can exploit direct geometric information when overlap is substantial, and AnyView does not surpass them universally on narrow-view tests. Its main advantage appears under extreme viewpoints where reprojection supplies insufficient information.
- vs Shape of Motion: That method uses depth and point tracks to construct an explicit dynamic representation and is closer to reconstruction. AnyView directly generates conditional video, making it relevant to viewpoint changes in world models or policy data augmentation, although those downstream benefits are not quantitatively established here.
Rating¶
- Novelty: 4/5. A simple combination whose main contributions are unified conditioning, cross-domain training, and a stricter task setting rather than complex new network modules.
- Experimental Thoroughness: 4/5. Covers in-distribution, zero-shot, geometry, and pretraining/data ablations, but test scale, interface differences, and component ablations remain limitations.
- Writing Quality: 4/5. The architecture and evaluation goals are clear, although some broad claims need to be read alongside exceptions in the tables.
- Value: 4/5. Provides a reusable approach and evaluation for extreme dynamic view synthesis, without making perceptual plausibility a substitute for true geometry or safety guarantees.