CameraAnything: Refilming Videos with Arbitrary Camera Control

Conference: ECCV2026
Paper: ECCV Paper
Project: CameraAnything
Area: 3D Vision / Video Generation
Keywords: camera control, video retakes, focal-length editing, resolution adaptation, multi-shot generation

TL;DR

CameraAnything injects per-pixel camera rays into the attention positional encoding of a video diffusion model and trains with independently varied synthetic camera supervision to jointly change viewpoint, focal length, output format, and shot transitions in one generation process, reducing DAVIS retake rotation error from ReCamMaster's 5.11982 degrees to 2.7600 degrees.

Background & Motivation

Video retaking is not generating a similar scene from text: it preserves the people, actions, and timeline of an existing video while changing how a camera observes them. Among existing approaches, TrajectoryCrafter recovers point clouds from monocular depth, renders along a target trajectory, and uses a generative model to fill occluded regions; geometric errors in complex dynamic scenes propagate into the output. ReCamMaster instead learns a generative mapping between synchronized camera views, but primarily controls extrinsics and does not fully express changes in focal length or output format. Moving a camera smoothly is also different from turning a continuous take into a multi-shot sequence with abrupt viewpoint changes.

The difficulty is that an object can grow larger in the image because the camera moves closer or because the focal length increases, yet these operations produce different perspective effects. A classic dolly zoom even changes position and focal length together to approximately preserve subject size while altering background perspective. If training examples always bind the operations together, the model cannot learn which visual change belongs to which control variable. Output format is likewise more than a final cropping operation: converting landscape to portrait can change the pixel grid, principal point, and corresponding rays, requiring composition under new imaging geometry.

The paper therefore changes both conditioning and training data rather than merely adding a focal-length scalar to the network. Per-pixel rays associate camera position, orientation, and intrinsics with individual visual tokens, while the target resolution determines the latent grid and its positional encoding. Synchronized synthetic videos supervise unchanged actions under changed camera settings, and independent control-axis sampling exposes both isolated edits and compound instructions. Core Idea: use geometric conditioning to specify where each target pixel should look, then train with paired examples that keep some control dimensions unchanged so viewpoint, focal length, and format can be edited independently or together.

Method

Overall Architecture

Inputs comprise a source video, source camera parameters, and per-frame target camera settings; the output preserves source content and dynamics while following the requested cinematography. The backbone is Wan2.1-T2V-1.3B, which encodes video using a 3D VAE and denoises target latents using a diffusion Transformer. Source video tokens and noisy target tokens are concatenated along the frame dimension, allowing every self-attention layer to connect source content with the target frames. The source branch supplies content references, the target branch supplies positions to synthesize, and both carry their respective camera geometry. Training first constructs synchronized multi-camera pairs and selects tasks along independent control axes; generation interprets the selected target settings through ray and RoPE conditioning. The pipeline does not first optimize an explicit 3D scene for each input video, nor does it generate a fixed format before outpainting it.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Synchronized Synthetic Pairs"] --> B["Orthogonal Control Sampling"]
    B -->|Training pairs and camera parameters| C["Per-Pixel Ray Conditioning"]
    S["Inference inputs<br/>Source video and target settings"] --> C
    C --> D["Resolution-Aware<br/>RoPE Injection"]
    D --> E["Source and target tokens<br/>Joint-attention denoising"]
    E --> F["Retake at the target format"]

Key Designs

1. Synchronized Synthetic Pairs: preserve action timing while changing cinematography

The authors construct dynamic environments with animated people in Unreal Engine 5 and record 20 strictly time-aligned videos per scene. Clips 1–5 in Group A provide fixed-focal, continuous single-shot base views that anchor input content and motion. Clips 6–10 in Group B introduce 1–3 instantaneous cuts per video, allowing abrupt changes in target camera position and orientation. Clips 11–20 in Group C reuse the extrinsic trajectories of A and B with different focal lengths, exposing the same camera motion under different fields of view. Crucially, changing focal length does not require a different performance of the action, which could otherwise be mistaken for a consequence of the camera control. Single-shot sources and multi-shot targets also share an action timeline, so a cut should change the observation point rather than restart the action.

Raw footage is rendered as high-resolution squares, and parameterized center cropping produces training samples with different aspect ratios and resolutions. This does not contradict avoiding cropping at inference: cropping constructs supervision, whereas the final generation operates directly on the target grid. The important distinction is that the model regenerates the entire target frame and does not guarantee exact pixel preservation of a source crop. The main paper does not report the complete number of training scenes, so 20 clips per scene must not be presented as the dataset's total size.

2. Orthogonal Control Sampling: learn disentanglement by holding some dimensions fixed

Each training sample takes its source from a single-shot, fixed-focal video, then independently selects changes in extrinsics, focal length, and aspect ratio. Extrinsic modes are same-camera, different single-shot, and single-to-multi-shot, sampled with probabilities 0.2, 0.4, and 0.4, respectively. A focal-length change is independently enabled with probability 0.5, and an aspect-ratio change with probability 0.2. The focal branch uses re-rendered versions of identical extrinsic trajectories, preventing optical zoom from being confounded with an additional pose change. When format editing is enabled, source and target resolutions are sampled independently from buckets while ensuring different aspect ratios.

The axes yield up to \(3\times2\times2=12\) nominal task combinations, but the combinations are not equally probable. If sampling produces the same camera with neither focal nor format changes, the authors convert it into a focal-change sample to avoid a completely unchanged training pair. Consequently, identity samples in the ablation mean examples with some unchanged control axes, not a large set of entirely unedited video pairs. Removing such examples makes every dimension change together: the network still sees variation but loses the controlled comparisons that identify each variable's independent effect. This is data-driven disentanglement rather than an additional explicit disentanglement loss.

3. Per-Pixel Ray Conditioning: turn frame-level camera parameters into local viewing directions

The frame-level baseline flattens the \(3\times4\) extrinsic and \(3\times3\) intrinsic matrices into a 21-dimensional vector and broadcasts one embedding to every spatial token in the frame. It describes the camera globally but does not directly distinguish the viewing rays of a corner pixel and the image center. The proposed approach constructs a six-dimensional per-pixel ray representation from camera intrinsics and extrinsics, then supplies it to every DiT block after patchification and learned projection. All camera poses are expressed relative to the first source frame, avoiding dependence on an arbitrary world coordinate system. Focal length enters the intrinsics and changes pixel viewing directions; camera rotation and translation place those rays in a shared coordinate system.

Changing target resolution reconstructs target intrinsics, the principal point, and the ray field rather than attaching a resolution label to an unchanged grid. The number of target tokens and their imaging geometry can therefore change together, supporting native outputs with different aspect ratios. However, cached Eq. (3) is badly corrupted, and the following prose describes the six-dimensional representation as origin plus direction, whereas the introduction describes direction plus moment. Standard Plücker coordinates commonly use direction and moment, so the name alone does not justify replacing the paper's expression with a cross-product formula. This note retains the supported per-pixel ray-conditioning mechanism; the precise coordinate convention requires verification against a clear original or implementation.

4. Resolution-Aware RoPE Injection: affect attention matching instead of directly rewriting features

The authors compare additive injection, AdaLN injection, and RoPE injection. Addition directly changes token hidden states; AdaLN predicts camera-conditioned scale and shift parameters that modulate normalized feature distributions. The selected design follows ReDirector's RoCE approach: a lightweight MLP maps camera-ray conditions to phase shifts, applied to self-attention queries and keys alongside the original 3D RoPE. Camera conditioning consequently enters token-similarity computation instead of simply adding a camera vector to visual content. Intuitively, whether a source token can inform a target token depends on their viewing geometry as well as their appearance. This injection is not an exact ray-correspondence solver: geometric consistency is learned rather than enforced as a hard constraint.

Target format also determines the spatial coordinates used by the original 3D RoPE. Page 9 gives the latent patch-grid relationship, with VAE spatial compression \(s_{\mathrm{vae}}\) and patch stride \(s_{\mathrm{patch}}\):

\[ h'=\frac{H_t}{s_{\mathrm{vae}}s_{\mathrm{patch}}},\qquad w'=\frac{W_t}{s_{\mathrm{vae}}s_{\mathrm{patch}}}. \]

A resolution change recomputes positional coordinates on the new grid and uses rays from the new target camera; it does not retain the old token layout and merely resize the decoded image. The paper specifies VAE spatial compression of 8 and temporal compression of 4, but does not explicitly give the numerical patch stride here. Source and target token counts can differ when their resolutions differ, so the preliminary description of a \(2N\) sequence for equal sizes is not a fixed length for every editing task. Inference also requires source camera parameters; the main paper does not detail their complete calibration pipeline for real inputs, so ViPE evaluation must not be assumed to describe that preprocessing.

A Worked Example

Consider the landscape-to-portrait setting in the paper: a person keeps walking in the source video, while the target requests an 832×480 portrait format and a cut to another viewpoint partway through. During training, a source can come from Group A and its target from the corresponding multi-shot trajectory and re-focal variant, with both aligned to the same walk. At inference, the user supplies target camera and focal settings without providing a real target video. The model constructs per-pixel rays from target intrinsics and poses, then builds target latents and RoPE spatial coordinates matching the portrait dimensions. During denoising, target tokens read source content through joint self-attention; a cut changes camera conditions abruptly but should not reset the action timeline. The result decodes directly into a portrait retake instead of generating a landscape retake and cropping the person from its center. This example explains module interaction and is not an additional per-example quantitative experiment reported by the paper.

Loss & Training

Training updates only self-attention layers, the camera encoder, and camera adapters; other backbone parameters are frozen, and newly introduced camera modules are zero-initialized. The model trains for 20k steps with a learning rate of \(5\times10^{-5}\) and a batch size of 32. Reported resolution buckets are 480×832, 832×480, 384×672, 672×384, 512×672, 672×512, 384×512, 512×384, 640×640, and 320×320. The work uses the pretrained video-generation backbone's training framework, but the readable main text does not separately provide the complete denoising objective, sampler settings, or inference step count. This note therefore does not substitute a generic diffusion loss for the authors' exact implementation or infer memory usage and speed from model size.

Key Experimental Results

Main Results

Synthetic evaluation uses 6 settings from each of 50 unseen scenes, totaling 300 video pairs; targets cover format, single-shot extrinsics, multi-shot extrinsics, and combinations with focal changes. Real evaluation uses 50 DAVIS videos with 6 synthesized target trajectories each, also totaling 300 pairs, but has no real target videos. Real camera errors are computed after ViPE estimates generated-video poses: RotErr is mean rotation error after first-frame normalization, and TransErr is mean translation error after trajectory-arc-length scale alignment. TransErr is therefore not an absolute position error in meters, and real evaluation cannot report paired-ground-truth PSNR.

The following selection comes from Table 1 on page 12 and Table 2 on page 13; higher PSNR and SSIM are better, while lower LPIPS, FVD, and camera errors are better.

Source and task	Metric	Baseline A	Baseline B	CameraAnything
Table 1, synthetic extrinsic/focal	PSNR	TrajectoryCrafter 12.24	ReCamMaster 12.87	15.88
Table 1, synthetic extrinsic/focal	LPIPS	TrajectoryCrafter 0.654	ReCamMaster 0.607	0.179
Table 1, synthetic extrinsic/focal	FVD	TrajectoryCrafter 462.9	ReCamMaster 351.5	231.0
Table 1, synthetic resolution editing	PSNR	Follow-Your-Canvas 19.91	VACE 19.45	20.54
Table 1, synthetic resolution editing	SSIM	Follow-Your-Canvas 0.765	VACE 0.714	0.758
Table 1, synthetic resolution editing	LPIPS	Follow-Your-Canvas 0.206	VACE 0.194	0.174
Table 2, DAVIS extrinsic/focal	RotErr, degrees	TrajectoryCrafter 13.9916	ReCamMaster 5.11982	2.7600
Table 2, DAVIS extrinsic/focal	TransErr	TrajectoryCrafter 0.9949	ReCamMaster 0.4628	0.3309

Extrinsic/focal PSNR in Table 1 exceeds ReCamMaster by 3.01, but resolution-editing SSIM is below Follow-Your-Canvas, so the method is not best on every metric. In Table 2, ReCamMaster achieves subject consistency of 0.9094 versus 0.8410 for this method; the authors attribute this to retaining the source first frame and avoiding multi-shot transitions, which requires generating fewer novel regions. This is a possible explanation for the metric difference, not controlled proof that evaluation bias has been eliminated.

Ablation Study

Table 3 on page 13 compares camera representations and injection mechanisms with other settings fixed; all three reconstruction metrics are retained below.

Representation	Injection	PSNR	SSIM	LPIPS
Linear, 21-dimensional frame-level	Addition	15.62	0.481	0.434
Plücker Ray, per-pixel	AdaLN	16.51	0.517	0.368
Plücker Ray, per-pixel	RoPE	16.66	0.528	0.372

RoPE exceeds AdaLN by 0.15 PSNR and 0.011 SSIM, but LPIPS slightly worsens from 0.368 to 0.372. Moving from the first row to the second changes both representation and injection, so that improvement cannot be attributed entirely to per-pixel rays. Absolute values in Table 3 differ from the Table 1 aggregates; improvements must not mix these tables, and the main text does not fully explain their aggregation difference.

Table 4 on page 15 compares training strategies on the synthetic benchmark; each cell reports PSNR / SSIM / LPIPS.

Config	Extrinsic tasks	Focal tasks	Resolution tasks
Full model	14.13 / 0.423 / 0.483	20.03 / 0.606 / 0.213	20.05 / 0.743 / 0.187
Extrinsic-only	14.00 / 0.414 / 0.497	Not applicable	Not applicable
Without partially unchanged control samples	13.87 / 0.406 / 0.503	15.94 / 0.475 / 0.399	14.64 / 0.456 / 0.414

Key Findings

• Removing partially unchanged control samples reduces focal-task PSNR from 20.03 to 15.94 and resolution-task PSNR from 20.05 to 14.64, showing that compound variation alone does not teach independent control.
• Joint training achieves extrinsic PSNR of 14.13 versus 14.00 for extrinsic-only training, showing no clear extrinsic-performance sacrifice on this benchmark.
• Better real-scene pose following is distinct from leading every perceptual dimension; the stronger evidence concerns camera accuracy, reconstruction quality, and the range of supported controls.

Highlights & Insights

• Task disentanglement depends on paired examples that keep some factors unchanged, not only on network architecture. Table 4 shows that controlled data variation is itself supervision.
• Resolution enters both ray geometry and the positional grid rather than only changing final image dimensions. Native-format generation therefore has explicit conditioning inputs.
• Cuts are represented as abrupt changes in per-frame camera conditions and learned from synchronized target videos. This places smooth motion and editing transitions in the same video-generation process.

Limitations & Future Work

• The main text has no detailed failure-case or limitations section; the following points primarily reflect this note's assessment of the method and evaluation rather than individually acknowledged author conclusions.
• Training depends on synthetic dynamic scenes, and transfer to DAVIS does not establish reliability for every complex material, occlusion, long-duration action, or extreme camera operation.
• Arbitrary control describes the supported interface, not comprehensive testing of every focal length, resolution, and pose range; the main text provides no full ranges or extrapolation curves.
• Real pose errors depend on ViPE estimation reliability, and the main text provides no dedicated real focal-recovery or cut-timing errors, leaving room for stronger evaluation of joint intrinsic control.
• Reliable acquisition of source camera parameters, precise cross-resolution token organization, inference latency, and memory cost are insufficiently documented for straightforward reproduction.
• Cached Eqs. (1), (3), and (6) are visibly corrupted, and ray descriptions disagree about origin versus moment; the supplementary user study is absent from this full-text cache, so its participant counts or preference rates cannot be cited.

Related Work & Insights

• Compared with ReCamMaster: the method retains frame-dimension source/target concatenation but expands frame-level extrinsic conditioning into per-pixel geometry, focal control, and native formats, with multi-shot transitions included in training.
• Compared with ReDirector: RoCE is an existing attention-injection idea; the contribution here is combining rays, target-resolution grids, and multi-axis supervision into unified editing, not inventing camera-aware RoPE itself.
• Compared with TrajectoryCrafter: the latter reconstructs explicit geometry before re-rendering, whereas this approach learns cross-view mappings through generative priors; removing reconstruction also means newly exposed regions depend largely on model completion.
• Compared with Follow-Your-Canvas and VACE: outpainting or reference-based editing tends to preserve existing regions, while this method regenerates complete target frames; they make different trade-offs between pixel preservation and new composition.
• Possible extension, not a reported result: independent calibration tests for each camera axis, followed by progressively stronger compound operations, could distinguish accepting control parameters from genuinely following them.

Rating

• Novelty: 4/5. The main contribution unifies cinematographic controls and data supervision rather than introducing a completely new backbone or attention operator.
• Experimental Thoroughness: 4/5. Synthetic, real, and training-ablation evidence is provided, but real intrinsic accuracy and efficiency need fuller reporting.
• Writing Quality: 3/5. Task and dataset design are clear, while ray definitions, cross-resolution details, and some cross-table evaluation conventions need clarification.
• Value: 4/5. The framework offers practical ideas for recomposition, shot design, and cross-platform adaptation, but the evidence does not establish consistently production-ready quality.