GimbalDiffusion: Gravity-Aware Camera Control for Video Generation

Conference: ECCV2026
Paper: ECCV Paper
Project: https://lvsn.github.io/GimbalDiffusion/
Area: Video Generation
Keywords: gravity alignment, absolute camera control, panoramic video, extreme rotation, null-pitch conditioning

TL;DR

GimbalDiffusion constructs extreme-view training samples from gravity-aligned panoramic videos and captions level-view crops to reduce prompt-angle entanglement, lowering WAN 2.2 5B PitchErr on SpatialVID-extreme from 14.39° without null-pitch conditioning to 8.36°, although not every visual quality metric improves.

Background & Motivation

Text-to-video models can synthesize detailed scenes, but convincing appearance does not guarantee obedience to camera commands. Cinematic shots require more than forward motion: a camera may look at the sky, point at the ground, roll around its optical axis, or complete a full rotation. Methods such as AC3D accept camera trajectories but typically define their coordinate system relative to the first frame. This represents how the camera subsequently moves without uniquely specifying its initial orientation relative to the physical ground. If the first frame already looks upward, relative transformations alone cannot communicate that state to the model. Human-captured videos also concentrate on approximately level views, leaving large pitch and roll underrepresented in training.

A gravity reference and more rotation samples are still insufficient because text implicitly specifies a viewpoint. Captioning a downward-looking video of grass often produces a grass-only description, allowing the model to learn a shortcut from “grass” to “look down.” When a user describes grass, trees, and sky but requests an upward-facing camera, text and camera conditions can compete over composition. A correct output should show what is visible from that viewpoint, rather than force every mentioned object into the image. This is also why ordinary text similarity cannot independently establish success on this task.

The paper uses a single 360° video to supply real dynamic content in different directions, constructing rotated crops for supervision and level crops for captioning. The former cover extreme viewpoints, while the latter reduce pitch and roll cues in the text; both remain connected through the same panoramic scene. Here, “absolute” specifically refers to gravity-aligned pitch and roll; gravity cannot determine a compass-like absolute yaw. Core Idea: supervise scene description separately from the actual viewing angle, letting text describe the scene and gravity-aligned camera conditions determine what the camera can see.

Method

Overall Architecture

Training starts from 360° panoramic videos: the pipeline estimates source camera motion and gravity, samples rotations and fields of view, and projects ordinary perspective videos. The rotated crops supervise generation, crops with pitch and roll removed provide captions, and UCPE or Plücker encodings inject the camera parameters into the video model. Inference requires only a scene prompt, per-frame intrinsics, and gravity-aligned extrinsics; users need neither a panorama nor a separately generated reference image. Panoramic videos are therefore a training source, not an intermediate panorama that must be generated and cropped at inference time. The four key designs are gravity calibration, trajectory sampling, null-pitch captions, and camera conditioning, in that order.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training: panoramic video"] --> B["Gravity calibration"]
    B --> C["Trajectory sampling"]
    C --> D["Null-pitch captions"]
    D -->|Training text| E["Camera conditioning"]
    C -->|Target video supervision and camera conditions| E
    P["Inference: text and absolute cameras"] -->|Inference conditions| E
    E --> F["Generated perspective video"]

Key Designs

1. Gravity calibration: establish a shared vertical reference across videos

Each panorama is projected into six perspective directions: front, back, left, right, top, and bottom; ViPE then estimates camera motion. These estimates still inhabit an arbitrary reference frame, so the pipeline first expresses motion relative to the initial frame instead of treating the estimated coordinates as physical world coordinates. Following Eq. (3), the relative pose is:

\[ \mathbf{E}_{\mathrm{rel},f}=\mathbf{E}_{\mathrm{SfM},0}^{-1}\mathbf{E}_{\mathrm{SfM},f}. \]

Here, \(f\) indexes frames and \(\mathbf{E}_{\mathrm{SfM},f}\) is an estimated camera pose; referencing the initial frame removes each sequence's arbitrary starting transformation. To determine the physical up direction, the pipeline extracts eight perspective views with a 90° field of view, evenly spaced in yaw, from the first panorama. Perspective Fields predicts their up vectors, which are rotated back to a common zero-yaw reference and averaged. Observations from several directions reduce reliance on whether any single crop contains a clear horizon or vertical structure. The relative motion, estimated gravity, and sampled rotations are then combined into gravity-aligned per-frame extrinsics, with the sequence initialized at zero yaw.

This convention permits subsequent yaw changes; it does not prohibit the camera from turning left or right throughout the video. It provides neither geographic position nor a globally shared north direction, only consistent meanings for pitch and roll across videos. Translation is transformed along with the coordinate alignment, but the available translational motion still comes from the source video rather than being created by spherical cropping. Eq. (4) is corrupted in the text extraction, so its exact rotation composition cannot be reliably recovered; this account explains the operations confirmed by the prose without inventing a precise matrix formula.

2. Trajectory sampling: derive many extreme shots from the same dynamic panorama

The pipeline samples an initial orientation and combines several smooth rotations around random 3D axes over the clip. Algorithm 1 samples initial pitch and roll from \([-90^\circ,90^\circ]\) and yaw from \([0^\circ,360^\circ]\). The number of additional rotation axes is written as \(N\in\{0,1,2,3\}\), which includes samples without any additional rotation. The prose instead mentions 1 to 3 additional rotations; this minor discrepancy is retained here, with the algorithm's range stated explicitly rather than silently reconciling the two. Each additional rotation has random start and end times, an axis, and an amplitude, distributed smoothly across frames using a cubic spline. Its maximum angular displacement is sampled as \(\operatorname{Beta}(1.0,5.0)\cdot720^\circ\cdot d_t\), where \(d_t\) is the normalized duration. This provides common small motions while retaining opportunities for large rotations; it does not require every sample to rotate through 720°.

Field of view is independently sampled at 1, 2, or 3 keyframes between 35° and 100°, then interpolated with cubic splines having randomized endpoint derivatives. The same source clip can consequently yield upward views, downward views, rolls, or perspective videos with changing fields of view. Because the panorama frames already contain dynamic objects, these samples are not novel views extrapolated solely from a static single-image point cloud. However, full directional coverage does not mean uniform sampling by spherical area: the authors note a slight bias toward the poles, which they find useful for extreme-view stability. The source camera's translation remains in place, so rich rotational augmentation does not imply that arbitrary translational augmentation has been solved.

3. Null-pitch captions: change the caption source, not the generation target

For each sampled trajectory, the pipeline creates an additional set of level crops with pitch and roll set to 0°, unchanged yaw, and a 90° field of view. InternVL-3-2B reads six uniformly spaced temporal frames from these crops to produce the training caption. The original rotated crops remain the targets for the diffusion or flow loss; the level crops must not be mistaken for generation targets. This repeatedly presents training pairs in which the caption describes the surrounding scene while the target image covers only a particular direction. The model must therefore use explicit camera conditions to determine visibility instead of guessing the angle from whether the caption mentions sky or ground. The level crop is not an exhaustive panoramic semantic inventory either; it supplies a scene description with fewer pitch and roll biases.

Panoramic capture introduces another problem: a tripod or a hand holding a selfie stick may appear directly below the camera but never in the level-view caption. Without intervention, the model can learn these objects as ordinary content that should automatically appear in downward views. The authors therefore create another crop set at a pitch of -90° and caption it with the vision-language model. During training, this description is appended to the null-pitch caption with 50% probability; at inference, the look-down description becomes a negative prompt for classifier-free guidance. The aim is to make unwanted capture equipment explicitly suppressible through text, not to add a post-processing repair network. The authors find this particularly useful for fully fine-tuned models prone to fitting training artifacts, but the main paper provides no separate quantitative ablation of this component.

4. Camera conditioning: reuse existing encodings with different coordinates and training data

GimbalDiffusion is not a new camera encoding; it is a data and training framework compatible with different backbones and conditioning mechanisms. The UCPE branch injects camera-ray transformations through additional trainable self-attention layers together with spatiotemporal RoPE; latitude and up-vector information is linearly mapped into the attention input. The authors use UCPE by default because it better preserves translation accuracy experimentally. The alternative branch uses a six-dimensional Plücker ray per pixel, with a ray moment in the first three dimensions and a unit direction in the last three:

\[ \mathbf{p}_{f,u,v}=\left(\mathbf{t}_f\times\mathbf{d}'_{f,u,v},\;\mathbf{d}'_{f,u,v}\right). \]

Here, \(\mathbf{t}_f\) is the camera center in world coordinates, and \(\mathbf{d}'_{f,u,v}\) is the normalized pixel ray obtained by inverse projection through the intrinsics and rotation into world coordinates. The same six-dimensional form can store either relative or gravity-aligned coordinates; the crucial differences are the pose convention and whether the model was trained with it. Consequently, feeding absolute Plücker rays at test time to AC3D, which originally saw relative cameras, does not substitute for the proposed training. The CogVideoX-2B branch uses a camera encoder and ControlNet-style parallel layers, whereas WAN variants use either full fine-tuning or UCPE module training. These configurations change both the backbone and the trainable parameter set, so their results must be interpreted within the original table's groups. UCPE Eqs. (1) and (2) are also damaged in the extraction; missing operators are not reconstructed here as implementation-ready attention equations.

A Worked Example

Consider a forest containing a dirt path, trees, and blue sky, with a requested camera fixed upward; this illustrates the mechanism rather than introducing a new experiment. During training, a level crop from a similar panorama may describe the path and forest, while the random target crop points upward at +90° and mostly shows the canopy and sky. The model receives the scene caption, a gravity-aligned upward-facing camera, and the corresponding target video, learning that the path need not appear from every viewpoint. At inference, the user supplies an ordinary forest description together with +90° camera conditions, and the model should prioritize camera direction over displaying every named object. Changing the pitch to -90° should reveal the ground; adding roll should rotate the view around the optical axis rather than make the scene automatically appear upright. The look-down negative prompt suppresses capture artifacts such as selfie sticks, serving a different purpose from the positive forest description.

Loss & Training

Training uses the YouTube subset of PanoVid, removing videos whose shortest side is below 900 pixels and clips with failed pose estimates or abnormal mean acceleration. Section 4.1 reports filtering 7797 videos down to 4798, discarding approximately 38%. Each training iteration generates 49 frames at 720 × 480 or a similar backbone-compatible resolution, resampling the trajectory and producing a new caption. Caption length is also sampled among short, medium, and long descriptions to avoid dependence on a fixed style. The method retains the chosen backbone's diffusion or flow loss; its contribution is the training pairing and conditioning convention, not an additional gravity-error loss. The main paper supplies no single learning rate, step count, or full loss expansion shared across all backbones, so implementation parameters cannot be filled in from this evidence. The AC3D ControlNet path is described as copying 8 layers and operating only during the first 40% of denoising steps; the WAN 2.2 5B version uses a smaller camera encoder to reduce parameter count. These describe different implementation configurations, not one architecture shared by every variant.

Key Experimental Results

Main Results

Source: Table 1, page 12; this excerpt contains matched-backbone comparisons on SpatialVID-extreme, and every listed entry supports dynamic scenes. The benchmark selects 138 clips from SpatialVID-HQ's 371K clips, covering pitch from -85° to +85° in 10° bins, then adds random roll trajectories between -40° and 40° by warping the images. Test conditions retain ViPE translations normalized to a unit maximum displacement but resample rotations, so the task is not pixelwise reconstruction of the original video. PitchErr measures pitch-angle error; GravityErr measures the angle between target and estimated generated-frame up vectors in camera coordinates; both use Perspective Fields. RotErr uses VGGT to estimate relative rotation error, and all three angular errors are in degrees; TransErr compares normalized translation trajectories rather than metric distances. Higher CLIP is better; lower angular errors, TransErr, FID, and FVD are better.

Backbone and conditioning	Method	PitchErr ↓	GravityErr ↓	RotErr ↓	TransErr ↓	CLIP ↑	FID ↓	FVD ↓
CogVideoX-2B / Plücker	AC3D + cam. text.	39.09	44.13	29.27	0.62	23.1	113.0	984.1
CogVideoX-2B / Plücker	Ours	21.19	24.84	18.48	0.68	21.1	110.9	915.7
WAN 2.1 1.3B / Plücker	PreciseCam + WAN-I2V-CC	29.16	33.48	24.54	0.62	22.3	109.2	776.3
WAN 2.1 1.3B / Plücker	Ours	10.50	13.05	25.84	0.68	20.5	114.7	1041.6
WAN 2.1 1.3B / UCPE	UCPE	16.25	18.88	18.38	0.44	22.5	110.7	957.9
WAN 2.1 1.3B / UCPE	Ours	13.98	17.16	12.12	0.46	21.2	112.5	978.1

For WAN 2.1 1.3B / UCPE, the proposed method achieves RotErr of 12.12° versus the original UCPE's 18.38°, but TransErr changes from 0.44 to 0.46. The WAN 2.1 Plücker branch substantially improves absolute orientation yet has worse FVD than PreciseCam + WAN, ruling out a claim of uniformly better visual quality. Table 1 also lists PreciseCam + GEN3C, but its static-scene novel-view synthesis should not be equated with dynamic-scene generation.

Ablation Study

Source: Table 1, page 12; the two groups use different backbones and training strategies, so ablation comparisons must remain within each group. “Without null-pitch” means captioning the actual rotated target video; “without absolute rotations” converts cameras back to poses relative to the first frame.

Backbone and conditioning	Config	PitchErr ↓	GravityErr ↓	RotErr ↓	TransErr ↓	CLIP ↑	FID ↓	FVD ↓
WAN 2.2 5B / Plücker	Without null-pitch	14.39	16.36	17.26	0.74	22.0	110.6	895.2
WAN 2.2 5B / Plücker	Full model	8.36	10.54	15.05	0.74	20.6	116.0	1014.4
WAN 2.1 1.3B / UCPE	Without null-pitch	18.92	21.96	11.21	0.47	22.6	110.6	979.2
WAN 2.1 1.3B / UCPE	Without absolute rotations	50.62	56.18	14.35	0.53	21.7	108.9	862.4
WAN 2.1 1.3B / UCPE	Full model	13.98	17.16	12.12	0.46	21.2	112.5	978.1

Key Findings

• The absolute reference is essential information: removing it raises the UCPE branch's PitchErr from 13.98° to 50.62°, approximately 3.62 times as high according to Section 4.5.
• Null-pitch improves absolute angles without guaranteeing improvement in every rotation metric: UCPE RotErr is 11.21° without it and 12.12° for the full model.
• The second benchmark appears in Section 4.6, page 14, and Figure 6, page 15: 20 PolyHaven panoramas are each tested at static pitches from -90° to +90° in 10° steps, producing 380 prompt-angle pairs.
• Figure 6's revised CLIP evaluation uses a caption of the real crop at each target angle instead of always using the level-view input caption; its curves support disentanglement at extreme pitch, but the cache provides no precisely transcribable per-angle values.

Highlights & Insights

• The transferable idea is to decouple the control variable from the caption source. Captioning a different reference crop changes the semantic shortcut used to infer the camera without requiring a new loss.
• Gravity supplies a specific inductive bias: it identifies up and down. This resolves a coordinate ambiguity rather than endowing the video model with complete physical simulation.
• Online panoramic cropping turns the same scene into many camera conditions, increasing viewpoint diversity rather than the number of independent scenes. That distinction clarifies both the augmentation benefit and its generalization limits.

Limitations & Future Work

• The authors explicitly note that augmentation freely changes rotations but retains source translations; novel-view synthesis such as 3D Gaussian Splatting could extend translation diversity, but that extension is not implemented here.
• They also acknowledge geometric artifacts, citing the Tokyo Skytree example in Figure 5; more accurate camera orientation does not imply more accurate scene geometry.
• In the evaluation design, absolute orientation depends on Perspective Fields and motion on VGGT, while training annotations depend on ViPE and gravity estimation, so reported errors are not pure generation errors measured against sensor ground truth.
• The 138 extreme-view clips and 20 panoramas offer targeted diagnostics but do not establish reliability for all scenes, long videos, or arbitrary translations; this is an assessment of evidence scope, not an additional failure experiment reported by the authors.
• Reproduction limits include damaged Eqs. (1), (2), and (4), plus the rotation-count discrepancy around Algorithm 1; unavailable supplementary details and unreadable curve values are not treated as verified facts in this note.

Related Work & Insights

• vs AC3D: AC3D primarily controls cameras in first-frame-relative coordinates; this work assigns gravity-aligned semantics to Plücker conditions and retrains with matching data, rather than expecting a coordinate change alone to confer the same capability.
• vs UCPE: UCPE already encodes latitude and up vectors, so this paper should not be read as introducing gravity to UCPE for the first time; the advances are richer trajectory generation and null-pitch training pairs.
• vs PreciseCam + WAN / GEN3C: controlling the first frame and extending it can improve some visual metrics, whereas this work conditions the complete text-to-video process on absolute cameras; GEN3C additionally assumes static scenes.
• Research direction: a controlled experiment could vary only the caption source while keeping scenes and trajectories identical to test whether other control tasks also shed semantic shortcuts; this is a reader-proposed direction, not a result established here.

Rating

• Novelty: 4/5. The combination of gravity alignment, panoramic augmentation, and caption disentanglement is clear, with contributions centered on data and conditioning design.
• Experimental Thoroughness: 4/5. Multiple backbones, two encodings, and two targeted benchmarks are covered, although evaluation scale and estimator dependence remain limitations.
• Writing Quality: 4/5. The central conflict and ablations are clear, but the algorithm and prose disagree on rotation counts, and the cached equations have separate extraction damage.
• Value: 4/5. Useful for fine-grained camera control and conflicting conditions, while deployment still requires joint assessment of visual quality and translation control.