TriMotion: Modality-Agnostic Camera Control for Video Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Full Text: Conference PDF
Project: TriMotion
Authors: Seunghyun Shin, Jifei Song, Wooseok Jeon, Hae-Gon Jeon, Jiankang Deng
Area: Video Generation
Keywords: camera control, unified motion embedding, cross-modal alignment, latent consistency, motion composition

Identity check: manifest ID 3398, the Chinese stub, and the cached paper title agree, with no subtitle change. This note uses Sections 3-5 and Tables 1-4 of the conference PDF; no verified arXiv ID or code repository is supplied in the cache, so neither is added.

TL;DR¶

TriMotion learns a shared camera-motion representation from geometry-grounded video-pose-text triplets and constrains generation with a frozen latent motion predictor, reaching 221.59 FVD and 3.6797 CamMC in pose-conditioned V2V while supporting cross-modal motion composition.

Background & Motivation¶

Camera-controlled video generation must preserve source content while executing a new viewpoint trajectory, but users express motion intent in different ways. CamI2V and ReCamMaster accept explicit poses, which offer precision but require geometric trajectories; MotionClone and CamCloneMaster transfer motion from reference videos, which are harder to edit into precise temporal stages. Text is easier to provide, yet lacks native frame-wise structure and rarely specifies complete translation scales or angular velocities.

The challenge is not merely adding three input interfaces: a sentence describing a forward dolly with a left pan, a matching video, and its camera extrinsics should produce the same control signal. Global semantic alignment alone can miss ordering, while cosine similarity alone can weaken absolute geometric information; even a correct conditioning embedding does not ensure that diffusion follows the requested trajectory. The framework therefore needs both a reliable shared motion space and a mechanism to evaluate whether the generated result follows its target.

Core Idea: derive synchronized language from geometry, jointly align global intent, temporal tokens, and recoverable poses, then predict motion directly from generated latents to supervise the output without repeated video decoding.

Method¶

Overall Architecture¶

Inputs comprise a source image or video providing content and a video, pose sequence, or text reference specifying camera motion; the output depicts that content under the requested trajectory. Training first constructs motion triplets and learns unified motion embeddings, then trains a latent motion predictor, and finally fine-tunes a Wan-family video diffusion Transformer with target-motion conditioning and a motion consistency loss. Source content and target motion play distinct roles: a motion-reference video need not depict the same scene as the source video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    data["Synchronized videos<br/>and extrinsics"] --> triplet["Geometry-grounded triplets"]
    triplet --> space["Unified motion embedding"]
    space --> condition["Target-motion conditioning"]
    source["Source image or video"] --> condition
    condition --> latent["Generated latents"]
    latent --> consistency["Latent motion consistency"]
    space --> consistency
    consistency -.->|Training constraint| condition
    latent --> output["VAE-decoded output video"]

The consistency branch supervises training; it is not post-processing that decodes a video and estimates its camera at every sampling step. Section 4.1 uses a 3D VAE with temporal and spatial downsampling factors of 4 and 8; clean source latents and noisy target latents are concatenated along time. A frozen T5 encoder processes the appearance description, a frozen CLIP image encoder processes the first source frame, and a separate pathway injects target motion.

Key Designs¶

1. Geometry-grounded triplets: extract motion phases before generating language

The Motion Triplet Dataset extends the Multi-Cam Video Dataset, whose underlying collection contains 136K Unreal Engine 5 videos, 13.6K dynamic scenes, 40 environments, and 122K unique camera trajectories. Passing continuous extrinsic matrices directly to a language model can obscure temporal relationships, so the pipeline first anchors trajectories at the first frame and computes frame-wise translation and rotation changes. Changes below fixed thresholds are treated as stationary; larger changes are mapped to canonical operations such as Dolly, Pan, and Tilt, then organized into ordered motion phases.

Qwen3-4B-Instruct converts these symbolic sequences into a short summary and a detailed paragraph covering temporal transitions and speed changes. The prompt distinguishes simultaneous from sequential operations and avoids explicit metric values, bringing the descriptions closer to ordinary user instructions. The result is geometrically grounded language supervision rather than manually authored text or unconstrained scene captions; the main paper does not specify the numerical stationarity thresholds.

2. Unified motion embedding: preserve global intent, temporal positions, and geometry

The video branch adopts VGGT feature aggregation, appending camera tokens to frame patches and collecting multi-view geometry through alternating frame-wise and global attention. The text branch encodes a sentence with frozen T5, then uses a fixed number of learnable motion queries that cross-attend to text tokens to produce an ordered motion sequence. The pose branch flattens each \(3\times4\) extrinsic matrix and applies a frame-wise MLP with GELU, mapping poses into the shared dimension while retaining trajectory structure.

Each branch produces \(N\) motion tokens; a learnable global token is prepended before a modality-specific lightweight temporal Transformer produces an \((N+1)\times D\) representation. The first token describes the whole trajectory, while the remaining tokens encode temporal dynamics. Symmetric InfoNCE aligns global tokens for every modality pair, and cosine distance aligns corresponding temporal tokens. A shared pose regressor also predicts extrinsics from each modality embedding with L1 supervision, preventing the space from merely clustering similar motions while losing recoverable geometry.

Written in normalized notation, Equation (9) combines these constraints as:

\[ \mathcal{L}_{\mathrm{align}}=\mathcal{L}_{\mathrm{NCE}}+\lambda_t\mathcal{L}_{\mathrm{temp}}+\lambda_p\mathcal{L}_{\mathrm{pose}}. \]

Correct global retrieval does not imply exact frame-wise pose recovery; Table 3 later exposes this distinction for text embeddings.

3. Target-motion conditioning: specify the desired trajectory without competing source-motion cues

Each diffusion Transformer block has a separate projection MLP that maps the target-motion embedding to the hidden feature layout and adds it residually. The clean source latent still supplies content, but motion conditioning uses only the target embedding, neither concatenating source motion nor using a target-minus-source difference vector. The desired camera trajectory is an absolute target: source-motion concatenation can introduce competing control cues, while subtraction can remove useful absolute trajectory structure.

I2V uses the same generator by encoding the input image into the first frame of a pseudo-video latent and zero-filling its remaining latent frames. V2V provides the complete source sequence instead. Both share the conditioning and denoising structure, unifying source formats and target-motion interfaces without reducing camera control to ordinary text prompting.

4. Latent motion consistency: supervise generated motion rather than only its conditioning representation

The motion embedding predictor combines 3D convolutions and a temporal Transformer to recover global and temporal motion tokens from clean video latents. It is first aligned with embeddings from the video motion encoder and exposed to mild noise at lower diffusion timesteps, preparing it for imperfect clean-latent estimates; it is then frozen. Equation (10) uses cosine distances at two granularities:

\[ \mathcal{L}_{\mathrm{motion}}(\hat e,e)=1-\mathrm{sim}(\hat e^{(0)},e^{(0)})+\frac{\lambda_h}{N}\sum_{k=1}^{N}\left[1-\mathrm{sim}(\hat e^{(k)},e^{(k)})\right]. \]

Under rectified flow, the noisy target is \(z_t=(1-t)z_0+t\epsilon\) and the velocity target is \(\epsilon-z_0\), giving the clean-latent estimate \(\hat z_0=z_t-t v_\theta\). This is the meaning of Equation (11): the frozen predictor processes that estimate, its embedding is compared with the current target-modality embedding, and gradients propagate through the predictor to the generator. Freezing the predictor does not block those gradients; it keeps the motion evaluation function stable instead of allowing it to change jointly with the generator. Avoiding repeated high-resolution pixel decoding is the computational advantage, but the paper does not report a measured speedup factor.

A Worked Example¶

Suppose the source image depicts an indoor space and the motion instruction is to dolly forward, then gradually pan left; this is an explanatory example, not an additional experiment. T5 and the motion queries produce temporal tokens representing forward movement followed by the left pan, and the unified embedding supplies the target condition at every generator block. The first source latent frame and zero-filled remaining frames form the I2V content input; during training, a predicted clean latent that fails to turn left is penalized through temporal-token consistency. Replacing the sentence with a reference video or extrinsics expressing that trajectory changes the motion encoder input, not the diffusion backbone.

Section 5.5 also explores cross-modal composition: sequential composition concatenates two motion sequences and offsets the second by the first sequence's final state, while interpolation linearly blends two embeddings. Figure 5 and user ratings support these applications, but the implementation of the final-state offset in embedding space is not fully developed in the text, so this is not a general trajectory algebra with guaranteed geometric continuity.

Loss & Training¶

Equation (12) defines \(\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{denoise}}+\lambda_m\mathcal{L}_{\mathrm{motion}}\), where denoising uses mean squared velocity error. Only the backbone's 3D spatial-temporal attention layers and block-specific projection MLPs are updated; I2V and V2V are sampled with equal probability at each iteration. The unified motion space is trained for 100 epochs, the predictor for 10 epochs, and the diffusion backbone for 10K iterations, with 1% of the dataset held out for validation. Training uses 4 NVIDIA H200 GPUs and AdamW with learning rate \(10^{-4}\), \(\beta_1=0.9\), \(\beta_2=0.999\), and weight decay 0.01. The main text does not specify numerical \(N\), \(D\), loss weights, or the exact Wan model size; these should not be replaced with assumed standard configurations.

Key Experimental Results¶

Main Results¶

Section 5.1 uses 500 source videos from Koala-36M and 500 video-pose pairs from RealEstate10K; missing motion descriptions are generated with the same geometry-grounded pipeline. The following V2V results are selected from Table 1. Lower FVD, translation error, and CamMC are better; higher CLIP-V indicates better source-content preservation. Camera errors are computed from MegaSaM-estimated poses, and CamMC follows the CamI2V metric; its complete formula is not restated in the main text and it is not an accuracy percentage.

V2V method	Control modality	FVD	Translation error	CamMC	CLIP-V
TrajectoryCrafter	Pose	299.96	3.2621	3.5711	0.8794
ReCamMaster	Pose	253.65	3.6559	4.3611	0.8833
CamCloneMaster	Video	241.38	3.8987	4.7964	0.8791
TriMotion	Text	237.87	3.7525	4.7778	0.9041
TriMotion	Video	254.55	3.6468	4.2384	0.8851
TriMotion	Pose	221.59	3.2356	3.6797	0.9071

Pose-conditioned TriMotion reduces FVD by 32.06 relative to ReCamMaster, but its CamMC is worse than TrajectoryCrafter's 3.5711, so it does not lead every metric. For I2V, pose-conditioned TriMotion obtains 317.85 FVD and 4.3087 CamMC versus CamCloneMaster's 312.13 and 4.9585, again showing that distributional video quality and motion accuracy need not improve together.

Ablation Study¶

Table 2 uses pose-conditioned V2V throughout. Concatenation refers to source and target motion embeddings, not the source/target video-latent concatenation at the backbone input.

Motion conditioning / loss	FVD	Translation error	CamMC	CLIP-V
Concatenated source and target motion embeddings	193.56	10.5327	11.2641	0.9214
Target-minus-source motion embedding	265.53	9.6904	10.9317	0.8903
Without motion consistency loss	249.85	4.2986	4.8660	0.8828
Full TriMotion	221.59	3.2356	3.6797	0.9071

Removing motion consistency increases CamMC from 3.6797 to 4.8660, a difference of 1.1863; adding it reduces CamMC by approximately 24.38% relative to the no-loss variant. Concatenation actually improves FVD to 193.56 and CLIP-V to 0.9214 while substantially worsening motion errors, demonstrating that preserving the source too strongly is not successful camera redirection.

Key Findings¶

Cross-modal retrieval in Table 3 uses the triplet validation split, with positives sharing a trajectory and negatives using different trajectories within the same scene; Recall@1 is the fraction of queries whose matching counterpart ranks first.

Modality pair	Positive cosine similarity	Negative cosine similarity	Recall@1
Video and pose	0.9605	0.0317	98.50%
Video and text	0.8887	0.0256	98.00%
Text and pose	0.9121	0.0243	99.50%

In the same table's RealEstate10K pose-regression evaluation, text embeddings obtain 3.8359 CamMC versus 0.6603 for pose embeddings: strong retrieval alignment does not imply equally precise geometric recovery from language. Table 4 reports 20 participants evaluating 30 video-text pairs, 10 sequential-composition examples, and 10 interpolation examples; generation-quality evaluation uses 10 examples each for I2V and V2V. On a five-point scale, detailed descriptions score \(4.11\pm0.22\), sequential composition \(3.97\pm0.23\), and interpolation \(3.88\pm0.27\); these small human studies support usability rather than broad generalization claims.

Highlights & Insights¶

Converting continuous extrinsics into temporally ordered camera operations reduces the numerical reasoning burden on the language model. It also makes caption supervision traceable to a geometric source.
Global alignment, temporal matching, and pose regression constrain different properties. Strong retrieval alongside weaker text-based geometry cautions against treating semantic agreement as geometric equivalence.
Motion supervision operates directly on generated latents, avoiding pixel decoding. A reusable idea is a frozen latent evaluator for controllable attributes, provided it remains robust to generation errors.

Limitations & Future Work¶

The authors acknowledge boundary outpainting artifacts under extreme trajectories that expose large unseen regions, suggesting explicit 3D background completion or temporal extrapolation.
Training supervision primarily comes from synthetic UE5 scenes; real-video evaluation is valuable, but broader breakdowns for complex dynamics, extreme viewpoints, and open-ended language are absent.
Generated text omits explicit metric values, and text-based pose regression trails pose inputs. Modality-agnostic therefore means a shared interface and representation, not equal information content across inputs.
Composition is supported mainly by qualitative examples and small human studies rather than cumulative geometric errors over long chains; length, turn-strength, and failure-rate sweeps would strengthen the evidence.
Runtime, memory savings from the latent constraint, and variance across repeated training runs are not reported, preventing quantitative efficiency and statistical-stability conclusions.

vs CamCloneMaster: joint reference/target token attention transfers reference-video motion, whereas TriMotion converts multiple reference types into a shared representation; both use efficient attention fine-tuning ideas.
vs ReCamMaster: TriMotion builds on its multi-camera data but adds geometry-grounded language and cross-modal learning, expanding explicit-trajectory rerendering into three camera-control interfaces.
vs TrajectoryCrafter: explicit point-cloud and geometry guidance gives stronger V2V CamMC, whereas pose-conditioned TriMotion provides better FVD and source preservation, illustrating a trade-off between control mechanisms.
Future direction: introduce scale-aware language conditions and separately evaluate semantic instruction following and absolute pose error, so a shared embedding does not hide information loss.

Rating¶

Novelty: 4/5. The three-modality motion space and latent output constraint form a coherent contribution, although individual components have precedents.
Experimental Thoroughness: 4/5. I2V, V2V, conditioning ablations, retrieval, and human evaluation are covered; efficiency and extreme-motion analysis remain limited.
Writing Quality: 4/5. Architecture and ablations correspond well, but some hyperparameters and motion-composition implementation details are underspecified.
Value: 4/5. The framework lowers the input barrier for interactive camera control, without establishing a lossless language interface to exact 3D trajectories.