FlexAM: Flexible Appearance-Motion Decomposition for Versatile Video Generation Control¶

Conference: ECCV2026
Paper: Official ECCV page
PDF: Full paper
Authors: Mingzhi Sheng, Zekai Gu, Peng Li, Cheng Lin, Hao-Xiang Guo, Ying-Cong Chen, Yuan Liu
Code: https://github.com/IGL-HKUST/FlexAM
Area: Video Generation
Keywords: appearance-motion disentanglement, 3D motion control, multi-frequency positional encoding, dynamic depth, video editing

TL;DR¶

FlexAM specifies appearance with masked video and motion with variable-density 3D point trajectories carrying multi-frequency positional encoding and dynamic depth, allowing one video diffusion model to perform appearance editing, camera control, and object manipulation while achieving text alignment of 32.55 and temporal consistency of 0.976 in the paper's motion-transfer evaluation.

Background & Motivation¶

Controllable video generation often translates user intent into depth maps, skeletons, 2D trajectories, or camera parameters, then trains a model to interpret these conditions. Each signal has useful applications, but they do not naturally share a control interface: changing a character's appearance requires retaining its actions, re-cinematography changes the camera while preserving scene dynamics, and moving an object requires resolving occlusions. Preparing separate modalities and specialized modules for each task turns task expansion into additional training and preprocessing work.

Separating appearance from motion can reduce this fragmentation, but existing representations leave specific gaps. First-frame appearance conditioning cannot directly constrain regions revealed later. Fixed-density motion guidance either overconstrains a new appearance or fails to express fine details. DaS already uses 3D tracking videos, showing that point trajectories can support multiple controls. However, similar attributes on neighboring points can still confuse their motion identities. Initial geometry alone is also insufficient to distinguish subsequent object scaling from movement along the depth axis.

Rather than add another task-specific condition, FlexAM improves the distinguishability of a shared motion representation and expands the temporal coverage of appearance conditioning. Core Idea: use masked video to specify which appearances should remain where, and use 3D trajectories with fixed identities, dynamic depth, and variable density to specify how those regions move, then supply both to one video generation backbone.

Method¶

Overall Architecture¶

The input can be an image or video, together with editable regions and a target control. FlexAM organizes two conditioning branches: masked appearance conditioning retains content that should remain unchanged, while 3D point trajectories pass through precise positional encoding, dynamic depth encoding, and density adaptation to form a motion video suitable for the generator. The two branches jointly condition a diffusion Transformer in latent space to produce the requested appearance and motion.

A motion video is neither an ordinary RGB video nor a rough preview of the output. It is a control tensor obtained by projecting attributed 3D points into images frame by frame. Its 18 channels comprise six three-channel streams: one identity stream, four multi-frequency positional streams, and one dynamic-depth stream. Projected locations indicate where points move, while their attached attributes indicate their identities and current depths. These roles are distinct.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input image or video<br/>editing target"] --> B["Masked appearance conditioning"]
    A --> G["3D point trajectories"]
    G --> C["Precise positional encoding"]
    C --> D["Dynamic depth encoding"]
    D --> E["Density adaptation"]
    B --> F["Video diffusion backbone<br/>controlled video output"]
    E --> F

Key Designs¶

1. Masked appearance conditioning: extending first-frame reference to arbitrary spatiotemporal regions

If all appearance information comes from the first frame, backgrounds revealed later have no direct reference. FlexAM instead uses a partially masked video and a binary edit mask. Unedited regions retain their original pixels, while editable regions are filled with gray value 127 and marked as 1 in the mask. The model can therefore distinguish genuine context from placeholder regions awaiting generation, rather than interpreting gray as the desired appearance.

This design expresses different tasks through the same conditioning format. Image-to-video generation retains the first frame and masks the remaining frames; local video editing masks only the target object or background while preserving available appearance information elsewhere in the sequence. The paper also uses the mask to assign distinct denoising timesteps to edited and preserved regions, separating content to regenerate from context to maintain. Motion signals are zeroed in unedited regions as well, avoiding irrelevant motion constraints there. These are control mechanisms and design goals, not guarantees of perfect pixelwise preservation in every output.

2. Precise positional encoding: distinguishing neighboring trajectory identities with multi-frequency attributes

When points approach, cross, or occlude one another in the image, projected locations alone do not clearly identify their original parts. FlexAM attaches attributes derived from initial 3D coordinates to each trajectory. The identity stream stores normalized initial coordinates, with inverse depth used for the depth component. These identity attributes remain fixed over time even as the point's projected location changes. A point therefore carries the same spatial identity marker across frames.

Continuous coordinates alone may still be too similar for neighboring points to remain distinguishable after encoding. The paper additionally applies cosine encodings at four frequencies to each of the three initial coordinates, organizing the result into four three-channel streams. Higher frequencies can amplify feature differences between nearby coordinates, supplementing the raw identity stream. Retaining multiple frequencies also gives the network access to both coarse position and finer distinctions. The central transformation is:

\[ \gamma_l(v)=\cos(2^l\pi v),\qquad l\in\{0,1,2,3\}. \]

Here, \(v\) is one normalized initial-coordinate component and \(l\) indexes the frequency level. The encoding describes a trajectory's initial attributes, not a newly estimated identity in every frame. The paper illustrates the issue through confusion between neighboring limbs: improved distinguishability is intended to reduce left–right identity swaps, not simply increase the output video's resolution.

3. Dynamic depth encoding: separating current depth ordering from fixed identity

Initial coordinates describe where a point originated, but not directly how far it is from the camera now. The 2D projection is ambiguous too: changing an object's size and moving it along the viewing direction can create similar image changes. With only projected trajectories and fixed colors, the model must infer depth changes itself, which can produce incorrect front–back relationships as objects approach the camera or occlude one another.

FlexAM therefore adds depth attributes that vary over time. Each frame's normalized depth is mapped to three channels using the Spectral colormap, concatenated with the time-invariant identity and multi-frequency attributes, and rendered at the point's projected location. The paper argues that this color mapping provides richer gradients than grayscale and helps distinguish small depth variations. Locations without projected points remain zero, while valid locations inside edited regions carry identity, frequency, and current-depth information together. This explicitly separates which point is being tracked from its present depth, rather than requiring the generator to infer both meanings from a static marker.

4. Density adaptation: teaching one generator to interpret sparse hints and dense constraints

Dense trajectories preserve more of the source video's local motion structure but may also impose the source object's geometry on a target with a different appearance. Sparse trajectories give the generative prior more freedom, yet cannot independently determine every local detail. Instead of using separate models for the two regimes, FlexAM randomly downsamples point clouds during training so that one model encounters motion conditions at different densities. A density embedding explicitly tells the network how dense the current constraints are, rather than treating every blank region in the control signal identically.

This stage also connects the motion representation to the pretrained backbone. The 18-channel motion video is split into six three-channel streams, each processed by the same pretrained VAE. A lightweight CNN Adapter fuses the resulting motion latents. The appearance condition and mask form the corresponding appearance latents, which enter the denoising network alongside the motion condition. Density is injected through an embedding analogous to the diffusion timestep embedding. Sharing the VAE reuses the backbone's latent space instead of training a separate motion encoder for every control task. Density adaptation does not guarantee lossless behavior at arbitrary sparsity; it teaches the model to use changing constraint strength.

A Worked Example¶

Consider replacing foreground appearance while preserving a source video's motion. DELTA first extracts 3D points and trajectories from the source, and a repainted reference first frame is prepared. The target foreground is masked with value 127 while other regions retain their original appearance. The reference frame is prepended to the conditioning sequence, and the edit mask identifies which regions may change.

Each source trajectory then carries its fixed identity, four multi-frequency attribute streams, and per-frame depth to form an 18-channel motion video at the selected point density. The generator reconstructs the masked region from the reference appearance while continuing the source motion according to the trajectories. No additional human skeleton is required, and the method does not simply warp source RGB pixels along optical flow. New appearance and original motion can therefore be specified separately.

Other tasks mainly change where the trajectories come from and how they are projected. Image-to-video camera control estimates input-image depth and reprojects the point cloud along the target camera trajectory. Video-to-video re-cinematography additionally estimates the source camera and builds a dynamic point cloud. Spatial object editing uses SAM2 and MoGe to obtain object segmentation and geometry, then translates, rotates, or scales the object's point cloud in 3D before reprojection. All use the same downstream motion-video interface.

Loss & Training¶

The paper builds on a pretrained latent video diffusion model such as Wan2.2Fun 5B Control. Explicitly described training ingredients include random point-cloud downsampling, density embeddings, and fusion of conditioning latents. The main text does not specify the full training loss, dataset size, learning rate, or optimizer settings, referring instead to supplementary material. That supplement is not included in the available full-text cache, so these settings are not invented here, nor is a conventional diffusion loss presented as a formula explicitly reported by this paper.

Key Experimental Results¶

Main Results¶

The motion-transfer comparison uses reference images generated by Qwen Image Edit throughout, but control inputs differ: VACE and Wan2.2Fun use depth maps, DaS uses 3D tracking videos, and FlexAM uses its motion video. The following table selects key metrics from the paper's Table 1. Tex-Ali measures CLIP-based text–video alignment, Tem-Con measures CLIP-based temporal consistency, i2v_sub is VBench++ subject consistency, and aes_ql and img_ql measure aesthetic and imaging quality. Higher is better for all listed metrics; their different scales should not be compared directly.

Method	Tex-Ali ↑	Tem-Con ↑	i2v_sub ↑	aes_ql ↑	img_ql ↑
DaS	32.14	0.968	0.98	0.60	0.68
Wan2.2 Fun	32.39	0.971	0.98	0.61	0.68
VACE	32.38	0.970	0.86	0.57	0.63
FlexAM	32.55	0.976	0.99	0.64	0.68

Relative to Wan2.2 Fun, FlexAM increases Tex-Ali by 0.16, Tem-Con by 0.005, and aes_ql by 0.03. These are absolute differences in the original score scales, not relative percentages. Its imaging quality of 0.68 ties DaS and Wan2.2 Fun, so the results do not establish strict superiority on every metric.

The randomized A/B user study of partial editing includes 27 participants and 8 video pairs. Overall-quality preferences are 59% for FlexAM, 35% for VACE, and 6% ties; preferences for preserving unedited regions are 51%, 28%, and 21%, respectively. The paper reports 91 seconds for FlexAM 5B on one GPU versus 348 seconds for VACE 14B on two GPUs. Different model sizes and GPU counts prevent interpreting this as a pure algorithmic speedup under identical compute configurations.

Ablation Study¶

Section 4.4 primarily compares the complete method with a vanilla 3D tracking representation through Figures 8–10. The cache preserves the experimental setup and qualitative findings, but not reliably readable numerical values from the sparse-reconstruction plots or a complete matrix of individual module removals. The following is therefore an ablation/analysis evidence table, without invented PSNR, SSIM, FVD, or per-module gains.

Analysis target	Comparison setting	Observation	Evidence boundary
Density adaptation	Video reconstruction from the first frame and sparse motion signals, compared with DaS	FlexAM has higher PSNR and SSIM, lower FVD, and less motion drift	Figure 8 and the text establish the direction; readable values are absent from the cache
Precise positional encoding	Vanilla 3D tracking representation versus the full representation	Less confusion between neighboring limb identities and more stable dynamic details	Qualitative evidence in Figure 9, not an isolated module-gain measurement
Dynamic depth encoding	Representation without time-varying depth versus the full representation	Better distinction between scaling and depth motion, with fewer implausible occlusions	Qualitative evidence in Figure 10; no explicit error difference is reported

Key Findings¶

The main quantitative evidence supports better appearance consistency, temporal consistency, and aesthetics, but imaging quality alone does not show a universal advantage. A unified control interface should be evaluated for both controllability and visual quality rather than only its strongest metric.
Camera control is evaluated on 100 randomly sampled RealEstate10K sequences. Pi3 estimates camera poses relative to the first frame from generated videos, and estimated rotations and translations are compared with the targets. FlexAM has the lowest rotation error, but Wan2.2 Fun Control Camera has slightly lower translation error. The cached formula and figure values are incompletely extracted, so exact equations and missing values are not reconstructed here.
For spatial object editing, GeoDiffuser is an image method and is compared with the final frames of FlexAM and DaS videos, not their full-video quality. The CLIP values and method labels in Figure 7 are merged in the cache; only the paper's qualitative conclusion of better semantic alignment for FlexAM is retained here.

Highlights & Insights¶

Fixed identity and dynamic depth jointly represent trajectory correspondence and current spatial relationships. Compared with forcing all information into one depth map, this explicitly distinguishes attributes that should remain constant across frames from those that should change with motion.
The appearance mask is more than a switch for local editing: it expresses first-frame-conditioned generation and within-video editing in one input format. It addresses the coverage of appearance conditions, rather than merely increasing the number of reference images.
Multitask capability mainly comes from an editable 3D control representation, not dedicated output heads for each task. The shared interface does not eliminate geometric preprocessing, but funnels its results into one motion-video format.

Limitations & Future Work¶

The authors acknowledge that control accuracy depends on estimated 3D trajectories. Fast motion, heavy occlusion, and tracking errors propagate into generation. Explicit geometric conditioning does not automatically guarantee correct geometry.
Moderate geometric mismatch between source motion and target appearance is tolerated, but severe mismatch can still cause implausible deformation. Limited lighting diversity in the training data can also cause illumination mismatches. The authors identify richer lighting data and larger video datasets as improvement directions.
The available evidence does not fully isolate the contribution of each encoding component, and the user study is small. Stronger conclusions require verifiable numerical ablations, efficiency comparisons with controlled compute budgets, and the training details in the supplement. These are qualifications of the existing evidence, not a new research proposal.

vs DaS: Both methods control generation with 3D point-tracking videos. FlexAM adds multi-frequency identity features, time-varying depth, and density adaptation, while extending appearance conditioning to masked videos. It is a targeted extension of an existing 3D control approach, not the first appearance–motion decomposition method.
vs VACE: VACE supports multiple tasks but requires task-dependent inputs such as depth or skeletons. FlexAM funnels control through a 3D motion video. It still requires reliable trajectories, depth, and segmentation where needed, rather than eliminating preprocessing.
vs specialized camera control and GeoDiffuser: FlexAM covers multiple video-control tasks within one framework, but specialized camera models can still lead in translation accuracy. Final-frame comparisons with image-editing baselines also do not establish comprehensive superiority over entire videos.

Rating¶

Novelty: 4/5. Targeted representation improvements build on 3D tracking control, with clear motivation for combining appearance conditions and motion attributes.
Experimental Thoroughness: 3/5. Multiple tasks and a user study are included, but the available main-text evidence lacks isolated numerical ablations, controlled efficiency comparisons, and training details.
Writing Quality: 4/5. Problems map clearly to representation choices; broad superiority claims require qualification by tied metrics and the camera-translation exception.
Value: 4/5. A useful interface for unified video generation control, with practical performance still constrained by geometry estimation and appearance compatibility.