DeRA: Decoupled Representation Alignment for Video Tokenization

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Authors: Pengbo Guo, Junke Wang, Zhen Xing, Chengxu Liu, Daoguo Dong, Xueming Qian, Zuxuan Wu
Area: Video Generation
Keywords: video tokenization, appearance-motion decoupling, representation alignment, gradient conflict, autoregressive generation

TL;DR

DeRA separately learns appearance and motion representations in a one-dimensional video tokenizer, supervises them with image and video foundation models, and uses symmetric gradient projection to mitigate objective conflicts in the shared encoder, reducing UCF-101 reconstruction rFVD from LARP's 20 to 15 with the same 1024-token budget while improving downstream autoregressive generation.

Background & Motivation

A discrete video tokenizer compresses pixels into indices from a finite vocabulary, allowing an autoregressive generator to predict those indices sequentially. Grid-based representations preserve local structure, but longer videos and higher resolutions generally require more downstream tokens. One-dimensional tokenizers such as LARP instead aggregate an entire video with learnable queries, decoupling the compressed sequence length from the input grid. Their queries must nevertheless learn to preserve textures, object identity, and motion simultaneously, making training difficult.

DeRA does not focus on shrinking the generator further. It aims to help the tokenizer learn a useful latent space more quickly. A single video encoding path mixes static content with temporal changes, while reconstruction alone provides limited guidance about what each token group should represent. Pretrained features can help, but image models specialize in appearance and video models are better suited to temporal information. Applying both supervision signals to shared parameters can make their updates cancel one another.

The paper therefore changes both representation organization and optimization: separate queries receive appearance and motion responsibilities, each stream aligns with an appropriate foundation model, and conflicting components are removed when the two alignment gradients oppose each other. Core Idea: give different kinds of visual knowledge dedicated representational roles, then coordinate their updates in the shared encoder instead of expecting a mixed latent space to discover every division of labor unaided.

Method

Overall Architecture

The inputs are a video clip and its first frame; the outputs are discrete tokens that can reconstruct the entire clip. Shared Dual-Stream Encoding first extracts appearance and motion representations with separate queries. Decoupled Representation Alignment supervises the corresponding streams with frozen image and video foundation models, while Symmetric Conflict Projection coordinates the two supervisory updates during training. The compressed query outputs are concatenated, quantized, and decoded; at generation time, an autoregressive model predicts the same kind of discrete sequence.

Two paths should be distinguished. The reconstruction path turns compressed queries back into video. The auxiliary training path aligns the encoder's image or video patch representations, influencing the compressed queries through the shared encoder. Although the overview and schematic use latent alignment as a broad description, the method text explicitly uses patch outputs rather than requiring a direct one-to-one match between a small set of compressed queries and teacher tokens.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video and first frame"] --> B["Shared Dual-Stream Encoding"]
    B --> C["Decoupled Representation Alignment<br/>Frozen image and video models"]
    C --> D["Symmetric Conflict Projection<br/>Coordinate alignment gradients"]
    D -.->|Training update| B
    B --> E["Concatenate and quantize queries"]
    E --> F["Decode reconstructed video"]

Key Designs

1. Shared Dual-Stream Encoding: separate compression inputs for appearance and motion

The appearance stream reads only the first frame, whereas the motion stream reads the entire video. This does not involve optical-flow estimation or explicit background subtraction: the motion stream still sees appearance information. Its input coverage, query set, and subsequent supervision jointly encourage attention to dynamics. The first frame is divided into spatial patches and the video into spatiotemporal blocks. Each set passes through a linear projection and is concatenated with its own learnable queries.

Both sequences pass separately through the same Transformer encoder, so decoupling does not mean training two independent backbones. Each pass produces features at query positions and patch positions. The former provide the compact representation to retain; the latter also receive auxiliary supervision. The default configuration uses 256 appearance queries and 768 motion queries, totaling 1024. Allocating more positions to motion divides representational capacity between changes across a clip and single-frame content rather than replicating appearance for every frame.

The two sets of query outputs are then concatenated and vector-quantized with a codebook of size 8192 and a quantized latent dimension of 16. The decoder processes these tokens together with its own learnable queries, then reshapes the outputs at decoder-query positions into video. This preserves a conventional encoder–quantizer–decoder interface without requiring the downstream generator to understand either teacher's internal structure. Sharing the encoder saves parameters but does not eliminate the two forward passes; the paper does not claim that dual-stream encoding itself is free.

2. Decoupled Representation Alignment: place each expert's supervision on the corresponding stream

Two query sets alone do not guarantee a stable division of labor. DeRA aligns encoded appearance-patch features with frozen DINOv3 ViT-B/16 image features and encoded motion-patch features with frozen InternVideo2-B/14 video features. A lightweight two-layer MLP projects student features into the appropriate teacher space, and the loss averages negative cosine similarity over corresponding tokens. This primarily constrains feature direction rather than forcing raw channel values from different models to match exactly.

The supervision complements reconstruction: reconstruction preserves information needed to recover pixels, appearance alignment supplies mature spatial semantics, and motion alignment supplies temporal cues from video representations. Supervision flows from patch outputs back into the shared encoder, benefiting compressed queries within the same representational system. It does not add a sequence of teacher tokens for the generator or require teacher inference at deployment. Both foundation models remain frozen and are removed with the auxiliary alignment modules after training, leaving the tokenizer's original inference path.

The method should not be interpreted as enforcing strict statistical independence. It does not directly optimize mutual information between appearance and motion, and the video teacher is not a pure motion sensor. Reconstruction, generation, and token-swapping results support a useful semantic division, but do not guarantee that either token group excludes information about the other. The cached method text also does not detail correspondence handling when student and teacher patch grids differ, which requires verification for reproduction.

3. Symmetric Conflict Projection: keep alignment objectives from pushing shared parameters in opposing directions

Shared parameters enable knowledge transfer between streams but also expose gradient conflicts directly. SACP, short for Symmetric Alignment-Conflict Projection, separately computes the gradients of appearance and motion alignment losses with respect to encoder parameters. If their inner product is nonnegative, it leaves the objectives unchanged. If it is negative, it removes each gradient's conflicting projection along the other. Unlike correcting only one objective, symmetric treatment does not designate appearance or motion as the side that must always yield.

The following standard notation restates Algorithm 1. Some typeset equations are corrupted in the cache, so this expression follows the algorithmic steps rather than introducing an additional objective. Let \(g_a\) and \(g_m\) denote the two alignment gradients and \(s=\langle g_a,g_m\rangle\). Under conflict:

\[ g'_a=g_a-\frac{s}{\|g_m\|_2^2+\varepsilon}g_m,\qquad g'_m=g_m-\frac{s}{\|g_a\|_2^2+\varepsilon}g_a,\qquad s<0. \]

The implementation need not turn projection into a new higher-order optimization problem. It computes projection coefficients, treats them as constants using stop-gradient, and subtracts a coefficient times the other alignment loss from each loss. Back-propagation through these reformulated losses produces the corrected directions above. The stability constant is \(\varepsilon=10^{-8}\) to prevent a zero denominator. This addresses conflicts between the two auxiliary alignment gradients, not every conflict involving reconstruction, perceptual, or adversarial objectives, and it does not provide a global convergence guarantee.

Loss & Training

The tokenizer's original objective combines pixelwise \(\ell_1\) reconstruction, LPIPS perceptual loss, GAN loss, and VQ loss, supplemented by the two SACP-reformulated alignment objectives. Appearance alignment has weight 1.0 and motion alignment has weight 0.5. These weights still require selection: the claim of no additional tuning applies specifically to SACP avoiding a separate conflict-penalty weight of the kind used by Soft Loss.

Training and evaluation use 16-frame, \(128\times128\) videos, temporal block size 4, spatial patch size 8, and encoder hidden dimension 768. Main results use a 632M-parameter LLaMA-style autoregressive generator. UCF-101 class-conditional generation prepends a class prompt, while K600 frame prediction uses a separator token between context and targets. The generator learns next-token prediction through cross-entropy, samples with CFG scale 1.2, and sends predicted indices to the tokenizer decoder.

Ablations train the tokenizer for only 75 epochs and use a 343M-parameter generator, so their generation FVD values should not be subtracted directly from the main-table results. For K600, the text specifies conditioning on the first 5 frames and predicting the following 11. Figure 7 displays fewer frames; the displayed counts in its caption should not be treated as a separate evaluation protocol.

Key Experimental Results

Main Results

Reconstruction rFVD measures Fréchet Video Distance between reconstructed and real videos; generation gFVD measures the corresponding distributional distance between generated and real videos. Lower is better for both, and neither is a percentage accuracy. The following selection from Table 1 reports UCF-101 reconstruction, UCF-101 class-conditional generation, and K600 frame prediction.

Method	Tokenizer parameters	Generator parameters	Tokens	UCF-101 rFVD ↓	UCF-101 gFVD ↓	K600 gFVD ↓
OmniTokenizer	82.2M	650M	1280	42	191	32.9
LARP	173M	632M	1024	20	57	5.1
DeRA	174M	632M	1024	15	50	4.1
MAGVIT-v2-MLM	Not reported	307M	1280	8.6	58	4.3

Relative to LARP, DeRA reduces reconstruction FVD by 5 units, or 25%, not by 25 percentage points. Generation FVD decreases by 7 on UCF-101 and 1.0 on K600. MAGVIT-v2-MLM still has lower reconstruction rFVD, so outperforming LARP should not be generalized to the best reconstruction among all tokenizers. MLM and AR also differ in generation procedure, token count, and parameter scale.

Ablation Study

The following selection from Tables 3 and 4 retains each table's own baseline. These experiments should not be combined into a single sequence of incremental module additions. Both use the ablation configuration of 75 epochs and a 343M-parameter generator.

Experiment group	Config	rFVD ↓	gFVD ↓
Table 3: alignment source	Base	24.47	112
Table 3: alignment source	+ DINOv3	20.14	98
Table 3: alignment source	+ InternVideo2	22.64	107
Table 3: alignment source	+ DINOv3 + InternVideo2	18.83	94
Table 4: conflict handling	No conflict regularization	19.81	97
Table 4: conflict handling	Soft Loss, weight 0.5	19.42	96
Table 4: conflict handling	Soft Loss, weight 1.0	20.12	103
Table 4: conflict handling	SACP	18.83	94

Image alignment alone provides a larger improvement than video alignment alone, and combining them improves results further. Against its own baseline without conflict regularization, SACP reduces rFVD by 0.98 and gFVD by 3. Increasing the Soft Loss weight from 0.5 to 1.0 instead worsens performance, showing that merely adding a conflict penalty is not automatically effective.

Key Findings

• Training acceleration should be understood as time to reach the same quality. In Table 8, LARP reaches rFVD 20 after 150 epochs and 145.6 hours; DeRA reaches it after 73 epochs and 78.2 hours, approximately a 1.86-fold wall-clock speedup. At 150 epochs each, DeRA takes 158.6 hours rather than being faster, but produces better quality. Measurements use 8 H100 GPUs.
• Encoder sharing is a trade-off with a measurable cost. In Table 7, separate encoders use 343M parameters and 0.88 seconds with rFVD 18.61; the shared version uses 174M and 0.54 seconds with rFVD 18.83, saving resources at a slight reconstruction cost.
• Deeper alignment is better overall, but improvement is not monotonic at every layer. In Table 6, DINOv3 rFVD worsens from 21.34 at layer 6 to 21.68 at layer 8, then reaches 20.14 at layer 12. The paper's statement of consistent improvement needs to be narrowed to match the table.

Highlights & Insights

• The method considers representational specialization and optimization coordination together. Simply adding two supervision streams without checking their update directions in shared parameters may not stabilize joint training, even with stronger teachers.
• Alignment positions need not coincide with the final discrete tokens. Patch outputs can receive teacher knowledge while fixed-length queries remain the generation interface, avoiding a dependence of autoregressive sequence length on the teacher feature grid.
• Token swapping offers more intuitive evidence of decoupling than reconstruction metrics alone. It remains a qualitative demonstration of controllable specialization, not proof that arbitrary videos support lossless appearance-motion swaps.

Limitations & Future Work

• Evaluation centers on low-resolution short clips from UCF-101 and K600, which does not establish performance on open-domain, high-resolution long videos. Visualized temporal consistency cannot replace longer-horizon testing.
• Video supervision remains the relatively weaker component. The authors identify a dedicated high-fidelity motion representation model as future work; the present experiments only show a larger benefit from the selected image teacher, not that image models universally outperform video models.
• Reproduction details have gaps: some cached equations are corrupted and patch-correspondence handling is not elaborated. The no-alignment rFVD is 24.48 in Table 2 but 24.47 in Table 3, and the Base configurations in Tables 3 and 4 also differ. These values should not be silently merged into one baseline.
• Removing teachers at inference does not make training free. Frozen models still require forward passes, and SACP requires both alignment gradients. Equal-epoch wall-clock time should be reported separately from convergence speed.

Related Work & Insights

• vs LARP: Both turn videos into compact one-dimensional discrete sequences. LARP emphasizes a learned autoregressive prior, whereas DeRA adds targeted appearance-motion supervision and conflict handling rather than redefining next-token generation.
• vs REPA / VideoREPA: These methods use foundation-model features to guide generative-model training. DeRA applies alignment to the video tokenizer and selects different teachers for the two representations, improving the representation interface on which the generator depends rather than only its internal denoising features.
• vs MAGVIT-v2: A tokenizer with stronger reconstruction does not necessarily produce lower generation FVD in every configuration. Table 1 motivates checking reconstruction, generator type, and sequence budget separately rather than ranking systems by one metric column.

Rating

• Novelty: 4/5. Dual-stream tokenization, heterogeneous alignment, and symmetric gradient coordination form a coherent design, although the component ideas have precedents.
• Experimental Thoroughness: 4/5. Generation, reconstruction, teacher selection, and training costs are examined, but scale and dataset coverage remain limited.
• Writing Quality: 3/5. The motivation is clear, but cross-table baselines and qualitative claims need greater precision, while corrupted cached equations also impede reading.
• Value: 4/5. Directly relevant to researchers improving compact video tokens and the training efficiency of autoregressive generation.