Video Generation Models are General-Purpose Vision Learners¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation / Video Understanding
Keywords: generative pretraining, single-pass perception, unified task representation, synthetic data, multi-task learning

TL;DR¶

GenCeption converts WAN text-to-video generation into text-controlled, single-pass perception through RGB-formatted targets, sparse query tokens, and unified L2 post-training, approaching or exceeding specialists on several benchmarks while still encountering substantial interference between sparse keypoint and dense tasks during joint training.

Background & Motivation¶

Foundation models such as Depth Anything and SAM are strong within their respective tasks, but a different instruction alone does not make them solve depth, normals, segmentation, and camera motion together. Attaching multiple task heads to a shared backbone reuses features, yet each additional task still requires maintaining output structures, loss functions, and weights. This paper asks an earlier question: which pretraining objective can accumulate sufficiently general visual knowledge that post-training primarily becomes a matter of defining task data rather than repeatedly redesigning networks?

Text-to-video generation is an appealing candidate. Producing continuous video requires handling appearance, occlusion, motion, and geometric changes, while text conditioning connects vision with semantics; these models have also already undergone large-scale training. However, generating plausible images does not imply precise scene measurement. Diffusion models originally sample iteratively from noise, whereas depth or segmentation benefits from stable, direct prediction. Their targets are also not natural RGB videos, so replacing the decoder or adding many modules may damage pretrained representations.

The paper therefore retains as much of the original video generation network as possible and shifts adaptation into the organization of inputs, outputs, and supervision. Core Idea: use video generative pretraining for spatiotemporal priors, replace iterative sampling with a single forward pass, and encode perception targets in continuous visual representations familiar to the pretrained model so that one backbone can handle different tasks through text instructions.

Method¶

Overall Architecture¶

The inputs are an RGB video and text specifying a task, such as depth estimation or a referring expression identifying an object. WAN's VAE encodes the video, the original text encoder processes the instruction, and both enter the pretrained diffusion Transformer (DiT). The adapted DiT executes one forward pass instead of repeatedly generating from random noise. Dense tasks use a shared VAE decoder to produce three-channel target videos, while sparse tasks read structured quantities such as per-frame coordinates from additional query tokens through an MLP.

The scope of unification matters: dense tasks share the backbone and decoder, whereas sparse tasks require extra tokens and an MLP; not every output uses the same head. Text selects the requested modality or target, and a single forward pass should not be interpreted as producing all tasks simultaneously. Solid arrows below indicate prediction data flow, while dashed arrows indicate training supervision. Target encoding and L2 are not additional generation steps at test time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB video and task text"] --> B["Single-Pass Perception Adaptation<br/>VAE and text encodings enter DiT"]
    B --> C["RGB Formatting of Dense Targets<br/>Shared VAE decoder"]
    B --> D["Sparse Query Extension<br/>Additional tokens and MLP"]
    C --> E["Depth, normals, segmentation, and raymaps"]
    D --> F["Keypoints and camera poses"]
    G["Training targets and unified L2"] -.->|Target-latent supervision| C
    G -.->|Output-space supervision| D

Key Designs¶

1. Single-Pass Perception Adaptation: let the generative backbone directly read noise-free video

The original diffusion process receives noisy latents, with a timestep indicating their noise level. GenCeption instead directly supplies clean RGB video latents and fixes the timestep to the one used at the end of the original sampling process, indicating noise-free inputs. This is more direct than treating a generated video as an answer: the backbone no longer gradually constructs a scene from random variables, but transforms an existing scene encoding into a task-target encoding. Post-training thus becomes a deterministic supervised mapping.

WAN predicts rectified-flow velocity, using the paper's convention \(v=\epsilon-x_0\). The output is therefore negated before entering the loss or decoder, giving \(-v=x_0-\epsilon\). The authors observe that this initialization is closer to RGB video latent encodings and helps convergence. Negation does not already yield the exact target depth, nor does post-training continue optimizing the original denoising objective. Task competence still comes from labeled-target supervision; negation simply respects the output semantics of the pretrained parameters.

2. RGB Formatting of Dense Targets: replace task-specific decoders with representation design

All dense outputs are organized into a three-channel space within \([0,1]\). Scalar outputs such as depth and segmentation are replicated across channels, whereas three-dimensional quantities such as normals and DensePose occupy distinct channels. Target videos are also VAE-encoded, allowing training to compare predicted and target latents directly without an independent decoder for each modality. The point is not to colorize results for visualization, but to present tensors compatible with generative pretraining while reusing the same video encoding and decoding mechanisms.

Camera information demonstrates the flexibility of this idea. A raymap includes both a three-dimensional ray origin and a three-dimensional direction, requiring six channels if stored directly. The proposed "Rothko" raymap puts origins in the central image region and directions around its boundaries, using spatial layout to hold six-dimensional information without increasing the channel count. This preserves the three-channel decoding interface, but also means that the output format itself embeds task knowledge. A task-agnostic network does not eliminate task-specific data processing.

3. Sparse Query Extension: read explicit structured outputs through per-frame tokens

Pixel representations suit depth maps but are not necessarily convenient for directly producing coordinates needed by robotics or motion analysis. The authors therefore append one learnable token per video frame to the video latents. After processing by the same DiT, an MLP maps each token to a per-frame \(K\)-dimensional target. Here, frames refer to the original video rather than VAE-compressed latent frames. Since the original frame count \(T\) exceeds the latent sequence length \(T'\), positional correspondence between these temporal resolutions must be handled.

The extension preserves the DiT's native 3D RoPE. Query-token spatial positions are unknown and are therefore learned; temporal positions are known but interpolated into the temporal range seen during pretraining. The paper states that this additional-query design outperforms adding attention layers, but also reports that joint training severely harms keypoint prediction and other dense tasks. The authors hypothesize that new tokens disturb existing attention interactions while coordinate regression departs from pixel-space pretraining. The extension therefore establishes a usable interface, not full compatibility between sparse and dense tasks.

A Worked Example¶

Consider an 81-frame human video for which depth is requested. The input is encoded at 480x832 resolution and text specifies the depth task. DiT reads clean video latents, executes one forward pass at the fixed terminal timestep, and its negated output is VAE-decoded into a three-channel depth representation. The three channels represent the same scalar, not three different kinds of depth. Training compares target-depth encodings with predicted latents; test-time prediction neither receives ground-truth depth nor performs iterative denoising.

For three-dimensional keypoints, the video still passes through the same backbone, but the additional query tokens corresponding to the 81 frames are read through an MLP to produce coordinates. This explains the distinction between switching tasks through instructions and sharing one output head across every modality. When both depth and keypoints are needed for an input, the single-pass description does not establish that one call suffices. Likewise, a specialist keypoint model's score must not be presented as a result already achieved by the unified multi-task model.

Loss & Training¶

Training uses only standard L2: dense tasks compare latent-space values, while sparse tasks compare output-space values. The difficulty moves into making target distributions appropriate. For example, monocular depth has scale ambiguity. The authors normalize each frame's depth map using the scene's median depth, then map the normalized depth \(d\) into the RGB range:

\[ d' = \operatorname{clip}(\alpha\log(d+1),0,1). \]

The parameter \(\alpha\) adjusts the emphasis on near-field details versus far-field structure. Median-depth normalization applies to training targets and does not require reading ground truth at inference time. This puts scale handling into data representation rather than a separate scale-invariant loss. Task balance is controlled through data mixture ratios, which still need to be chosen. A unified loss removes multiple objective functions, not every multi-task training hyperparameter.

The authors generate 7,500 synthetic videos using 800 RenderPeople human assets and 200 CMU motions, varying backgrounds, focal lengths, camera positions, and trajectories. Blender separately renders depth, normals, and segmentation, while rigged skeletons provide keypoint ground truth. Input and target video latents and text embeddings are cached in advance. Multimodal annotations arise from the same scene, naturally aligning geometry and appearance over time and avoiding the cost of separately collecting each task's labels in real videos.

The full training mixture also includes depth data from TartanAir, Virtual KITTI, and MVS Synth, plus TartanAir camera trajectories. Referring-expression segmentation is an exception that uses real MeViS, Ref-COCO, and YouTube-VOS data, so purely synthetic post-training is not a system-wide claim covering every task. Inputs contain 81 frames at 24 FPS; the VAE downsamples time by 4 and space by 8. Training uses Adam with batch size 64 on 256 v6e TPUs for 15,000 steps, a learning rate of 5e-5, and linear warmup over the first 250 steps. Gradient clipping and dropping batches with exceptionally large gradients stabilize optimization.

Key Experimental Results¶

Main Results¶

The following excerpts from the paper's Table 1 distinguish task-specific variants of the framework (Specialist) from joint training (Generalist). Normal mAE is mean angular error, depth AbsRel is absolute relative error, and foreground MSE is mean squared error; lower is better for all three. Referring-expression J&F averages region IoU and contour accuracy, with higher values preferred.

Task / Benchmark	Metric	Specialist Comparator	GenCeption Specialist	GenCeption Generalist
Normals / Sintel	mAE, degrees	Lotus-2: 30.3	29.7	29.3
Normals / Hi4D	mAE, degrees	Sapiens-2B: 12.14	10.99	11.47
Depth / Sintel	AbsRel	DepthAnything 3: 0.201	0.130	0.156
Depth / ETH3D	AbsRel	VGGT-Ω: 0.016	0.037	0.044
Foreground / VideoMatte	MSE	RVM: 0.0010	0.0027	0.0010
Referring expression / MeViS	J&F	ReferEverything: 60.3	70.0	69.0

Hi4D uses camera 4 from pairs 28, 32, and 37, totaling 1,195 frames. VideoMatte uses RVM's static composite split, and referring-expression evaluation follows VoCap's protocol. Section 4.3 states that training sets accompanying evaluation benchmarks were not used, but Section 4.1 lists MeViS training data while Table 1 evaluates on MeViS. These statements conflict. The cache does not clarify the specific MeViS split, so this result should not be described as verified cross-dataset zero-shot performance.

Ablation Study¶

The paper's Table 2 compares pretrained backbones on identical depth post-training data, then increases dataset coverage and model size. Averages cover Sintel, KITTI, and ETH3D. The δ1 metric is the percentage of pixels for which the larger of the prediction-to-ground-truth depth ratio and its reciprocal is below 1.25; higher is better.

Backbone	Parameters	Post-Training Videos / Frames	Average AbsRel	Average δ1 (%)
V-JEPA-H	0.6B	7.5K / 0.9M	0.281	52.2
VideoMAE V2-G	1B	7.5K / 0.9M	0.154	66.9
WAN 2.1-S	1.3B	7.5K / 0.9M	0.122	85.8
WAN 2.1-L	14B	7.5K / 0.9M	0.093	90.7
WAN 2.1-S, 4 datasets	1.3B	8.08K / 1.23M	0.094	90.6
WAN 2.1-L, 4 datasets	14B	8.08K / 1.23M	0.071	93.8

With the same 7.5K videos, WAN-S reduces average AbsRel from VideoMAE V2-G's 0.154 to 0.122. However, parameter counts are not identical, and pretraining data and compute are not controlled. Expanding to 4 datasets changes both quantity and distribution, so improvement cannot be attributed entirely to sample count. These resource figures describe post-training only, excluding WAN's original large-scale video pretraining cost.

Key Findings¶

Multi-task gains are not universal: VideoMatte MSE improves from 0.0027 to 0.0010, but Sintel depth AbsRel worsens from 0.130 to 0.156. Joint training creates genuine task trade-offs.
Both specialist and generalist variants underperform VGGT-Ω on ETH3D. Leading on selected tasks does not establish universal superiority over specialists.
In the paper's Table 1, EMDB MPJPE is 71.8 for the specialist versus 72.6 for GVHMR; the generalist result is marked as not obtained. Section 4.4 explicitly reports degradation during joint sparse-task training, so a generalist score must not be inferred.
Figure 7 shows an almost flat learning curve when training DiT from scratch, with improvement as more pretrained layers are transferred. This supports the importance of pretraining, not a strict causal proof that generation is superior to every alternative objective.

Highlights & Insights¶

Encoding task properties in representations rather than network branches reduces adaptation work for dense tasks. The Rothko raymap is a concrete example rather than a generic claim about multimodal unification.
Single-pass adaptation jointly handles input noise conventions, timesteps, and the sign of velocity outputs. Transfer therefore requires more than changing training data; respecting the pretrained interface helps retain useful initialization.
The negative sparse-query result is informative: unified input-output formats do not guarantee unified optimization behavior. Extending a task interface requires separately checking whether it damages existing representations.

Limitations & Future Work¶

The authors explicitly observe severe degradation from joint keypoint training and hypothesize a mismatch between coordinate regression and pixel-space pretraining, alongside attention interference from randomly initialized queries. These explanations have not been isolated as individual causal factors.
This note's assessment: comparisons between pretraining paradigms remain confounded by parameter counts, pretraining corpora, and compute. The claim of 7x to 500x less data should be restricted to post-training, not interpreted as a reduction in total data or total cost.
This note's assessment: transfer to animals, robots, and multiple instances is mainly illustrated qualitatively in Figure 8. This is not equivalent to establishing general physical understanding through comprehensive OOD benchmarks. Referring-expression segmentation also does not use purely synthetic training data.
This note's assessment: one forward pass does not guarantee low-latency deployment. A 14B model and a 256-TPU training configuration remain expensive, and the cached main paper provides insufficient practical latency and memory comparisons. Further work could explore pretraining that natively supports sparse queries and systematic measurements of task interference and deployment overhead.

vs Marigold / GenPercept / Diception: These methods transfer image generation priors to depth or general perception. GenCeption uses a native video backbone, builds on spatiotemporal modeling, and evaluates multiple video task families.
vs DepthCrafter / NormalCrafter / ReferEverything: These focus on depth, normals, or referring-expression segmentation, respectively. GenCeption attempts unified dense-task representations, but unification does not improve every result.
vs V-JEPA / VideoMAE V2: All seek transferable video representations. GenCeption already has text conditioning and a generative pretraining interface, and performs better after training on the same downstream data, but an equal-pretraining-budget comparison is absent.
vs Video models are zero-shot learners and reasoners / Vision Banana: The former emphasizes training-free prompting through multi-step generation, while the latter focuses on images. GenCeption uses dedicated post-training and single-pass video perception, so its results should not be mistaken for zero-shot abilities without adaptation.

Rating¶

Novelty: 4/5. The combination of video generation priors, single-pass prediction, and unified data representations is systematic, although several ingredients build on prior work.
Experimental Thoroughness: 4/5. Task and benchmark coverage is broad, and joint-training failures are reported; pretraining-budget controls and quantitative OOD evaluation remain limited.
Writing Quality: 4/5. The method is clearly motivated, but the scope of architectural unification, sparse branches, and MeViS training statements requires careful cross-checking.
Value: 4/5. The work offers a concrete approach to transferring video generators into perception and exposes obstacles to truly general models.