Control-DINO: Feature Space Conditioning for Controllable Video Diffusion

Conference: ECCV2026
Paper: Official page · PDF
Project: Control-DINO
Area: Video Generation
Keywords: Dense Feature Conditioning, Appearance Disentanglement, Video Style Transfer, 3D-Guided Generation, DINOv3

The heading follows the conference manifest; the cached PDF title additionally specifies “Image-to-Video,” which accurately describes the underlying generation setting. This note uses the paper's text and experimental tables. Code and weight availability have not been verified on the project page.

TL;DR

Control-DINO controls frozen CogVideoX with full-channel dense DINOv3 features, reduces source-appearance copying by pairing original features with multiple augmented appearances, reaches a COLMAP registration rate of 0.9932 on stride=16 T&T appearance transfer, and extends the same feature interface to 3D-guided generation.

Background & Motivation

Video appearance transfer starts with an existing scene rather than an empty canvas: a user wants different lighting or a different artistic style while retaining building outlines, object identities, and camera motion. Depth maps locate surfaces, edges identify sharp visual transitions, and segmentation maps specify category regions. Each offers useful control, but none alone expresses both geometry and fine-grained semantics. With larger camera movements, a reference first frame and weak spatial controls increasingly struggle to prevent structural drift.

Dense DINOv3 patch features offer a richer alternative because they describe objects, parts, and spatial layout together. Their strength is also a liability: color, texture, and style remain embedded in the features. If a control branch learns to reconstruct an original video from its original features, copying source appearance is an effective shortcut. At inference, this shortcut competes with the new appearance requested through an edited first frame. The paper's Real Only baseline illustrates why preserving geometry does not automatically mean following a new style.

Instead of relying primarily on PCA to remove appearance-bearing directions, the authors change the training correspondence: an unchanged feature sequence is paired with videos exhibiting different styles, photometric properties, or sharpness. Core idea: retain the structural information in high-dimensional features and teach the receiving control branch to use it selectively through multiple-appearance supervision, rather than compressing the representation into a presumed style-free subspace beforehand.

Method

Overall Architecture

Inputs consist of per-frame features obtained from a source video or a 3D scene, a first frame specifying target appearance, and text conditioning. The output follows the supplied scene structure and viewpoint changes. The dense feature interface unifies 2D and 3D sources, spatiotemporal residual adaptation injects those features into the frozen generator, and multi-appearance paired training determines what the generator should learn to read from them.

Training and inference use this interface differently. During training, features come from the original video while denoising targets come from an appearance-augmented version. During inference, only the first frame needs an appearance edit; source-domain features maintain the subsequent layout. The first-frame edit must preserve structure and semantics, otherwise appearance and structural conditions can still conflict.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Source video or 3D scene"] --> Features["Dense Feature Interface"]
    Features --> Adapter["Spatiotemporal<br/>Residual Adaptation"]
    Noisy["Noisy video latents"] --> Adapter
    Adapter --> Backbone["Frozen CogVideoX<br/>Target first frame and text"]
    Original["Original training video"] --> Features
    Original --> Pairing["Multi-Appearance<br/>Paired Training"]
    Backbone -->|Training prediction| Pairing
    Backbone -->|Inference| Output["Structurally controlled video"]

Key Designs

1. Dense Feature Interface: compensate for a coarse spatial grid with rich channels and accommodate 3D sources

The 2D path uses frozen DINOv3-ViT-S/16 to extract 384-dimensional patch descriptors from each frame. The method section specifies bicubic upsampling of input images before encoding, and the implementation gives a factor of 2; the resulting conditioning maps have shape \(384\times60\times90\). Rather than reducing these descriptors to RGB controls and passing them through the video VAE, the system preserves their original channels and delegates alignment to a dedicated adapter. Coarse spatial sampling therefore does not reduce the condition to a few category labels: each token retains part layout and regional semantics that can constrain the generation prior more richly than depth or edges alone.

The 3D paths share this interface but construct the descriptors differently. The mesh path renders untextured meshes along target camera trajectories, upsamples those views, and extracts DINOv3 features. The voxel path reconstructs a dense point cloud from an input video, associates image DINO features with 3D points, voxelizes the feature-augmented cloud, and projects it into target views. This second route does not infer DINO features from geometry alone: it uses the appearance and feature information available in the source video, so it is not information-equivalent to color-free geometric conditioning.

The Concerto path extracts point-cloud descriptors, voxelizes them, and renders DINO-like conditioning maps, allowing scenes without complete RGB coverage to use the same branch. The results discussion ultimately distinguishes the Concerto and Concerto (inpainted) rows in Table 4 as colored and uncolored point-cloud settings; they should remain separate. Holes in mesh or voxel projections must not be treated as valid scene observations. The authors propagate visibility masks and explicitly encode a binary mask through the CogVideoX VAE as an auxiliary condition in the mesh setting, identifying regions that require completion.

This interface assigns visibility and viewpoint alignment to 3D projection, regional meaning to high-dimensional descriptors, and missing textures or fine detail to the video prior. It does not guarantee that projected 3D descriptors match the distribution of image-derived DINO features. That distribution gap is one reason 3D-conditioned results remain below the 2D reference.

2. Spatiotemporal Residual Adaptation: mix features with the denoising state inside a frozen backbone

DINO produces a sequence of image representations, whereas the video VAE compresses time. Direct per-frame addition would misalign the temporal axes. Two causal Conv3D layers therefore compress 49 feature frames into 13 latent-time positions before channel-wise concatenation with noisy VAE latents. The causal organization follows the video encoder's temporal structure; this is not simply selecting 13 independent frames and discarding the rest.

The concatenated information enters a lightweight Transformer control branch with hidden dimension 256. Zero-initialized output projections add its residuals to the first 16 Transformer blocks of CogVideoX-5B-I2V. Freezing the large backbone preserves its pretrained generation prior while the small branch learns how to use the condition; zero initialization prevents an abrupt perturbation at the start of training. Inference scales residual strength by 0.8. A spatial mask can additionally modulate injection, with the unmasked setting treating all locations as valid.

The branch's Transformer does more than change channel counts at each location. It spatially mixes conditioning information with the current denoising state, allowing the model to reason about regional relationships under camera motion. Replacing this process with a direct linear projection makes condition following less reliable under large motion according to the ablation discussion. Semantic inputs do not remove the need for spatial reasoning during injection. The cached version of Eq. (2) is corrupted, so this note describes the verifiable residual mechanism in prose rather than inventing a complete equation and attributing it to the paper.

3. Multi-Appearance Paired Training: change the supervision instead of deleting suspected style directions

For each source video, DINO conditions always come from the original, unstylized sequence. Denoising targets are sampled uniformly across four groups: real video, photometric augmentation, GAN-based stylization, and mild blur. Photometric transforms include hue, saturation, brightness, contrast, and gamma, applied consistently across the sequence. The GAN group uses five neural style-transfer methods. Uniform sampling refers to the four groups, not equal probability for every individual filter and the original video.

This supervision makes source color an unreliable unique answer while spatial layout remains predictive across all augmented targets. The branch can consequently reduce its reliance on source appearance and let the target first frame specify the desired look. This is selective, functional disentanglement; it does not claim that DINO features themselves have become independent geometry and style variables.

The authors also examine the alternative of deleting style directions first. Differences between real and stylized feature pairs estimate style directions, which are projected out before a second PCA extracts structural components. Cross-appearance cosine similarity rises from 0.9194 to 0.9481, but explained variance falls to 30.4%. Selecting bottom eigenvectors reaches similarity 0.9929 while retaining only 2.9% variance. Making representations of two appearances more alike is therefore not sufficient evidence that useful structural information survives, motivating retention of all 384 channels.

A Worked Example

Consider a 49-frame video moving around a building, with a requested painterly appearance along the same path. Extract \(384\times60\times90\) feature maps from all original frames, compress their temporal dimension to 13 positions with the causal adapter, and edit only the first frame with InstantStyle to establish the target appearance.

During denoising, the lightweight branch reads both source features and current noisy latents. Its residuals constrain building and background layout through the first 16 backbone blocks, while the edited first frame supplies the painterly appearance. With a 3D source, feature maps and visibility masks are first rendered along the camera trajectory and then passed into the same branch. This is a walkthrough, not an additional quantitative experiment; the paper does not guarantee success if the first-frame edit changes the building's structure substantially.

Loss & Training

Training uses squared error for velocity prediction. Making the supervision correspondence explicit, the target comes from augmented video latents \(\tilde z_0\), while the condition comes from original-video features \(F_{1:T}\):

\[ v_t=\alpha_t\epsilon-\sigma_t\tilde z_0,\qquad \mathcal L=\mathbb E\left[\left\|v_t-v_\theta(z_t,t,F_{1:T})\right\|_2^2\right]. \]

Here \(t\) denotes diffusion time, whereas \(1:T\) indexes video frames; \(z_t\) is obtained by adding forward-process noise to augmented video latents. First-frame and text conditions remain present in the backbone despite being omitted from this compact expression. This notation restates the velocity target and augmented pairing described in the text, without adding a disentanglement loss. The cached forward-noising Eq. (1) and feature-shape Eq. (3) have missing characters, so they do not support reconstructing a complete noise schedule or further dimension conventions.

Only the temporal adapter and lightweight Transformer branch are trained from scratch. AdamW uses \(\beta_1=0.9\) and \(\beta_2=0.95\), peak learning rate \(2\times10^{-4}\), cosine annealing, 500 warmup steps, batch size 8, and gradient clipping at 1.0. Conditioning inputs are zeroed with probability 0.1. The main text does not provide verifiable total training steps, complete GPU costs, or inference speed; none are supplied here.

Key Experimental Results

Main Results

The base model is trained on DL3DV using approximately 4.5M unique training frames and evaluated on approximately 10k unique frames. Out-of-domain evaluation uses Tanks and Temples at sampling strides 8 and 16, with 15 lighting or stylistic variations per scene that differ from training. The following excerpt from Table 1 covers the larger-motion stride=16 setting. All displayed metrics are higher-is-better.

Method	Subject Consistency	Background Consistency	Aesthetic	Imaging Quality	First-Frame CLIP Similarity	COLMAP Registration
Wan 2.2 Fun Depth	0.8222	0.8527	0.5044	0.6497	0.4804	0.4956
Wan 2.2 Fun Canny	0.8226	0.8512	0.5104	0.7235	0.4630	0.5918
AnyV2V	0.7621	0.8579	0.5072	0.6299	0.4864	0.1875
Control-DINO Real Only	0.8544	0.8906	0.5485	0.6908	0.4406	0.9911
Control-DINO	0.8461	0.8862	0.5508	0.7109	0.4707	0.9932

Relative to Canny, registration improves by 0.4014, or 40.14 percentage points, but imaging quality falls from 0.7235 to 0.7109. This is not a universal win. Relative to Real Only, mixed augmentation increases first-frame CLIP similarity by 0.0301 while slightly reducing subject and background consistency. That is direct evidence of improved appearance transfer, accompanied by a measurable trade-off.

CLIP similarity uses the target first frame as a reference and serves as an appearance-following proxy, not an independent measure of pure style accuracy. COLMAP registration indicates successful view registration, not perfect geometry. Baselines use different pretrained backbones and official checkpoints, so cross-model differences cannot be attributed exclusively to the choice of DINO features.

Ablation Study

The following three rows are clearly readable in Table 6's DL3DV140 architectural ablation. Tail Drop constructs a 64-component PCA basis during training and randomly retains the first \(k\in\{8,16,32,64\}\) components. The reported variants retain 64 or 8 components at inference.

Configuration	FID, Lower Is Better	Subject Consistency	Background Consistency	Aesthetic	Imaging Quality
Control-DINO	71.67	0.959	0.955	0.504	0.542
Tail Drop, k=64	108.87	0.918	0.920	0.459	0.494
Tail Drop, k=8	156.67	0.845	0.908	0.468	0.490

The two Tail Drop settings increase FID by 37.20 and 85.00 relative to the full model. The cached rows for removing spatial mixing and using style-invariant PCA are merged, so their numbers are not guessed; the spatial-mixing ablation is discussed only qualitatively. Table 3 also uses DL3DV140 but reports a separate reconstruction and random-seed analysis with Control-DINO FID 20.23. That value must not be compared against Table 6's 71.67 to calculate an improvement.

3D Conditioning Analysis

For ScanNet++, the base model is fine-tuned on nearly 8k training clips sampled at stride=1 and evaluated on part of the official validation split. This excerpt from Table 4 preserves the distinction between the 2D reference and 3D-derived inputs.

Conditioning Source	PSNR, Higher Is Better	SSIM, Higher Is Better	LPIPS, Lower Is Better	FID, Lower Is Better
2D Control-DINO, reference	27.93	0.9084	0.0963	38.5853
Untextured mesh renderings	21.41	0.8010	0.1807	67.1730
Voxel features	22.80	0.8153	0.1572	65.3810
Concerto, colored setting	21.19	0.7805	0.1752	67.9680
Concerto (inpainted), uncolored setting	19.66	0.7542	0.2060	78.8411

Key Findings

• Structural control is particularly strong under larger camera motion, but not every setting favors the method: at stride=8, Canny registration is 0.9770, above Control-DINO's 0.9530.
• Rich conditioning accompanies lower output diversity. In Table 3, Control-DINO has Latent Var. 0.1944 and CLIP Var. 0.0051, compared with Canny's 0.4787 and 0.0150. The text does not fully specify how these variance statistics are computed, so they are used only as within-table evidence of seed sensitivity.
• A control adapter cannot remove missing input information. Voxel conditioning beats meshes but remains below the full-image-feature reference; uncolored Concerto reaches imaging quality 0.6326 without implying lower reconstruction error.

Highlights & Insights

• Disentanglement need not require channel removal. Pairing source features with multiple output appearances lets the receiving model learn which appearance cues are unreliable while avoiding PCA-induced loss of structural information.
• Dense semantic features provide a practical interface between 2D and 3D. High-dimensional descriptors supplement coarse geometry with semantics, while visibility masks distinguish unknown regions from valid conditions.
• Transfer evaluation must measure both structure and appearance. Real Only's high registration and low first-frame similarity show how reporting geometry alone can hide appearance-following failures.

Limitations & Future Work

• The authors acknowledge source-appearance leakage, especially for styles far from the training distribution. Weaker conditioning can mitigate leakage but also relax structural constraints; this is not strict independent control of style and geometry.
• First-frame editing must preserve structure and semantics. Object replacement and major layout edits are not established general capabilities, and appearance transfer should not be presented as arbitrary video editing.
• 3D generation depends on reconstruction fidelity, holes, and the alignment between projected 3D features and image-derived DINO distributions. Color-free point-cloud results favor plausible generation rather than exact recovery of real textures.
• From a critical reading perspective, unmatched backbones and training resources, absent significance analysis, and incomplete compute reporting constrain efficiency and causal claims. The cache additionally contains damaged equations, merged table rows, and scattered explanations of Concerto settings; reported results must be separated from details that can actually be verified.
• Future work could learn spatial conditioning weights to attenuate uncertain 3D regions. A controlled comparison of DINO, depth, edges, and other foundation feature spaces under matched backbones, datasets, and training budgets would also clarify their respective contributions.

Related Work & Insights

• vs ControlNet / T2I-Adapter: The method inherits frozen generation backbones with trainable conditioning branches. Its distinction is high-dimensional foundation features and handling their appearance entanglement, not the residual-control idea itself.
• vs DIVE: DIVE focuses on subject-driven editing with per-video MLP optimization. Control-DINO learns a reusable scene-level dense adapter and evaluates 3D-derived conditioning without per-video optimization.
• vs Driving with DINO: This concurrent work also injects DINOv3 through a control branch but targets driving sim-to-real transfer with PCA-based spectral pruning. Control-DINO instead retains raw high-dimensional features with multi-appearance supervision; its results do not establish that pruning is ineffective for every task.
• vs REPA / Concerto: REPA uses DINO representations as training alignment targets, whereas this method uses them as explicit spatial conditions at inference. Concerto supplies 3D descriptors and is an input source here, not a video generation baseline.

Rating

• Novelty: 4/5. The combination of multi-appearance paired training and a shared 2D/3D feature interface is useful, although residual adaptation has established precedents.
• Experimental Thoroughness: 4/5. Cross-domain transfer, 3D generation, and feature ablations provide broad evidence, but backbone differences and missing budget details limit strict attribution.
• Writing Quality: 3/5. The main argument is clear, but some protocols are insufficiently consolidated, and the available cache corrupts equations and selected ablation rows.
• Value: 4/5. A reusable approach to structure-first video editing and generative rendering, with an informative account of the control-diversity trade-off.