V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/HL-hanlin/V-Co
Area: Image Generation
Keywords: Diffusion Models, Pixel-Space Diffusion, Co-Denoising, Representation Alignment, DINOv2

TL;DR

V-Co systematically dissects visual co-denoising in a controlled JiT-based pixel diffusion framework and establishes an effective recipe comprising a fully dual-stream backbone, structural semantic-to-pixel CFG masking, a perceptual-drifting hybrid loss, and RMS-based SNR calibration, substantially outperforming prior pixel-space diffusion models on ImageNet-256.

Background & Motivation

Pixel-space diffusion models (e.g., JiT) have recently re-emerged as a compelling alternative to latent diffusion models (LDMs). By operating directly in native pixel space with scalable Transformer-based denoisers, pixel diffusion circumvents the reconstruction bottlenecks, information loss, and inductive biases introduced by pretrained variational autoencoders (VAEs). However, standard pixel-level velocity prediction or noise prediction objectives offer only weak high-level semantic supervision. Because pixel loss treats local low-frequency and high-frequency details uniformly, the denoiser learns semantic concepts and compositional structures with relatively poor sample efficiency, requiring immense model capacity and extended training schedules to approach LDM fidelity.

To inject rich visual priors into diffusion models, recent efforts have explored representation alignment (such as REPA aligning intermediate hidden states) and representation-latent diffusion (such as RAE). Meanwhile, visual co-denoising (e.g., Latent Forcing, ReDi) has emerged as a promising direction: by jointly generating image pixels alongside patch-level semantic features from a frozen self-supervised vision encoder (e.g., DINOv2), the two streams continuously exchange information across the entire generative trajectory. Nevertheless, existing co-denoising methods typically entangle multiple design choices—architectural topology, classifier-free guidance mechanisms, auxiliary training losses, and feature calibration—obscuring which ingredients are truly indispensable.

This paper builds a controlled testbed on top of JiT to systematically isolate and answer four fundamental questions: what architecture optimally balances feature-specific computation and cross-stream interaction; how to formulate an unguided baseline for CFG without semantic leakage; which auxiliary loss best complements flow matching in representation space; and how to physically calibrate cross-stream denoising difficulty. Core idea: adopt a fully dual-stream JiT backbone with structural semantic-to-pixel masking for CFG, combined with a perceptual-drifting hybrid loss and RMS-based SNR matching, delivering a simple and highly effective recipe for pixel-semantic co-denoising.

Method

Overall Architecture

V-Co is formalized under a continuous flow-matching formulation with velocity prediction (\(v\)-loss). Given a clean image \(x\) and its patch-level semantic features \(d\) extracted from a frozen DINOv2 encoder, independent Gaussian noise \(\epsilon_x, \epsilon_d \sim \mathcal{N}(0, I)\) is added at diffusion timestep \(t \in [0, 1]\): $\(z_t^x = t x + (1-t)\epsilon_x, \quad z_t^d = t d + (1-t)\epsilon_d\)$ The co-denoising network takes \((z_t^x, z_t^d, t, c)\) as input (where \(c\) is the class condition) and jointly predicts clean targets \((\hat{x}, \hat{d})\). These are transformed into velocity predictions \(\hat{v}_x = (\hat{x} - z_t^x)/(1-t)\) and \(\hat{v}_d = (\hat{d} - z_t^d)/(1-t)\) supervised by ground-truth targets.

The V-Co pipeline integrates four carefully isolated design choices: dual-stream dedicated processing with joint attention, structural masking that severs semantic influence during unconditional sampling, a hybrid loss balancing instance attraction and distributional repulsion, and RMS rescaling ensuring matched signal-to-noise ratios.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Noised Pixels and DINOv2 Semantic Features"] --> B["RMS Feature Rescaling<br/>Match cross-stream SNR via pixel-to-semantic RMS ratio"]
    B --> C["Dual-Stream JiT Architecture<br/>Separate Norm/MLP/QKV projections with joint self-attention"]
    C --> D["Structural Semantic-to-Pixel Masking<br/>Block cross-attention from semantic to pixel stream for CFG"]
    D --> E["Perceptual-Drifting Hybrid Loss<br/>Instance-level paired attraction plus repulsive drift"]
    E --> F["Output: Predicted Velocities and Denoised High-Fidelity Image"]

Key Designs

1. Dual-Stream JiT Architecture: Preserving Feature-Specific Processing with Flexible Cross-Stream Interaction

Pixel values and high-level DINOv2 representations lie in fundamentally disparate manifolds. When using single-stream shared backbones—such as early element-wise addition or channel concatenation followed by shared Transformer blocks—the forced parameter sharing constrains the expressive capacity of both streams. Under controlled evaluations with 12 JiT blocks, direct addition and channel concatenation result in poor unguided FIDs of 15.15 and 14.33, respectively. Even with token-concatenation fusion, dedicating fewer blocks to stream-specific processing harms performance.

V-Co adopts a fully dual-stream JiT architecture (Dual-Stream JiT Blocks). Across all layers, the pixel stream and semantic stream maintain dedicated LayerNorm layers, Feed-Forward Networks (MLPs), and attention projection matrices (Query, Key, Value). The two streams interact exclusively via joint self-attention by concatenating their projected tokens along the sequence dimension. This topology allows each stream to retain its specialized feature processing while enabling the model to learn flexible, data-driven cross-stream information transfer at every block, lowering unguided FID to 8.86 under identical parameter counts.

2. Structural Semantic-to-Pixel Masking: Eliminating Semantic Leakage in CFG Unconditional Paths

Classifier-free guidance (CFG) synthesizes guided velocity via an extrapolated direction \(\Delta_x = \hat{v}_x^{\text{cond}} - \hat{v}_x^{\text{uncond}}\). In visual co-denoising, the model is conditioned on both class labels and semantic feature inputs. Conventional methods adopt input-level corruption, replacing semantic tokens with zero vectors or learnable [null] tokens during unguided evaluation. However, replacing inputs with corrupted vectors causes severe distributional shift and semantic leakage, causing guided FID to collapse to 24.75 when class and semantic inputs are jointly dropped.

V-Co introduces structural semantic-to-pixel masking. Instead of altering input values, the unconditional branch structurally zeroes out the attention weights from the semantic Key/Value tokens to the pixel Query tokens. This cleanly severs any semantic information flow into the pixel generation path while preserving pixel-to-semantic interaction to maintain semantic branch stability. Paired with joint condition dropout during training, structural masking ensures that the unguided baseline reflects purely unconditional generation, sharply improving guided FID from 6.69 to 3.18.

3. RMS Feature Rescaling: Equivalent SNR Matching to Noise-Schedule Shifting

Because DINOv2 features and normalized pixel tensors have widely divergent magnitudes, assigning both streams to identical diffusion timesteps \(t \in [0, 1]\) introduces severe discrepancies in denoising difficulty. Under flow matching, denoising difficulty is governed by the signal-to-noise ratio: $\(\text{SNR}(t) \propto \frac{t^2 \mathbb{E}[\|s\|^2]}{(1-t)^2 \mathbb{E}[\|\epsilon\|^2]}\)$ When noise scale is held constant, mismatched signal norms cause one stream to denoise significantly faster or slower than the other, causing conflicting gradient updates across streams (omitting rescaling degrades guided FID to 5.28).

V-Co resolves this by rescaling the semantic features using the ratio of root-mean-square magnitudes: $\(d' = \alpha \cdot d, \quad \alpha = \frac{\text{RMS}(x)}{\text{RMS}(d)}\)$ The authors mathematically establish that scaling semantic features by \(\alpha\) is SNR-equivalent to keeping features unscaled while shifting the semantic diffusion schedule to \(t' = \frac{\alpha t}{1 + (\alpha - 1)t}\). RMS rescaling matches the effective noise schedules across streams without introducing cumbersome multi-schedule hyperparameters, driving guided FID to 2.52.

4. Perceptual-Drifting Hybrid Loss: Unifying Instance-Level Attraction and Distributional Repulsion

Standard velocity regression supervises local pointwise matching without enforcing topological structure in representation space. While REPA aligns intermediate diffusion features with DINOv2, its rigid mean-squared error constraint over-restricts model representations later in training (improving guided FID by only 0.05). Conversely, standard perceptual loss enforces pairwise attraction but is prone to local mode clustering, while drifting loss provides distributional coverage but lacks paired multi-step trajectory supervision.

V-Co designs a Perceptual-Drifting Hybrid Loss. The positive field acts as an instance-level perceptual pull toward the paired ground-truth feature \(\phi(x_i)\): $\(V_{\text{pos}}(u_i) = \phi(x_i) - \phi(\hat{x}_i)\)$ Concurrently, a negative field repels sample \(i\) away from other generated samples within the same class using kernel-normalized weights \(\alpha_{ij} \propto \exp(-\|\phi(\hat{x}_i) - \phi(\hat{x}_j)\|^2 / \tau_{\text{rep}})\): $\(V_{\text{neg}}(u_i) = \sum_{j \neq i} \alpha_{ij} (\phi(\hat{x}_j) - \phi(\hat{x}_i))\)$ A similarity-based gating factor \(s_i = \exp(-\|\phi(\hat{x}_i) - \phi(x_i)\|^2 / \tau_{\text{gate}})\) adaptively combines both fields into \(V_{\text{hyb}}(u_i) = s_i V_{\text{pos}}(u_i) + (1 - s_i) V_{\text{neg}}(u_i)\). The resulting drifting loss \(\mathcal{L}_{\text{drift}} = \|u_i - \text{sg}(u_i + V_{\text{hyb}}(u_i))\|_2^2\) lets repulsion dominate early when samples are far from the target to prevent mode collapse, shifting smoothly to pure attraction as samples converge, advancing guided FID to 2.44.

Loss & Training

The overall training objective combines the co-denoising velocity loss with the auxiliary hybrid loss: $\(\mathcal{L} = \mathcal{L}_{\text{v-co}} + \lambda_{\text{hyb}} \mathcal{L}_{\text{drift}}\)$ where the base velocity loss is defined as: $\(\mathcal{L}_{\text{v-co}} = \mathbb{E}\Big[\|\hat{v}_x - v_x\|_2^2 + \lambda_d \|\hat{v}_d - v_d\|_2^2\Big]\)$ A hyperparameter sweep over \(\lambda_d\) demonstrates that \(\lambda_d \in \{0.01, 0.1\}\) yields the lowest FID, corresponding to a pixel-to-semantic gradient norm ratio of \(4\times\) to \(2\times\). This confirms that semantic supervision is most effective when providing strong auxiliary guidance while remaining secondary to pixel-level reconstruction. All models are trained on ImageNet-256 following standard JiT optimization settings without task-specific tuning.

Key Experimental Results

Main Results

On the ImageNet 256×256 class-conditional benchmark, models are evaluated on 50,000 generated samples using FID and Inception Score (IS). V-Co demonstrates substantial parameter efficiency and convergence speed, outperforming the underlying JiT baseline as well as prior pixel-space and latent-space diffusion models.

Model	Space / Paradigm	#Params	#Epochs	FID↓	IS↑
JiT-B/16	Pixel-space diffusion	131M	600	3.66	275.1
JiT-L/16	Pixel-space diffusion	459M	600	2.36	298.5
JiT-H/16	Pixel-space diffusion	953M	600	1.86	303.4
JiT-G/16	Pixel-space diffusion	2.0B	600	1.82	292.6
Latent Forcing (JiT-B/16)	Pixel-space co-denoising	465M	200	2.48	-
PixelGen-XL/16	Pixel-space diffusion	676M	160	1.83	293.6
ReDi (SiT-XL/2)	Pixel-space joint synthesis	675M	350	1.72	278.7
DiT-XL/2	Latent diffusion (VAE)	724M	1400	2.27	278.2
SiT-XL/2	Latent flow matching (VAE)	724M	1400	2.06	277.5
V-Co-B/16 (200 ep)	Pixel-space co-denoising	260M	200	2.52	242.6
V-Co-B/16 (600 ep)	Pixel-space co-denoising	260M	600	2.35	261.1
V-Co-L/16 (200 ep)	Pixel-space co-denoising	918M	200	2.10	243.0
V-Co-L/16 (500 ep)	Pixel-space co-denoising	918M	500	1.72	245.3
V-Co-H/16 (300 ep)	Pixel-space co-denoising	1.9B	300	1.71	263.3

Ablation Study

Systematic controlled ablations on the JiT-B/16 backbone demonstrate the isolated impact of model architecture, CFG unguided pathways, feature calibration, and auxiliary objectives.

Component / Stage	Configuration Variant	CFG=1.0 FID↓	Guided FID↓	Note
Architecture	Direct Addition	15.15	-	156M params; early shared fusion severely limits capacity
	Channel Concat	14.33	-	157M params; unisolated forward processing
	Token Concat (6 spec / 6 shared)	12.35	-	198M params; increasing specific layers steadily improves FID
	Fully Dual-Stream (0 shared / 12 dual)	8.86	-	260M params; dedicated Norm/MLP with joint self-attention
CFG Unconditional	Zero Embedding Dropout	9.17	6.69	Input replacement corrupts feature distribution
	Learnable [null] Token	9.37	6.64	Input token fails to stop cross-stream semantic leakage
	Bidirectional Masking	11.08	7.17	Blocking pixel-to-semantic attention damages semantic stream
	Semantic-to-Pixel Masking	5.62	3.18	Structural blockage with joint condition dropout is optimal
Feature Calibration	w/o RMS Scaling	9.12	5.28	Unmatched cross-stream SNR leads to conflicting denoising
	Noise-Schedule Shifting	4.81	2.93	Effective but requires complex dual-schedule hyperparameters
	RMS Feature Rescaling (Default)	5.38	2.52	Simple dynamic rescaling produces the best guided quality
Auxiliary Loss	Default \(v\)-loss only	5.38	2.96	Evaluated at 300 epochs; lacks high-level geometric guidance
	+ REPA Loss (Hidden Alignment)	5.63	2.91	Rigid intermediate constraints provide marginal gain
	+ Perceptual Loss (Feature Space)	4.28	2.73	Instance-level paired attraction improves alignment
	+ Drifting Loss (Distribution Space)	4.86	2.85	Distributional repulsion enhances feature diversity
	+ Perceptual-Drifting Hybrid Loss	4.44	2.44	Gated integration of attraction and repulsion reaches lowest FID

Key Findings

• Dual-stream specialization surpasses brute-force scaling: V-Co-B/16 (260M parameters) achieves 2.35 FID at 600 epochs, matching the 459M-parameter JiT-L/16 (2.36 FID) while using nearly half the parameters. V-Co-H/16 reaches 1.71 FID in 300 epochs, outperforming the 2.0B-parameter JiT-G/16 (1.82 FID after 600 epochs).
• Structural masking stabilizes classifier-free guidance: In dual-stream generation, input-level zeroing leads to distorted guidance vectors that amplify high-frequency artifacts under CFG scaling. Structural attention masking cleanly separates conditioning without corrupting input statistics.
• Zero sampling overhead for external encoders: At test-time sampling, V-Co does not evaluate the frozen DINOv2 encoder. Consequently, V-Co-B achieves an inference throughput of 0.484 images/s, noticeably faster than JiT-L (0.329 images/s) while attaining superior synthesis fidelity.

Highlights & Insights

• Topological decoupling for condition extrapolation: By converting input-level condition replacement into directional attention masking, V-Co offers an elegant, generalizable solution to multi-condition CFG distortion in joint generative models.
• Unified SNR perspective on multimodality diffusion: Proving the mathematical equivalence between RMS amplitude normalization and noise-schedule shifting clarifies how to balance diverse multi-stream diffusion processes with a single scalar operation.
• Adaptive gating for distributional regularization: Adapting drifting loss from single-step generators to multi-step diffusion via similarity-based gating provides an effective paradigm for balancing paired instance fidelity with mode-collapse prevention.

Limitations & Future Work

• Training-time encoder compute: While sampling requires no external encoder, extracting DINOv2 patch tokens during training adds approximately 35 GFLOPs per sample, increasing overall pre-training computation.
• Single vision teacher explored: The current study evaluates DINOv2 exclusively. Investigating multimodal encoders (e.g., CLIP, SigLIP) or generative representations could expand applicability to text-to-image synthesis.
• Evaluation scale: Experiments are conducted primarily on ImageNet-256 class-conditional synthesis; scaling V-Co to megapixel resolutions (512×512, 1024×1024) and open-vocabulary prompt-guided generation remains open for exploration.

Related Work & Insights

• vs JiT: JiT demonstrates scalable pixel diffusion but suffers from weak semantic supervision. V-Co integrates a co-denoising semantic stream into JiT, showing that a fully dual-stream backbone is far more compute- and parameter-efficient than merely widening single-stream Transformers.
• vs Latent Forcing: Latent Forcing employs independent noise schedules and unaligned features. V-Co establishes the mathematical equivalence of RMS scaling to schedule shifting, achieving better generation fidelity (FID 2.35 vs. 2.48) with a unified timestep.
• vs REPA & PixelGen: While REPA over-constrains intermediate hidden states and PixelGen relies strictly on pairwise perceptual loss, V-Co introduces a hybrid objective that synergizes paired attraction with repulsive drifting, preventing mode collapse.

Rating

• Novelty: ⭐⭐⭐⭐ [Systematically decomposes co-denoising mechanisms, proposing structural CFG masking, RMS calibration equivalence, and a hybrid perceptual-drifting loss]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive controlled ablations covering architecture, CFG, calibration, and objectives, with compelling ImageNet-256 benchmarks and throughput metrics]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, tightly structured experiments, and insightful pedagogical narrative]
• Value: ⭐⭐⭐⭐⭐ [Provides an actionable, high-performance blueprint for training representation-aligned pixel-space diffusion models without VAE dependencies]

Related Papers

• [ECCV 2026] MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
• [ECCV 2026] Divide and Conquer: Decoupled Representation Alignment for Multimodal World Models
• [ECCV 2026] Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment
• [ECCV 2026] REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion
• [ECCV 2026] Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?