V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

Conference: ECCV2026
Paper: ECCV Paper
Area: Self-Supervised Learning
Keywords: dense features, deep self-supervision, video representation learning, joint-embedding prediction, world models

TL;DR

V-JEPA 2.1 extends video latent-prediction supervision to visible positions and multiple network depths, improving local spatiotemporal structure while recovering global semantic capabilities; its ViT-G reaches NYUv2 depth RMSE of 0.307 and SSv2 action recognition accuracy of 77.7% in frozen-feature evaluations.

Background & Motivation

Video representations must capture not only what happens but also where objects are, how they move, and when contact occurs. V-JEPA 2 already learns strong action and dynamic semantics from unlabeled videos, but its training loss evaluates predictions only at masked positions. Visible positions contribute to prediction without being required to retain information about their own local regions in their outputs. They can aggregate information into global summaries useful for completing masked regions instead of becoming features suitable for position-wise readout. This explains why strong action classification does not necessarily imply strong depth estimation or semantic segmentation. In contrast, the DINO family provides strong dense image features but does not directly learn temporal dynamics through video training.

Simply supervising visible positions does not solve the problem in one step. In Table 1, adding the context loss increases ADE20K mIoU from 22.2 to 33.8 but reduces SSv2 accuracy from 72.8% to 62.5%. Protecting local information therefore comes with a substantial cost to global understanding. The paper investigates how one representation can support both local structure and global dynamics, rather than merely produce smoother visualizations. The authors extend supervision to intermediate encoder layers so information from different depths jointly participates in latent prediction.

Image-video data composition, input tokenization, and model capacity then become conditions for further improvement. Images provide diverse appearances and videos provide motion and temporal structure; they share an encoder but should not be forced into the same static-video format. Core Idea: supervise visible and masked positions across space and multiple representations across network depth, then combine native image-video inputs with scaling to preserve both dense structure and global understanding.

Method

Overall Architecture

The input is an image or a video, and the output is a feature sequence retaining spatial positions and, for videos, temporal structure. Native Image-Video Tokenization converts inputs into tokens, after which the same input supplies a visible branch and a complete target branch. The visible x-encoder processes only tokens that were not randomly dropped; the y-encoder processes the complete input and supplies stop-gradient target representations. Deep Self-Supervision gathers information from multiple encoder layers, and the predictor produces layer-specific predictions for visible and masked positions. Dense Prediction Loss aligns these predictions with targets at corresponding positions and layers of the y-encoder. Data and Model Scaling subsequently increases the scale and resolution of this training recipe rather than adding an image generator at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image or video"] --> Tokenizer["Native Image-Video Tokenization"]
    Tokenizer -->|Drop masked positions| Student["Visible branch x-encoder"]
    Tokenizer -->|Complete input| Teacher["Target branch y-encoder"]
    Student --> Deep["Deep Self-Supervision<br/>Multi-level fusion and prediction"]
    Deep --> Dense["Dense Prediction Loss"]
    Teacher -.->|Multi-level targets, stop-gradient| Dense
    Student -.->|EMA parameter update| Teacher
    Dense --> Scale["Data and Model Scaling<br/>High-resolution cool-down"]
    Scale --> Output["Frozen representations and downstream tasks"]

Loss-related edges denote training supervision, not mandatory processing steps at deployment. Depth estimation and semantic segmentation train linear projections on frozen final-layer encoder features; action recognition uses an attentive probe. Action anticipation additionally uses the frozen encoder and predictor, while robotic applications separately train an action-conditioned predictor or a latent world model. General representation learning, downstream-head training, and closed-loop planning must therefore not be conflated into a system requiring no task-specific training.

Key Designs

1. Native Image-Video Tokenization: do not disguise a static image as a video

The previous recipe used a single 3D convolution for patch embedding and repeated images temporally into 16-frame static videos. This increased computation and imposed an unnatural zero-motion assumption on static appearance samples. V-JEPA 2.1 uses a 3D convolution with spatial size \(16\times16\) and temporal extent \(2\) for videos, and a \(16\times16\) 2D convolution for images. Tokens from both pathways enter a shared encoder, without duplicating images into redundant frames. Spatial and temporal relationships are represented through 3D RoPE, while a learnable modality token identifies the input type.

The modality indication appears in both encoder and predictor inputs, informing the shared network whether it should interpret static appearance or temporal change. This is not an image-text multimodal model: the two modalities here are images and videos. Nor does it train a complete independent backbone for each modality; the differences primarily concern input embeddings and modality conditioning. The image pathway is better suited to exploiting large appearance datasets, while the video pathway retains a route for learning dynamics. In Table 1, this step raises ADE20K from 40.8 to 41.4 mIoU while leaving classification results unchanged.

2. Deep Self-Supervision: feed local details and global semantics jointly into prediction

Instead of exposing only its final features, the x-encoder supplies outputs from three intermediate blocks and its final layer. These normalized representations are concatenated along the channel dimension and fused and reduced through a lightweight MLP. Learnable mask tokens carrying the spatiotemporal information of masked positions are then added along the sequence dimension to form the predictor input. The predictor produces corresponding outputs for four encoder layers, with targets taken from matching y-encoder layers. These levels are neither four independent encoders nor four outputs averaged into a single supervision target.

This retains more local information from intermediate layers while allowing higher-level semantics to participate. Both losses are computed at all four levels, distributing the learning signal beyond the network output. The authors argue that this also carries local information into the final layer, so dense downstream tasks need not concatenate additional intermediate features. The decisive evidence is that, after adding context loss, multi-level prediction restores SSv2 from 62.5% to 72.1% while further raising ADE20K from 33.8 to 38.6. It addresses the semantic capability loss introduced by local supervision rather than simply pursuing a deeper network.

3. Dense Prediction Loss: visible tokens must retain their own positional meaning

Masked positions retain the original latent-prediction objective, comparing predictor outputs against corresponding target-encoder representations with an L1 distance. The additional context loss applies analogous position-wise alignment to visible positions. Targets remain learned features, not RGB pixels, ground-truth depth, or manually annotated segmentation labels. The complete-input target branch supplies a reference for the visible branch, while stop-gradient and exponential moving average updates retain the JEPA self-supervised framework. A visible token can no longer act solely as a global aggregator serving other positions: its prediction must also match the target at its own position.

Rather than giving all visible positions equal importance, the authors emphasize context near masked regions. The weight is the inverse square root of the minimum spatiotemporal block distance from a visible patch to any masked token. Restating the weighting relationship explicitly described in the prose gives:

\[ w_{\mathrm{ctx}}(d_{\min})=\frac{1}{\sqrt{d_{\min}}}. \]

Here \(d_{\min}\) is measured in blocks rather than pixels; visible patches nearer a masked boundary receive a stronger penalty for context-prediction errors. This encourages continuity around visible-masked boundaries without imposing an equally strong constraint on distant context. Equations (1) and (2) are corrupted in the text extraction; the expression above only restates the weight defined in readable prose and does not claim to reconstruct the full original equations. No loss coefficients, exact distance norm, or unspecified implementation details are inferred from the damaged expressions.

4. Data and Model Scaling: expand appearance coverage and capacity after correcting the objective

The authors replace approximately 1M ImageNet images with LVD-142M and combine these with existing video sources to form VisionMix163M. The readable text describes approximately 19M video samples but does not provide a complete itemized accounting of the total here, so remaining sources cannot be filled in by inference. Additional static images broaden the appearance distribution, while videos continue to supply motion information. This changes the coverage of unlabeled pretraining inputs rather than adding manually labeled pixel supervision. In the cumulative ablation, image-data scaling improves depth and segmentation and slightly improves classification.

The model then scales from a 300M-parameter ViT-L to a 2B-parameter ViT-G. High-resolution cool-down lowers the learning rate while increasing image resolution from \(256\times256\) to \(512\times512\). Videos increase from 16 frames at \(256\times256\) to 64 frames at \(384\times384\). This changes spatial detail and temporal context together, so its gains cannot be attributed exclusively to higher image resolution. It reduces NYUv2 RMSE from 0.365 to 0.307, showing that the final training schedule also matters for dense capabilities.

A Worked Example

Consider a short video of a hand approaching a cup; this illustrates the pipeline and is not an additional experiment reported by the paper. The video tokenizer converts local regions across adjacent frames into spatiotemporal tokens, and masking removes some positions. The x-encoder sees only the remaining hand, cup, and background regions, whereas the y-encoder sees the complete clip. After four-level feature fusion, the predictor completes masked positions and predicts the complete-view representations of visible positions as well. For a visible cup rim adjacent to a masked boundary, the context loss receives a higher weight, encouraging the token to represent the rim rather than merely summarize the entire action. Deep Self-Supervision simultaneously retains more global dynamic information, such as the hand approaching the cup. In an action-anticipation system, a downstream probe reads these frozen representations; robot control additionally requires action-conditioned prediction and planning components.

Loss & Training

Pretraining supervises both masked tokens and visible context tokens at four encoder levels. The target encoder follows an EMA of student-encoder parameters rather than receiving direct back-propagation from the student loss. The paper describes the mechanism, but the supplied cache does not include the referenced supplementary material; learning rates, masking ratios, and layer indices are therefore not filled in. For distillation, a frozen ViT-G teacher replaces the EMA target encoder, and an EMA copy of the student becomes the final model. Distillation matches only the teacher's final layer without Deep Self-Supervision; the predictor is reduced to 12 blocks with a final projection. This branch produces a 300M ViT-L and an 80M ViT-B and should not be treated as identical to large-model pretraining from scratch.

Key Experimental Results

Main Results

The following selection comes from Table 6, page 11; depth and semantic segmentation use linear probes on frozen final-layer features, while video object segmentation uses non-parametric label propagation. Lower RMSE and higher mIoU and J&F are better; different parameter counts do not constitute compute-matched comparisons.

Model	Parameters	NYUv2 RMSE	KITTI RMSE	ADE20K mIoU	VOC12 mIoU	YouTube-VOS J&F
DINOv3 ViT-7B	7B	0.309	2.346	55.9	86.6	74.1
DINOv2	1B	0.372	2.624	49.5	83.1	65.6
V-JEPA 2 ViT-g	1B	0.642	4.650	24.4	63.9	53.7
V-JEPA 2.1 ViT-g	1B	0.350	2.601	47.8	84.7	72.3
V-JEPA 2.1 ViT-G	2B	0.307	2.461	47.9	85.0	72.7

The 1B-to-1B comparison already shows substantial dense-feature improvement, so gains do not come entirely from scaling to ViT-G. However, ViT-G does not outperform DINOv3 on every dense task: ADE20K remains 47.9 versus 55.9.

The next selection combines Tables 2 and 3 on page 9 with Table 7 on page 12, retaining same-protocol comparisons within the V-JEPA family. Ego4D All mAP jointly evaluates interaction objects, actions, and time to contact; EK100 reports validation-set mean-class Action Recall@5. EK100 uses 32 frames at 8 fps and \(384\times384\) resolution; SSv2 reports attentive-probe Top-1 accuracy on frozen features.

Model	Parameters	Ego4D All mAP	EK100 Action Recall@5	SSv2 Top-1 (%)
V-JEPA 2 ViT-g	1B	6.02	39.7	77.3
V-JEPA 2.1 ViT-g	1B	6.75	38.4	76.9
V-JEPA 2.1 ViT-G	2B	7.71	40.8	77.7

At 1B, localized interaction anticipation improves, but EK100 and SSv2 decline slightly; improvement on every task is not supported. The 2B EK100 increase from 39.7 to 40.8 is 1.1 percentage points, or approximately 2.8% relative improvement, not 2.8 percentage points.

Ablation Study

Table 1, page 6, adds components cumulatively; dense tasks use linear probes and classification tasks use attentive probes. This is a progressive addition experiment, not a set of independent removals with every other factor held constant.

Cumulative Config	IN1K Acc. (%)	SSv2 Acc. (%)	NYUv2 RMSE	ADE20K mIoU
V-JEPA 2 baseline	82.2	72.8	0.682	22.2
+ Context Loss	72.6	62.5	0.474	33.8
+ Multi-level Pred.	80.8	72.1	0.463	38.6
+ Vision Mix	81.6	72.6	0.418	40.8
+ Multi-modal Tok.	81.6	72.6	0.415	41.4
+ Model Scaling	84.8	76.1	0.365	47.1
+ Cool-down	85.5	77.7	0.307	47.9

The first discussion of context loss on page 7 gives 33.9 mIoU and 0.473 RMSE, slightly different from Table 1's 33.8 and 0.474; this note uses the table values and retains the discrepancy. Table 1's baseline belongs to the ablation setting and must not be mixed with Table 6's 1B main-experiment baseline of 0.642 RMSE and 24.4 mIoU.

Key Findings

• Context supervision and Deep Self-Supervision are complementary: the former supplies local constraints, and the latter substantially restores classification capabilities. Adding only the former does not represent the complete recipe.
• Table 4, page 10, reduces navigation planning time from NWM's 103.2 seconds to 10.6 seconds; ViT-G reaches Tartan Drive ATE of 5.687 against 5.831. This evaluates open-loop trajectories, not real-time closed-loop obstacle-avoidance success.
• Table 5, page 11, improves grasp success from 60% to 70% under the same budget of 800 samples, 10 iterations, horizon 1, and 3 seconds. Reaching 80% requires 300 samples, 15 iterations, horizon 8, and 14 seconds; the total gain is 20 percentage points, with only 10 tasks per skill.
• Table 8, page 14, shows ViT-L distillation improving SSv2 from 74.2% to 76.5% and ADE20K from 42.0 to 46.7. The accompanying prose's depth change from 2.490 to 2.461 does not match the scratch-to-distilled ViT-L rows; KITTI in the table changes from 2.914 to 2.490.

Highlights & Insights

• The paper provides a direct experimental counterexample to the assumption that visible positions do not need prediction. Which tokens receive supervision influences whether they become local descriptors or carriers of global information.
• Multi-level supervision matters not just for segmentation accuracy but also for mitigating the damage local objectives cause to action semantics. This exposes dependencies among designs more clearly than final dense metrics alone.
• Frozen representations support geometry, segmentation, anticipation, and planning, demonstrating broad applicability. Different tasks still use different downstream components, so a common representation should not be mistaken for a unified task head.

Limitations & Future Work

• The authors partly attribute the complex-scene segmentation gap to insufficient cluttered scenes in VisionMix. This is an explanatory hypothesis, not the sole cause established through an independent data ablation.
• The best global results depend on a 2B model and high-resolution cool-down; same-size video semantic tasks do not improve consistently. Fixed-compute and fixed-data comparisons would better isolate the sources of gains.
• Robot experiments are small, and the longer-horizon configuration increases runtime; the paper does not provide confidence intervals sufficient to establish reliable deployment. More scenes, objects, and control-budget-matched comparisons are needed.
• The supplied cache lacks supplementary material and contains damaged equations; exact reproduction still requires complete formulas, training hyperparameters, and downstream implementations. Qualitative PCA plots are not treated as independent causal proof of the mechanism.

Related Work & Insights

• Compared with V-JEPA 2: the method retains masked latent prediction while adding visible-position constraints and multi-level targets. The central change concerns what representations must preserve, not a switch to pixel reconstruction.
• Compared with DINOv2 / DINOv3: both pursue strong local features, but this paper learns dynamics through video training. DINOv3 still leads on several segmentation metrics, so the relationship is not wholesale replacement.
• Compared with NWM: navigation separately trains a latent model on the new representation and uses clean-representation prediction and DDIM sampling to handle high-dimensional features. Planning acceleration is a downstream-system result, not an isolated encoder-throughput measurement.
• Potential extension: compare masked-boundary weighting schemes at fixed model size and separate effects on boundary localization, temporal consistency, and action semantics. This is a research suggestion from the analysis, not an experiment completed in the paper.

Rating

• Novelty: 4/5. Context constraints and multi-level prediction have a clear mechanism and corresponding ablations, while remaining an improvement to JEPA training.
• Experimental Thoroughness: 4/5. Broad task coverage and cumulative ablations are strengths; small robot samples and scaling confounds limit some conclusions.
• Writing Quality: 4/5. The method is clearly organized, but several numerical discrepancies between prose and tables require checking.
• Value: 5/5. The work offers a reusable training approach for self-supervised representations that retain local structure and video semantics.