S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight

Conference: ECCV2026
Paper: ECCV Paper
Project: S-VAM
Authors: Haodong Yan, Zhide Zhong, Jiaguan Zhu, Junjie He, Weilin Yuan, Wenxuan Song, Xin Gong, Yingjie Cai, Guanyi Zhao, Xu Yan, Bingbing Liu, Ying-Cong Chen, Haoang Li
Affiliations: The Hong Kong University of Science and Technology (Guangzhou); Huawei Foundation Model Department
Area: Robotics & Embodied AI
Keywords: video-action models, self-distillation, geometric foresight, semantic foresight, diffusion policy

Identity check: the official-list title exactly matches the cached paper's title page; Haodong Yan and Zhide Zhong contributed equally, and Haoang Li is the corresponding author. The task is to predict robot actions from monocular observations and language, with video generation supplying policy priors, so the appropriate category is Robotics & Embodied AI rather than Image Generation.

TL;DR

S-VAM self-distills geometric and semantic foresight from future videos generated along the same diffusion trajectory into one-step video features, then trains an action expert that also retains the original features, reaching 4.16 average completed tasks on CALVIN and 72.8% average success on MetaWorld.

Background & Motivation

Vision-language-action models commonly map current images and language to actions, but their pretrained vision-language backbones primarily learn from static image-text pairs and do not directly represent how objects will move next. Video-action models instead exploit the dynamic priors of video diffusion models: they construct a future visual plan that an action expert converts into control signals. The difficulty is that synthesizing a complete future video requires iterative denoising; waiting for that process can delay the robot's response to fresh observations.

VPP takes a faster route by extracting intermediate features from the first video-denoising pass instead of waiting for a viewable video. These features contain future-layout information, but noise, object identity, and geometry remain entangled, leaving the action expert to learn both representation organization and control. Under monocular observations, transparent-object depth ambiguity and target-identity drift over long sequences can particularly disrupt manipulation. S-VAM therefore seeks a future representation that is easier for an action expert to consume without requiring online multi-step video generation.

The authors use the model's own completed generation trajectory as supervision, avoiding forced alignment between a future implicit in one-step features and a different demonstrated future. Core Idea: compress geometric and semantic knowledge from the same generation trajectory into two lightweight decouplers, turning one-step video features directly into future representations for control.

Method

Overall Architecture

The inputs are a current monocular RGB observation and a task instruction; the output is a robot action sequence. SVD supplies one-step multi-layer features, and its own multi-step generated videos provide teacher targets during training; geometric and semantic decouplers predict future representations anchored by the current observation, followed by a Uni-Perceiver and a diffusion action policy. Deployment omits the multi-step video-generation path used for teacher supervision, but still requires current-observation reference representations; this should not be described as removing all vision foundation model computation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Monocular observation<br/>and instruction"] --> FEAT["One-Step Multi-Layer Features"]
    FEAT --> TEACH["Same-Trajectory Self-Distillation<br/>Training-only teacher construction"]
    TEACH -.->|Geometric and semantic targets| DEC["Reference-Anchored Dual Decoupling"]
    FEAT -->|One-step features| DEC
    IN -->|Current reference representations| DEC
    DEC --> ACT["Joint Condensation and Action Prediction"]
    FEAT -->|Retained original features| ACT
    ACT --> OUT["Robot action chunk"]

Key Designs

1. One-Step Multi-Layer Features: retain future layout without waiting for a video

The video backbone is Stable Video Diffusion (SVD), which denoises latents conditioned on the current image and instruction; S-VAM executes only the first denoising forward pass and reads intermediate features from multiple upsampling layers. Because these layers have different spatial resolutions, their features are linearly interpolated to a common grid and concatenated along the channel dimension, forming a feature volume spanning future time steps. Here, "one-step" refers to the video-diffusion feature-extraction budget, not to predicting a single robot action or eliminating denoising in the action diffusion policy.

Multi-layer concatenation exposes cues at different scales, but does not itself remove noise or disentangle representations. The paper treats this feature volume as informative yet difficult-to-use input: subsequent decouplers map it into stable geometric and semantic spaces, while an additional route preserves the original features for the action expert so that specialized transformations do not discard useful control context.

2. Same-Trajectory Self-Distillation: supervise with the model's own generated future

During training, the frozen video model continues multi-step denoising along the same diffusion trajectory that produced the one-step features, yielding a predicted video. Frozen vision foundation models extract teacher representations from these predicted frames and align them to the target spatial grid: the geometric branch uses the model called DPAv3 in the paper, and the semantic branch uses DINOv2. Reference [25] identifies Depth Anything 3; this note retains the paper's DPAv3 spelling rather than treating it as a separately verified model name.

The teacher targets are not extracted from ground-truth future demonstration frames. For example, a grasp can be approached along different paths: one-step video features already imply one possible future, so supervising them with another demonstrated route can create an alignment conflict even when the real images have higher visual quality. Self-distillation therefore pursues consistency within a prediction process, not a claim that generated videos are more accurate than real videos. The two branches independently match teachers in their respective representation spaces; multi-step generation and future-frame teacher encoding are used only for training supervision.

3. Reference-Anchored Dual Decoupling: fix the starting point of future representations

Both decouplers are spatiotemporal Transformers, with output channels corresponding to the geometric and semantic teachers respectively. Their inputs include not only noisy one-step features but also branch-specific reference representations of the current observation, repeated across time and concatenated with future features. This anchor establishes what the objects currently are and where they are, reducing the need to reconstruct the scene solely from noisy latent features. The key input operation, corresponding to the paper's Eq. (5), is:

\[ \tilde{F}_i^0=\operatorname{Concat}\left(F,\operatorname{Repeat}\left(Y_i^{\mathrm{ref}}\right)\right),\qquad i\in\{\mathrm{geo},\mathrm{sem}\}. \]

Here, \(F\) denotes aligned one-step multi-layer features and \(Y_i^{\mathrm{ref}}\) the branch-specific reference representation of the current observation; Repeat operates over time and Concat over channels. A linear layer then projects the input into a compact hidden space, followed by \(K\) factorized Transformer blocks that apply spatial aggregation before temporal aggregation, and an output projection back to the teacher's channel dimension. Spatial modeling organizes objects and layout within each frame, while temporal modeling connects future frames so that object identities and positions do not drift independently.

The geometric branch supplies dynamic structural cues, while the semantic branch provides patch-level object distinctiveness; these are complementary representations rather than duplicate encodings. The prediction target is a vision foundation model representation, not a point cloud or depth map: depth visualizations use an additionally trained DPT probing head, which is not a required component of the action policy.

4. Joint Condensation and Action Prediction: specialized foresight does not replace all original context

The action expert concatenates geometric foresight, semantic foresight, and original one-step features along the channel dimension. To avoid processing the entire high-dimensional spatiotemporal volume directly, the Uni-Perceiver uses \(N\) learnable queries and cross-attention to gather information into compact tokens, followed by self-attention and a feed-forward network to integrate relationships among them. Unlike simply passing all features to the policy, this learned condensation determines which future locations and objects are most useful for action decisions.

The condensed tokens and task-text embeddings enter a Diffusion Transformer action policy through cross-attention. Training corrupts demonstration action sequences with noise and learns to predict that noise; inference recovers executable sequences from action noise. The original-feature route is important because geometric and semantic teachers need not preserve all global context, so the action expert must combine structured foresight with residual cues.

A Worked Example

Consider the task in Fig. 4, picking up a nut and placing it onto a peg: one video-denoising forward pass produces future features involving the nut, gripper, and target region. During training, the same-trajectory generated video supplies geometric and semantic teachers; at deployment, the decouplers predict future representations from current references and one-step features without generating this teacher video. Geometric foresight helps the action expert reason about spatial relationships during approach and placement, semantic foresight maintains attention on the nut rather than nearby background, and the Uni-Perceiver condenses these cues into policy conditions. This is an explanation of the illustrated mechanism, not an additional success-rate experiment or a claim that the model performs explicit online collision checking.

Loss & Training

Training has three stages rather than end-to-end joint optimization of every module. First, SVD is initialized from VPP's embodied-scene pretrained weights and fine-tuned for 100k steps on MetaWorld or 40k steps on real-world tasks using 4 NVIDIA H100 GPUs; CALVIN directly reuses VPP's fine-tuned model. Second, the video model is frozen while the two decouplers learn representation distillation for 50k steps per benchmark on 1 H100. Finally, SVD and the decouplers are frozen while the action expert is trained for 60k steps on CALVIN or 40k steps on the other benchmarks using 4 H100 GPUs.

Geometric and semantic objectives separately encourage predicted representations to match their teachers, while the action objective trains noise prediction. The cached extraction of Eqs. (7) and (9) has incomplete norms and subscripts, so this note does not reconstruct exact loss norms or weights from corrupted text; the input-concatenation equation above only restates the operation explicitly explained for Eq. (5).

Key Experimental Results

Main Results

CALVIN uses ABC→D: training takes place in environments ABC and consecutive tasks are evaluated in unseen environment D. Avg. Len. is the average number of completed tasks in a sequence of up to 5, measured in tasks rather than percent; stage-wise success measures completion through the corresponding stage. MetaWorld contains 50 tasks with 50 demonstrations each, split into 28 easy, 11 middle, and 11 hard tasks; both simulated benchmarks use the third-person primary monocular RGB camera. The following table selects representative results from Tables 1 and 2; MetaWorld proportions in the original table have been converted to percentages.

Method	CALVIN Avg. Len. (tasks, Table 1)	CALVIN stage 5 (%, Table 1)	MetaWorld average (%, Table 2)	MetaWorld hard (%, Table 2)
Spatial Forcing	3.94	64.6	60.9	45.1
HiF-VLA	4.08	69.4	57.7	40.4
VPP	3.58	51.8	68.2	52.6
S-VAM	4.16	68.9	72.8	68.4

Relative to VPP, S-VAM completes 0.58 more tasks on average in CALVIN and improves MetaWorld average and hard-task success by 4.6 and 15.8 percentage points respectively. It does not lead every column: CALVIN stage-5 success is below HiF-VLA, and MetaWorld easy-task success is 79.3%, below VPP's 81.8%.

Ablation Study

The following CALVIN ablations come from Table 3; decreases are calculated against the same full-model value of 4.16, in tasks.

Config	Avg. Len. (tasks)	Decrease from full model (tasks)	Actual change
Full S-VAM	4.16	0.00	Dual branches, same-trajectory teachers, joint condensation, original features
w/o geometric distillation	4.01	0.15	Geometric-branch supervision absent
w/o semantic distillation	3.99	0.17	Semantic-branch supervision absent
w/o self-distillation	3.82	0.34	Teachers extracted from ground-truth future frames instead
w/o Uni-Perceiver	3.72	0.44	Joint token-condensation module removed
w/o original diffusion features	3.93	0.23	Only distilled foresight used as the feature condition

The teacher analysis in Table 4 reports 4.01 for DINOv2 alone, 3.99 for DPAv3 alone, and 4.16 for their combination. Replacing the geometric teacher gives 4.04 for DINOv2 + VGGT; replacing the semantic teacher gives 4.06 for SigLIP + DPAv3. These comparisons apply to the paper's setup and do not establish universal superiority of a vision foundation model family across tasks.

Key Findings

Real-world experiments use an AgileX Robotics Cobot dual-arm platform based on the Mobile ALOHA system design, with 7 degrees of freedom per arm, parallel grippers, and only front monocular RGB input. A unified multi-task model uses roughly 50 human demonstrations per task and is evaluated over 25 trials per task; the following success rates come from Fig. 5(b), not Table 5.

Real-world task	VPP success (%)	S-VAM success (%)	Gain (percentage points)
Place-to-Pot	40	56	16
Place-to-Pot (Hard), transparent objects	16	32	16
Pour-Water	20	44	24
Lift-Pot	12	24	12

On a single NVIDIA RTX 3090 (24GB), S-VAM takes 307.6 ms per forward pass: 231.0 ms for the video backbone, 40.1 ms for the decouplers, and 36.5 ms for the action expert. VPP takes 265.7 ms, so S-VAM adds approximately 15.8% latency rather than accelerating VPP; its efficiency argument concerns avoiding online multi-step video generation. Each pass predicts an action chunk of length 8, and the paper reports an effective control frequency of 25 Hz; this does not mean 25 complete model replanning passes per second conditioned on fresh observations.

Highlights & Insights

• Teacher compatibility matters beyond visual quality. Worse results with ground-truth future supervision suggest that feature distillation for generative policies must respect the future branch implicit in the features.
• Foresight complements original features. Structured teachers reduce the action-learning burden, yet retaining undecoupled context still helps, supporting representation augmentation rather than wholesale replacement.
• Removing the Uni-Perceiver causes the largest decrease in Table 3. Effective integration of multiple kinds of foresight deserves attention alongside the ability to predict them.

Limitations & Future Work

• The authors do not provide a dedicated limitations section. This note's assessment is that self-distillation remains constrained by video-backbone prediction quality: trajectory consistency reduces supervision conflicts but does not guarantee physically correct imagined futures.
• Each real-world task has only 25 evaluation trials, and the difficult transparent-object task reaches only 32%. The results support relative improvement rather than highly reliable deployment, and the presented results do not report confidence intervals.
• Action chunking increases effective control throughput without eliminating the 307.6 ms inference latency. Sudden disturbances, chunk interruption, and more frequent feedback merit further evaluation.
• Three-stage training depends on pretrained video and visual teacher models; these results do not establish large-scale cross-embodiment transfer or safety-critical manipulation capability.

Related Work & Insights

• vs VPP: both extract one-step predictive features from video models; S-VAM additionally uses same-trajectory teachers to decouple geometry and semantics before joint condensation, improving control at extra inference cost.
• vs Spatial Forcing: the latter emphasizes spatial alignment of current-observation representations, whereas S-VAM emphasizes future geometric and semantic foresight; comparisons should still account for training-data and backbone settings.
• vs video-planning methods such as SuSIE: complete video plans offer visible futures but require costly multi-step generation; S-VAM uses that process for training supervision and retains a representation-based foresight path online.
• Transferable direction: for other generative control tasks, check whether teacher targets are compatible with the student's internal prediction trajectory before increasing teacher capacity or adding distillation branches; this is a research implication, not a result verified in this paper.

Rating

• Novelty: 4/5. Same-trajectory self-distillation and complementary foresight form a clear contribution, while building on existing video policies and representation models.
• Experimental Thoroughness: 4/5. Two simulation benchmarks, component and teacher ablations, and real dual-arm tasks are covered, but real-world scale and statistical reporting remain limited.
• Writing Quality: 4/5. The method and ablation definitions are clear, although real-time claims require an action-chunk interpretation and corrupted cached equations limit exact reproduction.
• Value: 4/5. A useful representation-distillation route for expensive video planning, particularly for studying predictive representations in low-data robot control.