GEM: Generative Supervision Helps Embodied Intelligence

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Project: https://zhaorw02.github.io/GEM/
Area: Robotics & Embodied AI
Keywords: Depth generative supervision, vision-language-action models, spatial reasoning, progressive training, embodied grounding

TL;DR

GEM adds conditional depth generation to embodied vision-language model training and progressively aligns its components to encode geometry in shared representations, raising the 8B model's VSI-Bench score from 68.6 with same-data SFT to 70.6, while its GEM-VLA extension reaches a 96.1% average success rate on LIBERO.

Background & Motivation

Vision-language models (VLMs) can identify objects and describe task steps without necessarily retaining the fine geometry needed to execute actions. Question-answering supervision only requires correct text. Unless object boundaries, relative distances, or available placement space affect that answer, the model may have little incentive to encode them fully in its visual tokens. Better semantic question answering therefore does not automatically make a robot more accurate at placement or manipulation.

Existing spatially enhanced vision-language-action models (VLAs) introduce depth, point clouds, or additional spatial features, but these signals often enter only during action learning. GEM addresses an earlier bottleneck: can embodied vision-language training already produce representations that provide the eventual action expert with both semantics and geometry? Rather than simply showing the model another depth image, it requires existing visual hidden states to support depth reconstruction, making missing structural information directly observable as training error.

Depth fits this goal more closely than RGB reconstruction, which also involves color and texture and need not imply accurate distance understanding. The paper also builds embodied question-answering data and initializes the new generative branch progressively to avoid disrupting the language backbone. Core idea: make the same visual representations support both embodied question answering and depth generation, then retain this geometric constraint during action learning instead of adding spatial information only at the action output stage.

Method

Overall Architecture

GEM takes images and language instructions and can output embodied question-answering responses; its GEM-VLA extension also outputs continuous action chunks. Training starts with GEM-4M data that supplies task semantics and depth targets, then trains the backbone through depth generative supervision and progressive alignment, before adding an action expert while retaining depth supervision.

The depth branch is a training constraint, not a serial planner that must first produce depth-related text and then invoke an action model. Language answers use the backbone's multimodal representations. The action expert reads key–value tokens from backbone attention, whereas the depth generator reads final-layer visual tokens: the two branches access different representations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["GEM-4M Data Pairing"] --> B["Depth Generative Supervision"]
    B --> C["Progressive Alignment"]
    C --> D["Embodied Vision-Language Representations"]
    D -->|Question answering| E["Text Answers"]
    D --> F["Action Transfer and Joint Supervision"]
    F --> G["Continuous Action Chunks"]

Key Designs

1. GEM-4M Data Pairing: expose structural supervision and task semantics to the same scene

GEM-4M is not a single depth-estimation dataset. It mixes question-answering data for embodied grounding, physical and spatial reasoning, and spatiotemporal planning. Its grounding component collects approximately 1M question-answer pairs covering open-vocabulary detection, instruction-based pointing, and affordance recognition, with approximately 100k additional point and bounding-box annotations generated from robot action data. Point and box coordinates are normalized to 0–1000 so that the same spatial location does not require different output scales at different image resolutions. Language supervision must therefore ground descriptions in concrete locations rather than merely describe scenes.

The reasoning component combines MindCube, ViCA, SPAR, VSI-590K, and other sources, adds approximately 100k manually annotated spatial-understanding samples, and aggregates approximately 1M robot-related question-answer pairs. For planning, frames are extracted from robot videos with subtask annotations: Qwen3 identifies manipulated objects, SAM3 supplies masks, and CoTracker3 tracks trajectories. Subtask descriptions and visualized trajectories then support approximately 50K planning question-answer pairs. Samples without ground-truth depth receive pseudo-depth from DepthAnythingv3, so supervision does not come exclusively from physical depth sensors. These figures describe source-reported subsets and should not simply be added to infer an exact deduplicated total.

2. Depth Generative Supervision: make visual tokens accountable for scene geometry

The backbone is Qwen3-VL, the depth generator is Sana, and a two-layer MLP connector bridges them. After the backbone processes images and instructions, the connector maps final-layer visual tokens into the generator's conditioning space. Sana's diffusion Transformer then learns to recover the depth map of the current observation from noisy states using this conditioning. The target is current scene structure, not future RGB frames, and depth maps are not inserted as additional language tokens.

The depth branch uses flow matching: at a given noise level, the generator learns the velocity field that moves a noisy sample toward the target depth. Crucially, joint training propagates this error through the connector into the VLM backbone, requiring it to retain cues needed for boundary and distance reconstruction. This adds a dense geometric constraint beyond answering questions correctly. Compared with RGB reconstruction, it reduces the auxiliary task's emphasis on texture and color. It nevertheless encourages geometric encoding rather than imposing a hard guarantee of physical consistency or action feasibility.

3. Progressive Alignment: establish a usable generative branch before changing the shared backbone

If the pretrained language backbone and generator are unfrozen together before the connector is aligned, generative error may primarily reflect an incompatible interface rather than missing geometric information in the backbone. GEM therefore first freezes both the backbone and generator and trains only the connector. It then keeps the backbone frozen while training the connector and generator to adapt to the new visual conditioning. Only the final stage jointly trains all three components. The first two stages use only depth generation; the third combines text cross-entropy with depth flow matching.

This order separates an unaligned conditioning space from a genuine need to improve shared representations. The connector is trained for 500 steps, the generative branch warms up for 4k steps, and joint training lasts 1 epoch. Components are not individually trained to final convergence and permanently frozen. The source ablation also shows that direct joint training without this initialization sequence is substantially weaker than the default, indicating that merely attaching a generative task does not reliably improve understanding.

4. Action Transfer and Joint Supervision: preserve learned geometry during action optimization

GEM-VLA adds a diffusion-Transformer action expert that reads key–value tokens representing multimodal observation history from the backbone's attention modules and generates continuous action chunks through a diffusion policy. Question-answering text is not the sole action interface, nor does the depth generator directly produce control signals. The action and depth branches share a backbone but have separate prediction targets.

During action fine-tuning, the backbone, depth branch, and action expert remain jointly optimized. Action flow matching moves noisy action states toward demonstrated action chunks, while depth flow matching maintains structural constraints on the representations. This uses previously learned spatial knowledge without discarding the geometric auxiliary objective during action-data optimization. Real-robot experiments use the RDT2 action expert. LIBERO experiments instead follow StarVLA and learn action prediction starting from GEM-2B; these are not the same training-budget setting.

A Worked Example

Consider table bussing from the paper's real-world experiments. A robot receives a tabletop image and an instruction to arrange the table and must place objects according to their locations. The backbone first forms shared visual and language representations. During training, the same observation supports task supervision and conditions depth generation, preventing the spatial relationships among a fork, a plate, and the table from being reduced to object names alone.

After embodied training, the action expert reads the corresponding key–value tokens and generates the next continuous action chunk, while depth supervision remains active during action fine-tuning. This explanation does not imply a runtime sequence of explicitly generating depth maps and then planning cutlery positions, and it does not assume an action-chunk length that the paper does not report.

Loss & Training

Vision-language joint training and action fine-tuning use different main objectives but share the same depth auxiliary term:

\[ \mathcal{L}_{\mathrm{VLM}}=\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{L}_{\mathrm{flow}},\qquad \mathcal{L}_{\mathrm{VLA}}=\mathcal{L}_{\mathrm{action}}+\lambda\mathcal{L}_{\mathrm{flow}},\qquad \lambda=0.1. \]

Cross-entropy supervises text answers, depth flow matching supervises depth generation, and action flow matching supervises demonstrated action chunks. The combinations above are explicitly described in the paper. The local PDF text extraction does not preserve the complete velocity-field equations, so their exact expectation terms are not reconstructed here.

Embodied vision-language training uses 32 NVIDIA A800 GPUs and a cosine learning-rate schedule from \(10^{-5}\) to \(10^{-6}\). Real-robot task-specific fine-tuning lasts 50k steps on 8 A800 GPUs with a linear schedule and a learning rate of \(10^{-5}\); LIBERO action training lasts 20k steps. Real-world tasks are evaluated after fine-tuning, not as demonstration-free zero-shot manipulation.

Key Experimental Results

Main Results

The following results come from Tables 1–3. Understanding tasks report each benchmark's All score, with higher being better. Differences are direct subtractions on the reported scale, not relative percentage improvements. SFT is the control trained on the authors' constructed data without depth generative supervision, making it more informative for isolating the auxiliary objective's contribution.

Task and Setting	Metric ↑	Baseline	Baseline Score	GEM / GEM-VLA	Difference
VSI-Bench, 8B	All score	Qwen3-VL-8B original backbone	57.9	70.6	+12.7
VSI-Bench, 8B	All score	Qwen3-VL-8B-SFT	68.6	70.6	+2.0
MMSI-Bench, 8B	All score	Qwen3-VL-8B-SFT	32.8	35.3	+2.5
RefSpatial, 8B	All score	Qwen3-VL-8B-SFT	45.8	44.4	-1.4
Where2Place, 8B	All score	Qwen3-VL-8B-SFT	62.0	65.0	+3.0
LIBERO	Average success rate (%)	Qwen3VL-SFT-VLA	94.9	96.1	+1.2
LIBERO	Average success rate (%)	DepthVLA	94.9	96.1	+1.2

LIBERO contains four suites—Spatial, Object, Goal, and Long—with 10 tasks per suite and 50 trials per task. GEM-VLA achieves respective suite success rates of 99.0%, 98.8%, 97.1%, and 89.3%. Comparisons with prior methods involve different pretraining data and training recipes; some entries in the source table are previously reported results.

Real-world experiments use a UR5 for cloth folding, table bussing, and unzipping. Figure 3 labels All Task Average as 43.0%, versus 28.7% for π0.5, a difference of 14.3 percentage points. However, table bussing uses a progress score defined as the average fraction of completed subtasks rather than binary whole-task success, and the main paper does not fully explain the aggregate calculation. This number is therefore best treated as an author-reported composite result, not a success probability with a uniform definition across all tasks.

Ablation Study

All three configurations in Table 4 are trained on VSI-590K, with other hyperparameters and strategies controlled as described for those experiments. They are not the same runs as full GEM-4M training above. Depth versus RGB reconstruction and progressive initialization versus direct joint training should be compared within this table.

Config	CV-Bench All ↑	VSI-Bench Absolute Distance ↑	VSI-Bench Relative Distance ↑	VSI-Bench All ↑	RoboSpatial All ↑
RGB reconstruction supervision	80.9	47.5	62.8	60.0	44.6
Direct end-to-end joint training	79.7	42.1	60.0	57.6	44.0
Default GEM: depth supervision and progressive training	81.1	47.8	65.2	63.0	48.9

Key Findings

• Depth supervision improves VSI-Bench All by 3.0 points and RoboSpatial by 4.3 points over RGB reconstruction. Absolute-distance performance improves by only 0.3 points, so not all distance metrics show large gains.
• Progressive training improves VSI-Bench All by 5.4 points over direct joint training, demonstrating the importance of initialization in this controlled experiment. It does not establish a global ranking of contributions from data construction, depth supervision, and training order.
• GEM-8B scores 44.4 on RefSpatial, below same-data SFT's 45.8. Its MMSI-Bench score of 35.3 also trails the table's Gemini-3-Pro result of 45.9. Claims of overall superiority should therefore not be expanded into leadership on every benchmark.

Highlights & Insights

• Generation serves representation learning rather than final image output. Depth reconstruction turns structural information easily overlooked by a text objective into supervised error, improving backbone features used for both question answering and actions.
• Controlled experiments separate data effects from auxiliary supervision. Differences from same-data SFT are more informative about the incremental value of depth supervision than large gains over the original backbone, and they also expose counterexamples in grounding.
• Geometric supervision spans two training phases. Depth is used during both embodied vision-language training and action fine-tuning, avoiding a complete separation of representation and action learning. This does not replace real manipulation evaluation.

Limitations & Future Work

• Pseudo-depth depends on DepthAnythingv3 and can import its errors into supervision. The main paper does not quantify downstream sensitivity to pseudo-depth quality, so the generative target should not be equated with ground-truth geometry.
• Real-robot evaluation covers three tasks on one UR5 platform and requires task-specific fine-tuning. The main paper provides insufficient trial counts, confidence intervals, and cross-platform statistics to establish broad deployment reliability.
• Some claims of global leadership exceed the support of individual table entries, and Figure 3's aggregate mixes success and progress measures. Explicit per-task results and matched controls should take priority when interpreting the evidence.
• Data-construction details and some implementation information are deferred to supplementary material absent from the local cache. In particular, missing depth-generation cost and action-latency analyses prevent an evidence-based assessment of deployment efficiency.

Related Work & Insights

• vs Qwen3-VL-SFT: Both use embodied question-answering supervision, but GEM additionally requires shared representations to support depth generation. This control shows that data alone does not explain every improvement and that an auxiliary objective need not benefit every benchmark.
• vs SpatialVLA / DepthVLA: These methods also emphasize spatial information for actions. GEM first strengthens the embodied VLM backbone and then retains depth supervision during action learning, rather than focusing exclusively on spatial enhancement inside the action model.
• vs RGB reconstruction and future-frame generation: GEM directly constrains the depth structure of the current observation instead of centering appearance reconstruction or future-world prediction. Its evidence supports this choice on the evaluated spatial tasks, not a universal rejection of other generative auxiliary objectives.

Rating

• Novelty: 4/5. Depth generation, shared representations, and action transfer form a coherent combination, although the individual components are not entirely new.
• Experimental Thoroughness: 3/5. Understanding, grounding, simulation, and real manipulation are covered with controls, but real-world statistics and some evaluation definitions remain incomplete.
• Writing Quality: 3/5. The methodological narrative is clear, but superiority claims, the action-expert figure caption, and some reference labels require scrutiny.
• Value: 4/5. The work offers a useful training recipe and empirical evidence for assessing whether embodied models retain geometry that can support execution.