Skip to content

Understanding the Impact of Geometric Foundation Models on Vision-Language-Action Models

Conference: ECCV2026
Paper: ECCV
Area: Robotics & Embodied AI
Keywords: vision-language-action model, geometric foundation model, VGGT, linear probing, spatial understanding

TL;DR

Using GR00T-N1.5 and VGGT as a fixed pair, this paper quantifies for the first time the "geometric gap" between a VLA and a geometric foundation model (GFM) via linear probing, distills the disparate geometry-injection schemes in the literature into three controlled, comparable routes—Early Fusion, Late Fusion, and Spatial Forcing—and systematically scans non-architectural factors such as training-data size, camera count, and reconstruction quality: under task-level fine-tuning geometric VLAs are not significantly better, and only when training data is scaled up or the setup degrades to a single camera does the value of geometric information become visible.

Background & Motivation

Vision-language-action models (VLAs) have become the dominant paradigm for robot manipulation over the past two years: a vision-language model (VLM) grounds the robot's camera images and the user instruction ("pick up the pink water bottle") onto the relevant pixels, and an action expert then turns the robot embodiment plus the VLM's understanding into a chunk of actions. This "VLM + action expert" pairing is highly effective at semantic grounding and generalization, but it inherits a known weakness from VLMs: poor spatial understanding. Prior work has repeatedly observed that VLMs are unreliable at geometric reasoning such as estimating distances, relative pose, or object-placement constraints (SpatialVLM, SpatialBot, and others). Manipulation, however, is decided by geometry—a few centimeters of error separate a successful grasp from a failed one. This motivated the 3D VLA line, which feeds the policy depth from RGB-D cameras or point clouds (PointVLA, RVT-2, 3D-VLA), at the cost of extra sensors and extra data collection.

A second thread comes from computer vision itself. Geometric foundation models (GFMs) such as DUSt3R, MASt3R, and VGGT regress camera poses, depth maps, and dense point maps directly from multi-view images with a single feed-forward transformer, eliminating the traditional SLAM / SfM pipeline of feature extraction, matching, bundle adjustment, and dense reconstruction, and requiring no camera calibration. This is especially attractive for VLAs: VLAs already consume only RGB, so GFMs make it nearly free to add 3D information without new data collection or new hardware. A wave of recent work (Evo-0, VGGT-DP, Spatial Forcing) has therefore injected GFM tokens into VLAs and reported gains.

The trouble is that these works leave more questions open than they settle. First, nobody has quantified how much geometric understanding a VLA actually lacks—"VLAs lack spatial understanding" has remained an intuition, and if a VLA already has enough, injecting geometry is wasted effort. Second, the proposed injection architectures differ wildly (some modify the vision encoder output, some the VLM output, some change nothing structurally and only add a training loss), and their implementation details differ as well, so "which injection is better" cannot be compared fairly, nor can anyone say why. Third, the performance of geometric VLAs clearly depends on a set of non-architectural factors—how many cameras are used, how much training data is available, how accurately the GFM reconstructs the scene at hand—none of which has been systematically examined. This paper deliberately narrows the scope to one fixed combination (GR00T-N1.5 as the VLA, VGGT as the GFM) and runs a controlled experimental analysis within it. Core idea: first turn "how much geometry does the VLA lack" into a measurable number via linear probing, then reduce the injection schemes in the literature to three routes whose implementation details are aligned as closely as possible for an apples-to-apples comparison, and finally isolate and sweep the non-architectural variables—data size, camera count, reconstruction quality—to see under which conditions geometric information actually buys success rate.

Method

Overall Architecture

This is not a new model but a controlled experimental study of how geometry should enter a VLA, so its "method" consists of three reusable instruments: a ruler (linear probing), a set of controllable injection structures (three strategies plus one shared fusion module), and a strict evaluation protocol. The baseline VLA follows the standard GR00T-N1.5 form: the inputs are the image set \(I\) captured by the robot cameras at the current time, a language instruction \(L\), and robot state \(R\) (joint angles, etc.). These are turned by a vision encoder, a text tokenizer, and a state encoder into visual tokens \(V_e\), language tokens \(L_e\), and state tokens \(R\); \(V_e\) and \(L_e\) go through the LLM backbone inside the VLM to produce new visual tokens \(V_l\) and language tokens \(L_l\); those tokens, together with \(R\), are handed to the action expert, which predicts the action sequence \(a_{0:T}\) for the next \(T\) steps. The action expert uses a diffusion-policy-style flow-matching formulation: during training it learns to integrate random noise back into plausible actions, and at inference the predicted denoising field is integrated (Euler) to obtain actions (the exact loss expression is corrupted in the cached text, so it is not reconstructed here).

On this skeleton, the GFM (VGGT) consumes the same images \(I\) and emits a set of geometric tokens \(G\). Everything in this paper is about answering how \(G\) should enter the pipeline above, in what form, at what position, and under what conditions. The three injection routes form the controlled independent variable: Early Fusion fuses \(G\) with the vision-encoder output \(V_e\) before the LLM; Late Fusion fuses \(G\) with the VLM output \(V_l\) before the action expert; Spatial Forcing leaves the architecture untouched and only adds an alignment loss at training time that pulls the LLM's internal tokens toward \(G\). Linear probing, meanwhile, attaches a light MLP to a frozen network and reads out the vision-encoder output, the VLM output, VGGT itself, and the VLM output of the two fusion models, measuring how much depth information survives at each point. The whole pipeline is then evaluated on the RoboCasa and LIBERO simulation benchmarks and on a real Unitree G1 humanoid.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: multi-view RGB<br/>+ instruction + robot state"] --> B["GR00T-N1.5 base<br/>VLM + action expert"]
    A --> C["VGGT forward pass<br/>emits geometric tokens G"]
    B --> D["Linear probing of the geometric gap"]
    C --> D
    D -->|geometry is lost before the VLM output| E["Controlled implementation of three injection routes"]
    C --> E
    E --> F["Gated cross-attention fusion"]
    F --> G["Geometric VLA (Early / Late Fusion)"]
    E -->|Spatial Forcing: no structural change, alignment loss| G
    G --> H["Controlled evaluation and significance testing<br/>RoboCasa / LIBERO / Unitree G1"]

Key Designs

1. Linear probing of the geometric gap: turning "VLAs don't understand geometry" from intuition into numbers

Prior work argued that VLAs lack spatial understanding on the basis of qualitative observations or downstream success rates, which cannot answer "how much is missing". This paper turns that into a measurement with linear probing: freeze the network under test, attach a simple MLP to one of its layer outputs, and train only that MLP to solve monocular depth estimation—if the layer's features genuinely contain geometric information, the light module can read it out; if not, it cannot. The probe is trained for 10 epochs on NYU Depth V2 (24,000 training pairs, 645 validation pairs) with a scale-invariant logarithmic (SILog) loss, and is scored by RMSE depth error (lower is better) and the \(\delta_1\) score (the fraction of pixels predicted within 25% of ground-truth depth, higher is better). What makes the comparison meaningful is the denominator: probing the VGGT backbone itself gives the performance ceiling "when geometric information is plentiful", and the gap between the VLA-side probes and that ceiling is the geometric gap. The design also exposes a caveat—the probe attached to the vision-encoder output has a hidden dimension of 1024 versus 2048 for the VLM and VGGT probes, so the vision-encoder row is depressed by the smaller MLP capacity; the authors state this explicitly in a footnote, and the two rows must not be read as a strictly equal-capacity comparison.

2. Controlled implementations of three injection routes: making "which injection is better" comparable for the first time

Existing geometric VLA works each modify their own architecture with their own implementation details, so their conclusions are not comparable. This paper reduces the literature to three routes and reproduces them on one shared base with implementation details kept as aligned as possible. Early Fusion treats the GFM as an extra encoder: VGGT emits geometric tokens \(G\) from the same images \(I\), these are fused with the vision-encoder output \(V_e\) into \(\text{fuse}(V_e, G)\), and the fused tokens replace the original visual tokens fed to the LLM; the intuition is to let geometry participate in the representation before the language model, which is what early works such as Evo-0 do. Late Fusion instead acts at the exit of the pipeline: \(V_l\) has already been processed by the LLM and is about to be handed to the action expert, and at that point \(G\) is fused with \(V_l\) into \(\text{fuse}(V_l, G)\), replacing the original tokens so that the VLM output is "topped up" with geometry right before the action expert sees it. The third route, Spatial Forcing, changes no architecture at all: at training time it adds an alignment loss that pulls the LLM's internal tokens toward the GFM tokens by cosine similarity, so the VLM retains geometric information in its internal representation during fine-tuning. Training supervision, sensor inputs, and the action expert are identical across the three, leaving "at which step geometry enters" as the only difference—that is the precondition for the paper's comparable conclusions. The authors also state plainly that no open-source code existed for Early and Late Fusion at the time of writing, so both are their own prototypes on GR00T-N1.5, whereas Spatial Forcing does have open-source code but based on OpenVLA and π0, so it too was ported to GR00T to equalize the comparison conditions.

3. Gated cross-attention fusion: keeping geometric tokens from washing out a pretrained action expert

Early and Late Fusion share one fusion module \(\text{fuse}(\cdot,\cdot)\), and the difficulty is not attention itself but the fact that the entire downstream architecture was pretrained without geometric tokens, so an abrupt injection leaves the action expert no time to adapt. Concretely, queries, keys, and values are projected from the VLA tokens \(X\) (with \(X = V_e\) for Early Fusion and \(X = V_l\) for Late Fusion) and the geometric tokens \(G\); standard cross-attention is computed and projected back into the VLA feature dimension, so that the fused tokens keep exactly the original size and the downstream interface is untouched. A learnable attention gate \(A\) then weights the cross-attention output element-wise, and the gated result is added back to the original tokens as a residual correction:

\[\tilde{X} = X + A \odot W_O\left(\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V\right)\]

(reconstructed from the prose; ⚠️ refer to the original paper for exact notation.) The gate \(A\) is initialized close to zero, so at the start of training the model is essentially the original VLA and geometric information seeps in gradually. The authors treat this as the crux of the strategy: ablations show the strategy essentially fails without the attention gate, and Early Fusion in particular depends on it. In addition, the projection matrices \(W_Q, W_K, W_V, W_O\) are parameterized as low-rank (LoRA) layers, and positional encodings are added to both the VLA tokens and the GFM tokens so that the spatial arrangement of the corresponding tokens is preserved.

4. Controlled evaluation and significance testing: separating random fluctuation from real improvement

Diffusion sampling in the action expert is itself stochastic: even with the scenario-generation seed fixed, the success rate of the same model varies across evaluation runs, so "this geometric VLA is 2 points above baseline" may mean nothing. The paper counters with three measures: fix the random seeds and the noise generation of the diffusion transformer, without any attempt at tuning the seed for performance; evaluate every model at checkpoints 1, 5, 10, 15, 18, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100 and report the best success rate, avoiding a lucky sampling run; and report a two-sided McNemar's test p value for every comparison, treating differences with \(p \ge 0.05\) as not significant. The p values in the "Average" column are not an average of the per-task p values but are recomputed from the success counts across all tasks. This design is what allows the paper to state a counter-intuitive result: after task-level fine-tuning, none of the three geometric VLAs is significantly better than the GR00T baseline in average success rate.

Loss & Training

The training configurations of the three model families are deliberately leveled. The GR00T-N1.5 baseline follows the standard protocol and trains only the action expert from pretrained weights; Early and Late Fusion train the fusion module plus the action expert with everything else frozen; Spatial Forcing applies the alignment loss to layer 9 of the 13-layer GR00T LLM and fine-tunes only the linear projection between the vision encoder and the LLM (the authors also tried fine-tuning the LLM under Spatial Forcing and got worse results, see the supplementary material). Probing is the opposite: the network under test stays fully frozen at its original pretrained weights and only the probe MLP is trained. All models are fine-tuned per task on RoboCasa with roughly 300 demonstrations per task for 100 epochs on a 320 GB A100 cluster, taking 2-3 days; the mid-training in Section 5.3 first continues training on the entire RoboCasa dataset (all 8 tasks) before task-level fine-tuning and takes about one week.

Key Experimental Results

Main Results

Evaluation covers two simulation benchmarks and one real-robot benchmark. RoboCasa uses 8 pick-and-place tasks with 5 episodes each (a different kitchen scenario and a different object to grasp), 15 trials per episode with randomized scene appearance, for 600 manipulation experiments in total, using left, right, and wrist cameras. LIBERO contains the LIBERO-SPATIAL / OBJECT / GOAL / 100 suites, with 500 trials per task. The real benchmark uses a Unitree G1 humanoid on three object classes (bottle, ball, box) with 90 tests, randomizing object location, orientation, and the specific instance within each class.

Method CabToCtr CtrToCab CtrToMicrowave CtrToSink CtrToStove MicrowaveToCtr SinkToCtr StoveToCtr Average
GR00T-N1.5 (baseline) 42.7 74.7 73.3 93.3 77.3 58.7 65.3 88.0 71.7
Early Fusion 32.0 (p=0.186) 69.3 (p=0.289) 65.3 (p=0.377) 88.0 (p=0.388) 73.3 (p=0.664) 62.7 (p=0.742) 80.0 (p=0.019) 86.7 (p=1.000) 69.7 (p=0.399)
Late Fusion 46.7 (p=0.710) 72.0 (p=0.625) 74.7 (p=1.000) 85.3 (p=0.146) 69.3 (p=0.307) 69.3 (p=0.115) 69.3 (p=0.689) 81.3 (p=0.267) 71.0 (p=0.806)
Spatial Forcing 29.3 (p=0.123) 68.0 (p=0.227) 66.7 (p=0.499) 76.0 (p<0.001) 72.0 (p=0.454) 60.0 (p=1.000) 84.0 (p=0.007) 90.7 (p=0.804) 68.3 (p=0.154)
Early Fusion (mid-trained) 52.0 (p=0.281) 72.0 (p=0.727) 69.3 (p=0.700) 94.7 (p=1.000) 80.0 (p=0.815) 68.0 (p=0.189) 81.3 (p=0.023) 84.0 (p=0.607) 75.2 (p=0.104)

All p values are computed against the GR00T-N1.5 baseline with a two-sided McNemar's test. For reference, the literature methods not shown in the table have markedly lower averages on RoboCasa (π0 33.8, π0-Fast 48.3, Video Policy 47.3, RS-CL 59.0), indicating that the carefully reproduced GR00T-N1.5 baseline is already far stronger than earlier methods.

Method Approach Grasp Lift Placement Overall
GR00T-N1.5 57.78 51.92 85.19 86.96 22.22
Early Fusion 84.44 (p<0.001) 60.53 (p=0.824) 89.13 (p=1.000) 65.85 (p=0.180) 27.78 (p=0.511)
Late Fusion 57.78 (p=0.855) 59.62 (p=1.000) 93.55 (p=1.000) 79.31 (p=0.625) 25.56 (p=0.710)

The Unitree G1 real-robot results are broken down by grasping stage: Approach succeeds when the robot approaches and touches the target, Grasp when it secures the object, Lift when the object is correctly lifted off the table, and Placement when it is correctly placed back on the table; an episode counts as successful only if all four stages succeed.

Ablation Study

The first ablation is linear probing, which measures the amount of geometric information at each stage (the VGGT row is the ceiling reference):

Probe location RMSE [m] (↓) δ1 (↑)
GR00T vision-encoder output 0.92 0.51
GR00T VLM output 0.73 0.63
VGGT backbone 0.41 0.89
Early Fusion (VLM output) 0.44 0.88
Late Fusion (VLM output) 0.45 0.87

The second ablation drops to a single camera (left only), testing whether the multi-view geometry a VLA can see for itself is what carries its success rate:

Config Average success rate p vs single-camera baseline
GR00T-N1.5 (single camera, no mid-training) 17.2 —
Early Fusion (single camera, no mid-training) 21.5 p=0.030

A third analysis links VGGT reconstruction quality to success rate: for each of the 8 tasks × 5 episodes, the average depth RMSE of VGGT across all images and cameras in that episode is compared with the success rate, giving a Spearman correlation of \(\rho = -0.202\).

Key Findings

  • Under task-level fine-tuning, geometric VLAs deliver no statistically significant gain in success rate. The three injection routes average 69.7 / 71.0 / 68.3 on RoboCasa, all below the baseline's 71.7, but with p values of 0.399 / 0.806 / 0.154, so the differences are not significant; the authors conclude these fluctuations more likely stem from randomness in the action expert. ⚠️ Note the prose discussing this table writes "69.7% vs. 71.2% for the baseline", which disagrees with the baseline's 71.7 in the table; the table value is taken as authoritative here.
  • On the real robot, geometry helps more, and more locally. Early Fusion raises the Approach stage from 57.78% to 84.44% (p<0.001, the only strongly significant gain in the paper), and its overall success rate is 27.78% vs. the baseline's 22.22% but not significant (p=0.511); Grasp and Lift also improve slightly while Placement degrades. The authors conjecture that the VLM (and in particular the LLM inside it) is more plastic in processing new information, whereas the action expert is relatively "rigid" and cannot make good use of additional data sources—consistent with the plasticity observed in the probing experiment. In practice Early Fusion reliably grasps small objects such as the small ball and small box that the baseline struggles with, while the gain of Late Fusion is much more modest.
  • Scaling the training data is what makes geometry pay off. After mid-training on the full RoboCasa dataset (all 8 tasks) followed by task-level fine-tuning, Early Fusion rises to 75.2 average, overtaking the 71.7 baseline (p=0.104), with SinkToCtr reaching 81.3 (p=0.023); the supplementary material further shows that Early Fusion still beats GR00T when both are mid-trained with the same protocol, ruling out information leakage from mid-training itself. This matters because scaling training requires no additional data collection, making it a practically viable route.
  • The fewer the sensors, the more the geometric model is worth. With a single camera all models drop sharply (17.2%), confirming that multi-view input is essential for grasping; but Early Fusion's advantage over the baseline widens (21.5% vs. 17.2%) and becomes significant for the first time (p=0.030), showing that when the VLA cannot triangulate geometry from multiple views itself, the GFM supplies something irreplaceable.
  • Probing provides quantitative evidence of the gap and localizes where the information is lost. VGGT features give a depth RMSE of 0.41, the GR00T VLM output 0.73 (nearly double), and the vision-encoder output already 0.92—geometric information is gone before it ever reaches the LLM, so the LLM is not the one losing it. Interestingly, after geometry injection the probes nearly match VGGT itself (Early Fusion 0.44, Late Fusion 0.45), meaning Early Fusion can bring out dense depth prediction capability even with the LLM completely frozen, which is non-trivial.
  • There is a mild negative correlation between VGGT reconstruction quality and success rate (\(\rho = -0.202\); lower RMSE corresponds to higher success). The correlation is weak, indicating success is dominated by other factors such as task complexity, but the trend suggests that fine-tuning VGGT in the target environment may be a viable way to squeeze a little more performance out of a geometric VLA.

Highlights & Insights

  • Turning the "geometric gap" into a scalar with probing: instead of arguing about whether VLAs understand geometry, freeze the network, attach an MLP, and have it solve dense depth prediction to read off how much geometric information each layer retains. This converts a vague qualitative judgment into a reproducible measurement and incidentally localizes the loss to after the vision encoder, which informs where injection should happen.
  • The attention gate with near-zero initialization is the hidden precondition of this route: both Early and Late Fusion are structurally plain, and what decides success is initializing the gate close to zero so geometric tokens enter gradually into an already-pretrained architecture. The authors state explicitly that this was not mentioned in prior work, and it is a lesson learned from failure—findings where "the implementation detail is the method" often transfer better than the architecture itself.
  • The single-camera experiment is an elegant reverse validation: if the gain of a geometric VLA comes from VGGT's supplementation, then the gain should grow when the VLA itself loses multi-view geometry—and that is exactly what happens, becoming significant for the first time. Using a setting that deliberately weakens the baseline to demonstrate a method's value is a design pattern transferable to any research that "supplies a missing kind of information".
  • Using significance testing against diffusion-policy randomness: fixed seeds, best-of-many-checkpoints, and two-sided McNemar's tests together keep noise-level conclusions such as "2 points better" out. That is why the paper can state a conclusion contrary to the prevailing narrative—that none of the three geometric VLAs is significantly better—and be more credible for it; the rigor itself is worth copying.

Limitations & Future Work

  • The authors admit the conclusions are tied to the GR00T-N1.5 + VGGT combination. Other VLAs and other GFMs (MASt3R, the Depth Anything family, MapAnything) are architecturally similar but cannot be extrapolated to; whether the conclusions hold on another base remains open.
  • Most evidence comes from simulation benchmarks, and some are close to saturated, which compresses the headroom for improvement; the real-robot study covers only one platform (G1), three object classes, and 90 tests, so Early Fusion's significant advantage on Approach is currently a single data point.
  • The design space is not exhausted: which layer Spatial Forcing should be applied to, whether non-cross-attention fusion exists, and other gate parameterizations were not ablated, because single training runs can take close to a week and the cost is prohibitive.
  • A small self-consistency blemish: the prose summarizing the RoboCasa main table writes the baseline average as 71.2% while the table says 71.7; and in the real-robot table Placement drops from 86.96 to 65.85, which the authors describe as only "slightly worse"—the wording does not match the magnitude.
  • Natural next steps: re-measure success rate after fine-tuning VGGT in the target environment (the paper only offers correlational evidence); scale mid-training further across robot datasets; and test whether other GFMs produce the same single-camera gain.
  • vs Spatial Forcing: it changes no architecture and only aligns the LLM's internal tokens with geometric features at training time, making it the lightest to engineer; after porting it to GR00T, this paper finds its 68.3 average on RoboCasa is the lowest of the three routes, yet individual tasks shift significantly (CtrToSink p<0.001, SinkToCtr p=0.007) in inconsistent directions, showing the alignment loss is highly sensitive to layer and task. The paper's value is placing this route in the same controlled table.
  • vs 3D VLAs (PointVLA, RVT-2, 3D-VLA): they feed explicit geometry from RGB-D cameras or point clouds, which is genuinely geometric and accurate, at the cost of extra sensors and the corresponding data collection. The GFM route studied here uses RGB only with zero extra hardware, making it suitable for large-scale training on existing RGB data; its weakness is dependence on GFM generalization quality, which the weak \(\rho = -0.202\) correlation reflects.
  • vs early-fusion pioneers (Evo-0 and others): they proposed fusing GFM tokens with visual tokens but left implementation details unspecified (whether to add an attention gate, how to initialize it, how to add positional encodings). This paper reproduces the route and shows it fails without the attention gate, separating "the architectural idea" from "the implementation conditions that make it work"—which is the paper's most practical contribution relative to existing work.

Rating

  • Novelty: ⭐⭐⭐ [no new model; the value lies in first quantifying the geometric gap and placing three injection routes under one controlled comparison]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [two simulation benchmarks plus the real G1, covering probing, fusion strategy, data scale, camera count, and reconstruction quality, all with significance testing]
  • Writing Quality: ⭐⭐⭐⭐ [the problem is decomposed clearly and the claims are restrained, but a few numbers and wordings are inconsistent (71.2 vs. 71.7, the "slightly worse" Placement)]
  • Value: ⭐⭐⭐⭐ [delivers practical conclusions—scaling training data beats swapping architectures, and geometry matters more with fewer cameras—that directly inform decisions for anyone building geometric VLAs]