Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics

Conference: ECCV 2026
arXiv: 2512.13660
Code: https://zhoues.github.io/RoboTracer (Project Page)
Area: Multimodal VLM / Embodied AI / LLM Reasoning
Keywords: Spatial Trace, Spatial Reasoning, 3D Vision-Language Models, Metric Perception, Reinforcement Fine-Tuning

TL;DR

This work proposes RoboTracer, a 3D-aware VLM that learns "3D spatial referring + metric measurement" via a regression-supervised scale decoder and a universal spatial encoder with plug-and-play geometric inputs. It is then elevated to multi-step, metric-grounded spatial trace reasoning using GRPO reinforcement fine-tuning with metric-sensitive process rewards. RoboTracer outperforms Gemini-2.5-Pro by 36 percentage points on the self-built TraceSpatial-Bench and can directly interface with motion planning to drive real robots such as the UR5 and the G1 humanoid.

Background & Motivation

Robots are increasingly receiving spatially constrained instructions such as "hover the watering can 1–5 cm above each flower and water them one by one from left to right." To execute this, a robot must infer a sequence of ordered 3D waypoints in a 3D scene—which this paper defines as a spatial trace—serving as an intermediate bridge between "understanding instructions" and "generating actions." However, this task is fundamentally challenging: each step requires both 3D spatial referring (precisely locating objects in a cluttered scene based on relationships like "the first flower from the left") and 3D spatial measurement (retrieving the real physical height of objects and absolute metrics like "1–5 cm above"), and then stringing these clues into multi-step reasoning. Recently, data-scarce VLA models that directly predict end-to-end 6D dense actions largely fail on such tasks.

Existing VLMs are also inadequate. While they can perform 2D spatial reasoning and even generate 2D visual traces (sequences of points on the image plane), they generally ignore the multi-step nature of the tasks. Specifically, intermediate objects that play critical roles along the trace are not explicitly supervised, leading to compromised generation quality. A more fundamental gap is that these outputs remain in 2D space, lacking 3D grounding and absolute metric understanding, which creates a barrier between 2D visual traces and actual 3D spatial traces. Directly fine-tuning off-the-shelf VLMs also runs into two walls: first, the lack of absolute scale supervision (especially when only RGB is available, the model has no concept of how long "one meter" is); second, existing absolute scale geometric cues like camera intrinsic parameters and absolute depth are not utilized.

The key insight of this paper is to decouple the problem into two stages: "learning perception first, then learning reasoning," while injecting metric awareness into all four stages of input, output, supervision, and training. During the SFT phase, the model is first equipped with precise 3D referring and measurement capabilities (relying on a dedicated regression-supervised scale decoder and a universal spatial encoder that can handle arbitrary geometric inputs). During the RFT phase, a set of metric-sensitive process rewards is utilized to supervise the key perceptual steps within the reasoning chain (which object to refer to and how many meters to measure at this step), organizing the perceptual capabilities into multi-step, metric-grounded reasoning. Core Idea: Represent spatial traces as decoupled \((u,v,d)\) point sequences, lay a solid foundation for 3D metric perception through SFT using "regression-supervised scale + plug-and-play geometric encoding", and then perform GRPO reinforcement fine-tuning with metric-sensitive process rewards supervising key perceptual steps to enable the VLM to learn multi-step, metric-grounded spatial trace reasoning.

Method

Overall Architecture

RoboTracer aims to solve the following problem: given an RGB image (optionally with geometric cues such as camera intrinsics and depth) and a spatially constrained instruction, output an ordered sequence of 3D sparse waypoints \(\tau=\{p_t\}_{t=1}^{T}\) (typically 6–12 points) to complete the instruction. Each waypoint is expressed as \(p_t=(u_t,v_t,d_t)\)—representing image plane coordinates paired with the corresponding absolute depth, rather than directly using \((x,y,z)\). This formulation allows trivial projection to 3D given camera intrinsics, eliminating the need for the VLM to implicitly learn camera geometry, which simplifies training and improves accuracy. This decoupled representation is also naturally dimension-reducible and reusable: discarding \(d\) degrades it to a 2D visual trace, while keeping only the start and end points degrades it to 3D/2D spatial referring data, allowing co-training with existing 2D datasets.

The entire pipeline consists of two main stages. Stage 1 (SFT): On top of a standard VLM (RGB encoder + LLM), two new components are integrated—a scale decoder that maps the <SCALE> token to a numerical scale factor, and a universal spatial encoder capable of flexibly accepting camera intrinsics, poses, and depth. Both components are aligned with the LLM via their respective projectors. The SFT process is decoupled into two sub-steps: "metric alignment" and "metric enhancement". First, only the projector and scale decoder are updated; then, the spatial encoder is frozen while the rest of the model is fine-tuned. This stage enables precise 3D referring and measurement and provides a "cold start" for the next stage using the multi-step reasoning processes embedded in the data. Stage 2 (RFT): GRPO is employed to perform reinforcement fine-tuning on multi-step reasoning data. The model reasons step-by-step in the format of <think>Step1…Step7</think><answer>…</answer>, explicitly declaring "[perception type] [target object]" at each step (e.g., "[Referring] [watering can]", "[Measuring] [the first flower from left] 0.195m") before producing the final trace, supervised by a set of metric-sensitive rewards (both outcome-level and process-level).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: RGB + Instruction<br/>(Optional Intrinsics/Depth)"] --> B["Decoupled (u,v,d) Representation<br/>Image Coordinates + Absolute Depth"]
    B --> C["Scale Decoder<br/>Regression-Supervised Scale Factor"]
    B --> D["Universal Spatial Encoder<br/>Plug-and-play Geometric Input"]
    C --> E["Two-step SFT: Metric Alignment → Metric Enhancement<br/>Learn 3D Referring + Measurement"]
    D --> E
    E --> F["GRPO + Metric-Sensitive Process Reward<br/>Supervise Key Perceptual Steps"]
    F --> G["Output: Multi-step Reasoning Chain<br/>+ 3D Spatial Trace τ"]
    G -->|Interface with Motion Planning| H["Robot Execution<br/>UR5 / G1 Humanoid"]

Key Designs

1. Decoupled \((u,v,d)\) trace representation: Unloading the burden of "learning camera geometry" from the VLM

The limitation is straightforward: if the VLM is forced to directly output \((x,y,z)\) world coordinates, it must implicitly internalize both camera intrinsics and extrinsics, leading to harder training and lower accuracy. This paper instead predicts image plane coordinates plus absolute depth, \(p_t=(u_t,v_t,d_t)\), because with intrinsics available, \((u,v,d)\to(x,y,z)\) is a trivial projection, sparing the VLM from geometric transformations. The beauty of this representation lies in its reusability: discarding \(d\) yields a 2D visual trace, while keeping only the start and end points gives 3D/2D referring samples, allowing seamless alignment with existing 2D datasets for co-training, which in turn boosts multi-task performance. Ablation results (Table 9 ID G vs H) show that modeling \((u,v,d)\) outperforms direct \((x,y,z)\) modeling on TraceSpatial-Bench (31% vs 30%) under the same training data, yielding a significantly lower Fréchet distance (0.1605 vs 0.2426). The authors attribute this to data reuse facilitated by dimension reduction and tighter alignment with 2D data.

2. Regression-supervised scale decoder: Forcing the model to develop an "absolute scale" sense from RGB-only inputs

Standard VLMs are typically pre-trained only on 2D data and have virtually no concept of absolute scales like "how long one meter is," making them particularly ineffective under RGB-only conditions. This work attaches a scale decoder to the LLM to map the embedding of a special <SCALE> token to a numerical scale factor, linking scale-invariant representations with absolute metric scales. Crucially, regression loss is used instead of classification/textual loss to supervise it, aligning the predicted scale with the ground truth in log space. The total SFT loss is:

\[\mathcal{L} = \mathcal{L}_{ntp} + 0.1\,\bigl\|\log(\hat{s})-\operatorname{stopgrad}(\log(s^{*}))\bigr\|_2^2\]

where \(\mathcal{L}_{ntp}\) is the next-token prediction loss, \(\hat{s}\) represents the predicted scale in log space, and \(s^{*}\) is the ground-truth scale (with stopgrad applied to the ground truth to restrict gradient backpropagation solely to the prediction branch). Ablation studies (Table 9 ID E/F/H) compare regression, next-token text supervision, and unsupervised approaches: regression performs the best, while pure text supervision offers only marginal gains. The reason is that standard next-token supervision requires massive amounts of data to cultivate a sense of scale, whereas explicitly regressing the scale factor—especially under mixed RGB/RGB+X training—forces the model to internalize scale information without relying on extra geometry.

3. Universal spatial encoder: Consuming whatever geometric cues are available, without altering model architecture during training/inference

In real-world embodied scenarios, absolute scale geometric cues like camera intrinsics, poses, and depth are often readily available, but standard VLMs cannot utilize them. This work constructs a plug-and-play spatial encoder built on a powerful feed-forward metric 3D geometric model: when extra geometry is available, it is fed in to refine spatial representations; the more geometric details provided, the more accurate the representation; if none are available, the model can still run on RGB alone. This provides two benefits: flexible training (leveraging various scale annotations in datasets for input augmentation to enrich spatial learning) and geometry-adaptive inference (using whatever geometric cues are available at inference time in a plug-and-play manner without retraining or architectural modification). Empirically, precise geometry yields up to an approximate 6% absolute improvement. Interestingly, ablation results (Table 9 ID D vs I) reveal that metric-grounded reasoning relies heavily on multi-step process training in RFT rather than the high-quality external 3D input provided by this spatial encoder during SFT—RFT without the spatial encoder is only slightly inferior to the version with it, still vastly outperforming SFT, and remains robust under noisy geometric inputs.

4. Metric-sensitive process rewards + GRPO: Looking beyond outcome accuracy to monitor perception at every intermediate step

This is the core of upgrading perceptual capabilities into multi-step reasoning. Following SFT, GRPO is first applied alongside several outcome-level rewards: format reward \(R_{OF}\) (enforcing structured outputs), point reward \(R_P\) (ensuring consistency of start and end points), and trace reward \(R_T\) (trajectory-level alignment). The point reward is computed as a truncated similarity of the start and end point distances:

\[R_P=\tfrac{1}{2}\bigl[f(p_1,\hat{p}_1)+f(p_T,\hat{p}_T)\bigr],\quad f(p,p')=\max\bigl(0,\,1-\|p-p'\|_2^2\bigr)\]

where all \((u,v,d)\) values are normalized to \([0,1]\) and depth is scaled according to the maximum depth of the scene. However, these outcome-level rewards are metric-agnostic and fail to monitor the critical intermediate perceptual steps (such as referring to the correct object or measuring the precise distance in meters). Therefore, this work introduces two process-level rewards leveraging the key-step perceptual annotations in TraceSpatial: a process format reward \(R_{PF}\) (forcing each step to be formatted as "[perception type] [target object]：") and an accuracy reward \(R_{Acc}\) (calculating prediction errors based on perception types only for steps that map to key-step annotations, such as using L1 distance for referring). \(R_{Acc}\) is designed to be order-invariant, allowing flexible step ordering. The final reward sums the outcome-level and process-level rewards, with the process-level component scaled by 0.25. Ablation studies show that adding process rewards boosts the overall success rate by an additional 4%, with its impact on 3D metrics being significantly larger than outcome-level rewards alone—proving that "supervising metric-grounded step-by-step perception" is key to generating accurate trajectories under complex spatial relations.

A Complete Example

Taking "watering flowers from left to right, hovering the watering can 1–5 cm above each flower" as an example, the 7-step inference after RFT proceeds as follows: Step 1 establishes the scene scale ([Scale][Scene] 2.406); Step 2 refers to the watering can (179,789,0.941); Step 3 measures its height as 0.104m; Step 4 refers to "the first flower from the left" (454,723,0.965); Step 5 measures its height as 0.195m; Step 6 refers to "the second flower from the left" (729,624,0.972); Step 7 measures its height as 0.403m... It step-by-step resolves the metric clues for the position and height of each flower as intermediate evidence, finally producing a complete spatial trace of 5 waypoints within the <answer> tag (e.g., (176,788,0.945)→…→(728,122,0.978)). This trace is then sent to a motion planner (incorporating collision avoidance, joint limits, and smoothing costs, with physical constraint refinement applied to the VLM-generated trace) to drive the robot to hover the watering can 1–5 cm above each flower sequentially.

Loss & Training

NVILA (2B/8B) is used as the base model, with RFT performed only on the 2B model due to GPU resource constraints. SFT is executed in two sub-steps: metric alignment (updating only the projector and scale decoder) \(\to\) metric enhancement (freezing the spatial encoder, fine-tuning the rest of the model, and training on both RGB-only and RGB+X inputs to preserve general VQA capabilities while adapting to any geometric configuration). The loss function is the next-token prediction loss plus the log-space scale regression term with a weight of 0.1, as formulated in Eq. (1). RFT uses GRPO, with the total reward being the sum of the outcome-level (format/point/trace) and process-level (process format/accuracy) rewards, where the process-level term is scaled by a factor of 0.25.

Key Experimental Results

Main Results

Evaluation is conducted comprehensively across spatial understanding, measurement, referring, 2D visual traces, and the self-built TraceSpatial-Bench. RoboTracer-8B (SFT only) achieves an average success rate of 85.7% on spatial understanding/measurement, outperforming Gemini-2.5-Pro by 8.58% and the base NVILA-8B by 20.3%. The performance gain in 3D/measurement tasks (23.6%) is markedly larger than in 2D tasks (14.7%).

Benchmark / Metric	RoboTracer (Ours)	Prev. SOTA / Best Baseline	Gain
Spatial Understanding / Measurement Avg. Success Rate (8B-SFT)	85.7%	Gemini-2.5-Pro 77.1%	+8.58%
Q-Spatial S.E. Success Rate (8B-SFT)	83.01%	RoboBrain 2.0-7B 69.11%	+13.9%
2D Visual Trace ShareRobot-Bench (Fréchet↓, 8B-SFT)	0.1384	RoboBrain 2.0-7B 0.1575	Lower
TraceSpatial-Bench Overall Success Rate (2B-RFT, R.I.D.)	45%	RoboRefer-2B 28%	+17%
TraceSpatial-Bench Overall Success Rate (vs Gemini)	39–45%	Gemini-2.5-Pro 3%	+36%
RoboTwin 2.0 hard Overall Avg. Success Rate (2B)	64.0%	π0 8.6%	+55.4%

On TraceSpatial-Bench, although existing strong VLMs perform decently on 2D referring/tracking, they often output 3D traces that "float in mid-air" or collide with objects due to their lack of metric depth understanding. RoboTracer-RFT significantly dominates in 3D metrics, surpassing LEO and RoboRefer which utilize 3D point cloud/depth inputs. On real robots (UR5 pick-and-place, G1 humanoid watering flowers), only the proposed method successfully executes long-horizon tasks requiring multi-step metric-grounded tracking in cluttered dynamic environments. Integrated with Code-as-Monitor, it achieves fast updates at 1.5 Hz, adapting and replanning dynamically when targets are moved.

Ablation Study

Configuration	Key Metric (TraceSpatial-Bench SR)	Description
Full (2B-SFT, ID H)	31%	2D+3D+Video + Spatial Encoder + Regression Scale + (u,v,d)
w/o 2D Data (ID A)	27%	Lacks indoor/outdoor scale supervision, Q-Spatial drops to 51.49
w/o 3D Data (ID B)	19%	Drops the most, Q-Spatial plunges to 33.52
w/o Video Data (ID C)	24%	End-effector tracking and Fréchet significantly degrade (0.4376)
w/o Spatial Encoder (RFT, ID D vs I)	36% vs 39%	RFT slightly degrades after removal but still far outperforms SFT
Unsupervised Scale (ID E)	24%	Q-Spatial 53.47
Scale supervised via N.T.P. text (ID F)	26%	Q-Spatial 57.43, inferior to regression
Modeling with (x,y,z) (ID G)	30%	Fréchet 0.2426, inferior to (u,v,d)
+ Process Reward (RFT)	+4% Overall Success Rate	Compared with outcome-only rewards, the gain in 3D metrics is particularly substantial

Key Findings

• 3D data contributes the most: Removing it drops TraceSpatial-Bench performance from 31% to 19% and Q-Spatial from ~69 to 33.52, as it provides metric-grounded supervision through precise 3D bounding boxes. 2D and video data complement indoor/outdoor scale perception and end-effector tracking, respectively, making all three sources indispensable.
• Regression-based scale supervision > Text-based supervision > Unsupervised: Explicitly regressing the scale factor forces RGB-only models to acquire a sense of scale without needing extra geometry, whereas pure next-token text-based supervision requires massive amounts of data to show even minor effects.
• Multi-step process-level RFT is more critical than high-quality external geometry: Metric-grounded reasoning relies heavily on step-by-step process training. RFT without the spatial encoder still far outperforms SFT, while remaining robust to noisy geometric inputs.
• The more accurate the geometry, the better: Feeding in accurate intrinsics/depth yields up to roughly a 6% absolute improvement, which is plug-and-play without requiring retraining.

Highlights & Insights

• Decoupled learning of "perception" and "reasoning": SFT first solidifies perceptual capabilities like 3D referring and measurement using regression scale decoders and plug-and-play geometry. RFT then organizes these into multi-step reasoning via process-level rewards. This "perception foundation first, reasoning organization second" approach is clear, elegant, and highly transferable.
• Process rewards supervising "key perceptual steps" is key: While most RFT models only offer outcome-level, metric-agnostic rewards, this work utilizes key-step annotations from the dataset to monitor "which object to refer to and how many meters to measure at this step". Designing the rewards to be order-invariant allows flexible step ordering and contributes most to accurate tracking in complex spatial relations.
• The decoupled \((u,v,d)\) representation serves multiple purposes: A single representation can seamlessly degrade to 2D visual traces or 3D/2D referring tasks, aligning naturally with existing 2D data for co-training to improve accuracy while saving data—a very practical engineering trick.
• Emphasis on "explicit geometry > pure implicit learning": In embodied settings, camera intrinsics/depth are often readily available. Instead of forcing the VLM to implicitly learn them from RGB, feeding them in as plug-and-play inputs is highly effective and offers broad design inspiration for embodied VLMs.

Limitations & Future Work

• The authors acknowledge that the method only predicts 3D spatial traces and relies on motion planning to solve for 6D end-effector poses, making it less effective for rotation-heavy operations (such as screwing or flipping) where orientation constraints are key. Dense spatial traces might address this and represent a future research direction.
• RFT was performed only on the 2B model due to compute limits (supplementary materials show that 8B-RFT yields a larger gain: 12% vs 8%), leaving the full potential of 8B-RFT not fully unleashed in the main text.
• Reliance on an extensive suite of foundation models for the data pipeline (RAM++, GroundingDINO, SAM 2.1, MoGe-2, etc.) and simulation generation makes the data quality and biases susceptible to these toolchains. TraceSpatial-Bench only features 100 real-world images (extended to 800 with double-annotated labels in supplementary materials).
• Traces need prior physical constraint refinement before deployment, indicating that the raw outputs from the VLM may still violate physical feasibility.

Related Work & Insights

• vs RoboRefer: RoboRefer performs 2D qualitative spatial referring on a single point from a single image. In contrast, this work tackles the harder task of 3D spatial traces, predicting entire trajectories in the \((u,v,d)\) space and injecting metric awareness into every phase of input, output, supervision, and training. It also collects full task-level, temporally consistent 3D keypoint sequences from manipulation/simulation videos—a feat unachievable with single-image, single-point features.
• vs End-to-end VLAs (e.g., MolmoAct, π0): VLAs directly predict 6D dense actions and require per-task training, largely failing on long-horizon, spatially and metrically constrained tasks (where RoboTwin hard overall success rate resides in the single digits). This work adopts the spatial trace as an intermediate representation, showing significantly stronger zero-shot generalization (surpassing the best baseline on unseen tasks by 32.8%) and can feed back generated action data to MolmoAct, boosting its performance from 0% to 25%.
• vs 2D Visual Trace Methods (Lift-to-3D / Overlap-on-2D, e.g., HAMSTER): These methods output in 2D and rely on depth projection or overlay rendering, which fails to capture full 3D dynamics and lacks supervision of key perceptual steps. This work natively operates on 3D tracking and addresses this gap with process rewards. Ablation studies also demonstrate that "lifting annotations to (u,v,d) 3D" outperforms "lifting outputs to 3D."

Rating

• Novelty: ⭐⭐⭐⭐⭐ Decoupling the task into spatial traces, utilizing metric-sensitive process rewards, and incorporating plug-and-play geometry alongside scale regression creates a highly coherent and recognizable 3D metric reasoning framework.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers spatial understanding/measurement/referring, 2D traces, a custom 3D benchmark, simulations, and real-world robots, with comprehensive ablations dissecting each component and dataset.
• Writing Quality: ⭐⭐⭐⭐ Clear structure and well-designed diagrams; however, some formulas in the original text had minor formatting issues, requiring cross-referencing with supplementary materials to verify notations.
• Value: ⭐⭐⭐⭐⭐ Can directly interface with motion planning to drive multiple real-world robots, and can further assist action generation for VLAs, showing strong potential for real-world embodied deployment.

Related Papers

• [ECCV 2026] ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision–Language Models
• [NeurIPS 2025] RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
• [ACL 2026] TRACE: Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
• [ICLR 2026] OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models
• [ICLR 2026] InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models