3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

Conference: ECCV2026
Official Paper: 4788
Paper: PDF
Code: https://github.com/ziqin-h/3DWay
Area: Robotics / Embodied Intelligence
Keywords: 3D consistent waypoints, multi-view geometry, trajectory representations, vision-language-action models, few-shot generalization

TL;DR

3DWay expresses 3D trajectory generation as multi-view 2D coordinate prediction in a VLM's text interface, reconstructs executable waypoints using calibrated geometry, reaches 64.0% success on unseen RLBench tasks with fixed top-down control, and improves few-shot pi0 adaptation through local spatial guidance.

Background & Motivation

Vision-language models can interpret instructions such as offering a red fruit to a named character, but producing continuous robot actions requires a substantial change in representation and supervision. Trajectories offer an interpretable intermediate step: determine where the arm should move before asking a controller to execute the motion. Unlike an object box or a grasp region, a trajectory also expresses movement order and intermediate locations.

The problem is that a point on a 2D trajectory does not uniquely identify a position in space. Looking up its depth is not always sufficient either: a depth image measures the visible surface behind that pixel, whereas a waypoint above a basketball hoop may lie in empty space. The resulting representation, called 2.5D waypoints in the paper, can incorrectly attach free-space motion to an observed surface. Directly regressing world coordinates from a VLM presents a different difficulty, requiring adaptation to camera parameters and 3D action outputs instead of retaining the pretrained image-text interface.

The paper preserves 2D prediction but requires different views to describe the same spatial waypoints, allowing calibrated geometry to recover their 3D locations. Core idea: let the VLM predict corresponding multi-view 2D waypoints, let triangulation recover depth, and use the resulting path either for simple direct execution or as guidance for a more capable VLA policy.

Method

Overall Architecture

Inputs are a language instruction, multi-view RGB observations, and known camera intrinsics and extrinsics; the default system uses two cameras. The VLM receives images and language and produces per-view waypoint coordinate sequences with gripper states. Camera parameters support training-label projection and inference-time triangulation, rather than requiring the VLM to generate world coordinates directly. The output describes a simplified tool center point (TCP) path, not a full joint-action sequence or a complete six-degree-of-freedom pose trajectory.

Training builds shared labels through Consistent Waypoint Supervision, then adapts NVILA through Multi-view Text Prediction. At inference, Geometric Triangulation recovers the spatial path, and Dual-mode Execution either applies fixed top-down control or supplies local waypoint conditions to pi0. The training branch in the diagram describes label construction; demonstrations are not required at deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training demonstrations<br/>and camera parameters"] --> B["Consistent Waypoint<br/>Supervision"]
    B -->|Training labels| C["Multi-view Text<br/>Prediction"]
    I["Multi-view RGB<br/>and instruction"] --> C
    C --> D["Geometric<br/>Triangulation"]
    K["Known camera parameters"] --> D
    D --> E["Dual-mode Execution"]
    E -->|Simple tasks| F["Top-down grasping<br/>and motion planning"]
    E -->|Dexterous actions| G["Local waypoint<br/>guidance for pi0"]

Key Designs

1. Consistent Waypoint Supervision: select points in 3D before projecting them into each view

The challenge is not merely predicting an accurate point in each image. Corresponding sequence entries must represent the same execution phase and the same spatial location. The authors compute the raw TCP trajectory from robot states and the end-effector model, simplify the 3D trajectory using Ramer-Douglas-Peucker (RDP), and project the retained points into each calibrated camera. Thus, every view receives labels derived from one shared 3D sequence, rather than separately simplified image-plane paths with potentially mismatched correspondences.

This preserves important changes in the path while reducing redundant outputs from dense trajectories. Training samples contain initial observations, task descriptions, and associated execution trajectories; they do not require the entire demonstration video as input. RLBench, DROID, and RH20T supply approximately 290 tasks and 134k trajectories. An automatic scoring process combining task prompts and an open-vocabulary detector filters low-quality examples. The cached main paper does not specify the scoring details or RDP threshold, so no filtering equation or threshold is assumed here.

2. Multi-view Text Prediction: learn corresponding waypoints through the language-modeling interface

The generator uses NVILA backbones with 2B, 8B, or 15B parameters. Initial fine-tuning on RoboPoint improves pixel-level localization; subsequent training teaches multi-view consistent 2D waypoints, abbreviated 2D-MCW. Targets encode per-camera coordinate sequences and gripper opening or closing states as text. Learning uses language-modeling cross-entropy rather than an added 3D regression head, an action expert head, or a complex geometric objective.

The motivation is specific to the modality mismatch: image-conditioned text generation is already native to the VLM, and emitting image coordinates is closer to that interface than predicting world-frame robot coordinates. Shared multi-view labels introduce spatial correspondences through supervision. Importantly, the paper does not introduce a dedicated loss that guarantees exact geometric agreement among predictions. Consistency primarily comes from joint multi-view prediction and labels constructed from shared 3D trajectories; geometric reconstruction follows afterward. It should not be described as a hard constraint already satisfied by every generated coordinate.

3. Geometric Triangulation: recover free-space locations instead of querying surface depth

Each corresponding image point defines a ray through a calibrated camera. Triangulation combines rays from multiple views to recover a spatial location without requiring that location to lie on a visible surface. The central conversion, expressing the meaning of Equation (4), is:

\[ \widehat W_{3D}=\operatorname{Tri}(\widehat{\mathcal W}_{2D},\mathcal K,\mathcal T). \]

The predicted multi-view 2D waypoints, camera intrinsics, and camera extrinsics jointly determine the reconstruction; the triangulation operator is not another neural network. Geometry handles depth recovery and coordinate conversion, while the VLM identifies where the instruction requires the end effector to pass. A waypoint above a container can therefore be located independently of the depth of the table or object visible behind its image projection.

This does not remove all uncertainty. If views predict different execution stages, a waypoint is occluded, or calibration is inaccurate, the reconstructed path can still be wrong. The main text provides the triangulation interface but does not detail the numerical solver, outlier rejection, or handling of unmatched output sequences. A specific robust solver should therefore not be attributed to the implementation from this cache alone.

4. Dual-mode Execution: follow waypoints directly for simple tasks and let pi0 complete dexterous actions

The direct-execution variant, 3DWay-TD, supplies a fixed top-down grasp orientation that the waypoint path itself does not predict. Waypoint-based motion planning and inverse kinematics then produce low-level commands. This tests whether the representation is actionable without training a task-specific low-level policy. However, a fixed orientation cannot naturally support rotation-dependent manipulation, so the strong direct-execution results must be read together with the rotation-invariant task selection.

For more complex actions, 3DWay conditions pi0 instead. During execution, distances between the current end-effector position and the waypoint sequence determine a local temporal window, which is concatenated with the robot's proprioceptive state. The default window contains two waypoints. The action policy therefore receives nearby spatial guidance relevant to its current execution phase rather than repeatedly processing the entire path. The paper states that the window's starting index depends on distance but does not fully specify its selection rule; a particular nearest-neighbor or monotonic-progress algorithm is not assumed here.

Because pi0 was not pretrained with waypoint conditioning, it must learn to use this information during in-distribution fine-tuning, although no architectural modification is needed. A zero-shot waypoint generator means that the generator receives no additional target-task fine-tuning. It does not mean that the downstream pi0 policy has never seen demonstrations of those tasks. This distinction is essential for interpreting the few-shot results.

A Worked Example

Consider the paper's illustrative instruction to put a ball in a hoop. From two initial views, the VLM identifies the ball and target and predicts corresponding waypoints and gripper states for grasping, lifting, moving, and releasing. These stages explain the workflow; they do not imply that the paper uses a fixed number of waypoints.

When the ball must pass through free space above the hoop, triangulation reconstructs that elevated target rather than using the depth of the visible surface behind the relevant pixel. If top-down grasping is adequate, the controller follows the path directly. If pi0 handles the finer action details, its local waypoint window updates as the arm progresses, while current images and proprioception inform action prediction. Updating that guidance window is not equivalent to dynamically replanning the waypoint sequence as the environment changes.

Loss & Training

The generator is fine-tuned for one epoch on RoboPoint and ten epochs on the processed trajectory data. Both stages use batch size 256, cross-entropy, AdamW, a learning rate of \(10^{-5}\), and cosine decay. Full-parameter VLM fine-tuning uses eight A100 GPUs and finishes in under two days even for NVILA-15B.

Unless otherwise specified, experiments use the 15B generator, two cameras, and a two-waypoint adaptive window. Preserving vision-language reasoning is a motivation for retaining the text-generation objective. Semantic manipulation results support that motivation, but they are not a comprehensive assessment of retained general-purpose VLM capabilities.

Key Experimental Results

Main Results

The following selection from Table 1 reports task success rates (%). RLBench uses ten rotation-invariant tasks; the unseen setting excludes demonstrations from test tasks, and each task is evaluated in three runs of 25 rollouts. VLABench training uses 500 demonstrations for each of five representative tasks, followed by evaluation across in-distribution, cross-category, common-sense, semantic-instruction, and unseen-texture settings, with 50 trials per task and setting.

Method	RLBench seen	RLBench unseen	VLABench five-dimension average
OpenVLA-OFT	8.8	Not reported	0.4
pi0	39.6	12.0	25.9
pi0.5	61.5	18.8	31.3
3DWay-TD	84.8	64.0	37.7

Relative to pi0.5, 3DWay-TD gains 45.2 percentage points on unseen RLBench tasks and 6.4 points in the VLABench average. It does not win every dimension: its semantic-instruction score is 24.3%, below pi0.5's 25.8%. Table 1 labels its parenthesized uncertainty as standard deviation, while the surrounding text calls it standard error. Only means are reproduced here to avoid conflating these statistics.

Few-shot Analysis

The following results come from Table 2 and compare pi0 fine-tuning on five RLBench tasks. Zero-shot and few-shot describe the waypoint generator; all three ten-demonstration pi0 configurations undergo target-task fine-tuning.

pi0 fine-tuning configuration	Demonstrations per task	Average success (%)
Standard	10	17.6
3DWay-augmented, zero-shot generator	10	38.1
3DWay-augmented, few-shot generator	10	46.1
Standard	30	31.2
Standard	50	46.7

Differences calculated from the table are 20.5 and 28.5 percentage points for the two augmented configurations. The cached prose instead states 18.4 and 26.4; this note uses the table entries and their arithmetic differences. Few-shot augmentation reaches 46.1% with ten demonstrations, close to standard fine-tuning's 46.7% with fifty. The claim of using only 20% of the data concerns target-task fine-tuning demonstrations, not the generator's pretraining data.

Ablation Study

Table 3 evaluates unseen RLBench tasks under a common protocol, using task success as an indirect measure of waypoint quality. Extrinsic-shift experiments change the test camera pose; they should not automatically be interpreted as corrupting calibration parameters.

Configuration	Backbone	Test camera pose shift	Multi-view consistent supervision	Success (%)
A0, default	NVILA-15B	None	Yes	64.0
B0	NVILA-8B	None	Yes	62.8
B1	NVILA-2B	None	Yes	50.1
C0	NVILA-15B	Small	Yes	61.6
C1	NVILA-15B	Large	Yes	58.4
D0	NVILA-15B	None	No, independent-view prediction	22.0

Key Findings

• D0 offers the strongest mechanism evidence: independent-view supervision and prediction reduce success from 64.0% to 22.0%, a 42.0-point drop, even though predictions are still triangulated. Geometric post-processing cannot automatically repair semantic or execution-stage mismatches across views.
• The 8B backbone loses only 1.2 points relative to 15B, whereas 2B loses 13.9 points. This suggests a practical smaller-model option in this setting, but without latency measurements it does not establish real-time operation.
• Real-world experiments use an AgileX PIPER arm, with twenty demonstrations per basic task and 100 trajectories overall. The generator receives no additional in-domain fine-tuning, and each basic or extended task is tested with 24 rollouts. Section 4.3 reports an improvement from 21.7% for standard pi0 to 65.8% for augmented pi0 on basic tasks, a gain of 44.1 percentage points.

Highlights & Insights

• Spatial reasoning does not necessarily require a neural 3D output head. Predicting corresponding 2D observations and recovering structure through known geometry reduces the coordinate transformations that the model must learn from data.
• The key supervision choice is a shared 3D source, not simply another camera image. Transferring this label-construction principle to other multi-view planning problems may be more useful than increasing the number of views alone.
• One representation supports both direct control and policy conditioning. Direct execution tests actionability, whereas VLA augmentation tests whether paths help complete orientation and low-level actions; these are complementary questions.

Limitations & Future Work

• The authors explicitly acknowledge that the current system focuses on translational waypoints without explicit orientation prediction or dynamic adaptation. SE(3) pose generation and hierarchical subtask reasoning are future directions, not existing full dexterous-planning capabilities.
• Pre-calibrated cameras and predominantly dual-view validation leave uncertainty under clutter, occlusion, and dynamic scenes. Robustness to changed viewpoints does not substitute for evaluating calibration errors or camera failures.
• VLA integration remains a proof of concept, and local state concatenation does not fully explore joint waypoint-action reasoning. Uncertainty-aware fusion and online waypoint updates are possible extensions, not modules already implemented in the paper.
• This note's assessment: the main comparison emphasizes foundation VLAs rather than a systematic, equal-budget comparison with other trajectory-guided methods. It cannot establish superiority over all 3D policies; the authors also note the difficulty of matching low-level controllers and post-processing.
• Reproduction requires supplementary details on filtering, output formats, and failure cases. Inconsistent uncertainty labels and some gains in the cached main paper warrant checking original experiment records instead of treating every reported number as internally reconciled.

Related Work & Insights

• Compared with HAMSTER / RT-Trajectory: these approaches use 2D trajectories to express motion intent. 3DWay adds multi-view correspondence and explicit 3D reconstruction, gaining spatial specificity while requiring calibrated multi-camera input.
• Compared with depth-lifting approaches such as MOKA / A0: under the paper's categorization, depth lookup supplies 3D surface cues but leaves free-space waypoints ambiguous. Multi-view correspondence localizes those waypoints instead, shifting the bottleneck toward correspondence prediction and calibration reliability.
• Compared with SpatialVLA: SpatialVLA integrates depth-aware spatial encoding into visual features, whereas 3DWay produces a separately inspectable and executable path representation. These choices are compatible; a spatially aware policy could potentially benefit from explicit path guidance.
• Relationship to pi0: pi0 is both a standard baseline and the policy used to test waypoint augmentation. Results support complementarity under scarce target-task data, not replacing the foundation policy's continuous control skills with waypoints alone.

Rating

• Novelty: 4/5. Shared 3D-projection supervision and actionable intermediate paths form a clear contribution built from established VLM adaptation and geometry.
• Experimental Thoroughness: 4/5. Two simulation benchmarks, few-shot adaptation, real robots, and consistency ablations provide breadth, but matched trajectory baselines, dynamic settings, and statistical reporting need further work.
• Writing Quality: 3/5. Motivation and architecture are understandable, but table-prose discrepancies and inconsistent uncertainty terminology weaken precision; reproduction details depend on supplementary material.
• Value: 4/5. Useful guidance for calibrated multi-view manipulation with scarce demonstrations, with important boundaries around rotation, occlusion, and dynamic adaptation.