R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation¶
Conference: ECCV2026
Paper: ECCV Paper
Project: R3DP Project Page
Area: 3D Vision / Robotic Manipulation
Keywords: imitation learning, diffusion policy, 3D foundation model, temporal feature prediction, multi-view fusion
TL;DR¶
R3DP runs expensive VGGT geometry processing only on sparse keyframes, updates its features with a lightweight temporal network, and fuses camera geometry, enabling an RGB diffusion policy to achieve 69.0% average success over ten RoboTwin tasks while reducing observation encoding latency from 73.1 ms to 40.3 ms under a different operating configuration.
Background & Motivation¶
Diffusion policies can learn multiple feasible actions from demonstrations, but action generation capacity does not itself provide spatial understanding. Standard Diffusion Policy (DP) encodes RGB with a 2D backbone and generates action sequences conditioned on visual features and robot state; distances, occlusions, and correspondences across cameras must be learned indirectly. Simply concatenating features from several images does not tell the policy where the cameras observe the same object from. The paper's multi-view DP baseline even underperforms its single-view counterpart, showing that additional camera inputs do not automatically yield better geometric representations in this training setup.
The point-cloud policy DP3 supplies explicit 3D structure for action learning, but its quality depends on depth observations and point-cloud preprocessing. Transparent cups, reflective surfaces, or incomplete depth can propagate perception errors into grasping and placement, while ground-truth simulation depth can conceal this fragility. 3D foundation models such as VGGT offer another route: extract intermediate features carrying geometric priors directly from RGB, without first constructing point clouds from sensor depth. However, running a large model on every frame slows the policy, and independently estimated geometry can fluctuate in scale over time; stronger static perception does not necessarily produce more stable continuous control.
R3DP therefore asks which information must be recomputed at every invocation and which can be updated from previous geometric understanding. Scene structure is often temporally continuous between policy calls, yet objects and robot arms move, so neither full VGGT processing on every frame nor unmodified reuse of stale features is appropriate. The authors use current RGB to drive lightweight updates, historical geometry as a temporal prior, and known camera parameters to constrain cross-view fusion. Core Idea: support action generation with sparse, high-quality geometric anchors and continuously updated temporal features, decoupling expensive 3D understanding from frequent policy calls.
Method¶
Overall Architecture¶
The inputs are multi-view RGB, camera intrinsics and extrinsics, and robot proprioceptive state; the output is a future action sequence rather than a depth map or reconstructed mesh. The multi-view simulation setup uses head and front cameras, while single-view DP uses only the head camera. The visual pathway retains 2D encoding of current images and adds three connected designs: asynchronous fast-slow collaboration, temporal feature prediction, and multi-view feature fusion. The slow VGGT branch produces Slow 3D-aware Features (S3DF) on sparse keyframes; the fast TFPNet branch produces Real-time 3D-aware Features (R3DF) from historical features and current images. The Multi-View Feature Fuser (MVFF) combines R3DF with 2D features and uses camera parameters to produce the visual condition for the diffusion head. Here, 3D information resides primarily in latent features: an explicit point cloud need not be decoded every control cycle, and the paper's depth visualizations mainly analyze whether geometric information is retained.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Images["Multi-view RGB"] --> Schedule["Asynchronous fast-slow collaboration<br/>Sparse VGGT anchors"]
Schedule --> Predict["Temporal feature prediction<br/>TFPNet updates R3DF"]
Images --> Predict
Predict --> Fuse["Multi-view feature fusion<br/>MVFF and PRoPE"]
Images --> Encoder["ResNet 2D features"]
Encoder --> Fuse
Cameras["Camera intrinsics and extrinsics"] --> Fuse
Fuse --> Policy["Diffusion head + robot state<br/>Future action sequence"]
Predict -.->|Historical feature recurrence| Predict
Teacher["Training-time VGGT<br/>Per-frame teacher features"] -.->|Pretraining supervision only| Predict
Solid edges summarize deployment data flow; the teacher's dashed edge denotes TFPNet pretraining supervision, not an additional teacher computation at every policy call. The slow branch refresh interval is controlled by \(\tau\), measured in policy inference calls rather than directly in camera frames or robot servo cycles. A policy can execute an action chunk per call, so temporal sampling must account for observations separated by multiple action steps.
Key Designs¶
1. Asynchronous fast-slow collaboration: use the large model only for sparse geometric anchors
AFSC treats VGGT as a slow system that processes current multi-view images every \(\tau\) inferences to obtain high-quality S3DF. Intermediate calls no longer run full VGGT processing on every image; instead, a fast system advances previous geometric information to the current time. This is not simple cache reuse: new observations still enter a lightweight network so that features can change with object motion and occlusion. Periodic VGGT refreshes provide renewed geometric evidence for the recurrent process, avoiding complete reliance on indefinitely extended historical predictions. The computational saving occurs in observation encoding, while the downstream action diffusion model still performs sampling and denoising. Consequently, the claimed real-time advantage cannot be translated into a complete control frequency by merely inverting encoding latency.
The design also makes refresh frequency a deployment-time parameter. The policy is trained with \(\tau=8\) and evaluated with both \(\tau=4\) and \(\tau=8\), without retraining for each interval. A shorter interval corrects geometry more frequently, whereas a longer one reduces large-model calls. Across the ten tasks, increasing the interval from 4 to 8 changes average success from 69.0% to 65.7%, so performance is not entirely unchanged. The main text does not detail asynchronous threads, queues for arriving slow results, or stale-feature handling; the supported description is the fast-slow feature collaboration mechanism, not a particular concurrency implementation.
2. Temporal feature prediction: correct historical geometry with current images instead of distilling frames independently
TFPNet first encodes each view with DINOv2-S, then aggregates views through 4 Alternating-Attention Transformer blocks. It subsequently combines historical 3D features through cross-attention to produce the current R3DF. The first timestep uses VGGT features as its initialization basis, and later steps recursively use previous predictions alongside current images. The fast network thus addresses how an existing scene representation has changed, rather than recovering the full scene's spatial structure from scratch each time. Historical state provides geometric continuity, while current images provide evidence of change; removing either would move the system toward independent prediction or stale caching. Figure 3 and the wording in Section 4.2 do not fully agree on the cross-attention query direction, so this note does not assert that historical features must serve as either keys/values or queries.
During pretraining, the authors sample 4 chronologically ordered frames from one episode, with random gaps of 1–8 frames between adjacent samples. VGGT supplies teacher features for all these frames, using the unified token representation from its 24 layers for dense outputs. The student is unrolled through the sequence, receives a teacher anchor at the first frame, and updates later frames from its own previously predicted features. This training exposes the student to recurrent prediction errors rather than learning only a conditionally independent RGB-to-VGGT mapping. Supervision uses a cosine similarity loss between predicted and teacher features to preserve geometric information in feature direction. Cached equations (9)–(10) are corrupted, preventing reliable recovery of the complete notation, reduction, or loss constants; a reconstructed standard cosine formula is therefore not presented as the authors' exact equation.
3. Multi-view feature fusion: incorporate camera relationships when combining 2D and 3D information
Although VGGT's multi-view tokens describe a shared scene, they retain view-specific properties, such as supporting separate depth decoding for each view. MVFF first fuses information within each view: ResNet supplies current 2D appearance, R3DF supplies geometric context, and cross-attention combines them into hybrid features. The equation text in Section 4.3 projects 2D features into queries and R3DF into keys and values, whereas Figure 2 appears to label Q/K/V in the opposite direction. This note follows the equation description while retaining that figure-text inconsistency, rather than treating it as an implementation fact verified against code. The central mechanism lets current appearance access geometric features instead of replacing the 2D input entirely with a point-cloud representation.
Projective Positional Encoding (PRoPE) then injects camera intrinsics and extrinsics into multi-view aggregation. Extrinsics describe relative camera poses, while intrinsics describe imaging properties such as focal length; encoding only SE(3) poses cannot fully capture differences between camera frustums. PRoPE organizes attention through relative projective relationships rather than treating camera indices as absolute positions, prioritizing geometry over arbitrary view order. It builds on an existing camera positional encoding method, not a new geometric theorem introduced by this paper. The authors emphasize that actual camera parameters are more reliable than relying entirely on VGGT's implicit estimates, but this also means that avoiding depth sensors does not remove the need for camera calibration. The fused visual embedding and robot state condition diffusion denoising to produce executable future action chunks.
A Worked Example¶
Consider the bimanual Block Handover task; this is an explanatory walkthrough of the architecture, not an additional trajectory released by the authors. At a key observation, head and front RGB images enter VGGT together, establishing geometric feature anchors for the block, robot arms, and background. After an action segment, the block has moved; the next policy call supplies both current views to TFPNet and retrieves the previous historical features. The fast network updates the representation around the block instead of passing its old position unchanged to the policy. MVFF combines the updated geometric features with current ResNet appearance features and constrains cross-view relationships using both cameras' parameters. The diffusion head generates a new handover action sequence conditioned on visual information and robot state; the slow branch supplies another anchor when the refresh interval is reached. Table 1 reports 95% success on this task at \(\tau=4\), but that result alone cannot attribute occlusion recovery within individual handovers to a particular module.
Loss & Training¶
Training separates geometric feature pretraining from downstream imitation learning; teacher-feature supervision and action supervision are different objectives. TFPNet first learns to predict VGGT features and is subsequently used together with VGGT as a frozen visual backbone during policy training. The policy stage uses a diffusion noise prediction objective to learn the demonstration action distribution; cached diffusion equations (1)–(2) contain missing terms and are not reproduced in their corrupted form. Section 4.1 explicitly states that only the diffusion head is updated at this stage while also calling training end-to-end; this should not be interpreted as joint fine-tuning of VGGT and TFPNet. That passage does not separately clarify whether ResNet and MVFF are updated or how their parameters are grouped, so their optimizer settings still require verification for reproduction.
Each downstream training tuple contains 8 frames sampled 8 timesteps apart, instead of the original 3 consecutive observations. The paper calls this an effective 64-frame window; strictly, the first and last sampled indices are 56 steps apart, reflecting different counting conventions. The first frame in each tuple runs through VGGT, subsequent frames run through TFPNet in order, and processing is serial within tuples but parallel across batch samples. The authors report 600 policy-training epochs on 4 NVIDIA 4090 GPUs using Adam, a learning rate of 2e-5, and batch size 8; the universal TFPNet is trained on 8 NVIDIA A800 GPUs. Components involving VGGT outputs use bfloat16; real-robot experiments use DDIM with 100 training scheduler steps and 10 denoising steps at sampling time. These settings show that a lightweight deployment branch does not imply inexpensive training of the entire system.
Key Experimental Results¶
Main Results¶
The following results come from Table 1, page 10, and all metrics are success rates; every model is evaluated for 100 trials per task under identical environmental configurations. The average covers all 10 tasks, while the task rows show only 4 representative examples and are not used to recompute that average. DP3 uses ground-truth simulation depth and cropping boxes, whereas DP3-DA2 uses estimated depth, so their input conditions differ.
| Task / Summary | DP-single | DP-multi | DP3 | DP3-DA2 | R3DP (\(\tau=4\)) | R3DP (\(\tau=8\)) | \(\pi_0\) | \(\pi_{0.5}\) |
|---|---|---|---|---|---|---|---|---|
| Block Handover | 1% | 2% | 48% | 31% | 95% | 93% | 71% | 41% |
| Blocks Stack Easy | 6% | 7% | 26% | 3% | 69% | 62% | 79% | 86% |
| Diverse Bottles Pick | 16% | 17% | 34% | 7% | 31% | 32% | 47% | 66% |
| Tube Insert | 92% | 64% | 97% | 32% | 97% | 97% | 68% | 75% |
| All 10 tasks, average | 36.1% | 17.6% | 57.6% | 28.2% | 69.0% | 65.7% | 59.9% | 66.7% |
Comparing 69.0% with 36.1% and 17.6% gives gains of 32.9 and 51.4 percentage points, not relative improvements of 32.9% and 51.4%. R3DP outperforms both VLA baselines on average but not on stacking or diverse-bottle picking; the paper's broad wording about consistent gains across all tasks must be interpreted against the individual rows. SpatialVLA is evaluated separately on 4 RoboTwin 2.0 tasks because of compatibility issues; Table 2 reports 63.8% for R3DP and 52.8% for SpatialVLA, which cannot be merged into Table 1's ten-task ranking.
Ablation Study¶
The following table comes from Table 4, page 12, and progressively adds modules on 4 tasks, again reporting success rates. The final R3DP column matches the per-task results of the \(\tau=4\) configuration in Table 1; averages retain the authors' one-decimal reporting.
| Task | DP-multi | DP+VGGT | DP+VGGT+MVFF | R3DP |
|---|---|---|---|---|
| Blocks Stack Easy | 7% | 30% | 51% | 69% |
| Block Hammer Beat | 0% | 44% | 74% | 77% |
| Shoe Place | 19% | 55% | 66% | 72% |
| Diverse Bottles Pick | 17% | 30% | 30% | 31% |
| Average | 10.8% | 39.8% | 55.3% | 62.3% |
Adding VGGT, then MVFF, and finally AFSC yields average gains of 29.0, 15.5, and 7.0 percentage points, respectively, computed from the displayed values. This supports the usefulness of geometric priors, explicit multi-view fusion, and temporal processing, but it is not a full factorial experiment establishing independent effects. In particular, Diverse Bottles Pick changes from 30% to 30% and then 31%, showing that these modules do not address every task bottleneck.
Table 3, page 11, measures latency over the same 4 ablation tasks; observation encoding runs from raw multi-view images to final fused features and excludes the subsequent action expert.
| Latency component, ms, lower is better | DP+VGGT | DP+VGGT+MVFF | R3DP (\(\tau=4\)) | R3DP (\(\tau=8\)) |
|---|---|---|---|---|
| Observation encoder | 73.1 | 78.3 | 50.5 | 40.3 |
| Action expert | 56.6 | 57.7 | 55.3 | 56.7 |
The change from 73.1 ms to 40.3 ms is a 44.8% reduction in observation encoding latency; this lowest latency uses \(\tau=8\), whereas the highest average success of 69.0% uses \(\tau=4\). The action expert still takes approximately 56 ms, so 40.3 ms must not be presented as complete camera-to-action latency.
Key Findings¶
- Real-world Table 5, page 14, reports average success over 4 tasks of 30.8% for DP, 48.4% for DP3, 66.7% for DP+VGGT+MVFF, and 71.7% for R3DP. Each bimanual task uses 100 demonstrations and each single-arm task uses 50, with 30 evaluation trials per task.
- The same table reports R3DP observation encoding at 62.18 ms and the action expert at 69.78 ms on an RTX 4090. Real-world encoding time and the 40.3 ms simulation result in Table 3 must not be treated as one configuration.
- Table 6, page 15, reports adjacent-frame Log Drift of 0.0272 for VGGT and 0.0066 for TFPNet, with lower values indicating greater scale stability. It measures adjacent-frame logarithmic changes in the optimal scale factors needed to align predicted depth with ground truth, not absolute depth accuracy; cached equation (14) lacks complete absolute-value or other reduction notation, so its exact formula is not reconstructed.
Highlights & Insights¶
- The fast network is not merely an accelerator: it explicitly uses historical information. Its ablation improves success while reducing encoding cost, shifting the objective from replacing a large model with a cheaper one to supplying control with a representation better suited to temporal tasks.
- Geometric priors and camera calibration provide complementary information. VGGT supplies transferable spatial representations, while MVFF uses known imaging relationships on the particular robot to reduce uncertainty in cross-view fusion.
- The refresh interval offers a deployment-level adjustment. The practical insight is to reduce expensive representation recomputation while checking action quality and the complete control pathway, rather than optimizing encoder latency alone.
Limitations & Future Work¶
- The authors plan extensions to more contact-rich, longer-horizon, and reinforcement-learning-driven manipulation; current evidence mainly concerns task-specific imitation learning and should not be equated with open-world general-purpose robotics.
- Real-world evaluation covers only 4 tasks with 30 trials each, and the result tables provide no multi-seed confidence intervals. Small differences require caution, especially where one additional success noticeably changes the reported percentage.
- Camera intrinsics and extrinsics must be reliable, and fast-branch errors may accumulate under sudden occlusion, rapid deformation, or large viewpoint changes. Uncertainty-triggered keyframes are a reader-proposed extension, not a validated module in this paper.
- Several cached equations are corrupted, and attention directions and MVFF optimizer membership are insufficiently clear. The available material supports understanding the method and checking tables, but not unambiguous reproduction of every implementation detail.
Related Work & Insights¶
- Compared with Diffusion Policy: R3DP retains action diffusion and improves its conditional visual representation, placing its main contribution in perception and temporal collaboration rather than a new action-generating distribution.
- Compared with DP3: RGB-based latent geometry reduces dependence on actual depth quality and point-cloud preprocessing, although DP3 remains strong on some tasks when simulation ground-truth depth is reliable.
- Compared with eVGGT: both address geometric encoding efficiency, but R3DP incorporates historical feature recurrence and sparse teacher refreshes into control rather than emphasizing only single-frame encoder compression.
- Compared with PRoPE: it reuses relative positional encoding of complete camera geometry for multi-view fusion of policy conditions; the transferable lesson is to inject existing reliable geometry directly into attention.
Rating¶
- Novelty: 4/5. Sparse geometric anchors, recurrent prediction, and calibration-aware fusion form a coherent policy perception module, but rely on existing backbones and encoding methods.
- Experimental Thoroughness: 4/5. Simulation, real-world tasks, component ablations, and temporal scale analysis are included, but real-task coverage and statistical uncertainty reporting remain limited.
- Writing Quality: 3/5. The overall argument is accessible, but figure-text directions, some training boundaries, and percentage terminology need clarification; equation corruption is a separate cache-extraction issue.
- Value: 4/5. The framework offers practical guidance for integrating costly geometric foundation models into robot policies, while deployment evaluation still needs to cover the complete action pathway.