ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation

Conference: ECCV 2026
Paper: ECCV Original
Code: https://github.com/Tencent-Hunyuan/ROAR-3D
Area: 3D Vision
Keywords: 3D generation, multi-view conditioning, view routing, dual-stream attention, pose-free 3D reconstruction

TL;DR

ROAR-3D introduces a lightweight multi-view 3D generation framework that augments a pretrained single-view diffusion model with a token-wise view router, dual-stream cross-attention, and an orientation perturbation strategy, enabling flexible integration of arbitrary unposed multi-view inputs without camera poses.

Background & Motivation

Diffusion-based 3D generative models have achieved remarkable progress in synthesizing detailed and plausible geometry from a single 2D image. By learning rich generative priors from large-scale 3D datasets, these models complete unobserved regions even when input views are limited. However, single-view conditioning inherently introduces geometric ambiguity: the model must hallucinate unobserved surfaces, often producing geometries that deviate significantly from a user's multi-view concept sketches, reference images, or sequential frames.

Multi-view conditioned generation offers a natural bridge between pure generation and classical multi-view reconstruction, preserving generative robustness while drastically reducing shape ambiguity. Yet, current solutions remain hindered by severe limitations. On one hand, feed-forward models and commercial platforms demand rigid, canonical camera setups, failing on unconstrained captures. On the other hand, recent attempts to inject features from external reconstruction backbones (such as VGGT) incur prohibitive training overheads due to substantial domain gaps, while their generation fidelity remains strictly bottlenecked by the external module's reconstruction errors. Furthermore, while pretrained single-view models already possess strong implicit 2D-to-3D grounding across diverse angles, naively concatenating multi-view tokens into cross-attention layers creates destructive conflicts between differing viewpoint orientations.

Core idea: reuse the rich spatial grounding of pretrained single-view DiTs by using 3D latent tokens as geometric anchors, dynamically routing each 3D token to its most informative view via a lightweight token-wise view router, and decoupling global orientation anchoring from local geometric transfer using dual-stream cross-attention and orientation perturbation.

Method

Overall Architecture

ROAR-3D builds upon a flow-matching Diffusion Transformer (DiT) and a 3DShape2VecSet-based 3D geometry VAE (adopting Hunyuan3D 2.1 as the foundational backbone). The system takes a designated primary view alongside an arbitrary number of \(V-1\) unposed auxiliary reference views, outputting a high-fidelity 3D shape within a unified canonical frame. The pipeline coordinates three essential stages: a token-wise view router that assigns each 3D latent token to the most informative 2D view, a dual-stream cross-attention backbone separating orientation locking from geometry enrichment, and a second-stage geometric refinement DiT that elevates structural details.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Image Inputs<br/>Primary view + arbitrary auxiliary views"] --> B["Token-Wise View Router<br/>3D token affinity with mean-pooled patch features"]
    B --> C["Dual-Stream Cross-Attention<br/>Primary stream for orientation + auxiliary stream for geometry"]
    C --> D["Orientation Perturbation Strategy<br/>Discrete latent rotations break alignment shortcut"]
    D --> E["Coarse 3D Latent Representation"]
    E --> F["Geometric Refinement Stage<br/>High-resolution voxel guidance with multi-view conditioning"]
    F --> G["High-Fidelity 3D Mesh Output"]

Key Designs

1. Token-Wise View Router: implicit 2D-to-3D grounding via feature affinity

Concatenating all multi-view tokens into a unified attention module causes computational cost to surge linearly with the number of views and forces 3D tokens to battle conflicting viewpoint signals. The token-wise view router resolves this by operating per transformer block, allowing each spatially grounded 3D latent token to select its single most relevant view. To capture representative view-level context without losing localized correlations, the router pairs the LayerNorm-projected 3D token query with the mean-pooled DINOv2 patch representation \(\bar{f}_v\) of each view. Query and key projections (\(W_q^{(\ell)}\) and \(W_k^{(\ell)}\)) are directly initialized from the pretrained cross-attention layers, drastically expediting convergence. Differentiable routing is achieved via hard Gumbel-Softmax with straight-through gradient estimation during training:

\[r_{i, v}^{(\ell)} = \mathbf{w}_{\text{agg}}^\top \left[ \frac{(q_{i,1}^{(\ell)})^\top k_{v,1}^{(\ell)}}{\sqrt{d}}, \;\ldots,\; \frac{(q_{i,H}^{(\ell)})^\top k_{v,H}^{(\ell)}}{\sqrt{d}} \right]\]

2. Dual-Stream Cross-Attention: decoupling global orientation from geometric structure transfer

In single-view DiTs, cross-attention simultaneously handles two entangled roles: establishing the canonical global orientation from the camera perspective and transferring structural geometry from the image. Naively mixing views causes orientation signals to collide, generating severe distortion. ROAR-3D factors these operations into two structural streams: the primary stream \(\text{CA}_p\) retains the pretrained weights and processes tokens routed to the primary view \(v_p\), firmly establishing global orientation and baseline geometry; the auxiliary stream \(\text{CA}_a\), copied from \(\text{CA}_p\) at initialization, handles tokens assigned to auxiliary views, restricting its role strictly to supplementary geometry transfer. This dual-stream separation ensures the per-token attention computation remains identical to the single-view baseline.

3. Orientation Perturbation: enforcing orientation-invariant geometry transfer

During standard training, the ground-truth 3D latent remains canonically aligned with the primary view, allowing the auxiliary stream \(\text{CA}_a\) to exploit this fixed spatial relationship as a shortcut rather than learning genuine orientation-invariant feature mapping. To eliminate this reliance, ROAR-3D activates an orientation perturbation strategy with probability \(p_{\text{pert}} = 0.2\). In perturbed iterations, the 3D latent point cloud is randomly rotated azimuthally to \(\{0^\circ, 90^\circ, 180^\circ, 270^\circ\}\) (strictly excluding orientations aligned with any conditioning view). Concurrently, primary view status is stripped so all views pass exclusively through \(\text{CA}_a\), and only \(\text{CA}_a\) parameters receive gradient updates while \(\text{CA}_p\) is frozen. This adversarial setup forces \(\text{CA}_a\) to establish 2D-to-3D correspondence purely through semantic feature affinity.

4. Geometric Refinement Stage: fine-grained detail synthesis in high-resolution latent space

To sharpen surface details and suppress meshing noise, ROAR-3D incorporates a second-stage geometric refiner adapted from LATTICE. The refinement DiT operates on 6144 latent tokens and integrates visual features extracted by a larger DINOv2 ViT-G/14 encoder. By adopting the identical token-wise view router, dual-stream attention, and orientation perturbation mechanisms conditioned on the first stage's coarse voxel-grid positional embeddings, this stage enriches subtle surface contours, engravings, and fine structural features.

Loss & Training

The architecture is trained using a flow-matching conditional objective. The first-stage DiT backbone contains 4096 latent tokens of hidden dimension 2048, extracting image features with a frozen DINOv2 ViT-L/14 model. During training, 1 to 4 auxiliary views are uniformly sampled per instance, enabling flexible test-time view scaling. Training employs the AdamW optimizer with an initial learning rate of \(1 \times 10^{-5}\) under a cosine decay schedule over 50K iterations, taking approximately 3 to 4 GPU days. The model seamlessly generalizes to 1 to 12+ input views at inference time without architectural modifications.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on a held-out test set of 100 high-quality 3D assets from Objaverse and ObjaverseXL. Methods are evaluated on Chamfer Distance (CD, \(\times 10^{-3}\), lower is better), F1-Score at thresholds 0.1 and 0.05 (%, higher is better), and semantic image-shape alignment using ULIP-I and Uni-I (higher is better).

Method	Category	CD↓	F1(0.1)↑	F1(0.05)↑	ULIP-I↑	Uni-I↑
Hunyuan3D 2.0	Single-view generative	35.421	80.5	64.2	0.141	0.302
Hunyuan3D 2.1	Single-view generative	34.031	82.6	68.7	0.147	0.317
Trellis2	Single-view generative	33.722	84.4	71.0	0.149	0.321
LGM	Multi-view reconstruction	58.009	71.0	52.3	0.089	0.215
VGGT	Multi-view reconstruction	34.686	81.3	65.4	0.103	0.211
Hunyuan3D 2.0-MV	Multi-view conditioned	33.954	82.4	67.1	0.147	0.301
ReconViaGen	Multi-view conditioned	32.055	85.1	72.7	0.165	0.345
ROAR-3D (Stage 1)	Multi-view conditioned (coarse)	25.372	88.7	76.8	0.168	0.346
ROAR-3D (Stage 2)	Multi-view conditioned (full model)	21.039	91.2	81.6	0.173	0.349

Ablation Study

The contribution of each individual component is systematically analyzed on the first-stage model across 50K training iterations, starting from a baseline that naively concatenates multi-view tokens:

Config	CD↓	F1(0.1)↑	F1(0.05)↑	ULIP-I↑	Uni-I↑	Note
Baseline	39.412	79.6	66.3	0.154	0.322	Naive token concatenation
+ Token-wise View Router	28.143	86.3	73.5	0.159	0.335	Resolves multi-view conflict; CD drops by 11.27
+ Dual-Stream Attention	26.732	87.5	75.1	0.163	0.339	Decouples orientation from geometry transfer
+ Orientation Perturbation (Ours)	25.372	88.7	76.8	0.168	0.346	Eliminates orientation bias; optimal stage 1

Key Findings

• The view router is the primary performance driver: Introducing the token-wise view router over naive concatenation reduces CD from 39.412 to 28.143 (a 28.6% drop), proving that per-token selection resolves visual feature conflict across diverse viewing perspectives.
• Orientation decoupling prevents symmetric artifacts: Qualitative observations on asymmetrical objects (e.g., fighter jets and motorcycles) confirm that without dual-stream cross-attention and orientation perturbation, models replicate front nose structures onto tailplanes or duplicate wheels. The proposed decoupling eliminates these orientation-entanglement errors.
• Robust test-time view scaling: Although trained with only 1 to 4 auxiliary views, the router's per-token isolation allows linear scaling to 12+ uncalibrated views at test time, yielding monotonic improvements in geometric reconstruction accuracy.

Highlights & Insights

• 3D latent tokens as natural routing anchors: Using 3D geometric tokens as queries to match pooled 2D patch features bypasses the need for 2D pixel-level correspondence matching, exploiting the pretrained cross-modal knowledge already embedded in the DiT.
• Dual-stream role separation: Assigning a single reference view to lock the coordinate frame while treating auxiliary views strictly as structural donors resolves the fundamental contradiction between multi-view pose variability and 3D coordinate consistency.
• Orientation perturbation as shortcut prevention: Deliberately rotating the latent representation and disabling primary view privileges forces the auxiliary branch to learn pure content-level geometric correspondence, offering a practical training paradigm for pose-free 3D tasks.

Limitations & Future Work

• Reliance on primary view selection: The global coordinate frame depends entirely on the designated primary image; if the primary view suffers from extreme perspective distortion or severe occlusion, overall shape orientation may degrade.
• Extreme occlusions and ultra-thin structures: Deep internal cavities and microscopic thin-walled components uncaptured across all input views remain bound by the resolution limit of the discrete geometric VAE.
• Joint material and texture modeling: The current pipeline focuses primarily on geometric mesh reconstruction, leaving joint PBR material synthesis and relightable reflectance modeling for future extensions.

Related Work & Insights

• vs ReconViaGen: ReconViaGen relies on an external VGGT reconstruction network to feed geometric priors into a diffusion backbone, suffering from domain gap discrepancies and reconstruction bottlenecks; ROAR-3D requires no external reconstruction backbones, operates end-to-end, and delivers a 34% lower CD (21.039 vs 32.055).
• vs LGM / InstantMesh: Feed-forward reconstruction models require pre-calibrated canonical camera configurations; ROAR-3D natively processes arbitrary uncalibrated views, rough sketches, and video frames while preserving generative completion capabilities.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant token-wise view routing and decoupled dual-stream conditioning for pose-free multi-view 3D generation
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous benchmark evaluation on Objaverse, uncurated Anyview-200 testing, and thorough ablations
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive motivation, clear problem formulation, and structured technical explanations
• Value: ⭐⭐⭐⭐⭐ Highly practical for concept art, design sketch workflows, and downstream 3D asset production