UniQueR: Unified Query-based Feedforward 3D Reconstruction¶

Conference: ECCV2026
Paper: ECCV 2026 / Project Page
Area: 3D Vision
Keywords: feedforward 3D reconstruction / 3D queries / Gaussian Splatting / pose-free reconstruction / novel view synthesis

TL;DR¶

UniQueR reformulates feedforward 3D reconstruction from per-pixel prediction into inference over a sparse set of 3D queries: 4096 learnable anchors with explicit 3D coordinates interact with multi-view features in a shared scene frame and each spawns 64 Gaussians, so a single forward pass turns unposed images into camera poses, point maps and renderable Gaussians — beating feedforward baselines on Mip-NeRF 360 / VR-NeRF in both rendering quality and depth accuracy with roughly 1/15 the number of primitives.

Background & Motivation¶

Feedforward reconstruction has largely settled into one pattern over the past two years. DUSt3R, VGGT and Pi3 showed that a single forward pass can regress 3D geometry straight from 2D observations, by feeding multi-view features into a transformer and producing 2.5D intermediates such as depth maps and point maps; MVSplat, NoPoSplat, FLARE and AnySplat then swapped the output for pixel-aligned or voxel-aligned 3D Gaussians, so geometry and appearance share one differentiable representation, rendering quality improves, and downstream operations such as relighting and editing become possible. The trouble lies in the granularity of the output: these primitives are generated on the image plane (or a voxel grid), one primitive per pixel or per cell, so their density and spatial coverage are pinned to the input views — they can only express observed surfaces. As soon as a novel view departs from the input cameras, there are no primitives left for occluded or unobserved regions, and the rendering shows holes and artifacts.

This stands in contrast to per-scene optimization. NeRF and 3D Gaussian Splatting do not grow their primitives on the image plane; their positions are optimized, so they can express geometry beyond visible surfaces — at the cost of re-optimizing every scene for minutes to hours, with no way to exploit priors learned from large-scale data. In other words, feedforward methods buy speed and generalization and pay in representational freedom: they hand the decision of "where do primitives go" to the sampling grid of the input views rather than to the network. Keeping the speed of feedforward while recovering the representational freedom of optimization requires decoupling the scene representation from any particular viewpoint.

This paper borrows its angle from the query paradigm in detection and scene understanding (DETR, DETR3D, PETR): if a detector can use a set of learnable queries to "find" objects in 3D space, reconstruction can let a set of queries "grow" primitives in 3D space directly. UniQueR therefore keeps a fixed budget of sparse queries (4096), each carrying an explicit 3D coordinate and a latent embedding, with coordinates living in a scene frame shared by all views rather than in any single camera frame; after interacting with multi-view image features, each query spawns a Gaussian cluster, and the primitives are rendered to RGB and depth by differentiable splatting for supervision — with no 3D ground truth required. Core idea: replace the scene representation "a function of pixels" with "a set of learnable queries carrying explicit 3D coordinates", and make the supervision views a superset of the input views (three input views plus three held-out views), which forces queries to allocate Gaussians to regions nobody observed.

Method¶

Overall Architecture¶

The input is \(N\) unposed RGB images of the same scene. A DINOv2 ViT first extracts per-frame tokens, an alternating-attention transformer (alternating intra-frame and inter-frame attention) aggregates cross-view information, and task decoders output three per-frame geometric annotations: a camera-to-world pose \(P_i\in SE(3)\), a local point map \(X_i\in\mathbb{R}^{3\times H\times W}\), and a confidence map \(C_i\). These point maps play a double role — they are both the geometric output to be reported and the geometric prior used later to initialize query coordinates.

The actual scene representation is a set of \(Q=4096\) learnable queries: each query has an explicit 3D coordinate \(p_i\) (living in a non-metric, per-scene gauge shared by all views — note this frame is not the first camera's frame) plus a high-dimensional latent embedding. The queries pass through 12 Query Transformer layers that interact with image tokens: each layer first performs query→image cross-attention, then self-attention among queries only. Each refined query spawns \(K=64\) Gaussians, giving \(Q\times K = 262\text{K}\) colored primitives; these are rasterized differentiably into RGB and depth and supervised against both the input views and the held-out novel views. Everything happens in one forward pass, emitting poses, point maps and a directly renderable set of Gaussians; if extra per-scene optimization is allowed, these Gaussians also serve as the initialization for test-time optimization (TTO), which pushes rendering quality up another notch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Unposed multi-view images"] --> B["DINOv2 + alternating attention<br/>predict pose / point map / confidence"]
    B -->|point maps as geometric prior| C["Hybrid-initialized<br/>sparse 3D queries"]
    B -->|Plücker ray positional encoding| D["Decoupled cross-attention<br/>query propagation"]
    C --> D
    D --> E["Query-spawned<br/>Gaussian splatting"]
    E --> F["Novel-view supervision<br/>render RGB and depth"]
    F --> G["Output: poses / point maps / Gaussians"]

Key Designs¶

1. Hybrid-initialized sparse 3D queries: let the network, not the pixel grid, decide where primitives go

The query set is the representational core of the paper: sparse (4096), explicitly located in 3D, and fixed in budget — it does not change with the number of input views or the image resolution, which is a fundamental difference from per-pixel methods (the image-token count still grows linearly with the number of views, the query count does not). Initialization is the first obstacle. Detection-style sparse queries learn sensible positions because box-level supervision constrains their coordinates directly; dense reconstruction only has 2D rendering losses and no 3D boxes, so purely random coordinates are unstable — the ablation shows this variant simply collapses (PSNR 12.11). UniQueR therefore uses a hybrid initialization: half of the query coordinates are sampled from the predicted non-metric point maps so they cling to observed surfaces and carry geometric grounding, while the remaining anchors are spread uniformly in the normalized 3D scene range. Uniformity applies to initialization only: after transformer refinement and the predicted query deformation, these anchors are free to leave their initial samples. The final deformed query is the center of a local Gaussian cluster, and neither the query centers nor the spawned Gaussians are constrained to sit on a surface — which is exactly the precondition for placing geometry into occluded regions.

2. Decoupled cross-attention query propagation: splitting the quadratic cost of concatenated self-attention

The most straightforward way to let queries absorb multi-view evidence is to concatenate query tokens and image tokens and run full self-attention, at a cost of \(O\big((Q + NT)^2\big)\) where \(T=HW/p^2\) is the token count per image. For 4096 queries plus high-resolution multi-view input this is prohibitive, and the concatenated image tokens would have to be discarded anyway (their updates are useless downstream). UniQueR instead splits each layer into two steps: queries first draw features from image tokens via cross-attention, then run one self-attention among themselves, reducing the cost to \(O(QNT + Q^2)\) — the image side and the query side go from "all-pairs" to "fetch what you need, then confer", and the resulting memory and latency gains show up directly in the end-to-end comparison (39% less GPU memory and 2.35× faster than AnySplat with 32 views). The accompanying positional parameterization matters just as much: image tokens are encoded with Plücker ray embeddings derived from the predicted poses, while queries naturally carry their own 3D coordinates as positional encodings, so the two sides interact in one shared 3D semantics rather than being aligned across two separate 2D coordinate systems.

3. Query-spawned Gaussian splatting: sparse queries growing dense primitives

A query cannot itself be a rendering unit — 4096 points are far too sparse to draw detail. Each query is therefore treated as the seed of a Gaussian cluster: its updated embedding first predicts a deformation \(\delta q_i\) that moves the cluster center away from the initial anchor to wherever the network deems appropriate; an MLP then decodes \(K\) local offsets \(\delta g_{ik}\) around that center, while attribute heads output opacity, scale, rotation and color. As a set,

\[ G = \bigcup_{i=1}^{Q}\bigcup_{k=1}^{K}\ \{\, p_i + \delta q_i + \delta g_{ik},\ \Sigma_{ik},\ c_{ik},\ \alpha_{ik} \,\} \]

(⚠️ Equation (8) of the original paper is garbled in the cached text; the form above is reconstructed from the prose — refer to the original paper for the exact notation.) This "sparse queries + local dense clusters" representation is the source of the paper's efficiency advantage: AnySplat needs 3.85M primitives at 32 views, UniQueR only 262K — 14.7× fewer — yet covers more completely because the positions are chosen by the network rather than fixed by the input grid. The absence of 3D ground truth is not a problem either: Gaussian Splatting acts as the differentiable bridge between 3D and 2D, and signals such as RGB and depth are far more available than 3D annotations, so supervision can be carried out entirely in the image domain.

4. Novel-view supervision: making the loss force Gaussians into places nobody saw

Representational freedom is only a necessary condition; something has to drive the model to use it. If supervision only rendered the input views, the model could cram all 4096 queries onto observed surfaces and still drive the training loss very low. UniQueR instead makes the supervision views a superset of the input views: given three input views, the model renders six — the three inputs plus three held-out training views — and compares all of them against the corresponding ground truth. Any missing appearance or geometry in a held-out view turns into a hole-shaped error in the rendering loss, and the only way to remove it is to allocate Gaussians to regions the input views cannot see. The elegance of the design is that it converts "fill in occluded regions" into a rendering objective that needs no 3D annotation: nobody has to tell the network where something is missing, because the target image of the held-out view exposes the gap automatically. The paper also confirms the direction of this signal in ablation — removing depth-rendering supervision drops PSNR from 20.23 to 19.96, indicating that geometric and photometric cues jointly stabilize query localization.

A Worked Example¶

Take three input views at Stage-1 resolution \(224^2\). At initialization, roughly 2048 query coordinates are sampled from the predicted point maps (clinging to observed surfaces) and roughly 2048 are spread uniformly over the normalized scene range; through 12 Query Transformer layers each query first draws features from image tokens encoded as Plücker rays, then confers once with the other queries; each query's embedding then predicts a deformation that relocates its cluster center and decodes 64 local offsets → 4096 × 64 = 262K Gaussians. These are rendered into 6 views (3 inputs + 3 held-out), and holes in the held-out views enter the loss directly. Stage 2 raises the resolution to \(448^2\) and fine-tunes for 20 epochs.

Loss & Training¶

The total loss combines a photometric rendering term, a scale-invariant rendered-depth term, and the backbone camera term:

\[ \mathcal{L} = \mathcal{L}_{\text{rgb}} + \lambda_d \mathcal{L}_{\text{depth}} + \lambda_c \mathcal{L}_{\text{cam}}, \qquad \lambda_d = 0.1,\ \lambda_c = 0.2 \]

Here \(\mathcal{L}_{\text{rgb}}\) consists of an \(\ell_1\) reconstruction term plus an LPIPS perceptual term in Stage 1; for the \(448^2\) Stage-2 fine-tuning only the \(\ell_1\) term is kept, because LPIPS substantially increases training time.

Training is staged: the DINOv2 encoder, the alternating-attention transformer and the point-map head are initialized from Pi3 and kept frozen (890.99M parameters), while the camera head, the 12-layer Query Transformer and the Gaussian spawning/attribute heads are trained (502.31M trainable parameters, 1.393B in total). The optimizer is AdamW with a learning rate of \(10^{-4}\) and cosine decay, training for 100 epochs at \(224^2\) on 32 A100 GPUs and then fine-tuning for 20 epochs at \(448^2\); each sample contains 2–64 input views while the query budget remains \(Q=4096\). The paper additionally offers an optional test-time optimization (TTO) protocol: with the predicted Gaussians as initialization and the predicted poses as input, the Gaussians are rendered back into the input views and a reconstruction loss against the original images is back-propagated, used for further gains in the dense-view setting.

Key Experimental Results¶

Main Results¶

Sparse-view novel view synthesis (input and test views are disjoint; five input-view subsets are averaged per scene and metrics are computed only on their held-out targets; inference time is one forward pass on a single A100 80GB):

Dataset	Method	PSNR↑	SSIM↑	LPIPS↓	Time (s)↓
Mip-NeRF 360	NoPoSplat	18.21	0.482	0.426	0.416
Mip-NeRF 360	AnySplat	20.08	0.606	0.274	0.279
Mip-NeRF 360	UniQueR	22.70	0.660	0.261	0.213
VR-NeRF	NoPoSplat	21.28	0.744	0.381	0.406
VR-NeRF	AnySplat	19.67	0.745	0.313	0.290
VR-NeRF	UniQueR	21.99	0.708	0.446	0.198

On RealEstate10K with two views (150 randomly sampled scenes, 2 input and 2 disjoint target views): FLARE 17.74 / 0.586 / 0.371, AnySplat 18.93 / 0.615 / 0.226, UniQueR 20.50 / 0.672 / 0.196 — leading on all three metrics.

Camera pose estimation (RRA@30 / RTA@30 / AUC@30, all at a 30-degree threshold):

Method	RealEstate10K RRA↑	RTA↑	AUC↑	Co3Dv2 RRA↑	RTA↑	AUC↑
Fast3R	99.05	81.86	61.68	97.49	91.11	73.43
CUT3R	99.82	95.10	81.47	96.19	92.69	75.82
VGGT	99.97	93.13	77.62	98.96	97.13	88.59
Pi3	99.99	95.62	85.90	99.05	97.33	88.41
UniQueR	99.99	95.44	83.69	99.05	97.44	88.52

The authors explain the slight gap to Pi3 as most likely due to more limited training data (Pi3 uses a private dataset) — this comparison should be read with that caveat.

Dense-view and per-scene optimization (Mip-NeRF 360, 64 input views; top rows are feedforward only, bottom rows are feedforward initialization followed by per-scene optimization):

Method	PSNR↑	SSIM↑	LPIPS↓
AnySplat (feedforward)	21.26	0.607	0.303
UniQueR (feedforward)	21.58	0.641	0.335
3DGS + AnySplat	23.71	0.664	0.266
3DGS + UniQueR	26.00	0.784	0.176
MipSplatting + AnySplat	23.84	0.675	0.257
MipSplatting + UniQueR	25.99	0.782	0.178

The contrast is even starker on VR-NeRF: at 32 views 3DGS + baselines reach 21.90 (AnySplat) / 21.74 (VGGT) while 3DGS + UniQueR reaches 27.03, and 28.56 at 64 views.

Ablation Study¶

Stage-1 (\(224^2\), three input views, Mip-NeRF 360) component ablation:

Config	PSNR↑	SSIM↑	LPIPS↓	Note
Full model (hybrid initialization)	20.23	0.713	0.182	—
(a) w/o depth rendering	19.96	0.718	0.234	depth cues stabilize geometry
(b) Point-map queries only	18.58	0.627	0.313	geometric grounding but limited coverage
(c) Random initialization only	12.11	0.259	0.574	collapses without 3D supervision

Efficiency and rendered-depth comparison (32 input views, \(448^2\), one A100):

Method	# Gaussians	GPU Mem.	Time (s)	Depth Abs Rel↓
AnySplat	3.85M	18.42 GB	4.63	0.062
UniQueR	262K	11.19 GB	1.97	0.038

Key Findings¶

Hybrid initialization is what makes optimization stable, not a nicety: random initialization fails outright (PSNR 12.11), while point-map-only is stable but clearly weaker than the hybrid (18.58 vs 20.23). The two anchor sources play different roles — point maps guarantee surface grounding and hence optimizability, while uniform anchors supply the spatial coverage that lets the network extrapolate into occluded regions.
Rendering quality and perceptual quality do not always move together: UniQueR has the best PSNR on both Mip-NeRF 360 and VR-NeRF at sparse views, but on VR-NeRF it loses SSIM / LPIPS to NoPoSplat (0.708 / 0.446 vs 0.744 / 0.381). This matches the authors' admission that compactness trades away some high-frequency perceptual detail, and points to the fixed query budget as the weak spot.
Feedforward-only loses to per-pixel methods in the dense setting, but serves as a far better initialization: at 32 / 64 views UniQueR's feedforward PSNR is slightly below AnySplat (21.51 / 21.58 vs 22.32 / 21.26), a gap the authors attribute to a nearly two-orders-of-magnitude difference in primitive count; yet once the predicted Gaussians initialize per-scene optimization, 3DGS + UniQueR leads across the board (26.00 / 0.784 / 0.176 on Mip-NeRF 360 at 64 views).
Pose accuracy cascades into per-scene optimization: on VR-NeRF, 3DGS + AnySplat is occasionally worse than standalone AnySplat; the authors attribute this to AnySplat's pose errors degrading the optimization, since its predicted Gaussians are aligned with its own predicted poses. UniQueR supplies both more accurate poses and a better Gaussian initialization, so this degradation does not appear.
Clear scaling behavior: query count, Gaussians per query (16 / 32 / 64) and model capacity all yield monotone gains, suggesting this representation has not yet hit a capacity ceiling.
Geometric quality: rendered depth Abs Rel drops from AnySplat's 0.062 to 0.038, consistent with the qualitative observation that AnySplat shows depth holes in occluded regions while UniQueR produces cleaner boundaries.

Highlights & Insights¶

Decoupling output granularity from the input sampling grid is the paper's central move: the implicit assumption of prior feedforward reconstruction was "one pixel, one primitive"; UniQueR replaces it with "one learned 3D anchor, one cluster of primitives," so density and coverage are no longer dictated by the input views. The same reframing transfers to any task where generated primitives are tied to an input grid — sparse point-cloud completion, SLAM map representations, dynamic scene reconstruction.
Novel-view supervision turns occluded-region completion into a purely 2D supervision problem: no 3D annotation is needed; as long as the supervision views strictly include targets beyond the input views, missing content automatically becomes visible error. It is an almost free trick reusable in any feedforward reconstruction or generation pipeline.
The observation that purely random sparse queries are unstable when only 2D rendering losses are available is valuable in itself: it quantifies how much DETR-style query learnability depends on box-level supervision, and explains why hybrid initialization is a requirement rather than an option.
Encoding queries with their own 3D coordinates and image tokens with Plücker rays is a quiet but essential design — it guarantees that query-to-image attention happens in one shared 3D semantics rather than across two mismatched 2D coordinate frames.

Limitations & Future Work¶

As the authors acknowledge: the formulation assumes static scenes and models no temporal dynamics; the fixed query budget can underrepresent highly complex scenes; the method depends on a large frozen geometry backbone; and it trades some high-frequency perceptual detail for compactness. The authors suggest adaptive query allocation, lighter backbones, controlled component ablations, and temporally persistent queries.
The frozen backbone carries a real inference cost: 890.99M frozen parameters out of 1.393B total, and a single forward pass still takes 1.97s on an A100 at 32 views — hardly real time, even though per-pixel methods such as AnySplat are only in the same order of magnitude despite far more primitives (4.63s).
The scene coordinate frame is non-metric (determined only up to a similarity transform, and not the first camera frame), so the predicted geometry has no absolute scale and needs an extra scale-recovery step before use in robotics or autonomous driving. The paper notes that a global similarity change does not alter the rendered supervision — convenient for training, a limitation in deployment.
The ablation is shallow: it is run only at Stage-1 (\(224^2\), three input views), with no component-level comparison for the fixed query count, the construction of the supervision superset (how many held-out views are best), or the Plücker positional encoding. The scaling curves in the figure report PSNR only, with no SSIM / LPIPS, so it is impossible to tell whether perceptual quality improves along with capacity.
Pose evaluation covers only RealEstate10K and Co3Dv2 and uses different training data than Pi3, so "comparable" cannot be read as a direct comparison; the paper is fairly honest about this, but readers should not treat it as a settled claim about pose accuracy.
Concrete improvements: replace the fixed query budget with complexity-adaptive allocation (e.g., driven by point-map confidence or image coverage); persist queries over time for dynamic scenes; distill away the dependence on the large frozen backbone.

vs DUSt3R / VGGT / Pi3: they regress per-pixel point maps in one pass (2.5D, density tied to input pixels), whereas UniQueR outputs Gaussians spawned by sparse queries with explicit 3D coordinates. UniQueR inherits this line's geometric priors (the point-map head is initialized from Pi3 and frozen) without inheriting its 2.5D output — a direct inheritance plus a local substitution; the price is keeping a large backbone.
vs AnySplat / NoPoSplat / FLARE / MVSplat: they predict pixel- or voxel-aligned Gaussians whose coverage matches the input views, and therefore show holes under novel views (the qualitative figures show AnySplat's blank RGB regions and depth holes). UniQueR decouples coverage through queries; the trade-off is fewer primitives and slightly lower metrics in the feedforward-only dense setting, but a much stronger initialization for per-scene optimization.
vs per-scene optimization (NeRF / 3DGS / Scaffold-GS): they can express geometry beyond visible surfaces but must re-optimize for every scene; Scaffold-GS's anchors are the closest idea to this paper's queries, but those anchors are optimized and remain per-scene reconstruction rather than feedforward prediction. UniQueR effectively makes anchor-style representations feedforward.
vs LRM / LVSM / SceneTok / TokenGS: they also represent scenes with learned tokens, but those tokens are latent, carry no explicit 3D coordinate, and are decoded into target views or generative representations. In UniQueR each token carries an explicit 3D location and spawns a local Gaussian cluster, with Plücker-ray-encoded attention collecting multi-view evidence — the key distinction from "latent token" work.
Insight: treating the choice of representation as a first-class design variable in feedforward reconstruction (rather than only touching the backbone or the loss) extends naturally to 4D dynamic reconstruction, editable scene representations, and robotic perception tasks that need joint pose and geometry output.

Rating¶

Novelty: ⭐⭐⭐⭐ Importing DETR-style sparse queries into dense feedforward reconstruction with novel-view supervision is a clean reframing; but anchor/query-based 3D representations have precursors in Scaffold-GS and TokenGS, and the paper acknowledges concurrent work.
Experimental Thoroughness: ⭐⭐⭐⭐ Three datasets covering rendering, depth, pose and efficiency, plus scaling curves; but ablations are Stage-1-only at low resolution, with no component-level controls and no perceptual-metric scaling curves.
Writing Quality: ⭐⭐⭐⭐ The causal chain from motivation to method is crisp, and the comparison table (Tab. 1) makes the representational difference from per-pixel methods very clear; a few metric outcomes (SSIM / LPIPS on VR-NeRF) are mentioned but not explained.
Value: ⭐⭐⭐⭐ Achieving this level of rendering and geometric accuracy with 262K Gaussians is valuable, especially as an initialization for per-scene optimization; the non-metric frame and the large frozen backbone are practical obstacles before deployment.