Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantification¶

Conference: ECCV2026
Paper: ECCV Paper
Project: Auto3R
Area: 3D Vision
Keywords: active reconstruction, uncertainty quantification, Gaussian Splatting, depth reliability, scanning path planning

TL;DR¶

Auto3R learns uncertainty from the current 3D model's color and depth renders, selects the next scanning viewpoint through depth weighting and reliability reweighting, achieves 27.00 dB average PSNR on Objaverse with a final budget of 20 views, and extends view selection to continuous robotic scanning paths.

Background & Motivation¶

3D Gaussian Splatting (3DGS) can fit photographs quickly and render novel views, but it does not automatically determine where a camera should move next. When photographs repeatedly cover the front of an object, rear recesses, occluded structures, or reflective regions can remain inaccurate; photographing already well-observed regions adds acquisition and optimization costs. Active reconstruction therefore places acquisition inside a loop: train a model from existing photographs, identify informative viewpoints, capture them, and update the model. The crucial decision is not how to generate a better-looking image, but which candidate views are worth capturing before their real photographs are available.

FisherRF and Gauss-MI use Fisher information or mutual information to estimate acquisition value, connecting reconstruction state to view selection through approximate calculations. The paper argues that analytical metrics can diverge from actual rendering errors, particularly when materials, occlusion, and lighting produce complex failures. Learned approaches treat blur, ghosting, and texture distortion as evidence, but color alone has a blind spot: visually similar pixels may represent very different amounts of spatial coverage. Conversely, geometry alone can overlook reflections and color artifacts, so the two kinds of evidence are not interchangeable.

Auto3R starts from the color and depth actually rendered by the current model, learns where those predictions appear unreliable, and aggregates pixel evidence into a view score. An important complication is that depth is both a geometric cue and a potentially erroneous prediction during sparse reconstruction; it should not amplify color errors without qualification. Core Idea: use color uncertainty to locate appearance defects, depth to represent their spatial influence, and depth reliability to limit how incorrect geometry distorts view selection.

Method¶

Overall Architecture¶

The inputs are a small set of captured RGB images and their camera poses, while the reconstruction backend maintains an evolving set of 3D Gaussians. For every candidate camera location, the system renders an RGB image and a depth map from the current Gaussian representation; these are predictions, not previously unavailable real observations. A dual-branch uncertainty prior produces color and depth uncertainty maps, preserving the pixel locations of suspected errors. Depth-aware blending and depth-reliability reweighting convert these local predictions into a scalar view score, and the highest-scoring candidate is selected for real capture. After adding the photograph to the reconstruction training set, 3DGS continues optimization and the next round reevaluates candidates using updated renders rather than reusing initial scores. The robotic version changes the candidate unit from a single pose to a continuous path and uses path-sequence estimation to exploit neighboring frames; this is an acquisition-loop extension, not an extra mandatory stage in discrete-view experiments.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Existing images and poses<br/>Update 3DGS and render candidates"] --> B["Dual-branch uncertainty prior"]
    B --> C["Depth-aware blending"]
    C --> D["Depth-reliability reweighting"]
    D -->|Discrete viewpoints| F["Select and capture<br/>Add real observations"]
    D -.->|Robotic path extension| E["Path-sequence estimation"]
    E --> F
    F --> A
    G["Training: renders and ground truth<br/>Produce SSIM supervision"] -.-> B
    G -.-> E

Key Designs¶

1. Dual-branch uncertainty prior: learn missing observational support from rendering artifacts

Each 3DGS Gaussian has a position, covariance, color, and opacity, and projected contributions are composited in depth order. The same representation renders color and depth, but the two expose different failures: color can show ghosting or blurred textures, while depth can reveal distorted or discontinuous structures. The authors use two lightweight ResNet-50-based image-to-image networks to process rendered RGB and depth separately and predict pixel-wise uncertainty. The network does not directly output a camera pose or learn an independent posterior distribution for every Gaussian; its initial predictions are made in rendered image space. Geometric awareness primarily comes from depth inputs and subsequent weighted aggregation, so the claimed joint 2D–3D formulation should not be mistaken for an additional learned 3D occupancy field.

Supervision uses local SSIM between reconstruction renders and corresponding ground truth, associating visible artifacts with reconstruction errors. The authors call this self-supervised because no human uncertainty annotations are needed, but training still requires corresponding ground-truth images rather than being reference-free. Low SSIM indicates poor agreement, whereas the acquisition algorithm selects high-uncertainty views; these directions must be distinguished. The available main text does not fully specify the transformation from SSIM to network targets, target normalization, or the regression loss, so a particular error formula cannot be presented as an explicit author specification. At inference time, candidate ground truth is no longer required: the networks estimate defects from current renders, enabling active acquisition.

2. Depth-aware blending: equal pixel errors need not have equal spatial influence

Under perspective projection, distant Gaussians occupy smaller image regions and one pixel can mix more spatial structure; simply summing pixel errors can underweight distant regions. The authors therefore weight color uncertainty by depth rather than merely averaging the color network's output. With \(\mathbf{D}(u,v)\) denoting rendered depth and \(\mathcal{R}(u,v)\) denoting color uncertainty, source equation (3) is:

\[ l_{\mathrm{blend}}=\sum_{(u,v)}\mathbf{D}(u,v)\mathcal{R}(u,v). \]

This quantity is an aggregated candidate-view scoring term, not a network training loss, despite the source notation \(l\). It gives the same color error different contributions at different depths, but does not directly calculate the actual error reduction after capturing a view. The paper motivates depth dependence through projection geometry and experimentally compares linear and squared depth weighting; the final design uses linear weights. This is consequently a geometrically motivated scoring rule, not a proven optimal information-gain estimator.

3. Depth-reliability reweighting: prevent unreliable depth from distorting the score

With few early observations, rendered depth can itself be wrong; multiplying large depth values by larger weights may amplify structural artifacts rather than valuable acquisition directions. The second branch's depth uncertainty \(\mathcal{D}(u,v)\) suppresses this unreliable contribution, giving source equation (4):

\[ l_{\mathrm{blend*}}=\sum_{(u,v)}\mathbf{D}(u,v)\mathcal{R}(u,v)\bigl(1-\mathcal{D}(u,v)\bigr). \]

The distinction between \(\mathbf{D}\) and \(\mathcal{D}\) matters: the former is distance, while the latter expresses uncertainty about that distance prediction. Reliable depth preserves more of the geometrically weighted color-error contribution; unreliable depth reduces its contribution along this scoring route. This does not mean the algorithm always ignores depth-error regions, because the final score also contains the original blending term and global aggregates of the color and depth uncertainty maps. Source equation (5) has damaged symbol extraction in the supplied text, so its exact full formula is not reconstructed here; only the components and coefficient roles confirmed by the prose are retained. The prose associates \(\lambda_0\) with reweighted blending and \(\lambda_1\) and \(\lambda_2\) with the global color and depth uncertainty terms, respectively. The settings are \(\lambda_0=0.5\), \(\lambda_2=0.8\), and \(\lambda_1=0.8\times(\mathrm{iteration}/\mathrm{max\ iteration})\). The global color term gains weight over optimization, while the other two listed coefficients stay fixed; the combined scalar \(U\) ranks candidates and the highest-scoring viewpoint is selected.

4. Path-sequence estimation: make continuous acquisition during motion the decision unit

Capturing frames along a real robot arm's movement adds relatively little cost, whereas repeatedly moving and stopping at isolated poses can be expensive, so the best single view need not yield the most efficient scanning path. The authors construct candidate paths between nearby viewpoints and interpolate camera poses between endpoints to obtain rendered sequences. The path extension adopts the TMFI-Net encoder and adds an upsampling module for per-frame uncertainty prediction, allowing sequence context rather than isolated image processing. An overall uncertainty score selects a candidate path, the robot captures continuously along it, and the real sequence returns to the reconstruction training set. This changes the granularity of scoring and capture without changing the loop of rendering the current model, predicting weak regions, and adding real observations.

To train the path network, a subset of views again fits 3DGS, after which predicted and ground-truth frames are rendered along interpolated paths and pixel-wise SSIM supplies supervision. The main text describes the sequence input, encoder, and per-frame output, but does not provide a precise, reproducible formula for aggregating the path score. Candidate-path generation and robotic system details are deferred to Supplementary Material B, which is absent from the supplied full text, so collision checking and motion constraints cannot be filled in. The diagram's dashed connection represents extending the discrete UQ workflow to sequences, not a claim that TMFI-Net must consume discrete scalar scores as inputs.

A Worked Example¶

Consider an object with a recess as an explanatory scenario, not an additional quantitative case reported by the paper. With only a few initial photographs, the current Gaussian model may render the recess as a blurred plane, prompting the color network to identify texture inconsistency and the depth network to identify unreliable structure. Depth-aware blending first accounts for the pixels' spatial weights, and reliability reweighting then prevents incorrect depth from disproportionately inflating a candidate's contribution. Meanwhile, global uncertainty terms retain attention to overall defects, so reweighting is not equivalent to discarding every difficult region. After comparing candidate scores, the system captures the selected oblique side view and adds it to training, grounding the next scoring round in improved geometry and appearance. With the robot path extension, it instead compares rendered sequences spanning neighboring poses and captures multiple frames along the selected path rather than one photograph at its endpoint.

Loss & Training¶

The object-level prior uses 3,000 Objaverse objects from the MaterialAnything subset, rendered under randomized HDR environment lighting in Blender 3.2.2. Each object has 60 camera views, with 12–25 used to train 3DGS and the remaining 35–48 providing novel views for uncertainty supervision. Varying the reconstruction view count produces models at different quality levels, exposing the estimator to errors from coarse structure through more accurate reconstruction rather than a fixed noise regime. The scene-level prior uses map-free-reloc with the same protocol; the authors explicitly state that object training and testing categories do not overlap. Offline uncertainty-network training and repeated 3DGS optimization on a target object are separate layers of learning; the latter directly incorporates newly scanned photographs. The main experiments optimize 3DGS for 30,000 iterations, adding one photograph at each of 16 scheduled iteration points to reach 20 views in total. Thus, the main protocol does not replan after every gradient update, and the 12–25 offline reconstruction views should not be confused with the final 20 views at evaluation. The paper reports approximately 3 ms per image for uncertainty prediction, but this figure does not explicitly cover all candidate rendering, reconstruction optimization, and robot motion costs.

Key Experimental Results¶

Main Results¶

Object experiments randomly generate 256 candidate and 256 test viewpoints without supplying initial 3D points; scene experiments use Mip-NeRF360. PSNR is measured in dB and higher is better; higher SSIM and lower LPIPS are better; “worst” denotes the reported worst 5% of test views. The following excerpts come from source Table 1 on page 10 and Table 2 on page 11, both under active selection with a final total of 20 views; AVS is the authors' CNN reimplementation retrained under matched conditions.

Dataset	Method	PSNR avg	PSNR worst	SSIM avg	SSIM worst	LPIPS avg	LPIPS worst
Objaverse	FisherRF	23.08	17.85	0.8055	0.6320	0.1063	0.2689
Objaverse	AVS	23.94	18.18	0.8479	0.7232	0.0935	0.2191
Objaverse	Auto3R	27.00	21.88	0.8882	0.7819	0.0711	0.1813
Mip-NeRF360	FisherRF	18.74	10.32	0.5771	0.3299	0.2619	0.4788
Mip-NeRF360	AVS	18.95	9.28	0.6062	0.3709	0.2679	0.5185
Mip-NeRF360	Auto3R	19.13	10.75	0.6055	0.3862	0.2520	0.4374

On Objaverse, average PSNR exceeds AVS by 3.06 dB and worst-5% PSNR by 3.70 dB, supporting improvement on difficult viewpoints rather than only in the average. Scene results are closer: average PSNR is only 0.18 dB higher, and average SSIM is 0.6055 versus AVS's 0.6062, so superiority on every metric would be an overstatement. Source Table 3 on page 12 also evaluates the GIR inverse-rendering backend for specular objects: GIR + Auto3R obtains average/worst PSNR of 30.29/22.92, versus 29.53/20.36 for GIR + AVS. GIR uses adjusted view-addition and optimization schedules, so this demonstrates compatibility with material-aware reconstruction rather than attributing every cross-backend difference to view selection.

Ablation Study¶

The following values reproduce source Table 4 on page 13; the accompanying passage does not explicitly identify its test subset, so its absolute values should not be merged directly with Table 1's 27.00 dB. The six columns report average and worst-5% metrics under the same ablation protocol; depth2-blending uses squared depth weights.

Config	PSNR avg	PSNR worst	SSIM avg	SSIM worst	LPIPS avg	LPIPS worst
w/o depth-uq	23.41	17.95	0.8124	0.6494	0.1016	0.2570
w/o depth-blending	22.60	15.94	0.7910	0.5664	0.1085	0.2791
depth2-blending	23.59	17.60	0.8131	0.6388	0.1004	0.2593
Full model	23.77	18.49	0.8252	0.6703	0.0978	0.2145

The source contains an explicit conflict: Section 5.6 describes the no-depth-blending starting point as PSNR 22.48 and SSIM 0.7818, whereas the corresponding Table 4 row gives 22.60 and 0.7910. The prose also says linear depth weighting slightly outperforms squared weighting, but w/o depth-uq and depth2-blending have average PSNR of 23.41 and 23.59; without confirming that all other settings match, these rows are not a clean isolated linear-versus-squared comparison. This note retains the table values and discrepancies without silently reconciling them; one directly readable comparison is the full model's 0.36 dB improvement over w/o depth-uq.

Key Findings¶

The depth-blending-related ablation has a larger gap: the full model exceeds w/o depth-blending by 1.17 dB average PSNR and 2.55 dB worst PSNR; these are configuration differences, not additional independent experiments.
Depth-reliability reweighting further raises average SSIM from 0.8124 to 0.8252, supporting the need to assess depth reliability when using geometric weights.
In robotic Table 5 on page 14, Auto3R's Origin and Path-extension results are 22.91/28.56 dB and 0.8346/0.8953 SSIM; AVS's path-extension results are 26.98 dB and 0.8894 SSIM.
Robotic experiments begin with 4 images along a forward path, but the main text does not establish identical total frame counts, travel distances, and runtime across Table 5 configurations, so the entire gain cannot be called a matched-budget improvement.

Highlights & Insights¶

The uncertainty estimator inspects errors in the current model rather than relying only on object categories or a single initial photograph. Acquisition can consequently adapt as reconstruction evolves.
Depth serves both as a geometric weight and as a prediction that needs scrutiny. Separating those roles helps avoid treating unreliable geometry as a trustworthy prior.
Worst-5% metrics expose local failures caused by missing coverage. Since active scanning should reduce blind spots, average image quality alone can obscure an important objective.
A transferable idea is to predict local defects in an online system's outputs and define decision units around acquisition costs; applying it elsewhere still requires validating the connection between error and acquisition value.

Limitations & Future Work¶

The authors explicitly acknowledge interference from robot self-occlusion shadows, which may cause acquisition-induced appearance changes to be mistaken for reconstruction defects.
Reader assessment: an SSIM-driven error prior is not equivalent to true 3D accuracy; the main text primarily reports novel-view image metrics rather than direct geometric distances or surface completeness.
Reader assessment: prioritizing uncertain views does not explicitly predict their post-acquisition marginal benefit; difficult reflections could remain highly ranked despite limited potential for improvement.
Reproducibility boundary: the supplied full text excludes the cited supplementary material, leaving the SSIM target transformation, exact path aggregation, and robot motion constraints unresolved.
Evaluating or learning from actual post-acquisition error reduction together with scanning time is a possible next step, not an experiment already completed in this paper.

FisherRF [17] / Gauss-MI [45]: these estimate view value through Fisher information and Shannon mutual information, respectively; Auto3R mainly learns rendering errors and adds depth weighting, changing the evidence source rather than merely computation speed.
AVS [42] / PUN [49]: the paper characterizes the former as reference-image-based 2D quality estimation and the latter as single-input uncertainty prediction; Auto3R emphasizes current reconstruction state and depth reliability.
GIR [38]: GIR handles inverse rendering and material representation for specular objects, while Auto3R selects acquisition viewpoints; their combination should not be misread as Auto3R independently learning material decomposition.
TMFI-Net [50]: this supplies the video encoder adapted for per-frame uncertainty prediction along paths; citation numbers follow the source bibliography, and the project entry is linked above.

Rating¶

Novelty: 4/5. It combines a data-driven error prior, depth weights, and reliability correction for active acquisition, with innovation concentrated in formulation and integration.
Experimental Thoroughness: 4/5. Objects, scenes, specular materials, and robots are covered, but conflicting ablation descriptions and incomplete acquisition-budget reporting limit conclusion strength.
Writing Quality: 3/5. The main workflow is clear, but uncertainty supervision, table–text consistency, and path details require more precise treatment.
Value: 4/5. It offers directly useful ideas for reducing redundant 3D asset capture, with further validation needed for actual motion costs and geometric accuracy.