Geo-ID: Test-Time Geometric Consensus for Cross-View Consistent Intrinsics¶

Conference: ECCV 2026
arXiv: 2603.13859
Code: None (project page includes release notes https://alaradirik.github.io/geoid/)
Area: 3D Vision / Intrinsic Image Decomposition
Keywords: Intrinsic Image Decomposition, Multi-View Consistency, Diffusion Models, Test-Time Geometric Guidance, 3D Correspondence

TL;DR¶

Geo-ID proposes a training-free, test-time framework that couples the predictions of independently run pre-trained single-view intrinsic diffusion models using sparse 3D geometric correspondences. It aggregates these predictions into a confidence-weighted, voxel-level robust consensus, which is back-projected and injected as a guide into the subsequent diffusion sampling process via gradient updates on latent variables. This enforces consistency in albedo, roughness, and metallicity across views without degrading single-view decomposition quality, and enables direct use in downstream editable neural scene representations.

Background & Motivation¶

Background: Intrinsic Image Decomposition aims to recover light-independent surface properties—albedo, roughness, and metallicity—from a single image. Recent diffusion models (such as RGB↔X, Marigold IID Appearance, and PRISM) have achieved high-quality single-view predictions, demonstrating strong generalization to diverse scenes. However, when these models are independently applied to different views of the same scene, the same surface area is often assigned different albedo or roughness values, leading to cross-view inconsistency.

Key Challenge: Single-view intrinsic decomposition is a highly ill-posed problem where geometry, illumination, and materials are deeply entangled; infinitely many equivalent decomposition solutions exist for a single image. Varying illumination directions and occlusions across different viewpoints induce the model to make different "decomposition decisions". This inconsistency severely hinders downstream applications (e.g., editable neural scene representations, 3D reconstruction) which require mutually consistent material predictions in 3D space.

Limitations of Prior Work: Video-based methods (e.g., Diffusion Renderer, Ouroboros) regularize predictions through temporal coherence, but they require dense, ordered sequences and cannot handle sparse, unordered image sets. Inverse rendering pipelines (e.g., NeRFactor, GS-IR, Intrinsic Image Fusion) optimize geometry and materials per scene, which relies on dense capture and precise geometry and takes over an hour per scene. Multi-view feed-forward networks (e.g., IDT, IDArb) require designing specialized architectures on multi-view training data, limiting their generalization on in-the-wild scene-level data.

Core Idea: The key insight of Geo-ID is that sparse 3D geometric correspondences are sufficient to resolve cross-view intrinsic ambiguities, even when the estimated geometry is noisy or incomplete. Instead of modifying the training process, it couples pre-trained single-view diffusion models during the inference phase using geometrically guided consensus constraints, achieving consistent predictions without modifying any model parameters.

Method¶

Overall Architecture¶

Geo-ID operates entirely at test time and is compatible with any existing diffusion-based intrinsic prediction model. The pipeline is divided into three stages, requiring only a sparse, uncalibrated multi-view image collection (4–32 images) of the same scene as input:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Images<br/>{I_1,...,I_V}"] --> B["Stage 1: 3D Geometric Guidance<br/>Sparse Correspondence Estimation"]
    B --> C["Stage 2a: Initial Intrinsic Prediction<br/>Independent Per-View Diffusion"]
    B --> D["Stage 2b: Voxelization<br/>Robust Consensus Aggregation"]
    C --> D
    D --> E["Stage 3: Consensus Guidance<br/>Diffusion Sampling Strategy"]
    E --> F["Consistent Intrinsic Decomposition<br/>{Ŷ_1,...,Ŷ_V}"]

Geometrically Guided Correspondence Estimation: A feed-forward geometric Transformer (VGGT) is used to predict the world-coordinate point cloud, depth map, camera pose, intrinsics, and pixel-wise confidence for each image. High-confidence 3D points are retained, and their source views and pixel locations are recorded.
Voxelized Consensus Computation: The base diffusion model is first run independently to generate initial intrinsic predictions for each view. High-confidence 3D point clouds are partitioned into axis-aligned voxels. Within each voxel, intrinsic predictions from different views are collected, and a robust consensus is computed using a confidence-weighted median, while the dispersion is estimated as an uncertainty metric to filter out outliers.
Consensus-Guided Diffusion: The diffusion sampling process is re-run for each view. During the last 80% of denoising steps, a gradient update (Huber loss) based on the consensus target is applied to the current latent variables, forcing the predictions to converge toward the consensus without deviating excessively from the prior of the base generator.

Key Designs¶

1. 3D Geometric Guidance for Sparse Correspondence Estimation: Replacing Traditional Matching with a Feed-forward Geometric Model

The core prerequisite for Geo-ID is solving "where the same surface is across different views". Traditional multi-view stereo (e.g., COLMAP) requires dense images and is slow, making it incompatible with sparse, unordered inputs. Geo-ID employs VGGT—a feed-forward geometric Transformer—to directly regress world-coordinate point clouds and camera parameters from single images without pairwise matching or iterative optimization.

Specifically, VGGT outputs for each image \(I_i\): a point cloud \(P_i \in \mathbb{R}^{H\times W\times 3}\), depth map \(D_i\), camera extrinsic \(E_i\), intrinsic \(K_i\), and pixel-wise confidence maps \(\sigma^P_i\) and \(\sigma^D_i\). All 3D quantities are registered in the coordinate system of the first frame. Geo-ID retains high-confidence points where \(\sigma^P_i \geq \tau_c\) (default 0.35) and discards unreliable correspondences. This ensures that even in difficult regions like glass or specular surfaces, only geometrically reliable pixels participate in subsequent consensus constraints, laying a robust foundation for the pipeline. When the number of available images exceeds the designated view count \(V\), they are randomly subsampled. The entire geometric estimation is a single feed-forward pass, taking only around 6 seconds for 16 views.

2. Voxelized Robust Consensus Aggregation: Weighted Median + Outlier Rejection + Uncertainty Modeling

Extracting a reliable consensus from \(V\) potentially inconsistent or erroneous initial predictions is a major challenge. The key innovation of Geo-ID is performing voxelized aggregation in 3D space rather than feature matching in 2D image space, which fundamentally avoids drift accumulation typical of 2D tracking under sparse, unordered settings.

After merging the high-confidence point clouds from all views, they are partitioned into axis-aligned voxels with adaptive edge lengths: \(\delta = \alpha \cdot \tilde{d}\) (where \(\tilde{d}\) is the median nearest-neighbor distance, default \(\alpha=2.5\)), resulting in an average of approximately 6.2 observations per voxel. Voxels with fewer than \(n_{\text{min}}=2\) views are discarded. In each voxel \(v\), intrinsic predictions \(\hat{Y}^{(0)}_j(u^j_v)\) from different views \(j\) are gathered, and the consensus is computed using a weighted median:

\[s_v = \operatorname{weighted-median}_{j\in\mathcal{V}(v)} \hat{Y}^{(0)}_j(u^j_v)\]

The weights \(w^j_v = \sigma^P_i(u^j_v) \cdot \log(1+|\mathcal{V}(v)|)\) combine two sources of information: geometric confidence from VGGT (giving more weight to reliable 3D points) and the number of observing views (more viewpoints indicate richer information). Simultaneously, a robust dispersion is computed as \(\hat{\sigma}_v = 1.4826 \cdot \operatorname{median}_j |\hat{Y}^{(0)}_j(u^j_v) - s_v|\), which serves a dual purpose: (a) observations deviating from the consensus by more than \(3\hat{\sigma}_v\) (or a scale of \(\hat{\sigma}_v\)) are excluded from guiding; (b) the inverse variance \(\hat{\sigma}_v^{-1}\) is used as the consensus confidence weight—giving higher weight to voxels with strong consensus (low dispersion). Finally, the voxel centers are projected back to each view using the known camera parameters. Visibility is verified via depth consistency (with a tolerance \(\epsilon=0.05\)), yielding a sparse target set \(\{(u^i_v, s_v, w^i_v)\}\) for each view.

Ablation studies show that the weighted median outperforms the mean (which is sensitive to outliers and degrades decomposition quality), and 3D voxel aggregation outperforms VGGT's 2D tracking head (which suffers from severe drift under sparse, unordered inputs).

3. Consensus-Guided Diffusion Sampling: Phased Insertion of Sparse Constraints

With the sparse consensus targets established, Geo-ID's strategy is to inject constraints as gradient updates during the mid-to-late stages of diffusion sampling. Specifically, the diffusion process is re-run independently for each view. At selected denoising steps \(t \in \mathcal{S}\) (the last 80% of steps), the clean intrinsic map \(\hat{Y}_{t,i}\) is estimated from the current latent \(x_{t,i}\) using Tweedie's formula, and the consensus loss is computed:

\[\mathcal{L}_{t,i} = \sum_{v: m^i_v=1} w^i_v \cdot \rho\big(\hat{Y}_{t,i}(u^i_v) - s_v\big)\]

where \(\rho\) represents the Huber loss (threshold \(\delta_H=1.0\)) and \(w^i_v\) is the confidence weight. The latent variable is then modified via a single-step gradient update: \(x_{t,i} \leftarrow x_{t,i} - \eta_t \nabla_{x_{t,i}}\mathcal{L}_{t,i}\), after which the standard diffusion denoising step proceeds. All model parameters are frozen, and only the latent variables are manipulated.

There are two key design choices: (a) Applying guidance only during the last 80% of steps—early denoising steps determine the global structure and material layout, and premature constraints would interfere with the generation prior, degrading decomposition quality (ablation shows full guidance drops PSNR from 16.4 to 15.8); applying constraints in the mid-to-late steps performs "alignment correction" on top of the already formed structure. (b) Applying gradients purely to the latent variables—this allows Geo-ID to run entirely as a wrapper during inference without any training or fine-tuning.

An Illustrative Example¶

Consider an input of 16 indoor scene images. VGGT outputs a \(512\times512\) point cloud for each image. Filtering out low-confidence points retains about 5000–8000 3D points. After voxelization, around 800–1200 voxels satisfy the minimum threshold of 2 viewing angles. Taking a voxel on a wall surface as an example: it is visible in 8 views, with initial albedo predictions of [0.72, 0.75, 0.68, 0.73, 0.81, 0.70, 0.74, 0.76]. The weighted median yields 0.73, with a dispersion of \(\hat{\sigma}=0.028\). Due to this low dispersion, this voxel receives a high confidence weight. Once these consensus values are projected back to each view, they guide each view's prediction via gradients during steps 10 to 50 of the second-phase diffusion sampling (the last 80% of 50 total DDIM steps). Ultimately, the wall's albedo stabilizes around 0.73 across all 16 images, whereas the unguided baseline predictions fluctuated between [0.68, 0.80].

Loss & Training¶

Geo-ID is a training-free method and has no traditional training loss. However, the consensus-guided process utilizes an online gradient loss—the Huber consensus loss (Eq. 3). Key hyperparameters: learning rate \(\eta=10\) (for RGB↔X with 50 DDIM steps) or \(\eta=50\) (for Marigold Appearance with 10 LCM steps—fewer steps require stronger single-step correction), using the Adam optimizer with only one gradient update per step; Huber threshold \(\delta_H=1.0\); and all model parameters frozen. Additionally, 20% of voxels are reserved as a hold-out set for consistency evaluation.

Key Experimental Results¶

Main Results¶

Cross-view consistency comparison. MAD (Median Absolute Deviation) is evaluated on the hold-out voxels, where lower values indicate better cross-view consistency. Geo-ID consistently improves performance across all datasets and view counts:

Method	Views	Mip-NeRF Ind.			Mip-NeRF Out.			T&T
		Alb.↓	Rou.↓	Met.↓	Alb.↓	Rou.↓	Met.↓	Alb.↓	Rou.↓	Met.↓
RGB↔X (base)	32	0.114	0.100	0.206	0.070	0.144	0.192	0.109	0.105	0.219
+ Geo-ID	4	0.110	0.074	0.138	0.067	0.121	0.115	0.106	0.096	0.195
+ Geo-ID	8	0.107	0.072	0.128	0.061	0.118	0.101	0.102	0.089	0.180
+ Geo-ID	16	0.103	0.071	0.115	0.058	0.104	0.097	0.100	0.082	0.174
+ Geo-ID	32	0.098	0.065	0.114	0.054	0.079	0.085	0.097	0.076	0.166
Marigold (base)	32	0.091	0.096	0.111	0.073	0.080	0.070	0.098	0.100	0.118
+ Geo-ID	4	0.086	0.088	0.101	0.067	0.072	0.062	0.089	0.090	0.104
+ Geo-ID	8	0.081	0.086	0.103	0.064	0.069	0.059	0.085	0.085	0.099
+ Geo-ID	16	0.078	0.079	0.099	0.063	0.064	0.049	0.083	0.082	0.097
+ Geo-ID	32	0.076	0.082	0.100	0.061	0.062	0.044	0.082	0.080	0.095

Preservation of single-view decomposition quality. While Geo-ID significantly enhances consistency, it does not degrade single-view prediction accuracy. For instance, on the synthetic InteriorVerse dataset (with ground truth):

| Method | Views | InteriorVerse Albedo | | | Rou. | Met. | HyperSim Alb. | | | | | | PSNR↑ | SSIM↑ | LPIPS↓ | RMSE↓ | RMSE↓ | PSNR↑ | SSIM↑ | LPIPS↓ | | RGB↔X (base) | 1 | 16.4 | 0.78 | 0.19 | 0.44 | 0.38 | 17.4 | 0.81 | 0.18 | | + Geo-ID | 32 | 16.4 | 0.78 | 0.19 | 0.42 | 0.35 | 17.3 | 0.81 | 0.18 | | Marigold (base) | 1 | 19.5 | 0.85 | 0.19 | 0.20 | 0.25 | 18.5 | 0.78 | 0.20 | | + Geo-ID | 32 | 19.5 | 0.86 | 0.19 | 0.19 | 0.23 | 18.5 | 0.78 | 0.20 |

Most metrics remain within \(\pm 0.1\) dB (PSNR) and \(\pm 0.01\) (SSIM) of the baseline, with some showing slight improvements (e.g., Marigold's roughness RMSE drops from 0.25 to 0.23), indicating that multi-view consensus signals can correct certain single-view errors.

Ablation Study¶

16 views, RGB↔X base model, consistency on MipNeRF-360 indoor + decomposition quality on InteriorVerse:

Configuration	Alb. MAD↓	Rou. MAD↓	Met. MAD↓	PSNR↑	Notes
Full Model (Geo-ID)	0.103	0.071	0.115	16.4	Weighted median + Outlier rejection + Last 80% steps guidance
Mean instead of Median	0.098	0.066	0.099	16.1	Slightly better consistency but degraded decomposition quality due to outlier sensitivity of mean
No Outlier Rejection	0.100	0.067	0.112	16.2	Similar trend: slightly better consistency but degraded quality
100% Steps Guidance	0.096	0.067	0.106	15.8	Best consistency but worst decomposition quality (PSNR −0.6dB)
Last 20% Steps Guidance	0.111	0.093	0.188	16.4	Guidance is too weak; metallicity shows almost no improvement
2D Tracking Alternative	0.109	0.074	0.135	16.2	Severe drift in 2D tracking, offering insufficient constraint
Baseline (No Guidance)	0.114	0.100	0.206	16.4	Independent prediction with no constraints

Key Findings¶

More views yield better consistency: From 4 \(\rightarrow\) 8 \(\rightarrow\) 16 \(\rightarrow\) 32 views, MAD decreases almost linearly without saturation, demonstrating that the sparse constraints remain effective across the studied view range. Using ground-truth geometry on InteriorVerse yields similar gains, ruling out the concern that the method only aligns with VGGT's geometric bias.
Geometric quality directly determines the magnitude of improvement: A scene-by-scene analysis on Tanks&Temples shows a strong correlation (Pearson \(r=0.94\)) between VGGT geometric F-score and consistency improvement. High-quality geometry scenes (e.g., Barn, F=0.82) achieve over 12% reduction in MAD, while lower-quality scenes (e.g., Courthouse, F=0.48) improve by only 5%. Crucially, Geo-ID does not degrade performance on any scene, as outlier rejection and confidence weighting provide adequate protection.
Guidance timing is critical: Full guidance (100% of steps) yields the best consistency but significantly degrades decomposition quality (PSNR drops from 16.4 to 15.8), as the early denoising steps responsible for structure are corrupted by external constraints. Applying guidance only in the last 20% of steps is too weak, showing almost no improvement in metallicity. Guidance in the last 80% of steps serves as the key balance point.
3D voxel aggregation outperforms 2D tracking: Replacing the 3D voxel solution with VGGT's 2D tracking head degrades consistency (albedo MAD 0.109 vs. 0.103, metallicity 0.135 vs. 0.115) because 2D tracking accumulates drift in sparse and unordered scenes and relies on visibility-ordering assumptions frequently violated by unordered inputs.

Highlights & Insights¶

Philosophy of "Sparse Correspondence over Dense Matching": Geo-ID demonstrates that constraining fewer than 1000 voxels in 3D space is sufficient to resolve cross-view intrinsic ambiguities, dismantling the conventional assumption that dense correspondences or complete geometry are required. This makes the method highly applicable to sparse, unordered real-world image collections.
Two-tier Robust Mechanism with Weighted Median + Inverse Variance Confidence: The framework simultaneously accounts for geometric confidence (VGGT's \(\sigma^P\)) and prediction uncertainty (intra-voxel dispersion \(\hat{\sigma}_v\)), automatically downweighting regions with poor geometry or inconsistent predictions. This two-tier filtering ensures that the method never degrades, even in scenes where geometric quality fluctuates by \(\pm50\%\).
Two-phase Diffusion (Exploration then Controlled Convergence): The model is first allowed to generate freely to establish initial consensus targets, and then re-sampled under consensus constraints. This preserves the richness of the base generative prior, avoids mode collapse, and naturally provides a way to perform unbiased evaluations on hold-out voxels.
Wrapper Design with Frozen Parameters and Latent-only Manipulation: Geo-ID can be plugged directly into any diffusion-based intrinsic predictor. At test time, the total overhead is only about 3 minutes for 16 views when using Marigold Appearance (joint prediction of three modalities), which is orders of magnitude faster than inverse rendering approaches (>1 hour).

Limitations & Future Work¶

Inherited Upstream Failure Modes: When VGGT produces severe mismatches on textureless or repetitive texture regions, the consensus target becomes unreliable, which may degrade prediction quality. The authors do not propose an active geometric quality detection or downweighting mechanism to address this.
Inability to Correct Systematic Biases: If the base intrinsic predictor systematically misinterprets shadows as albedo (e.g., Marigold Appearance on dark metallic surfaces), the geometric consensus will reinforce rather than correct this error, since the shadow remains geometrically consistent across viewpoints.
Voxel Resolution Limits: Fine geometric details smaller than voxel size \(\delta\) (such as railings or thin decorative lines) are merged into neighboring surfaces, causing local blurring. Reducing \(\alpha\) partially alleviates this but reduces consensus stability.
Currently Restricted to Diffusion Models: The method relies on the progressive denoising mechanism of diffusion models to inject guidance, making it not directly applicable to non-diffusion predictors (such as direct ViT regression). However, the core idea of geometrically guided consensus can in principle be adapted.
Downstream Quality Capped by Base Models: Current diffusion intrinsic predictors are biased toward indoor Lambertian scenes, which determines the absolute upper limit of accuracy for any test-time method, including Geo-ID.

vs. Video-based Methods (Diffusion Renderer / Ouroboros): Video methods rely on dense, ordered sequences, and their performance drops sharply on unordered inputs (e.g., Diffusion Renderer's albedo MAD increases from 0.068 to 0.104, almost a \(2\times\) degradation, on unordered 32-view sequences). Geo-ID is permutation-invariant, meaning unordered inputs perform just as well as ordered ones, presenting a significant advantage. However, video-based methods still yield better temporal smoothness in dense, sequential scenarios.
vs. Multi-view Feed-forward Networks (IDT / IDArb): IDArb requires training specialized diffusion architectures on large-scale multi-view synthetic datasets, and its cross-view attention is only effective for isolated objects. Geo-ID requires no training, directly reuses single-view models, and generalizes better to scene-level data.
vs. Inverse Rendering Pipelines (NeRFactor / Intrinsic Image Fusion): Inverse rendering methods achieve higher final fidelity but require precise geometry and dense capture, taking up to an hour per scene. Geo-ID takes only 3 minutes from 16 sparse images, which is orders of magnitude faster and a more realistic option for fast prototyping and low-resource scenarios.
vs. Diffusion Guidance Techniques (Classifier Guidance / ControlNet): Classifier guidance requires training an auxiliary classifier, while ControlNet necessitates fine-tuning a copy of the model. Geo-ID merely performs single-step gradient updates on latent variables during inference, demanding no training data or parameter duplication, thus maintaining minimal overhead.

Rating¶

Novelty: ⭐⭐⭐⭐ The idea of introducing "test-time geometric consensus guidance" to intrinsic decomposition is elegant and simple. The consensus design using a weighted median and inverse variance is far more sophisticated than naive averaging, showing medium-to-high innovation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The evaluation covers 4 datasets (real + synthetic, indoor + outdoor), 3 base models, 4 view counts, and 6 ablation variants. The appendix also features scene-by-scene analyses, quantitative correlations with geometric quality, and full hyperparameter sensitivity curves, exhibiting excellent engineering execution and transparency.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is clear and compelling, the three-stage methodology is well-structured, and the transition from motivation to experiments to limitations flows seamlessly. The extensive appendix (scene-by-scene breakdowns, failure mode categorization, hyperparameter grid search) makes this a textbook example of CVPR/ECCV methodology paper writing.
Value: ⭐⭐⭐⭐ It addresses the practical pain point of adapting single-view models to achieve multi-view consistency without training or fine-tuning. It has direct application value for sparse smartphone captures, though the absolute downstream editing quality is still bounded by the accuracy of the base models.