2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction¶

Conference: ECCV 2026
arXiv: 2603.19964
Code: None (promised to be open-sourced upon acceptance)
Area: 3D Vision
Keywords: 2K Resolution, Sparse Refinement, Entropy-Guided Selection, Depth Estimation, 3D Reconstruction

TL;DR¶

This paper proposes 2K Retrofit, a generic framework that enables existing 3D foundation models (e.g., Depth Anything, VGGT) to output 2K-resolution geometric predictions without any architectural modifications or retraining of the base models. The core idea is to utilize a frozen base model for fast coarse predictions, and then employ an entropy-guided sparse refinement mechanism to apply full-resolution corrections only to high-uncertainty pixel regions. This design achieves or exceeds the accuracy of full-resolution methods while significantly reducing computational and memory overheads.

Background & Motivation¶

Background: High-resolution geometric prediction (depth estimation, 3D reconstruction) is a foundational capability for applications such as autonomous driving, embodied AI, and augmented reality. In recent years, 3D foundation models like Depth Anything V2, DUSt3R, and VGGT have demonstrated powerful generalization capabilities after training on large-scale datasets. However, they are generally limited by low-resolution training data (typical inference resolution is around 960x480) and cannot directly output fine 2K-level geometric predictions.

Limitations of Prior Work: To enable foundation models to output 2K-resolution results, existing solutions fall into three categories, each with its drawbacks: (1) directly training native 2K models—which incurs enormous memory and computational overheads (e.g., VGGT requires 76.5 GB VRAM and 495 GFLOPs for inference at 2K resolution); (2) upsampling after low-resolution inference—which is fast but fails to recover thin structures (e.g., railings, handles, cables); (3) patch-wise refinement—which requires multiple low-resolution inferences and complex patch fusion, introducing boundary artifacts and spatial inconsistency.

Key Challenge: There is a fundamental trade-off between high-resolution geometric accuracy and computational efficiency—full-resolution dense inference is too costly, while lightweight approximation methods lose critical high-frequency details.

Goal: (1) To enable frozen 3D foundation models to output 2K-resolution geometric predictions without any architectural modifications or retraining; (2) to achieve comparable or even superior accuracy at a computational cost far lower than full-resolution dense inference; (3) to ensure the method is generic across various backbone models for both monocular and multi-view scenarios, as well as depth and point map estimation.

Key Insight: The authors perform a systematic analysis of the error distribution of coarse predictions and find a key pattern—the error between low-resolution predictions and high-resolution ground truth is not uniformly distributed, but highly concentrated in sparse, semantically critical regions (object boundaries, thin structures, category transition zones). This implies that if refinement computations can be concentrated on this small subset of pixels, significant quality improvements can be achieved with extremely low additional overhead. Furthermore, the authors find that these high-error regions are highly correlated with prediction entropy (information entropy calculated from the backbone head features), making entropy a cheap and effective signal for deciding "whether to refine or not."

Core Idea: Use prediction entropy as a proxy metric for pixel-level uncertainty to identify a sparse subset of pixels (only ~10%) that need correction the most. Then, a lightweight sparse convolutional network is used to perform high-resolution residual correction only on these pixels. Finally, the coarse predictions and refined results are adaptively blended via gated fusion.

Method¶

Overall Architecture¶

2K Retrofit is a two-stage pipeline that decomposes high-resolution geometric prediction into "coarse prediction + sparse refinement": The first stage uses a frozen 3D foundation model for fast inference on downsampled low-resolution images, which is then upsampled to obtain a dense coarse estimation. The second stage localizes high-uncertainty pixels via an entropy selector, applies high-resolution residual correction only to these sparse pixels using a lightweight network, and finally outputs the high-precision geometric map through gated fusion. The entire process does not modify the base model, nor does it require retraining the base.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["2K Input Image I_HR"] --> B["Downsample to 256px<br/>Frozen Base Inference"]
    B --> C["Upsample to obtain coarse estimate<br/>Geo_HR_coarse"]
    C --> D["Entropy Selector<br/>Compute pixel-wise entropy"]
    D --> E["Filter high-entropy pixel subset P<br/>(~ top 10%)"]
    E --> F["Sparse Refinement Module<br/>MinkowskiUNet Residual Prediction"]
    F --> G["Gated Fusion<br/>Adaptive blending of coarse estimate + residual"]
    C --> G
    G --> H["Final 2K Geometry Output<br/>Geo_HR"]

Key Designs¶

1. Entropy Selector: Localizing Pixels Needing Refinement via Prediction Entropy

Coarse prediction errors vary dramatically across different regions—errors at object boundaries, thin structures, and occlusion edges are much larger than in flat areas. Performing full-resolution refinement on all pixels is computationally wasteful. The authors identify a crucial correlation: the information entropy of the backbone head features aligns highly with the mathematical prediction error (see Fig. 4), making entropy a cheap and effective signal for "which pixels require refinement."

Specifically, for the coarse prediction results, logits vector \(\mathbf{q}_p \in \mathbb{R}^C\) is extracted from the backbone head features (before the regression projection layer). After softmax normalization, the pixel-wise information entropy is computed as:

\[\mathcal{H}(p) = -\sum_{c=1}^{C} \mathbf{q}_p^{(c)} \log \mathbf{q}_p^{(c)}\]

By setting a threshold \(\alpha\), pixels with \(\mathcal{H}(p) > \alpha\) are selected to form the sparse subset \(\mathcal{P}\) for refinement. This step is effective because entropy naturally reflects the model's "hesitation" at a given pixel—features at boundaries, thin structures, and texture transitions often span multiple modes, resulting in high entropy, which corresponds precisely to the zones of maximum coarse prediction error. Experiments show that the entropy selector can cover approximately 80% of high-error pixels by refining only about 10% of total pixels, adding a latency of only ~2.3 ms (RTX 4090). This considerably outperforms random selection (ineffective), edge heuristics (insufficiently accurate), and learnable selectors (accurate but nearly twice as slow).

2. Sparse Refinement Module: Efficient Refinement on Irregular Pixel Sets via MinkowskiUNet

The selected pixels \(\mathcal{P}\) are irregularly distributed in image space; they do not form a dense grid but rather resemble a sparse 3D point cloud with uneven density, occlusion, and complex topology. For such inputs, standard CNNs would need to perform dense convolutions over the entire high-resolution feature map, wasting most computation on regions that do not require refinement. To solve this, the authors employ MinkowskiUNet (a sparse convolutional network) as the refinement module \(\mathcal{R}\), which executes sparse convolutions only at active positions and maintains sparsity throughout the network.

For each selected pixel \(p \in \mathcal{P}\), the refinement module extracts features from the corresponding high-resolution image patch and predicts a local residual correction:

\[\Delta \mathbf{Y}_p = \mathcal{R}(\mathbf{I}_{\mathrm{HR}}|_p), \quad \forall p \in \mathcal{P}\]

The module itself has a small parameter footprint (lightweight design) and operates only on sparse locations, keeping the inference latency at 2K resolution in the millisecond tier. Compared to Point Transformer, MinkowskiUNet achieves over 3x speedup with slightly lower accuracy (5.5 vs 1.7 FPS), which is key to achieving real-time 2K inference.

3. Gated Fusion: Adaptively Blending Global Consistency of Coarse Prediction and Local Accuracy of Sparse Refinement

After obtaining the sparse residual correction \(\Delta \mathbf{Y}\), the easiest approach is to directly overwrite the corresponding pixels in the coarse prediction. However, this introduces new issues: sparse refinement lacks global context, and direct replacement easily introduces local inconsistencies and noise. The authors design a pixel-wise gated fusion mechanism that adaptively assigns weights based on the respective confidence of the coarse prediction and refinement results.

For each refined pixel \(p\), the final prediction is defined as:

\[\mathbf{Y}_p = f\left(w_p \cdot \hat{\mathbf{Y}}_p + (1 - w_p) \cdot \Delta \mathbf{Y}_p\right)\]

where the fusion weight \(w_p\) is computed via a two-layer MLP with a sigmoid activation, taking the coarse prediction value, the refined value, and their respective entropies as inputs:

\[w_p = \sigma\left(\mathrm{MLP}\left([\hat{\mathbf{Y}}_p; \Delta \mathbf{Y}_p; \mathcal{H}(\hat{\mathbf{Y}}_p); \mathcal{H}(\Delta \mathbf{Y}_p)]\right)\right)\]

Intuitively, when the entropy of the coarse prediction is low (the model is confident about the pixel), the gate tends to preserve the global consistency of the coarse prediction. Conversely, when the entropy of the refinement is low (the refinement network is confident), the gate adopts more of the fine correction. Ablation studies confirm that direct replacement (Direct) performs poorly due to the loss of global context, pure entropy-weighted fusion (Entropy) improves performance but is still inflexible, and gated fusion (Gated) achieves the best accuracy-speed trade-off among the three.

A Complete Example: From 2K Image to High-Precision Depth Map¶

Taking a 1440x1920 indoor RGB image from the ARKitScenes dataset as an example: the image is first downsampled to a long edge of 256 pixels and fed into the frozen Depth Anything V2 backbone to obtain a low-resolution depth map. This is then upsampled back to 1440x1920 using nearest-neighbor interpolation to obtain a coarse depth estimate. Next, the pixel-wise entropy is computed from the backbone head features (~2.3 ms), filtering out approximately 10% (~276k) of high-entropy pixels with a threshold of \(\alpha=0.3\). These pixels are clustered around desk and chair edges, cabinet handles, and curtain folds. MinkowskiUNet extracts features and predicts depth residuals only for the 2K image patches corresponding to these pixels. Finally, gated fusion pixel-wise blends the coarse depth and the residuals. The final output is significantly superior to the coarse estimate in terms of object boundary sharpness and thin structure integrity, with an overall pipeline latency of about 8.1 FPS (RTX 4090), far surpassing full-resolution dense inference schemes.

Loss & Training¶

The training objective directly inherits the loss function of the base model used (e.g., the scale-invariant loss of Depth Anything V2) without additional design. Regarding training data, since there was no publicly available training set for 2K-level geometric prediction, the authors synthesized 50K high-quality rendered images using NVIDIA Omniverse, covering diverse geometric and texture scenes (see Appendix for details). Training configuration: 13 epochs, batch size 8, two NVIDIA RTX A6000 GPUs, completed in about one day. Inference latency was evaluated on a single RTX 4090 with FP16 precision. The base models (e.g., Depth Anything V2, VGGT) were frozen throughout, and only the entropy selector, sparse refinement module, and gated fusion components were trained.

Key Experimental Results¶

Main Results¶

Monocular 2K Depth Estimation, Zero-Shot Setting (trained only on synthetic data, no in-domain fine-tuning):

Method	ARKitScenes AbsRel↓	ARKitScenes δ0.5↑	ScanNet++ AbsRel↓	ScanNet++ δ0.5↑
MSPF	0.0149	0.9721	0.0226	0.9674
Depth Anything V2*	0.0326	0.9615	0.0371	0.9437
PromptDA	0.0131	0.9825	0.0175	0.9781
2K Retrofit (Ours)	0.0118	0.9866	0.0146	0.9825

Multi-view 2K Point Map Estimation (ETH3D Zero-Shot Test):

Method	Acc.↓	Comp.↓	Overall↓	FPS↑
DUSt3R	1.462	0.927	1.195	1.1
VGGT	1.167	0.639	0.903	2.6
StreamVGGT	1.140	0.613	0.870	3.5
2K Retrofit (Ours)	0.935	0.602	0.839	5.5

2K Retrofit reduces the Overall error of VGGT on point map estimation by approximately 7% while delivering an over 2x speedup (5.5 vs 2.6 FPS). If VGGT were retrained to 2K resolution, it would require 76.5 GB VRAM and 495 GFLOPs, whereas 2K Retrofit only requires 32.8 GB and 172 GFLOPs (both reduced by more than 50%). Its inference speed is 17x faster than the retrained version (5.5 vs 0.5 FPS, where the latter is extremely slow due to frequent page swapping from VRAM out-of-memory).

Ablation Study¶

Pixel Selector Comparison (Point Map Estimation on ETH3D):

Pixel Selector	Acc.↓	Comp.↓	FPS↑	Analysis
Random Selection	1.126	0.835	4.1	Ineffective, worse than no refinement
Edge Heuristic	0.942	0.625	4.1	Improved but not accurate enough
Bilateral Upsampling	0.940	0.617	4.0	Similar to edge heuristic
Learnable Selector	0.921	0.635	3.8	Slightly better accuracy but nearly twice as slow
Entropy Selector (Ours)	0.935	0.602	5.5	Best accuracy-speed balance

Fusion Strategy Comparison:

Fusion Strategy	Acc.↓	Comp.↓	FPS↑	Analysis
Direct Replacement	1.113	0.735	5.9	Severe loss of global context
Entropy-Weighted Fusion	0.940	0.607	5.0	Improved but inflexible
Gated Fusion (Ours)	0.935	0.602	5.5	Adaptive and optimal

Key Findings: - The entropy selector is the core prerequisite for the method's effectiveness—random selection failed completely (Acc. 1.126, which is even worse than the pure low-resolution baseline of VGGT at 1.167, indicating that unguided sparse refinement at 2K resolution is a negative optimization). This proves that "selecting the right pixels" is more critical than "how to refine them." - The benefit of gated fusion stems from its explicit modeling of uncertainty: trusting the coarse estimation in regions with high coarse prediction confidence, and adopting the residual in regions where the refinement network is confident, preventing inconsistencies caused by crude replacement. - The entropy threshold \(\alpha=0.3\) is the recommended trade-off: a lower threshold (\(\alpha=0.1\)) yields slightly higher accuracy (Acc. 0.917) but drops the speed to 3.4 FPS; a higher threshold (\(\alpha=0.8\)) is faster (8.0 FPS) but significantly degrades accuracy (Acc. 1.135). - In extreme scenarios such as textureless regions and highly reflective surfaces, errors become dense (no longer sparse), and the benefit of the sparse refinement mechanism decreases—this is a known blind spot of the method.

Highlights & Insights¶

Entropy as a free lunch: In most works, entropy is discarded after being used as an uncertainty quantification metric; this work turns it into a cheap signal for pixel selection—no extra network, no learnable parameters, locking high-error regions with 80% recall in 2.3 ms. This idea can be migrated to any dense prediction task requiring "global coarse prediction + local refinement" (semantic segmentation, optical flow, image super-resolution), as long as the coarse prediction network provides positional logits/feature vectors to compute entropy.
Sparse convolutions for irregular selection: Modeling irregularly distributed selected pixels as 3D point clouds and applying MinkowskiUNet for sparse convolution is an ingenious method migration. This avoids costly dense CNN filling on sparse pixel sets and bypasses the speed bottleneck of PointNet-like methods. Any scenario requiring "selective processing of sparse sub-regions in image space" can benefit from this approach of re-casting 2D sparse selection as a 3D sparse convolution problem.
Philosophy of "Retrofitting" rather than "Rebuilding": 2K Retrofit works entirely as a plug-in without modifying a single line of the base model's code or retraining its parameters. This means it is naturally compatible with any future 3D foundation model—one only needs to swap the frozen backbone when the model is upgraded. This "retrofit" design philosophy is highly practical in the era of rapidly iterating foundation models, and similar ideas can be extended to other tasks requiring low-resolution models to adapt to high-resolution scenarios.
Zero-shot generalization trained on synthetic data: Training only on 50K images synthesized by NVIDIA Omniverse achieves SOTA performance on three real-world datasets (ARKitScenes, ScanNet++, ETH3D). This demonstrates that what the sparse refinement mechanism learns—"where to refine and how to fuse"—is a cross-domain generic skill that does not rely on the specific appearance distribution of real data.

Limitations & Future Work¶

Degradation in extreme scenarios: On highly reflective surfaces, transparent objects, and extremely low-texture regions, prediction errors become dense instead of sparse, leading to a significant drop in the benefit of the sparse refinement mechanism. The authors acknowledge this limitation and suggest incorporating stronger geometric priors (e.g., multimodal information, physical constraints) to handle these scenarios in the future.
Dependency on the quality of backbone head feature entropy: The effectiveness of the entropy selector depends on whether the backbone head features indeed encode pixel-wise uncertainty. If the backbone head is designed without generating meaningful logits (e.g., direct regression without usable softmaxed features), the entropy selector will fail. This restricts the plug-and-play scope of the method, although current mainstream models like Depth Anything and VGGT are fortunately applicable.
Gap between synthetic and real domains: Although zero-shot generalization performs well, synthetic data still differs from real-world data in terms of lighting, materials, and sensor noise. Systematically studying the impact of synthetic data domain randomization strategies on generalization performance is a valuable direction for future work.
Unexplored dynamic scenes and multi-frame temporal consistency: The current method processes static single/multi-frame images and does not leverage temporal consistency in videos. Tracking sparse pixel selections across time could further reduce per-frame computational load and improve temporal stability.

vs SparseRefine (ECCV 2024): SparseRefine first introduced the concept of sparse refinement in semantic segmentation, selecting uncertain areas for high-resolution feature refinement. 2K Retrofit extends this idea from 2D semantic tasks to 3D geometric prediction. The key differences are: (1) using entropy instead of a learnable gate for pixel selection (faster); (2) using MinkowskiUNet sparse convolution instead of dense CNN for refinement (better suited for irregular distribution); (3) introducing gated fusion instead of simple overwriting.
vs PatchRefiner / PatchFusion (CVPR/ECCV 2024): Both are patch-based refinement schemes that require multiple low-resolution inferences and fuse overlapping patches, introducing boundary artifacts and high computational redundancy. 2K Retrofit only requires a single low-resolution inference and subsequently processes only the sparse pixel set, offering order-of-magnitude advantages in speed and memory.
vs PromptDA (CVPR 2025): PromptDA leverages a prompt mechanism to enable Depth Anything to output 4K depth, but still requires modifying the inference pipeline. 2K Retrofit employs a more thorough "wrapper-style" design, leaving the base model completely untouched, which offers stronger versatility.
vs Direct retraining of base models to 2K: Experiments clearly show that although retraining VGGT to 2K yields slightly better accuracy (Acc. 0.911 vs 0.935), it requires over 2x more memory and computation, with an inference speed that is only 1/17th. For practical deployment, "accurate enough + fast enough" is far more valuable than "most accurate + extremely slow."

Rating¶

Novelty: ⭐⭐⭐⭐ Introducing sparse refinement to 3D geometric prediction is novel in itself. The combination of using entropy as a pixel selection signal and MinkowskiUNet for irregular sparse refinement is ingenious and practical. However, the core "coarse-to-fine" two-stage pipeline and the entropy selection idea have precedents in semantic segmentation (SparseRefine), making this essentially a cross-task transfer rather than a completely new paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ It covers two major tasks: monocular depth estimation and multi-view point map estimation, using three real-world datasets (ARKitScenes, ScanNet++, ETH3D) along with synthetic training data. It compares more than 10 SOTA methods under both zero-shot and non-zero-shot settings. The ablation studies cover four dimensions: selectors, fusion strategies, thresholds, and feature extractors, with a systematic efficiency analysis of VRAM, FLOPs, and FPS—making the experimental design highly comprehensive and solid.
Writing Quality: ⭐⭐⭐⭐ The motivation is clear, and the framework diagram in Fig. 3 and the error-entropy correlation analysis in Fig. 4 are of high quality. The equations are well-defined, and the experimental section is clearly presented. A minor limitation is that some comparison methods (e.g., MSPF, BoostingDepth) are not introduced in detail in Section 1, requiring readers to consult the original papers.
Value: ⭐⭐⭐⭐ High practical value—the plug-and-play "retrofit" design without modifying the base model is extremely friendly to industrial deployment. Training on synthetic data with zero-shot generalization also lowers the barrier to adoption. The method itself is generic enough that future 3D foundation models will likely be able to adopt it directly. However, as the first retrofit method for 2K geometric prediction, it has not yet undergone sufficient community validation and subsequent improvements, so its long-term impact remains to be seen.