2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction¶

Conference: ECCV 2026
arXiv: 2603.19964
Code: None (promised to be open-sourced after acceptance)
Area: 3D Vision
Keywords: 2K Resolution, Sparse Refinement, Entropy-Guided Selection, Depth Estimation, 3D Reconstruction

TL;DR¶

This paper proposes 2K Retrofit, a universal framework enabling existing 3D foundation models (e.g., Depth Anything, VGGT) to output 2K resolution geometry predictions without modifying or retraining the base models. The core idea is to perform fast coarse prediction with a frozen base model and apply full-resolution correction only to high-uncertainty pixel regions using an entropy-guided sparse refinement mechanism, achieving or exceeding the accuracy of full-resolution methods while significantly reducing computational and GPU memory overhead.

Background & Motivation¶

Background: High-resolution geometry prediction (depth estimation, 3D reconstruction) is a foundational capability for applications such as autonomous driving, embodied AI, and augmented reality. Recently, 3D foundation models like Depth Anything V2, DUSt3R, and VGGT have demonstrated strong generalization after training on large-scale datasets. However, they are commonly constrained by low-resolution training data (typical inference resolution is around 960x480) and cannot directly output detailed 2K-level geometry predictions.

Limitations of Prior Work: To enable foundation models to output 2K resolution results, existing approaches fall into three categories, each with limitations: (1) Direct training of native 2K models, which incurs massive GPU memory and computational overhead (e.g., VGGT requires 76.5 GB VRAM and 495 GFLOPs for inference at 2K resolution); (2) Upsampling after low-resolution inference, which is fast but fails to recover fine-grained structures (such as railings, handles, or thin cables); (3) Patch-wise refinement, which requires multiple low-resolution inferences and complex patch merging, leading to boundary artifacts and spatial inconsistencies.

Key Challenge: There is a fundamental trade-off between high-resolution geometry accuracy and computational efficiency—full-resolution dense inference is extremely costly, whereas lightweight approximation methods lose critical high-frequency details.

Goal: (1) Enable frozen 3D foundation models to output 2K resolution geometry predictions without any architectural modifications or retraining; (2) Achieve comparable or superior accuracy at a computational cost far lower than full-resolution dense inference; (3) Ensure the method is highly generalizable across various backbone models for both monocular and multi-view scenes, as well as depth and point map estimations.

Key Insight: A systematic analysis of the coarse prediction error distribution reveals a key pattern: the error between low-resolution predictions and high-resolution ground truth is not uniformly distributed but highly concentrated in sparse, semantically critical regions (object boundaries, thin structures, and category transition areas). This implies that focusing refinement computations on this small subset of pixels can yield substantial quality improvements at minimal extra overhead. Furthermore, these high-error regions are strongly correlated with prediction entropy (computed from the backbone network's head features), rendering entropy a cheap and effective indicator for "whether to refine".

Core Idea: The core idea is to use prediction entropy as a proxy for pixel-level uncertainty to identify the sparse subset of pixels that require refinement the most (only ~10%). A lightweight sparse convolutional network is then applied to perform high-resolution residual correction solely on these pixels, and a gated fusion module adaptively blends the coarse predictions with the refinement results.

Method¶

Overall Architecture¶

2K Retrofit is a two-stage pipeline that divides high-resolution geometry prediction into "coarse prediction + sparse refinement": the first stage uses the frozen 3D foundation model for fast inference on the downsampled low-resolution image, which is then upsampled to obtain a dense coarse estimation; the second stage locates high-uncertainty pixels via an entropy selector, performs full-resolution residual correction on only these sparse pixels using a lightweight network, and finally outputs the high-precision geometry map through a gated fusion. The entire process does not modify or retrain the base model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["2K Input Image I_HR"] --> B["Downsample to 256px<br/>Frozen Base Inference"]
    B --> C["Upsample to obtain coarse estimate<br/>Geo_HR_coarse"]
    C --> D["Entropy Selector<br/>Compute pixel-wise entropy"]
    D --> E["Filter high-entropy pixel subset P<br/>(approx. top 10%)"]
    E --> F["Sparse Refinement Module<br/>MinkowskiUNet Residual Prediction"]
    F --> G["Gated Fusion<br/>Adaptive blending of coarse estimate + residual"]
    C --> G
    G --> H["Final 2K Geometry Output<br/>Geo_HR"]

Key Designs¶

1. Entropy Selector: Locating Pixels Requiring Refinement via Prediction Entropy

The error of coarse predictions varies significantly across different regions—errors at object boundaries, thin structures, and occlusion edges are far higher than in flat regions. Directly performing full-resolution refinement on all pixels wastes computational power. The authors identify a key correlation: the information entropy of backbone head features is highly consistent with prediction errors (see Figure 4), allowing entropy to serve as a cheap signal for identifying "which pixels need refinement".

Specifically, for coarse prediction results, a logits vector \(\mathbf{q}_p \in \mathbb{R}^C\) is extracted from the backbone head features (before the regression projection layer). After applying softmax normalization on this vector, the pixel-wise information entropy is computed as:

\[\mathcal{H}(p) = -\sum_{c=1}^{C} \mathbf{q}_p^{(c)} \log \mathbf{q}_p^{(c)}\]

By setting a threshold \(\alpha\), pixels satisfying \(\mathcal{H}(p) > \alpha\) are selected to form the sparse subset \(\mathcal{P}\) for refinement. This step is effective because entropy naturally reflects the model's "hesitation" at a given pixel—features at boundaries, thin structures, and sudden texture changes usually span multiple modes, resulting in higher entropy, which precisely corresponds to the regions with the largest coarse prediction errors. Experiments demonstrate that the entropy selector, at a cost of only ~2.3ms (RTX 4090), covers approximately 80% of high-error pixels while refining only ~10% of pixels, which significantly outperforms random selection (ineffective), edge heuristics (insufficiently accurate), and learnable selectors (accurate but nearly twice as slow).

2. Sparse Refinement Module: Efficient Refinement on Irregular Pixel Sets via MinkowskiUNet

The selected pixels \(\mathcal{P}\) are irregularly distributed in image space—they do not form a dense grid but rather resemble a sparse 3D point cloud (with uneven density, occlusions, and complex geometry). For such inputs, standard CNNs would have to perform dense convolutions over the entire high-resolution feature map, wasting most computations on regions that do not require refinement. To resolve this, the authors adopt MinkowskiUNet (a sparse convolutional network) as the refinement module \(\mathcal{R}\), which only performs sparse convolutions at activated positions, maintaining sparsity throughout the network.

For each selected pixel \(p \in \mathcal{P}\), the refinement module extracts features from the corresponding high-resolution image patch to predict a local residual correction:

\[\Delta \mathbf{Y}_p = \mathcal{R}(\mathbf{I}_{\mathrm{HR}}|_p), \quad \forall p \in \mathcal{P}\]

The module itself has a very small parameter footprint (lightweight design) and operates only at sparse locations, resulting in millisecond-level inference latency at 2K resolution. Compared to Point Transformer, MinkowskiUNet offers slightly lower accuracy but a speedup of over 3\(\times\) (5.5 vs 1.7 FPS), which is the key to achieving real-time 2K inference.

3. Gated Fusion: Adaptively Blending Global Consistency of Coarse Prediction and Local Accuracy of Sparse Refinement

After obtaining the sparse residual correction \(\Delta \mathbf{Y}\), the simplest approach is to directly overwrite the corresponding pixels in the coarse prediction. However, this introduces a new problem: sparse refinement lacks global context, and direct replacement easily introduces local inconsistencies and noise. To address this, the authors design a pixel-wise gated fusion mechanism that adaptively assigns weights based on the respective confidence scores of the coarse prediction and the refinement result.

For each refined pixel \(p\), the final prediction is defined as:

\[\mathbf{Y}_p = f\left(w_p \cdot \hat{\mathbf{Y}}_p + (1 - w_p) \cdot \Delta \mathbf{Y}_p\right)\]

where the fusion weight \(w_p\) is computed via a two-layer MLP with sigmoid activation, taking the coarse prediction, refined value, and their corresponding entropies as inputs:

\[w_p = \sigma\left(\mathrm{MLP}\left([\hat{\mathbf{Y}}_p; \Delta \mathbf{Y}_p; \mathcal{H}(\hat{\mathbf{Y}}_p); \mathcal{H}(\Delta \mathbf{Y}_p)]\right)\right)\]

Intuitively, when the entropy of the coarse prediction is low (indicating high confidence in the coarse estimation), the gate tends to preserve the global consistency of the coarse prediction. Conversely, when the entropy of the refinement result is low (indicating confidence in the refinement network), the gate relies more on the detailed correction. Ablation studies confirm that direct replacement (Direct) performs poorly due to the loss of global context, pure entropy-weighted fusion (Entropy) improves but remains inflexible, and gated fusion (Gated) achieves the best accuracy-speed trade-off among the three.

A Complete Example: From 2K Image to High-Precision Depth Map¶

Taking an indoor 1440x1920 RGB image from the ARKitScenes dataset as an example: the image is first downsampled to a maximum edge of 256 pixels and fed into a frozen Depth Anything V2 backbone, resulting in a low-resolution depth map. This is upsampled back to 1440x1920 via nearest-neighbor interpolation to obtain a coarse depth estimation. Pixel-wise entropy is computed from the backbone head features (taking ~2.3ms), and a threshold of \(\alpha=0.3\) is applied to select approximately 10% (~276k) high-entropy pixels—concentrated in regions like table and chair edges, cabinet handles, and curtain folds. MinkowskiUNet extracts features and predicts depth residuals only for the 2K image patches corresponding to these pixels. Finally, gated fusion blends the coarse depth and the residuals pixel-by-pixel. The final output is significantly superior to the coarse estimate in terms of boundary sharpness and the completeness of thin structures, while achieving an end-to-end pipeline latency of approximately 8.1 FPS (RTX 4090), vastly outperforming full-resolution dense inference alternatives.

Loss & Training¶

The training objectives directly inherit the loss functions of the base models utilized (e.g., the scale-invariant loss of Depth Anything V2) without additional modifications. For training data, as there was no public dataset specifically targeting 2K-level geometry predictions, the authors synthesized 50K high-quality rendered images using NVIDIA Omniverse, covering diverse geometries and texture scenarios (see Appx. for details). Training configuration: 13 epochs, batch size 8, on two NVIDIA RTX A6000 GPUs, taking approximately one day to complete. Inference latency is evaluated in FP16 precision on a single RTX 4090. The backbone models (such as Depth Anything V2 and VGGT) are kept entirely frozen, and only the entropy selector, sparse refinement module, and gated fusion module are trained.

Key Experimental Results¶

Main Results¶

Monocular 2K depth estimation under a zero-shot setting (trained purely on synthetic data without domain-specific fine-tuning):

Method	ARKitScenes AbsRel↓	ARKitScenes δ0.5↑	ScanNet++ AbsRel↓	ScanNet++ δ0.5↑
MSPF	0.0149	0.9721	0.0226	0.9674
Depth Anything V2*	0.0326	0.9615	0.0371	0.9437
PromptDA	0.0131	0.9825	0.0175	0.9781
2K Retrofit (Ours)	0.0118	0.9866	0.0146	0.9825

Multi-view 2K point map estimation (ETH3D zero-shot evaluation):

Method	Acc.↓	Comp.↓	Overall↓	FPS↑
DUSt3R	1.462	0.927	1.195	1.1
VGGT	1.167	0.639	0.903	2.6
StreamVGGT	1.140	0.613	0.870	3.5
2K Retrofit (Ours)	0.935	0.602	0.839	5.5

2K Retrofit reduces Overall error by approximately 7% compared to VGGT in point map estimation, while improving speed by over 2\(\times\) (5.5 vs 2.6 FPS). Retraining VGGT to 2K resolution requires 76.5 GB VRAM and 495 GFLOPs, whereas 2K Retrofit consumes only 32.8 GB and 172 GFLOPs (reducing both by over 50%). Furthermore, its inference speed is 17\(\times\) faster than the retrained version (5.5 vs 0.5 FPS, where the latter is highly inefficient due to frequent page-swapping caused by out-of-memory overhead).

Ablation Study¶

Comparison of pixel selectors (point map estimation, ETH3D):

Pixel Selector	Acc.↓	Comp.↓	FPS↑	Analysis
Random Selection	1.126	0.835	4.1	Ineffective, worse than no refinement
Edge Heuristic	0.942	0.625	4.1	Improved but insufficiently accurate
Bilateral Upsampling	0.940	0.617	4.0	Similar to edge heuristic
Learnable Selector	0.921	0.635	3.8	Slightly better accuracy but nearly twice as slow
Entropy Selector (Ours)	0.935	0.602	5.5	Best accuracy-speed balance

Comparison of fusion strategies:

Fusion Strategy	Acc.↓	Comp.↓	FPS↑	Analysis
Direct Replacement	1.113	0.735	5.9	Severe loss of global context
Entropy-Weighted Fusion	0.940	0.607	5.0	Improved but inflexible
Gated Fusion (Ours)	0.935	0.602	5.5	Adaptive and optimal

Key Findings¶

The entropy selector is central to the efficacy of the method—random selection fails completely (Acc. 1.126, which is even worse than the pure low-resolution baseline of VGGT at 1.167, indicating that unguided sparse refinement at 2K resolution yields a negative optimization constraint). This proves that "selecting the right pixels" is more critical than "how to refine".
The benefit of gated fusion stems from its explicit modeling of uncertainty: trusting the coarse estimation where coarse prediction confidence is high and adopting residuals where the refinement network is confident. This avoids the inconsistencies introduced by abrupt replacement.
The entropy threshold \(\alpha=0.3\) is the recommended trade-off point: a lower threshold (\(\alpha=0.1\)) yields slightly higher accuracy (Acc. 0.917) but drops the speed to 3.4 FPS; a higher threshold (\(\alpha=0.8\)) runs faster (8.0 FPS) but degrades accuracy significantly (Acc. 1.135).
In extreme scenarios like textureless areas and specular surfaces, errors become dense rather than sparse, which degrades the performance gain of the sparse refinement mechanism—representing a known limitation of the method.

Highlights & Insights¶

Entropy as a Free Lunch: In most works, entropy is discarded after serving as an uncertainty metric. This work transforms it into a cheap signal for pixel selection—requiring no extra networks or learnable parameters, locking onto high-error regions with an 80% recall rate in just 2.3ms. This methodology can be transferred to any dense prediction task requiring a "global coarse prediction + local fine-tuning" structure (such as semantic segmentation, optical flow, and image super-resolution), provided the coarse prediction network provides position-wise logits/feature vectors to calculate entropy.
Sparse Convolution for Irregular Selection: Mapping the irregularly distributed selected pixels as 3D point clouds and processing them with MinkowskiUNet for sparse convolution is an ingenious transfer of methodologies. It avoids costly dense CNN padding over sparse pixel domains and bypasses the performance bottlenecks of PointNet-related methods. Any scenario requiring selective processing of sparse sub-regions in 2D image space can borrow this idea of re-casting 2D sparse selection as a 3D sparse convolution problem.
Philosophy of "Retrofitting" rather than "Rebuilding": 2K Retrofit does not modify a single line of code in the base model or retrain its parameters. It operates completely as a plug-in, making it inherently compatible with future 3D foundation models. Simply swapping the frozen backbone facilitates model upgrades. This design philosophy is highly practical in an era of rapidly evolving foundation models, and similar principles can be applied to other tasks adapting low-resolution models to high-resolution scenarios.
Zero-Shot Generalization via Synthetic Training: Trained only on 50K synthetic images from NVIDIA Omniverse, the model achieves SOTA results on three real-world datasets (ARKitScenes, ScanNet++, and ETH3D). This demonstrates that the "where to refine and how to fuse" mechanism learned by the sparse refinement module is a domain-agnostic capability, independent of the visual appearance distributions of real data.

Limitations & Future Work¶

Degradation in Extreme Scenarios: On highly specular surfaces, transparent objects, and textureless regions, prediction errors become dense rather than sparse, which significantly diminishes the benefit of the sparse refinement mechanism. The authors acknowledge this limitation and suggest integrating stronger geometric priors (e.g., multi-modal information or physical constraints) to handle such environments in the future.
Dependency on the Quality of Base Model Head Feature Entropy: The efficacy of the entropy selector is contingent on whether the base model's head features successfully encode pixel-wise uncertainty. If a base model's head design does not produce meaningful logits (e.g., direct regression without usable softmax-compatible features), the entropy selector will fail. This restricts the plug-and-play coverage of the method, although it remains compatible with current mainstream architectures like Depth Anything and VGGT.
Gap Between Synthetic Data and Real Domains: Although zero-shot generalization is strong, synthetic data still exhibits discrepancies in illumination, materials, and sensor noise compared to real-world data. Conducting a systematic study on domain randomization strategies for synthetic data to improve generalization remains valuable future work.
Unexplored Dynamic Scenes and Temporal Consistency: The current method handles static single-frame/multi-frame images without utilizing temporal consistency in videos. Tracking sparse pixel selections across time steps could further minimize frame-by-frame computational overhead and improve temporal stability.

vs SparseRefine (ECCV 2024): SparseRefine first introduced the concept of sparse refinement in semantic segmentation, selecting high-uncertainty regions for fine-tuning using high-resolution features. 2K Retrofit extends this to 3D geometry prediction, with key differences being: (1) using entropy instead of a learnable gate for pixel selection (making it faster); (2) employing MinkowskiUNet sparse convolutions rather than dense CNNs for refinement (better suited for irregular distributions); (3) incorporating gated fusion rather than raw replacement.
vs PatchRefiner / PatchFusion (CVPR/ECCV 2024): Both are patch-wise refinement schemes that require multiple low-resolution inferences and the fusion of overlapping patches, leading to boundary artifacts and heavy computational redundancy. 2K Retrofit requires only a single low-resolution inference and processes only a sparse set of pixels thereafter, demonstrating order-of-magnitude advantages in both speed and memory.
vs PromptDA (CVPR 2025): PromptDA uses a prompt mechanism to enable Depth Anything to output 4K depth, but it still requires modifying the inference process. 2K Retrofit adopts a more thorough "wrapper" design, keeping the base model entirely untouched to improve compatibility.
vs Direct Base Model Retraining to 2K: Experiments clearly show that although retraining VGGT to 2K is marginally more accurate (Acc. 0.911 vs 0.935), its VRAM usage and compute exceed 2\(\times\), and its inference speed is 17\(\times\) slower. For real-world deployment, "accurate enough + fast enough" is far more valuable than "extremely accurate + painfully slow".

Rating¶

Novelty: ⭐⭐⭐⭐ Extending sparse refinement into the domain of 3D geometry prediction exhibits novelty. The combination of using entropy as a pixel selection signal and MinkowskiUNet for irregular sparse refinement is elegant and practical. However, the core "coarse-to-fine" two-stage pipeline and the entropy selection concept have precedents in semantic segmentation (e.g., SparseRefine), making the contribution a cross-task transfer rather than an entirely new paradigm.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covering both monocular depth estimation and multi-view point map estimation across three real datasets (ARKitScenes, ScanNet++, ETH3D) along with synthetic training data. It evaluates more than 10 SOTA methods under zero-shot and full-training configurations. The ablation studies cover four dimensions (selectors, fusion strategies, thresholds, and feature extractors), accompanied by solid system efficiency analysis covering VRAM/FLOPs/FPS. The experimental design is exceptionally thorough.
Writing Quality: ⭐⭐⭐⭐ The methodology motivation is clear, with high-quality architecture diagrams (Figure 3) and error-entropy correlation analyses (Figure 4). Equations are rigorously defined, and the experimental evaluation is presented transparently. A minor drawback is that some compared methods (e.g., MSPF, BoostingDepth) are not fully detailed in Section 1, requiring readers to consult the original papers.
Value: ⭐⭐⭐⭐ Highly practical—the "retrofit" plug-and-play design that operates without modifying the base models is extremely friendly for industrial deployment. Synthetically-trained zero-shot generalization also lowers deployment thresholds. The method is sufficiently general to be applied to future 3D foundation models. However, as the first retrofitting approach for 2K geometry prediction, it has not yet undergone extensive community verification and subsequent iterations, meaning its long-term impact remains to be watched.