EPO: Boosting 3D Foundation Models with Edge-based Pose Optimization

Conference: ECCV 2026
arXiv: 2607.00579
Code: https://github.com/mattiadurso/EPO
Area: 3D Vision
Keywords: 3D reconstruction, Structure-from-Motion, pose optimization, edge alignment, 3D foundation models

TL;DR

This paper proposes EPO, a geometric optimization framework that does not require explicit feature tracks. It utilizes Canny edge maps and distance transform fields to construct a differentiable edge reprojection loss, refining the camera parameters and depth maps output by 3D foundation models through a two-stage adaptive optimization. It achieves or exceeds the geometric accuracy of traditional bundle adjustment (BA) methods within seconds on consumer-grade GPUs.

Background & Motivation

Background: Restoring the 3D structure of a scene and camera motion from an unorganized collection of images (Structure-from-Motion, SfM) is one of the most fundamental problems in computer vision. Traditional SfM pipelines establish 2D correspondences between images through feature extraction and matching, and then jointly optimize camera poses and 3D point coordinates via Bundle Adjustment (BA). Although this framework has evolved over decades, it often struggles in texture-poor scenes (such as indoor white walls, large floor areas) and sparse multi-view scenarios—point features like SIFT are unreliable in these environments, leading to matching failures or a significant drop in accuracy.

Limitations of Prior Work: The rise of 3D foundation models (3DFMs) in recent years has opened up an entirely new path. Models such as VGGT, MapAnything, and Pi3, trained on massive 3D datasets, can predict camera parameters and depth maps for all images in a purely feed-forward manner, completing reconstruction in seconds. However, these models have a fundamental limitation: although feed-forward inference is fast, their geometric accuracy is far inferior to traditional SfM. To run post-processing BA for accuracy improvement, one must re-extract feature points and establish 3D-2D tracks across views. This step is not only time-consuming (taking minutes) but also extremely memory-intensive (often over 40GB), making it impossible to run on consumer-grade GPUs. Applying BA to traditional methods essentially compromises the "second-level" speed advantage of 3DFMs, whereas omitting BA results in insufficient accuracy, seeming to make the two objectives mutually exclusive.

Key Insight: This paper fundamentally departs from the "point feature -> track -> BA" mindset. The authors observe that the edge structure inherent in images remains relatively stable under viewpoint changes, and when the edges of one image are projected into another, their sampled values on the distance transform field naturally form a differentiable geometric alignment metric—completely eliminating the need to establish any explicit 3D-2D point correspondences.

Core Idea: Utilizing edge map alignment as a geometric constraint signal instead of feature tracks, and constructing a differentiable reprojection loss via Canny edge detection and distance transform fields. First-order optimizers are then used to refine the camera parameters and depths output by 3DFMs, achieving accuracy beyond BA without extracting any features.

Method

Overall Architecture

The input to EPO is a set of unorganized RGB images and initial geometric estimates G = {(K_i, P_i, Z_i)} output by a 3D foundation model (such as VGGT). In the preprocessing phase, Canny edge maps are extracted for each image and used to compute Distance Transform Fields (DTFs). A viewgraph is then constructed via loop-consistency reprojection to determine which image pairs participate in the optimization. The optimization is conducted in two sequential stages: first optimizing only camera intrinsics and poses, and then incorporating depth refinement once the poses stabilize, all driven by first-order gradients.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Input Image Set"] --> VGGT["3DFM Feed-forward Inference<br/>Initial G{K,P,Z}"]
    Input --> Edge["Canny Edge Map + DTF"]
    VGGT --> VG["Viewgraph Construction<br/>Loop-Consistency Reprojection"]
    Edge --> VG
    VG --> P1["Phase 1: Camera Optimization<br/>MLP Refines Poses<br/>γ Scales Focal Length"]
    P1 --> C1{"Pose Converged?<br/>θ95&lt;ε1 over w1 steps"}
    C1 -->|No| P1
    C1 -->|Yes| P2["Phase 2: Joint Optimization<br/>+Pixel-wise Depth α,β"]
    P2 --> C2{"Pose Converged?<br/>θ95&lt;ε2 over w2 steps"}
    C2 -->|No| P2
    C2 -->|Yes| Out["Refined Geometry G*"]

Key Designs

1. Edge-Based Differentiable Geometric Alignment—Replacing Feature Tracks with DTF

This is the core design of EPO. For each image I_i, a Canny detector is used to extract an edge map E_i, which is then processed with an exact L2 distance transform to obtain DTF_i—where each pixel value represents its Euclidean distance to the nearest edge. For an image pair (i, j) in the viewgraph, each edge pixel of I_i is projected onto I_j based on the current camera parameters, and the distance value at this projected location is sampled in DTF_j. The average distance across all edge points constitutes the reprojection error in the i→j direction. Summing bidirectionally yields the edge alignment loss for the image pair, where a Huber loss suppresses the impact of outliers, and the truncation threshold λ is annealed from 10 to 6 to further enhance robustness. The entire projection function π is fully differentiable from 3D to 2D, allowing gradients to propagate all the way back to camera parameters and depth.

The significance of this lies in: the density of edge points is much higher than that of sparse feature points (generating 1.7 times as many 2D observations in experiments), providing richer geometric constraints in texture-poor regions; meanwhile, the sampling operation of the DTF at projected locations can be implemented as an efficient Triton kernel without the need to compute dense feature correlation and self-attention to establish tracks, significantly reducing GPU memory overhead.

2. Dual-Track Pose Refinement with MLP and Residuals

Directly performing gradient optimization on the pose matrix can easily cause the rotation part to deviate from the SO(3) manifold. EPO adopts a 6-layer MLP (128-dimensional hidden layers, skip connection after the third layer) that takes the current pose as input and outputs offsets. The offsets are represented as continuous 6D rotations and converted to valid rotation matrices via Gram-Schmidt orthogonalization. All MLP parameters are zero-initialized, ensuring that the optimization starts smoothly from the initial estimation.

The authors found that it is difficult for the MLP alone to accurately predict translation offsets—due to the wide scale variation of translation, the network tends to underfit fine-grained adjustments. To address this, an additional learnable parameter δ_i is introduced as a translation residual offset. The final pose update is \(P_i^s = \phi(P_i^0 + \mathrm{MLP}(P_i^0)) + [0 | \delta_i]\). Ablation studies show that the MLP contributes +2.0 AUC, and δ_i further contributes +1.8 AUC—the complementarity between the two is highly evident: the MLP is responsible for structural adjustments of rotation, while δ_i specifically absorbs the residual error of translation.

3. Two-Stage Adaptive Scheduling and Pose-Based Early Stopping

The optimizer needs to simultaneously process camera parameters, poses, and depth, but different parameters converge at different paces. EPO adopts a two-stage scheduling: Phase 1 optimizes only camera intrinsics and poses (focal length is updated via a scaling factor γ_i: \(f_i^s = f_i^0 \cdot (1 + \gamma_i)\)), while depth remains frozen. When the pose changes tend to stabilize—meaning the 95th percentile rotation change ΔR_95 and translation change Δt_95 are both below ε_1=0.5° for w_1=25 steps—Phase 2 begins, incorporating depth optimization. Depth is parameterized via pixel-wise affine transformation: \(Z_{i,(x,y)}^s = Z_{i,(x,y)}^0 \cdot \alpha_{i,(x,y)} + \beta_{i,(x,y)}\), which yields better results than a single offset or global scaling. Phase 2 terminates under stricter conditions (ε_2=0.1°, w_2=50 steps).

A noteworthy detail is the choice of stopping criterion. Loss-based early stopping often terminates prematurely because the loss converges before the pose, significantly compromising accuracy (an average of only 80.8 AUC vs. the optimal 83.6). In contrast, the pose-based stopping criterion directly monitors the pose itself, reducing runtime from 25 seconds to approximately 7 seconds while achieving 82.2 AUC—representing the optimal balance between accuracy and efficiency.

Loss & Training

The global loss is the average bidirectional edge reprojection Huber loss over all valid image pairs in the viewgraph ε: $$ \mathcal{L} = \frac{1}{|\mathcal{E}|} \sum_{(i,j) \in \mathcal{E}} \mathcal{L}_{ij} $$

The optimizer uses AdamW (β1=0.9, β2=0.999, weight decay=0.01, and the weight decay of the δ term is increased to 0.1 to strengthen regularization). The learning rate warms up from 0 to 3e-3 over 25 steps, followed by cosine annealing decay up to a maximum of 2000 steps. MLP and edge extraction use BF16 mixed-precision, while the remaining components use FP32. In each step, M=1024 image pairs are randomly sampled from the viewgraph (mini-batch SGD), ensuring that the computational cost does not scale with the scene size.

Key Experimental Results

Main Results

Dataset	Metric	VGGT	+BA	+Ref+BA†	+EPO
ScanNet++	AUC@5°↑	55.6	70.0	70.9	77.1
	Time (s)↓	37.1	171.6	303.5	52.1
TerraSky3D	AUC@5°↑	56.8	71.1	75.5	79.2
	Time (s)↓	31.0	226.1	337.7	44.6
Mip-NeRF 360	AUC@5°↑	72.2	85.5	87.8	90.5
	Time (s)↓	41.9	103.9	210.2	49.6

EPO achieves the best AUC across all three datasets, while the running time is only a dozen seconds longer than the original VGGT (whereas BA methods require several minutes). Note that BA and Ref+BA require an H200 to run (marked with †), whereas EPO reaches higher accuracy on an RTX 4090.

Ablation Study

Config	AUC@5°↑	LPIPS↓	Time (s)↓
VGGT baseline	72.2	0.422	0
+ Free Variables (100 steps)	77.0	0.343	4
+ Pose MLP	79.0	0.339	4
+ Parametric K & Z	80.2	0.323	4
+ Translation Offset δ	82.0	0.308	4
+ 10× Iterations (1000 steps)	91.0	0.269	14
+ 20× Iterations (2000 steps)	92.1	0.266	25
+ Adaptive Early Stopping (Avg. 387 steps)	90.5	0.284	7

Key Findings

• In texture-poor indoor scenes (ScanNet++), the advantage of edge constraints is most prominent—where BA is limited by unreliable feature points, EPO improves the AUC from 70.9 to 77.1, more than doubling the gain of BA-based methods.
• Excellent cross-model generalization: EPO not only improves VGGT (+22 pts) but also brings massive performance gains to MapAnything (+25 pts) and Pi3 (+14 pts).
• In downstream NVS tasks, 3DGS rendering with EPO fully outperforms BA methods (PSNR 23.93 vs. 22.70, LPIPS 0.284 vs. 0.339), as more accurate poses directly translate into clearer rendering details.
• Triton kernel fusion reduces the optimization time per step from 28-39ms to ~7.3ms, achieving a 4.6× speedup.

Highlights & Insights

• Track-free global geometric optimization is feasible: The counter-intuitive aspect of this work—using edges instead of feature points for global BA-style optimization—is thoroughly validated across three mainstream datasets, opening up a brand-new technical path for 3DFM post-processing.
• The design wisdom of pose-based early stopping: The loss often stops decreasing before the pose converges. Directly monitoring pose changes avoids premature termination while saving iterations. This physics-driven stopping criterion offers valuable reference for similar optimization problems.
• Practical utility on consumer-grade GPUs: EPO runs entirely on an RTX 4090 while BA methods require an H200, significantly lowering the barrier to improving 3DFM accuracy and demonstrating direct engineering value.

Limitations & Future Work

• Inherent vulnerability of edge reliance: In high-frequency unstructured texture scenes (such as dense foliage or reflective surfaces), the "edges" extracted by Canny are noise rather than stable geometric primitives, polluting the gradient signals and leading to optimization failure. The authors explicitly state this as a major limitation.
• Viewgraph topology limitations: Low-connectivity viewgraphs (due to wide-baseline or small-overlap image pairs) fail to provide sufficient geometric constraints, making optimization difficult to converge.
• Future directions: Integrating with learning-based edge detection could enhance robustness to complex textures. Additionally, leveraging temporal continuity in video sequence scenarios to further compress the search space holds potential.

Related Work & Insights

• vs. VGGT+BA / VGGT+Ref+BA: These methods rely on the entire pipeline of feature extraction -> matching -> track -> BA, introducing overhead at each step. EPO simultaneously enhances accuracy and speed through edge alignment and first-order optimization without establishing tracks at all.
• vs. Line-based SfM: Line features are stronger than point features in man-made environments, but line matching itself is a difficult problem. The edge map essentially provides denser, line-like constraints while completely bypassing explicit line matching.
• vs. Edge SLAM: SLAM can leverage temporal continuity, spatial priors, and metric depth to assist edge alignment. In contrast, EPO faces unorganized image collections, has no temporal priors, and starts solely from the noisy depths of 3DFMs, making the setting significantly more challenging.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Using edges instead of feature tracks for global geometric optimization; the idea is novel and counter-intuitive.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive experiments across 3 datasets, multiple 3DFMs, ablation/early stopping, and downstream NVS.
• Writing Quality: ⭐⭐⭐⭐ Clear methodological exposition with abundant algorithmic pseudocode and Triton implementation details.
• Value: ⭐⭐⭐⭐⭐ Addresses the speed-accuracy dilemma of 3DFMs, runs on consumer-grade GPUs, and has high practical value.

Related Papers

• [CVPR 2026] Foundry: Distilling 3D Foundation Models for the Edge
• [CVPR 2026] Emergent Extreme-View Geometry in 3D Foundation Models
• [CVPR 2026] Global-Aware Edge Prioritization for Pose Graph Initialization
• [CVPR 2026] E2EGS: Event-to-Edge Gaussian Splatting for Pose-Free 3D Reconstruction
• [CVPR 2026] Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds