Stable and Scalable Bundle Adjustment of Holistic 3D Structures

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/cvg/holistic-ba ⚠️ Pending public release (refer to original paper)
Area: 3D Vision
Keywords: bundle adjustment, structure from motion, Schur complement, wireframe, geometric primitives

TL;DR

This paper introduces a unified bundle adjustment framework that incorporates higher-order geometric relations (planes, vanishing points, wireframes) by treating groups as camera-like non-eliminated variables and formulating constraints as 2D reprojection residuals, preserving the block-diagonal Schur complement efficiency of classical point BA while yielding substantially richer and more accurate 3D geometry.

Background & Motivation

Bundle adjustment (BA) serves as the foundational optimization engine in 3D computer vision and structure from motion (SfM), jointly refining camera intrinsics, poses, and 3D landmarks by minimizing image reprojection errors. For decades, the remarkable scalability of BA—optimizing scenes with millions of 3D landmarks—has hinged on the bipartite visibility structure of the problem. Because each 3D point is observed independently across cameras with no direct inter-point coupling, the landmark block in the normal equations Hessian matrix remains strictly block-diagonal. This allows solvers to marginalize 3D points via Schur complement elimination at trivial computational cost, reducing the normal equations to a compact system over camera parameters alone.

However, real-world man-made environments are dominated by holistic structural regularities far beyond isolated point clouds: boundaries form 3D line segments governed by vanishing points, surfaces form planar facets and geometric primitives, and points and lines intersect at junctions to form wireframe skeletons. Integrating such higher-order relations into bundle adjustment has historically confronted two major roadblocks. First, structural coupling destroys Hessian sparsity. Directly penalizing pairwise or higher-order feature relationships (e.g., coplanarity, collinearity, or wireframe connectivity) introduces off-diagonal entries between features in the landmark Hessian block, obliterating the block-diagonal structure and causing expensive dense matrix fill-in during Schur complement elimination. Second, 3D metric regularization introduces severe scale mismatch and ill-conditioning. Formulating geometric priors as 3D Euclidean distances (such as 3D point-to-plane distances) mixes uncalibrated 3D metric residuals with 2D pixel reprojections. In monocular SfM where absolute scale is unobservable, such 3D penalties break the scale gauge invariance of bundle adjustment, incentivizing the optimizer to arbitrarily shrink the scene to minimize metric costs. Furthermore, they lack an isotropic uncertainty noise model and fail to balance measurements across varying view depths.

This paper addresses both challenges by reframing the computational taxonomy of geometric entities and converting spatial constraints into pure pixel-space reprojections. Core idea: model higher-order geometric groups as camera-like entities placed in the non-eliminated parameter block to absorb inter-feature couplings, and reformulate both group-induced and wireframe constraints as 2D reprojection errors in pixel space, naturally anchoring structural primitives according to each feature's Fisher information without empirical weight tuning.

Method

Overall Architecture

The system takes multi-view images alongside detected 2D points, lines, vanishing points, and segmentations, jointly estimating camera poses, 3D features (points and lines), and higher-order structural groups (planes, vanishing points, quadric primitives). Variables are formally partitioned into a scalable feature set \(\mathcal{F}\) and an augmented non-eliminated set \(\mathcal{A} = \mathcal{C} \cup \mathcal{G}\) consisting of cameras \(\mathcal{C}\) and groups \(\mathcal{G}\). Structural constraints are converted into group-induced and cross-feature pixel reprojection residuals. The normal equations maintain a block-diagonal feature Hessian \(\mathbf{H}_{ff}\), enabling standard, highly scalable Schur complement elimination.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view image input<br/>Extract points, lines, VPs, and planes"] --> B["Taxonomy of Features & Groups<br/>F=points/lines, A=cameras C + groups G"]
    B --> C["Group-induced reprojection error<br/>3D surface projection reprojected to 2D"]
    B --> D["Cross-feature wireframe error<br/>Decouple point-line with fixed 2D partner"]
    C --> E["Augmented normal equations<br/>Strict block-diagonal H_ff preserved"]
    D --> E
    E --> F["Schur complement & solver<br/>Eliminate F and solve compact reduced S"]

Key Designs

1. Taxonomy of Features & Groups: Preserving Block-Diagonal Schur Elimination To resolve the structural coupling that breaks Schur complement elimination, the authors categorize geometric entities based on their computational roles in the normal equations. Geometric primitives that possess direct 2D image measurements are classified as features \(\mathcal{F}\) (3D points \(\mathbf{X}_j\) and Plücker lines \(\mathbf{L}_k\)), which scale into the thousands or millions. Entities encoding higher-order relations without direct per-image measurements are classified as groups \(\mathcal{G}\) (planes with normal and offset \((\mathbf{n}, d)\), vanishing point directions \(\mathbf{v}_g\), and parametric surfaces), which number at most in the dozens or hundreds. Crucially, group parameters \(\mathcal{G}\) are placed alongside cameras \(\mathcal{C}\) in the non-eliminated block \(\mathcal{A} = \mathcal{C} \cup \mathcal{G}\). Each feature-group residual links exactly one feature and one group, mimicking the bipartite camera-feature observation structure. Consequently, the feature Hessian \(\mathbf{H}_{ff}\) remains strictly block-diagonal, and inverting \(\mathbf{H}_{ff}\) requires only inverting small, independent diagonal blocks. Inter-group relations (such as Manhattan orthogonality between plane normals or vanishing points) couple only parameters within \(\mathcal{A}\), contributing strictly to \(\mathbf{H}_{gg}\) without perturbing \(\mathbf{H}_{ff}\).

2. Group-Induced Reprojection Error: Scale Invariance and Implicit Fisher Anchoring To overcome the scale gauge degradation and hyper-parameter sensitivity of 3D metric penalties, positional group constraints are reformulated as 2D pixel-space reprojection differences. Given a 3D point \(\mathbf{X}_j\) associated with a plane group \(G_g = (\mathbf{n}, d)\), it is orthogonally projected onto the plane manifold via \(\mathrm{Proj}_{G_g}(\mathbf{X}_j) = \mathbf{X}_j - \frac{\mathbf{n}^\top \mathbf{X}_j - d}{\|\mathbf{n}\|^2}\mathbf{n}\). The group-induced reprojection error in camera \(C_i\) is formulated as the displacement between the projected point's image coordinate and the projection of its planar counterpart: $$ \mathbf{r}g^{(p)} = \pi_p(C_i, \mathbf{X}_j) - \pi_p(C_i, \mathrm{Proj}^2 $$ For lines on planes, the residual }(\mathbf{X}_j)) \in \mathbb{R\(\mathbf{r}_g^{(l)} \in \mathbb{R}^4\) similarly measures the difference between line reprojection vectors before and after planar projection. First-order Taylor linearization yields \(\mathbf{r}_g^{(p)} \approx \mathbf{J}_i \boldsymbol{\delta}\), where \(\boldsymbol{\delta} = \mathbf{X}_j - \mathrm{Proj}_{G_g}(\mathbf{X}_j)\) and \(\mathbf{J}_i\) is the camera projection Jacobian. Summed across all observing views, the effective objective simplifies to \(\boldsymbol{\delta}^\top (\sum_i \mathbf{J}_i^\top \mathbf{J}_i) \boldsymbol{\delta}\). Because \(\sum_i \mathbf{J}_i^\top \mathbf{J}_i\) is precisely the Fisher information matrix \(\boldsymbol{\Sigma}_X^{-1}\) of point \(\mathbf{X}_j\), the residual is mathematically equivalent to the Mahalanobis distance \(\boldsymbol{\delta}^\top \boldsymbol{\Sigma}_X^{-1} \boldsymbol{\delta}\). Consequently, features with high multi-view triangulation confidence and wide baselines firmly anchor the group surface parameters, while geometrically degenerate or noisy features exert negligible influence, achieving principled, parameter-free uncertainty weighting in pixel space.

3. Cross-Feature Wireframe Reprojection: Decoupling Point-Line Constraints via Fixed Observations To integrate wireframe structural junctions (points incident on lines) without creating direct point-line couplings in \(\mathbf{H}_{ff}\), the authors re-conceptualize wireframe constraints using single-variable asymmetric residuals. For a junction edge \((\mathbf{X}_j, \mathbf{L}_k)\), two complementary residuals are defined across their non-overlapping visibility graphs: In any view \(i\) where line \(\mathbf{L}_k\) is observed as segment \(\boldsymbol{\ell}_{ik}\) but point \(\mathbf{X}_j\) is not detected, the point-to-line error treats the 2D segment measurement \(\boldsymbol{\ell}_{ik}\) as a fixed constant and measures the perpendicular distance from the projected 3D point to the observed line: $$ r_{p \to l} = d(\pi_p(C_i, \mathbf{X}j), \boldsymbol{\ell}) $$ Conversely, in any view \(i\) where point \(\mathbf{X}_j\) is observed as keypoint \(\mathbf{x}_{ij}\) but line \(\mathbf{L}_k\) is missing, the line-to-point residual treats \(\mathbf{x}_{ij}\) as a fixed constant measurement, penalizing distance \(r_{l \to p} = d(\mathbf{x}_{ij}, \pi_l(C_i, \mathbf{L}_k))\). In views where both primitives are directly detected, standard reprojection residuals already enforce the constraint, so cross-feature terms are omitted. Because each residual involves only one feature and one camera variable, the Jacobian contains zero off-diagonal blocks between point and line indices, keeping \(\mathbf{H}_{ff}\) completely uncoupled.

A Worked Example

Consider an indoor corner junction where a structural line \(L\) and a vertex point \(X\) meet. In camera 1, due to heavy illumination changes, the vertex detector fails, but the edge line segment \(\boldsymbol{\ell}_1\) is sharply extracted. In camera 2, the edge line is occluded by foreground clutter, but the corner point \(\mathbf{x}_2\) is reliably tracked. In camera 3, both are detected. Under traditional 3D regularized BA, establishing a 3D distance penalty \(d(X, L) = 0\) creates a non-zero off-diagonal Hessian block coupling \(X\) and \(L\), increasing the elimination complexity. In the proposed framework, camera 1 instantiates residual \(r_{p \to l}(C_1, X; \boldsymbol{\ell}_1)\), constraining the ray of \(X\) to lie on the back-projected plane of \(\boldsymbol{\ell}_1\). Camera 2 instantiates \(r_{l \to p}(C_2, L; \mathbf{x}_2)\), constraining the 3D line \(L\) to project through image point \(\mathbf{x}_2\). Camera 3 evaluates standard point and line reprojections independently. Throughout optimization, \(X\) and \(L\) remain mathematically decoupled in the normal equations, preserving \(O(N)\) linear Schur marginalization while fully exploiting cross-view wireframe geometry.

Loss & Training

The overall joint optimization problem minimizes: $$ E(\mathcal{C}, \mathcal{G}, \mathcal{X}, \mathcal{L}) = \sum_{\mathcal{O}p} \rho(\mathbf{r}}}) + \sum_{\mathcal{Ol} \rho(\mathbf{r}}}) + \sum_{\mathcal{O{g}} \rho(\mathbf{r}}}) + \sum_{\mathcal{O{wf}} \rho(\mathbf{r}}}) + \sum_{\mathcal{O{inter}} \rho(\mathbf{r}) $$ where }\(\rho(\cdot)\) denotes the robust Cauchy loss function. Structural associations are instantiated only when validated across \(\ge 3\) images via geometric voting. Because all primary residuals are measured in pixel units, convergence can be reliably monitored via the relative cost reduction \(\frac{|\Delta E|}{E} < \epsilon\), eliminating the empirical tuning required when balancing 2D pixels with 3D metric units.

Key Experimental Results

Main Results

The method was evaluated on synthetic benchmarks (varying from 100 to 2000 images), the synthetic Hypersim benchmark (8 scenes with ground-truth meshes), large-scale real-world 1DSfM datasets, and camera pose benchmarks across ScanNet++, ETH3D, and 7Scenes.

Table 1: Geometry evaluation on Hypersim (8 scenes) against ground-truth meshes | Method / Configuration | Point Inlier @1mm (%) ↑ | Point Inlier @5mm (%) ↑ | Point Median Err (mm) ↓ | Line Recall @1mm (m) ↑ | Line Prec. @1mm (%) ↑ | Line Prec. @10mm (%) ↑ | |---|---|---|---|---|---|---| | Per-point/line refinement | 9.3 | 31.7 | 15.10 | 72.1 | 21.3 | 77.3 | | + Group constraints | 11.5 | 35.6 | 13.80 | 93.2 | 27.3 | 81.0 | | + Group & Wireframe constraints (Ours) | 11.9 | 36.3 | 13.55 | 93.3 | 27.3 | 81.0 |

Table 2: Relative camera pose AUC across four standard structure-from-motion benchmarks | Dataset | Metric | COLMAP (Point) | Point-Line SfM | Holistic SfM (Ours) | Notes | |---|---|---|---|---|---| | Hypersim (8 scenes) | AUC@3° / AUC@5° / AUC@10° | 89.3 / 90.4 / 92.2 | 92.2 / 93.2 / 94.0 | 92.5 / 93.4 / 94.2 | Consistent geometric gain | | ScanNet++ (20 scenes) | AUC@3° / AUC@5° / AUC@10° | 84.0 / 86.0 / 87.5 | 84.7 / 86.7 / 89.2 | 87.4 / 89.4 / 90.6 | +3.4° AUC gain in planar indoor rooms | | ETH3D (11 scenes) | AUC@3° / AUC@5° / AUC@10° | 50.2 / 66.4 / 77.6 | 49.4 / 67.1 / 79.6 | 50.4 / 68.1 / 80.4 | Best camera accuracy outdoors | | 7Scenes (7 scenes) | AUC@3° / AUC@5° / AUC@10° | 22.9 / 48.7 / 75.5 | 23.1 / 48.9 / 76.8 | 24.3 / 50.8 / 78.4 | Outperforms baselines under low texture |

Ablation Study

Comparison of the proposed 2D group-induced reprojection error against direct 3D point-to-plane distance regularization under different hyper-parameter weights \(w\).

Table 3: Ablation of 2D group reprojection error vs. direct 3D metric regularizations | Variant | ETH3D Acc. @1cm (%) ↑ | ETH3D Acc. @5cm (%) ↑ | ScanNet++ AUC@1° ↑ | ScanNet++ AUC@3° ↑ | ScanNet++ AUC@10° ↑ | |---|---|---|---|---|---| | Point BA | 45.17 | 75.53 | 66.1 | 84.0 | 87.5 | | 3D group error (\(w=1\)) | 45.17 | 75.55 | 67.4 | 85.0 | 89.1 | | 3D group error (\(w=10\)) | 45.24 | 75.61 | 68.1 | 85.0 | 90.5 | | 3D group error (\(w=100\)) | 45.72 | 76.12 | 68.3 | 87.8 | 91.1 | | 3D group error (\(w=1000\)) | 46.69 | 76.71 | 67.1 | 85.1 | 88.3 | | Holistic BA (Ours, 2D error, no tuning) | 47.05 | 76.84 | 69.4 | 87.4 | 90.6 |

Key Findings

• Breakdown of 3D Regularization under Scale Ambiguity: As shown in the ablation study, 3D metric penalties require delicate weight tuning. A low weight (\(w=1\)) leaves structural cues inactive, while an aggressive weight (\(w=1000\)) causes scale conflicts that degrade camera rotation/translation accuracy (ScanNet++ AUC@1° collapses from 68.3 to 67.1). In contrast, the parameter-free 2D reprojection formulation achieves superior geometry (47.05% @1cm on ETH3D) and best pose accuracy (69.4 AUC@1°) without manual tuning.
• Runtime Scalability Comparable to Classical BA: On 1DSfM large-scale reconstructions (e.g., Tower of London with 963 registered images and 99k points), the full holistic SfM pipeline took 59.2 minutes versus 46.4 minutes for the point-only baseline—an overhead of only \(\sim 1.3\times\), in contrast to previous hybrid feature SfM pipelines that incurred \(2\times\) to \(4\times\) overhead. Under both Ceres SPARSE_SCHUR and DENSE_SCHUR solvers, asymptotic per-iteration complexity remains unchanged (\(\sim n^{1.6-1.9}\) and \(\sim n^{2.6-2.8}\) respectively).

Highlights & Insights

• Algebraic placement of groups alongside cameras: Positioning low-cardinality geometric groups within the non-marginalized block \(\mathcal{A}\) elegantly absorbs structural coupling without altering the block-diagonal structure of \(\mathbf{H}_{ff}\), solving a decades-old matrix fill-in hurdle in structured BA.
• Natural Fisher information weighting in pixel space: Linearizing the 2D group-induced reprojection error proves that the implicit weighting matches the landmark's inverse covariance \(\boldsymbol{\Sigma}_X^{-1}\), providing a theoretically grounded anchoring mechanism where confident multi-view tracks determine the structure.
• Decoupled wireframe constraints via constant measurements: Formulating cross-feature residuals using the partner feature's 2D image observation as a fixed target decouples 3D variables, offering an extensible blueprint for other hybrid geometric associations (e.g., point-to-mesh or line-to-surface).

Limitations & Future Work

• Vulnerability to noisy upstream association: The optimization relies on accurate detection and clustering of planes and wireframes. While robust Cauchy kernels and multi-view consensus voting mitigate outliers, widespread association failures can cause geometric distortion.
• Degenerate viewing geometry recovery: Features viewed at grazing angles (optical axes parallel to the plane) yield tiny reprojection errors regardless of out-of-plane displacement; while this protects plane estimation from corruption, such features cannot be pulled back onto the plane.
• Future directions: Integrating foundation models (e.g., SAM3) to prompt and detect higher-order semantic shapes (cylinders, spheres, bounding boxes), and formulating learned uncertainty weights for end-to-end differentiable structural BA.

Related Work & Insights

• vs. Bartoli & Sturm (Explicit Parameterization): Classical reparameterization hard-constrains features onto planar degrees of freedom, which requires perfect, irreversible associations prior to optimization; the proposed soft reprojection formulation naturally tolerates uncertain associations via robust loss functions.
• vs. Struct-VIO / PLP-SLAM (3D Regularization): Prior works define Euclidean point-to-plane distances in metric space, causing scale collapse under monocular gauge ambiguity; the proposed 2D reprojection preserves scale invariance and shares identical pixel units with standard BA.
• vs. 3D Line Mapping (Liu et al., CVPR 2023): Existing point-line wireframe solvers introduce 3D point-to-line residuals that create dense inter-feature off-diagonal blocks; the proposed cross-feature formulation retains strict block-diagonal sparsity and fast Schur marginalization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant mathematical reformulation of group topologies and pixel-space structural residuals.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across four real-world benchmarks, synthetic scaling tests, and geometry ablation studies.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear algebraic derivations and intuitive geometric insights.
• Value: ⭐⭐⭐⭐⭐ Establishes a highly practical foundation for modern structure-from-motion and SLAM systems incorporating higher-order geometric primitives.