City-Level 3D Surface Reconstruction with Viewpoint Orientation Partitioning and Scene Completion¶

Conference: ECCV2026
Paper: ECCV Paper
Project: VOP-GS
Area: 3D Vision
Keywords: city-level surface reconstruction, viewpoint orientation partitioning, initial point cloud completion, multi-view consistency, Gaussian Splatting

TL;DR¶

The method groups camera views that support each other's reconstruction, selectively completes the initial point cloud using keypoint distribution entropy, and optimizes Gaussian representations with multi-view consistency, increasing MatrixCity surface reconstruction F1 from CityGS-X's 0.581 to 0.803.

Background & Motivation¶

City-level reconstruction must recover building facades, roads, and other scene surfaces from many multi-view photographs, rather than merely synthesize a convincing novel-view image. 3D Gaussian Splatting (3DGS) improves rendering efficiency through explicit Gaussian primitives, but urban scale still introduces memory pressure, lengthy optimization, and complex occlusion. A common solution partitions space into blocks, assigns photographs to each block, trains them independently, and organizes the results into a complete scene. This partitioning is computationally natural but not necessarily suitable for geometry estimation: nearby cameras may face opposite directions, while occluders or supporting views needed by boundary cameras may lie outside a block. Consequently, a local model can fit colors yet recover discontinuous or incorrect depth at boundaries.

The paper shifts the partitioning criterion from spatial location toward which photographs can provide reliable geometric constraints for one another. Similar camera orientations often make substantial image overlap easier to obtain; subsequent position-based subdivision organizes useful observations locally. However, grouping only improves how existing observations are used and cannot automatically repair regions with no initialization points. COLMAP may recover no reliable 3D points in low-texture or poorly covered regions, leaving densification through Gaussian splitting and cloning without suitable starting primitives. Running dense reconstruction on every image across a city is expensive, motivating the selection of photographs that genuinely need additional points.

The two designs address whether supervision is mutually supportive and whether optimization has a geometric starting point, while retaining an established surface reconstruction framework. Core Idea: organize geometrically compatible training groups by viewpoint orientation and use keypoint distribution entropy to select missing regions for completion, directing multi-view constraints and initialization computation toward reliable surface recovery.

Method¶

Overall Architecture¶

Inputs are multi-view images, camera poses, a COLMAP sparse point cloud, and valid 2D keypoints associated with reconstructed 3D points. The pipeline performs viewpoint orientation partitioning, initial point cloud completion, and multi-view consistency optimization to produce a city-level surface reconstruction. Partitioning produces camera groups that may share supporting views, not mutually exclusive spatial cells; completion provides geometry before optimization rather than repairing a mesh afterward. Groups are trained separately across multiple GPUs, while the pretrained matching network participates only in point completion, not final per-pixel scene rendering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Images, poses, and<br/>COLMAP sparse point cloud"] --> PART["Viewpoint orientation<br/>partitioning"]
    PART --> FILL["Initial point cloud<br/>completion"]
    FILL -->|Completed points initialize each group| OPT["Multi-view consistency<br/>optimization"]
    PART -->|Expanded camera groups and source views| OPT
    INPUT -->|Observed-image supervision| OPT
    OPT --> OUT["City-level surface reconstruction"]

The arrows distinguish geometric initialization from training supervision: additional points provide starting Gaussian positions, while observations and source views constrain subsequent optimization. The main paper primarily explains grouping, completion, and losses; it does not adequately specify final mesh extraction or inter-group fusion, so no particular fusion algorithm should be inferred.

Key Designs¶

1. Viewpoint orientation partitioning: ensure compatible viewing directions before controlling local scale

The first step applies DBSCAN to normalized camera directions, using cosine distance, defined as 1 minus the dot product of the two direction vectors. This prioritizes similar orientations rather than mixing observations of opposite facades merely because their camera positions are close. Similar orientations can still span distant regions, so large groups undergo a second partition based on 3D camera positions. For a group containing \(n\) cameras, the paper takes \(K=\lceil n/M_{\max}\rceil\) and applies position-based K-means, with \(M_{\max}=200\) by default. Position therefore acts as a secondary constraint, reducing both local spatial extent and per-group resource requirements. Ordinary K-means does not strictly guarantee that every group satisfies the capacity limit, and the main paper provides no additional capacity-constrained step.

Subdivision alone can still damage observation relationships at group boundaries, so each reference view retrieves useful source views from the full view set and adds them to its group. The implementation selects 4 source views per reference view, allowing training groups to overlap rather than enforcing hard boundaries. The intention is to retain the orientation-consistent structure while providing sufficient observations for local matching. Expanded groups may consequently exceed the nominal subdivision limit; 200 should not be treated as a strict guarantee of final memory use. The main paper does not fully specify source-view scoring and selection, so no particular overlap threshold is inferred here. This design changes how supervision is organized rather than introducing a new Gaussian primitive or replacing the renderer.

2. Initial point cloud completion: detect uneven coverage before adding points for selected views

The authors reuse COLMAP's valid 2D keypoints, avoiding the need to project the entire 3D point cloud into every image again. These keypoints correspond to successfully reconstructed 3D points, making their image coverage a proxy for the adequacy of geometric initialization. Each image is divided into an \(N\times N\) grid, with \(N=10\) by default; \(p_j\) is the fraction of that image's valid keypoints located in cell \(j\). Concentrated proportions indicate that points occupy only a few regions, potentially leaving others without geometric support. Shannon entropy measures this uniformity and is normalized by its maximum so that images can share a threshold. The following expression restates the textual definition using consistent notation; the cached equations mix the symbols \(E/H\):

\[ E_{\mathrm{norm}}=-\frac{\sum_{j=1}^{N^2}p_j\log(p_j+\epsilon)}{\log(N^2)}. \]

Images satisfying \(E_{\mathrm{norm}}<\tau\) enter the candidate set, with \(\tau=0.7\) by default. Each candidate is paired with neighboring source views and passed to a pretrained matching network to obtain dense correspondences, which are triangulated into 3D points. The matching method cited in the main text is reference [36], MINIMA; MASt3R, mentioned in related discussion, should not be mistaken for the completion network actually cited by this method. New points undergo geometric consistency filtering and duplicate removal before being concatenated with the original COLMAP cloud to initialize Gaussian training. Completion thus relies on recoverable cross-view image correspondences, not on a generative model inventing occluded city structures.

This selection saves computation relative to globally dense initialization, but entropy measures relative spatial distribution rather than directly measuring point count or geometric quality. A small number of uniformly distributed points can still have relatively high entropy; defining proportions for images with no valid points also requires implementation handling that the main paper does not describe. These are reader analyses of the detector's scope, not failure experiments established by the paper. The authors observe that most images have normalized entropy between 0.6 and 1.0 and use a common threshold to select the poorly covered portion. Increasing the threshold includes more photographs in completion, improving surface completeness while increasing matching time and initialization memory.

3. Multi-view consistency optimization: make grouping and completion affect geometric learning

The method is implemented on PGSR and trains Gaussian representations within camera groups, rather than training another reconstruction network to generalize across cities. Alongside color reconstruction and depth-normal consistency, optimization uses PGSR's multi-view photometric and geometric consistency constraints. The photometric term maps a reference-image patch into a source view through a homography, computes normalized cross-correlation (NCC), and penalizes its difference from perfect correlation. This local patch comparison provides cross-view supervision: a surface explanation must agree with neighboring observations rather than only fit the current image. The geometric term constrains consistency across views, while the depth-normal term constrains local shape consistency; the main paper does not expand the former into a complete computational formula.

Orientation-compatible, overlapping groups provide more reliable matching conditions for these constraints, and the completed cloud supplies optimizable primitives in missing regions. Neither replaces the other: more initialization points do not guarantee consistent supervision, and good supervision does not ensure automatic recovery where no starting points exist. Matching-network predictions must still be filtered to avoid injecting erroneous correspondences into later geometric optimization. After training, the optimized scene representation is used for surface recovery; the paper does not introduce a separate inference stage that reruns entropy detection on input photographs.

A Worked Example¶

Consider a low-texture building facade in aerial city imagery; this example illustrates the mechanism without adding experimental measurements. First, camera orientations determine groups, and large groups are subdivided by position into smaller local regions. A facade photograph at a group boundary brings 4 useful supporting views into its expanded training group. Valid keypoints are then counted in its \(10\times10\) grid; if they concentrate around roof edges, entropy may fall below 0.7 and trigger completion. The matching network establishes correspondences between the candidate and its source views, and triangulated, filtered points are added to the initially sparse facade region. These points initialize Gaussians, which are constrained by the original photographs and neighboring observations so that added geometry fits real imagery rather than merely filling a hole.

Loss & Training¶

Each group is trained for 30K iterations, using 8 RTX 3090 GPUs in the experiments. The objective combines color reconstruction, patch-level photometric consistency, geometric consistency, and depth-normal consistency; weights for the latter three terms are 0.15, 0.03, and 0.05, respectively. The cached total-loss equation is missing some operators, so the textual definition and weights are retained without treating the damaged equation as an exact reproducible expression. The implementation section states that \(\epsilon_{\mathrm{dir}}=0.71\) corresponds to \(45^\circ\), but the preceding cosine-distance definition yields \(1-\cos45^\circ\approx0.2929\). This conflict may concern similarity versus distance conventions, but the paper does not resolve it; angular ablations below retain the table's original values without correcting parameters on the authors' behalf. PGSR and 2DGS baselines are trained for 100K iterations with spherical harmonic degree reduced to 1 to avoid running out of memory, so methods do not all share the same training configuration.

Key Experimental Results¶

Main Results¶

Experiments cover 6 GauU-Scene scenes, a small aerial city region from MatrixCity, and the Residence and Sci-Art scenes from UrbanScene3D. GauU-Scene contains approximately 400 to 1,200 images per scene, UrbanScene3D exceeds 2,500 per scene, and MatrixCity exceeds 5,000 images. Images from the two real-world datasets are downsampled by a factor of 4; the longer image edge in MatrixCity is resized to 1,600 pixels. The table selects city-scale methods from original Table 1 on page 10 and Table 2 on page 12; higher is better, and F1 is the harmonic mean of precision and recall. Geometric precision measures proximity of reconstructed points to ground truth, while recall measures coverage of ground-truth geometry; the cached main text does not provide the distance threshold used for matching.

Method	GauU-Scene mean F1	MatrixCity precision	MatrixCity recall	MatrixCity F1
CityGaussian-V2	0.506	0.441	0.752	0.556
CityGS-X	0.527	0.444	0.840	0.581
Ours	0.560	0.713	0.920	0.803

On MatrixCity, the absolute F1 gain over CityGS-X is 0.222; precision and recall both improve, so the gain cannot be interpreted solely as adding more points. Mean GauU-Scene F1 increases by 0.033, but the method does not win every scene: on HAV, PGSR scores 0.573 and the proposed method scores 0.554. The authors note that HAV contains only 424 images, making non-partitioned optimization potentially advantageous for this smaller scene. GauU-Scene's ground-truth LiDAR boundaries are incomplete, so evaluation follows CityGaussian-V2's central-region cropping; F1 does not describe all scene boundaries. UrbanScene3D receives primarily qualitative comparison in the main text, with no corresponding F1 table available to reproduce here.

Ablation Study¶

Original Table 4 on page 13 changes the angular threshold while retaining the nominal camera capacity \(M_{\max}=200\); group counts are totals across all GauU-Scene scenes, not counts for one scene.

Angular threshold	Total groups	Mean F1
15°	36	0.534
30°	35	0.544
45°	27	0.560
60°	27	0.532

Both 45° and 60° produce 27 groups, yet their F1 scores differ by 0.028, showing that group count or parallel budget alone cannot explain the quality difference. Smaller thresholds are not uniformly better either: 15° produces more groups but worse quality, supporting the motivation to retain sufficient shared observations. Original Table 5 on page 13 varies the completion entropy threshold and reports reconstruction F1; disabling completion retains grouping and the remaining training design.

Completion threshold	GauU-Scene F1	MatrixCity F1
Without completion	0.546	0.772
0.6	0.551	0.797
0.7	0.560	0.803
0.8	0.561	0.805
0.85	0.563	0.806
0.9	0.565	OOM

The default threshold of 0.7 improves F1 over no completion by 0.014 on GauU-Scene and 0.031 on MatrixCity. Increasing the MatrixCity threshold from 0.7 to 0.85 adds only 0.003, while 0.9 runs out of memory, showing that adding more points is not always more practical.

Key Findings¶

Without completion, MatrixCity F1 already reaches 0.772 versus CityGS-X's 0.581; this reflects the remaining design combination, not a strictly isolated contribution from partitioning alone.
Original Table 6 on page 15 raises PGSR's mean F1 from 0.333 to 0.382 by replacing its initialization with the completed cloud, supporting reuse with other reconstructors without implying improvement in every scene.
Original Table 3 on page 12 reports GauU-Scene training times of 2 hours for the proposed method, 3 hours for CityGS-X, and 6.5 hours for CityGaussian-V2.
MatrixCity times are 6.5, 5, and 13 hours, respectively; the proposed method beats CityGaussian-V2 but is slower than CityGS-X, so it is not universally fastest.
MatrixCity is divided into 30 groups trained in 4 rounds on 8 GPUs; matching and completion costs are not itemized in the main text, so training time should not be equated with complete end-to-end cost.

Highlights & Insights¶

Partitioning determines which photographs jointly constrain geometry, not just memory usage. Including supervision compatibility aligns task organization more closely with surface reconstruction than spatial location alone.
Keypoint distribution entropy provides a low-cost trigger for completion. Reusing SfM intermediates concentrates pretrained matching computation on locally undercovered images.
Completeness and accuracy should be considered together. Simultaneous MatrixCity improvements support more reliable surface recovery rather than merely broader geometric coverage.

Limitations & Future Work¶

The authors demonstrate diminishing gains and memory exhaustion from excessive completion. More targeted candidate prioritization and a point budget are promising extensions beyond simply raising the threshold.
Similar orientations do not guarantee unoccluded overlap, and high entropy does not guarantee sufficient geometry. Combining visibility with absolute point counts may improve selection, but this is a reader suggestion not tested in the paper.
The orientation distance-angle conflict, K-means capacity enforcement, source-view selection, and handling of images without valid keypoints require implementation-level clarification.
The geometric distance threshold, central-region cropping, and the reported discrepancy between CityGS-X's MatrixCity reconstruction here and in its original paper limit direct cross-paper comparisons.
This note uses the supplied main-text cache; supplementary material and code were not read, and the promised PSNR, SSIM, LPIPS results and mesh-fusion details were not verified.

Compared with PGSR: the method reuses its surface representation and consistency constraints while changing view organization and initialization at scale; HAV also shows that grouping is not the default best choice for small scenes.
Compared with CityGaussian-V2: spatial blocks and multi-stage training support scalability, whereas this method prioritizes orientation compatibility and expands groups with source views to address insufficient boundary observations.
Compared with CityGS-X: the latter emphasizes voxel-level parallel efficiency; on MatrixCity, this method trades more training time for higher geometric quality, motivating task-dependent selection rather than a single ranking.
Transferable insight: sparse reconstruction systems can explicitly decide whether additional initialization evidence is needed before invoking an expensive matcher; transfer should validate coverage entropy, point count, and visibility rather than blindly reuse 0.7.

Rating¶

Novelty: 4/5. Orientation-first grouping and selective point completion form a concise city-scale surface recovery strategy, although most underlying components come from existing methods.
Experimental Thoroughness: 4/5. Multiple datasets, angular and entropy ablations, and cross-method initialization experiments are included, but some reproduction details and a complete cost breakdown remain absent.
Writing Quality: 3/5. The central motivation is clear, while inconsistent parameter definitions and damaged cached equations introduce reproduction ambiguity.
Value: 4/5. The work provides useful view scheduling and initialization strategies for large-scale Gaussian surface reconstruction, with applicability dependent on scene scale.