SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models¶

Conference: ECCV 2026
arXiv: 2606.27444
Code: https://albertchen98.github.io/SemCityLoc
Area: 3D Vision
Keywords: Aerial localization, 6DoF pose estimation, Semantic 3D city models, Semantic-geometric alignment, UAV localization

TL;DR¶

SemCityLoc reformulates UAV aerial 6DoF pose estimation as a "structured surface registration" problem between foundation model visual priors (semantic segmentation + monocular depth) and standardized semantic 3D city models. Through a two-stage pipeline consisting of "4D semantic cost volume coarse localization \(\rightarrow\) joint semantic-depth fine alignment via particle filtering", it improves the localization recall in urban canyon scenarios from 35.11% to 69.15% (under the 2m-2° threshold) without requiring dense radiometric reconstruction, while reducing the translation error from 9.89m to 2.62m. Furthermore, it introduces SemCityLockeD, the first real-world UAV localization benchmark with centimeter-level accuracy.

Background & Motivation¶

UAV aerial 6DoF localization is a core capability for urban navigation, inspection, mapping, and digital twins, and is particularly indispensable in GNSS-denied environments. Traditional approaches rely on RTK-GNSS combined with dense, textured 3D meshes. Although highly accurate, such radiometrically rich models demand significant storage, incur high computational overheads, and are difficult to deploy at scale, potentially raising privacy concerns in urban areas. Consequently, there is a strong demand for lightweight alternative solutions.

Existing lightweight methods attempt to reduce dependency on dense textures, yet each possesses distinct limitations: OrthoLoC relies on 2D orthophotos and DSMs, remaining essentially texture-dependent; LoD-Loc v1/v2 utilizes abstract building models for localization but relies solely on silhouette-level wireframe alignment, lacking exploitation of semantic surfaces and depth geometry, which leads to poor robustness in low-altitude, close-range urban canyon scenarios. The deeper contradiction lies in the fact that aerial images exhibit large baselines, sparse structural details, and dominant vanishing directions, which weakens pure geometric constraints—meanwhile, the dense SfM reconstructions relied upon by traditional visual localization are precisely the burden one seeks to discard.

The core challenge is clear: must one pay the price of dense radiometric reconstruction to obtain geometric constraints sufficient to support high-precision pose estimation? The key insight of this work is that over 216 million semantic 3D city models are publicly available worldwide under the CityGML standard. These models natively include semantic surface classifications (such as roofs and facades/walls), structured geometry, and centimeter-level georeferencing. If modern foundation models (e.g., DINOv3 for semantic segmentation, MoGe-2 for monocular depth) can be used to extract semantic-geometric priors from a single aerial view and structurally register them with these lightweight city models, sufficient pose observability can be achieved without relying on texture reconstruction.

Core Idea: Upgrade aerial localization from "edge/silhouette matching" to "structured regional registration of semantic surfaces and depth geometry." Introducing semantic surface partitioning provides region-level structural constraints, which, when combined with complementary depth signals, significantly enhances pose discriminability under repetitive building facades and occluded conditions.

Method¶

Overall Architecture¶

SemCityLoc adopts a coarse-to-fine semantic-geometric alignment strategy. The input consists of a query UAV image \(I_q\), a coarse prior pose \(\boldsymbol{\xi}_{pri}\) (from GNSS/IMU, with errors up to tens of meters), and a standardized semantic LoD city model \(\mathcal{M}\) of the target area. The output is the precise 6DoF camera pose \(\boldsymbol{\xi}^* = (x,y,z,\phi,\theta,\psi)\). Since roll and pitch can be reliably estimated via gravity measurements, the search space for \((\phi,\theta)\) is fixed, making the search operate only in the 4D space \((x,y,z,\psi)\).

The entire pipeline comprises two perception modules and two localization stages. The two perception modules utilize foundation models to extract the semantic mask \(M_q\) (DINOv3 backbone + lightweight DPT decoder) and the monocular depth map \(D_q\) (MoGe-2 pre-trained model) from the query image. The coarse localization stage uniformly samples around the prior pose in the 4D space \((x,y,z,\psi)\). For each candidate pose, a semantic mask \(M_c\) is rendered from the city model, and a 4D cost volume is constructed by computing class-wise IoUs with \(M_q\). The maximum of this volume yields the coarse pose \(\boldsymbol{\xi}_c\). The fine alignment stage starts from \(\boldsymbol{\xi}_c\), uses particle filtering to generate perturbed candidates, and evaluates each particle against a high-resolution semantic alignment cost \(C^s\) and a depth alignment cost \(C^d\). After weighted fusion and iterative updates, the particles converge to the final pose \(\boldsymbol{\xi}^*\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query Image + Pose Prior<br/>+ Semantic LoD City Model"]
    B["Perception Modules<br/>DINOv3 Segmentation → M_q<br/>MoGe-2 Monocular Depth → D_q"]
    C["4D Semantic Cost Volume Coarse Localization<br/>Uniform sampling in (x,y,z,ψ) → Render semantic masks<br/>Class-wise IoU → argmax → ξ_c"]
    D["Joint Semantic-Depth Alignment<br/>2× resolution semantic mask rendering<br/>Semantic IoU + Scale-aligned depth L1"]
    E["Particle Filter Iterative Optimization<br/>Perturbed ξ_c → Cost evaluation → Weighted update<br/>Convergence in N_iter rounds → Final pose ξ*"]
    A --> B
    B --> C
    C --> D
    D --> E

Key Designs¶

1. 4D Semantic Cost Volume Coarse Localization: Replacing Sparse Contour Matching with Semantic Surface Registration

Prior works like LoD-Loc rely solely on edge matching between building contours/wireframes and images during the coarse localization stage. Such sparse constraints are highly prone to ambiguity under low-altitude oblique views, repetitive facades, and occlusions. The core alternative approach of SemCityLoc is to leverage the native semantic surface classification (e.g., roofs, facades) in CityGML semantic 3D city models, transforming the localization problem into dense regional registration over multi-class semantic masks.

Specifically, in the vicinity of the prior pose \(\boldsymbol{\xi}_{pri}\), the search space is uniformly sampled in the 4D space \((x,y,z,\psi)\) based on the sampling range \(\mathbf{e}_c = [e_{cx}, e_{cy}, e_{cz}, e_{c\psi}]\) and the sampling counts \(\mathbf{n}_c = [n_{cx}, n_{cy}, n_{cz}, n_{c\psi}]\), generating \(n_{cx} \times n_{cy} \times n_{cz} \times n_{c\psi}\) candidate poses. For each candidate pose \(\boldsymbol{\xi}_i^{\text{samp}}\), the corresponding semantic mask \(M_r(\boldsymbol{\xi}_i^{\text{samp}})\) is rendered from the city model via GPU rasterization, and the mean class-wise IoU with the query semantic mask \(M_q\) is calculated as the semantic cost:

\[C_i^s = \frac{1}{N_c}\sum_{k=1}^{N_c}\frac{|M_r^k(\boldsymbol{\xi}_i^{\text{samp}}) \cap M_q^k|}{|M_r^k(\boldsymbol{\xi}_i^{\text{samp}}) \cup M_q^k|}\]

where \(N_c\) is the number of semantic classes, and \(M_r^k\) and \(M_q^k\) denote the rendered and predicted masks for the \(k\)-th class, respectively. After constructing the complete 4D cost volume, the pose corresponding to the maximum cost value is selected as the coarse localization result \(\boldsymbol{\xi}_c\).

The effectiveness of this design stems from three advantages of semantic surface registration: first, semantic surfaces provide region-level constraints instead of sparse edges, which are more robust to local occlusions and view perspective changes; second, class-wise IoU inherently forces spatial alignment between the rendered and predicted masks, providing much stronger spatial regularization than edge alignment; third, distinct semantic classes feature different geometric spatial distribution patterns (e.g., roofs are on top, facades are on the sides), and their inter-class combinations further enhance pose discriminability. Ablation studies confirm that the coarse localization stage alone improves the localization recall from 41.49% to 56.38% (under the 2m-2° threshold).

2. Joint Semantic-Depth Fine Alignment: Complementary Geometric Constraints for Disambiguation of Repetitive Structures

While the coarse localization stage uses low-resolution semantic masks to curb search overheads, its accuracy is bounded by the grid resolution and relies purely on a single semantic modality. In the fine alignment stage, SemCityLoc introduces two key enhancements: doubling the semantic mask resolution to capture fine-grained local geometric details, and incorporating monocular depth as an independent and complementary geometric signal.

The primary challenge of depth alignment is that the monocular depth estimation \(D_q\) outputs scale-ambiguous relative depth, which cannot be directly compared with the metric depth \(D_r(\boldsymbol{\xi}_i^{\text{samp}})\) rendered from the city model. The paper estimates a global scale \(s^*\) and shift \(t^*\) for each match via least squares, aligning the predicted depth with the rendered depth within the valid region \(\mathcal{M}_{valid}\):

\[C_i^d = \sum_{p \in \mathcal{M}_{valid}} \frac{1}{D_r(p)}\left\|s^* D_q(p) + t^* - D_r(p)\right\|_1\]

where the inverse depth weighting \(1/D_r(p)\) de-emphasizes highly uncertain distant depth estimates (as distant points have larger depth values and thus receive smaller weights). The final total cost is a weighted combination of semantic and depth costs:

\[C_i^{final} = \lambda_1 C_i^s + \lambda_2 C_i^d\]

The joint utilization yields highly complementary and significant improvements: when semantic features are indistinguishable across repetitive building facades (e.g., arrays of identical windows or balconies), depth provides independent geometric distance constraints to eliminate ambiguity; conversely, when depth estimation becomes unreliable in textureless regions, the semantic surface boundaries offer strong structural priors. Ablation studies show that adding depth alignment further improves the 2m-2° recall from 56.38% to 69.15%, and reduces the yaw error from 1.04° to 0.42°, proving that the semantic and depth cues are complementary and mutually crucial in the fine alignment stage.

3. Particle Filter Iterative Optimization: Stochastic Exploration Replacing High-Dimensional Exhaustive Search

Coarse localization yields a discrete optimum in the 4D space, but its grid resolution is constrained by computational costs. Proceeding with finer-grained exhaustive searches in a continuous 4D space would lead to an exponential explosion in computational complexity. Inspired by particle filtering in visual localization, the authors design a stochastic optimization strategy to achieve highly efficient convergence.

Specifically, starting from the coarse pose \(\boldsymbol{\xi}_c\), small perturbations \(\delta\) are applied to its translation and rotation parameters to generate \(N_f\) particle candidate poses:

\[\{\boldsymbol{\xi}_{p_i}\}_{i=1}^{N_f} = \boldsymbol{\xi}_c + \{\delta\boldsymbol{\xi}_i\}_{i=1}^{N_f}\]

Each particle is evaluated against the comprehensive cost \(C_i^{final}\) in Equation (7), and the pose is updated via weighted averaging based on probability scores derived from the costs. This process iterates for \(N_{iter}\) rounds, enabling the particles to progressively cluster in high-cost regions and eventually converge to the final pose \(\boldsymbol{\xi}^*\).

The elegance of particle filtering in this context lies in the fact that it is not merely random sampling followed by picking the best candidate; instead, it adaptively shrinks the particle population toward high-probability regions through weighted updates, thereby enabling sub-grid pose recovery with a limited number of particles. Compared to exhaustive search, the complexity is reduced from \(O(n^4)\) to \(O(N_f \times N_{iter})\). It is also proven in the ablation study to be an indispensable complement to coarse localization—omitting the fine alignment stage causes the translation error to rise from 2.62m to 3.20m, and the yaw error to increase from 0.42° to 1.04°.

Loss & Training¶

SemCityLoc's localization pipeline (cost volume search + particle filtering) is a parameter-free optimization process that requires no training. The only component requiring training is the semantic segmentation module, where the DINOv3 ViT backbone is frozen and only a lightweight DPT decoder is trained on top. Training is conducted separately for each dataset, achieving 88% mIoU on SemCityLockeD, 85% on UAVD4L-LoD, and 78% on Swiss-EPFL, with all stable convergences occurring within approximately 15 epochs, significantly faster than LoD-Loc's training-from-scratch. The MoGe-2 depth estimator is used directly with pre-trained weights without any fine-tuning.

Key hyperparameters during the localization phase include: the coarse search sampling range \(\mathbf{e}_c\) and sampling counts \(\mathbf{n}_c\) (which determine the cost volume resolution and search coverage), the particle count \(N_f\) and the iteration rounds \(N_{iter}\) (which control the exploration-exploitation trade-off of the fine alignment), and the cost weights \(\lambda_1\) and \(\lambda_2\) (which regulate the relative contributions of semantics and depth). The total runtime of the pipeline is 0.878 seconds per single image (0.116s for perception inference, 0.409s for coarse search, and 0.353s for particle filtering), achieving sub-second real-time performance.

Key Experimental Results¶

Main Results¶

On the SemCityLockeD benchmark, SemCityLoc comprehensively outperforms all baseline methods across all recall thresholds and error metrics. Feature matching-based methods (CAD-Loc and MC-Loc based on RoMa/e-LoFTR) fail almost completely in dense urban canyon scenarios (with recall values near 0%) because the texture features they rely upon cannot establish reliable correspondences under repetitive building facades and occlusions. The wireframe alignment methodology of LoD-Loc performs slightly better but achieves only a 35.11% recall at the strictest 2m-2° threshold, whereas SemCityLoc reaches 69.15%, while simultaneously reducing the average yaw error from 1.78° to 0.42° and the translation error from 9.89m to 2.62m.

Method	2m-2° (%)	3m-3° (%)	5m-5° (%)	Yaw (°)	XYZ (m)
CAD-Loc (e-LoFTR)	0	0	0	-	-
CAD-Loc (RoMa)	0	0	0	-	-
MC-Loc (DINOv2)	5.32	8.51	18.09	5.90	10.71
MC-Loc (RoMa)	0	1.06	3.19	6.67	18.61
LoD-Loc	35.11	47.87	53.19	1.78	9.89
SemCityLoc (Full)	69.15	84.04	89.36	0.42	2.62

On the two public benchmarks, UAVD4L-LoD and Swiss-EPFL, SemCityLoc demonstrates consistent advantages. Particularly in the out-of-place generalization scenario of Swiss-EPFL, SemCityLoc improves the 2m-2° recall from 17.41% to 35.36%, the 5m-5° recall from 48.55% to 89.18%, and reduces the translation error from 12.54m to 3.09m—demonstrating that the semantic-geometric alignment strategy possesses much stronger cross-scene generalization capability than wireframe matching.

Ablation Study¶

Configuration	2m-2° (%)	3m-3° (%)	5m-5° (%)	Yaw (°)	XYZ (m)	Description
Full model	69.15	84.04	89.36	0.42	2.62	Full two-stage pipeline
w/o Refinement	56.38	70.21	81.91	1.04	3.20	Without particle filter fine alignment
w/o Coarse Selection	41.49	56.38	72.34	1.11	5.94	Without 4D cost volume coarse search

The ablation results clearly reveal the complementarity of the two stages: the coarse search contributes to the first major leap from 41.49% to 56.38% (+14.89% @ 2m-2°), while the fine alignment drives the second leap from 56.38% to 69.15% (+12.77%). Removing the coarse search (using only the particle filter starting from the prior) causes the translation error to inflate to 5.94m, showing that without a good initial value from coarse localization, the local exploration capability of the particle filter is insufficient to cover the large error range of the prior. Omitting the fine alignment increases the yaw error from 0.42° to 1.04°, indicating that the depth alignment's contribution to rotation accuracy is particularly prominent.

Key Findings¶

Impact of LoD Levels: Evaluation on SemCityLockeD's unique standardized LoD1-LoD3 models indicates that localization accuracy scales with geometric detail (LoD1: 48.94% \(\rightarrow\) LoD2 semantic: 69.15% \(\rightarrow\) LoD3: 70.21% @ 2m-2°); however, semantic LoD2 even outperforms LoD3 at the 3m-3° threshold (84.04% vs 75.53%). This indicates that semantic surface structures can partially compensate for deficiencies in geometric resolution—pose accuracy depends not only on geometric density, but more crucially on the joint availability of discriminative structural cues and semantically consistent surface representations.

Robustness to Pose Priors: Under prior translation perturbations of up to 50m per axis, the 2m-2° recall remains steady at 62.77%, highlighting the system's strong tolerance to large initial errors. However, once the perturbation exceeds 100m per axis, the recall drops precipitously to 18.09%, revealing the operational boundaries of the method—the sampling range of the coarse search must encompass the ground-truth pose; otherwise, recovery becomes impossible.

Sensitivity to Segmentation Quality: Replacing the lightweight fine-tuned head of DINOv3 with zero-shot alternatives (e.g., CLIP + Semantic-SAM or Grounded-SAM2) drops the mIoU from 88% to 32-37%, causing the recall to plummet from 69.15% to 15.93% (2m-2°). This represents both a limitation (current dependency on some in-domain supervision) and an opportunity (as foundation models continue to improve under aerial perspectives, zero-shot localization accuracy is expected to rise organically).

Cross-dataset Generalization: SemCityLoc consistently outperforms LoD-Loc across three highly diverse datasets (dense urban canyons in SemCityLockeD, hybrid suburban areas in UAVD4L-LoD, and high-altitude mixtures in Swiss-EPFL), demonstrating that the semantic-geometric alignment framework has broad scene adaptability and is not a specialized solution only valid under restricted conditions.

Highlights & Insights¶

"Semantic Surface Registration" reformulates the problem definition of aerial localization: Prior methods improved within the framework of "which geometric features to match" (points, lines, contours, textures). This paper directly leaps to "which surface regions to match"—leveraging native semantic surface classifications (roofs, facades, etc.) in CityGML to transform localization from a sparse correspondence problem into a dense regional registration process. This paradigm shift is the core insight of the paper, offering high transferability to any localization task operating over structured semantic maps.
Semantic LoD2 models rival or even outperform LoD3 on multiple metrics: This counter-intuitive finding reveals an important principle—for localization tasks, "knowing what each surface represents" can be more critical than "knowing how detailed the geometric details of the surface are." This has direct implications for real-world deployment: with over 216 million public LoD2 models available globally, high-precision localization can be achieved without waiting for higher LoD coverage.
Foundation models designed as "pluggable perception frontends": Semantic segmentation (DINOv3) and depth estimation (MoGe-2) are treated as modular perception priors, leaving the localization pipeline itself independent of specific model architectures. Consequently, as foundation models continue to evolve (especially in adaptation to aerial views), localization accuracy will naturally improve without requiring modifications to the core alignment algorithm—marking a highly sustainable system design pattern.
Inverse-depth weighted L1 loss: Weighting the depth error inversely by \(1/D_r(p)\) in Equation (6) ensures that distant pixels, having larger depth values, receive weaker weights. This seemingly simple trick precisely addresses the fundamental issue of monocular depth estimation—uncertainty at a distance—and prevents unreliable signals from polluting the overall matching cost. It serves as a widely applicable and directly reusable depth alignment technique.
Sub-second full-pipeline inference: A processing speed of 0.878 seconds per frame (including both localization stages) translates to practical viability for online relocalization, making it directly applicable for autonomous UAV flight in GNSS-denied environments.

Limitations & Future Work¶

Dependence on city model availability and accuracy: SemCityLoc's performance upper bound is limited by the LoD model coverage and geometric precision within the target region. Although global CityGML data is abundant, its distribution remains uneven (highly dense in Europe, sparse elsewhere), and models in certain regions may lack semantic annotations or suffer from geometric deviations.
Failure under extremely low geometric observability: When the field of view contains only a single building facade or sparse visible structures, even joint semantic-depth constraints may not suffice to uniquely determine the pose. This represents an inherent limitation of structural registration approaches, in contrast to dense feature matching methods.
Bottleneck of zero-shot semantic segmentation: Current zero-shot schemes (such as CLIP + SAM models) achieve only 32-37% mIoU under aerial views, which is insufficient to support reliable localization, thereby still requiring lightweight dataset-specific fine-tuning. This is a limitation of this work and an open problem for the entire field—visual foundation models in aerial domains are far less mature than those for ground views.
Unexplored expansion directions: The evaluation was restricted to static scenes, without addressing the interference of dynamic objects (vehicles, pedestrians) on mask rendering and depth alignment. Additionally, multi-frame temporal fusion for further robustness enhancement remained unexplored, as did the potential to extend the methodology to other CityGML semantic classes like roads and vegetation.

vs LoD-Loc / LoD-Loc v2: Both utilize LoD building models for aerial localization, but LoD-Loc relies solely on contour/wireframe alignment, and LoD-Loc v2 extends this to silhouette alignment—both essentially being edge-level matching techniques. The key difference in SemCityLoc lies in the introduction of semantic surface class information and monocular depth cues, upgrading matching from sparse edges to joint dense regional and depth registration. Experiments prove that this upgrade yields substantial benefits across all scenarios, particularly expanding the performance gap under occluding and repetitive structures.
vs OrthoLoc: OrthoLoc utilizes aerial images for cross-modal matching with orthophotos/DSMs to perform 6DoF localization, which inherently depends on 2D texture information. SemCityLoc completely discards textures, relying entirely on the alignment of semantic labels and geometric structures, which is highly advantageous in privacy-sensitive scenarios and areas lacking updated orthophotos.
vs OrienterNet / MapLocNet: These methods use navigation maps (2D vector graphics of road networks and building footprints) for localization but are restricted to 3-DoF (\(x, y, \text{yaw}\)). SemCityLoc utilizes facade information from 3D city models to extend localization to full 6DoF, which is a necessary prerequisite for downstream tasks requiring precise height and pitch/roll information (such as 3D reconstruction and building detection).
vs Traditional SfM + Feature Matching: The classic pipeline of SfM reconstruction + SuperGlue/LoFTR matching + PnP offers high accuracy but relies on dense textured scene models, incurring massive storage and computational overheads. SemCityLoc's philosophy fundamentally replaces "privately collected dense texture reconstruction" with "publicly available structured semantic-geometry," paving a more viable path for scalable deployment.

Rating¶

Novelty: ⭐⭐⭐⭐ The paradigm shift of upgrading aerial localization from contour/edge matching to semantic surface registration is creative; the concept of joint semantic and depth structured alignment is proposed in localization for the first time.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducts comprehensive evaluations across 3 datasets, multiple LoD levels, different prior noise magnitudes, and varying segmentation qualities; the ablation studies clearly dissect the contributions of each of the two stages; includes efficiency analysis and qualitative comparisons.
Writing Quality: ⭐⭐⭐⭐ Clear problem formulation, logically consistent methodology flow, and an experimental design that thoroughly covers key dimensions. The resonance between the LoD analysis and the semantic surface registration concept is particularly outstanding.
Value: ⭐⭐⭐⭐ Provides solid empirical support for the pathway of "substituting dense reconstructions with publicly available structured maps for localization"; the SemCityLockeD benchmark fills a significant void in the field; sub-second inference endows it with immediate potential for real-world deployment.