SemCityLoc: Aerial 6DoF Localization Using Semantic 3D City Models¶

Conference: ECCV 2026
arXiv: 2606.27444
Code: https://albertchen98.github.io/SemCityLoc
Area: 3D Vision
Keywords: Aerial Localization, 6DoF Pose Estimation, Semantic 3D City Models, Semantic-Geometric Alignment, UAV Localization

TL;DR¶

SemCityLoc redefines UAV aerial 6DoF pose estimation as a structured surface registration problem between foundation model visual priors (semantic segmentation + monocular depth) and standardized semantic 3D city models. Through a two-stage pipeline consisting of "4D semantic cost volume coarse localization \(\rightarrow\) particle filter-based joint semantic-depth fine alignment", without requiring dense radiometric reconstruction, it improves the localization recall in urban canyon scenarios from 35.11% to 69.15% (under the 2m-2° threshold) and reduces the positional error from 9.89m to 2.62m. Concurrently, it releases SemCityLockeD, the first real UAV localization benchmark with centimeter-level precision.

Background & Motivation¶

UAV aerial 6DoF localization is a core capability for urban navigation, inspection, mapping, and digital twins, and is especially indispensable in GNSS-denied environments. Traditional solutions rely on RTK-GNSS combined with dense textured 3D meshes. Although highly accurate, such radiometrically rich models have huge memory footprints, high computational overheads, and are hard to deploy at scale. They also pose privacy concerns in urban settings, driving a strong industry demand for lightweight alternatives.

Existing lightweight methods attempt to reduce dependency on dense textures, but each has its drawbacks: OrthoLoC relies on 2D orthophotos and DSMs, remaining texture-dependent in essence; LoD-Loc v1/v2 use abstract building models for localization but rely solely on contour-level wireframe alignment, lacking the exploitation of semantic surfaces and depth geometry, which leads to insufficient robustness in low-altitude close-range urban canyon scenarios. The deeper challenge lies in: aerial images have large baselines, lack structural details, and are dominated by vanishing directions, leading to weak purely geometric constraints—whereas the dense SfM reconstruction that traditional visual localization relies on is precisely the burden to be discarded.

The key challenge is clear: is it necessary to pay the price of dense radiometric reconstruction to obtain geometric constraints sufficient for high-precision pose estimation? The key insight of this paper is that: internationally, over 216 million semantic 3D city models under the CityGML standard are publicly available, which natively possess semantic surface classifications (such as roofs, walls, etc.), structured geometry, and centimeter-level georeferencing. If modern foundation models (DINOv3 semantic segmentation, MoGe-2 monocular depth) can be used to extract semantic-geometric priors from a single aerial image and perform structured registration with these lightweight city models, sufficient pose observability can be achieved without relying on texture reconstruction.

Core Idea: Upgrade aerial localization from "edge/contour matching" to "structured region registration of semantic surfaces + depth geometry". Semantic surface classification introduces region-level structural constraints, which, combined with complementary depth signals, significantly enhances pose discriminability under repetitive building facades and occlusion.

Method¶

Overall Architecture¶

SemCityLoc adopts a "coarse-to-fine" semantic-geometric alignment strategy. The input is a UAV query image \(I_q\), a rough pose prior \(\boldsymbol{\xi}_{pri}\) (from GNSS/IMU, with errors up to tens of meters), and a standardized semantic LoD city model \(\mathcal{M}\) of the target area; the output is the precise 6DoF camera pose \(\boldsymbol{\xi}^* = (x,y,z,\phi,\theta,\psi)\). Since roll and pitch can be reliably estimated via gravity measurements, the method fixes \((\phi,\theta)\) in the search space and searches only within the 4D space \((x,y,z,\psi)\).

The entire pipeline is split into two perception modules + two localization stages. The two perception modules extract the semantic mask \(M_q\) (using a DINOv3 backbone + a lightweight DPT decoder) and the monocular depth map \(D_q\) (using a MoGe-2 pre-trained model) of the query image respectively. The coarse localization stage uniformly samples near the prior pose in the 4D space \((x,y,z,\psi)\). For each candidate pose, it renders a semantic mask \(M_c\) from the city model, computes class-wise IoU with \(M_q\) to construct a 4D cost volume, and selects the maximum value as the coarse pose \(\boldsymbol{\xi}_c\). The fine alignment stage starts from \(\boldsymbol{\xi}_c\), uses particle filtering to generate perturbed candidates, evaluates both high-resolution semantic alignment cost \(C^s\) and depth alignment cost \(C^d\) for each particle, and iteratively updates them through weighted fusion to eventually converge to \(\boldsymbol{\xi}^*\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query Image + Pose Prior<br/>+ Semantic LoD City Model"]
    B["Perception Modules<br/>DINOv3 Semantic Segmentation → M_q<br/>MoGe-2 Monocular Depth → D_q"]
    C["4D Semantic Cost Volume Coarse Localization<br/>Uniform sampling of (x,y,z,ψ) → Render Semantic Mask<br/>Class-wise IoU → argmax → ξ_c"]
    D["Joint Semantic-Depth Alignment<br/>2× Resolution Semantic Mask Rendering<br/>Semantic IoU + Scale-aligned Depth L1"]
    E["Particle Filter Iterative Optimization<br/>Perturb ξ_c → Cost Evaluation → Weighted Update<br/>Convergence in N_iter Rounds → Final Pose ξ*"]
    A --> B
    B --> C
    C --> D
    D --> E

Key Designs¶

1. 4D Semantic Cost Volume Coarse Localization: Replacing Sparse Contour Matching with Semantic Surface Registration

Prior works such as LoD-Loc solely rely on edge matching of building contours/wireframes with the image in the coarse stage. Such sparse constraints easily cause ambiguity under low-altitude oblique perspectives, repetitive facades, and occlusions. The core replacement strategy of SemCityLoc is to leverage the semantic surface classification (roofs, walls, etc.) inherent in the CityGML semantic 3D city model, converting the localization problem into multi-class semantic mask dense region registration.

Specifically, uniform sampling is performed near the prior pose \(\boldsymbol{\xi}_{pri}\) with sampling ranges \(\mathbf{e}_c = [e_{cx}, e_{cy}, e_{cz}, e_{c\psi}]\) and sampling numbers \(\mathbf{n}_c = [n_{cx}, n_{cy}, n_{cz}, n_{c\psi}]\) in the 4D space \((x,y,z,\psi)\), generating \(n_{cx} \times n_{cy} \times n_{cz} \times n_{c\psi}\) candidate poses. For each candidate pose \(\boldsymbol{\xi}_i^{\text{samp}}\), the corresponding semantic mask \(M_r(\boldsymbol{\xi}_i^{\text{samp}})\) is rendered from the city model via GPU rasterization, and the mean of class-wise IoUs with the query image semantic mask \(M_q\) is calculated as the semantic cost:

\[C_i^s = \frac{1}{N_c}\sum_{k=1}^{N_c}\frac{|M_r^k(\boldsymbol{\xi}_i^{\text{samp}}) \cap M_q^k|}{|M_r^k(\boldsymbol{\xi}_i^{\text{samp}}) \cup M_q^k|}\]

where \(N_c\) is the number of semantic classes, and \(M_r^k\) and \(M_q^k\) are the rendered and predicted masks for the \(k\)-th class, respectively. After constructing the complete 4D cost volume, the pose corresponding to the maximum value is taken as the coarse localization result \(\boldsymbol{\xi}_c\).

The effectiveness of this design stems from a three-fold advantage of semantic surface registration: First, semantic surfaces provide region-level constraints rather than sparse edges, making them more robust to local occlusions and viewpoint variations. Second, class-wise IoU naturally requires spatial consistency between the rendered and predicted masks, supplying much stronger spatial registration than edge alignment. Third, different semantic classes exhibit different geometric distribution patterns in space (e.g., roofs on top, walls on sides), and their combination further enhances pose discriminability. Ablation studies confirm that the coarse localization stage alone improves the localization recall from 41.49% to 56.38% (at the 2m-2° threshold).

2. Joint Semantic-Depth Fine Alignment: Complementary Geometric Constraints Eliminating Ambiguity of Repetitive Structures

The coarse localization stage uses low-resolution semantic masks to manage computational costs, but its accuracy is limited by the resolution and relies solely on a single semantic modality. In the fine alignment stage, SemCityLoc introduces two key enhancements: doubling the semantic mask resolution to capture local geometric details, and introducing monocular depth as an independent and complementary geometric signal.

The main challenge in depth alignment is that the monocular depth estimation \(D_q\) outputs a scale-ambiguous relative depth, which cannot be compared directly with the metric depth \(D_r(\boldsymbol{\xi}_i^{\text{samp}})\) rendered from the city model. The paper estimates a global scale \(s^*\) and shift \(t^*\) for each match via least squares, aligning the predicted depth to the rendered depth within the valid region \(\mathcal{M}_{valid}\):

\[C_i^d = \sum_{p \in \mathcal{M}_{valid}} \frac{1}{D_r(p)}\left\|s^* D_q(p) + t^* - D_r(p)\right\|_1\]

where the inverse weighting of \(1/D_r(p)\) suppresses the unreliability of distant depth estimates (where larger depth values have smaller weights). The final composite cost is a weighted fusion of semantic and depth costs:

\[C_i^{final} = \lambda_1 C_i^s + \lambda_2 C_i^d\]

The joint usage delivers complementary and significant results: when semantic features are indistinguishable between repetitive building facades (e.g., arrays of identical windows/balconies), depth provides independent geometric distance constraints to disambiguate; when depth estimation is unreliable in textureless regions, semantic surface boundaries offer strong structural priors. Ablation studies show that adding depth alignment further improves the 2m-2° recall from 56.38% to 69.15%, and reduces the yaw error from 1.04° to 0.42°, proving that the semantic and depth signals are complementary and jointly critical in the fine alignment stage.

3. Iterative Particle Filter Optimization: Stochastic Exploration Replacing High-Dimensional Exhaustive Search

Coarse localization yields a discrete optimum in 4D space, but its grid resolution is restricted by computational budget. Conducting a finer-grained search in the 4D continuous space using an exhaustive approach would cause exponential computational growth. Drawing inspiration from particle filtering in visual localization, the paper designs a stochastic optimization strategy to achieve efficient convergence.

Specifically, starting from the coarse pose \(\boldsymbol{\xi}_c\), small perturbations \(\delta\) are applied to the translation and rotation parameters to generate \(N_f\) particle candidate poses:

\[\{\boldsymbol{\xi}_{p_i}\}_{i=1}^{N_f} = \boldsymbol{\xi}_c + \{\delta\boldsymbol{\xi}_i\}_{i=1}^{N_f}\]

Each particle is evaluated according to the composite cost \(C_i^{final}\) in Equation (7), and the poses are then updated via weighted averaging based on their cost probability scores. This process is iteratively executed for \(N_{iter}\) rounds, progressively concentrating particles towards high-cost regions, and eventually converging to \(\boldsymbol{\xi}^*\).

The ingenuity of the particle filter here lies in its adaptive contraction of the particle swarm towards high-probability regions through weighted updates during iterations, rather than simple random sampling to find the best candidate. This enables much finer pose recovery than uniform grid searches under a limited number of particles. Compared to exhaustive search, the complexity of particle filtering is reduced from \(O(n^4)\) to \(O(N_f \times N_{iter})\), and ablation studies prove it to be a necessary complement to coarse localization—removing the fine alignment stage increases the positional error from 2.62m to 3.20m, and the yaw error from 0.42° to 1.04°.

Loss & Training¶

The localization pipeline of SemCityLoc itself (cost volume search + particle filtering) is a parameter-free optimization process that requires no training. The only component requiring training is the semantic segmentation module: the DINOv3 ViT backbone is frozen, and only a lightweight DPT decoder is trained on top. Training is conducted separately for each dataset, achieving 88% mIoU on SemCityLockeD, 85% on UAVD4L-LoD, and 78% on Swiss-EPFL, all of which stably converge within about 15 epochs, significantly faster than training LoD-Loc from scratch. The MoGe-2 depth estimator utilizes pre-trained weights without any fine-tuning.

Key hyperparameters during the localization phase include: the coarse search sampling ranges \(\mathbf{e}_c\) and sampling numbers \(\mathbf{n}_c\) (which determine the cost volume resolution and coverage), particle count \(N_f\), iteration rounds \(N_{iter}\) (controlling the exploration-exploitation balance of fine alignment), and cost weights \(\lambda_1, \lambda_2\) (regulating the relative contributions of semantics and depth). The total inference time of the entire pipeline is 0.878 seconds per image (perception inference 0.116s + coarse search 0.409s + particle filtering 0.353s), achieving sub-second real-time performance.

Key Experimental Results¶

Main Results¶

On the SemCityLockeD benchmark, SemCityLoc comprehensively outperforms all baseline methods across all recall thresholds and error metrics. Feature matching-based methods (CAD-Loc, MC-Loc based on RoMa/e-LoFTR) fail almost completely in dense urban canyon scenarios (recalls close to 0%), because the texture features they rely on cannot establish reliable correspondences under repetitive building facades and occlusion. The wireframe-alignment-based LoD-Loc performs slightly better but only achieves a 35.11% recall under the strictest 2m-2° threshold, whereas SemCityLoc reaches 69.15%, while simultaneously reducing the average yaw error from 1.78° to 0.42° and the positional error from 9.89m to 2.62m.

Method	2m-2°(%)	3m-3°(%)	5m-5°(%)	Yaw(°)	XYZ(m)
CAD-Loc (e-LoFTR)	0	0	0	-	-
CAD-Loc (RoMa)	0	0	0	-	-
MC-Loc (DINOv2)	5.32	8.51	18.09	5.90	10.71
MC-Loc (RoMa)	0	1.06	3.19	6.67	18.61
LoD-Loc	35.11	47.87	53.19	1.78	9.89
SemCityLoc (Full)	69.15	84.04	89.36	0.42	2.62

On the two public benchmarks, UAVD4L-LoD and Swiss-EPFL, SemCityLoc demonstrates a consistent superiority. In particular, in the out-of-place generalization scenario of Swiss-EPFL, SemCityLoc improves the 2m-2° recall from 17.41% to 35.36%, the 5m-5° recall from 48.55% to 89.18%, and reduces the positional error from 12.54m to 3.09m—indicating that the semantic-geometric alignment strategy possesses much stronger cross-scenario generalization capabilities than wireframe matching.

Ablation Study¶

Configuration	2m-2°(%)	3m-3°(%)	5m-5°(%)	Yaw(°)	XYZ(m)	Description
Full model	69.15	84.04	89.36	0.42	2.62	Full two-stage pipeline
w/o Refinement	56.38	70.21	81.91	1.04	3.20	Remove particle filter fine alignment
w/o Coarse Selection	41.49	56.38	72.34	1.11	5.94	Remove 4D cost volume coarse search

Ablation results clearly reveal the complementarity of the two stages: coarse search contributes the first significant leap from 41.49% to 56.38% (+14.89% @ 2m-2°), while fine alignment contributes the second leap from 56.38% to 69.15% (+12.77%). Removing the coarse search (directly optimization with particle filtering from the prior) causes the positional error to inflate to 5.94m, demonstrating that the local exploration capability of particle filtering is insufficient to cover the large error range of the prior without a good initial value provided by coarse localization. Removing fine alignment increases the yaw error from 0.42° to 1.04°, indicating that depth alignment is particularly key to improving rotation accuracy.

Key Findings¶

Impact of LoD levels: Assessment on the standardized LoD1-LoD3 models unique to SemCityLockeD shows that localization accuracy increases with geometric detail (LoD1: 48.94% \(\rightarrow\) LoD2 semantic: 69.15% \(\rightarrow\) LoD3: 70.21% @ 2m-2°), but semantic LoD2 even surpasses LoD3 on the 3m-3° threshold (84.04% vs 75.53%). This indicates that semantic surface structures can partially compensate for deficiencies in geometric resolution—pose accuracy depends not only on geometric density, but more so on the joint availability of discriminative structural clues and semantically consistent surface representations.

Robustness to pose priors: When the prior translation perturbation is up to 50m per axis, the 2m-2° recall remains at 62.77%, showing high tolerance to large initial errors. However, when the perturbation exceeds 100m per axis, the recall drops abruptly to 18.09%, revealing the operating boundary of the method—the sampling range of the coarse search must cover the ground truth pose, as recovery is impossible if it falls outside.

Sensitivity to segmentation quality: Replacing the DINOv3 lightweight fine-tuned decoder with zero-shot schemes (such as CLIP + Semantic-SAM or Grounded-SAM2) causes the mIoU to drop from 88% to 32-37%, and the recall to plunge from 69.15% to 15.93% (2m-2°). This represents both a limitation (current reliance on some in-domain supervision) and an opportunity (as foundation models in aerial perspectives continue to advance, zero-shot localization accuracy is expected to naturally improve).

Cross-dataset generalization: SemCityLoc consistently outperforms LoD-Loc on three significantly different datasets (SemCityLockeD with dense urban canyons, UAVD4L-LoD with mixed suburban areas, and Swiss-EPFL with high-altitude mixed areas), proving that the semantic-geometric alignment framework has broad scene adaptability and is not a specialized solution only valid under specific conditions.

Highlights & Insights¶

"Semantic surface registration" reframes the aerial localization problem: Prior works improved "what geometric features to match" (points, lines, contours, textures); this paper directly leapfrogs to "what surface regions to match"—leveraging the semantic surface classification (roofs, walls, etc.) inherent in CityGML to convert localization from a sparse correspondence problem into a dense region registration problem. This paradigm shift is the core insight of the paper and can be generalized to any localization task with structured semantic maps.
LoD2 semantic models match or even outperform LoD3 on multiple metrics: This counter-intuitive finding reveals an important principle—for localization tasks, "knowing what each surface is" can be more important than "knowing how detailed the surface geometry is". This has direct implications for practical deployment: over 216 million public LoD2 models are already available worldwide, enabling high-precision localization without waiting for higher LoD coverage.
Foundation models as "pluggable perception frontends": Semantic segmentation (DINOv3) and depth estimation (MoGe-2) are treated as modular perception priors in the system, and the localization pipeline itself does not depend on specific model architectures. This means that as foundation models advance (especially in aerial-domain adaptation), localization accuracy will naturally improve without修改 core alignment algorithms—representing a long-lasting system design pattern.
Reciprocal depth-weighted L1 loss: Using \(1/D_r(p)\) to inverse-distance-weight depth errors in Equation (6) assigns smaller weights to distant pixels due to their larger depth values. This simple trick precisely addresses the fundamental issue of monocular depth estimation—unreliability of distant depth—preventing unreliable signals from contaminating the overall matching cost. It is a highly transferable and reusable depth alignment technique.
Sub-second full pipeline inference: 0.878 seconds per frame (including two-stage localization) implies that this method is practical for online relocalization, viable for autonomous UAV flights in GPS-denied environments.

Limitations & Future Work¶

Dependency on the availability and accuracy of city models: The upper performance bound of SemCityLoc is constrained by the LoD model coverage and geometric accuracy of the target area. Although CityGML data is widely open, its coverage is uneven globally (dense in Europe, sparse in other regions), and models in some areas may lack semantic annotations or contain geometric deviations.
Failure in scenarios with extremely low geometric observability: When the field of view contains only a single building facade or very few visible structures, even the joint semantic-depth constraint may fail to uniquely determine the pose. This is an inherent boundary of structure-registration-based methods, in contrast to dense feature-matching methods.
Zero-shot semantic segmentation bottleneck: Currently, zero-shot schemes (CLIP + SAM series) only achieve an mIoU of 32-37% in aerial views, which is far from sufficient for reliable localization, still requiring lightweight fine-tuning per dataset. This is both a limitation of this paper and a pending issue for the field—visual foundation models for aerial viewpoints are far less mature than ground viewpoints.
Unexplored extension directions: The paper only evaluates static scenarios, without considering the interference of dynamic objects (vehicles, pedestrians) on rendering masks and depth alignment. Temporal fusion across multiple frames to further enhance robustness also remains to be explored. Furthermore, the potential to extend the method from buildings to other CityGML semantic categories, such as roads and vegetation, has not been fully investigated.

vs LoD-Loc / LoD-Loc v2: Both use LoD building models for aerial localization, but LoD-Loc only uses contour/wireframe alignment, and LoD-Loc v2 extends this to silhouette alignment—both being edge-level matching. The key difference in SemCityLoc is the introduction of semantic surface class information and monocular depth signals, upgrading matching from sparse edges to joint dense region + depth registration. Experiments show this upgrade provides significant gains in all scenarios, with the performance gap widening especially under occlusions and repetitive structures.
vs OrthoLoc: OrthoLoc matches aerial images with orthophotos/DSMs for 6DoF localization but essentially relies on 2D texture information. SemCityLoc entirely discards textures, relying solely on the alignment of semantic labels and geometric structures, which is more advantageous in privacy-sensitive scenarios and regions lacking updated orthophotos.
vs OrienterNet / MapLocNet: These classes of methods use navigation maps (road networks, 2D vector maps of building contours) for localization, but are restricted to 3-DoF (\(x,y,yaw\)). SemCityLoc utilizes the facade information from 3D city models to extend localization to full 6DoF, which is a necessary condition for downstream tasks requiring precise altitude and pitch/roll information (such as 3D reconstruction and building detection).
vs Traditional SfM + Feature Matching: Classical pipelines utilizing SfM reconstruction + SuperGlue/LoFTR matching + PnP are highly accurate but depend on dense textured scene models, requiring substantial storage and computational overhead. SemCityLoc's philosophy is essentially to "replace privately captured dense textured reconstructions with publicly available structured semantic geometries", paving a more viable path for large-scale deployment.

Rating¶

Novelty: ⭐⭐⭐⭐ The perspective shift of converting aerial localization from contour/edge matching to semantic surface registration is novel; the scheme of joint semantic and depth structured alignment is proposed for the first time in the localization domain.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations were conducted across three datasets, multiple LoD levels, different prior noise magnitudes, and varying segmentation qualities; the ablation study clearly dissects the individual contributions of the two stages; and efficiency analysis and qualitative comparisons are provided.
Writing Quality: ⭐⭐⭐⭐ Clear problem definition, self-consistent method pipeline logical flow, and experimental design that fully covers key dimensions. The LoD analysis resonates exceptionally well with the concept of semantic surface registration.
Value: ⭐⭐⭐⭐ Provides strong empirical evidence for the approach of "replacing dense reconstruction with publicly available structured maps for localization"; the SemCityLockeD benchmark fills a gap in the field; sub-second inference makes it immediately viable for real-world deployment.