Defending from GeoLocalization through Adversarial Road Trips

Conference: ECCV 2026
Paper: ECCV 2026
Area: AI Safety
Keywords: Adversarial Attack, Image GeoLocalization, Privacy Preservation, Beam Search, Geographic Annealing

TL;DR

This paper introduces RoadTrip Attack (RTA), an adversarial defense method against retrieval-based visual geolocalization that reconceptualizes the attack as finding an optimal journey composed of intermediate geographic targets, utilizing adaptive-radius sampling and beam search to achieve state-of-the-art targeted deception and black-box transferability at minimal perturbation budgets.

Background & Motivation

Deep learning-based visual geolocalization has advanced rapidly over recent years. From contrastive retrieval networks matching visual queries against millions of geotagged prototypes to multimodal vision-language architectures performing retrieval-augmented spatial reasoning, modern models can accurately identify the country, city, and even precise geographic coordinates within a few kilometers from a single photograph. However, the widespread availability of such models introduces critical privacy and physical safety risks: malicious scraping entities can effortlessly track users' whereabouts and daily itineraries from casual social media uploads. Adversarial machine learning offers a promising countermeasure by inoculating images with imperceptible perturbations to defend user location privacy.

Standard gradient-based adversarial attacks—such as FGSM, PGD, and Carlini & Wagner (C&W)—face fundamental challenges when directly applied to retrieval-based geolocalization. The multi-modal embedding spaces of contrastively trained vision models (e.g., GeoCLIP) exhibit severe non-convexity and intra-modal discontinuities, exacerbated by deceptive visual similarities across distant geographic landscapes (such as look-alike vegetation or modern architecture). When conventional optimization algorithms greedily drive image features straight toward a distant target embedding, they frequently get trapped in sub-optimal local minima. Consequently, they necessitate excessive perturbation budgets \(\epsilon\) that introduce visually perceptible artifacts. Meanwhile, GeoShield, the only dedicated geolocalization defense baseline, relies on heavy vision-language exposure segmentation to paste localized adversarial patches, suffering from sluggish execution and suboptimal fooling rates under low noise budgets.

The key insight of this paper is to exploit the intrinsic spherical geometry of the physical Earth rather than blindly navigating the unconstrained visual feature space. Instead of marching directly toward the destination in a single rigid direction, the authors conceptualize the adversarial trajectory as an "adversarial road trip"—guiding the model through a succession of dynamically sampled intermediate geographic waypoints that progressively converge toward the chosen destination. The core idea is to reframe the targeted adversarial attack on geolocalization as a multi-path heuristic journey across Earth's physical geometry, dynamically adjusting the exploration radius via Haversine distance and employing beam search to prune suboptimal perturbation trajectories, thereby bypassing feature-space local minima with minimal perturbation budget.

Method

Overall Architecture

RoadTrip Attack (RTA) begins with a query image \(I_q\) located at ground-truth coordinates \(x_0\) and an adversary-designated destination \(x_T\) located over 2,500 km away. The framework operates through an alternating iterative process: at each macro iteration, it dynamically computes a sampling radius proportional to the Haversine distance between the current candidate prediction and the destination. Within this geographic disk, it samples several intermediate geographic targets. It then executes local Projected Gradient Descent (PGD) optimization on each target, and prunes the resulting candidate trajectories using Beam Search to retain only the top-\(K\) candidates closest to \(x_T\) in physical space until the trip converges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query image $I_q$ & adversarial target $x_T$"] --> B["Adaptive Geographic Sampling Radius<br/>Scale radius $R = \eta \mathcal{G}(x, x_T)$ dynamically"]
    B --> C["Retrieval Target Cross-Entropy Optimization<br/>Local PGD updates on candidate perturbations $\delta$"]
    C --> D["Geographic Beam Search Pruning<br/>Retain Top-K candidates closest to $x_T$"]
    D -->|Step < N and distance > 1km| B
    D -->|Converged or reaching 1km threshold| E["Output optimal adversarial image $I_q + \delta^*$"]

Key Designs

1. Adaptive Geographic Sampling Radius: balancing global exploration and fine convergence Directly attacking the final target \(x_T\) traps the gradient ascent in non-convex local minima, while choosing a fixed sampling radius for intermediate targets presents an unavoidable dilemma: a large radius leads to wild oscillations near the goal, whereas a small radius makes cross-continental transitions prohibitively sluggish. RTA introduces an adaptive radius scaling mechanism grounded in physical geodesy. At step \(i\), for any surviving candidate state with predicted coordinates \(x_{m,i}\), the sampling disk radius \(R\) is dynamically computed as \(R = \eta \cdot \mathcal{G}(x_{m,i}, x_T)\) (empirically optimal at scaling factor \(\eta = 0.5\)), where \(\mathcal{G}(\cdot, \cdot)\) is the Haversine distance. When the current prediction is geographically distant, a larger radius permits expansive exploratory leaps across feature barriers. As the adversarial trajectory nears the destination, \(R\) shrinks automatically, contracting the candidate disk into a tight focus around \(x_T\) to enable high-precision local refinement in a manner akin to physical simulated annealing.

2. Retrieval Target Cross-Entropy Optimization: recasting nearest-neighbor retrieval into linear classification Targeted retrieval-based geolocalizers evaluate similarity scores between an encoded query and a pre-computed gallery \(\mathcal{D} = \{\mathbf{e}_0, \dots, \mathbf{e}_C\}\) of geotagged prototype embeddings. RTA exploits the mathematical equivalence between prototype matching and linear classification by formulating the attack objective as minimizing the cross-entropy loss \(\mathcal{L}_{CE}\) targeting the prototype corresponding to sampled waypoint \(\tau_{m,j}\). For each sampled waypoint, local Projected Gradient Descent (PGD) perturbs the image in the direction of the gradient sign while strictly projecting the noise into the \(l_\infty\) norm ball \(B_\epsilon(I_q) = \{\delta \in \mathbb{R}^n : \|\delta\|_\infty \le \epsilon\}\). Under an exceptionally subtle noise budget, this redirects the model's similarity ranking away from the original coordinates toward the desired gallery cluster.

3. Geographic Beam Search Pruning: parallel multi-path exploration with physical distance heuristics Navigating a complex multi-modal landscape along a single optimization trajectory often leads to suboptimal adversarial noise. RTA employs beam search to concurrently explore multiple high-potential paths. The algorithm maintains a beam of \(K\) (default \(K=4\)) candidate pairs \(S_i = \{(x_{m,i}, \delta_{m,i})\}_{m=1}^K\). In each macro iteration, every candidate branches out to \(J\) (default \(J=5\)) independently sampled geographic waypoints, producing \(K \times J\) refined adversarial perturbations via parallel PGD. The surrogate model evaluates the resulting predictions, and the top-\(K\) candidates exhibiting the smallest Haversine distance to destination \(x_T\) are retained for the subsequent round. This systematic pruning keeps computation tractable while unlocking non-obvious optimization pathways that greedy descent fails to discover.

A Worked Example

Consider an example where a user takes a photo in Rome, Italy (\(x_0\)), and seeks to protect location privacy by misleading any geolocalization system to Tokyo, Japan (\(x_T\)), roughly 9,800 km away. 1. Initialization: The initial perturbation is randomly seeded within the \(B_\epsilon\) ball (\(\epsilon = 2/255\)), initializing candidate set \(S_1\). 2. First Leg of the Journey: Since the distance is vast, the adaptive radius is set to \(R = 0.5 \times 9800 = 4900\) km. The algorithm samples 5 geographic waypoints uniformly within 4,900 km of Tokyo (falling across Central Asia or Siberia). Running 5 local PGD steps for each waypoint produces 5 candidate images. Beam search queries the surrogate model and prunes the pool down to the \(K=4\) closest geographic predictions to Tokyo, forming \(S_2\). 3. Progressive Annealing: By step 15, the candidate predictions have migrated to East Asia, now within 600 km of Tokyo. The sampling radius automatically contracts to \(R = 300\) km, densely clustering subsequent waypoints strictly around the Greater Tokyo area. 4. Early Convergence: As soon as any candidate prediction falls within 1 km of the target coordinates, the algorithm triggers early stopping and returns the final adversarial image \(I_q + \delta^*\). The resulting image preserves near-perfect perceptual fidelity (LPIPS of only 0.027), while the geolocalization model outputs Tokyo with high confidence.

Loss & Training

During each internal optimization iteration directed toward an intermediate waypoint \(\tau_{m,j}\) with corresponding gallery index \(T\), the cross-entropy loss is minimized: $\(\mathcal{L}_{CE}\left(\text{enc}(I_q + \delta), \mathbf{e}_T\right) = -\sum_{i=1}^C q_i \log p_i = -\log \frac{\exp(s_T / \tau_s)}{\sum_{j=1}^C \exp(s_j / \tau_s)}\)$ where \(q_i = \mathbb{I}(i=T)\) represents the one-hot target indicator, and \(s_i = \langle \text{enc}(I_q + \delta), \mathbf{e}_i \rangle\) denotes the cosine similarity between the query feature and prototype \(i\). Perturbations are updated via the projected gradient operator: $\(I_q^{(t+1)} = \Pi_{B_\epsilon(I_q)} \left( I_q^{(t)} - \alpha \cdot \text{sign}\left( \nabla_{I_q^{(t)}} \mathcal{L}_{CE} \right) \right)\)$ with step size \(\alpha = \epsilon / 2\). The physical distance across the Earth's surface between two sets of coordinates \((\phi_1, \lambda_1)\) and \((\phi_2, \lambda_2)\) is calculated using the Haversine formula: $\(\mathcal{G}(x_1, x_2) = 2 R_{\text{earth}} \arcsin \sqrt{ \sin^2\left(\frac{\phi_2 - \phi_1}{2}\right) + \cos\phi_1 \cos\phi_2 \sin^2\left(\frac{\lambda_2 - \lambda_1}{2}\right) }\)$ where \(R_{\text{earth}}\) is the mean radius of the Earth. If any candidate achieves \(\mathcal{G}(x, x_T) \le 1\) km, the optimization for that path terminates early.

Key Experimental Results

Main Results

The primary white-box evaluation was conducted on the standard Im2GPS3k benchmark attacking the state-of-the-art GeoCLIP retrieval architecture. Both Ground Truth Accuracy \(\text{ACC}_{\text{GT}}\) (lower is better, indicating successful defense) and Target Accuracy \(\text{ACC}_{\text{Target}}\) (higher is better, indicating precise deception) were evaluated across distance thresholds ranging from 1 km to 2,500 km.

Method	Budget \(\epsilon\)	LPIPS (↓)	\(\text{ACC}_{\text{GT}}\)@1km (↓)	\(\text{ACC}_{\text{GT}}\)@25km (↓)	\(\text{ACC}_{\text{GT}}\)@2500km (↓)	\(\text{ACC}_{\text{Target}}\)@1km (↑)	\(\text{ACC}_{\text{Target}}\)@25km (↑)	\(\text{ACC}_{\text{Target}}\)@2500km (↑)
C&W (\(c=0.1\))	-	0.031	0.43%	1.63%	10.88%	53.79%	55.59%	74.74%
C&W (\(c=1.0\))	-	0.079	0.10%	0.43%	3.97%	81.95%	83.35%	90.52%
FGSM	2/255	0.007	6.64%	15.98%	55.39%	0.33%	0.33%	9.41%
PGD	2/255	0.007	0.40%	1.03%	8.34%	71.81%	73.97%	85.82%
RTA (Ours)	2/255	0.027	0.03%	0.03%	1.03%	93.43%	94.99%	98.50%
FGSM	4/255	0.042	4.57%	12.08%	49.15%	0.40%	0.40%	11.64%
PGD	4/255	0.044	0.00%	0.07%	1.27%	96.63%	98.13%	99.03%
RTA (Ours)	4/255	0.059	0.00%	0.00%	0.73%	98.93%	99.73%	99.93%

In black-box transferability experiments, adversarial perturbations were generated against GeoCLIP and transferred directly to two leading vision-language geolocalization models, Img2Loc and G3. Notably, the victim models utilized the MP16 reference gallery, which is over 40× larger than GeoCLIP's gallery. The results on Im2GPS3k are benchmarked against the specialized defense method GeoShield:

Victim Model	Attack Method	\(\epsilon=2/255\) @1km (↓)	\(\epsilon=2/255\) @2500km (↓)	\(\epsilon=4/255\) @1km (↓)	\(\epsilon=4/255\) @2500km (↓)	\(\epsilon=16/255\) @1km (↓)	\(\epsilon=16/255\) @2500km (↓)
Img2Loc	GeoShield	8.54%	63.43%	7.57%	55.52%	2.74%	32.10%
Img2Loc	RTA (Ours)	2.77%	27.86%	2.20%	24.52%	1.80%	21.19%
G3	GeoShield	12.38%	78.14%	9.71%	68.67%	3.74%	45.01%
G3	RTA (Ours)	1.17%	19.12%	0.80%	15.42%	0.73%	16.08%

Ablation Study

To confirm that RTA's performance gains stem from geographic-aware trajectory optimization rather than extra random restarts or computation, the authors benchmarked several PGD variants under identical perturbation constraint \(\epsilon = 2/255\) on Im2GPS3k: - Multi-start: runs 5 independent PGD optimizations in parallel and selects the best candidate. - Random restart: introduces stochastic noise re-initialization during optimization. - Interpolated: uses 5 fixed intermediate waypoints uniformly interpolated along the great-circle path connecting source and target coordinates.

Configuration	\(\text{ACC}_{\text{GT}}\)@1km (↓)	\(\text{ACC}_{\text{GT}}\)@2500km (↓)	\(\text{ACC}_{\text{Target}}\)@1km (↑)	\(\text{ACC}_{\text{Target}}\)@2500km (↑)	Note
Clean	14.11%	83.82%	-	-	Unperturbed baseline
Standard PGD	0.40%	8.34%	71.81%	85.82%	Greedy direct path
Multi-start (5 paths)	0.90%	5.41%	72.07%	89.19%	More compute, lacks spatial annealing
Random restart	0.45%	8.55%	64.27%	82.90%	Stochastic perturbation exploration
Interpolated (linear path)	0.99%	13.77%	32.13%	68.32%	Deterministic path traps in feature gaps
RTA (Full Model)	0.03%	1.03%	93.43%	98.50%	Adaptive radius + beam search

Perceptual quality evaluations across noise budgets demonstrated: - At \(\epsilon = 2/255\), GeoShield yielded LPIPS = 0.047 and PSNR = 35.26 dB, whereas RTA achieved LPIPS = 0.027 and PSNR = 41.42 dB. - At \(\epsilon = 16/255\), GeoShield's LPIPS surged to 0.305 (producing noticeable visual artifacts), whereas RTA remained at 0.174 with ~3 dB higher PSNR.

Key Findings

• Low-budget supremacy: At \(\epsilon = 2/255\), RTA boosted target accuracy at 1 km from PGD's 71.81% to 93.43%, while suppressing the residual ground-truth accuracy \(\text{ACC}_{\text{GT}}\)@2500km from 8.34% down to 1.03%.
• Failure of deterministic interpolation: Interpolating fixed waypoints along the direct geographic chord degraded \(\text{ACC}_{\text{Target}}\)@1km to 32.13%, confirming that the visual manifold does not align linearly with geographic lines and requires stochastic disk exploration.
• Sensitivity to radius scaling: The scaling factor \(\eta = 0.5\) strikes the optimal balance between global exploration and local precision. Larger scaling factors (e.g., \(\eta = 1.5\)) induce excessive spatial jumps that hinder fine-grained convergence.
• High throughput via parallelization: Because the \(J\) intermediate waypoints can be evaluated concurrently in batch tensors and the high target precision enables frequent early stopping, RTA achieves faster wall-clock execution than standard PGD and runs 8× faster than GeoShield (GeoShield averages 20 s/img vs. RTA's ~2.5 s/img).

Highlights & Insights

• Formulating adversarial search as geographic simulated annealing: RTA moves beyond arbitrary feature-space noise injection by exploiting physical Earth coordinates and Haversine distances to smoothly transition from broad exploratory steps to fine-grained convergence.
• Remarkable black-box transfer across model paradigms: Adversarial samples crafted on a lightweight contrastive retrieval model effectively deceive complex vision-language models equipped with 40× larger galleries, demonstrating that geographic deceptive trajectories uncover fundamental representational vulnerabilities.
• Preserving superior visual fidelity: By navigating optimal low-resistance paths across the feature manifold, RTA accomplishes targeted deception with substantially less noise intensity than prior art, achieving lower LPIPS and higher PSNR.

Limitations & Future Work

• Admitted Limitations: The attack is currently evaluated under unperturbed digital scenarios; robustness against defensive input purifications—such as JPEG compression, feature squeezing, diffusion-based purification (DiffPure), or recovery perturbations—remains to be systematically analyzed.
• Spotted Limitations: In geographic regions characterized by extreme sparsity of visual landmarks (such as deserts, oceans, or polar regions), prototype gradients may become uninformative, potentially slowing down convergence. Furthermore, surrogate model gradient access is still required; fully gradient-free hard-label black-box settings remain unaddressed.
• Improvement Directions: Future extensions could incorporate diffusion priors to inject semantic-level geographic perturbations without relying on pixel-level gradients, or integrate differentiable JPEG transforms during optimization to guarantee robustness across social media compression pipelines.

Related Work & Insights

• vs PGD / C&W: Standard gradient-based attacks pursue a greedy direct trajectory toward the target embedding, frequently stalling in sub-optimal local minima within non-convex multimodal spaces. RTA maintains a beam of diverse paths guided by adaptive geographic waypoints, achieving substantially higher target accuracy under tighter noise constraints.
• vs GeoShield: GeoShield relies on heavy VLM segmentation to overlay localized adversarial patches on position-revealing regions, resulting in high latency (~20 s/img) and conspicuous visual artifacts. RTA applies imperceptible global noise, runs 8× faster, and achieves far stronger black-box transferability at lower perturbation budgets.

Rating

• Novelty: ⭐⭐⭐⭐⭐ (Formulating adversarial evasion as a physical geographic road trip with adaptive distance annealing is remarkably elegant and conceptually refreshing.)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive empirical validation across Im2GPS3k and YFCC4k, covering white-box attacks, large-scale black-box transfer, four PGD variants, perceptual quality metrics, runtime, and hyperparameter sensitivity.)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation, compelling motivation, well-structured algorithms, and insightful analysis of optimization landscapes.)
• Value: ⭐⭐⭐⭐⭐ (Provides an indispensable framework for safeguarding location privacy against automated visual tracking and surveillance technologies.)