Recurrent Cross-View Object Geo-Localization¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Temperature-ai/ReCOT.git
Area: Remote Sensing / Cross-View Object Geo-Localization
Keywords: point prompts, learnable tokens, recurrent refinement, SAM distillation, hierarchical attention

TL;DR¶

ReCOT repeatedly reads satellite features with learnable tokens carrying the queried object's semantics, supported by SAM distillation and reference feature enhancement, achieving 48.10% Ground-to-Satellite and 72.05% Drone-to-Satellite [email protected] on the CVOGL test set with 29.9M parameters.

Background & Motivation¶

Cross-view image geo-localization typically asks where a photograph was taken, whereas this paper asks where the object clicked in that photograph appears in a satellite image. These questions are not interchangeable: finding the camera's vicinity does not identify the target building in its field of view. The query can be a ground or drone image, the reference is a high-resolution satellite image, and the output is the specified object's bounding box in the reference. The model must therefore bridge appearance and viewpoint differences while distinguishing the selected object from other objects in the query scene. Methods such as DetGeo encode the point prompt, fuse it with query features, generate reference-image attention, and regress the location in a single detection pass. When several satellite candidates look similar, one-shot aggregation can emphasize background or the wrong building, without a subsequent opportunity to reconsider the evidence.

Simply adding iterations is insufficient because recurrence can repeatedly reinforce the same misleading evidence. A point identifies a pixel but does not inherently specify the object's extent, while reference features are often dominated by background. ReCOT consequently addresses two interdependent requirements: clarify query semantics and reference evidence, then allow the localization state to be revised repeatedly. SAM is not a deployed segmentation system here; it is a training teacher that helps identify which object contains the prompt point. Reference enhancement is likewise not the final one-shot localization decision, but preparation of less distracting evidence for recurrent reading. Core Idea: maintain cross-view object matching as an updatable token state, refine localization through weight-shared attention steps, and use semantic supervision plus hierarchical reference enhancement to prevent the recurrence from accumulating misleading cues.

Method¶

Overall Architecture¶

The inputs are a query image, a point prompt on that image, and a satellite reference image containing the target. A Swin-t image encoder extracts query features and multiscale reference features, with positional embeddings following DETR. The pipeline comprises three designs: Prompt Semantics and Distillation, Hierarchical Reference Enhancement, and Recurrent Localization and Token Supervision. The query branch jointly encodes learnable tokens and image features before injecting the point prompt; the reference branch prepares fixed object-relevant features for repeated token access. Each step predicts boxes, and evaluation selects the highest-confidence output box as the final result, rather than producing a satellite segmentation mask.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Query["Query image and point"] --> Prompt["Prompt Semantics<br/>and Distillation"]
    Teacher["SAM query mask"] -.->|Training supervision only| Prompt
    Reference["Multiscale satellite features"] --> Enhance["Hierarchical<br/>Reference Enhancement"]
    Prompt --> Enhance
    Prompt -->|Initial tokens| Recur["Recurrent Localization<br/>and Token Supervision"]
    Enhance -->|Fixed reference features| Recur
    Boxes["Ground-truth reference box"] -.->|Training supervision only| Recur
    Recur -->|Weight-shared state update| Recur
    Recur --> Output["Object box and confidence"]

Key Designs¶

1. Prompt Semantics and Distillation: understand the scene before identifying the clicked object

The model initializes a set of learnable tokens whose count and channel dimension determine the capacity of the semantic state. They are concatenated spatially with query image features and processed with multi-head self-attention to absorb global query context, producing query tokens. At this point, the tokens describe the image contents but are not yet explicitly constrained by the point prompt. The point embedding then interacts with the concatenated features through cross-attention, introducing information about the specified object into both image features and tokens. Retaining these two states matters: query tokens before prompt interaction support coarse reference filtering, whereas features after prompt interaction inject more precise object semantics. The reference branch therefore does not require a single point to determine a unique satellite candidate immediately.

To teach query features about object extent, training feeds the query image and point prompt into SAM to obtain a pseudo ground-truth mask. The query-image portion of the prompt-interacted features passes through a lightweight convolutional head and sigmoid to predict a mask. Binary cross-entropy and Dice supervision against the SAM mask transfer segmentation priors into the query representation. This supervision operates in the query view; SAM is not asked to directly find the corresponding object in the satellite image. SAM is unnecessary at inference, so its computational cost is not added to the deployed recurrent localization path. However, the supervision still depends on pseudo-mask quality, and the paper does not establish reliable teacher masks under erroneous point prompts.

2. Hierarchical Reference Enhancement: filter scene-relevant regions before injecting object detail

RFEM uses both low-resolution semantic features and high-resolution detailed features, with the latter having 2 times the spatial dimensions of the former. Low-resolution features support scene relevance, while high-resolution features retain boundaries but also contain more background interference. RFEM first aggregates query tokens from before prompt interaction using softmax weighting, yielding a global descriptor of the query image. A dot product between this descriptor and low-resolution reference features, followed by sigmoid, produces a query-relevant spatial attention map. The map is upsampled and multiplied element-wise with high-resolution features to suppress satellite regions unrelated to the query scene. This stage finds potentially relevant regions rather than claiming that object identity has already been resolved.

The spatially filtered high-resolution reference features then use cross-attention to read the prompt-aware query and token features. Coarse filtering uses global scene context, while refinement uses object-level cues, forming the spatial-to-cross-attention hierarchy. The result is downsampled and refined with spatial attention and self-attention to produce the final reference representation. This reference representation stays fixed during recurrent localization; the token state reading it is what changes. Recurrence should therefore not be interpreted as repeatedly running the backbone or repeatedly invoking SAM. Figure 4 and the structural description support this flow; some prime marks differ between prose and diagrams, so the note distinguishes states using the stated prompt semantics.

3. Recurrent Localization and Token Supervision: reinterpret the same evidence step by step

Each step uses current tokens as queries and enhanced reference features as keys and values in multi-head cross-attention. Updated tokens enter a linear prediction layer to estimate that step's location, and the next step reads the reference using the updated state. The recurrent cross-attention and linear prediction layers share weights, so more steps increase computation without adding parameters to these modules. The authors interpret attention visualizations as showing some tokens progressively focusing on the target while others weaken and stabilize on background patterns. This is qualitative evidence, not a requirement that every token represent a separate object or that error decrease strictly at every step. Later prose describes the behavior as residual correction, but the presented structure primarily specifies token updates and per-step predictions; the note does not infer an additional box-residual implementation.

To prevent tokens from learning abstract semantics that cannot localize the target, every step also receives direct region supervision. Its tokens are aggregated along the token dimension using softmax weighting, correlated with the fixed reference features, and passed through sigmoid to produce a reference response map. The ground-truth box becomes a box-level binary mask, and binary cross-entropy plus Dice supervise that response, constituting the token-guidance loss. This mask comes from box annotation, not precise satellite pixel segmentation, and it is not the SAM query mask. The two auxiliary signals respectively constrain which query object the prompt indicates and whether the recurrent state can highlight that object in the reference. Recurrence consequently depends on readable reference evidence and spatially useful semantic states, rather than merely executing the same detection head several times.

A Worked Example¶

Consider clicking a building in a ground photograph and finding it among similar roofs in a satellite image; this illustrates the mechanism rather than introducing an experimental sample. Query self-attention first encodes neighborhood context, and prompt interaction distinguishes the target building from other buildings in the same image. RFEM suppresses irrelevant satellite regions using neighborhood-level correlation, then injects object cues into the high-resolution features that preserve detail. The first token update may still favor a visually similar building, while later updates redistribute attention across candidates using the revised semantic state. Training supervises each step's box predictions and token responses; testing uses only images, the point prompt, and weight-shared recurrence, without ground-truth boxes. The default model trains with 6 unfolded steps and selects 5 inference steps on validation; training with 6 steps does not require deploying all 6.

Loss & Training¶

The overall objective combines per-step DETR-style detection losses, per-step token-guidance losses, and one SAM distillation loss. The following is an equivalent summary of the prose in Section 3.4, not a reconstruction of the damaged attention or aggregation equations in the extraction:

\[ \mathcal{L}=\sum_{i=1}^{m}\mathcal{L}_{\mathrm{Det},i}+\alpha\left(\sum_{i=1}^{m}\mathcal{L}_{\mathrm{Token},i}+\mathcal{L}_{\mathrm{SAM}}\right),\qquad \alpha=1. \]

Both auxiliary mask objectives include binary cross-entropy and Dice, but their supervision sources, views, and target representations differ. The readable main text does not enumerate the internal DETR detection-loss weights, so conventional settings cannot be treated as reported hyperparameters. Experiments use 4 NVIDIA GeForce RTX 4090 GPUs, AdamW, batch size 16, an initial learning rate of \(2.5\times10^{-5}\), and weight decay of \(1\times10^{-4}\). CVOGL methods are trained to convergence, with checkpoints selected on validation before test evaluation. Inference depth is selected per dataset on validation rather than by per-instance adaptive stopping; unfolding 6 training steps supports learning multistep updates.

Key Experimental Results¶

Main Results¶

Each CVOGL direction contains 6,239 image pairs, split into 4,343 training, 923 validation, and 973 test samples. Ground queries have size \(512\times256\), drone queries \(256\times256\), and satellite references \(1024\times1024\). [email protected] and [email protected] are the percentages of samples whose predicted and ground-truth boxes meet the corresponding IoU threshold; IoU is intersection area divided by union area. These are neither geographic errors in meters nor pixelwise segmentation accuracies; only the highest-confidence output box is evaluated for each image pair. The following selection is from Table 4, page 13, with accuracies in %; FPS is measured on Drone-to-Satellite using one RTX 4090.

Method	Ground [email protected]	Ground [email protected]	Drone [email protected]	Drone [email protected]	Parameters M	FPS
DetGeo	45.43	42.24	61.97	57.66	73.8	29.5
TROGeo	51.08	46.56	76.16	68.96	71.3	13.5
GeoFormer	49.54	45.32	73.79	68.96	739.1	11.1
ReCOT, 1 step	49.74	46.25	78.31	71.74	29.9	27.2
ReCOT, 5 steps	52.00	48.10	77.60	72.05	29.9	24.3

Against TROGeo, 5-step ReCOT gains 1.54 and 3.09 percentage points on the two [email protected] metrics, but it is not faster than DetGeo. Within ReCOT, moving from 1 to 5 steps improves Ground [email protected] from 46.25% to 48.10%, whereas Drone [email protected] only rises from 71.74% to 72.05%. Drone [email protected] instead decreases from 78.31% to 77.60%, so the table does not support the claim that recurrence improves every metric.

Ablation Study¶

The following results are from Table 2, page 12, all on the CVOGL test set with 5 recurrent steps and accuracies in %.

Config	Ground [email protected]	Ground [email protected]	Drone [email protected]	Drone [email protected]
Without RFEM	46.45	42.24	50.15	46.04
Without SAM distillation loss	49.74	44.81	70.91	65.78
Without token-guidance loss	50.57	47.58	72.05	66.80
Full model	52.00	48.10	77.60	72.05

Removing RFEM lowers Drone [email protected] by 26.01 percentage points, much more than the 0.31-point gain from increasing recurrent steps within the model. Reference enhancement is therefore a strong dependency, and differences against earlier methods cannot all be attributed to recurrence. Table 3, page 12, further reports Drone [email protected] of 70.30% without the spatial attention map and 67.11% when low-resolution features replace the high-resolution branch. The two resolutions are not interchangeable, and object detail contributes to reference enhancement.

Key Findings¶

Table 1, page 11, selects the step count on validation: Ground [email protected] is 42.25%, 43.66%, and 43.34% at 1, 5, and 6 steps, respectively. The corresponding Ground FPS values are 26.9, 24.1, and 23.4, showing computation cost and nonmonotonic returns; these validation accuracies must not be mixed with test results. CVOGL-few-shot contains only 4 new categories, 28 training samples, and 24 test samples, with 7 training examples per category. Models are initialized from standard CVOGL checkpoints and fine-tuned, so the following is not a zero-shot generalization evaluation. The following selection is from Table 5, page 14, on the Drone-to-Satellite few-shot test set, with accuracies in %.

Method / Steps	[email protected]	[email protected]
DetGeo	16.67	16.67
GeoFormer	45.83	29.17
OCGNet	29.17	25.00
ReCOT, 1 step	37.50	25.00
ReCOT, 2 steps	41.67	29.17
ReCOT, 4 steps	45.83	33.33
ReCOT, 6 steps	50.00	33.33

From 1 to 6 steps, few-shot [email protected] increases by 8.33 percentage points, but on only 24 test samples this represents approximately 2 additional correct predictions. The few-shot results motivate further investigation of recurrent refinement rather than establishing that large-scale open-category generalization is solved.

Highlights & Insights¶

Learnable tokens represent both query intent and recurrent state, connecting query understanding with location updates; weight sharing makes inference depth adjustable without adding parameters.
SAM contributes training supervision to the query branch without placing a large foundation model inside every inference iteration; this suits settings with more training resources than deployment budget.
RFEM uses prompt-free coarse filtering followed by object-aware refinement, explicitly distinguishing scene relevance from object relevance instead of requiring one attention operation to handle both.

Limitations & Future Work¶

The authors explicitly leave per-instance adaptive stopping to future work; fixed dataset-level depth can overcompute easy examples while under-refining difficult ones.
Evidence is mainly from two cross-view tasks and a small few-shot set, without systematic evaluation across cities, seasonal changes, or references missing the target.
SAM pseudo-masks may supervise the wrong region around a point; robustness to prompt perturbations and teacher quality is a reader-proposed direction worth testing separately.
Several extracted equations contain missing symbols, and the supplementary material mentioned in the paper was not supplied for this task; additional hyperparameters such as token count and fine-grained implementation details cannot be confirmed.

Compared with DetGeo, VAGeo, and TROGeo: these methods emphasize one-shot cross-view aggregation and detection, whereas ReCOT repeatedly updates a semantic state against fixed reference features; component ablations nevertheless show that recurrence is not the sole source of improvement.
Compared with CVM-Net, SAFA, and GeoDTR+: image-level methods primarily locate the camera or scene, while this task uses a point prompt to locate a specific object, so image-retrieval accuracy cannot stand in for object-localization ability.
Connections to DETR, SAM, and recurrent matching: ReCOT combines detection-style supervision, prompted segmentation priors, and weight-shared iteration; the transferable insight is to supervise both whether a state points to the target and whether its final location is correct, rather than copying a particular remote-sensing architecture.

Rating¶

Novelty: 4/5. Recurrent semantic states are well motivated for object-level cross-view localization, while the underlying attention and distillation components are established.
Experimental Thoroughness: 4/5. Main comparisons, step counts, component and multiscale ablations, and few-shot analysis are included, although the latter is small.
Writing Quality: 4/5. Task and module motivations are clear, but damaged formula extraction limits implementation-level verification.
Value: 4/5. Stronger localization with 29.9M parameters is useful, though inference speed and gains remain scenario-dependent.