Skip to content

From Visual Primitives to Semantic Masks: Fine-Grained Visual-Linguistic Alignment for Open-Vocabulary Remote Sensing Image Segmentation

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/blackdyc/SANO3
Area: Segmentation
Keywords: open-vocabulary semantic segmentation, fine-grained remote sensing, SAM 3, visual primitive discovery, information-gain prompting

TL;DR

Addressing visual self-similarity and text representation collapse in fine-grained remote sensing scenes, SANO3 introduces a training-free framework that extracts visual primitives from DINOv2 self-similarity priors, aligns them via Plackett-Luce ranking structure matching, injects hybrid prototypes into the text stream, and greedily selects positive/negative point prompts via entropy reduction, achieving state-of-the-art performance on the newly constructed FG-OVSSRS Bench.

Background & Motivation

Open-vocabulary semantic segmentation (OVSS) enables pixel-level recognition guided by arbitrary textual descriptions, unlocking remarkable flexibility for real-world remote sensing tasks such as environmental protection, precision agriculture, post-disaster assessment, and urban mapping. Prior training-free approaches typically build upon CLIP, altering its attention pooling layers to produce dense feature activations or incorporating auxiliary vision foundation models (VFMs) like SAM and DINO to improve spatial localization. However, remote sensing imagery exhibits severe inter-class visual similarity (e.g., distinguishing deciduous trees from coniferous trees, subtle cloud thicknesses, or progressive building damage levels) alongside drastic scale variations. Because CLIP is pre-trained primarily for global vision-language correspondence, it struggles to capture nuanced sub-category distinctions, resulting in rampant semantic confusion and blurred boundaries.

The advent of SAM 3, equipped with promptable concept-driven segmentation objectives, offers substantially stronger geometric and mask-level priors than CLIP. Nonetheless, transferring SAM 3 directly to fine-grained remote sensing OVSS reveals critical limitations. On the semantic side, SAM 3's native text encoder suffers from pronounced feature collapse when handling fine-grained categories, clustering semantically adjacent labels tightly in the latent space and causing naive text queries to cross-activate incorrect regions. On the operational side, existing SAM 3-based remote sensing frameworks rely on simplistic post-processing heuristics that combine segmentation heads with Transformer decoder outputs in a single forward pass, entirely neglecting the iterative, feedback-driven refinement capability intrinsic to promptable architectures. Furthermore, established remote sensing benchmarks predominantly evaluate coarse land-cover categories (e.g., generic water, tree, road), leaving a void for systematic evaluation under subtle semantic distinctions.

To overcome the twin barriers of ambiguous vision-language alignment and coarse boundary delineation, this work turns to the topological self-similarity priors inherently preserved within self-supervised vision representations. Core idea: extract discriminative visual primitives via DINOv2 topological clustering, align them with fine-grained text queries using cross-modal token attention and Plackett-Luce ranking consistency, inject hybrid multi-modal prototypes into the prompt stream, and iteratively select positive and negative spatial point prompts via greedy entropy reduction to collaboratively optimize textual and geometric guidance without any training.

Method

Overall Architecture

The SANO3 framework operates across four integrated stages: unsupervised visual primitive discovery, visual-conditioned relation-aware alignment, hybrid prototype injection, and iterative information-gain visual prompting. Given an input remote sensing image, a frozen DINOv2 backbone extracts dense topological feature maps, which are over-clustered and hierarchically merged into visual primitive masks and prototype vectors. Next, visual prototypes perform cross-attention over text token embeddings to form conditioned representations, and a Plackett-Luce ranking model aligns the visual and textual neighborhood permutation structures to assign semantics reliably. Dynamically filtered instance prototypes and online momentum-updated static prototypes from a global memory bank are concatenated with text embeddings to form a hybrid prompt for SAM 3. Finally, the framework evaluates global prediction entropy and iteratively introduces the most informative positive or negative point prompts in a greedy loop to progressively refine ambiguous boundaries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Remote Sensing Image + Fine-Grained Text Queries"] --> B["Unsupervised Visual Primitive Discovery<br/>DINOv2 Over-Clustering and Hierarchical Merging"]
    B --> C["Visual-Conditioned Relation-Aware Alignment<br/>Token-Level Cross-Attention and Plackett-Luce Ranking"]
    C --> D["Hybrid Prototype Injection<br/>Dynamic Instance and Global Static Bank Prototypes to Text Stream"]
    D --> E["Iterative Information-Gain Visual Prompting<br/>Candidate Pool Mining and Greedy Entropy-Reduction Points"]
    E --> F["Fine-Grained Semantic Mask Output"]

Key Designs

1. Unsupervised Visual Primitive Discovery: Topological Over-Clustering and Hierarchical Merging

Directly seeking fine-grained cross-modal correspondence is prone to noise caused by text ambiguity. In contrast, self-supervised vision models like DINOv2 naturally exhibit intra-modal patch-level self-similarity and topological coherence. SANO3 leverages this visual prior to decompose the input image \(I\). Dense visual feature maps \(F_{vfm}\) are extracted from DINOv2. To capture subtle sub-class distinctions, the feature map is partitioned into \(K_{init} = Q \times C\) disjoint regions using K-Means over-clustering, where \(Q\) is the number of queried classes and \(C\) governs clustering granularity. Because over-clustering can fragment coherent entities, a hierarchical merging mechanism computes cosine similarities between cluster prototypes and iteratively merges adjacent clusters with similarity exceeding 0.85, producing a refined mask set \(\mathcal{M} = \{M_1, \dots, M_K\}\). Aligning these masks to the SAM 3 vision encoder feature space \(F_{sam}\), visual prototypes \(v_k\) are extracted via masked average pooling: $\(v_k = \frac{1}{|M_k|}\sum_{(x,y):M_k(x,y)=1} F_{sam}(x,y)\)$ These prototypes act as grounded physical anchors that retain both low-level geometric boundaries and high-level visual structures for downstream assignment.

2. Visual-Conditioned Relation-Aware Alignment: Cross-Modal Attention and Plackett-Luce Ranking Consistency

Due to feature collapse in SAM 3's text encoder, fine-grained categories (such as thick cloud versus thin cloud) exhibit high cosine similarity, causing naive [EOS] sentence-level matching to misassign labels. SANO3 addresses this via a two-stage alignment protocol: visual-conditioned text refinement followed by dynamic relation-aware matching. In the first stage, each visual prototype \(v_k\) queries the token embeddings \(T_c \in \mathbb{R}^{L \times d}\) of text query \(c\) through cross-attention, yielding a conditioned representation \(t_{k,c} \in \mathbb{R}^d\). This dynamic query mechanism sharpens informative discriminative tokens while suppressing redundant words.

In the second stage, rather than comparing absolute cosine similarities that falter in collapsed spaces, the framework aligns the structural ordering distributions between visual and textual representations using the Plackett-Luce ranking model. For each prototype \(v_k\), the top-\(N\) candidate queries (\(N=4\)) form a local permutation space \(\Pi_N\). The probability of a permutation \(\pi\) under visual-anchored similarities and textual intra-modal similarities are respectively defined as: $\(P(\pi | v_k) = \prod_{i=1}^N \frac{\exp(\cos(v_k, t_{k,(i)})/\tau_t)}{\sum_{j=i}^N \exp(\cos(v_k, t_{k,(j)})/\tau_t)}\)$ $\(P(\pi | t_{k,c}) = \prod_{i=1}^N \frac{\exp(\cos(t_{k,c}, t_{k,(i)})/\tau_t)}{\sum_{j=i}^N \exp(\cos(t_{k,c}, t_{k,(j)})/\tau_t)}\)$ The structural consistency score \(S(k, c) = \sum_{\pi \in \Pi_N} P(\pi | v_k) \cdot P(\pi | t_{k,c})\) measures agreement between visual neighborhood structures and linguistic semantic neighborhoods, assigning the prototype to the highest-scoring query. This distribution-level verification effectively eliminates spurious high cosine scores caused by latent collapse.

3. Hybrid Prototype Injection: Dual Representation Injection into the Text Stream

Relying exclusively on current-image prototypes can expose the model to local illumination artifacts or occlusion noise, whereas using purely static prototypes lacks instance-level adaptability. SANO3 balances both by injecting a hybrid multi-modal prompt into SAM 3's fusion encoder. For each class \(c\), dynamic prototypes \(\mathcal{P}_{dyn}^{(c)}\) passing the confidence threshold \(S(k, c) \ge \tau\) are selected. Concurrently, an online global memory bank updates a class-level static prototype \(m_c\) with momentum: $\(m_c^{(t)} = \eta \cdot m_c^{(t-1)} + (1-\eta) \cdot \text{Mean}(\mathcal{P}_{dyn}^{(c)})\)$ where \(\eta = 0.9\). The text token sequence \(T_c\), dynamic instance prototypes \(\mathcal{P}_{dyn}^{(c)}\), and the global static prototype \(m_c\) are concatenated into an enriched prompt: $\(T_{hybrid} = \text{Concat}(T_c, \mathcal{P}_{dyn}^{(c)}, m_c)\)$ Feeding \(T_{hybrid}\) into SAM 3's fusion encoder allows cross-attention to attend dynamically to fine-grained image details while anchoring predictions against stable category semantics, simultaneously tightening intra-class variance and expanding inter-class margins.

4. Iterative Information-Gain Visual Prompting: Uncertainty-Driven Greedy Spatial Point Optimization

While hybrid prompts resolve semantic assignment, ambiguous boundaries require explicit spatial guidance. Instead of heuristic point sampling, SANO3 formulates prompt placement as an entropy-reduction optimization process. Candidate pools of positive points \(\mathcal{X}_{pos}\) (sampled from the assigned cluster masks) and negative points \(\mathcal{X}_{neg}\) (sampled from other clusters) are ranked by prototype similarity \(S_{proto}(x)\). An initial semantic probability map \(P_t\) is computed, yielding global entropy \(U_{prev} = -\frac{1}{HW}\sum_{i} [P_t^{(i)}\log P_t^{(i)} + (1-P_t^{(i)})\log(1-P_t^{(i)})]\).

At each iteration \(t\), the top positive candidate \(x^*_{pos}\) and negative candidate \(x^*_{neg}\) are simulated to produce candidate probability maps and updated entropies \(U_{new}\). The information gain \(\Delta U = U_{prev} - U_{new}\) is evaluated for both: the candidate producing greater entropy reduction is selected and appended to the prompt set, and \(U_{prev}\) is updated. This greedy iteration dynamically switches between positive guidance in under-segmented cores and negative guidance along ambiguous boundaries, terminating when maximal gain drops below threshold \(\tau_e\).

Key Experimental Results

The FG-OVSSRS Benchmark

To establish rigorous assessment in fine-grained remote sensing OVSS, the paper presents FG-OVSSRS Bench, integrating eight multi-platform datasets: Forest (10 classes; vegetation & fallen trees), Cloud (4 classes; cloud thickness & shadows), Rescue (11 classes; post-disaster structural damage), YRCC (3 classes; river ice & water), SkyScapes (20 classes; aerial urban land cover), GIS (3 classes; urban infrastructure), Fine-RSMI (3 classes; mineral categories), and Barley (3 classes; crop varieties). Synonymous and real-world clutter descriptions are expanded to evaluate robustness against prompt variation.

Main Results

Method Backbone Forest Cloud Rescue GIS YRCC SkyScapes RSMI Barley Avg. mIoU (%)
GEM (CVPR 24) CLIP 12.97 30.25 18.88 31.57 26.56 9.70 7.24 9.06 18.28
ProxyCLIP (ECCV 24) CLIP + DINO 27.37 32.51 33.22 32.82 39.68 15.43 10.61 8.06 24.96
FSA (ICCV 25) CLIP + DINO 27.73 32.70 33.40 33.04 43.23 15.41 10.32 8.03 25.48
SCLIP (ECCV 24) CLIP 9.22 28.10 14.11 30.02 32.02 6.40 12.22 11.05 17.89
ClearCLIP (ECCV 24) CLIP 13.89 32.85 22.31 31.46 26.96 13.23 14.68 9.38 20.60
CorrCLIP (ICCV 25) CLIP+SAM2+DINO 29.33 19.24 33.89 31.81 50.57 15.46 12.10 10.51 25.36
SegEarth-OV (CVPR 25) CLIP 22.91 24.41 26.83 33.78 56.10 14.79 17.95 9.06 25.73
SAM 3 (ICLR 26) SAM 3 33.10 29.56 35.76 41.46 52.66 26.84 29.17 10.55 32.39
SegEarth-OV3 (arXiv 25) SAM 3 33.05 28.80 35.85 41.46 52.24 26.77 29.17 10.53 32.23
SANO3 (Ours) SAM 3 + DINOv2 33.90 30.70 36.84 43.06 57.78 26.83 28.50 18.78 34.55

Ablation Study

1. Component Contribution Analysis (Tab. 3)

Config Dynamic Prototype (DPI) Static Prototype (SPI) Iterative Prompting (IGVP) Cloud (mIoU %) YRCC (mIoU %) Rescue (mIoU %)
SAM 3 Baseline - - - 29.56 52.66 35.76
+ DPI ✓ - - 29.68 55.25 36.67
+ DPI + SPI ✓ ✓ - 30.11 56.46 36.82
Full Model (SANO3) ✓ ✓ ✓ 30.70 57.78 36.84

2. Alignment Strategy and VFM Backbone Selection (Tab. 5 & Tab. 6)

Ablation Dimension Variant / Setting Cloud (mIoU %) YRCC (mIoU %) Rescue (mIoU %) Note
Alignment Strategy Max Cosine Similarity 30.30 57.11 36.42 Relies on absolute similarity; vulnerable to feature collapse
Alignment Strategy VARA (Ours) 30.70 57.78 36.84 Ranking-based alignment consistently improves discrimination
VFM Backbone MAE 30.44 58.64 36.70 Reconstruction-based VFM provides viable visual priors
VFM Backbone DINOv2 (Ours) 30.70 57.78 36.84 Robust and balanced across diverse domains
VFM Backbone DINOv3 30.26 58.84 36.66 Advanced foundation model yields comparable strong gains

Key Findings

  • Dynamic prototype injection provides substantial initial gains on structurally complex targets like YRCC (+2.59%), while static global prototypes add cross-sample semantic regularization (+1.21%). Iterative spatial prompting unlocks the remaining potential (+1.32%), achieving an overall +5.12% surge on YRCC and an +8.23% breakthrough on Barley crops over baseline SAM 3.
  • Hyper-parameter evaluations show that setting clustering granularity \(C \in [6, 8]\) achieves the optimal trade-off between primitive expressiveness and spatial coherence, with hierarchical merging effectively suppressing over-segmentation. Confidence threshold \(\tau \in [0.4, 0.8]\) and termination threshold \(\tau_e \in [0.05, 0.20]\) provide broad stability windows across diverse sensing platforms.
  • SANO3 demonstrates strong architectural generalizability: replacing DINOv2 with MAE or DINOv3 preserves state-of-the-art segmentation accuracy, confirming that the framework's benefits stem from topological visual-linguistic alignment rather than handcrafted feature memorization.

Highlights & Insights

  • Ranking Consistency Resolves Feature Collapse: Replacing flat cosine thresholds with Plackett-Luce permutation distribution alignment elegantly bypasses the severe embedding collapse in SAM 3's text encoder, anchoring fine-grained concepts through relative topological neighborhoods.
  • Active Uncertainty-Driven Spatial Prompting: Formulating prompt placement as an explicit entropy-reduction optimization process transforms prompt-based segmentation from a passive single-pass pipeline into an active feedback loop, sharpening fragile remote sensing boundaries without manual intervention.
  • First Dedicated Fine-Grained RS-OVSS Benchmark: Integrating eight heterogeneous datasets with multi-sensor imagery and expanded real-world vocabularies fills a long-standing gap in fine-grained remote sensing benchmark standardization.

Limitations & Future Work

  • Multi-Pass Simulation Overhead: Calculating information gain requires simulating predictions for positive and negative candidate pairs across multiple iterations, which increases inference latency compared to single-forward baselines and poses throughput bottlenecks for massive gigapixel remote sensing tiles.
  • Vulnerability to Tiny Fragmented Objects: For tiny, sparse targets in extreme high-resolution drone imagery (such as fragmented debris or narrow road markings), unsupervised K-Means and similarity merging can occasionally merge rare categories into dominant background primitives.
  • Future Directions: Integrating test-time adaptation (TTA) with adaptive spatial search budgets could accelerate convergence and improve small-object primitive separation.
  • vs SegEarth-OV / CorrCLIP: Prior CLIP-based methods focus on modifying attention pooling or patch correlations, yet remain capped around 25% mIoU on fine-grained benchmarks due to CLIP's global objective. SANO3 adopts SAM 3's promptable architecture and enriches it with dense DINOv2 self-similarity priors, reaching 34.55% mIoU.
  • vs SegEarth-OV3: SegEarth-OV3 incorporates SAM 3 but applies only rudimentary post-processing over decoder features, failing to overcome text feature collapse and achieving minimal improvement over raw SAM 3 (32.23% vs 32.39%). In contrast, SANO3 explicitly restructures textual features and introduces interactive spatial feedback, delivering consistent domain-wide gains.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant combination of Plackett-Luce ranking alignment with greedy entropy-reduction spatial prompting for training-free segmentation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across an 8-dataset fine-grained benchmark with comprehensive ablations on backbones, components, and hyper-parameters.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clearly articulated motivation, sound mathematical formulations, and coherent architectural presentation.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational paradigm and benchmark for fine-grained open-vocabulary Earth observation intelligence.