Skip to content

CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentation

Conference: ECCV2026
Paper: ECCV 2026 / project page
Area: Segmentation
Keywords: Referring Remote Sensing Segmentation / Vision-Language Model / Cascaded Distillation / Contrastive Learning / SAM

TL;DR

CROSS distills text-filtered patch-to-patch geometric affinities from the SAM 2 encoder into the shallow, deep, and final layers of SigLIP 2 as a soft structural regularizer, then adds dual-constrained contrastive learning (perceptive hard negatives plus spatial counterfactual texts) to break the object-centric shortcut, reaching 83.25 cIoU and 41.32 [email protected] on RefSegRS.

Background & Motivation

Referring Remote Sensing Image Segmentation (RRSIS) asks a model to segment a specified target in satellite or aerial imagery given one natural-language description. It looks like ordinary referring image segmentation (RIS), but the difficulties are entirely different: remote sensing images span an immense visual scale while targets often occupy only tens of pixels, co-occurring instances of the same class (cars in a parking lot, airplanes on an apron, identical courts in a stadium complex) are extremely homogeneous, and small targets are dense and spread across scales. Because the task demands precise pixel-level localization rather than generative text capability, the field has moved away from hand-designed architectures (LAVT; RMSIN with rotation-aware convolutions for arbitrary orientations; FIANet with multi-scale fine-grained alignment) toward an assembly of discriminative foundation models: a VLM such as SigLIP/CLIP produces a dense semantic prompt, which SAM/SAM 2 decodes into a mask. RSRefSeg and RSRefSeg 2 are the representative works on this line.

Yet these pipelines largely coast on the pre-trained capabilities of the base models rather than achieving genuine multimodal synergy for the dense, cluttered remote sensing scene, and the authors dissect this superficial integration into two bottlenecks. The first is architectural weak-coupling: the VLM produces a prompt and SAM consumes it, so information flows strictly one way and SAM's image encoder — the component that excels at pixel-level structure — is entirely unused while the prompt is being generated. Since a CLIP-style vision encoder carries no structural constraint of its own, its dense prompt response is spatially diffuse on cluttered scenes, which shows up as localization drift. The second is object-centric semantic bias: CLIP-style models are pre-trained for global image-text alignment, and the laziest strategy they learn is to attend to the dominant object category ("vehicle") while receiving very weak supervision for spatial modifiers such as "left" or "inside". The authors' perturbation probe makes this failure mode concrete — swap the spatial words in the referring expression and the baseline still reports a high image-text similarity while its mask is already entirely wrong. The model is doing category retrieval, not logical grounding, which also explains why it segments both instances in expressions such as "the ground track field inside the gray large stadium".

Core idea: SAM's encoder already holds class-agnostic structural priors, so feed them back into the VLM by distilling text-filtered pairwise affinity matrices into multiple intermediate layers as a soft regularizer (LGCD), keeping the VLM's semantic localization continuously constrained by geometry; at the same time, mine hard negatives from both the visual and the linguistic side for things that look right but violate the spatial constraint (PSCL), explicitly penalizing the semantic shortcut so the model is forced to decode the spatial relations in the sentence.

Method

Overall Architecture

The inference backbone of CROSS is still "VLM emits a prompt, SAM 2 decodes the mask": SigLIP 2 encodes the remote sensing image and the referring expression into a visual feature map and text embeddings, a cross-modal prompter computes the dense prompt \(P\) from their similarity, and \(P\) together with the SAM 2 encoder's image features goes into the SAM 2 mask decoder to produce the binary mask. Two things are actually changed. At training time, LGCD cascadedly distills the SAM 2 encoder's pairwise geometric affinity matrices into layers 9, 18, and 27 of SigLIP 2, so that the visual state at each cascade stage is constrained by structural priors; concurrently, PSCL builds two groups of hard negatives on top of the mask prediction — the background distractors outside the mask that look most like the target, and counterfactual texts with swapped subject/object or inverted spatial words — and pushes both away with an asymmetric InfoNCE. The first treats architectural isolation, the second treats semantic bias. At inference only the "text → prompt → mask" trunk remains; the distillation teacher branch and the contrastive branch exist during training only.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Remote sensing image + referring expression"] --> B["SigLIP 2 dual encoder<br/>visual features + text embeddings"]
    K["SAM 2 encoder (frozen)<br/>teacher: patch-pair affinity"] -.->|training only| C["Linguistic-Guided Cascaded<br/>Distillation (LGCD)"]
    B --> C
    C --> D["Cross-modal prompt → SAM 2 mask decoding"]
    D --> E["Predicted mask"]
    E --> F["Perceptive Hard Negatives"]
    E --> G["Spatial Counterfactual Negatives"]
    F --> H["Asymmetric InfoNCE dual constraint"]
    G --> H

Key Designs

1. Linguistic-Guided Cascaded Distillation (LGCD): distill SAM 2's pairwise geometric affinity into the VLM layer by layer as a soft regularizer

This design targets the localization drift caused by the unidirectional flow, and it consists of two steps. The first is cascaded representation extraction. Instead of relying on a single terminal output, it harvests one feature map from each of the shallow, deep, and final blocks of the SigLIP 2 encoder (layers 9, 18, 27 in the experiments), unrolls them into patch token sequences, and injects them residually stage by stage as \(X_i = H_{i-1} + F_v^{(i)}\) with \(H_0=0\) (all three maps share the same patch count, so direct summation is valid). Each stage then adds a bidirectional cross-attention that anchors this visual state to the linguistic context using the text embeddings as keys and values, yielding the refined \(H_i\) and, as a by-product, the attention matrix \(A_i\) over visual patches and text tokens.

The second step is what actually transfers the structural prior, and two design choices here are worth recording. First, remote sensing backgrounds are extremely cluttered, so densely distilling SAM's class-agnostic features indiscriminately would inject a large amount of structural noise; therefore the cross-attention matrix is averaged along the text-token dimension into a per-location "text relevance" soft mask that weights the distillation, routing only linguistically relevant topology, with a lower bound \(\alpha\) that preserves basic background consistency so the background topology does not collapse entirely. Second, SAM's latent space and the VLM's are heterogeneous, and pixel-wise alignment between them distorts features, so the method aligns relative topology instead: both sides compute a Gram affinity matrix of pairwise cosine similarities between patch features, and the VLM is asked to match SAM's pairwise similarity structure. The distillation loss is a text-weighted difference between the two affinity matrices over all spatial pairs, averaged over the \(K\) cascade stages:

\[\mathcal{L}_{\text{distill}}=\frac{1}{K}\sum_{k=1}^{K}\frac{1}{|\Omega|}\sum_{(p,q)\in\Omega} w_{p,q}^{k}\cdot\big\|\mathcal{G}(H_k)_{p,q}-\mathcal{G}(S)_{p,q}\big\|^{2},\qquad w_{p,q}=\alpha+(1-\alpha)\cdot\frac{M_{\text{text}}(p)+M_{\text{text}}(q)}{2}\]

where \(S\) is the SAM 2 encoder feature map, \(\Omega\) contains all spatial location pairs, and \(\alpha=0.1\). (⚠️ This equation is garbled in the cached text; it is reconstructed here as a text-weighted difference of affinity matrices — refer to the original paper for the exact norm.) This is fundamentally different from distilling SAM's masks as hard pseudo-labels: CROSS does not treat SAM 2 as a segmentation oracle and only transfers its preference for "which two patches belong to the same structure", so SAM's decoding errors cannot be baked into the VLM. The cascade provides deep supervision — shallow layers receive coarse structure and deep layers fine topology, and a shared geometric anchor regularizes the representational trajectory into something smoother (corroborated by CKA layer-wise similarity heatmaps, which show smoother transitions between adjacent layers and more stable alignment between the final layer and preceding deep hierarchies), while the layer-specific text soft mask prevents this shared constraint from flattening feature diversity across layers.

2. Perceptive Hard Negatives: mask out the target with the GT mask and mine the background distractors that look most like it

This is the visual side of the attack on object-centric bias. The recipe is direct: use the ground-truth mask to physically remove the target region from the feature map, and among the remaining background \(\Omega_{bg}\) pick the Top-K locations with the highest cosine similarity to the text embedding as hard negatives (\(K=8\)). The motivation comes from a concrete pathology of remote sensing: in an expression like "the ground track field inside the gray large stadium", the identical court excluded by the language is precisely the background region most similar to the text — it satisfies category matching while violating the spatial constraint. That is exactly the trap that misleads the baseline into segmenting both same-class instances. Labelling these look-alike background regions as negatives creates a dedicated penalty for "categorically right but spatially wrong"; randomly sampled background negatives are nearly useless because their similarity to the text is too low to produce meaningful gradients.

3. Spatial Counterfactual Negatives: swap left/right and subject/object so the model must read syntax, not keywords

This is the linguistic-spatial side of the same attack. Rather than building a geometric model, the authors synthesize logical contradictions on the text side: absolute spatial words are inverted ("top" becomes "bottom"), while relative relations have their subject and object swapped wholesale ("A golf field is on the left of the green airport" becomes "The green airport is on the left of a golf field"). The perturbed sentences are generated offline in batches by a lightweight LLM (Qwen2.5-7B-Instruct in the experiments). Such a negative pair — almost identical wording, opposite logic — closes off the keyword-matching shortcut: to keep scoring well, the model has to actually decode the syntactic structure. Designs 2 and 3 together are what the title calls the "Dual-Constraint": one constrains the visual side by suppressing spatially misplaced distractors, the other constrains the linguistic side by penalizing spatial-logical inconsistency. The ablation shows that keeping only the linguistic-spatial perturbation actually hurts (RefSegRS cIoU −0.05), because spatial cues without visual grounding are themselves ambiguous in cluttered scenes; both must be present.

A Worked Example

Take the nested expression "The airplane is on the right of the airplane on the left" from RRSIS-D. After SigLIP 2 encoding, the three LGCD stages filter SAM's affinity structure with the text soft mask and press the pairwise topology of "each airplane's connectivity plus who is left of whom" into the VLM's intermediate layers; the dense prompt produced by the prompter therefore concentrates on the left airplane (in the paper's visualization CROSS's prompt heatmaps are markedly more concentrated than RSRefSeg 2's). During training, PSCL masks out the left airplane with the ground truth and picks the 8 highest-similarity remaining locations — the top ones are typically the fuselage of the right airplane — while the counterfactual text \(t^-\) obtained by swapping the spatial words is pushed away from the positive prototype \(\mathbf{v}^+\), and the embedding of that right airplane in the background is pulled away from the original text \(t\). On this same sentence the baseline RSRefSeg 2 collapses into the partial clause "on the right" and cuts out only the right airplane; CROSS, squeezed from both sides by structural constraints and the two negative sets, lands the full mask on the left target.

Loss & Training

PSCL unifies the two constraints in one asymmetric InfoNCE: the positive pair is the mask-aggregated visual prototype \(\mathbf{v}^+\) with the original text \(t\), while the negative side holds both the visual hard-negative set \(\mathcal{V}^-\) and the counterfactual text \(t^-\), the latter weighted by a scaling coefficient \(\eta\):

\[\mathcal{L}_{\text{PSCL}}=-\log\frac{\exp(\langle \mathbf{v}^{+},F_t\rangle/\tau)}{\exp(\langle \mathbf{v}^{+},F_t\rangle/\tau)+\sum_{\mathbf{v}^{-}\in\mathcal{V}^{-}}\exp(\langle \mathbf{v}^{-},F_t\rangle/\tau)+\eta\cdot\exp(\langle \mathbf{v}^{+},F_{t^{-}}\rangle/\tau)}\]

The asymmetry lives in \(\eta\) (default \(\eta=2\)): the spatial-logical constraint is amplified twofold, so "the position was stated backwards" is punished more heavily than "there is something similar in the background". \(\tau=0.07\). (⚠️ This equation is garbled in the cached text; it is reconstructed from the surrounding description — refer to the original paper for the exact term-by-term form.)

The overall objective is the usual segmentation losses plus the two auxiliary terms: \(\mathcal{L}_{total}=\lambda_{ce}\mathcal{L}_{ce}+\lambda_{dice}\mathcal{L}_{dice}+\lambda_1\mathcal{L}_{distill}+\lambda_2\mathcal{L}_{PSCL}\), with \(\lambda_{ce}=\lambda_{dice}=5.0\) (following RSRefSeg 2) and the auxiliary weights tuned to \(\lambda_1=0.5\) and \(\lambda_2=0.2\). The backbones are siglip2-so400m-patch16-512 and sam2.1-hiera-large, both encoders fine-tuned with LoRA (\(r=16\)); only the newly introduced low-rank modules, the LGCD module, and the SAM decoder are trainable while the backbone weights stay frozen. Inputs are resized to 512 and 1024 respectively with no data augmentation. Training uses AdamW with a peak learning rate of \(1\times10^{-4}\), batch size 8, 300 epochs, BF16 precision, DeepSpeed ZeRO-2, and 8 NVIDIA RTX PRO 6000 GPUs.

Key Experimental Results

Main Results

Evaluation is on two standard benchmarks: RefSegRS (512×512, 2,172/413/1,817) and RRSIS-D (800×800, 12,181/1,740/3,481). Metrics are cIoU, gIoU, and Pr@X.

Dataset Method Publication [email protected] [email protected] cIoU gIoU
RefSegRS LAVT CVPR'22 51.84 2.09 71.86 47.40
RefSegRS RMSIN CVPR'24 79.20 3.25 75.72 62.58
RefSegRS FIANet TGRS'24 84.09 7.10 78.32 68.67
RefSegRS RS2-SAM 2 AAAI'26 84.31 21.19 80.87 73.90
RefSegRS RSRefSeg-2 TGRS'26 88.22 34.40 81.24 77.39
RefSegRS CROSS — 88.61 41.32 83.25 79.51
RRSIS-D LAVT CVPR'22 69.52 24.94 77.19 61.04
RRSIS-D RMSIN CVPR'24 74.26 24.53 77.79 64.20
RRSIS-D RS2-SAM 2 AAAI'26 77.56 29.73 78.99 66.72
RRSIS-D RSRefSeg-2 TGRS'26 80.23 31.05 79.45 69.17
RRSIS-D CROSS — 81.24 32.82 79.89 68.92

Against the strongest competitor RSRefSeg 2: on RefSegRS, cIoU +2.01, gIoU +2.12, [email protected] +6.92; on RRSIS-D, cIoU +0.44, [email protected] +1.01, [email protected] +1.77 — but gIoU is 0.25 lower, and [email protected] (74.56 vs 75.78) and [email protected] (64.80 vs 65.41) are also slightly lower. The paper explains this by the extreme scale diversity of dense small objects in RRSIS-D: the model prioritizes global logical grounding and high-precision localization over boundary fitting.

On parameters (Table 6): the baseline (SigLIP 2 + SAM 2) has 1.438 B total / 88.41 M trainable (6.15%), RSRefSeg 2 has 1.447 B / 97.87 M (6.76%), and CROSS has 1.455 B / 106.09 M (7.29%) — only 1.23% more trainable parameters than the baseline on a 1.4 B+ backbone. The paper does not report inference latency, FPS, or FLOPs; see Limitations below.

Ablation Study

Component ablation (Table 3a, cIoU %, baseline is SigLIP 2 + SAM 2):

Config CRE TGD \(\mathcal{L}_{PSCL}\) RefSegRS RRSIS-D
Baseline ✗ ✗ ✗ 80.82 77.80
+ CRE ✓ ✗ ✗ 82.55 78.63
+ CRE + TGD ✓ ✓ ✗ 83.15 79.02
PSCL only (no LGCD) ✗ ✗ ✓ 82.11 78.12
Full ✓ ✓ ✓ 83.25 79.89

PSCL negative composition (Table 3b, cIoU %):

Perceptive Spatial RefSegRS RRSIS-D
✗ ✗ 83.15 79.02
✓ ✗ 83.21 79.34
✗ ✓ 83.20 78.70
✓ ✓ 83.25 79.89

Distillation layer selection (Table 5, cIoU %):

Layer indices RefSegRS RRSIS-D
{18, 24, 27} 83.19 79.71
{5, 15, 27} 82.76 79.42
{25, 27, 27} 83.14 79.50
{9, 18, 27} (used) 83.25 79.89

Key Findings

  • The gain comes mainly from LGCD, not from the contrastive learning the title advertises. CRE alone adds +1.73 (80.82→82.55) and adding TGD reaches 83.15; PSCL then contributes only +0.10 on RefSegRS. The "PSCL without LGCD" configuration already reaches 82.11 (+1.29), which indicates the two mechanisms partially overlap in function: structural constraints themselves suppress spatial ambiguity. The authors' own reading is that PSCL supplies the discriminative power against visually similar distractors, making it a refinement rather than the main source.
  • The gains are concentrated at strict thresholds. On RefSegRS, [email protected] rises from 34.40 to 41.32 (+6.92) while [email protected] gains only +0.39; on RRSIS-D, [email protected] gains +1.77 versus +1.01 at [email protected]. This distribution is consistent with the claim of mitigating representational drift so masks adhere strictly to target boundaries — at loose thresholds the baseline already hits the target, and the difference only becomes visible in the high-precision regime.
  • Neither side of the negatives can be dropped. Perceptive negatives only give 83.21/79.34; spatial counterfactual negatives only give 83.20/78.70 (below the 79.02 with no negatives at all); both together give 83.25/79.89. The authors attribute the drop to spatial cues without visual grounding being too ambiguous in cluttered remote sensing scenes.
  • Hyper-parameters are fairly robust: \(\lambda_1=0.5\), \(\lambda_2=0.2\), \(K=8\), \(\alpha=0.1\) all stay above the baseline across the wide sweep in Fig. 3; \(\lambda_1\) beyond 0.5 causes gradient dominance and destabilizes joint optimization, and \(\alpha=0.5\) over-suppresses potential target features. The layer indices are explicitly not claimed to be strictly optimal — the goal is simply to cover different intermediate stages — and only excessively shallow choices ({5,15,27}) cost 0.49.
  • Two internal inconsistencies are worth noting: the implementation details state that 7.2% of parameters are updated while Table 6 reports 7.29%; and the RRSIS-D discussion mentions a "Filter-Refine-Verify paradigm" that is never defined elsewhere. ⚠️ Treat the tables/original text as authoritative.

Highlights & Insights

  • Replacing the teacher's masks with affinity matrices is the trick most worth stealing. Mask-level distillation forces the student to pay for the teacher's wrong masks, whereas a Gram matrix only expresses "which two patches belong to the same structure", decoupled from both category and the teacher's decoding errors; adding a text soft mask for conditional routing yields a soft regularizer that injects geometry without polluting the semantic space. Any setting that wants structural priors but fears semantic contamination (medical imaging, remote sensing, depth-guided segmentation) can adopt this form directly.
  • Using the text attention matrix as a soft mask kills two birds with one stone: it is a by-product of the cross-attention (no extra computation) and it inherently carries linguistic relevance, so it can serve as the spatial weight of the distillation. Recycling a mask from an intermediate quantity that has already been computed is far cheaper than training a separate relevance-scoring head.
  • Both hard-negative constructions match a concrete pathology of remote sensing. On the visual side, picking "the background region outside the mask that most resembles the text" mirrors "the same class of ground objects is everywhere"; on the linguistic side, swapping subject and object rather than swapping words mirrors the "matches keywords without decoding syntax" failure. The negatives are reverse-engineered from two specific failure chains, not sprinkled on top.
  • Generating the counterfactual texts offline with Qwen2.5-7B-Instruct, outside the training loop, is a lightweight way to land this: it turns "the model must understand syntax" from a structural problem into a data-construction problem.

Limitations & Future Work

  • No inference-cost report. The paper only gives parameter counts (Table 6) — no latency, FPS, memory, or FLOPs. Given that the N×N Gram matrices are non-trivial at 1024×1024 inputs and the method still requires a SAM 2 encoder forward pass, the deployment cost is unknown, and this is the largest information gap for judging practical value.
  • The emphasis in the title does not match the ablation weights. The title highlights "Dual-Constraint Grounding", yet PSCL's net gain is only +0.10 cIoU on RefSegRS and the bulk of the improvement comes from LGCD; meanwhile "PSCL without LGCD" already reaches 82.11, so the two are not cleanly independent and readers cannot easily judge the marginal value of each constraint.
  • The win on RRSIS-D is lopsided: cIoU / [email protected] / [email protected] lead, but gIoU and [email protected], [email protected] all trail RSRefSeg 2. The paper frames this as prioritizing global logic over heuristic boundary alignment, but it can equally be read as degraded boundary quality on dense small objects — if downstream use requires areas from the mask or change detection, this trade-off needs further validation.
  • The counterfactual texts have limited coverage. The perturbation templates are only "invert spatial words" and "swap subject and object"; multi-clause and nested relations (e.g. "the one to the right of the airplane on the left"), quantifiers, and comparatives are not covered — and the hardest example the paper itself cites is a nested expression. The diversity and correctness of the perturbations are also not manually audited.
  • Validation is limited to two remote sensing benchmarks (RefSegRS, RRSIS-D), with no cross-sensor (SAR, multispectral, infrared), cross-resolution, or open-vocabulary generalization experiments, and no discussion of whether the SAM 2 teacher overlaps in domain with the training data.
  • Possible improvements: replace the Gram-matrix distillation with a low-rank or randomly sampled approximation so the cascaded distillation can span more layers; generate richer counterfactuals automatically from relation graphs or syntax trees instead of template substitution; add an explicit relative-position prediction head to test whether geometric supervision can better carry what the linguistic-side constraint is currently doing.
  • vs RSRefSeg / RSRefSeg 2: same "VLM produces prompt, SAM decodes" family, and the direct baseline here. The difference is that their flow is strictly unidirectional (the SAM encoder idles while prompts are generated) whereas CROSS distills SAM's structure back into multiple VLM layers and adds a contrastive constraint; the price is an extra cascaded distillation module and the corresponding training overhead, with the payoff most visible at high precision thresholds.
  • vs MLLM-based methods such as LISA / EVF-SAM: these use multimodal large language models to produce text-driven embeddings for SAM and excel at holistic understanding and complex reasoning, but the paper deliberately avoids that route — RRSIS needs pixel-level localization rather than generative text capability, and the authors state that preliminary evidence shows MLLM-based methods remain less competitive on such dense tasks. CROSS is an enhanced member of the CLIP-based family instead.
  • vs dedicated architectures such as LAVT / RMSIN / FIANet: they redesign fusion modules for referring segmentation (language-aware ViT, rotation-aware convolutions, multi-scale fine-grained alignment) without relying on the foundation-model prompting paradigm. CROSS beats them by a wide margin in the tables (RRSIS-D cIoU 79.89 vs 77.79 for RMSIN), but that comparison deserves a caveat: CROSS uses a 1.4 B+ pre-trained backbone, so it is a cross-paradigm comparison of foundation-model adaptation against from-scratch architecture design.
  • vs cascaded alignment work such as Cascade-CLIP: those refine semantics internally stage by stage, whereas CROSS uses an external geometric prior (SAM's affinities), replacing structure from the model's own knowledge with an observation from another foundation model — which is what "cascaded distillation" actually means here.

Rating

  • Novelty: ⭐⭐⭐⭐ Distilling SAM's pairwise affinities (rather than masks) into multiple VLM layers as a soft regularizer, with text attention reused as a spatial soft mask, is a rare reverse information flow in this VLM+SAM line; the contrastive half is on-topic but conventional.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Two benchmarks with four ablation groups (components, negatives, layer selection, hyper-parameters), a parameter comparison, and visualizations plus CKA analysis are fairly complete, but there is no inference cost, no cross-domain or cross-sensor validation, and the explanation for the inferior RRSIS-D metrics is thin.
  • Writing Quality: ⭐⭐⭐⭐ The two bottlenecks and the perturbation evidence are laid out clearly, and the figures (paradigm comparison plus spatial logic probe) are persuasive; points off because the dual constraint emphasized in the title contributes very little and because of numeric inconsistencies such as 7.2% vs 7.29%.
  • Value: ⭐⭐⭐⭐ Comprehensive SOTA on RRSIS with a small parameter overhead, and the affinity-matrix distillation mechanism transfers to other tasks that want structural priors without semantic contamination; but the inference cost is unknown and the gains are concentrated at strict thresholds, so practical value still needs validation.