Blind to Position, Biased in Language: Probing Mid-Layer Representational Bias in Vision-Language Encoders for Zero-Shot Language-Grounded Spatial Understanding¶

ECCV 2026 Interpretability Mid-layer probing Spatial sensitivity Multilingual representation shifts Zero-shot referring image segmentation Vision-language encoders

Conference: ECCV 2026
Paper: Official paper page · Paper PDF
Code: https://github.com/kaist-cvml/Biased2Grounded
Area: Interpretability
Keywords: Mid-layer probing, Spatial sensitivity, Multilingual representation shifts, Zero-shot referring image segmentation, Vision-language encoders

TL;DR¶

B2G probes frozen vision-language encoders and reuses spatially sensitive visual features and more language-consistent text features for region ranking, increasing average multilingual referring segmentation mIoU from 42.42 to 49.62 across nine benchmarks, with additional multilingual inference cost.

Background & Motivation¶

Zero-shot referring image segmentation (RIS) identifies the pixels belonging to the object described by a sentence. A common pipeline generates candidate masks with SAM or Mask2Former, then uses a frozen vision-language encoder (VLE) to rank the corresponding regions against the query. This avoids training a task-specific segmentor, but makes the scoring interface a bottleneck: recognizing that an image contains birds is not equivalent to identifying which bird the description refers to. A final-layer embedding optimized for global image–text matching may discard distinctions needed to choose between nearby instances, even when the correct mask is already available.

Multilingual queries expose another weakness of this interface. The image and target mask can remain unchanged while a change in query language produces a different localization result. Here, “bias” concerns language-dependent geometry in the shared embedding space, rather than social stereotypes. The paper uses RIS as a diagnostic testbed to ask whether positional and cross-language information is genuinely absent or merely difficult to access through the final layer. Its central contribution is representation probing followed by intervention, which motivates the Interpretability classification rather than treating it primarily as a new mask-generation architecture.

The authors find spatially sensitive intermediate vision features and intermediate text features with stronger consistency across equivalent multilingual expressions. Instead of retraining the encoder, they select these internal representations and connect them back to the existing ranking pipeline. Core idea: distinguish information retained inside a model from information exposed by its final interface, and test whether a mid-layer spatial map and multilingual centroid injection can recover useful grounding signals.

Method¶

Overall Architecture¶

Biased to Grounded (B2G) takes an image, a referring expression, and candidate masks from an existing segmentor; its multilingual version also uses semantically equivalent expressions in several languages. Its output remains one of the supplied masks, not a newly generated segmentation. The visual branch constructs a spatial map, P-Map, through layer probing, while the text branch constructs a stabilized query through multilingual centroid injection; spatial reranking then selects the target.

The proposal generator, region cropping or masking operation, and encoder weights remain unchanged. The non-multilingual setting uses EConText, which combines sentence and local noun-context information; the multilingual setting substitutes EM-ConText for both patch matching and candidate scoring. CT denotes contextual text fusion, and TC denotes P-Map-based candidate selection. Experiments also use a spatial guider, SG, inherited from earlier work; SG should not be presented as an independent probing mechanism introduced by B2G.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image, query, and candidate masks"] --> B["Visual Layer Probing<br/>and Spatial Map"]
    A --> C["Multilingual<br/>Centroid Injection"]
    C -->|stabilized query constructs P-Map| B
    B --> D["Spatial Reranking"]
    C -->|stabilized candidate semantic scoring| D
    D --> E["Selected target mask"]

Key Designs¶

1. Visual Layer Probing and Spatial Map: locate positional sensitivity before reading target responses

A final-layer similarity score cannot reveal whether a localization failure reflects missing spatial knowledge or an unsuitable readout. B2G partitions an image into non-overlapping tiles, shuffles their arrangement, and encodes both the original and shuffled images. It then reverses the permutation on the shuffled patch features to restore correspondence with the original content. At each candidate layer, the probe score Γ is the mean patchwise cosine similarity between the original features and these correspondence-restored features. A larger Γ means the features remain similar despite changed spatial surroundings, indicating weaker dependence on arrangement; a smaller value indicates stronger positional sensitivity. This is sensitivity to a controlled perturbation, not proof that the model understands every spatial relation.

The paper considers a fixed intermediate layer, B2Gfix, permutation-based selection, B2Gdyn-perm, and cluster-coherence selection, B2Gdyn-clus. Automatically selected layers tend to lie in the middle of the encoders, and the three variants achieve similar accuracy, although the best layer depends on the backbone. The fixed option is consequently attractive when inference cost matters, while dynamic selection avoids assuming that every encoder produces a sharp spatial map at the same depth. Exact cluster-coherence definitions and full dynamic-selection rules are delegated to an appendix absent from the local source, so they are not reconstructed here.

At the selected visual layer, patch features pass through the appropriate normalization and pretrained projection interface before comparison with text. Raw intermediate-layer cosine responses can exhibit “opposite visualization”: target foreground features are negatively correlated with the query, leaving high responses on the background. B2G negates patch features before normalization and the shared projection to obtain responses better suited to foreground localization; SigLIP-family models without an image-to-text projection use the corresponding post-layer normalization interface. This is an empirical, deterministic correction rather than a learned objective. The cached equations for this transformation are extraction-corrupted, and negation through a complete layer-normalized transformation should not be equated with an exact cosine sign-flip identity. The mechanism is therefore explained in prose without reconstructing the damaged formulas.

2. Multilingual Centroid Injection: fuse meaning before later layers amplify language differences

Averaging final-layer multilingual embeddings can reduce variation, but accepts the language-dependent geometry already produced by that interface. B2G instead encodes equivalent expressions to candidate intermediate layers and reads the end-of-text (EOT) hidden state for each language. Its consistency score Φ is the average cosine similarity over all unordered language pairs, and the layer with the highest Φ is selected. The probe uses neither target masks nor image ground truth, but assumes that the language versions actually preserve the same meaning.

At the chosen layer, the arithmetic mean of the multilingual EOT states forms a centroid that replaces the EOT state of the anchor-language expression. The anchor is the original query language, not necessarily English. The modified state then continues through the remaining text layers and final projection, rather than being used directly as the final image–text embedding. This is a test-time intervention on a frozen model's activations, not an update to its parameters.

The text representation also combines global sentence context with local noun-phrase context, with γ controlling their relative contribution, to produce EM-ConText. The intention is to retain both the sentence-level relation and the local identity of the referred object, rather than merely concatenate multilingual final vectors. The main paper identifies the centroid, EOT replacement, and contextual fusion roles, but delegates noun-context construction and complete fusion implementation details to the appendix. Those details cannot be turned into a verified runnable implementation from the available source alone. At inference, each language is processed only up to the selected layer, followed by a single continuation pass; the resulting query is computed once and reused for patch matching and candidate scoring. This sharing limits overhead without making multilingual inference free.

3. Spatial Reranking: preserve global semantics while distinguishing similar candidates

P-Map is normalized to the range 0–1 and thresholded with δ, after which connected components are extracted on the patch grid. These components are upsampled to image resolution and merged into a binary guidance mask G. This converts dense responses into spatial support that can be compared with the supplied candidate masks, rather than treating the highest-response patch as the final answer. The candidate set remains a hard constraint: a useful P-Map cannot create a target mask omitted by the proposal generator.

The baseline score S is the cosine similarity between a candidate region's final visual embedding and the query embedding. B2G additionally computes the candidate's intersection over union with G and adds the weighted overlap to the baseline score. Only the top κ candidates under the baseline score undergo reranking, limiting interference from poor proposals. The additive relationship in the paper's Equation (2) is recoverable:

\[ S'_k = S_k + \lambda\,\operatorname{IoU}(M_k,G). \]

Here λ controls spatial guidance strength, and the highest reranked score determines the selected mask. The semantic term asks whether a region matches the described object; the overlap term asks whether it covers the location indicated by the intermediate features. The paper also mentions replacing overlap with mean P-Map response inside a candidate, but its detailed comparison is in the unavailable appendix and is not assigned invented results here.

Loss & Training¶

B2G introduces no training objective, performs no gradient updates, and does not require Grad-CAM-style backpropagation or attention recomputation. The fixed visual exit layer and initial threshold are determined on an unlabeled calibration subset comprising 10% of RefCOCOg val (U), then held fixed at test time. Thus, zero-shot means no RIS-specific parameter training, not absence of calibration or prior training of the encoder and segmentor. The main text does not fully specify all λ, γ, and δ settings or calibration criteria, limiting exact reproduction rather than preventing an explanation of the verified mechanism. Original-image intermediate features can be read during a forward pass, but permutation probing still processes a shuffled image, and region encoding and multilingual partial passes still incur computation.

Key Experimental Results¶

Main Results¶

The table selects verifiable comparisons from Table 1 and Section 3.4 instead of pooling unlike backbones into a single ranking. The English averages cover nine splits: RefCOCOg val/test/valG, RefCOCO val/testA/testB, and RefCOCO+ val/testA/testB. Both English comparisons use Mask2Former, with CT and SG applied to the compared methods. The multilingual figures instead aggregate SigLIP2 results over nine translated benchmarks and should not be directly compared with the English backbone results. All metrics are higher-is-better, and gains are percentage points, not relative percentages.

Setting and source	Metric ↑	Control	B2G	Gain
English nine splits, CLIP ViT-B/16, Global-Local → B2Gfix, Table 1	Average mIoU	45.74	50.20	+4.46
English nine splits, DFN ViT-H/14, Global-Local → B2Gfix, Table 1	Average mIoU	44.64	51.13	+6.49
Multilingual nine benchmarks, SigLIP2 → B2G, Section 3.4	Average mIoU	42.42	49.62	+7.20
Multilingual nine benchmarks, SigLIP2 → B2G, Section 3.4	Average IoU@50	45.13	53.41	+8.28

mIoU averages intersection over union across samples; IoU@50 is the proportion whose prediction–ground-truth IoU exceeds 0.5. A closer text-fusion control, final-layer centroid aggregation, reaches 46.07 mIoU and 48.67 IoU@50, below B2G. However, full B2G also includes visual processing, so the entire difference should not be attributed exclusively to EOT injection. Section 3.4 reports approximately 1.5–1.7× text-side latency, precluding a zero-cost interpretation of the accuracy gain.

Ablation Study¶

The following selection from Table 3b evaluates zero-shot RIS on RefCOCOg val (U), varying candidate count under the original and B2G scoring approaches. The controls are Global-Local+CT+SG; oIoU aggregates intersections and unions across samples before division, unlike sample-averaged mIoU, and both metrics are higher-is-better. The subtable does not separately identify a backbone, so these results retain its original comparison scope rather than being joined to a backbone-specific row from Table 1.

Candidate count	Control mIoU ↑	B2G mIoU ↑	Control oIoU ↑	B2G oIoU ↑
2	50.47	52.08	40.28	40.72
3	50.79	52.24	41.29	41.43
4	49.42	51.92	40.81	41.40

B2G improves both metrics at each fixed candidate count; three candidates give its highest mIoU in this comparison, not a universal optimum across backbones and datasets. This examines candidate selection and spatial scoring, rather than independently ablating multilingual centroid injection.

Key Findings¶

Gains depend on representations and proposal quality: the improvements differ across backbones, and the authors report better synergy with Mask2Former than with SAM.
Mid-depth concentration in automatic selection supports an interface-bottleneck interpretation, but does not isolate a single alignment objective as the cause of every layerwise change.
Multilingual intervention is also evaluated for zero-shot text-to-image retrieval, extending the diagnostic question beyond segmentation; however, some Table 2 aggregate averages disagree with their listed language entries, so those retrieval-average gains are not reproduced here.

Highlights & Insights¶

Probing is followed by an intervention: spatially sensitive and language-consistent features are tested through downstream localization rather than supported only by embedding visualizations.
Global and dense matching need not replace one another: reranking lets semantic recognition and local spatial evidence serve complementary roles within an existing pipeline.
The fusion location matters: replacing EOT at an intermediate layer and continuing computation intervenes in representation formation rather than simply ensembling final outputs.

Limitations & Future Work¶

The authors acknowledge backbone-dependent optimal layers and P-Map sharpness, together with additional multilingual inference latency; a fixed layer need not work equally well on every new encoder.
The paper discusses semantic drift in translation-derived benchmarks; centroid effectiveness depends on equivalent inputs, and benchmark gains do not establish fairness across all real multilingual settings.
Reader assessment: tile permutation measures positional sensitivity, not a complete causal account of compositional spatial reasoning; the authors also leave deeper theory of layerwise geometry for future work.
Reader assessment: proposal quality, calibration data, and access to additional language versions are deployment conditions that should be reported separately from the absence of parameter updates.
Evidence boundary: the cached official source includes the complete main Method and Experiments, but no separate supplement; detailed CT construction, complete dynamic-selection definitions, translation-system and donor-language ablations remain unverified, and damaged equations have not been reconstructed.

vs Global-Local / HybridGL: These improve global–local candidate representations; B2G preserves those interfaces while adding mid-layer diagnosis and spatial reranking, making it an additive enhancement rather than a replacement segmentor.
vs Grad-CAM-based approaches: They derive guidance through gradients, whereas B2G reads a spatial map from forward intermediate features; end-to-end speed must still account for region encoding, probing, and multilingual overhead.
vs ClearCLIP / ResCLIP: Intermediate representations for dense prediction are not new by themselves; B2G's more distinctive contribution combines visual positional probing and textual language-consistency probing within one frozen-model intervention framework.
vs multilingual pretraining and final-layer centroid aggregation: B2G does not train another encoder, instead injecting an averaged cross-language state inside the existing one, shifting costs toward multilingual inputs and test-time computation.

Rating¶

Novelty: 4/5 — Combining two-modality probing and intervention is valuable, although intermediate-layer localization and contextual fusion have precedents.
Experimental Thoroughness: 4/5 — Multiple backbones, three evaluation protocols, and candidate analysis provide breadth; some ablations require the missing appendix and certain aggregates need checking.
Writing Quality: 3/5 — The diagnostic narrative is clear, but contextual fusion and selection details depend on the appendix, and extraction-corrupted equations impede implementation checks.
Value: 4/5 — Useful for studying frozen VLE readouts and zero-shot grounding, subject to proposal quality and inference-budget constraints.