From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

Conference: ECCV 2026
Paper: ECCV
Area: Multimodal VLM
Keywords: multimodal large language model, semantic segmentation, mechanistic interpretability, linear probing, attention knockout

TL;DR

Without adding any new module to the MLLM, this paper dissects its segmentation competence through three interventional experiments: layerwise linear probing reveals a representation drop-off at the adapter when visual features are projected into the language space, followed by progressive recovery of mIoU across LLM layers; attention knockout shows that this recovery is not a passive by-product of residual connections but is actively mediated by cross-token attention, in which correctly classified tokens act as semantic anchors pulling misclassified neighbours toward the correct label; finally, causal attention is shown to starve the earliest image tokens of global context, a cost that bidirectional attention among image tokens eliminates.

Background & Motivation

Semantic segmentation underpins robotics, AR/VR, autonomous driving, and medical imaging. On the pure-vision side, ViTs pretrained with contrastive or self-supervised objectives have long been strong feature extractors — their patch-level embeddings already carry rich dense semantics, and attaching a linear probe or a lightweight decoder transfers them to segmentation; the SAM family pushed the ceiling of dedicated encoders further still. In parallel, multimodal large language models (MLLMs) excel at multimodal reasoning, instruction following and language-conditioned control, and among them adapter-style architectures are especially popular for their simplicity: a pretrained vision encoder, a trained adapter network, and a pretrained LLM, with the adapter mapping visual embeddings into the LLM's token space. This has motivated a wave of work that adapts MLLMs into segmentation models, attaching SAM or introducing special [SEG] tokens to bridge language reasoning with mask prediction, arguing that language supervision improves interpretability and enables referring, instruction-driven segmentation that classical segmenters cannot offer.

What these works report, however, is system-level performance — how much a fine-tuned model with SAM and special tokens scores — and nobody has answered the more basic question: treating the MLLM as a whole and under a controlled probing protocol, is it actually better at segmentation than the vision encoder underneath it? Existing diagnostic analyses suggest a discouraging answer: without task-specific fine-tuning, MLLMs can regress on classical vision tasks and even ignore the visual evidence already encoded in their own vision backbone, falling behind that backbone on vision-centric tasks such as depth estimation and correspondence, which made "the LLM is the bottleneck" a popular verdict. Yet the opposite evidence exists too: generative MLLMs have been shown to extract more visual information than CLIP from the same frozen encoder, and intermediate MLLM layers have been shown to hold richer region-level descriptions than the final layer. These analyses either probe representations without performing causal interventions, or focus on general perceptual tasks and classification rather than semantic segmentation — and segmentation is precisely the task that most demands per-patch spatial fidelity and therefore most sharply exposes representational degradation.

This paper's angle is to stop asking vaguely whether MLLMs can segment, and instead split the question into three separately intervenable, individually falsifiable sub-questions. At which step of the pipeline is segmentation competence destroyed? Is the subsequent climb a passive smoothing by residual connections and normalization, or is attention actively moving information? If there is a structural positional defect, how much can a change to the attention mask recover, and at what cost? Core idea: use an intervention chain of "layerwise linear probing to localize the phenomenon → attention knockout to verify the mechanism → bidirectional attention among image tokens to remove the structural bottleneck" to decompose MLLM segmentation competence into three separately measurable facts — a representation drop-off at the adapter, cross-token self-refinement driven by semantic anchors, and context starvation of early tokens caused by causal masking.

Method

Overall Architecture

The input is an image plus a text prompt, and the output is not a segmentation mask but a mechanistic account of which layer of the MLLM stack segmentation competence lives in and what mechanism sustains it. The entire analysis rests on one dissected object: a standard adapter-style MLLM, in which the vision encoder splits the image into \(T\) non-overlapping patches and encodes them into a token sequence, the adapter (a two-layer MLP) projects them into the LLM's \(d\)-dimensional input space, and image tokens are processed jointly with system tokens and text prompt tokens. The paper adds no structure to this pipeline; it only places a scalpel at three positions, and every scalpel measures with the same ruler — a linear probe trained on image patch token hidden states, whose per-token predictions are reassembled into a 2D segmentation map and scored by mIoU and pixel accuracy. The conclusions of the three experiments are therefore layer-aligned and mutually corroborating.

The first scalpel is layerwise linear probing, which answers "at which layer is the signal lost, and at which layer is it recovered"; the curve it produces is the starting point of the whole analysis and raises the next question — is the climb passive smoothing or active transport? The second scalpel is attention knockout, designed to rule out the "residual connections + LayerNorm" passive explanation and attribute recovery to cross-token attention; it further exposes that recovery depends on correctly classified anchors, a condition that does not hold for the earliest image tokens in the sequence. The third scalpel releases bidirectional attention among image tokens only, directly testing and removing this positional bottleneck created by causal masking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["image + text prompt<br/>frozen adapter-style MLLM"] --> B["layerwise linear probing<br/>localize drop-off and recovery"]
    B --> C["attention knockout<br/>test cross-token attention as the active mechanism"]
    C --> D["image-only bidirectional attention<br/>remove context starvation at early tokens"]
    D --> E["mechanistic conclusions + architectural guidance"]

Key Designs

1. Layerwise linear probing: a CT scan of the segmentation signal across the whole pipeline

Prior discussion of "can MLLMs segment" stops at the final output or at system-level metrics, leaving it unknown whether, inside the pipeline, the spatial information carried by the encoder is preserved, lost, or enhanced. The paper's approach is to reduce the pipeline to three comparable observation points: the vision encoder output, the adapter output, and every intermediate LLM layer. For a target layer \(\omega\), the whole MLLM is frozen and only the image-token slice of the hidden states, \(X^{(\omega)}\), is extracted; an independent linear probe is then trained per layer on the training set with cross-entropy so that every token is classified independently, and the per-token predictions are reshaped into a 2D grid, upsampled to the original resolution, and scored as pixel-level mIoU. The crucial point is "one procedure, one protocol": the encoder output, the adapter output and every LLM layer all use exactly the same probe training procedure, differing only in which hidden state is extracted and what its dimensionality is. Because of this, differences in mIoU across layers can only be attributed to the representations themselves rather than to probe capacity or training conditions — this is what makes the layerwise profile valid. Concretely, a 336×336 input with patch size 14 yields \(T = 576\) image tokens; since each token corresponds to a fixed spatial position, per-token predictions can be rebuilt into a 2D segmentation map.

The dissected models are three LLaVA-1.5 variants, all using Vicuna-7B as the LLM and differing only in the vision encoder: CLIP ViT-L/14@336, DINOv2 Large@336, and SigLIP SO400M/14. All three follow the standard two-stage recipe (pretrain the adapter on 558K image-caption pairs, then finetune the full model on 665K visual instruction data), so the comparison is not contaminated by differing training recipes. Evaluation uses three standard segmentation benchmarks: ADE20K (150 classes, mixed indoor and outdoor scenes), PASCAL VOC 2012 (20 foreground classes, augmented training set), and Cityscapes (urban street scenes), with mIoU as the primary metric and pixel accuracy as a secondary one. The differing degree of text alignment across the three encoders is deliberately retained as a variable — CLIP and SigLIP align visual features to text via contrastive objectives, DINOv2 is vision-only — and it later explains the differing strength of recovery.

2. Attention knockout: separating active recovery from passive bypass

Layerwise probing produces a curve, and a rising curve has two very different explanations: a passive one, in which residual connections and LayerNorm carry information smoothly across layers so that mIoU naturally recovers, and an active one, in which self-attention genuinely transports and integrates information across tokens. Distinguishing them requires an intervention. The paper borrows the attention knockout technique that Geva et al. introduced for tracing information flow in autoregressive language models, but applies it to a different question: earlier work used knockout to find which tokens influence the generated text, whereas this paper asks whether cross-token attention among image patch tokens themselves drives the self-refinement of spatial representations.

Concretely: given a test image, probes first produce a predicted label for every image patch; a target class \(c\) is chosen and its token set \(B_c\) (all image tokens predicted as \(c\)) is collected; then, at every LLM layer, attention from any image token to any token in \(B_c\) is set to negative infinity before the softmax:

\[A^{(\omega)}_{t \to s} = -\infty, \quad t \in I,\ s \in B_c\]

This makes class \(c\) completely invisible: no image token, including tokens of class \(c\) itself, can attend to them any more. Blocked tokens can still attend to all non-blocked tokens, so their representations keep evolving but without self-reinforcement from same-class neighbours. The same per-layer linear probes are then rerun to obtain a layerwise segmentation profile under intervention. The experiments select ADE20K images where the unmodified model shows characteristic class confusions (for example, ceiling patches predicted as sky) and run two complementary conditions on them — block the incorrect class, and block the correct class — against the unmodified baseline. The design's strength is that the two conditions are mirror images of each other: if misclassified tokens are actively reinforcing an error through attention, silencing them should make self-correction faster; if correctly classified tokens are semantic anchors, silencing them should make self-correction fail. The two directions predict opposite outcomes, so the conclusion is hard to explain away as "any intervention perturbs the model".

3. Bidirectional attention among image tokens: removing context starvation at early positions

The third scalpel targets a structural defect inherent to standard causal attention. Image tokens are laid out in raster order (left to right, top to bottom), and each token can only attend to tokens preceding it in the sequence: the first image token sees no other image token at all, while the last sees all \(T\) of them. For segmentation this is unreasonable — ideally every patch should access scene-level context, and early tokens cannot. The qualitative visualisations of layerwise probing confirm it directly: tokens in the first row, and especially the top-left corner, are frequently misclassified, and their accuracy does not improve as LLM layers deepen, indicating a persistent positional penalty rather than something later layers gradually repair.

The modification releases attention only among image tokens, leaving every other token pair under the causal mask — bidirectional inside the image, causal everywhere else:

\[M(q,k)=\begin{cases}\text{bidirectional} & q \in I \ \text{and}\ k \in I\\ \text{causal} & \text{otherwise}\end{cases}\]

(The typesetting of Eq. 2 is corrupted in the cached text; this is a rewrite based on the prose description, ⚠️ refer to the original paper.) This choice deliberately differs from PaliGemma's prefix-LM strategy: the latter makes all input tokens bidirectional, covering image, system prompt and task prefix alike, whereas this variant targets spatial self-refinement among image tokens specifically and fully preserves the sequential structure required for autoregressive text generation. Training follows the LLaVA-1.5 two-stage recipe (adapter pretraining on 558K image-caption pairs, full-model finetuning on 665K visual instruction data) with the same attention type in both stages, so the comparison reflects the cumulative effect of attention directionality across the entire training process; apart from the mask, all hyper-parameters, training data and the three vision encoders (CLIP, DINOv2, SigLIP) are identical, leaving the mask as the only variable.

A Worked Example

Take one indoor ADE20K validation image in which the ground truth has a large region of ceiling in the upper left, and the unmodified model predicts part of it as sky — the characteristic ceiling/sky confusion that the paper uses as a running case.

Step one, layerwise probing. For this image, the hidden states of all 576 image tokens are extracted at the encoder output, the adapter output and every LLM layer, classified by the corresponding layer's probe, and stitched back into a 2D map. The observation is: predictions at the adapter output have noisy boundaries and visible class confusion; after entering the LLM the confusion in the affected region starts to shrink in the early layers; by the mid-to-late layers it stabilises, though a few patches remain on sky.

Step two, run both blocking conditions on the same image. After blocking the incorrect class (setting attention from all image tokens to tokens predicted as sky to \(-\infty\)), the affected region flips back to ceiling already in the early LLM layers, and by the final layer the residual misclassifications are fewer than in the unmodified model — with the tokens that produced the wrong attractor silenced, the contextual cues from correctly classified neighbours (wall, floor) dominate the attention field. Blocking the correct class (silencing ceiling tokens) is markedly worse: patches misjudged as sky persist into the mid-to-late layers, the regional segmentation quality there drops below the unmodified model, and more residual errors remain. The asymmetry between the two directions pins down that cross-token attention is actively correcting errors.

Step three, look at patch 0 (top-left) of the same image. Under causal attention it has no visual context to attend to, and its prediction barely changes with depth and disagrees with its neighbours; under bidirectional attention among image tokens it sees all 576 image tokens in a single forward pass, and its pixel accuracy rises by 14.35 percentage points, a 23.2% relative improvement. Chained together, the three results give the mechanistic picture: correctly classified ceiling tokens are semantic anchors, and causal masking is exactly what prevents the front-most tokens in the sequence from reaching them — the recovery mechanism exists, but is locally blocked by the positional structure.

Key Experimental Results

Main Results

Layerwise linear probing is the paper's core result, and its main conclusions come from the inter-layer curves on ADE20K for the three encoders (Fig. 3 of the original). The original reports these as curves and gives only directional conclusions plus a few concrete values in the text; the table below marks what the original does not report rather than filling in numbers.

Vision encoder (LLM: Vicuna-7B)	Encoder output → adapter output	Strength of LLM-layer recovery	Peak vs. encoder baseline
CLIP ViT-L/14@336	modest decline	strongest	ultimately exceeds the vision encoder baseline
SigLIP SO400M/14	larger degradation	meaningful	whether it exceeds the baseline is not reported
DINOv2 Large@336	larger degradation	weakest	not reported; the original only states recovery is clearly weaker than the other two

The comparison between causal and image-only bidirectional attention is likewise a core segmentation-side result (Fig. 8), with some values given in the text:

Vision encoder	Peak mIoU, causal	Peak mIoU, bidirectional	Peak gain
CLIP ViT-L/14@336	not reported	not reported	+0.42
DINOv2 Large@336	not reported	not reported	+0.32
SigLIP SO400M/14	35.39 (layer 12, then declining with depth)	41.26 (layer 32, monotonic improvement)	+5.87 (subtraction of the two values)

The main risk of bidirectional attention is damaging language ability, so the original runs a controlled comparison on nine VQA benchmarks (Tab. 1; the LLM is Vicuna-7B in all cases):

Vision encoder	Attention	GQA	MMB	MMEP	MMEC	MMMU	POPE	SQAI	TextVQA	VizWiz
CLIP ViT-L/14	causal	62.6	66.4	1483	284	35.3	86.8	68.7	46.9	56.0
CLIP ViT-L/14	bidirectional	62.7	65.5	1538	288	36.2	86.9	69.7	47.0	57.5
DINOv2 ViT-L/14	causal	62.1	57.7	1304	324	34.6	87.2	66.1	14.0	51.4
DINOv2 ViT-L/14	bidirectional	60.5	55.3	1247	326	32.4	85.1	66.3	13.7	45.8
SigLIP SO400M/14	causal	61.1	63.7	1414	275	34.7	84.4	70.4	50.2	58.2
SigLIP SO400M/14	bidirectional	62.1	66.9	1402	298	34.9	84.8	70.7	53.6	53.3

Note: MMEP and MMEC are scores on MMEP-Bench and MMEC-Bench reported as-is by the original, on a different scale from the percentage metrics and therefore not directly comparable across columns (⚠️ refer to the original paper). DINOv2's TextVQA is only 13.7–14.0 under both settings, far below the other encoders, and the original does not explain this separately.

Ablation Study

The paper's "ablation" intervenes on attention pathways and its output is a qualitative layerwise segmentation comparison (Fig. 6); no corresponding numbers are given, so the table records directional conclusions only:

Condition	Intervention	Layerwise behaviour	Relative to the unmodified baseline
unmodified baseline	none	class confusion in early layers, progressively corrected in mid-to-late layers	—
block incorrect class (e.g. sky)	attention from all image tokens → tokens predicted as sky set to \(-\infty\)	confusion in the affected region drops already in early layers	self-correction accelerates; fewer residual misclassifications at the final layer than the baseline
block correct class (e.g. ceiling)	attention from all image tokens → tokens predicted as ceiling set to \(-\infty\)	misclassified tokens persist longer across layers	segmentation quality in mid-to-late layers falls below the baseline, with more residual errors

The positional analysis provides concrete numbers: pixel accuracy is tallied for the first 50 patches one by one (Fig. 7), and the gap between causal and bidirectional attention is heavily concentrated at the very front of the sequence — patch 0, which under causal attention sees no visual context at all, gains 14.35 percentage points under bidirectional attention (a 23.2% relative improvement), and the gap decays rapidly with patch index.

Key Findings

• The bottleneck is the adapter, not the encoder. All three encoders lose mIoU at the adapter, showing that projecting visual features into the LLM's embedding space trades fine-grained spatial fidelity for cross-modal alignment; the magnitude varies, with CLIP declining only modestly while DINOv2 and SigLIP degrade more.
• LLM layers do restore segmentation quality, and how they do so depends on whether the encoder is text-aligned. The recovery curve rises sharply in the early layers and then plateaus in the mid-to-late layers; text-aligned CLIP recovers most strongly and its LLM-layer representations ultimately exceed the vision encoder baseline, the similarly contrastive SigLIP shows meaningful recovery, and vision-only DINOv2 recovers least. This points to an actionable judgement: the degree of compatibility between the encoder's representation space and the LLM's embedding space governs how much the LLM layers can refine visual features.
• Geometric evidence corroborates the mIoU curve. UMAP projections of the 576 patch hidden states of a single image show that at the adapter output patches of different semantic classes are heavily interleaved with no category-level organisation; as depth increases, same-class patches progressively cluster, and by layer 20 floor, ceiling, wall and building occupy clearly separated regions, with wall and building close together because of semantic similarity. The LLM does not merely make features more linearly separable — it reorganises them into semantically coherent clusters.
• Recovery is actively performed by cross-token attention, not a passive by-product. Blocking the incorrect class accelerates self-correction and blocking the correct class impairs it; both directions hold, showing that correctly classified tokens act as semantic anchors whose attention signals pull misclassified neighbours toward the right label. This also explains why such segmentation errors tend to appear in patches and pairs (the ceiling↔sky kind of confusion), and why recovery fails in regions where no correct anchor exists at all.
• Recovery has clear boundaries. LLM layers provide structure- and constraint-aware refinement: they improve local consistency and resolve some class conflicts, but they cannot recover fine-grained spatial details absent from the encoder features, nor overcome systematic encoder biases where an entire region is misjudged and no correct anchor can be found.
• The cost of causal masking is sharply localised. The size of patch 0's gain (+14.35 percentage points) together with the rapid decay of the gap shows that once a token can reach even a handful of visual neighbours, partial context suffices for self-refinement; only the few positions at the very front of the sequence are genuinely blocked by causal masking.
• For SigLIP, causal masking is not just a local effect. Bidirectional gains for CLIP and DINOv2 are only a few tenths of a point of mIoU, consistent with the positional analysis; SigLIP is different: under causal attention recovery peaks at layer 12 with 35.39 and then declines with depth, whereas bidirectional attention improves monotonically to 41.26 at layer 32. For some encoders causal masking prevents deeper layers from continuing to benefit, rather than merely handicapping the first few patches.
• The language cost of image-only bidirectional attention is controllable, but encoder-dependent. CLIP and SigLIP are broadly on par with their causal baselines across the nine VQA benchmarks (individual metrics such as MMB and VizWiz move both ways), while DINOv2 shows broader regressions (MMB 57.7→55.3, VizWiz 51.4→45.8, MMMU 34.6→32.4), consistent with the weak text alignment it exhibits throughout the paper.
• The mechanism holds beyond LLaVA. Besides LLaVA, the paper repeats layerwise probing and attention knockout on OneVision (SigLIP SO400M + Qwen2-7B) and DeepSeek-VL (a hybrid SigLIP-L and SAM-B vision encoder + DeepSeek-LLM 7B); the supplementary shows that both reproduce the drop-off/recovery profile and the semantic-anchor effect — blocking the incorrect class accelerates self-correction while blocking the correct class impairs it.

Highlights & Insights

• Rewriting "is the MLLM better than its own encoder" into three falsifiable causal questions is the most transferable move in this paper: probing localises the phenomenon, knockout rules out the passive explanation, and mask ablation localises the structural bottleneck. Each step sets up a pair of opposite predictions, so the conclusion cannot be dismissed as an artefact of the intervention itself.
• The notion of a semantic anchor connects the phenomenon to the mechanism. Correctly classified tokens are not a passive outcome but an information source that actively pulls misclassified neighbours back; this viewpoint simultaneously explains why segmentation errors appear in connected patches and why regions without correct anchors cannot self-repair — far more concrete than simply saying "attention has global context".
• The uniform probing protocol is what makes the layerwise analysis valid. The encoder output, adapter output and every LLM layer use the same linear probe under the same training conditions, with the extracted hidden state as the only variable, so inter-layer differences can only come from the representations themselves. This "same-protocol probe across layers" recipe transfers directly to any diagnostic study asking at which layer of a pipeline some capability appears or disappears.
• Image-only bidirectional attention is a nearly free change that independently converges on an existing architectural choice. STAMP, starting from system design, makes mask tokens bidirectional while keeping text causal; this paper, starting from internal probes and mask ablation, reaches the same architectural conclusion without building a system. Two independent routes to one design is itself strong evidence.
• The three-step analysis transfers to other vision-centric tasks. For depth estimation, correspondence or counting, one can similarly use layerwise probing to locate the bottleneck layer, knockout to test whether recovery depends on cross-token attention, and then decide whether to change the adapter or the attention mask — a far more targeted route than simply fine-tuning a larger model.
• The negative conclusion about the boundary of recovery is equally valuable. It redirects effort on MLLM segmentation away from "more LLM layers / more data" back toward the spatial fidelity of encoder features, because LLM layers cannot restore details that the encoder never had.

Limitations & Future Work

• One limitation admitted by the authors: the experiments start from LLaVA-type models, and although three vision encoders were swapped and OneVision and DeepSeek-VL were added in the supplementary, the conclusions may not generalise to architectures that differ substantially (for instance non-adapter designs, or architectures where visual and text tokens are heavily interleaved).
• A second admitted limitation: linear probes may underestimate true segmentation capacity, since a richer task head might decode spatial information that still exists in the adapter output but is not linearly separable. The absolute magnitude of the adapter drop-off should therefore be read as a decline in linear separability, not as total information loss — the paper states this carefully, but readers citing the result should keep this qualification in mind.
• A third admitted limitation: attention knockout is applied globally across all layers, so the contribution of individual layers cannot be isolated. Moreover, since blocking hides the target class while the blocked tokens keep evolving, indirect influence mediated by non-blocked tokens cannot be excluded, so the "correct-class tokens are anchors" conclusion is not yet fully isolated at the level of mechanistic detail.
• A limitation I noticed: the paper lacks one key control — "vision encoder features plus a decoder of comparable capacity" evaluated under the same protocol. Without that external baseline, a reader learns only about relative changes inside the MLLM, not whether the MLLM as a whole beats a pure-vision alternative. Probe training details (iterations, regularisation, early stopping) are also relegated to the supplementary, and such details directly affect the fairness of cross-layer comparisons.
• The language-cost conclusion is bounded too: bidirectional attention is released only among image tokens, and if text prompt tokens were also made bidirectional (a prefix-LM-style design), the additional segmentation benefit and the damage to language ability remain unknown; SigLIP's VizWiz dropping from 58.2 to 53.3 also shows that "no language loss" does not hold for every metric.
• Concrete improvement directions: refine the global knockout into per-layer and per-head knockout to locate the heads that genuinely carry the semantic-anchor effect; replace manual image selection with automatic mining of characteristic class pairs, turning the knockout experiment from a case study into a statistical analysis; and add the encoder + comparable-capacity decoder baseline so that the question "is the MLLM worth it" gets a clean answer.

Related Work & Insights

• vs LISA / GSVA / PixelLM / GLaMM / SAM4MLLM / OMG-LLaVA and other MLLM segmentation methods: They propose architectural variants and report system-level segmentation performance, asking "how do we make it stronger" — LISA connects an MLLM to SAM via a [SEG] token, GSVA extends this with multiple [SEG]/[REJ] tokens for multi-target and empty-target cases, and PixelLM takes a SAM-free route with multi-scale segment tokens and a lightweight decoder. This paper trains no segmentation model and answers only "where does segmentation competence live in this stack and what mechanism restores it"; it is diagnostic rather than constructive, complementary rather than competing.
• vs STAMP: STAMP starts from system design and performs non-autoregressive simultaneous textual mask prediction, granting bidirectional attention among mask tokens while keeping text causal — an empirical architectural choice. Section 5 of this paper reaches the same architectural conclusion independently from layerwise probing and mask ablation. The two corroborate each other, but this paper offers no usable segmentation system.
• vs Fu et al. (VLMs overlook their visual representations) / Zhang et al. (classification) / Tong et al. (BLINK): They show that MLLMs regress on vision-centric tasks and identify the LLM as the primary bottleneck; this paper focuses on semantic segmentation, supplies the causal intervention they lack, and paints a more nuanced picture — the LLM is not only a bottleneck but also the source of recovery, and the strength of that recovery depends on how well the encoder's representation space aligns with the LLM's embedding space, with vision-only encoders benefiting markedly less.
• vs Li et al. (generative MLLMs extract more than CLIP from the same frozen encoder) / Liang et al. (intermediate layers hold richer region descriptions): They provide the observation that richer information lives in LLM layers; this paper quantifies that observation into a concrete trajectory using the mIoU profile from layerwise probing and UMAP projections of patch hidden states — classes interleaved at the adapter, progressively re-clustering into semantic groups within a few tens of layers — and further shows that this re-clustering is attention-driven and locally limited by causal masking.

Rating

• Novelty: ⭐⭐⭐⭐ The first work to combine layerwise linear probing, attention knockout and image-only bidirectional attention mask ablation into a single intervention chain that systematically answers where MLLM segmentation competence comes from and where its limits are, turning "active recovery vs. passive bypass" into a testable causal question; each individual technique is pre-existing, and the novelty lies in the decomposition and combination.
• Experimental Thoroughness: ⭐⭐⭐ Three encoders × three segmentation benchmarks plus two additional architectures (OneVision / DeepSeek-VL, in the supplementary) and a language-cost check on nine VQA benchmarks is solid coverage; however, the key knockout and layerwise probing results are mostly presented as curves with no per-layer mIoU values in the main text, and the comparison against an "encoder + comparable-capacity decoder" external baseline is missing.
• Writing Quality: ⭐⭐⭐⭐ The three experiments build on each other, each step has an explicit hypothesis, and the conclusions carry an awareness of their boundaries (it states plainly that details absent from the encoder cannot be recovered); the narrative is clear, though a few equations are slightly rough in their typesetting.
• Value: ⭐⭐⭐⭐ Directly actionable for designing segmentation-capable MLLMs: the main bottlenecks are the adapter and causal masking rather than the vision encoder itself, and image-only bidirectional attention is a low-cost change that largely preserves language ability (with gains concentrated at early tokens, most visibly for SigLIP).