Exploring Efficient Reasoning Segmentation with Small Language Models¶

Conference: ECCV2026
Authors: Changsong Wen, Zelin Peng, Yu Huang, Xiaokang Yang, Wei Shen
Paper: https://eccv.ecva.net/virtual/2026/poster/4052
PDF: https://media.eventhosts.cc/Conferences/ECCV2026/pdfs/4295.pdf
Code: https://github.com/downdric/LReSeg
Area: Multimodal Reasoning
Keywords: reasoning segmentation, small language models, register tokens, unified visual encoding, text-conditioned spatial features

TL;DR¶

Rather than relying entirely on a small model's single <SEG> token, LReSeg supplies dense spatial information through a unified visual encoder and text-guided register tokens, achieving 50.8 gIoU and 50.0 cIoU on the ReasonSeg test set with 805M parameters while reducing LISA-7B's latency from 320 ms to 98 ms under the paper's timing setup.

Background & Motivation¶

Reasoning segmentation requires identifying a target from an implicit description and then producing its pixel-level mask. Unlike a direct reference such as “segment the person on the left,” a query may describe a function, a state, or a commonsense association, so the model must first determine which object it denotes. LISA connects this task to a multimodal language model by using the response's <SEG> token as a segmentation prompt for SAM decoding. Large language models provide strong semantic reasoning, but they are often accompanied by two visual encoders, making the overall memory and computation demands substantial.

Simply replacing the language backbone with a small language model does not guarantee equally reliable masks. <SEG> remains a semantic representation in a language sequence, and a smaller model has more difficulty compressing fine spatial information into this single interface. Reusing the language model's visual tokens seems attractive, but these tokens precede the text and cannot read subsequent instructions under causal attention. Their spatial representations therefore do not know what the user is asking about. The problem is not just reduced parameter count: the existing information pathway does not adequately connect target semantics with spatial localization.

The paper addresses this pathway rather than applying quantization, knowledge distillation, or pruning. It appends a small set of learnable register tokens that can read both the image and the text, then explicitly injects the visual information most attended to by the text. On the visual side, reasoning and segmentation share DINOv3, avoiding a separate heavyweight SAM image encoder for pixel prediction. Core Idea: supplement the small model's segmentation interface with a few text-conditioned spatial carriers, separating semantic target prompting from dense spatial guidance instead of requiring a single <SEG> token to perform both roles.

Method¶

Overall Architecture¶

The input is an image and a textual instruction that implicitly describes a target; the output is a binary target mask. LReSeg first produces shared image features through Unified Visual Encoding, then builds text-conditioned spatial representations inside Qwen2.5-0.5B through Selective Register Enhancement, and finally combines <SEG> target semantics with register-derived spatial guidance through Dual-Path Decoding.

High-resolution features remain available to the segmentation side, while pooled and projected visual tokens enter the language model. The language model therefore does not need to process every spatial position in the original image. The encoder-to-decoder branch in the diagram preserves detail, whereas the language-model branch identifies which details matter for the current query.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image and implicit instruction"] --> B["Unified Visual Encoding"]
    B --> C["Selective Register Enhancement"]
    C --> D["Dual-Path Decoding"]
    B --> D
    D --> E["Binary target mask"]

Key Designs¶

1. Unified Visual Encoding: share image representations between reasoning and pixel prediction

Conventional dual-encoder systems often use CLIP or SigLIP for language reasoning and a SAM image encoder for mask decoding. This means more than storing an extra set of weights: the same high-resolution image undergoes two visual forward passes, and information selected on the language side may not align naturally with the segmentation-side representation. LReSeg uses DINOv3 as its unified visual encoder because it preserves fine spatial structures while supporting multimodal understanding. The removed component is the separate SAM image encoder, not the mask decoder.

DINOv3 features follow two routes: one preserves spatial structure for mask decoding, while the other applies adaptive average pooling and a two-layer MLP to map features into the small language model's embedding space. Multimodal pretraining uses 224 × 224 images and 144 language-side visual tokens; segmentation training switches to 1024 × 1024 images and increases the language-side visual token count to 256. These resolutions belong to different training stages, so 144 and 256 should not be treated as two input groups within the same forward pass. Sharing the encoder removes duplicate visual computation, whereas pooling reduces language-model sequence computation; the savings occur at different points.

2. Selective Register Enhancement: carry text-attended spatial information into subsequent language-model layers

LReSeg appends 16 randomly initialized, learnable register tokens to the input sequence. Earlier visual tokens cannot read later text under the causal mask, but the trailing registers can read both modalities and combine their information. These registers are neither an additional pixel grid nor final category labels; they are a small set of trainable spatial information carriers. Keeping the causal structure avoids wholesale changes to the small language model's established language-modeling behavior.

Letting registers learn solely through self-attention is insufficient. At layer 6, the method reads text-to-image attention, averages across attention heads and instruction-text positions, and obtains a relevance score for each visual token. It then selects the most attended top-k visual tokens. The following combines the paper's Eqs. (4)–(5) into equivalent, clearer notation:

\[ \alpha_j^{(l)}=\frac{1}{H N_t}\sum_{n=1}^{H}\sum_{i=1}^{N_t}A_n^{(l)}(i,j). \]

Here, \(A_n^{(l)}(i,j)\) is an attention weight in layer \(l\), head \(n\), from text position \(i\) to visual position \(j\); \(H\) is the number of heads and \(N_t\) is the number of text tokens. A higher score means that the current instruction attends more strongly to that image position. It is neither an additional ground-truth mask nor a confidence score for reasoning correctness. The analysis finds the best performance when the number of selected visual tokens reaches \(k=64\); increasing it further introduces redundancy and background information.

The selected visual hidden states are concatenated with the current register representations and passed through an MLP to update the registers. The updated registers then traverse subsequent language-model layers rather than immediately predicting a mask. Layer 6 is chosen because text attention has begun distinguishing the target region from the background, while enough layers remain to integrate spatial and semantic information. The main text describes concatenation followed by an MLP but does not fully specify tensor rearrangement between differing token counts. Reproduction therefore requires checking the implementation rather than assuming a one-to-one correspondence between registers and selected visual tokens.

3. Dual-Path Decoding: use semantic prompts for targets and dense features for spatial detail

The small register set cannot directly serve as a pixelwise representation of the entire image. After language-model processing, LReSeg uses visual tokens as queries and registers as keys and values in residual cross-attention. This writes instruction-aware register information back into visual tokens. Following the meaning of the paper's Eq. (7), the core transformation is:

\[ \mathbf{h}_d=\mathbf{h}_v+\operatorname{Softmax}\!\left(\frac{(\mathbf{h}_v\mathbf{W}_q)(\mathbf{h}_r\mathbf{W}_k)^\top}{\sqrt{d}}\right)(\mathbf{h}_r\mathbf{W}_v). \]

Here, \(\mathbf{h}_v\) denotes language-model-processed visual tokens, \(\mathbf{h}_r\) denotes output registers, \(d\) is the feature dimension used for attention scaling, and \(\mathbf{W}_q,\mathbf{W}_k,\mathbf{W}_v\) are learnable projections. The resulting \(\mathbf{h}_d\) inherits the visual layout while retrieving instruction-conditioned information from the registers. This write-back occurs on the decoding side, so earlier visual tokens inside the language model do not need to violate the causal mask to read subsequent text.

The <SEG> hidden state is then projected into a sparse semantic prompt that indicates the target. Enhanced visual tokens are restored to spatial features and upsampled into dense guidance, which enters SAM-style mask decoding together with the image features preserved by the unified encoder. These paths do not generate separate masks for voting; they provide target semantics and positional detail within the same mask prediction. The paper summarizes prediction through interaction between the <SEG> projection and enhanced image features. Its focus is supplementary segmentation information, not additional textual reasoning steps or reinforcement learning.

A Worked Example¶

Consider a house image with a query asking for the part of the house that can be opened. This example follows the task type in the paper's Fig. 1 and illustrates information flow without claiming measured attention values. Unified Visual Encoding preserves structures such as doors and windows. The 256 pooled visual tokens enter the small language model alongside the instruction, with 16 register tokens at the end of the sequence.

At layer 6, the model selects relevant visual positions using text-to-image attention and injects their information into the registers. Later layers further contextualize the registers, while <SEG> supplies target semantics. Dual-Path Decoding writes register content back to visual positions and combines it with high-resolution image features to produce the target mask. The pipeline does not hard-code which object “can be opened” must denote. If language understanding or attention-based selection is incorrect, the registers do not automatically guarantee the correct answer.

Loss & Training¶

The multimodal model first undergoes two training stages. The first trains only the projector for image–text alignment on a CC3M subset, using 1 epoch, batch size 256, and learning rate 5e-5. The second fine-tunes the entire multimodal model on LLaVA instruction data for 1 epoch with batch size 128 and learning rate 2e-5. The visual encoder and small language model must first learn to cooperate; the registers should not be interpreted as a training-free, plug-and-play addition to any existing small model.

Segmentation training mixes semantic segmentation, referring segmentation, ReasonSeg reasoning segmentation, and LLaVA visual question answering data. Supervision combines text-generation loss with binary cross-entropy and Dice mask losses, whose respective weights are 2.0 and 1.5. Training uses 8 RTX 3090 GPUs for 5 epochs with batch size 128. The authors specify fine-tuning the small language model and decoder for segmentation, with registers learned jointly. Some cached equations have damaged formatting. This note only clarifies attention mechanisms confirmed by the surrounding text and describes the loss combination verbally rather than filling in unverified implementation details.

Key Experimental Results¶

Main Results¶

The table combines parameter counts from the paper's Table 1 with overall ReasonSeg test results from Table 2. gIoU averages per-example IoU, whereas cIoU aggregates intersection and union areas before computing IoU; higher is better for both, and the two aggregation rules are not interchangeable. Lower latency, computation, and memory are better. LISA and READ are variants using the same Qwen2.5-0.5B backbone, not their original large-model versions.

Method	Parameters	Test gIoU ↑	Test cIoU ↑	Latency ms ↓	TFLOPs ↓	Memory GB ↓
LISA-7B	Not listed in this comparison table	47.3	48.4	320	7.16	18.4
LLaVA-OneVision	897M	46.8	46.0	264	6.11	11.9
LISA*	1.5B	46.5	45.2	233	3.08	6.2
READ*	1.5B	48.5	47.1	238	3.09	6.5
LReSeg	805M	50.8	50.0	98	1.42	3.6

LReSeg improves test cIoU by 2.9 percentage points over READ*. Compared with LISA-7B, it improves test cIoU by 1.6 percentage points and is approximately 3.27 times faster by the 320/98 latency ratio. The 805M count covers the full LReSeg model, whereas 0.5B refers to the language backbone; these should not be conflated. Timing numbers come from the paper's setup, but the text does not fully specify hardware, precision, and batch controls for the separate inference benchmark. Using RTX 3090 GPUs for training does not establish that inference was timed on the same hardware.

As supplementary referring-segmentation evidence, LReSeg reaches 71.2 cIoU on RefCOCOg test versus 68.0 for READ*. However, its 63.1 cIoU on RefCOCO+ val is below ReLA's 66.0. The evidence supports competitiveness across several settings, not superiority over every listed method on every split.

Ablation Study¶

The following subset of the paper's Table 3 reports ReasonSeg cIoU throughout. The last column subtracts each variant's test score from the full model's test score, in percentage points.

Config	Val cIoU ↑	Test cIoU ↑	Test drop from full model
`<SEG>` only	47.8	46.6	3.4
Without `<SEG>`, keeping registers and visual tokens	48.2	46.9	3.1
Without registers, keeping `<SEG>` and visual tokens	48.5	47.7	2.3
Fixed, non-learnable registers	48.7	47.4	2.6
Random visual token selection	50.8	48.1	1.9
Visual fusion at every layer	52.1	49.0	1.0
Removing the causal attention mask	49.5	48.2	1.8
DINO and SAM dual encoders	51.2	48.7	1.3
Full LReSeg	52.8	50.0	0.0

Key Findings¶

Among the listed ablations, retaining only <SEG> causes the largest test drop, at 3.4 percentage points. Removing <SEG> also substantially hurts performance, supporting complementary semantic prompting and dense spatial guidance rather than complete replacement of the semantic prompt by registers.
Fusing visual information at every layer does not outperform selective injection at layer 6, and the authors report lower speed. More injection does not necessarily mean better information quality.
The dual-encoder variant trails the full model by 1.3 percentage points, so the single encoder is not merely sacrificing accuracy for speed in this experiment. This ablation alone cannot establish that every dual-encoder design suffers from the same feature-consistency problem.

Highlights & Insights¶

The paper locates a small-model bottleneck in the information interface rather than attributing it solely to weak reasoning. Keeping target semantics in <SEG> and text-conditioned spatial information in registers is more targeted than expanding the responsibilities of one semantic token.
Causal order itself determines where information can be collected. Placing carriers at the sequence end and writing their content back during decoding obtains text-conditioned features without globally changing the language model's attention mask.
A unified encoder supports both parameter efficiency and representation consistency. Detail remains on the visual-backbone path while the small language model processes a shorter visual sequence, avoiding the need to discard all spatial resolution to reduce computation.

Limitations & Future Work¶

The demonstrated advantages mainly concern similarly sized methods and the specified LISA-7B baseline, not all large reasoning-segmentation models. The paper also does not validate cross-device deployment or long-video scenarios.
Performance depends on whether intermediate attention locates relevant regions. Layer 6 and a small register set work for the current backbone, but are not guaranteed to remain optimal for more complex relations, background distractions, or different pretrained models.
The cached main text does not fully explain tensor handling when concatenating top-k visual tokens with registers or the inference timing controls. Reproduction should first clarify these boundaries and add run-to-run variability and complex-instruction subgroup evaluation rather than relying only on overall averages.

vs LISA: LISA establishes the interface from language-generated <SEG> to mask prediction. LReSeg keeps that interface but adds text-conditioned dense features and removes the separate SAM image encoder. Its contribution is spatial compensation and architectural efficiency for small models, not a redefinition of reasoning segmentation.
vs READ: READ extracts high-response points from <SEG> activations to guide SAM. LReSeg instead injects text-attended visual tokens into registers inside the language model and writes them back as dense features. The table's READ* uses a substituted small language backbone, so the comparison should remain specific to that version.
vs LLaVA-OneVision: Its AnyRes approach produces more language-side visual tokens and additional attention computation. LReSeg retains a separate high-resolution feature pathway for pixel prediction while using compact tokens for language-conditioned interaction. This is not a comprehensive comparison of general visual question answering capability.

Rating¶

Novelty: 4/5. Registers, attention-based selection, and shared encoding have precedents, but their combination targets a clear spatial-interface bottleneck in small-model reasoning segmentation.
Experimental Thoroughness: 4/5. The study covers similarly sized baselines, efficiency, referring segmentation, and multiple ablations, but lacks complete timing conditions and statistical variability.
Writing Quality: 3/5. The overall mechanism is understandable, although some tensor connections and experimental settings remain underspecified, and cached equation formatting complicates reproduction-oriented reading.
Value: 4/5. The work offers an interpretable architecture for resource-constrained language-guided pixel prediction, with evidence supporting practical potential in the evaluated settings.