Toward Robust In-Context Segmentation via Concept Guidance¶

Conference: ECCV2026
arXiv: 2606.28149
Code: https://github.com/MAC-AutoML/CG-ICS
Area: Semantic Segmentation
Keywords: in-context segmentation, robustness, concept guidance, SAM3, tree search

TL;DR¶

CG-ICS reformulates the in-context segmentation problem as a concept-guided prompt segmentation problem: an MLLM is employed to generate candidate concepts, which are then scored by SAM3 and selected via tree search to find the optimal textual concept. Concurrently, an image-stitching technique extracts query-side visual exemplars from the reference images. Both cues jointly drive a frozen SAM3 to complete the segmentation, which significantly reduces performance variance caused by reference selection while maintaining high accuracy.

Background & Motivation¶

In-context segmentation (ICS) aims to segment target objects in a query image given only a few reference images and corresponding masks, without any parameter updates. While this paradigm flexibly adapts to the open world, existing methods almost entirely overlook a critical issue: system robustness. With the same query image, changing the reference images can lead to vastly different segmentation results. For instance, using GF-SAM, the current state-of-the-art (SOTA) training-free method, switching reference samples for the same image can cause the IoU to plummet from over 90 to under 30. Prior works have noticed this sensitivity, but their solution is to "select the optimal reference from a candidate pool." In actual deployment, however, users provide a single reference on the fly, leaving no opportunity for selection. A truly usable ICS system should deliver stable results under any reasonable reference, rather than only functioning when a "good" reference is encountered.

The root cause of this vulnerability lies in the reliance of existing ICS methods on low-level visual matching (points, regions, or feature correspondences) between the reference and query images. When changes in view, occlusions, or lighting variations occur, low-level matching drifts, leading to semantic misalignment. The key insight of this paper is: if ICS can first extract high-level semantic concepts indicating "what the target is" (e.g., "dog," "airplane") and use these concepts to guide segmentation, it can fundamentally escape the reliance on low-level visual correspondence. SAM3 provides exactly this capability—given a textual concept description, it understands the semantics and generates precise segmentation masks. However, the key challenge is that the standard setup of ICS only provides reference images and masks, without any category labels or textual descriptions, meaning the system must derive the appropriate concepts from the references autonomously.

The core idea of this work is to leverage an MLLM to autonomously generate candidate textual concepts from reference image-mask pairs, employ SAM3 for scoring and tree search to select the optimal concept, and simultaneously use the reference bounding boxes on a stitched image to derive query-side visual exemplars. The optimal text and visual exemplars then jointly drive SAM3 to produce the final segmentation. This reformulates ICS as a prompt concept segmentation (PCS) problem, enabling the system to work stably despite highly variable references.

Method¶

Overall Architecture¶

CG-ICS reformulates ICS as a prompt concept segmentation problem with SAM3, employing two parallel pathways to extract guidance information from reference images. One pathway is concept reasoning—using an MLLM to generate candidate textual concepts, which are iteratively refined to select the optimal concept via tree search (expansion \(\rightarrow\) scoring \(\rightarrow\) pruning \(\rightarrow\) re-expansion). The other is visual exemplar extraction—horizontally stitching the reference and query images, utilizing the bounding boxes of the reference masks to invoke SAM3 on the stitched image, and cropping the mask from the query half to convert it into a visual exemplar bounding box. Finally, the selected textual concept and the derived visual exemplar are fed together into SAM3 to obtain the final segmentation result. All model parameters remain frozen throughout.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Image + Mask + Query Image"] --> B["Concept Reasoning Branch"]
    A --> C["Visual Exemplar Extraction Branch"]

    B --> B1["MLLM Dual-View Input<br/>Generate N Candidate Concepts"]
    B1 --> B2["SAM3 Scoring<br/>Reference Fidelity × Query Matching"]
    B2 --> B3{"Score >= Stop Threshold?"}
    B3 -->|Yes| B4["Select Highest-Scoring Concept T*"]
    B3 -->|No| B5{"Score < Pruning Threshold?"}
    B5 -->|Yes| B6["Prune"]
    B5 -->|No| B7["MLLM Re-expansion<br/>→ Sub-node Concepts"]
    B7 --> B2

    C --> C1["Horizontally Stitch Reference + Query"]
    C1 --> C2["SAM3 Inference on Stitched Image<br/>→ Query-side Mask"]
    C2 --> C3["Convert Mask to Bounding Box V*"]

    B4 --> D["SAM3 Joint Prompting<br/>Text T* + Visual V*"]
    C3 --> D
    D --> E["Final Query Mask"]

Key Designs¶

1. Tree-Search Concept Reasoning: MLLM Expansion + SAM3 Scoring and Pruning

To enable SAM3 to segment the target correctly, a precise concept description is required. However, reference images only provide pixel masks without textual labels. The most direct approach is to let the MLLM "describe the image." However, a single-turn generation may produce descriptions that are too broad ("animal") or too specific ("Golden Retriever"), necessitating finding the most suitable level of granularity. This paper formulates the problem as tree search over concepts: the root node is empty, and the MLLM first generates \(N\) candidate concepts on the root as the first layer. Each candidate node is scored by SAM3 across two dimensions: Reference Fidelity (the IoU between the semantic mask of the concept on the reference image and the ground truth mask) multiplied by Query Matching (the probability of the concept's presence in the query image). The overall score is computed as \(S = RF \times QM\). Nodes with scores exceeding a stopping threshold (\(0.8\)) terminate early, while nodes scoring below a pruning threshold (\(0.5\)) are discarded. The remaining nodes are sent back to the MLLMs for further expansion (generating synonyms or finer-grained sub-concepts). This process iterates for a maximum of \(K = 3\) rounds, and the highest-scoring concept in the entire tree is selected. This dual-phase scoring ensures that the concept faithfully describes the reference sample while maintaining semantic consistency with the query image, suppressing both types of mismatch (incorrect descriptions that happen to exist in the query, or correct descriptions that are absent in the query).

2. Visual Exemplar Extraction: Propagating Reference Boxes to the Query Side via Stitched Images

Textual concepts may still be insufficient in certain scenarios; for rare categories or objects with highly specific appearances, even precise noun descriptions fail to capture all visual details. SAM3 natively supports visual exemplar prompting (target bounding boxes), but the exemplar boxes must be defined on the target image. This paper designs a simple propagation mechanism: horizontally stitching the reference and query images, and using the reference mask's bounding box as a prompt to let SAM3 perform inference on the entire stitched image. Since SAM3 can segment all matching instances, it generates masks on both the reference and query sides. Extracting the mask from the query half and converting it to a bounding box yields the visual exemplar on the query image. This design cleverly bypasses the difficulty of cross-image exemplar propagation. Without requiring any training or additional learning, this simple geometric operation of stitching enables SAM3 to respond consistently on both sides.

3. Joint Textual and Visual Prompting: Complementing Semantic Concepts with Spatial Anchoring

The final segmentation stage fuses both cues: the textual concept \(T^*\) provides category-level semantics (telling SAM3 "what it is"), while the visual exemplar \(V^*\) provides spatial anchoring on the query image (telling SAM3 "roughly where it is and what it looks like"). Both serve as joint prompts for SAM3 to produce the final mask in a single forward pass. Ablation studies show that using only the textual concept yields a Coefficient of Variation (CV) of \(14.7\%\), which drops to \(12.9\%\) when the visual exemplar is added. This indicates that visual cues provide orthogonal gains in spatial localization, especially when the concept words are imprecise, confining attention to the correct region.

Multi-reference Extension¶

When multiple references are available, CG-ICS performs independent concept reasoning for each reference to obtain their respective optimal concepts, and then rescores them across all references to select the one with the highest consistency. For visual exemplars, stitching is executed once for each reference-to-query pair, collecting all bounding boxes as a joint prompt. This "divide-and-conquer" design is more reliable than processing multiple images at once, as the reasoning capabilities of MLLMs are limited when handling multiple images simultaneously.

Key Experimental Results¶

Main Results¶

Comparison results on four standard ICS benchmarks:

Benchmark	Setting	CG-ICS	Strongest Training-Free Baseline	Strongest Trained Baseline (w/ In-Domain Data)
Pascal-5i	1-shot	89.3	72.1 (GF-SAM)	88.3 (DiffwS)
COCO-20i	1-shot	72.3	58.7 (GF-SAM)	77.8 (UNICL-SAM)
LVIS-92i	1-shot	55.4	35.2 (GF-SAM)	50.3 (SANSA)
FSS-1000	1-shot	90.2	88.0 (GF-SAM)	90.0 (SANSA)

As a training-free method, CG-ICS outperforms all methods trained with in-domain data on Pascal-5i, LVIS-92i, and FSS-1000, achieving state-of-the-art (SOTA) performance.

Robustness Evaluation (Reference Selection)¶

By fixing the query and sampling 50 different references for each, the standard deviation and coefficient of variation (CV) were measured:

Method	Pascal-5i CV	COCO-20i CV	LVIS-92i CV
CG-ICS (Ours)	7.8%	12.9%	30.1%
GF-SAM	27.3%	28.1%	44.3%
SANSA (Trained)	11.1%	17.6%	35.8%

CG-ICS achieves the lowest CV across all benchmarks, indicating a substantial reduction in sensitivity to reference selection.

Robustness Evaluation (Reference Corruption)¶

Applying 6 types of degradations (color shift, blur, compression, spatial transformation, domain shift, and deformation) to the reference images, the relative drop in mIoU was measured:

Method	Clean mIoU	Average Drop Ratio
CG-ICS (Ours)	74.6	-3.1%
UNICL-SAM (Trained)	79.8	-3.9%
GF-SAM	61.5	-4.1%

CG-ICS maintains remarkable stability under various degradations, being virtually unaffected by color shifts or blur (drop \(\le 0.3\) mIoU).

Ablation Study¶

Configuration	mIoU	Std	CV	Description
Single-turn MLLM Concept	67.6	15.1	22.3%	Directly generate a single concept for segmentation
+ Multi-candidates	66.5	15.3	23.0%	Merely increasing candidates yields no benefit
+ RF Scoring	69.3	13.9	20.5%	Reference fidelity aids filtering
+ QM Scoring	62.8	18.2	29.0%	Using only query matching degrades performance
+ RF + QM	70.1	11.3	16.1%	The combination of both is crucial
+ Tree Search	71.2	10.5	14.7%	Iterative refinement brings further improvements
+ Visual Exemplar	72.1	9.3	12.9%	Full system

Each component contributes measurable gains, with the joint RF+QM scoring serving as the turning point for variance reduction. Performance benefits saturate after setting the number of search nodes to \(N=5\) and the number of rounds to \(K=3\).

Highlights & Insights¶

Redefining ICS as a PCS problem is an elegant and profound conceptual shift: While prior ICS works were confined to "visual correspondence between reference and query," this paper breaks free—instead of matching pixels, reference images are used to extract semantic concepts, which then drive the segmentation. This shift eliminates the vulnerability of low-level matching (changes in view, occlusions, and illumination variations are entirely absorbed by the semantic understanding of the MLLM).
The two scoring dimensions of the tree search design (Reference Fidelity \(\times\) Query Matching) exhibit a multiplicative synergy: Ablations show that using either dimension alone degrades performance (using QM alone increases the CV to \(29.0\%\)), but multiplying them drops the CV to \(16.1\%\)—since RF filters out inaccurate concepts and QM ensures the concept is present in the query, both are indispensable.
The technique of propagating visual exemplars via stitched images is minimal yet elegant: Without introducing any parameters or learning, it simply stitches the reference and query together, utilizing the spatial consistency of SAM3 on the identical image to propagate bounding boxes across images. It is simple enough to require almost no explanation, yet serves as a critical, stabilizing complement to the full system.

Limitations & Future Work¶

The inference cost of the MLLM is non-negligible: each sample requires multiple rounds of MLLM calls (with \(N=5\) candidates per node for up to \(K=3\) rounds), which significantly increases latency and computational overhead compared to purely visual matching methods like GF-SAM.
When the target category is extremely rare or the MLLM hallucinates, concept reasoning may converge on erroneous semantic concepts—although RF scoring provides some defense, it is powerless if the MLLM misidentifies the object entirely.
Visual exemplar extraction relies on the consistency of SAM3 on stitched images: if there is an extreme appearance discrepancy between the target in reference and query images (e.g., completely different species), SAM3 may output empty masks on the query side, causing the visual branch to fail and leaving the model reliant solely on the textual concept.

vs GF-SAM / PerSAM / Matcher: These training-free methods rely on low-level visual matching (point/region correspondences) between reference and query. This paper replaces low-level matching with high-level semantic concepts, fundamentally improving robustness.
vs SegGPT / SINE / Other Trained Methods: Trained methods require fine-tuning on vast in-domain datasets. In contrast, the proposed method completely freezes the models, requiring no training while delivering stronger generalization.
vs UNICL-SAM: UNICL-SAM focuses on robustness under reference corruption, but its training paradigm is highly costly. In a training-free setting, the proposed method not only handles corrupted references but also systematically tackles the variance issues caused by arbitrary reference selection.

Rating¶

Novelty: ⭐⭐⭐⭐ Reformulating ICS as a PCS problem and introducing tree-search concept reasoning is highly creative and effective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Two carefully designed robustness evaluation protocols (reference selection + reference corruption) + 4 benchmarks + comprehensive ablation studies make the evaluation highly solid.
Writing Quality: ⭐⭐⭐⭐ Clear motivation, detailed methodological exposition, and excellent alignment between text and figures.
Value: ⭐⭐⭐⭐ Important obstacles to the practical deployment of ICS (reference sensitivity) are cleared, and the core ideas can be transferred to other in-context tasks.