title: >- [Paper Note] CoCo-IR: Conversational Composed Image Retrieval description: >- [ECCV 2026][Multimodal VLM][Composed Image Retrieval] Introduces Contextual Composed Image Retrieval (CoCo-IR) and TIE, an LMM-based architecture utilizing a dedicated bottleneck token and hybrid attention, supported by an autonomous self-reflective data engine. tags: - ECCV 2026 - Multimodal VLM - Composed Image Retrieval - Conversational Search - Large Multimodal Models date: 2026-09-19 content_hash: a575f3d715f04b6a

CoCo-IR: Conversational Composed Image Retrieval¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://CoCo-IR.github.io
Area: Multimodal VLM
Keywords: composed image retrieval, conversational visual search, large multimodal models, representation learning, hard negative mining

TL;DR¶

Addressing the single-turn constraint of conventional composed image retrieval, this paper introduces Contextual Composed Image Retrieval (CoCo-IR) alongside TIE, an LMM-based retriever featuring an \(\langle\text{EMB}\rangle\) global bottleneck token, hybrid attention, and an autonomous data engine, achieving a 4-turn R@1 of 44.1% where prior methods collapse.

Background & Motivation¶

Instruction-based visual search has progressed from simple keyword queries to Composed Image Retrieval (CIR). In the standard CIR setup, a user provides a reference source image \(I_{\text{src}}\) and a natural language instruction \(T\) describing a target transformation (e.g., "change the car to red") to retrieve a target image \(I_{\text{tgt}}\) from a massive database. Powered by dual-encoder alignments and recent multimodal foundation models, single-turn CIR has witnessed considerable success in benchmarks like CIRR and CIRCO.

However, real-world visual exploration is inherently iterative, progressive, and exploratory. In practical applications, users rarely express a complex visual search goal in an isolated prompt; instead, they refine intermediate discoveries, explore branching options, or backtrack across steps (e.g., "remove the watermark, change the daytime lighting to sunset, but keep the initial viewpoint"). Traditional CIR models cannot support this dynamic because their architectures strictly accept a single image and text query. When users attempt multi-turn refinement, existing systems force them to start from scratch by formulating an increasingly cumbersome text query. Even when employing external frontier models (such as Gemini 2.5 Pro) to summarize interaction histories into a single instruction, fine-grained visual details and cross-turn references inevitably get compressed or lost.

This creates a fundamental disconnect between current single-turn models and interactive search needs. Core idea: formalize the multi-turn Contextual Composed Image Retrieval (CoCo-IR) task under strict trajectory evaluation metrics, propose Transformable Image Embeddings (TIE) with a dedicated \(\langle\text{EMB}\rangle\) global information bottleneck and within-turn bidirectional / cross-turn causal hybrid attention, and build an autonomous data engine powered by LMM self-reflection and hard negative verification.

Method¶

Overall Architecture¶

CoCo-IR enables users to iteratively query an image corpus \(\mathcal{C}\) over multiple turns. At turn \(t\), the user inputs instruction \(T_t\), and the model processes the cumulative interaction history \(H_t = (I_0, T_1, I_1, \dots, T_{t-1}, I_{t-1})\) alongside \(T_t\) to retrieve target image \(I_t\). TIE builds upon a Large Multimodal Model (LMM) backbone, serializing visual patch tokens and text tokens into a unified interleaved sequence. At the end of each query turn, a specialized \(\langle\text{EMB}\rangle\) token is appended. The sequence undergoes hybrid masked attention, after which the last hidden state of \(\langle\text{EMB}\rangle\) is projected via a linear layer into a compact retrieval vector \(q_{t,i}\) trained with single-side contrastive loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-turn Interleaved Input<br/>I_0, T_1, I_1, ..., T_t"] --> B["Multimodal Tokenization<br/>Vision Encoder + Text Tokenizer"]
    B --> C["Hybrid Attention Mechanism<br/>Within-turn Bidirectional + Cross-turn Causal"]
    C --> D["Global Information Bottleneck<br/>Dedicated EMB Token"]
    D --> E["Linear Projection & Contrastive Training<br/>Pull Target Image, Push Hard Negatives"]
    E --> F["Corpus Nearest-Neighbor Retrieval<br/>Retrieve Target Image I_t"]

Key Designs¶

1. Dedicated EMB Token: Overcoming Autoregressive Recency Bias Prior embedding approaches derived from causal language models commonly extract the hidden state of the final text token. In extended dialogues, however, causal self-attention creates a strong local bias toward immediate semantic predecessors, causing the final token to overweight the latest few words while discounting earlier images or long-range coreferences (such as referencing "the original viewpoint from the first image"). To establish an unbiased query representation, TIE inserts a dedicated \(\langle\text{EMB}\rangle\) token at the end of each query turn. This token acts as a global information bottleneck, compelling the transformer layers to compress the cumulative visual and textual transformations across the entire history into a unified embedding. The final hidden state of \(\langle\text{EMB}\rangle\) is passed through a linear projection layer, yielding superior dialogue aggregation compared to last-token pooling.

2. Within-Turn Bidirectional and Cross-Turn Causal Hybrid Attention: Fusing Multimodal Nuance while Preserving Sequence Flow Interactive retrieval demands two competing attention behaviors: within a given turn, visual image tokens and modifying text tokens must attend to each other without restriction to resolve fine-grained grounding; across different turns, information must strictly respect temporal causality to prevent future leakage. TIE introduces a tailored hybrid attention mask. For all tokens belonging to the same turn, bidirectional (full) attention is applied, enabling complete vision-language interaction. Across distinct turns, causal masking is enforced so that turn \(t\) can attend to all prior history steps \((0 \dots t-1)\) but cannot look ahead. This design preserves historical context while capturing immediate multimodal relationships.

3. Autonomous Data Engine with Self-Reflection and Negative Verification: Scalable Multi-Turn Dialogue Synthesis Because large-scale multi-turn contextual retrieval datasets do not exist and manual annotation is cost-prohibitive, the authors design a fully automated data engine: - Pair Discovery and Dual-Dimensional Self-Reflection: Image pairs with natural transformations are identified from web corpora via visual feature clustering and URL co-occurrence. An LMM inspects each pair, synthesizes a transformation instruction, and performs self-reflection by rating both image-pair transformation clarity (1–5) and instruction ambiguity (1–5). Only samples scoring high on both dimensions are retained. - Model-Guided Hard Negative Mining: Effective contrastive representation requires challenging negatives. Candidate negatives are retrieved from the corpus and validated by an LMM verifier using pairwise comparisons against the ground-truth target. Only candidates confirmed to fail the modification instruction are kept as verified hard negatives. - Dialogue Chaining with Contextual Rewriting: Single-turn transitions \((I_0 \to I_1 \to I_2 \dots)\) are concatenated into chains of up to 4 turns. An LMM rewrites subsequent instructions \((T_2 \dots T_m)\) to introduce explicit conversational coreferences (e.g., "compared to the original image", "without the hands from image 2"), producing natural, coherent trajectories.

A Worked Example¶

Consider an exploratory architectural visual search: 1. Turn 1: The user provides daytime photo \(I_0\) of Nyhavn canal and instruction \(T_1\): "Remove the text overlay at the bottom"; TIE maps \((I_0, T_1)\), aggregates context into \(\langle\text{EMB}\rangle\), and retrieves clean daytime image \(I_1\). 2. Turn 2: The user follows with \(T_2\): "Now that the text overlay is gone, change the scene to sunset and recover the original viewpoint"; TIE ingests history \((I_0, T_1, I_1, T_2)\). Hybrid attention lets \(T_2\) resolve the "original viewpoint" from \(I_0\) and the clean state from \(I_1\), successfully retrieving wide-angle sunset photo \(I_2\). 3. Turn 3: The user pivots with \(T_3\): "Actually, let's change back to daytime, but make it brighter and sunnier than the second image"; TIE parses the backtracking request and the explicit comparative reference to image 2, successfully retrieving the bright daytime canal view. Under the trajectory evaluation, all 3 consecutive turns must succeed for a successful query.

Loss & Training¶

The model is trained using a single-side InfoNCE contrastive objective. For query embedding \(q_{t,i}\) of sample \(i\) at turn \(t\), the positive pair is target image embedding \(p_{t,i}\). The candidate negative pool \(\mathcal{C}_e\) encompasses all target images, initial source images, and verified hard negatives across the batch:

\[ \mathcal{L}_{t,i} = -\log \frac{\exp(\text{sim}(q_{t,i}, p_{t,i})/\tau)}{\sum_{e \in \mathcal{C}_e} \exp(\text{sim}(q_{t,i}, e)/\tau)} \]

where \(\text{sim}(\cdot, \cdot)\) is cosine similarity and \(\tau\) is a fixed temperature parameter. Averaging across all samples and turns yields the total training objective. Ablation results verify that this asymmetric contrastive formulation outperforms symmetric CLIP-style loss for retrieval candidate discrimination.

Key Experimental Results¶

Main Results¶

Because prior CIR models accept only single-turn queries, they are adapted via three proxy strategies: Concatenated Instructions (pairing \(I_0\) with all accumulated text), Latest Inputs (pairing previous image \(I_{t-1}\) with current text \(T_t\)), and Summarized Context (using Gemini 2.5 Pro as an oracle to distill full history into an optimized single prompt). Performance is measured via \(m\)-turn Recall@k (requiring correct top-k retrieval at every intermediate turn).

Model	Context Input Strategy	1-turn R@1	1-turn R@5	2-turn R@1	2-turn R@5	4-turn R@1	4-turn R@5
TIPS-SO400M	Concat Instructions	25.2	51.0	0.4	21.3	0.0	5.1
MagicLens (CLIP-L)	Concat Instructions	36.7	57.8	8.1	26.7	0.0	2.3
BGE-VL-MLLM-S1	Concat Instructions	60.0	79.5	28.7	58.3	4.9	34.7
BGE-VL-MLLM-S1	Latest Inputs	60.0	79.5	36.2	64.1	27.0	54.0
MagicLens (CLIP-L)	Gemini 2.5 Summarized	43.5	66.5	19.8	47.0	10.7	33.0
BGE-VL-MLLM-S1	Gemini 2.5 Summarized	60.8	79.4	35.5	62.7	28.2	53.9
TIE-4B (Ours)	Full Context (Native)	74.5	89.8	48.3	79.5	31.1	65.8
TIE-12B (Ours)	Full Context (Native)	76.3	90.4	54.8	82.1	44.1	78.3

On standard single-turn benchmarks, TIE establishes new SOTA: on CIRCO it scores 39.4 mAP@5 (surpassing MagicLens's 34.1 and BGE-VL's 39.2) and 38.7 R@1 on CIRR, achieving the best overall average rank (1.50).

Ablation Study¶

The table below highlights the ablation on core architectural decisions (Bidirectional Attention BA, Single-side Contrastive loss SC, and Embedding Token ET) from Table 4:

Config (BA / SC / ET)	FIQ R@10	CIRR R@1	CIRCO mAP@5	CoCo-IR 1-turn R@1	Average Rank ↓	Note
Full Model (✓ / ✓ / ✓)	40.1	37.7	37.8	75.4	1.50	Optimal combination across all metrics
w/o Embedding Token (✓ / ✓ / ✗)	39.2	37.2	35.9	75.5	3.00	Last-token extraction drops CIRCO by 1.9%
w/o Single-side Loss (✓ / ✗ / ✓)	39.6	35.9	37.4	75.6	2.25	Symmetric loss is suboptimal for candidate ranking
w/o Bidirectional Attention (✗ / ✓ / ✓)	39.7	33.9	36.6	74.2	3.25	Pure causal masking degrades within-turn fusion (-3.8% CIRR)

In terms of data scale (Table 5), TIE-4B trained on just 320K autonomously synthesized samples outperforms MagicLens trained on 36.7M samples across FIQ (38.5 vs 38.0 R@10), CIRR (34.8 vs 33.3 R@1), and CIRCO (36.7 vs 34.1 mAP@5), delivering over 100× data efficiency.

Key Findings¶

Multimodal Context Cannot Be Losslessly Compressed: Even with a frontier oracle like Gemini 2.5 Pro summarizing full visual-text history into a single instruction, the strongest baseline reaches only 28.2% 4-turn R@1, whereas native TIE-12B achieves 44.1%. Visual details and cross-turn references are fundamentally lossy when flattened to text.
Causal Attention Impairs Intraturn Vision-Language Grounding: Eliminating within-turn bidirectional attention causes CIRR R@1 to drop sharply from 37.7 to 33.9, underscoring that image patches and instruction tokens must attend bidirectionally for precise composition.
Data Quality Outweighs Sheer Quantity: Filtering candidates via self-reflection and mining LMM-verified hard negatives allows synthetic data to generalize effectively to out-of-domain human-annotated benchmarks with exceptional sample efficiency.

Highlights & Insights¶

Strict Trajectory Recall Metric: Introducing \(m\)-turn Recall@k captures the unforgiving reality of interactive retrieval where an intermediate retrieval failure derails all subsequent context-dependent instructions.
\(\langle\text{EMB}\rangle\) Global Information Bottleneck: Repurposing conversational foundation models for dense retrieval via a specialized summary token eliminates the recency bias inherent to autoregressive causal representations.
Closed-Loop Data Engine: Autonomous image discovery, dual-dimension self-reflection, and iterative negative mining form a self-sustaining data flywheel that bypasses expensive manual annotation.

Limitations & Future Work¶

Author-Acknowledged Limitations: Trajectory length is currently bounded to 4 turns due to the natural sparsity of longer coherent image transformation chains in web data; computing budgets restricted hard negative mining to one negative per instance and a single mining iteration.
Potential Bottlenecks: The benchmark assumes ground-truth intermediate history during evaluation and cannot simulate dynamic human recovery strategies when a bad retrieval occurs; memory footprint for multi-image KV-cache grows linearly with conversation depth.
Future Directions: Implementing reinforcement learning with multi-agent rollouts to simulate adaptive human feedback, alongside token pruning and KV-cache compression for low-latency interactive serving.

vs MagicLens [ICML 2024]: MagicLens relies on 36.7M pairs and remains restricted to single-turn retrieval; TIE supports arbitrary conversational turns and outperforms MagicLens using <1% data volume.
vs E5-V [2024] & BGE-VL [ACL 2025]: While E5-V and BGE-VL explore LMMs for embedding tasks, they rely on last-token representations and single-turn architectures that fail on conversational reference grounding.
vs IRR [ACM MM 2023] & MAI [ICLR 2025]: Prior multi-turn CIR works are confined to the narrow fashion domain and only permit an image query at the initial turn; CoCo-IR provides an open-domain benchmark supporting interleaved images and text across every interaction step.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates multi-turn contextual CIR and introduces a native LMM-based retrieval framework with an elegant bottleneck token design.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across three single-turn benchmarks, multi-turn trajectory metrics, extensive baseline adaptations, and thorough ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, structured motivations, and transparent experimental reporting.
Value: ⭐⭐⭐⭐⭐ Serves as a vital benchmark and methodological foundation for conversational visual search, e-commerce exploration, and multimodal agents.