OCTOPUS: Multi-Agentic Universal Compositional Visual Retrieval

Conference: ECCV 2026
Paper: Official page · PDF
Code: https://github.com/zbzzbzzbz/OCTOPUS
Area: Multi-Agent Systems / Composed Visual Retrieval
Keywords: composed retrieval, tool use, semantic refinement, multimodal proxies, candidate verification

TL;DR

OCTOPUS uses three task-training-free agents for perception, creation, and retrieval to turn a reference image or video and an editing instruction into visual and textual proxies, then improve ranking through bidirectional cross-modal retrieval and sequential candidate verification, reaching 67.15% WebVid R@1 and 37.00% average FashionIQ R@10 with ViT-L/14.

Background & Motivation

Composed visual retrieval is not simply about finding content that resembles a reference. It must preserve relevant reference properties while implementing a requested change. For a printed shirt, an instruction asking for a green four-leaf clover design without text requires removing the lettering constraint without abandoning the garment itself. Supervised approaches learn this composition from reference-modification-target triplets, but image and video systems are often trained separately, and their supervision does not transfer easily across domains.

Zero-shot approaches move composition into language. Pic2Word and LinCIR represent reference content through pseudo-word embeddings, while methods such as CIReVL produce reference captions and use a language model to infer a target description. Language makes edits easier to express but creates another bottleneck: captions can omit objects, instructions can be ambiguous, and words that make sense to an LLM may not be understood reliably by CLIP. A single target sentence provides little additional evidence about what was missed. Adding a generated image naively can instead elevate arbitrary generated background details into retrieval criteria.

OCTOPUS therefore focuses on supplementing and checking evidence at inference time rather than training another fusion encoder. It establishes the requested transformation, imagines the target, filters object-level clues, and checks whether retrieved candidates actually satisfy the change. Core idea: use language to specify modification constraints, visual and textual proxies to recover potentially missing semantics, and candidate verification to correct similarity-based rankings within one tool workflow for both images and videos.

Method

Overall Architecture

The inputs are reference visual content and a modification instruction. The output is a ranked list of real database images or videos, not the generated image. The Perceiver converts the inputs into a target caption; the Creator builds a visual proxy and multiple textual proxies; the Retriever combines two cross-modal similarity paths and verifies highly ranked candidates.

Here, agents are primarily inference modules with distinct responsibilities and access to existing tools, not three independent policies trained through reinforcement learning. Caption self-refinement occurs before retrieval, while ranking correction occurs afterward. The paper does not specify an unrestricted outer loop that returns to image generation whenever retrieval fails.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Reference image or video<br/>and modification text"] --> Perceiver["Perception and<br/>Self-Refinement"]
    Perceiver --> Creator["Visual and Textual<br/>Proxy Creation"]
    Creator --> Retrieval["Bidirectional<br/>Cross-Modal Retrieval"]
    Database["Candidate visuals<br/>and captions"] --> Retrieval
    Retrieval --> Verification["Candidate Verification<br/>and Re-ranking"]
    Verification --> Output["Ranked real targets"]

Key Designs

1. Perception and Self-Refinement: describe what must stay and what must change

The Perceiver first invokes visual captioning. An image receives a global description; a video is uniformly sampled into \(N\) keyframes and summarized using the same type of tool. Both tasks can consequently share downstream language reasoning. A modification tool then supplies the reference caption and user instruction to an LLM to produce an initial target caption. This is semantic editing rather than string concatenation: replacing an object requires removing the old object constraint while preserving scene properties that the user did not ask to change.

The initial caption can retain incorrect details or add unsupported information during reasoning. A semantic refinement tool therefore considers both the initial target caption and the reference description, highlights manipulated elements, filters inaccurate content, and preserves contextual coherence. Its question is whether the target sentence expresses the requested edit, not whether a matching target exists in the database. This distinction matters: a linguistically coherent description neither guarantees that CLIP understands it nor replaces candidate verification. The paper describes prompt-based refinement without specifying a fixed multi-round convergence criterion.

2. Visual and Textual Proxy Creation: discover missing semantics without accepting every generated detail

Reference content alone cannot fully depict objects or attributes that have yet to appear. The Creator uses the target and reference captions to infer a plausible scene and layout, then invokes a generator such as Qwen-Image to produce a visual proxy. This proxy supports retrieval; it is not intended as a pixel-faithful edited output. The authors argue that composed retrieval tolerates visual variation, so generation need not reproduce every reference detail. Even for video inputs, the system primarily generates images to avoid the cost of video generation.

The next step is more specific than retrieving directly with the generated image. An object detection tool analyzes the reference and visual proxy to extract regions and salient objects. An LLM then distinguishes objects that should exist in the target from those that should not, using the composed query as a constraint. This prevents every incidental generated background object from entering the query. The retained objects and target caption are used to generate \(K\) textual proxies with diverse descriptions of the same intended target. Generation thus proposes semantics, while the requested modification controls which proposals become retrieval evidence. This filtering is not a factual guarantee: generation, detection, and LLM judgments can all be wrong.

3. Bidirectional Cross-Modal Retrieval: match the visual proxy to candidate captions, not candidate pixels

The first path matches textual proxies against database visuals, or text-to-vision. The second matches the visual proxy against descriptions of database candidates, or vision-to-text. The latter should not be mistaken for image-to-image retrieval. The authors specifically argue that directly comparing a generated proxy with candidate images can overweight irrelevant visual details, so they instead compare generated visual content with candidate language semantics. Candidate-side captions and text features are consequently part of the required retrieval infrastructure.

For a single textual proxy \(T_A\), visual proxy \(V_p\), candidate visual \(I\), and its caption \(C_I\), the basic cosine similarities described in the prose can be written consistently as:

\[ s_{t2v}(T_A,I)=\cos\bigl(\Psi_t(T_A),\Psi_v(I)\bigr),\qquad s_{v2t}(V_p,C_I)=\cos\bigl(\Psi_v(V_p),\Psi_t(C_I)\bigr). \]

Here, \(\Psi_t\) and \(\Psi_v\) denote CLIP's text and visual encoders. This notation restates the two similarities explicitly described in the paper; it does not reconstruct the damaged gating equation. The aggregation of multiple textual proxies and the exact final fusion implementation cannot be inferred from these expressions. The authors use a parameter \(\lambda\) to control fusion and report that excessive visual-proxy influence can obscure textual semantics. Operators in cached equation (2) are incomplete, so no addition, multiplication, or normalization rule is guessed here. The phrase "adaptive gating" alone also does not establish that a gate is learned.

4. Candidate Verification and Re-ranking: test whether the edit holds beyond embedding similarity

A highly similar candidate may depict the correct subject while violating a fine-grained condition such as no lettering or a higher neckline. The Retriever therefore invokes an LLM evaluation tool that considers the reference, composed query, and candidate descriptions, returning Yes/No with a justification. Pairwise comparison here concerns the reference and a candidate, not an all-pairs tournament among retrieved items.

Candidates are checked sequentially in their initial ranking order. The first acceptable candidate is moved to the top, or checking stops after at most \(\alpha\) candidates. This is neither database-wide rescoring nor an exhaustive new ordering of every top candidate. The finite budget limits calls but makes success depend on initial retrieval placing a suitable target within the inspected range; verification cannot recover a missing target from outside that range. Cached equation (3) also has extraction damage, so this account follows the explicit stopping rule in the prose without inventing the missing ranking operator.

A Worked Example

The clothing modification in the paper's Figure 4 includes green, a four-leaf clover, and no text. The following walkthrough explains the mechanism rather than reconstructing an unreported execution trace. Perception and Self-Refinement first produces a target garment description and removes lettering information that conflicts with the instruction instead of retaining the reference caption verbatim.

Visual and Textual Proxy Creation imagines a green clover garment, then filters object evidence to retain the required pattern and exclude unwanted text. The textual proxies express the same joint constraints, rather than independently searching for unrelated green objects and clovers. Bidirectional Cross-Modal Retrieval uses these descriptions and the visual proxy to retrieve real garment candidates.

If a high-ranked candidate has the correct color but still contains lettering, Candidate Verification and Re-ranking should reject it and continue until an acceptable candidate is found or the \(\alpha\) budget is exhausted. The paper does not provide per-candidate scores, actual verification counts, or proxy counts for this example, so no numerical candidate-reduction trace is asserted.

Loss & Training

OCTOPUS does not train task-specific parameters or introduce a new training loss. Training-free means reusing pretrained tools, not that the tools were never trained. The default LLM is GPT-4o-Mini, and visual generation uses Qwen-Image. The implementation section lists CLIP ViT-B/32, ViT-L/14, and ViT-G/14 with feature dimensions of 512, 768, and 1024 respectively. All main results and ablations selected below use ViT-L/14.

The paper reports relatively stable Toptee and CIRCO performance for \(\lambda\in[0.2,0.4]\), with the best WebVid result at \(\lambda=0.2\). The cache does not clearly establish reproducible default values for \(N\), \(K\), and \(\alpha\), or the full proxy aggregation rule. Sensitivity curves should not be used to invent complete default configurations for each task.

Key Experimental Results

Main Results

The following values come from Tables 1 and 2. OCTOPUS and the comparator in each row use ViT-L/14. Scores are percentages; gains are percentage-point differences, not relative percentages. Each comparator is the strongest other method in the corresponding source-table column, so the comparator is not identical across rows.

Dataset	Metric	Strongest comparator	Comparator score	OCTOPUS	Gain (percentage points)
WebVid	R@1	Pic2Word	48.47	67.15	+18.68
FineCVR	R@1	CVRDM	17.25	20.16	+2.91
CIRCO	mAP@5	IP-CIR	26.43	28.18	+1.75
FashionIQ	Average R@10	OSrCIR	33.26	37.00	+3.74
FashionIQ	Average R@50	OSrCIR	54.37	57.60	+3.23

R@k measures the proportion of queries whose target appears among the top \(k\) results. CIRCO has multiple ground-truth targets per query and uses mAP@k to account for relevance and ranking position. FashionIQ averages cover Shirt, Dress, and Toptee. The experimental setup calls a video benchmark WebCVR, while the results use WebVid; this note follows the table label. CIRR is listed in the setup, but the available cache contains no verifiable numerical CIRR table, so the claim of five benchmarks is not presented as five verified result groups.

Ablation Study

The following raw metrics are taken directly from Table 3. Its Avg. columns and prose about relative declines are not reused, avoiding ambiguity from rounding and aggregation across metrics. The backbone remains ViT-L/14 and the default LLM is GPT-4o-Mini.

Configuration	WebVid R@1	WebVid R@5	CIRCO mAP@5	CIRCO mAP@10
Full OCTOPUS	67.15	85.28	28.18	28.66
Without modification reasoning	60.34	81.01	26.40	27.51
Without semantic refinement	60.45	80.70	24.67	25.62
Without textual proxies	62.06	81.28	23.84	25.02
Without visual proxy	61.90	81.01	25.26	26.24
Without the entire Creator	59.44	79.21	22.84	23.76
Replacing bidirectional retrieval	61.51	81.48	26.91	27.48
Without candidate verification	52.70	78.58	26.36	27.30

Replacing bidirectional retrieval retains text-to-vision but substitutes vision-to-vision for vision-to-text; it does not disable retrieval. Removing textual proxies corresponds to skipping semantic increment and should not be treated as independent evidence that every text-diversification strategy is effective.

Key Findings

• WebVid is particularly sensitive to candidate verification: R@1 falls from 67.15 to 52.70, a 14.45-point decline, supporting the usefulness of checking fine-grained errors among leading candidates.
• CIRCO depends more on the Creator: removing it reduces mAP@5 from 28.18 to 22.84, a 5.34-point decline. Removing textual proxies alone costs 4.34 points. The dominant bottleneck differs across domains.
• The authors report 5.3 seconds per query, versus 33.8 seconds for IP-CIR, 6.07 seconds for AutoCIR, and 0.6 seconds for OSrCIR. More than 95% of OCTOPUS runtime is spent on external API calls. Training-free does not mean low latency, and these timings also depend on the service environment.

Highlights & Insights

• A generative model can propose retrieval semantics without producing the final answer. Generation followed by object filtering and textual proxy construction turns imagination into intermediate evidence checked against user constraints.
• Bidirectional retrieval changes what is compared, not merely how many scores are added. Matching visual proxies to candidate captions gives a concrete way to reduce distraction from generated backgrounds.
• Candidate verification separates embedding proximity from satisfaction of the edit. This could transfer to product or media retrieval, provided evaluation also measures rejection of correct candidates and additional call costs.

Limitations & Future Work

• Author-reported conceptual mismatch: Figure 5 shows that "exotic patterns" is meaningful to the LLM but poorly handled by the retriever; replacing it with "bold floral print" improves retrieval. An interpretable language interface can still require manual wording changes rather than eliminating prompt sensitivity.
• Limits of video coverage: the shared input interface relies on keyframe captions, while creation mainly produces static images. Our assessment is that this does not establish equally strong handling of action ordering, motion speed, or long-duration event modifications.
• Compute and dependency costs: external reasoning, generation, captioning, and verification add latency, while candidate captions add indexing work. The authors propose a lighter framework; budget-adaptive verification, proxy caching, and local tool combinations are useful further experiments.
• Incomplete reproducibility evidence: fusion and ranking equations are damaged in the cached extraction, while some defaults and CIRR results are not clearly provided. The original PDF or code is needed to resolve these points; this note is not a complete implementation specification.
• Comparison boundaries: sharing a CLIP backbone does not equalize inference resources because OCTOPUS additionally calls generative models and LLMs. The ablations support the components within this system but do not isolate agent organization from extra computational budget.

Related Work & Insights

• Versus Pic2Word / LinCIR: pseudo-word mappings compress visual information into language queries, whereas OCTOPUS explicitly expands target descriptions, creates proxies, and verifies candidates. Its readable intermediate process comes with substantially higher inference cost.
• Versus CIReVL / LDRE / OSrCIR: these approaches emphasize language reasoning, divergent descriptions, or pre-retrieval reflection. OCTOPUS further connects object filtering, visual proxies, and post-retrieval verification rather than merely replacing a prompt template.
• Versus IP-CIR / AutoCIR: imagined proxies and automatic collaboration have precedents, so an unconditional first-ever multi-agent retrieval claim is unwarranted. A more defensible contribution is the shared image-video workflow and its specific cross-modal proxy integration.
• Versus CoVR / CVRDE / CVRDM: these methods emphasize composed video representations, while OCTOPUS reuses tools through a language interface across domains. Unifying a tool workflow and learning high-quality temporal representations remain different problems.

Rating

• Novelty: 4/5. Visual-textual proxies, bidirectional matching, and verification form a clear combination, although several underlying ideas have precedents.
• Experimental Thoroughness: 4/5. Cross-domain results and component ablations are useful, but visible CIRR evidence and budget-matched comparisons remain insufficient.
• Writing Quality: 3/5. The workflow is intuitive, but inconsistent benchmark naming and damaged cached equations limit precise reproduction.
• Value: 4/5. Useful for interpretable modification-based retrieval when second-scale inference latency is acceptable.