Learning Sample-wise Rank-Aware Interpolation Weights for Composed Visual Data Retrieval

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: composed visual retrieval, sample-wise weights, spherical linear interpolation, rank-aware supervision, prototype memory bank

TL;DR

SRAIN estimates query-specific spherical interpolation weights from within-batch target ranks, then replaces query-time multimodal large-model fusion with two-stage training and prototype-conditioned prediction, achieving 61.07% R@1 on WebVid-CoVR while retaining a substantial gap to an oracle that uses true test targets.

Background & Motivation

Composed visual retrieval takes a reference image or video and a requested modification, rather than a complete standalone description. The system must retain reference content that should not change without allowing the original visual semantics to overwhelm the requested edit. For example, "make it yellow" needs the reference to identify "it," whereas "make the jogger a young girl" provides more explicit target semantics. Visual and textual inputs should therefore not contribute equally to every query. Existing high-performing approaches often use a multimodal large language model to rewrite both inputs into a target description before text-based retrieval, paying for this fusion capability in query latency.

A lighter starting point is spherical linear interpolation in the shared embedding space of a vision-language model. Previous Slerp work shows that this operation can be effective, but tuning one global weight per dataset cannot accommodate different semantic strengths of modifications within that dataset. Training a weight predictor directly also lacks supervision: the weight that places a query closest to its correct target need not rank that target above similar distractors. If interpolation also increases similarity to many negatives, positive similarity can improve while the target rank deteriorates. The desired weight depends on candidate competition, rather than an abstract notion of visual or textual importance.

The paper turns this into a practical approximation by comparing candidate weights within sufficiently large training batches and approximating difficult-candidate geometry with compact prototypes at deployment. This avoids searching the entire training gallery to construct labels and avoids assuming that the correct target is known at test time. Core Idea: supervise sample-wise interpolation with the target's rank relative to negatives, then teach a lightweight predictor this rank-dependent fusion using real candidates during training and prototype candidates during inference.

Method

Overall Architecture

SRAIN stands for Sample-wise Rank-Aware INterpolation and supports composed image retrieval, CoIR, and composed video retrieval, CoVR. It uses BLIP-2 with ViT-G/14 and a Q-Former, but encodes the reference and modification separately instead of merging them through a heavy joint generation process. The output is a query-specific scalar weight that produces a fused query through spherical interpolation, followed by cosine-similarity ranking against targets. Here, "linear interpolation" specifically means spherical linear interpolation on the unit sphere, not an unnormalized element-wise weighted sum.

A reference image goes directly into the frozen ViT; a reference video supplies only its middle frame, following prior work. Learnable Q-Former queries extract tokens from image patch representations, followed by projection, token averaging, and normalization to obtain the reference vector. The modification text independently enters the Q-Former without visual conditioning; the first-token hidden state is projected and normalized to obtain the text vector. The target visual branch stays frozen, producing a normalized representation for a target image. For a target video, the model uniformly samples frames, extracts per-frame query tokens, and applies modification-dependent query scoring to aggregate each query token across frames. It then averages the aggregated tokens and normalizes the result to obtain the target representation used for comparison. The single-frame reference approximation and multi-frame target aggregation are therefore asymmetric and do not explicitly represent the complete reference action sequence.

Training proceeds through rank-aware labels and two-stage conditioned regression; deployment replaces real positive and negative candidates with prototype-conditioned inference. The diagram separates label construction from the actual test path: a test query never passes through label search that uses its correct target.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Train["Training triplets<br/>Separate encoding"] --> Rank["Rank-aware labels"]
    Rank --> Fit["Two-stage<br/>conditioned regression"]
    Train -->|Target embeddings update prototypes| Memory["Training prototype bank"]
    Fit -->|Freeze predictor| Infer["Prototype-conditioned<br/>inference"]
    Memory -->|Freeze memory bank| Infer
    Query["Test reference and modification<br/>Separate encoding"] --> Infer
    Infer --> Output["Weight prediction and spherical interpolation<br/>Cosine ranking"]

Key Designs

1. Rank-aware labels: select weights by whether the correct target beats distractors

For a batch containing \(B\) triplets, the method uniformly samples \(K\) candidate weights from \([0,1]\). Each candidate interpolates the reference and modification vectors into a query, which is compared with every target in the batch. Each query consequently has a \(B\times K\) similarity table, with one column per weight and one row per candidate target. Sorting each column by descending similarity reveals the position of the query's correct target; smaller ranks indicate fusion directions that better avoid batch distractors. All weights attaining the best rank are acceptable candidates, and their median becomes the supervision label instead of further maximizing positive similarity. The following equivalent set expression follows the prose in Section 3.2 rather than reproducing the damaged equation layout in the extracted text:

\[ \alpha_i^*=\operatorname{median}\{\alpha_k:\operatorname{rank}_i^k=\min_{\ell}\operatorname{rank}_i^{\ell}\}. \]

This requires no manual negative labels because other targets in the batch provide competitors, but it is not an exact solution for the optimal weight over the entire gallery. Small batches or batches lacking genuinely similar negatives may produce labels that do not reflect gallery difficulty; the paper mitigates this with a relatively large batch. Rank search constructs discrete supervision rather than a differentiable ranking loss that requires back-propagation through sorting. Weights approaching \(1\) emphasize text, while those approaching \(0\) preserve reference visual content, although spherical interpolation coefficients are not ordinary linear weights.

2. Two-stage conditioned regression: stabilize the representation before predicting its optimal weights

The first stage forms fused vectors using these labels and fine-tunes the query-side Q-Former to align fused queries with their targets while separating hard negatives. The ViT and target visual branch remain frozen, so encoder training does not mean jointly updating every BLIP-2 parameter. This stage uses a bidirectional hard-negative contrastive objective with adaptive weighting of confusing negative pairs. The second stage freezes the aligned query encoder and trains only the weight predictor, avoiding supervision drift caused by changes in the feature space. This addresses instability from representation updates rather than guaranteeing identical approximate optimal weights across different batches.

The predictor receives the reference and modification vectors together with the correct target and the hardest negatives in the batch. Hard negatives are selected by similarity to the fused query formed with the label weight, aligning the conditioning information with the ranking basis of supervision. The input concatenates a reference token, a text token, the candidate conditioning tokens, and a learnable prediction token. A Transformer lets the prediction token aggregate query and competitor information, after which a three-layer MLP and sigmoid produce a scalar in \([0,1]\). Attention thus remains inside the lightweight weight predictor; the method does not eliminate all cross-token interaction. Providing the correct target during training supports conditioned regression but cannot be retained as an answer-revealing input at deployment.

3. Prototype-conditioned inference: replace unknown positive and negative targets with deployable candidate approximations

Training maintains a memory bank of target-embedding prototypes; each target identifies its most cosine-similar prototype and updates that prototype through an exponential moving average. The prototypes are frozen after training and represent the training-target distribution, not test-gallery labels or newly generated images and videos. The synthetic hard negatives are representative embedding conditions, not verified negatives for the current query. Because the interpolation weight is initially unknown at test time, prototype selection cannot start from an assumed final fused query. The method first constructs queries with multiple uniformly sampled candidate weights, retrieves the top-\(k\) prototypes for each, and selects the \(n\) prototypes retrieved most frequently across weights. Repeated prototypes represent competitive regions encountered along multiple fusion directions, reducing dependence on a single preset interpolation weight.

The selected prototypes and actual query enter the trained predictor; only after the final weight is predicted does the system perform the actual spherical interpolation and target ranking. It does not search for the weight that optimizes the correct target's test-gallery rank, making it fundamentally different from the oracle experiment. Whether prototype context retains useful structure across domains requires empirical testing; the memory bank does not automatically acquire target-domain knowledge. The main paper specifies prototype capacity and momentum, but places further sampling and capacity analysis in supplementary material absent from this cache, so unverified \(k\) and \(n\) settings are not supplied here.

A Worked Example

Figure 4 shows two actual WebVid-CoVR predictions: "make the jogger a young girl" receives \(\hat\alpha=0.612\), whereas "make it yellow" receives \(\hat\alpha=0.394\). The former specifies clearer target-subject semantics, while the latter relies more on the reference to resolve its referent, illustrating different degrees of textual reliance. At deployment, both weights come from the query and prototype conditions rather than a search that reads the correct target. The full path is separate encoding, prototype selection across candidate weights, final weight prediction, spherical interpolation, and ranked output. These are qualitative examples from the paper, not evidence that the weights offer reliable causal explanations for every modification type.

Loss & Training

Section 3.3 explicitly trains the second stage with mean squared error; the following expresses that textual definition:

\[ \mathcal{L}_{\mathrm{reg}}=\frac{1}{B}\sum_{i=1}^{B}(\hat\alpha_i-\alpha_i^*)^2. \]

The first-stage hard-negative contrastive loss uses \(\gamma=1\), \(\tau=0.07\), and \(\beta=0.5\); missing symbols in the extracted original prevent reproducing its precise expanded form. Each stage runs for 5 epochs with batch size 512 and AdamW, using a Q-Former learning rate of \(2\times10^{-5}\) and \(10^{-3}\) for other parameters. There are 101 candidate weights; the predictor has a 2-layer Transformer and a 3-layer MLP, totaling 1.63M parameters or 0.14% of the full model. Training uses 50 conditioning tokens, comprising 1 positive target and 49 hard negatives; the memory bank contains 1,024 prototypes with EMA momentum \(0.99\). The paper reports additional hard-negative processing at 0.03 ms per query, or 0.08% of total wall-clock time; these are author measurements, not measurements reproduced for this note.

Key Experimental Results

Main Results

The first table excerpts source Table 1 on page 10, evaluated on WebVid-CoVR-Test with all compared methods fine-tuned on the WebVid-CoVR training set. Training contains 1.64M triplets and testing contains 2,556 triplets; R@K is the proportion of queries whose correct target appears among the top K results, expressed as % below. CA denotes cross-attention fusion, Average denotes element-wise averaging of reference and modification embeddings, and external knowledge refers to video descriptions generated by an additional model.

Method	Backbone	External knowledge	R@1	R@5	R@10	R@50
Average	BLIP-2	None	58.69	83.88	90.49	98.21
CoVR-2, CA	BLIP-2	None	59.82	83.84	91.28	98.24
Thawakar et al., CA	BLIP	MiniGPT4	60.12	84.32	91.27	98.72
SRAIN	BLIP-2	None	61.07	84.90	91.20	98.40

SRAIN improves R@1 over CoVR-2 CA by 1.25 percentage points and over the MiniGPT4-description method by 0.95 points, but does not lead every R@10 or R@50 comparison. Image protocols also matter: Table 2 reports fine-tuned FashionIQ average R@10 of 49.26 and R@50 of 70.27; the former exceeds CoVR-2's 48.86, while the latter trails its 70.46. In Table 3 on page 12, SRAIN without CIRCO fine-tuning reaches 26.16 mAP@5, below the 30.47 of OSrCIR using external knowledge. CIRCO uses 800 test queries and a gallery of 123,403 images; mAP@K aggregates truncated average precision over relevant targets under the standard protocol and is not directly comparable to single-target R@K.

Ablation Study

The second table excerpts source Table 5 on page 14, all on WebVid-CoVR and expressed as %; frozen Average is not the same setting as trained Average in the main table. Stage 1 only uses the globally optimal fixed test weight \(\alpha=0.5\); the oracle instead selects weights using true target ranks over the entire test set and is not deployable performance.

Config	R@1	R@5	R@10	R@50
Average, frozen BLIP-2	45.66	71.71	81.30	94.80
Stage 2 only, frozen BLIP-2	48.08	73.40	83.22	95.74
End-to-end training	57.36	82.71	89.20	97.77
Stage 1 only	60.50	83.90	91.09	98.40
No conditioning tokens in training or testing	60.33	84.66	90.92	98.36
Real candidates only in training, no test memory bank	60.62	84.65	91.09	98.40
Stage 1 + oracle weights	71.95	89.95	94.41	99.10
SRAIN	61.07	84.90	91.20	98.40

The third table excerpts source Table 6 on page 15, evaluating a WebVid-CoVR-trained model on zero-shot image retrieval; the FashionIQ column averages R@10 over its three clothing categories.

Conditioning source	CIRCO mAP@5	FashionIQ average R@10
No memory bank	23.79	27.37
Random vectors	23.72	27.23
SRAIN prototype memory bank	26.16	29.42

Key Findings

• Two-stage training exceeds end-to-end training by 3.71 R@1 percentage points, but predicted weights add only 0.57 points over the best fixed weight after first-stage alignment; these gains should be distinguished.
• The prototype memory bank adds 2.37 mAP@5 points on CIRCO over no memory, whereas random vectors do not help, supporting a contribution from prototype content rather than merely extra tokens.
• The oracle's 71.95 R@1 exceeds actual performance of 61.07 by 10.88 percentage points, showing that the predictor has not realized all the potential available within the spherical interpolation space.

Highlights & Insights

• Turning retrieval performance into weight labels better matches the task than maximizing positive similarity alone. Taking the median among tied optimal candidates avoids treating an arbitrary tied solution as uniquely correct.
• Separating representation learning from weight regression addresses supervision drift caused by encoder changes. This can inform other tasks whose pseudo-labels depend on embedding geometry.
• The memory bank stores a compressed reference for the candidate distribution without requiring a large model to generate descriptions at query time. The work shows that some query-fusion complexity can be shifted into training-time supervision design.

Limitations & Future Work

• The authors mainly attribute the oracle gap to difficulty regressing extreme optimal weights; complete distribution statistics are not in the main text, and the supplementary analysis was unavailable, so this explanation is not independently verified here.
• Using only the middle reference frame may omit evidence for action order or transient-event modifications. This is an architectural inference by the reader, not a newly reported failure experiment.
• Section 3.1 states both that target-video aggregation weights depend on modification text and that these weights can be precomputed and cached. Frozen encoders permit caching frame features, but do not establish that aggregation weights for arbitrary new text can be fixed offline; the end-to-end caching strategy needs implementation or supplementary clarification.
• Figure 1 reports approximately 24-fold throughput relative to MLLM methods, but this main-text cache lacks sufficient hardware and timing breakdown to verify it, so the number should not be generalized to every deployment scenario.
• Batch labels and training-domain prototypes are approximations that may degrade with larger galleries or stronger distribution shifts. Improving hard-candidate coverage or extreme-weight prediction is a direction suggested by the results, not an experiment already completed.

Related Work & Insights

• Versus Slerp / TAT: SRAIN retains efficient spherical interpolation but moves from dataset-level fixed weights to sample-wise, rank-supervised predictions; it does not introduce a new underlying interpolation operator.
• Versus CoVR-2: both use BLIP-2, but SRAIN separately encodes the reference and modification before interpolation. Gains should first be compared within matched-backbone table entries rather than conflating all methods with different pretraining data.
• Versus CIReVL / OSrCIR: these approaches generate composed descriptions using external models, whereas SRAIN avoids that query-time step. OSrCIR remains stronger on CIRCO, making SRAIN better understood as a performance-versus-query-cost tradeoff.
• Transferable insight: retrieval fusion gates can learn how to combine inputs given candidate competition rather than only static modality preferences; transferring this idea still requires testing whether prototypes represent relevant candidates.

Rating

• Novelty: 4/5. The contribution is the combination of rank supervision, conditioned regression, and deployment-time prototype approximation rather than interpolation itself.
• Experimental Thoroughness: 4/5. Image, video, zero-shot, and component ablations are covered, but efficiency details and oracle-error analysis still depend on supplementary material.
• Writing Quality: 4/5. Supervision and two-stage motivation are clear, while target-video caching needs clarification.
• Value: 4/5. A useful design for low-latency composed retrieval with quantifiable remaining room for better weight prediction.