CaPCL: Caption-Preserved Continual Learning for Text-to-Image Retrieval¶

Conference: ECCV2026
Authors: Naoya Sogi, Ren Ohkubo, Takashi Shibata, Makoto Terao, Yusuke Hosoya, Takayuki Okatani
Official Paper: 5212
PDF: Full paper
Code: https://github.com/NEC-N-SOGI/CaPCL
Area: Information Retrieval & RAG
Keywords: Continual learning, text-to-image retrieval, unbalanced cross-modal shift, auxiliary captions, semantic preservation

TL;DR¶

When learning a new image-text retrieval task, CaPCL requires the model to reproduce auxiliary captions generated for current images before the update, protecting prior semantics omitted by new queries; averaged over four BLIP-2 task orders, it reduces ModX's forgetting measure FM from 0.0890 to 0.0654 while raising AA from 0.6387 to 0.6942.

Background & Motivation¶

In text-to-image retrieval, “text-to-image” means finding existing images from a query, not generating images. As user interests change, a system must learn new image-text pairs, but an image rarely has only one valid description: a street scene containing vehicles and buildings can be retrieved for either its traffic or its architecture. A change in emphasis does not mean the earlier interpretation should disappear from the representation. Ordinary fine-tuning rewards only the current query and can shift image features toward its semantic focus. Methods such as ModX and DKR preserve pre-update image-text similarities, but primarily constrain the current training pairs and their relationships, which may not cover content omitted by the new queries.

The authors call the combination of substantial visual-domain overlap and markedly changing textual queries an unbalanced cross-modal shift. This is more specific than a generic domain shift: visually unrelated tasks may interfere less with prior concepts, whereas familiar images paired with supervision emphasizing a different facet can overwrite old semantics. The paper characterizes the two modalities using image-subspace overlap and the negative log-probability of text conditioned on images. Their combination correlates more strongly with forgetting than either alone; this is an empirical association, not an established causal law of forgetting.

Rather than freezing the entire embedding space, the method asks the pre-update model to express what it already sees in the current images and preserves those descriptions as training targets. Core Idea: anchor existing semantics with auxiliary captions generated by the pre-update model, then train jointly on these captions and the new image-text task so that learning a new retrieval perspective does not erase the ability to express an earlier one.

Method¶

Overall Architecture¶

The inputs are images and query texts from the current task; the output remains a vision-language model used for similarity-based retrieval. Before entering each dataset, the current model is frozen as a snapshot and “Semantic anchor generation” produces auxiliary captions. “Caption-preservation training” then teaches the updated model both the new queries and these earlier captions. The same captions can also enter “Auxiliary-caption relational constraints” for additional contrastive learning or similarity distillation, but these are not mandatory components of base CaPCL.

The teacher's timing matters: it is the model immediately before the current task, not a permanently fixed copy of the initial pretrained model. It therefore preserves interpretations acquired through preceding tasks. The auxiliary captions also come from current-task images, without introducing a replay buffer of old images here. Which prior knowledge they cover depends on the content of those images and what the pre-update model can describe.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current images and new queries"] --> B["Semantic anchor generation"]
    B --> C["Caption-preservation training"]
    B --> D["Auxiliary-caption relational constraints<br/>Optional ACC / ACS"]
    C --> E["Joint optimization and model update"]
    D --> E
    E --> F["Retrieval model enters the next task"]

Key Designs¶

1. Semantic anchor generation: retain the pre-update interpretation of each image

Training only on new queries does not tell the model which unmentioned content should remain accessible. Before a task begins, CaPCL uses the frozen snapshot's captioning capability for one offline sampling pass over the new images. The implementation generates 5 auxiliary captions per image through nucleus sampling with \(p=0.9\). These captions need not repeat the new query; their purpose is to retain other semantic facets. Multiple samples increase the range of possible perspectives, but do not guarantee exhaustive coverage or imply that the generated captions have been fact-checked by humans.

This differs from simply having a stronger external model relabel the data: self-generated captions preserve this model's pre-update interpretation, so the objective is knowledge retention rather than maximizing caption quality alone. The paper also allows captions from Qwen2.5-VL to replace or complement self-generated captions. External captions, however, anchor the external model's understanding, which may not coincide with the retrieval model's prior knowledge. Once generated, the caption set remains fixed throughout the current task to prevent the supervision target from drifting with the model.

2. Caption-preservation training: retain expressible content without freezing the entire representation space

Each current image now has two sources of supervision: the dataset's new query and the auxiliary captions produced by the pre-update model. The new query drives adaptation through the native VLM objective. Auxiliary captions supervise the updated model through autoregressive negative log-likelihood: given the same image and the preceding caption tokens, it should still predict the subsequent tokens of its earlier description. For one auxiliary caption, the objective can be written as:

\[ \mathcal{L}_{\mathrm{cap}}(I,\hat T) = -\sum_{t=1}^{|\hat T|}\log p_{\theta}(\hat T_t\mid I,\hat T_{<t}). \]

Here \(I\) is the current image, \(\hat T\) is an auxiliary caption generated by the frozen snapshot, and \(\theta\) denotes the updated model's parameters. In practice, the loss is averaged over the captions for each image. Some operators are missing from Eq. (6) in the cached text; the expression above restores the negative-log-likelihood meaning explicitly stated in the adjacent prose rather than transcribing the equation verbatim. The constraint preserves the ability to generate particular content conditioned on an image. Features may change to accommodate a new task, provided the old captions remain expressible. This leaves more room for representation updates than directly fixing every similarity, although maintaining captions does not strictly guarantee preservation of every earlier retrieval relationship.

3. Auxiliary-caption relational constraints: make semantic anchors useful in retrieval space as well

Beyond generation targets, captions can act as additional texts for retrieval training. Auxiliary Caption Contrastive loss (ACC) treats images and their own auxiliary captions as positive pairs and uses other images and captions in the batch for a bidirectional contrastive objective, reinforcing associations between images and texts expressing prior semantics. Auxiliary Caption Similarity distillation (ACS) instead compares pre-update and updated similarity matrices over batch images and auxiliary captions. It applies temperature-scaled softmax from both image and caption perspectives and constrains the resulting distributions through KL divergence. ACC reinforces pairs; ACS preserves relational structure. Both are optional uses of the same anchors and should not be conflated with the base caption-preservation loss.

This distinction determines how the framework extends to retrieval models without an accessible captioning head: an external model can generate captions, after which ACC or ACS can be used without pretending that the retrieval model computes an autoregressive caption loss. The paper's SigLIP2 experiment uses precisely this route—external Qwen2.5-VL captions with ACC—because its captioning head is not publicly available. That experiment supports the cross-architecture usefulness of auxiliary captions, but does not directly validate base caption-preservation training on SigLIP2.

A Worked Example¶

Consider a street scene containing vehicles, buildings, and a road. Suppose earlier queries emphasized vehicles, while a new task emphasizes buildings. Before the new task, the model generates 5 captions for this current image, some of which may retain details about vehicles and the street. During training, the new-query loss encourages the new architectural association, while caption preservation requires the model to remain capable of generating its earlier vehicle-related content from the same image.

With ACC, the vehicle-related auxiliary captions also enter retrieval contrastive learning as additional positive texts. With ACS, their previous similarity relationships with images in the batch are constrained as well. Only after the task is complete does the updated model become the frozen snapshot for the next task. This is a walkthrough of the paper's motivating scenario, not a newly measured example, and it does not imply that every street image produces the same phrases.

Loss & Training¶

The base version adds caption preservation to the native VLM objective. The fully extended version also includes ACC and ACS. For readability, the additive relationships described in the paper are summarized as:

\[ \mathcal{L}_{\mathrm{base}}=\mathcal{L}_{\mathrm{VLM}}+\mathcal{L}_{\mathrm{cap}}, \qquad \mathcal{L}_{\mathrm{extended}}=\mathcal{L}_{\mathrm{VLM}}+\mathcal{L}_{\mathrm{cap}}+\mathcal{L}_{\mathrm{acc}}+\mathcal{L}_{\mathrm{acs}}. \]

The base/extended subscripts are notation introduced here to distinguish the versions. Addition signs in cached Eqs. (5) and (12) are incompletely extracted, so these relationships follow the corresponding prose. For BLIP-2, the native objective contains image-text contrastive loss and language modeling loss for generating new queries. All Q-Former parameters and the captioning head are trainable. Training uses Adam with learning rate \(10^{-4}\), 10% warmup and cosine annealing, for 10 epochs per dataset on 8 NVIDIA L40S GPUs. Contrastive learning and similarity distillation use temperature \(\tau=0.0295\).

Diagnosing unbalanced shift is not an additional training module and does not dynamically control regularization in the loss. The image-overlap metric first applies PCA to frozen BLIP-2 image features, selecting subspaces with cumulative explained variance 0.95, then measures the proportion of principal directions whose principal-angle cosine exceeds 0.95. Larger values indicate stronger subspace overlap. Textual query divergence is the average negative log-probability of query text conditioned on its image: it is length-normalized over predicted tokens for each pair, then averaged over the dataset. A larger value means the model was less likely to generate that kind of query from the image before adaptation. It is not a direct KL divergence between two text datasets. The cached paper does not include the supplementary implementation details for aligning different subspace dimensions, so none are invented here.

Key Experimental Results¶

Main Results¶

The task orders comprise Incidents1M (I), BLIP3-OCR (B), RSICD (R), TextCaps (T), and PixMo-Docs (P): IBRTP, PTRBI, BTIRP, and PRITB. After each task, evaluation covers previously seen datasets and Flickr30K, which is not trained on in this sequence. Incidents1M uses mAP; the other datasets use MRR. MRR averages the reciprocal rank of the relevant image across queries, while mAP averages per-query average precision for retrieval with multiple relevant images.

AA measures average retrieval performance during continual learning and is better when higher; FM measures degradation of prior knowledge and is better when lower. The main text cites existing definitions without specifying the complete AA/FM aggregation equations. The corresponding supplement is not in the current cache, so Flickr30K's weight in that aggregation cannot be independently established here. The overall metric is explicitly:

\[ \mathrm{HM}=\frac{2\,\mathrm{AA}(1-\mathrm{FM})}{\mathrm{AA}+1-\mathrm{FM}}. \]

The following values are the four-order averages from Table 1. All methods use the same BLIP-2 model, native losses, and trainable modules. Entries are metric values, not percentages.

Method	AA ↑	FM ↓	HM ↑
FT	0.6459	0.1241	0.7435
EWC	0.6460	0.1241	0.7436
ModX	0.6387	0.0890	0.7509
DKR	0.6300	0.1000	0.7409
C2MR	0.6300	0.0955	0.7427
CaPCL (base)	0.6942	0.0654	0.7966

Relative to ModX, the strongest baseline on the combined metric, base CaPCL increases AA by 0.0555, reduces FM by 0.0236, and increases HM by 0.0457. These are absolute metric differences. On the hardest order, PTRBI, FM is 0.0965 for CaPCL, 0.1303 for ModX, and 0.2674 for FT, illustrating the effect of task order more clearly than averages alone. Average HM is the average of the individual orders' HM values; it need not exactly equal the result of substituting average AA and FM back into the formula.

Ablation Study¶

The next table selects the three individually applied components and the fully combined configuration from Table 2. All retain the native VLM objective and are averaged across the four orders. “Only” refers to the additional constraint, not to removing the native training loss.

Additional constraint	AA ↑	FM ↓	HM ↑
Caption preservation only	0.6942	0.0654	0.7966
ACC only	0.6666	0.0929	0.7684
ACS only	0.6301	0.1401	0.7272
Caption preservation + ACC + ACS	0.6941	0.0596	0.7987

Caption preservation is the strongest individual constraint; merely adding auxiliary captions and distilling similarities does not automatically yield good results. Combining all components reduces FM from 0.0654 to 0.0596 and raises HM from 0.7966 to 0.7987 relative to the base version, but AA slightly decreases from 0.6942 to 0.6941. It would therefore be inaccurate to claim that every metric improves further.

Key Findings¶

Caption source matters: Table 3 reports HM of 0.7966, 0.7884, and 0.8001 for self-generated captions, Qwen2.5-VL captions, and their combination, respectively. Self-generation outperforming the external source alone is consistent with preserving the model's own earlier interpretation. The stronger combined result suggests complementarity, but does not establish that a stronger external captioner must always help.
In the shift analysis, the product of image overlap and textual divergence has correlation 0.59 with forgetting, compared with 0.33 and 0.45 for the individual metrics. This supports examining both modalities, but the association comes from the paper's diagnostic dataset collection and is not a precise predictor for arbitrary future data.
Auxiliary captions are not free: for 50K image-text pairs and 5 captions per image, generation takes about 35 minutes on 8 L40S GPUs, versus approximately 160 minutes for 10 training epochs. Generation is a one-time preprocessing step per dataset, but its cost must still be included in the budget.

Highlights & Insights¶

Knowledge retention is operationalized as preserving a generatable interpretation of an image, rather than only a collection of similarities. Generation supports retrieval training here; retrieval remains the task.
Separating visual overlap from textual change exposes how familiar images can still introduce forgetting. Evaluation should consider not only whether visual domains are similar, but also which facet of the images the text supervision emphasizes.
The distinction between the base mechanism and optional extensions makes the scope of each experiment clearer. In particular, successful external-caption ACC should not be presented as evidence that a model without a generation head can directly use caption negative log-likelihood.

Limitations & Future Work¶

The authors acknowledge that captioning-head quality determines which content is retained, and that this knowledge selection is implicit. They suggest explicit preservation priorities when specified knowledge is available, but do not evaluate that extension.
Current images and self-generated text cannot guarantee coverage of all prior-task semantics, and erroneous descriptions may also be preserved. The base method offers empirical forgetting mitigation, not a formal guarantee of complete retention.
The main paper covers five heterogeneous datasets and four orders but does not provide repeated-run error bars here, so very small differences should not automatically be treated as stable advantages. SigLIP2 tests only external captions with ACC, not a cross-architecture replication of the core caption-preservation loss.
The local full text contains the complete main paper and references, but not the cited supplementary material. Additional ablations, exact AA/FM aggregation details, and the controlled C-CLIP comparison are outside the verified scope of this note. The code link comes from a planned-release statement in the paper; current availability was not checked online.

vs ModX, DKR, and C2MR: These methods primarily stabilize image-text relationships or features. CaPCL adds auxiliary semantics beyond current queries and uses caption generation as its base preservation objective. Similarity distillation is not rejected altogether—it can be combined through ACS. The ablations show that both the preserved target and the combination strategy matter.
vs EWC: EWC discourages changes to important weights at the parameter level, whereas CaPCL constrains output semantics. EWC does not require a captioner; CaPCL can exploit existing multimodal generation capabilities, but inherits their coverage limits and errors.
Relation to BLIP-2 and SigLIP2: These are primarily retrieval backbones in this paper, not the proposed continual learning mechanism. BLIP-2 supports self-generation and caption preservation; the SigLIP2 experiment instead uses external captions, illustrating why a capability present during pretraining should not be assumed accessible in a public implementation.

Rating¶

Novelty: 4/5. Connecting unbalanced cross-modal shift to caption preservation addresses a specific failure mode rather than merely adding another distillation loss.
Experimental Thoroughness: 4/5. Multiple datasets, task orders, and component ablations offer substantial coverage, but the main text lacks uncertainty estimates and some details depend on an unavailable supplement.
Writing Quality: 4/5. The motivation and mechanism connect clearly, and the base and optional formulations can be distinguished; damaged equation extraction requires care when consulting the cache.
Value: 4/5. Directly relevant to continually updated image-text retrieval systems with captioning capabilities, although benefits remain constrained by caption quality and generation overhead.