ECCV 2026 Multimodal VLM vision-language model personalization in-context prompt tuning dynamic token routing contextual variation memory concept separation

Personalize Your Large Vision-language Models With In-context Prompt Tuning¶

Conference: ECCV 2026
Paper: ECCV Official Page
Area: Multimodal VLM
Keywords: vision-language model personalization, in-context prompt tuning, dynamic token routing, contextual variation memory, concept separation

TL;DR¶

ICPT uses a shared Adaptive Concept Projector to compress multiple reference images into label anchors and variable-length visual prompts, with contextual debiasing and cross-concept separation enabling immediate personalization of a frozen LVLM; on LLaVA-NeXT-7B, weighted existence-recognition recall rises from MC-LLaVA's 0.811 to 0.868.

Background & Motivation¶

A general-purpose vision-language model (VLM) can recognize a person or a weapon without knowing which particular instance a user calls <sks1>. Personalization requires establishing a mapping between a new label and a visual identity, then applying that mapping to questions, relationships, and image descriptions. Putting every reference image directly into the context avoids parameter updates, but consumes many visual tokens and makes performance sensitive to how examples are organized, sometimes causing identity confusion or ignored labels.

MyVLM, Yo'LLaVA, and MC-LLaVA compress identities into soft prompts to reduce input redundancy, but commonly require vocabulary expansion and separate training for new concepts. PLVM avoids test-time training through an additional vision encoder and a prompt-generation module. This paper targets a more demanding setting: several concepts coexist, and each has multiple reference images with varying backgrounds, lighting, and viewpoints. A prompt must aggregate shared identity across images without treating incidental context as identity or collapsing similar objects into nearly identical representations.

The paper therefore learns a reusable mapping from reference images to contextual representations instead of relearning model parameters for every user. More reference images do not automatically justify more tokens, and similar concepts need not be forced into complete orthogonality. Core Idea: generate compact, dynamically prunable concept prompts with the native vision encoder, suppressing contextual variation directions and excessive cross-concept similarity during shared training so that new identities enter a frozen LVLM through forward computation alone.

Method¶

Overall Architecture¶

The input comprises labels and reference images for several concepts, a user question, and an optional query image. Adaptive Concept Projection first generates a label embedding and up to 20 visual prompt tokens per concept; Dynamic Token Routing then compresses the visual prompts before the frozen large vision-language model (LVLM) consumes them alongside the query and generates an answer.

Contextual Variation Memory and Margin-constrained Concept Separation are training-side representation regularizers, not sequential filters applied at test time. Both operate on full-capacity prompts before pruning, and concept separation also constrains label embeddings; inference retains only projection, routing, and model generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Concept labels + reference images"] --> ACP["Adaptive Concept Projection"]
    ACP --> DTR["Dynamic Token Routing"]
    ACP -.->|Training: single-image differences| CVM["Contextual Variation Memory"]
    ACP -.->|Training: full prompts and labels| MCS["Margin-constrained<br/>Concept Separation"]
    CVM -.-> Geo["Geometric loss"]
    MCS -.-> Geo
    DTR --> Model["Frozen LVLM"]
    Query["Question + optional query image"] --> Model
    Model --> Output["Personalized answer"]
    Model -.->|Training: answer supervision| Task["Answer loss"]

Key Designs¶

1. Adaptive Concept Projection: aggregate multiple reference images into one contextual identity representation

The Adaptive Concept Projector (ACP) uses neither a separate personalization vision encoder nor an auxiliary segmenter to crop the target. It takes features from the third and third-to-last layers of the LVLM's native vision encoder, concatenates shallow local detail and deeper semantic features along the channel dimension, and projects them linearly into the language model's hidden space. This is not explicit target localization: it enables subsequent learnable queries to extract identity-discriminative evidence from reference images containing distractions.

The projector combines single-layer cross-attention with a three-layer MLP. A fixed set of 21 learnable queries attends to each image's dense patch features to produce consistently shaped compressed representations. Representations of the same concept are averaged position by position across images, then passed through the shared MLP to generate one label embedding and 20 full-capacity visual prompt tokens. More images therefore do not directly increase the final context length linearly, but averaging requires the model to align different views to corresponding query slots instead of simply concatenating image summaries.

The label embedding supplies a semantic anchor, while the visual prompt supplies identity details. Textual markers such as <sks1> remain in the input, organized as <sks1> is w1 : P1, allowing the model to emit the marker with its existing vocabulary rather than permanently adding user-specific vocabulary entries. Here, w1 and P1 indicate positions occupied by continuous embeddings; users are not expected to type these letters as descriptive text.

2. Dynamic Token Routing: soft gates during training and shorter contexts during inference

Dynamic Token Routing (DTR) applies a linear scoring head and Sigmoid to every full-capacity visual token, producing a retention score between 0 and 1. Training scales each token by its score rather than deleting it immediately, allowing the generation loss to back-propagate through the gates. Inference retains tokens whose scores exceed 0.5, while ensuring at least one visual token remains for every concept. The label anchor is always retained and is never pruned.

Using the paper's notation, soft masking and the inference retention set can be summarized as:

\[ P_i=s_i\odot P_i^{\mathrm{full}},\qquad \mathcal{S}_i=\{\ell:s_{i,\ell}>\tau\},\quad \tau=0.5. \]

Multiplication broadcasts row-wise. An empty retention set still requires a fallback satisfying the at-least-one-token condition, but the main text does not specify the selection rule. Dynamic length is not obtained by explicitly calculating a visual-complexity score and looking up a budget. Instead, answer supervision and sparsity jointly teach the router which prompt tokens merit retention. Soft-mask training and discrete-pruning inference also differ, making the threshold and capacity empirical choices.

3. Contextual Variation Memory: learn directions to ignore from cross-image differences of the same identity

Contextual Variation Memory (CVM) addresses which changes in reference images should not alter identity. During training, a pair of images is randomly sampled for each eligible concept with at least two references. Their already-computed single-image ACP representations bypass multi-image averaging and independently pass through the shared MLP to produce full-capacity prompts. The difference is normalized by its Frobenius norm and inserted into a FIFO queue of capacity 128, replacing older entries as new ones arrive. This is not a test-time retrieval store of user identities.

The mechanism assumes that reference images of one concept retain stable or correlated identity components, which largely cancel under subtraction, leaving directions that predominantly reflect changes in background, lighting, or other states. The geometric regularizer suppresses Frobenius inner products between the current aggregated full-capacity prompt and stored variation directions. With both operands normalized, this inner product equals cosine similarity after vectorizing the matrices. Applying the constraint before pruning keeps prompts and memory entries directly comparable despite eventual differences in prompt length.

Theorem 1 makes a bounded claim: if the prompt is exactly orthogonal to the subspace spanned by memory, the component of query-state variation within that subspace no longer contributes to similarity error; the remaining error is bounded by the norm of the out-of-subspace residual. It does not prove that practical soft regularization attains exact orthogonality or that every new environment lies inside the learned variation subspace. Separating identity from environment through differences remains a central assumption.

4. Margin-constrained Concept Separation: distinguish related identities without erasing shared semantics

Margin-constrained Concept Separation (MCS) addresses overlap in both visual prompts and label anchors. Two concepts representing different dogs should retain shared dog semantics; forcing them to be entirely dissimilar can destroy useful prior knowledge. Allowing them to become arbitrarily similar, however, risks label substitution. The method therefore places separate similarity ceilings on normalized full-capacity visual prompts and normalized label embeddings of different concepts, applying separation pressure when the margin is exceeded.

The default margin is 0.15, permitting some positive similarity rather than pushing all concepts toward unrelated representations. CVM suppresses contextual variation directions, whereas MCS limits excessive similarity between identities: they target distinct sources of confusion. In Figure 6, constraining labels alone degrades performance more than constraining visual prompts alone, suggesting that visual-space overlap is the dominant source of confusion in these experiments, although jointly constraining both modalities remains more effective.

A Worked Example¶

Consider the three concepts in Figure 3(a): <sks1> denotes a weapon, while <sks2> and <sks3> denote two characters. Each concept's reference images are encoded and aggregated separately to obtain three sets of label anchors and full-capacity visual prompts. Each set contains at most 20 visual tokens; reference images belonging to different concepts are not averaged together.

DTR then independently prunes the three visual prompts while keeping all label anchors. The paper reports an average length of 13.6 tokens per concept, but does not give the pruning lengths for this particular example, so that mean cannot be treated as its actual token count. Given the three concept definitions and the query image, the LVLM must identify who holds what and who is fighting whom before producing a description with the user labels. The answer should connect <sks2> wielding <sks1> with fighting <sks3>, rather than merely saying that two people are fighting.

This inference path performs no gradient updates for the three new identities and constructs no online CVM regularization loss. Contextual debiasing and concept separation are learned during prior shared ACP training. Without a query image, the model can still answer appearance questions about the reference concepts, but cannot infer new scene relationships that it has not observed.

Loss & Training¶

Training freezes the LVLM and updates only ACP and DTR. The objective combines answer-token cross-entropy, geometric regularization, and a sparsity penalty on routing scores, summarized from the main-text description as:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{VQA}}+\alpha\mathcal{L}_{\mathrm{Geo}}+\beta\mathcal{L}_{\mathrm{Spa}}. \]

Answer cross-entropy averages the negative log-likelihood of target answer tokens conditioned on the question, optional query image, concept anchors, and soft-masked prompts. The geometric term combines CVM and MCS. Equation (12) is severely corrupted in the cached extraction, preventing reliable recovery of all normalization factors and penalty terms, so its exact expansion is not reconstructed here. The sparsity term averages routing scores over concepts and candidate visual tokens:

\[ \mathcal{L}_{\mathrm{Spa}}= \frac{1}{|\mathcal{C}|L_{\max}} \sum_{i\in\mathcal{C}}\sum_{\ell=1}^{L_{\max}}s_{i,\ell}. \]

Sparsity encourages fewer tokens, while answer supervision discourages solving the optimization merely by driving every score downward. LLaVA-NeXT-7B uses \(\alpha=0.3\) and \(\beta=0.5\). AdamW uses a learning rate of \(10^{-4}\) and a batch size of 8; the MLP hidden width is 2 times the LVLM embedding dimension, with GeLU activations. Experiments use two NVIDIA H200 GPUs, so the absence of test-time training should not be confused with an absence of offline training cost.

The training set contains 350 concepts with 10 reference images prepared per concept. Each training instance samples 1 to 6 references from a non-uniform count distribution. The 2000 query images contain 0 to 5 concepts each. Gemini3.1-Pro generates candidate questions, followed by two rounds of manual filtering retaining 5 questions per image; single-concept and multi-concept examples account for 35% and 60%, respectively. The main text does not explicitly identify the remaining proportion, so it should not automatically be described entirely as concept-absent negative examples.

Key Experimental Results¶

Main Results¶

The custom test set contains 200 concepts absent from training, with 1 to 6 reference images per concept. Its 300 query images contain 0 to 6 concepts, with 6 concepts representing an out-of-distribution setting relative to training. The paper reports 1050 single-concept cases and 2100 multi-concept cases, including 150 open-ended questions without query images. Besides existence recognition, multiple-choice VQA, and captioning used in training, evaluation includes open-ended VQA as a previously unused task type.

The following excerpt from Table 1 reports the paper's weighted results on LLaVA-NeXT-7B, not a common aggregate across all four metrics. Recognition uses recall, MVQA uses accuracy, OVQA uses BLEU, and captioning averages concept recall and DeBERTa-v3-large caption-embedding similarity. The columns should not be compared as if they were the same accuracy metric.

Method	Recognition	MVQA	OVQA	Captioning
GPT-4o + ICL	0.702	0.661	0.619	0.630
ICL	0.608	0.630	0.504	0.188
PLVM	0.769	0.692	0.628	0.559
MC-LLaVA	0.811	0.733	0.644	0.587
ICPT	0.868	0.791	0.702	0.654

ICPT's absolute gains over MC-LLaVA are 0.057, 0.058, 0.058, and 0.067 across the four metrics. Table 1 gives single-concept OVQA as 0.716 versus 0.689 and multi-concept OVQA as 0.697 versus 0.630, corresponding to gains of 0.027 and 0.067. The results discussion calls these single-image and multi-image settings, but the table headings actually indicate Single/Multi concept. This note follows the table headings rather than treating the numbers as an independent experiment on reference-image count.

Table 2 provides cross-backbone analysis. For compactness, the following excerpt includes only weighted recognition and MVQA for PLVM and ICPT. These results establish effectiveness after adaptation to each backbone, not direct transfer of one ACP checkpoint across backbones.

Backbone	PLVM Recognition	ICPT Recognition	PLVM MVQA	ICPT MVQA
LLaVA-NeXT-34B	0.782	0.904	0.704	0.822
InternVL3-8B	0.773	0.932	0.723	0.896
Qwen3VL-8B	0.792	0.950	0.805	0.879

Ablation Study¶

The cached extraction retains the settings and textual interpretations of Figures 4 through 6, but not reliably readable per-configuration values from the curves and bars. This table therefore records supported qualitative findings without inventing numerical drops.

Config or Analysis	Reported Finding	Evidence
Remove CVM, MCS, or geometric loss	Both single-concept and multi-concept performance suffer; gains are larger for multiple concepts	Figure 4, Section 4.3
DTR versus same-ratio random pruning and no pruning	Increasing capacity is not always beneficial; dynamic pruning helps suppress redundancy	Figure 5, Section 4.3
Label-only or visual-prompt-only MCS, margin 0.15	Label-only constraints degrade more; joint constraints work better	Figure 6, Section 4.3
Change MCS margin from 0.15 to 0	Excessive separation weakens useful shared semantic priors	Figure 6, Section 4.3

Key Findings¶

Multi-concept OVQA improves more than single-concept OVQA: 0.067 versus 0.027. This supports the goal of reducing cross-concept confusion but does not replace evaluation of identity generalization in the open world.
Average prompt length is 13.6 tokens per concept, below MC-LLaVA's fixed 16. Section 4.4 reports 12% and 18% lower full-test-set inference latency than PLVM and MC-LLaVA, respectively, without millisecond-level absolute latency available for verification here.
Table 3 reports cross-model average MMPB scores of 0.853 for ICPT, 0.824 for MC-LLaVA, and 0.814 for PLVM, supporting gains beyond the custom test set. This uses a different aggregation protocol from the individual metrics in Table 1.

Highlights & Insights¶

Separating labels from visual evidence gives the model both an outputtable name and identity features it can compare. This aligns personalized question answering more directly than compressing an undifferentiated visual representation alone.
CVM stores variation directions rather than identity prototypes, converting contextual disturbances shared across training concepts into a regularization signal. Its transferable insight is learning which changes should not alter a judgment, provided that the differences really contain less identity information.
MCS uses a finite margin rather than indiscriminately pushing concepts apart, acknowledging that distinguishable identities can share category semantics. This suits fine-grained concept expansion without requiring the entire model's knowledge to be rewritten.

Limitations & Future Work¶

Additional failure cases, data-diversity analyses, and hyperparameter ablations are assigned to the appendix, but the available cache ends with the references. Those details cannot be verified here, and this note does not claim to have checked every failure mode.
CVM assumes stable components cancel across reference images of the same identity; pose, occlusion, and target appearance changes can also enter the differences. Future work should distinguish identity changes from background changes and examine residuals in unseen environments rather than infer robustness solely from the orthogonality theorem.
Recall and BLEU do not fully characterize personalization reliability. Concept-absent false positives, similar-identity substitution, privacy risks, and confusion in larger concept collections deserve evaluation. Query images contain at most 6 concepts in the reported tests, which does not establish scalability to an unbounded personal knowledge base.
Synthetic reference images and model-generated questions followed by human filtering may introduce visual-style and annotation biases despite explicit exclusion of training-concept overlap. Real user-collected data, confidence intervals, and absolute latency would clarify deployment value.

vs MyVLM, Yo'LLaVA, and MC-LLaVA: These methods acquire identity knowledge through learned concept-specific prompts; ICPT learns a shared mapping from reference images to avoid retraining for each new concept. It still requires an offline multi-concept training set and a projector compatible with the target backbone.
vs PLVM: Both aim to generate prompts without test-time training. ICPT additionally combines the native vision encoder, multi-layer features, multi-image aggregation, dynamic routing, and explicit contextual and cross-concept constraints. Final scores alone do not attribute all gains to any single component.
vs ICL and PVIT: Direct in-context learning (ICL) trades long reference contexts for immediate adaptation, while PVIT strengthens personalized ICL capability. ICPT places adaptation primarily in continuous prompt generation. A useful next question is whether reference aggregation can detect anomalous views instead of weighting every reference image equally.

Rating¶

Novelty: 4/5. Combines multi-image prompt generation, dynamic budgets, and two geometric constraints for immediate multi-concept integration.
Experimental Thoroughness: 4/5. Covers four backbones and four task types, but the appendix is unavailable for verification, and the main text lacks uncertainty intervals and absolute latency.
Writing Quality: 3/5. Clear framework and motivation, with concept-count/image-count terminology mixed in the results discussion and corrupted cached equations limiting exact reproduction.
Value: 4/5. A reusable personalization interface for frozen LVLMs with practical insights for multi-identity questions, while large-scale real-user validation remains limited.