Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs¶

Conference: ECCV2026
Paper: ECCV official page
PDF: Full paper
Code: https://github.com/zengkaiya/CaT
Area: Multimodal VLM
Keywords: personalization, concept tree, synthetic data, hard negatives, perturbation-based filtering

TL;DR¶

CaT organizes user concepts into editable trees, generates positive, easy-negative, and hard-negative samples, and retains concept-relevant information through perturbation-based filtering; with an increased positive-sample budget, MC-LLaVA recognition accuracy on its namesake dataset rises from 0.917 to 0.963.

Background & Motivation¶

A vision-language model (VLM) may recognize the public category “cat” without knowing whether a photograph depicts a user's own pet. Personalization requires binding a user-defined identifier to a particular object and supporting recognition, selection, question answering, and captioning about it. MyVLM, Yo’LLaVA, and MC-LLaVA offer different mechanisms for injecting concepts, but their training still depends on positives and negatives: positives establish identity, while negatives specify which related-looking objects are not that identity. Users often provide only 1–3 photographs, and retrieved negatives can be dominated by background, visual style, or retrieval bias.

Simply generating more images does not resolve this problem automatically. Easy negatives belong to different categories and help establish basic recognition boundaries; hard negatives share the target category but depict different instances, more closely matching the discrimination needed for fine-grained question answering. The paper's observational experiments indicate that neither negative type can replace the other, and greater hard-negative diversity is not always beneficial. The relevant control variables are therefore which information changes and how much it changes, rather than data volume alone.

The paper replaces unrestricted text prompts as the intervention unit with a structured concept description, then checks whether image similarity actually depends on the target concept. Core Idea: control sample difficulty and diversity through category and attribute edits to concept trees, then filter concept-relevant images using the similarity drop caused by perturbation, supplying more effective training data to existing personalized VLMs.

Method¶

Overall Architecture¶

The input is a small set of user-provided concept photographs. The output is a training dataset for existing personalization methods, not a new VLM architecture. “Concept Tree Construction” summarizes photographs into a category, attribute dimensions, and attribute values; “Tree-Controlled Sample Synthesis” generates three types of training images; “PCS Perturbation Filtering” removes samples whose similarity is mainly background-driven. Selected images receive training instructions and join real reference images in personalization training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User concept photographs"] --> B["Concept Tree Construction"]
    B --> C["Tree-Controlled Sample Synthesis"]
    C --> D["PCS Perturbation Filtering"]
    D --> E["Image–instruction pairing"]
    E --> F["Personalized VLM training"]

Key Designs¶

1. Concept Tree Construction: turn object descriptions into addressable editing units

The root stores a high-level category such as cat. Intermediate nodes represent dimensions such as appearance, behavior, and location, while leaves contain specific attributes, such as fur color, sitting posture, or grass. Separating category identity from visible attributes is important: editing the root controls category differences, whereas editing a dimension changes local semantics while preserving the category. The tree is a control representation for data generation, not an additional reasoning tree inside the downstream VLM, and individual leaves are not treated as definitive identity tests.

A pretrained VLM first describes the reference photographs, and a batch of descriptions is summarized into dimensions and attribute sets. Multi-round voting then supplies feedback for self-refinement: if an image cannot repeatedly be assigned to the same attribute, the partition may be redundant or unsuitable and should be adjusted. This reduces overlapping attributes and missing descriptions, making operations such as adding or modifying a dimension more interpretable. The tree nevertheless depends on the model's understanding of limited photographs and does not guarantee recovery of every true object attribute.

2. Tree-Controlled Sample Synthesis: separate identity preservation, category changes, and within-category differences

A positive sample cannot merely depict another similar-looking cat. The paper therefore fine-tunes the image generator on user photographs before generating positives under the root-category condition. Identity information comes from reference-driven fine-tuning, not from the word “cat” itself. Negatives use a generator without that concept-specific fine-tuning. Easy negatives replace the root category and adjust the dimensions accordingly, for example changing cat to dog. Hard negatives preserve the root category but change dimensions or attributes, producing images that are close at the category level but different at the instance level. Distinguishing these generation paths prevents ordinary category members from being mistaken for positives.

For hard negatives, CaT provides three tree operations: adding, removing, and modifying. Adding a dimension such as mood expands the space of attribute combinations; removing a dimension such as scene objects contracts it; modification replaces an existing dimension or its attribute content. Both operation type and application count control generated diversity rather than merely changing wording. The paper describes diversity through distances between negative samples and their cluster centers, with the figure specifying K-means. The main text does not fully specify distance aggregation or normalization, so the reported score should not be treated as a universal metric directly comparable across implementations. The observations support moderate expansion rather than maximizing diversity unconditionally, since excessive variation can introduce noise.

Multi-concept settings combine individual trees into a forest and impose shared-scene attribute constraints to reduce attribute overlap between concepts. They reuse the sample-generation approach rather than introducing another multi-concept model. When two user objects appear together, for example, attributes should remain associated with their respective objects instead of transferring one object's description to the other. Trees provide explicit locations for such constraints, but whether the generated images satisfy them still requires filtering and empirical evaluation.

3. PCS Perturbation Filtering: distinguish object similarity from background similarity

CLIP cosine similarity between generated and reference images can admit incorrect objects against matching backgrounds. The authors distinguish concept-specific information (CS), such as object appearance, from concept-agnostic information (CA), such as background. PCS assumes that when similarity depends on intact object structure, disturbing image patches should substantially reduce it; when similarity mainly reflects background or style, the change may be smaller. The question is therefore not simply how similar an image is, but how much similarity remains after local structure is disrupted.

Let the CLIP features of the reference image, original generated image, and disturbed image be \(F_r\), \(F_o\), and \(F_d\), respectively. The main text's verbal definition can be expressed as:

\[ \operatorname{PCS}(I)=\operatorname{cos}(F_o,F_r)-\operatorname{cos}(F_d,F_r),\qquad I\text{ is retained if }\operatorname{PCS}(I)>\tau_{\mathrm{PCS}}. \]

This is an equivalent expression of similarity before perturbation minus similarity afterward; the authors associate higher scores with more concept-relevant information. The figure calls the perturbation patch shuffling, whereas the text describes mixing patches between reference and generated images. The cache does not uniquely specify the patch partition or mixing procedure and gives no numerical threshold. PCS should therefore be understood as a heuristic filtering signal, not a proven estimator that strictly separates identity from background. A high score for a negative sample also does not imply the same identity as the user's object: the intended benefit is useful concept-discriminative information, not a replacement for positive/negative labels.

A Worked Example¶

Consider 2 user-provided pet-cat photographs as an illustrative example, not an additional experiment from the paper. The root is cat, with possible appearance, behavior, and location dimensions. A concept-fine-tuned generator supplements positives of that pet; changing the root produces easy negatives such as dogs; preserving cat while modifying appearance or behavior descriptions produces same-category, different-instance hard negatives.

Each generated image is then compared with the references before and after perturbation. PCS aims to reject an image that mainly reproduces the floor and walls, while potentially retaining one with meaningful object structure. The three sample types receive training instructions appropriate to their roles and are passed to an existing personalization model. This illustrates how the tree controls candidate data; it does not mean that changing posture alone necessarily guarantees a different identity for a negative sample.

Loss & Training¶

CaT's primary contribution is on the data side, and the main text does not introduce a new loss replacing those of downstream methods. GPT-4o generates instruction text for the synthetic images, after which training follows the personalization schemes of MyVLM, Yo’LLaVA, or MC-LLaVA. Fine-tuning the positive-image generator and training the downstream VLM are distinct steps.

The experiments distinguish Real, Syn, Real+Syn, and Plus. Real uses original data; Syn uses synthetic data; Real+Syn combines 1–3 real concept images with synthetic positives to match the original positive count; Plus additionally expands the positive set. Negative counts are controlled, but Plus is not a fixed-positive-budget replacement experiment. The main paper refers baseline details, training hyperparameters, and further implementation information to an appendix. The available cache contains only the main paper and references, so it does not support specifying a learning rate, diffusion-model version, or filtering threshold.

Key Experimental Results¶

Main Results¶

The following results select the MC-LLaVA method from Tables 1–2. Rec denotes recognition accuracy; Choice-V/Choice-T denote visual/text-based multiple-choice accuracy; VQA uses BLEU; Caption uses recall. Higher is better for every metric. Entries use a 0–1 scale, and the last column is the absolute Plus-minus-Real difference, not a relative percentage. Recognition increasing from 0.917 to 0.963, for example, is a gain of 4.6 percentage points.

Evaluation set / task	Metric direction	Real	Syn	Real+Syn	Plus	Absolute gain
MC-LLaVA / Rec	Accuracy ↑	0.917	0.920	0.928	0.963	+0.046
MC-LLaVA / Choice-V	Accuracy ↑	0.890	0.892	0.890	0.928	+0.038
MC-LLaVA / VQA	BLEU ↑	0.684	0.695	0.704	0.726	+0.042
MC-LLaVA / Caption	Recall ↑	0.750	0.755	0.768	0.803	+0.053
Yo’LLaVA / Rec	Accuracy ↑	0.947	0.951	0.958	0.977	+0.030
MyVLM / Rec	Accuracy ↑	0.975	0.978	0.984	0.987	+0.012
MC-LLaVA multi-concept / Rec	Accuracy ↑	0.845	0.832	0.849	0.881	+0.036

Budget-matched Real+Syn already helps some tasks, but not every metric improves: MC-LLaVA Choice-V remains at 0.890. Multi-concept Syn has lower Rec than Real, showing that synthetic data is not a lossless replacement for real data. Strong Plus results combine generation quality with positive-set expansion.

Ablation Study¶

Table 3 compares filtering strategies for Yo’LLaVA on the MC-LLaVA evaluation. VQA is BLEU and Caption is recall; the other three columns report accuracy. Higher is better throughout.

Filtering strategy	Rec ↑	Choice-V ↑	Choice-T ↑	VQA ↑	Caption ↑
No filtering	0.835	0.793	0.691	0.633	0.697
Cosine similarity	0.862	0.816	0.716	0.652	0.726
PCS	0.885	0.845	0.738	0.682	0.754

Relative to cosine filtering, PCS increases Rec by 0.023 and VQA by 0.030. Unfiltered Rec of 0.835 is even below the original Yo’LLaVA result of 0.841. Gains do not arise automatically from adding generated images.

The tree-editing analysis in Table 4 further examines hard negatives. The table below retains three strengths of dimension addition. Diversity is a cluster-distance-based descriptor and is not assigned a “higher is better” direction.

Tree editing	Count	Diversity	Rec ↑	VQA ↑	Caption ↑
No editing	0	0.497	0.841	0.643	0.701
Add dimension	1	0.563	0.852	0.655	0.723
Add dimension	2	0.642	0.866	0.674	0.736
Add dimension	3	0.708	0.834	0.654	0.717

Key Findings¶

Increasing dimension additions from 2 to 3 raises diversity but reduces Rec from 0.866 to 0.834, directly contradicting the assumption that richer data is necessarily better.
PCS outperforms ordinary cosine filtering on all five reported tasks, supporting the importance of the source of similarity rather than its magnitude alone. These results do not establish strict feature disentanglement.
The final row of Table 5 reports VQA of 0.680, whereas the corresponding main-result and filtering rows report 0.682. This note preserves the respective table values rather than silently reconciling the discrepancy.

Highlights & Insights¶

The tree makes hard-negative diversity an actionable variable. Separating categories, dimensions, and attributes lets experiments discuss explicit interventions rather than only contrasting unrestricted prompts.
PCS extracts additional filtering information from a candidate image's response to perturbation. It targets background-inflated CLIP similarity without requiring a separately retrained filtering VLM for every user identifier.
The data pipeline is separate from downstream concept-injection methods. Several existing methods benefit, so the contribution primarily concerns training-data design rather than a new personalized model architecture.

Limitations & Future Work¶

The authors identify potential distribution shift in purely synthetic data, and multi-concept results also demonstrate the continued value of real examples. Requiring few reference images does not mean eliminating dependence on real images.
PCS assumes that perturbation predominantly disrupts concept information. The main paper lacks sufficient breakdowns for identities defined by local textures or objects strongly coupled to their backgrounds, so filtering reliability requires caution.
The main text does not report full generation and filtering costs or all required reproduction details. Plus also expands the positive budget, leaving compute-matched conclusions and threshold sensitivity insufficiently resolved.

vs MyVLM, Yo’LLaVA, and MC-LLaVA: These methods address how to inject user concepts into a VLM; CaT addresses which training samples to supply. They are complementary rather than alternatives requiring a model replacement.
vs SSDLLM: CaT draws on discovering structure from data descriptions but uses dimensions and attributes to control concept-centric generation, focusing on positive/negative difficulty and diversity.
vs ordinary CLIP filtering: Conventional filtering selects by original image similarity; PCS examines the change under perturbation. The additional signal targets similarity caused by irrelevant scene content but remains an empirical quality proxy.

Rating¶

Novelty: 4/5. Concept-tree control and perturbation-based filtering form a clear combination tailored to personalized data construction.
Experimental Thoroughness: 4/5. Multiple methods, datasets, and multi-concept settings are covered, with filtering and tree-editing ablations, though budget and reproduction details remain limited.
Writing Quality: 3/5. The motivation is clear, but perturbation details and a small number of table values require verification.
Value: 4/5. The work offers practical guidance for few-shot personalized VLM data construction, without establishing unconditional replacement of real data.