Compositional Caching for Training-free Open-vocabulary Attribute Detection

Conference: CVPR 2025
arXiv: 2503.19145
Code: Yes (project page)
Area: Multimodal VLMs
Keywords: Attribute Detection, Open-vocabulary, Training-free method, Vision-Language Model, Caching mechanism

TL;DR

ComCa proposes a training-free open-vocabulary attribute detection method. By leveraging web-scaled image databases and an LLM, the method constructs an auxiliary image cache labeled with soft attribute probabilities. During inference, it aggregates the similarities of cached images to enhance the VLM's attribute prediction capabilities, competing effectively with training-based methods without any additional training.

Background & Motivation

Background: Attribute detection (e.g., identifying visual properties like color, texture, and material of objects) is a fundamental computer vision task crucial for downstream applications like image understanding, text-to-image retrieval, and visual question answering. Current approaches heavily rely on large-scale human-annotated attribute-object pairs to perform classification within a fixed set of attributes.

Limitations of Prior Work: (1) The annotation process is extremely time-consuming and prone to ambiguity, as describing an object's attributes can occur at arbitrary granularities (e.g., "red" vs. "scarlet" vs. "dark red"), leading to annotator inconsistency. (2) Methods based on fixed attribute sets fail to generalize to new attributes or domains, resulting in poor scalability. (3) Existing training-based approaches require fine-tuning for specific datasets and attribute sets, lacking the flexibility to adapt to different downstream applications.

Key Challenge: The compositional nature of attributes (the same attribute manifests differently on different objects, e.g., "shiny metal" vs. "shiny skin") makes simple attribute classification difficult, and the open-vocabulary setting further requires the model to handle unseen attribute categories during training.

Goal: Design a completely training-free open-vocabulary attribute detection method that works simply given a target list of attributes and objects.

Key Insight: It is observed that although VLMs (like CLIP) exhibit strong vision-language alignment capabilities, they perform poorly when directly applied to attribute detection because attributes are highly fine-grained and context-dependent. The authors suggest that external images can serve as "reference cases" to calibrate the VLM's predictions.

Core Idea: Build a compositional cache (Compositional Cache) utilizing web-scale image knowledge and LLMs to determine attribute-object compatibility. This assigns soft attribute labels to each cache image. At inference, a similarity-weighted aggregation of these cached soft labels is used to enhance the VLM's zero-shot prediction.

Method

Overall Architecture

The pipeline of ComCa consists of two phases: (1) Cache construction phase — given a target list of attributes and objects, a web engine searches for reference images for each attribute-object composition, and an LLM determines which attribute-object combinations are semantically valid to assign soft attribute labels to each image; (2) Inference phase — for a query image, its visual feature similarities with cached images are calculated, and the weighted aggregation of soft labels in the cache is used to refine the VLM's zero-shot prediction.

Key Designs

1. LLM-based Attribute-Object Compatibility Filtering:

   • Function: Automatically determine which attribute-object combinations are semantically plausible.
   • Mechanism: Given an attribute list \(\{a_1, ..., a_M\}\) and an object list \(\{o_1, ..., o_N\}\), an LLM is prompted to judge whether each \((a_i, o_j)\) pair is visually plausible. For example, "wooden car" is unlikely to be valid, whereas "wooden table" is highly reasonable. This compatibility info is used to filter out implausible cache entries and avoid introducing noise.
   • Design Motivation: Directly enumerating all attribute-object combinations would generate a massive number of invalid entries, which wastes storage and introduces incorrect soft labels. Utilizing the world knowledge of LLMs enables efficient filtering without requiring manual annotations.

2. Soft Attribute Label Assignment:

   • Function: Assign continuous labels reflecting attribute probabilities for each cached image.
   • Mechanism: Unlike hard labels (0/1), ComCa computes a soft probability representing each attribute for every cached image. Specifically, the VLM is used to compute the similarity scores between each cached image and the text descriptions of all attributes. These scores are normalized using softmax to produce soft label vectors. This process accounts for the compositional nature of attributes — a single "red car" image might also carry a minor "glossy" attribute besides the "red" attribute.
   • Design Motivation: Hard labels fail to reflect the ambiguity and co-occurrence of attributes, whereas soft labels enable the model to better capture subtle differences and overlaps between attributes.

3. Similarity-weighted Aggregation for Inference:

   • Function: Enhance the VLM's attribute prediction during inference by leveraging cached images.
   • Mechanism: Given a test image, its features are first extracted via the VLM's visual encoder, and the cosine similarity with all cached image features is computed. The top-\(K\) most similar cached images are retrieved. Their soft attribute labels are then weighted by similarity and summed to obtain the cache-based attribute prediction. The final prediction is a weighted combination of the VLM's zero-shot prediction and the cache aggregation result, where the weight is a hyperparameter \(\alpha\).
   • Design Motivation: Similar to the caching ideas in methods like Tip-Adapter, ComCa specially designs a soft-labeling mechanism tailored for the compositionality of attribute detection, allowing for more fine-grained utilization of cached information.

Loss & Training

ComCa is completely training-free. All components (VLM encoders, LLM, cached image search) leverage existing off-the-shelf pre-trained models, making the entire method plug-and-play during inference.

Key Experimental Results

Main Results

Dataset	Metric	ComCa (SigLIP)	Zero-shot CLIP	Tip-Adapter	Training-based SOTA	Gain (vs ZS)
VAW	mAP	~38.5	~28.2	~33.1	~40.2	+10.3
OVAD	mAP	~32.7	~22.8	~28.5	~34.1	+9.9
LSA	mAP	~45.3	~35.6	~40.2	~47.8	+9.7

Ablation Study

Configuration	mAP (VAW)	Description
Full ComCa	~38.5	Complete model
w/o LLM Compatibility Filtering	~35.8	No filtering for implausible combinations, leading to increased noise
w/o Soft Labels (using hard labels)	~36.2	Soft labels capture the compositionality of attributes better than hard labels
w/o Cache (pure VLM zero-shot)	~28.2	Caching mechanism contributes the most
Different VLM backbones	~32-38	Method is robust to different VLM selections

Key Findings

• The caching mechanism is the primary source of performance gains, improving by approximately 10 mAP points relative to the zero-shot baseline.
• LLM compatibility filtering effectively mitigates interference from noisy cached images, contributing about 2-3 mAP points.
• Soft labels are better suited for attribute detection tasks than hard labels due to the inherent ambiguity and co-occurrence of attributes.
• As a training-free method, ComCa's gap from training-based state-of-the-art is only about 1-2 mAP on some datasets, demonstrating the strong potential of the caching strategy.
• The approach is effective across various VLM backbones (CLIP, SigLIP, OpenCLIP), validating its model-agnostic nature.

Highlights & Insights

• Compositional Cache Design: The core challenge of attribute detection, namely "attribute-object compositionality," is explicitly modeled as soft labels, which is more refined than simple hard-labeled caching. This design philosophy can be migrated to other tasks requiring fine-grained attribute handling.
• LLM as a Knowledge Source: Utilizing the world knowledge of LLMs to judge attribute-object compatibility avoids the tedious task of manually defining compatibility relationships. The role of LLMs as structured knowledge providers has broad prospects in visual tasks.
• Training-free and Plug-and-play: It requires no training phase and does not require access to target dataset annotations, significantly lowering the barrier to real-world deployment. This paradigm is particularly valuable for rapid adaptation to new domains.

Limitations & Future Work

• Cache quality is heavily dependent on web image search results; acquiring high-quality reference images might be difficult in specialized domains (e.g., texture attributes in medical imaging).
• When the attribute list is very large (hundreds of attributes), the cache size and retrieval overhead during inference may become a bottleneck.
• LLM compatibility judgments are not completely reliable and might misjudge unconventional or creative attribute-object combinations (e.g., "fluffy car," which could be valid in modified car scenes).
• Soft label assignment depends on the representation quality of the VLM itself, potentially cascading errors when the VLM understands certain attributes poorly.
• It lacks specialized optimization for highly fine-grained attributes (e.g., color shades, texture density) where cached images might not be highly discriminative.

Related Work & Insights

• vs Tip-Adapter: Tip-Adapter builds a few-shot cache to enhance CLIP classification, but it uses hard labels and lacks attribute-object compatibility modeling. ComCa specializes the caching approach for attribute detection, achieving significant improvements through soft labels and LLM filtering.
• vs OvarNet / OvAD: These training-based methods learn attribute detection by fine-tuning on large-scale datasets, offering higher accuracy but requiring training. ComCa achieves comparable performance under zero-training conditions.
• The cache-and-aggregate inference paradigm is worth exploring in other fine-grained visual tasks (e.g., material recognition, style classification).

Rating

• Novelty: ⭐⭐⭐⭐ Cleverly combines caching mechanisms with attribute compositionality, and the soft label design is highly creative.
• Experimental Thoroughness: ⭐⭐⭐⭐ Validated across multiple datasets with comprehensive ablation studies; validation on different VLM backbones enhances credibility.
• Writing Quality: ⭐⭐⭐⭐ Clear problem motivation and fluent method description.
• Value: ⭐⭐⭐⭐ The training-free and open-vocabulary settings are highly practical, offering a new paradigm for attribute detection.

Related Papers

• [ICCV 2025] Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection
• [CVPR 2026] Vocabulary Scaling Law: Tuning Open-vocabulary Predictors for Their Openness
• [AAAI 2026] O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model
• [ECCV 2024] MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection
• [ICLR 2026] Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach