Skip to content

ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling

Conference: ECCV 2024
arXiv: 2408.04102
Code: None (VGARank dataset has been released)
Area: Multimodal VLM
Keywords: Attribute Recognition, Generative Retrieval, PrefixLM, Contrastive Learning, Conditional Dependency Modeling

TL;DR

This paper proposes to reformulate the visual attribute recognition problem as a sentence generation probability problem under an image-conditioned Prefix Language Model (PrefixLM). By replacing traditional "contrastive retrieval" with "generative retrieval", the model explicitly captures the conditional dependency between objects and attributes, significantly outperforming contrastive methods on both the VAW and the newly proposed VGARank datasets.

Background & Motivation

Background: Visual attribute recognition (e.g., color, material, shape) is a fundamental task in computer vision. Large-scale image-text models like CLIP have achieved grand success in zero-shot object recognition, naturally leading to the question of whether they can be applied to attribute recognition.

Limitations of Prior Work: Directly employing CLIP for attribute recognition suffers from two core issues: - Attribute information is ignored: During pre-training with contrastive learning, if the object alone is sufficient to distinguish image-text pairs, the model does not need to learn fine-grained attribute information, leading to poor performance on downstream attribute tasks. - Inability to model conditional dependency: Contrastive retrieval treats the text as an unordered whole for global alignment and cannot capture word order relationships, thus failing to correctly evaluate the plausibility of counterfactual combinations (e.g., "bell-shaped sky", "graffitied sky").

Key Challenge: Attribute recognition inherently requires understanding the co-dependency between objects and attributes, but the global alignment paradigm of contrastive learning naturally ignores this structured dependency.

Goal: How to correctly extract and utilize the conditional dependency between objects and attributes from the pre-training knowledge of large-scale vision-language models.

Key Insight: Using the autoregressive generation probability of a Prefix Language Model (PrefixLM) instead of contrastive similarity to measure the matching degree between an image and an attribute description sentence.

Core Idea: Attribute recognition = given an image, comparing the conditional probabilities of different attribute description sentences, which can be precisely calculated through the generation cross-entropy of PrefixLM.

Method

Overall Architecture

Input image \(v\) → CoCa model (comprising contrastive and generative branches) → construct a description sentence \(t^{(c)}\) for each candidate attribute category \(\rightarrow\) calculate the generative cross-entropy loss \(L^{(gen)}(v, t^{(c)})\) \(\rightarrow\) select the category with the minimum loss as the prediction.

Key Designs

  1. Image-Conditioned Prefix Language Model (PrefixLM):

    • Function: Generates text descriptions autoregressively conditional on the image.
    • Mechanism: Decomposes the text generation probability into the product of token-by-token conditional probabilities: \(p(x|v) = \prod_{i=1}^{n} p(s_i | v, s_1, \dots, s_{i-1})\) This decomposition naturally captures the sequential dependency between words in the text. Consequently, during pre-training, the model can learn the conditional co-occurrence patterns of object-attribute combinations such as "orange cat" and "blue sky".
    • Design Motivation: Compared to contrastive learning, which encodes text into a single vector for global matching, the token-by-token generation in PrefixLM preserves word order information and dependency structures, thereby distinguishing semantic differences such as "fluffy cat" and "cat is fluffy".

  2. Generative Retrieval:

    • Function: Uses generative cross-entropy instead of contrastive L2 distance to evaluate image-text matching.
    • Mechanism: For a candidate attribute category \(c\), construct a sentence \(t^{(c)}\) and calculate: \(L^{(gen)}(v, t) = -\sum_{i=1}^{N} \hat{p}(t_i) \log q_\theta(v, t_{j|j<i})\) where \(\hat{p}(t_i)\) is the one-hot encoding of the \(i\)-th token, and \(q_\theta(v, t_{j|j<i})\) is the model's predicted probability distribution for the \(i\)-th token given the image and the preceding text. The classification result is \(c = \arg\min_i L^{(gen)}(v, t^{(i)})\).
    • Design Motivation: Contrastive retrieval \(L^{(con)} = \|f(v) - g(t)\|_2\) "disorders" the text, losing the structured relationship between objects and attributes.

  3. Conditional Dependency Modeling — Sentence Template Design:

    • Function: Constructs different probabilistic graphical models during inference by engineering various sentence templates.
    • Mechanism: Four sentence templates correspond to four different conditional dependency graphs:
      • "{A}": The simplest classification, only modeling \(p(\text{A}|v)\)
      • "{O} is {A}": Similar to MLM, approximating \(p(\text{A}|v, \text{O})\)
      • "{A}{O}": Recognizes the attribute first and then validates object compatibility, modeling \(p(\text{A}|v) \cdot p(\text{O}|v, \text{A})\)
      • "{A}{O} is {A}": A hybrid model that simultaneously captures three probability factors: attribute classification \(p(\text{A}|v)\), attribute-object compatibility \(p(\text{O}|v,\text{A})\), and attribute prediction conditioned on the object \(p(\text{A}|v,\text{O})\)
    • Design Motivation: Different applications require different probabilistic modeling approaches. The proposed framework allows flexible switching during inference by simply changing the sentence template without retraining, and is thus referred to as a "meta-model".

  4. Fine-tuning Strategy:

    • Function: Lightweight fine-tuning on downstream datasets.
    • Mechanism: Introduces a learnable bias \(\mu_c\) and a scaling factor \(\sigma_c\) to convert raw generative retrieval scores into probabilities: \(p_c = \text{sigmoid}\left(-\frac{L^{(gen)}(v, t^{(c)}) - \mu_c}{\sigma_c}\right)\)
    • Design Motivation: Since the generative cross-entropy scores of different attribute categories have similar distributions (due to similar sentence lengths), learning a simple rescale parameter is sufficient to adapt to new datasets.

Loss & Training

  • Pre-training: Uses CoCa (ViT-B/16, 224×224), with joint contrastive and generative pre-training on the LAION dataset.
  • Fine-tuning: Optimizes \(p_c\) using cross-entropy loss, with a batch size of 4, learning rate linearly decaying to 0 from 1e-5, and training for 100k steps (approximately 1.8 epochs).
  • Initialization: \(\mu_c = -15.0\), \(\sigma_c = 0.5\), selected based on the empirical distribution of generative retrieval scores.
  • Trained on a single-node TPUv3 machine, with an average fine-tuning time of about 7 hours.

Key Experimental Results

Main Results

Dataset Metric Ours (Best Gen) Contrastive Retrieval (Best Con) Gain
VAW (Zero-shot) Rank↓ 56.0 ("{A}{O}is{A}") 95.1 ("{A}") -39.1
VAW (Zero-shot) mR@15↑ 35.9 32.0 +3.9
VAW (Fine-tuned) Rank↓ 10.6 12.2 -1.6
VAW (Fine-tuned) mR@15↑ 62.6 59.8 +2.8
VAW (Fine-tuned) mAP↑ 72.0 69.6 +2.4
VGARank-A (Zero-shot) Rank↓ 12.0 16.1 -4.1
VGARank-A (Zero-shot) R@1↑ 17.6 9.7 +7.9
VGARank-O (Zero-shot) Rank↓ 5.8 5.9 -0.1

Ablation Study (Sentence Template Comparison - VAW Zero-shot)

Configuration Rank↓ mR@15↑ Description
Gen "{A}" 82.1 28.7 Pure classification, no object prior
Gen "{A}{O}" 63.9 35.9 PrefixLM mode
Gen "{O}is{A}" 61.9 32.9 MLM-like mode
Gen "{A}{O}is{A}" 56.0 31.7 Hybrid model, optimal
Con "{A}" 95.1 32.0 Contrastive baseline
Con "{A}{O}" 149.8 22.4 Adding objects worsens performance

Comparison with SOTA (VAW Fine-tuned mAP)

Method Overall Head Medium Tail
SCoNE 68.3 76.5 64.8 48.0
TAP (Without in-domain pre-training) 65.4 - - -
TAP (With in-domain pre-training) 73.4 77.6 72.9 58.8
Ours "{O}is{A}" 72.0 74.9 72.0 60.6

Key Findings

  • Generative retrieval significantly outperforms contrastive retrieval in both zero-shot and fine-tuned settings, with a particularly pronounced advantage in zero-shot scenarios (Rank: 56.0 vs 95.1).
  • Adding object information (e.g., "{A}{O}") in contrastive retrieval actually leads to a substantial decline in performance, indicating that contrastive learning indeed fails to effectively utilize structured object-attribute relationships.
  • The hybrid sentence template "{A}{O} is {A}" is optimal for the attribute recognition task, but is inferior to "{A}{O}" on the object recognition task (VGARank-O).
  • The proposed method shows a particularly strong advantage in medium and tail categories, embodying the generalization capability of pre-trained knowledge.

Highlights & Insights

  • Novel perspective on probabilistic modeling: Formulating attribute recognition as inference over joint/conditional probability graphs, where different probabilistic structures are constructed on-the-fly during inference via sentence templates, is an elegant "meta-modeling" concept.
  • PrefixLM is inherently suited for structured reasoning: While contrastive learning discards word order, PrefixLM preserves dependency relationships. This insight can be extended to other vision-language tasks requiring structured understanding.
  • Lightweight fine-tuning strategy: Extremely efficient as it only learns two parameters, bias and scale, per category.
  • Outstanding performance on long-tail attributes: Leverages pre-trained knowledge rather than overfitting to dataset distributions, thereby performing better on rare attributes.

Limitations & Future Work

  • The CoCa Base model (ViT-B/16, 224 resolution) is relatively small, leaving the effects of larger models unexplored.
  • Generative retrieval requires computing a forward pass for each candidate attribute, which is inefficient when the candidate set is large.
  • Sentence templates require manual engineering; automated methods to search for optimal templates are yet to be explored.
  • Evaluated only on the attribute recognition task; this can be further extended to structured reasoning tasks such as visual relation detection and scene graph generation.
  • vs CLIP + Contrastive Retrieval: CLIP lacks sensitivity to attribute-level fine-grained information, and contrastive matching fails to model conditional dependency. This work fundamentally addresses these two issues by shifting to a generative paradigm.
  • vs SCoNE/TAP: These methods rely on in-domain training and task-specific modules. This paper focuses on extracting general knowledge from pre-training, thus offering a greater advantage in long-tail categories.
  • vs VQA methods (LXMERT/UNITER): While these methods use MLM or image-text matching, this paper demonstrates that PrefixLM can approximate MLM while possessing stronger expressiveness.

Rating

  • Novelty: ⭐⭐⭐⭐ Modeling attribute recognition as conditional probability graph inference is a highly novel perspective, and the concept of using sentence templates as meta-models is very inspiring.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Detailed comparisons of various templates across two datasets, and the introduction of a new benchmark VGARank, though experiments are restricted to a single model scale.
  • Writing Quality: ⭐⭐⭐⭐ Clear logic, rigorous derivation in the probabilistic modeling section, and intuitive illustrations.
  • Value: ⭐⭐⭐⭐ Provides a systematic answer to the question of "how to correctly extract structured knowledge from VLMs", offering inspiration for attribute recognition and the broader field of visual reasoning.