Caption Bottleneck Models¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/bariscagliyan/CaptionBottleneckModels
Area: Interpretability
Keywords: caption bottleneck, concept discovery, text-only classification, phrase erasure, test-time intervention

TL;DR¶

CaBM converts images into taxonomy-censored captions, recognizes classes with an independent text-only classifier, and discovers concepts from its textual evidence, removing the visual-feature bypass while achieving competitive but not uniformly leading classification results.

Background & Motivation¶

Concept Bottleneck Models (CBMs) route image classification through human-understandable attributes such as color, shape, and local texture before predicting a class. The challenge is not merely training a concept predictor, but first deciding which concepts adequately cover the dataset's discriminative information. Expert image-level annotation is expensive, so methods including LaBo, LF-CBM, and VLG-CBM use language models to propose concepts and vision-language alignment to establish supervision. However, vocabularies generated from class names can contain encyclopedic knowledge, redundant synonyms, or properties that are not visible in the image, while missing dataset-specific details. Automatically generating a vocabulary therefore does not establish that its concepts come from evidence in the current image.

A separate issue is whether the concept interface genuinely restricts information flow. Continuous concept scores can carry information beyond their intended semantics, allowing high classification accuracy even when decisions use uninterpretable signals. Hardening concepts and training stages independently can reduce this leakage but often loses fine-grained information; retaining additional image embeddings weakens the bottleneck constraint again. Rather than expanding an attribute dictionary, this paper uses discrete free-form text as the intermediate representation, preserving visual detail through richer descriptions. The structural restriction must be distinguished from caption correctness: inspectable text does not make every upstream statement reliable.

Existing Large Multimodal Models (LMMs) provide a usable image-to-caption interface, while pretrained text encoders can learn class differences expressed in those captions. The authors consequently separate perception from classification and allow the downstream model to access captions but not additional visual features. Concepts are no longer a dictionary required before classification; they are extracted after training from the caption spans that the classifier uses. This also establishes the task: interpretable image recognition and concept discovery, rather than detecting whether content is AI-generated. Core Idea: use taxonomy-censored natural language as the only interface between stages, first learn caption-based classification, and then discover open-vocabulary concepts through phrase-erasure evidence.

Method¶

Overall Architecture¶

The input is a labeled image dataset, and the outputs include image-level predictions and a per-class concept set extracted after training. A frozen LMM performs "Multi-Caption Generation and Censoring," an independent text encoder performs "Text-Only Classification and Voting," and "Evidence-Driven Concept Extraction" produces the explanatory vocabulary. Classification does not require an initial concept list; the extracted concepts are post-hoc analysis results, not an additional bottleneck that must be fed back into training. Figure 2 on page 5 separates caption generation, text classification, and concept discovery; pages 6–8 provide the method details. The diagram preserves this ordering and separates the prediction branch from the post-training analysis branch.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input image"] --> B["Multi-Caption Generation<br/>and Censoring"]
    B --> C["Text-Only Classification<br/>and Voting"]
    C --> D["Image-level prediction"]
    C -->|Post-training: correctly classified training images| E["Evidence-Driven<br/>Concept Extraction"]
    E --> F["Per-class concepts and importance"]

"Text-only" describes the downstream recognition channel, not an entire system that no longer needs to inspect images. At test time, the frozen LMM must still convert an image into captions before the classifier reads only text. This distinction matters: CaBM removes direct access to image embeddings by the classifier, but not the visual computation required for caption generation.

Key Designs¶

1. Multi-Caption Generation and Censoring: preserve visual detail while removing explicit label shortcuts

The authors first use an LLM to design a fixed prompt template from the dataset domain and class granularity, rather than enumerating class labels to generate an attribute dictionary. The template emphasizes color, shape, texture, parts, and spatial relationships while constraining the output format and discouraging taxonomic naming. It is then reused for all images instead of designing a separate manual concept inventory for each image. Frozen Qwen3-VL-2B-Instruct generates complementary captions at different sampling temperatures: lower temperatures favor common, concise descriptions, while higher temperatures increase lexical and compositional variety. These captions are textual views of the same image, not independent image samples. Different descriptions can recover attributes omitted by one caption, but the method does not guarantee that every description is complete or accurate.

Prompt constraints cannot fully prevent the model from naming a class, so deterministic taxonomy censoring follows generation. The censoring function matches class strings and higher-level category terms, replacing matches with a dataset-generic referent such as "the object." This does not require a predefined visual concept set, but it still requires knowledge of the labels or taxonomic terms to filter. Lightweight text normalization and conservative stochastic decoding controls help reduce repetition and degenerate output. This removes explicit lexical shortcuts; it does not establish that context, synonyms, or incorrect descriptions cannot introduce bias. Page 6 describes this stage, while page 9 specifies the caption counts and sampling temperatures used in experiments.

2. Text-Only Classification and Voting: make classification depend exclusively on captions

All captions inherit the label of their source image, producing caption-label samples for training RoBERTa-base and a linear classification head. After tokenization and length truncation, the final-layer classification-position representation passes through the linear head to produce class logits. Training computes a classification loss for each caption, but dataset partitioning groups samples by image, keeping all captions of an image in the same split. Otherwise, near-duplicate descriptions with different wording could enter both training and validation, inflating apparent generalization. This split-level data leakage is distinct from information leakage through a concept bottleneck. The upstream LMM remains frozen, classification gradients cannot reshape caption generation, and the downstream classifier receives neither pixels nor image embeddings.

At test time, all captions of an image are encoded separately, their class logits are averaged, and the class with the largest average is selected. This is neither majority voting over predicted labels nor averaging probabilities; retaining the logit definition matters for reproduction. Multi-caption aggregation can reduce sensitivity to wording in an individual caption, but cannot recover visual attributes omitted by every caption. The aggregation rule in Equation (5), page 7, is:

\[ \bar{\mathbf z}_i=\frac{1}{K}\sum_{k=1}^{K}\mathbf z_i^{(k)},\qquad \hat y_i=\arg\max_c\bar z_{i,c}. \]

Here, \(K\) is the image's caption count, and every \(\mathbf z_i^{(k)}\) comes from the same text classifier. Structural inspectability comes from restricting the classifier to explicit text, not from assigning a human concept name to every hidden dimension. The recognition channel is therefore constrained, but RoBERTa remains a neural network whose decisions require attribution analysis.

3. Evidence-Driven Concept Extraction: propose candidates, then test their contribution through erasure

Concept extraction uses only correctly classified training images to reduce the risk of treating incidental phrases in erroneous decisions as stable class evidence. For captions of retained images, the method differentiates the target-class logit with respect to token embeddings and combines gradients with those embeddings to obtain saliency. Special-token scores are zeroed, non-padding token scores are normalized, and subword scores are summed into word-level scores. Variable-length word spans are then proposed using their summed saliency, with generic filler terms and edge stopwords removed. Greedy non-maximum suppression retains at most \(P\) candidates per caption, allowing at most 50% span overlap. Each image thus contributes at most \(KP\) candidate phrases; \(P\) is the paper's notation, and its fixed experimental value is not specified in the main text.

Gradients only reduce the candidate-search cost; the final importance measure comes from an actual erasure test. The authors replace a phrase's tokens with padding and disable their attention mask, then measure whether the target-class logit falls, rather than relying solely on a visually striking saliency map. Equation (7), page 8, defines phrase importance as the logit decrease clipped at zero:

\[ \Delta(p)=\max\left(0,\;z_y(\tilde c)-z_y(\tilde c^{\setminus p})\right). \]

Here, \(\tilde c\) is the censored caption, \(\tilde c^{\setminus p}\) removes phrase \(p\), and \(y\) is the ground-truth class during extraction. If removing a phrase does not reduce the target score, it receives no positive importance; a substantial decrease indicates support for the current textual decision. This measures dependence on a text span, not whether the phrase describes something genuinely present in the image. The trained CaBM text encoder then embeds candidate phrases, and HDBSCAN clusters them within each class while marking outliers as noise. Post-processing based on centroid similarity, substring relations, and word overlap merges near-duplicate expressions to reduce synonymous redundancy. Each cluster is represented by the actual phrase with the highest cosine similarity to its normalized centroid, and ranked by the sum of its phrase-erasure importance values (page 8, Equations (8)–(9)). The resulting concepts are therefore literal phrases from captions, rather than names assigned to continuous latent variables through an external dictionary. Because the aggregate score is a sum rather than an occurrence-normalized mean, concept frequency can also influence the ranking.

Loss & Training¶

The objective is caption-level cross-entropy, with label smoothing of 0.1 in the experimental setup; no additional concept-supervision loss is introduced. Each image receives 5 captions at temperatures 0.7, 0.9, 1.1, 1.3, and 1.5, except ImageNet-1K, which uses 3 captions. The main text does not specify which temperatures are used for those 3 ImageNet-1K captions, so that subset should not be assumed. When no official validation split exists, 10% of training images are held out with class stratification, and every caption stays with its source image. Optimization uses AdamW with weight decay 0.05, a cosine learning-rate schedule, and 6% warm-up. The batch size is 16, gradient accumulation gives an effective batch size of 48, and the maximum sequence length is 512. Training runs for up to 30 epochs with early-stopping patience 6 and layer-wise learning-rate decay factor 0.85. Embeddings and the two lowest encoder layers remain frozen; the remaining trainable encoder layers and linear head are optimized together. Training-only caption augmentation uses word dropout probability 0.1 and short-span dropout probability 0.03, with augmentation applied with probability 0.3. Validation and testing do not use these augmentations; prediction uses caption encoding and logit aggregation, without requiring concept discovery before every prediction. These implementation settings come from Section 4.1 on page 9; compute and runtime details are deferred to supplementary material, which is not included in the local cache.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 4, page 14: test Top-1 accuracy on six datasets, all in %. Baseline results come from their respective papers, with different visual backbones, concept construction procedures, and classifier inputs; this is not a strictly controlled architecture ranking.

Method and classifier input	CIFAR-10	CIFAR-100	CUB-200	Food-101	Flowers-102	ImageNet-1K
LF-CBM, image-derived concepts	87.30	68.80	58.60	77.70	94.40	67.50
PS-CBM, image-derived concepts	89.80	72.10	70.10	83.00	97.90	74.00
HybridCBM, image-derived concepts	97.93	86.22	84.25	92.62	99.23	83.67
CaBM, caption text only	96.42	81.14	76.92	93.68	86.57	75.12

CaBM achieves 93.68% on Food-101 but 86.57% on Flowers-102, below all three selected baselines on the latter, indicating that preserving discriminative information in language is dataset-dependent. Its CUB-200 accuracy is 76.92%, still below HybridCBM's 84.25%; the paper argues for competitiveness under a constrained channel, not uniform superiority.

Ablation Study¶

The cached full text contains no conventional component-removal ablation table; the following are genuine concept-quality and controlled vocabulary-replacement analyses, not relabeled component ablations. Concept quality comes from Table 1, page 10, evaluated on CUB-200; all three metrics are raw values rather than percentages. Purity averages, across classes, the cosine similarity between a class's mean concept embedding and its class-name CLIP text embedding. Separation is the average pairwise cosine distance between class-mean concept embeddings; Semantics is the fraction of GPT-3.5 judgments associating a concept with its intended class.

Method	Purity	Separation	Semantics
VLG-CBM	0.52	0.07	0.86
HybridCBM	0.40	0.80	0.46
CaBM	0.52	0.16	0.92

CaBM's Semantics score of 0.92 exceeds VLG-CBM's 0.86, but this is not accuracy from human verification of image attributes. HybridCBM has the largest Separation; the paper notes that its orthogonality regularization directly increases inter-class distance, so this metric alone should not stand for explanation quality. Table 2, page 10, additionally fixes the VLG-CBM evaluation pipeline and replaces only its original LLM-generated concepts with CaBM-discovered concepts. The table reports Top-1 accuracy (%): ANEC-5 uses effective concept count NEC=5, while ANEC-avg averages results for NEC ∈ {5, 10, 15, 20, 25, 30}.

Dataset	Original vocabulary ANEC-5	CaBM vocabulary ANEC-5	Original vocabulary ANEC-avg	CaBM vocabulary ANEC-avg
CIFAR-10	88.55	88.87	88.63	88.97
CIFAR-100	65.73	66.13	66.48	66.66
CUB-200	75.79	76.25	75.82	76.34

This measures the downstream utility of concept vocabularies within VLG-CBM, not the accuracy of the CaBM text classifier again. Small, consistent improvements across the three datasets support discriminative vocabulary quality, but do not isolate the individual contributions of caption count, taxonomy censoring, or clustering.

Key Findings¶

Table 3, page 12, injects concepts into a test subset drawn from the 20 most frequently misclassified classes; CUB-200 improves from 41.7% to 55.5%, a gain of 13.8 percentage points. The highest-ranked concept of the known ground-truth class is injected while the classifier remains frozen; this is a label-informed intervention, not ordinary full-test accuracy or a blinded user study.
Figure 5, page 13, shows an individual prediction changing from Shiny Cowbird to Brewer Blackbird after an eye-related attribute is added. It demonstrates that editing the textual interface can affect a prediction, but does not independently establish the visual faithfulness of every extracted concept.
Vocabulary replacement controls the downstream evaluator and therefore isolates the concept list's role better than cross-backbone classification comparisons. The main text still lacks controlled ablations of caption count, temperature, censoring, and clustering choices, so their contributions cannot be ranked.

Highlights & Insights¶

The central design restricts information flow rather than merely making explanations easier to read. The classifier cannot bypass text to inspect image embeddings, so text interventions modify the input on which predictions depend.
Concept discovery follows task training, and candidates come from literal spans in captions read by the model. This reduces external-dictionary coverage constraints while preserving traceable phrase origins.
Gradient attribution and erasure have different roles: fast candidate selection versus testing an actual score change. This division can inform other textual intermediate representations, provided distribution shifts introduced by erasure are also examined.

Limitations & Future Work¶

The authors explicitly state that CaBM is not an explanation head that can be attached to an arbitrary existing vision black box (pages 14–15). It requires deploying a new LMM-to-text classification pipeline and cannot directly explain an existing visual backbone's internal representations.
Structurally, upstream omissions, hallucinations, and residual category hints can still become downstream evidence. Future work could evaluate caption-image factual consistency, but the paper reports no results for such a verification module.
Mining concepts only from correctly classified training images may reduce coverage of failures and unusual appearances; this is a methodological risk identified in this reading. Ground-truth-informed concept injection also cannot replace evaluation of real users correcting predictions without knowing the answer.
The cached main text does not provide inference costs, variance across multiple random seeds, or conventional component ablations. These tables alone therefore cannot establish generation overhead, statistical stability, or the necessity of each design choice.

vs VLG-CBM / LaBo: these methods prepare external concepts before learning their association with visual evidence; CaBM classifies through free-form captions before extracting concepts. Vocabulary replacement in Table 2 shows reuse in another concept bottleneck pipeline, but that reuse experiment does not change CaBM's own text-based recognition architecture.
vs HybridCBM / CaptionCBM: HybridCBM combines static and dynamic concepts; its CaptionCBM variant introduces captions but still relies on CLIP image features at inference (page 4). A title or representation involving captions is therefore insufficient to establish the same information-channel constraint as CaBM.
vs XBM / DN-CBM: XBM retains both image and text inputs in its classifier, whereas DN-CBM discovers latent concepts before naming them through an external vocabulary. CaBM differs by restricting classification to text and deriving concept names directly from caption spans rather than dictionary matching for latent variables.

Rating¶

Novelty: 4/5. Combines a strict textual interface with post-training concept discovery, unlike methods that only use captions to assist visual classification.
Experimental Thoroughness: 3/5. Classification, concept quality, vocabulary reuse, and interventions complement one another, but component ablations and cost evidence are insufficient.
Writing Quality: 4/5. The pipeline is clear, but "leakage-free" requires its structural qualification and is not a guarantee of semantic correctness.
Value: 4/5. Useful for studying inspectable new classification pipelines, rather than a universal explanation plugin for existing black boxes.