CFM: Language-aligned Concept Foundation Model for Vision

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/kawi19/CFM
Area: Interpretability / Multimodal VLM / Segmentation
Keywords: concept bottlenecks, local concepts, sparse autoencoders, hierarchical naming, open-vocabulary segmentation

TL;DR

CFM decomposes language-aligned visual features into sparse concepts with spatial locations, hierarchical relations, and textual names, making downstream predictions explainable and steerable while largely preserving classification, segmentation, and captioning performance.

Background & Motivation

Vision-language models such as CLIP place images and text in a shared semantic space, allowing one visual representation to support classification, open-vocabulary segmentation, and visual inputs to language models. However, proximity between a feature and a word does not identify the image region providing evidence or explain which visual details drove the decision. Existing concept bottleneck models try to decompose opaque vectors into nameable concepts, but often start from globally pooled image features and recover global associations rather than local evidence. For example, when grass and cows frequently appear together, a purported cow concept might primarily respond to the background; even correct classification would not establish that the explanation identifies the relevant object.

This gap becomes more consequential for segmentation and captioning: a global concept vector loses the correspondence between regions and concepts, making it unsuitable as a replacement for spatial visual tokens. Simply increasing the number of concepts is insufficient because broad categories, fine-grained species, and object parts can coexist without explicit relations or a naming procedure that handles all granularities. A Matryoshka sparse autoencoder encourages coarse-to-fine representations, but having multiple granularities does not automatically reveal which concept is the parent of another. If fine-grained vectors primarily encode residual information missing from coarse representations, naming them through direct textual nearest neighbors can produce words detached from the object's context.

CFM therefore addresses the spatial origin of concepts, their relationships, and the semantic reference used for naming, rather than merely improving explanatory text attached to a classifier. It retains the language-aligned space as a downstream interface so explanations arise from the concept representation used by the task instead of a separate post-hoc narrative. Core Idea: learn coarse-to-fine sparse concepts from semantically smoothed local visual features, discover parent-child relations through same-location co-occurrence, and use parent concepts to restore the context needed to name fine-grained concepts.

Method

Overall Architecture

An input image passes through a frozen CLIP-DINOiser backbone to obtain patch features that remain text-aligned while exhibiting greater local semantic consistency. Local Concept Encoding maps each patch to sparse activations over a shared dictionary, and the activations across patches form spatial concept maps. After training, Local Relation Discovery aggregates same-location co-occurrences across the dataset, and Hierarchy-aware Naming turns the dictionary and its relations into readable labels. Relations and names organize explanations; downstream tasks can directly consume concept activations or reconstructed features without first converting concept names into textual prompts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image"] --> Features["Frozen backbone<br/>CLIP-DINOiser"]
    Features --> Local["Local Concept Encoding"]
    Local -->|Post-training statistics| Relations["Local Relation Discovery"]
    Relations --> Naming["Hierarchy-aware Naming"]
    Local -->|Inference time| Tasks["Concept activations or reconstructed features<br/>Classification, segmentation, captioning"]
    Naming -.->|Explanation labels| Explain["Concept explanations and interventions"]
    Tasks --> Explain

Spatial location is not a heat map inferred backward from a global classification score; it is an index retained during concept encoding. A shared dictionary neuron can activate at different patches in different images, making concepts reusable across examples while revealing their locations in an individual image. The SAE decoder maps sparse activations linearly back to the original visual space, allowing tasks that previously consumed visual embeddings to retain that interface. Classification, segmentation, and captioning read the representation differently and should not be mistaken for a single task head requiring no training.

Key Designs

1. Local Concept Encoding: clean local features before learning coarse-to-fine concepts

Learning concepts directly from CLIP's local tokens encounters noisy features and unstable local semantics. CLIP-DINOiser uses semantic affinities reflected in DINO self-supervised features for guided pooling, allowing semantically similar patches to support one another rather than pooling the whole image into one vector. The paper describes this as semantic voting: related regions reinforce stable information and attenuate isolated noise. For efficient inference, its existing mechanism uses a small \(3\times3\) convolution trained on patch tokens to approximate DINO affinities, avoiding a separate full DINO branch on every forward pass. CFM trains its SAE on these smoothed final-layer patch features, retaining language alignment while making object parts and local attributes more likely to form stable concepts.

The SAE's affine encoder maps each patch feature to a high-dimensional concept space, and its affine decoder reconstructs the original feature. Matryoshka training requires not only the full dictionary but also several nested prefixes of neurons to reconstruct the complete input. The smallest prefix has limited capacity and tends to represent coarse information, while later neurons can contribute fine-grained residuals. This tendency is induced by the training structure, not by manually assigning every neuron a semantic level. BatchTopK permits different samples to receive different numbers of active concepts, providing more flexibility than retaining the same number for every patch. Consequently, the main paper's \(K=12\) is a sparsity setting, not evidence that every patch has exactly 12 nonzero activations at every stage.

2. Local Relation Discovery: identify parent-child candidates through directed same-location co-occurrence

Once concept maps are available, the authors ask whether a parent concept is present wherever a child concept appears. For example, turtle should activate in the corresponding local region when turtle shell activates, whereas seeing a turtle does not necessarily mean its shell is visible. This directional asymmetry is more appropriate for part relations or specializations than an undirected co-occurrence count. Let \(A_{jp}\) be the activation strength of concept \(j\) at patch \(p\), and let \(B(p)\) contain concepts exceeding the activation threshold there; Equation (2) defines:

\[ C_{ij}=\frac{\sum_{p:\,i,j\in B(p)}A_{jp}}{\sum_{p:\,j\in B(p)}A_{jp}}. \]

The denominator accumulates the active evidence for child candidate \(j\), while the numerator retains locations where parent candidate \(i\) is also active. When \(C_{uv}\geq\tau\), the method treats \(u\) as a parent candidate of \(v\), producing a directed relation graph. Weighting by child activation strength gives strong evidence more influence than weak responses. Crucially, the unit of aggregation is a patch rather than an image, reducing erroneous relations between concepts such as field and cow caused merely by sharing a scene. These relations remain observational statistics, not causal relationships, and do not guarantee a unique, error-free semantic tree.

3. Hierarchy-aware Naming: restore parent context around a fine-grained residual

Standard automatic naming searches a candidate vocabulary for the CLIP text embedding closest to an SAE decoder dictionary vector. This is natural for the coarsest Matryoshka concepts but can misinterpret later neurons, which supplement a coarse representation rather than independently encode a complete object. Before retrieving a name for a child concept, CFM therefore incorporates dictionary vectors of its discovered parents to recover the relevant semantic context. Intuitively, a direction encoding shell details is named together with turtle information instead of requiring a residual direction to correspond to a natural-language noun on its own. The authors also expand the fine-grained vocabulary so categories, parts, and attributes are not restricted to a small label set.

Naming must be distinguished from recognition: a name describes an already learned neuron and is not a new target used to supervise the SAE. Several parents can inform naming, and the relation graph can organize prediction contributions into broad concepts and their finer descendants rather than an unrelated list of words. This helps explain why concepts appear together but does not guarantee every label is accurate or every neuron has a single meaning. Operators or formatting in Equations (1), (3), and (4) are corrupted in the cache, so this note follows the surrounding prose for prefix reconstruction and parent-vector correction without guessing bias signs or exact equations. The cited appendices are absent from the cache; the exact vocabulary size, relation thresholds, and finer training settings are therefore not supplied here.

A Worked Example

Consider the seaside kite image in Figure 8: the kite, sea, and sand first produce local sparse concept activations. For segmentation, the SAE reconstructs language-aligned vectors from these activations and computes cosine similarities with the category-word embeddings supplied for the task. The base version obtains patch-level predictions and then applies bilinear upsampling; the AnyUp version first upsamples concept activations to pixels, then decodes and computes category similarities. These alternatives change the spatial reconstruction path rather than train a separate closed-category classifier on pixel annotations. For the kite segment in Figure 8, the leading concept contributions are Sport Kite 32.0%, Kite String 26.9%, and Sail Boats 5.0%. This is an explanation for one image, not overall kite accuracy; the Sail Boats contribution also illustrates that local concepts can remain confused.

Captioning uses a different interface: spatial output tokens pass through an MLP adapter into Gemma-2-2B Instruct instead of concatenating segmentation labels into a description. Figure 7 shows that setting selected concept activations to zero can remove cake content while retaining drink content in the caption, or perform the reverse intervention. The intervention changes the visual representation consumed by the language model, making it closer to changing visual evidence than editing an already generated sentence. However, the example demonstrates operability rather than guaranteeing that a targeted semantic deletion preserves all unrelated content in every scene.

Loss & Training

The main experiments use CLIP ViT-B/16 and train the SAE without concept supervision on CC12M visual features, with a default dictionary of 8192 concepts and sparsity parameter \(K=12\). The central optimization combines reconstruction from multiple nested prefixes with BatchTopK, rather than learning manually annotated concepts or segmentation masks. Language alignment comes from the pretrained backbone; unsupervised concept learning does not mean that the entire vision-language system has never used paired image-text data. Classification trains a linear head on max- and mean-pooled patch concept activations, with precise aggregation details delegated to the unavailable appendix. Captioning follows a LLaVA-style setup and fine-tunes both the adapter and language model on CC12M DreamLIP long captions. Open-vocabulary segmentation directly compares reconstructed features with category text without new pixel-level supervision over a fixed class set. Relation discovery and naming construct explanations after concept learning; they should not be depicted as traversing the training dataset again for each prediction.

Key Experimental Results

Main Results

The following selects columns from main-paper Table 1, page 13; the metric is mIoU, higher is better, and Avg averages all 8 settings in the original table rather than only the columns shown here. VOC20, C59, and ADE exclude a background prompt, while VOC includes one; the authors reran the backbone baselines for inference consistency.

Method	VOC20	C59	ADE	VOC (with background)	Avg
CLIP-DINOiser	80.8	36.0	20.5	62.2	41.4
CFM	80.7	36.5	20.7	62.2	41.3
CLIP-DINOiser + AnyUp	81.6	37.2	20.9	64.0	42.2
CFM + AnyUp	81.8	37.6	21.1	63.6	41.9

CFM trails its corresponding backbone by 0.1 average mIoU, or 0.3 with AnyUp; the evidence supports near-backbone performance rather than uniform superiority over opaque models. For classification, Figure 6 on page 11 reports 78.9 / 55.4 for the OpenAI CLIP version of CFM on ImageNet / Places365, versus 80.2 / 55.1 for the standard CLIP linear probe. For captioning, Figure 7 on page 12 reports CLIPScore of 0.72 and CAPTURE of 0.35 for both CFM and CLIP-DINOiser; RefCLIPScore is 0.72 versus 0.73. The complete captioning evaluation setup is deferred to the appendix and cannot be verified from this cache, so these results are reported as shown without broader generalization claims.

Ablation Study

The following selects columns from Table 2 on page 14; classification uses OpenCLIP CLIP ViT-B/16 and must not be conflated with the OpenAI-weight results in Figure 6 as one capacity experiment. Loc. evaluates concept localization against annotated parts on PartImageNet, while C2 measures attribution consistency in DINOv2 feature space; higher is better for both.

Dictionary size	ImageNet	Places365	OVS Avg	Part Loc.	C2
4k	78.2	55.5	41.1	43.9	0.488
8k (default)	78.6	55.6	41.3	44.3	0.465
16k	78.6	56.0	40.8	44.6	0.418

Doubling capacity does not improve every metric: moving from 8k to 16k increases Places365 by 0.4 but decreases OVS Avg by 0.5 and C2 by 0.047. This establishes a trade-off between dictionary capacity, task performance, and explanatory consistency; it does not isolate the contribution of local encoding, relation discovery, or naming individually. The paper places additional structural ablations in Appendix B, which is absent from the cache, so numerical module-removal results cannot be supplied.

Key Findings

• Figure 3 on page 7 reports PartImageNet localization of 44.3 for CFM, 32.2 for PatchSAE, and 16.4 for SALF-CBM; improved local concept quality is more pronounced than downstream accuracy gains.
• The corresponding ImageNet C2 scores are 0.465, 0.295, and 0.298, supporting the authors' consistency claim without establishing causal faithfulness.
• The human study covers 1000 concepts with 5 annotators per task; the main text reports that over 80% of annotators assign consistency scores of at least 3, which must not be rewritten as over 80% of concepts being entirely correct.

Highlights & Insights

• Moving the concept bottleneck to patches makes spatial location part of the representation itself. The same concept layer can then answer both what is present and where it is located.
• Hierarchy does more than organize a visualization: it corrects naming. Using context to interpret residual features specifically addresses the structure of Matryoshka representations.
• Reconstruction reconnects interpretable representations to the existing visual semantic space. A reader takeaway is to preserve the downstream interface and evaluate information loss before building a separate concept system for every task.

Limitations & Future Work

• The authors acknowledge imperfect naming and the lack of a unified, broadly applicable evaluation framework for unsupervised concept explanations; larger, more diverse data and VQA applications are proposed directions.
• Local conditional co-occurrence reduces image-level confounding but cannot exclude local ambiguity from textures, occlusion, or context; the graph is an empirical semantic organization, not a ground-truth ontology.
• Caption interventions are primarily illustrated qualitatively. A reader recommendation is to evaluate intervention success and preservation of unrelated content to test whether concept removal is reliably selective.
• The main paper's language about all downstream tasks exceeds the demonstrated scope: the evidence covers classification, captioning, and open-vocabulary segmentation rather than guaranteeing arbitrary visual reasoning capabilities.
• The available full-text cache lacks the appendices and contains corrupted equations; this note records readable main-paper mechanisms and results without claiming implementation-level reproduction.

Related Work & Insights

• vs DN-CBM / LF-CBM: these concept bottlenecks primarily explain image-level classification; CFM retains local concept maps for spatial tasks and language-model inputs.
• vs PatchSAE: both study local SAE concepts; CFM combines semantic smoothing, a coarse-to-fine dictionary, and relation-aware naming, with stronger localization and consistency in Figure 3.
• vs CLIP-DINOiser: the latter supplies a locally language-aligned backbone rather than a readable concept system; CFM adds concept decomposition, and Table 1 measures its associated performance cost.
• vs Matryoshka SAE: nested reconstruction first induces granularity differences, after which CFM discovers explicit relations from local activations; this is not a predefined concept tree.

Rating

• Novelty: 4/5. Integrates local concept representations, data-driven relations, and structure-aware naming into a system usable across tasks.
• Experimental Thoroughness: 4/5. The main paper covers three task families, explanation proxies, and human evaluation, but the missing appendices prevent verification of all structural ablations.
• Writing Quality: 4/5. The method is clear, although important implementation and evaluation boundaries depend on appendices and corrupted cached equations complicate reproduction.
• Value: 4/5. Useful for studying interpretable interfaces to visual foundation models, provided that names and relations are not treated as reliable causal explanations.