Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations¶

Conference: ECCV 2026
arXiv: 2606.24716
Code: https://github.com/JonasKlotz/sae-concept-eval
Area: Interpretability / Self-Supervised Representation Learning
Keywords: Sparse Autoencoders, Interpretability Evaluation, Concept Matching, Causal Validation, Feature Disentanglement

TL;DR¶

This paper proposes an interpretability evaluation framework for SAEs based on human concept annotations. It features the FBMP algorithm supporting many-to-one matching and the TAPAScore causal validation metric based on target attribute perturbation. Furthermore, two synthetic perturbation datasets, synCUB and synCOCO, are constructed. Experiments show that existing automated evaluation metrics (FMS, MS, CKNNA) fail the sanity checks, whereas the proposed matching metrics and TAPAScore reliably distinguish trained from untrained SAEs. Additionally, increasing overcompleteness is found to degrade the quality of perturbation alignment.

Background & Motivation¶

Background: Sparse Autoencoders (SAEs) have recently been migrated from the field of language model interpretability to vision models. They are used to extract interpretable, sparse latent features from high-dimensional representations of vision encoders like CLIP and DINOv2, corresponding to concepts such as object parts, textures, and attributes. Currently, the evaluation of vision SAEs relies heavily on structural metrics (e.g., reconstruction error, sparsity) or qualitative demonstrations, lacking a systematic measurement of whether the learned SAE features truly correspond to human-understandable concepts.

Limitations of Prior Work: SAEs are known to exhibit three systematic failure modes: feature splitting (a single concept fragmented across multiple latents), feature absorption (generic features overshadowed by specialized latents, creating blind spots), and feature collapsing/composition (co-occurring concepts merged into a single latent). These modes imply that architectural design or reconstruction quality alone cannot guarantee the alignment of SAE features with human cognitive concepts. More critically, existing evaluation metrics are either structural (not directly measuring semantic correspondence) or require manual qualitative inspection (not scalable). While functional evaluation in the language domain relies on controlled intervention experiments, counterfactual data isolating changes in a single attribute are extremely scarce in computer vision.

Key Challenge: While the gold standard for evaluating SAE interpretability is human judgment, large-scale human experiments are unfeasible. An alternative approach requires dual validation through "human concept annotation + controlled intervention." However, the vision domain lacks both intervention benchmarks with dense attribute annotations and many-to-one matching methods capable of handling feature splitting.

Goal: (1) To design latent-concept matching metrics capable of handling many-to-one mappings; (2) To construct synthetic image datasets isolating single-attribute changes to support interventional evaluation; (3) To propose causal validation metrics to test whether matched latents respond in the expected direction under attribute perturbations.

Key Insight: Starting from the operational definition that "interpretability equals the semantic alignment between SAE features and human-annotated concepts," the authors argue that an evaluation framework must incorporate both statistical alignment (consistency between latent activation patterns and annotations) and causal alignment (latents responding as expected when attributes are changed), and that these two dimensions are mutually irreplaceable.

Core Idea: To use Fully Binarized Matching Pursuit (FBMP) to achieve many-to-one latent-concept alignment, and directional attribute perturbation of synthetic image pairs (TAPAScore) to validate the causality of alignment, establishing a dual-dimensional statistical and causal evaluation framework.

Method¶

Overall Architecture¶

The proposed evaluation framework consists of three components forming a "statistical matching \(\rightarrow\) causal validation" evaluation pipeline. Given an SAE trained on frozen vision encoder (CLIP or DINOv2) features, the SAE latent activations are first binarized (set to 1 if activation > 0, and 0 otherwise). The FBMP algorithm is then applied to match a subset (coalition) of latents to each human-annotated attribute, and MATCHScore is calculated to measure the statistical alignment quality. Next, on the synCUB or synCOCO synthetic datasets, TAPAScore is computed on image pairs where only one attribute is changed, verifying whether the matched latents show selective and directionally correct responses to targeted attribute perturbations. The entire framework is independent of user studies, and all metrics can be automatically computed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Images + Attribute Annotations"] --> B["SAE Inference<br/>Binarized Latent Activations"]
    B --> C["FBMP Matching<br/>Each Attribute → Latent Coalition"]
    C --> D["MATCHScore<br/>Statistical Alignment Evaluation"]
    C --> E["synCUB/synCOCO<br/>Single-Attribute Perturbed Image Pairs"]
    E --> F["Compute Signed Response<br/>δadd / δrem"]
    F --> G["TAPAScore<br/>Causal Alignment Evaluation"]
    D --> H["Comprehensive Evaluation Conclusion"]
    G --> H

Key Designs¶

1. Fully Binarized Matching Pursuit (FBMP): Supporting Many-to-One Latent-Concept Matching

To address the feature splitting problem where a single attribute is co-encoded by multiple latents, FBMP models the matching problem as the sparse reconstruction of binary signals. It treats the binary attribute annotation vector as the target signal and the binary activation vectors of all latents as candidate atoms, progressively building a complementary subset of latents through greedy sequential selection. Unlike standard Matching Pursuit, FBMP operates entirely in the binary domain. It uses the \(F_\beta\) score (rather than inner product) to select the latent that best matches the current residual, accumulates the contribution of selected latents using logical OR (\(\lor\)), and updates the residual using logical AND and NOT (\(\land \lnot\)) (i.e., removing covered positive examples from the target). After each selection step, the joint \(F_1\) score is checked; the process terminates early if no improvement is found, avoiding redundant latents.

Why binarize? The authors deliberately discard activation magnitude information. If a latent encodes multiple distinct concepts via magnitude variation, it is classified as "insufficiently disentangled" under their definition and should be penalized rather than rewarded. Experiments confirm that FBMP outperforms Non-Negative Orthogonal Matching Pursuit (NN-OMP), which retains magnitudes, under equivalent sparsity levels. The overall MATCHScore is defined as the mean of joint \(F_1\) scores across all attributes, subtracted by the baseline score of an untrained SAE to obtain \(\Delta\)MATCHScore, thereby eliminating the spurious inflation effect of dictionary size on matching scores (as larger dictionaries provide more candidate latents, increasing random correlation).

2. TAPAScore: Causal Alignment Validation Based on Target Attribute Perturbations

Statistical alignment (a high MATCHScore) does not equate to causal encoding. A latent might correlate with attribute annotations due to confounders, co-occurrence patterns, or background cues without genuinely encoding that semantic attribute. TAPAScore distinguishes correlation from causality through intervention experiments: on image pairs \((\mathbf{x}, \hat{\mathbf{x}})\) where only one attribute is changed, it checks whether the subset of latents matched to that attribute responds in the correct direction. For an added attribute, the matched latents should show enhanced activation after perturbation (\(\delta_{\text{add}} = \max_{i \in I_{\text{add}}} \hat{\mathbf{z}}^i_{\text{bin}} - \max_{i \in I_{\text{add}}} \mathbf{z}^i_{\text{bin}}\), expected to be positive); for a removed attribute, the matched latents should show weakened activation (\(\delta_{\text{rem}}\), expected to be negative). The final TAPAScore is defined as:

\[\text{TAPAScore} = \Delta_{\text{add}} - \Delta_{\text{rem}} = \frac{1}{P}\sum_p \delta_{\text{add}}^{(p)} - \frac{1}{P}\sum_p \delta_{\text{rem}}^{(p)}\]

A positive TAPAScore indicates that the matched latents exhibit selective, directionally correct responses under attribute perturbation. The authors additionally compute the \(\Delta\)stay metric (the drift of latents for unperturbed attributes) to verify that high TAPAScore values do not stem from indiscriminate latent drift (which is very low, with \(\Delta\)stay = 0.21 on synCUB and 0.12 on synCOCO).

3. Synthetic Perturbation Datasets synCUB and synCOCO: Image Pairs with Single-Attribute Changes

The premise of TAPAScore is the existence of image pairs where only one semantic attribute is altered while all other factors remain identical, which is virtually non-existent in natural images. The authors generate two synthetic benchmarks using the Flux2 image editing model: synCUB targets fine-grained attribute perturbations, starting from a 33-class subset and 45 attribute concepts of CUB-200-2011, editing a target attribute on each base image guided by a reference image (e.g., changing the breast pattern from solid to striped) while maintaining identity, pose, and background; synCOCO targets object removal, selecting a target object from MS-COCO (prioritizing the object with the least instances, or the largest area in case of ties) and instructing Flux2 to remove it with a fixed prompt while keeping the rest of the scene intact. Following generation, automated validation via a ResNet-50 classifier and manual image-by-image audit and filtering were performed. In the end, synCUB retains 2,933 pairs (from 3,063 generated), and synCOCO retains 2,534 pairs (from 9,000 generated), covering 79 out of 80 COCO classes and 43 CUB attributes.

A Complete Example¶

Taking the CUB attribute "has breast pattern: striped" as an example to walk through the complete evaluation pipeline: First, FBMP is executed on the 312-attribute annotation matrix and the SAE binary activation matrix of the CUB training set. In the first round, latent A (which might correspond to "striped texture"), matching best with the target attribute in terms of \(F_{0.5}\), is selected from all latents. In the second round, latent B (corresponding to "breast region") is selected based on the residual. After two rounds, joint \(F_1\) stops increasing, and the process terminates, yielding the coalition \(\mathcal{S} = \{A, B\}\) with a matching \(F_1\) of 0.72. Then, we take a base/edited image pair from synCUB (in the base image, the bird's breast is solid, and in the edited version, it is changed to striped with all other attributes remaining constant). We compute \(\delta_{\text{add}}\): in the base image, latent A acts 0 and B acts 0; in the edited image, A acts 1 and B acts 1, so \(\delta_{\text{add}} = 1 - 0 = 1\). Since this attribute is not removed from anything, \(\delta_{\text{rem}}\) is non-existent. Averaging over all such image pairs yields a positive TAPAScore contribution for this attribute. Ultimately, a higher overall average TAPAScore across all attributes indicates that the SAE features are not only statistically aligned with annotations but also causally encode the corresponding semantics.

Loss & Training¶

SAEs are trained on frozen CLIP ViT-L/14 and DINOv2 ViT-S/14 features using the entire training sets of CUB and COCO. Four SAE variants are compared: TopK, BatchTopK, Matryoshka, and JumpReLU, with dictionary sizes covering \(\{128, 256, 512, 1024, 2048, 4096\}\). TopK variants fix \(K=32\) (approximately equal to the average frequency of CUB attributes) and use an L2 reconstruction loss + an L1 sparsity penalty + an auxiliary loss (activating dead latents to model residuals); JumpReLU uses a learnable per-latent threshold + an L0 sparsity penalty (\(\lambda = 0.001\)); Matryoshka averages the reconstruction loss over nested intermediate reconstructions. All SAEs are trained for 50 epochs using the Adam optimizer with a learning rate of \(5\times10^{-4}\).

Key Experimental Results¶

Main Results¶

Sanity Check is a fundamental component of the experiments in this paper, testing whether various metrics can distinguish between trained SAEs, untrained SAEs, and random activations. The results are summarized below (CUB + CLIP, aggregating all dictionary sizes and SAE variants):

Evaluation Metric	Trained SAE	Untrained SAE	Random Activation	Passed Sanity Check?
MATCHScore (FBMP F1, k=3)	High (~0.35-0.50)	Significant Decrease	Significant Decrease	Yes
MATCHScore (F1, k=1)	Medium	Significant Decrease	Significant Decrease	Yes
TAPAScore	Positive (~0.10-0.25)	Close to Zero	Close to Zero	Yes
FMS	Medium	No Significant Change	No Significant Change	No (on CUB)
MS	Medium	No Significant Change	No Significant Change	No
CKNNA	Medium	Increased Instead	Decrease	No

Only the matching metrics proposed in this paper and TAPAScore reliably distinguish between these three conditions; FMS only passes on COCO, while MS and CKNNA fail entirely. CKNNA even exhibits a counter-intuitive increase on untrained SAEs, indicating that structural preservation metrics cannot measure semantic alignment.

Matching and perturbation alignment results on CUB and COCO (CLIP backbone, optimal dictionary size chosen for each SAE variant):

Dataset	SAE Variant	\(\Delta\)MATCHScore (FBMP F0.5)	TAPAScore (FBMP F0.5)	Best Dictionary Size
CUB	BatchTopK	Highest (~0.45)	Consistent with matching trend	512
CUB	Matryoshka	Second Highest (~0.40)	Consistent with matching trend	256
CUB	TopK	~0.50 (at largest dictionary size)	Decreases sharply for large dictionaries	Matching 2048 / Perturbation 128-256
CUB	JumpReLU	Increases steadily to 4096	Stable except for one-to-one	4096
COCO	BatchTopK	~0.65	Decreases at large dictionaries	512-1024
COCO	TopK	~0.70 (large dictionaries)	Decreases at large dictionaries	Matching 2048 / Perturbation 256-512
COCO	JumpReLU	~0.55 (2048+)	Decreases only at 4096	2048
COCO	Matryoshka	Overall low	Peaks at 2048	2048

Key Discovery: Overall, matching scores and TAPAScore are positively correlated at medium dictionary sizes. However, TopK on CUB exhibits a clear divergence—while the matching score continues to rise with larger dictionary sizes, the TAPAScore drops sharply. This indicates that while overcompleteness can improve statistical matching, it does not guarantee causal validity. The linear probe upper bound (dashed gray line) is far higher than the SAE matching scores in all settings, suggesting that SAEs only recover a fraction of the attribute information embedded in the representations.

Ablation Study¶

Comparison of matching strategies (FBMP variants vs. one-to-one, aggregating all dictionary sizes and SAE variants, CLIP backbone):

Matching Strategy	CUB \(\Delta\)MATCHScore	synCUB TAPAScore	COCO \(\Delta\)MATCHScore	synCOCO TAPAScore	Recommended?
F1 one-to-one (k=1)	Lowest	Significant decrease for large dictionaries	Lowest	Negatively correlated	No
FBMP F1 (k=3)	Medium	Medium	Medium	Weakly positively correlated	Feasible
FBMP F0.5 (k=3)	Highest	Highest	Highest	Highest	Yes
FBMP F0.25 (k=3)	Medium-High	Medium-High	Medium-High	Medium-High	Feasible

FBMP F0.5 achieves optimal or near-optimal matching scores and TAPAScores across all settings, and is thus recommended as the default matching strategy. \(F_{0.5}\) biases toward precision over recall, which aligns with the design logic of multi-latent coalitions—an individual latent does not need a high recall, as the coalition will combine multiple latents to cover the complete attribute.

Influence of sparsity K (TopK SAE, CLIP, d=1024):

K value	CUB \(\Delta\)MATCHScore	synCUB TAPAScore	COCO \(\Delta\)MATCHScore	synCOCO TAPAScore
8	Medium	Highest on FBMP F0.25	Medium	Medium
16	Highest	Highest	High	High
32	Highest	High	High	Highest
64	Significant Decrease	Decrease	Stable	Decrease
128	Sharp Decrease	Low & Fluctuating	Decrease	Sharp Decrease

Moderate sparsity (\(K=16\) or \(32\)) achieves the best balance between statistical and causal alignment, validating the choice of \(K=32\) as the default throughout the paper. Overly active latent sets (too large \(K\)) similarly degrade the quality of concept alignment.

Key Findings¶

The improvement in matching quality provided by FBMP is particularly pronounced with smaller dictionary sizes, as a limited pool of candidate latents heightens the need for complementary coalitions over single latents to fully cover an attribute.
TopK SAE exhibits the most pronounced "matching-TAPAScore divergence" on CUB: statistical matching improves as the dictionary size increases, whereas causal alignment collapses for larger dictionaries. This indicates that TopK SAEs with large dictionaries capture more "spurious correlations" rather than genuine semantic encoding. JumpReLU and Matryoshka exhibit milder levels of this divergence.
Overall matching scores on CUB are lower than those on COCO. This is driven by CUB's finer attribute granularity (312 fine-grained attributes vs. 80 COCO object classes), where fine-grained properties (e.g., distinguishing upper-body vs. lower-body color) present a harder challenge for SAEs.
The authors verify that TAPAScore is unaffected by indiscriminate latent drift: \(\Delta\)stay for unperturbed attributes remains low across all SAE variants (0.21 on synCUB and 0.12 on synCOCO).

Highlights & Insights¶

The binary design of FBMP is a theoretically conscious choice: The authors explicitly justify the decision to discard activation magnitude information: if a single latent encodes multiple concepts using different activation magnitudes (a form of polysemanticity), it constitutes insufficient disentanglement under their definition. Consequently, the evaluation metric should penalize rather than reward this behavior. This approach of working backward from the definition of "what constitutes good interpretability" to design metrics is highly instructive.
Dual-dimensional statistical + causal evaluation paradigm: Relying on MATCHScore alone can easily overestimate interpretability (e.g., TopK under large dictionary sizes yields high matching scores but low TAPAScores), while evaluating TAPAScore in isolation is bottlenecked by the matching quality. Combining both dimensions exposes the true performance of SAEs across different configurations far more comprehensively than any single-metric evaluation.
Building evaluation benchmarks via controlled attribute editing using Flux2: Compared to previous approaches that synthesized simple scenes through collages (e.g., the Soft Identifiability Benchmark by Fel et al.), using diffusion models for fine-grained editing generates more natural test samples that are closer to the real distribution, while keeping all confounding factors like background, pose, and identity constant. This methodology can be transferred to other vision tasks requiring counterfactual evaluation.

Limitations & Future Work¶

Framework is limited by annotation quality and granularity: Both MATCHScore and TAPAScore rely heavily on human attribute annotations. Unannotated concepts cannot be matched, and annotation noise can degrade evaluation reliability (e.g., spatially precise annotations of upper/lower-body colors in CUB may mismatch the global color representations learned by SAEs). The authors suggest utilizing VLMs to automatically generate attribute vocabularies as a future direction to mitigate dependence on human annotations.
Limited scale and diversity of synCUB/synCOCO: synCUB only covers 33 bird categories and 43 attributes, while synCOCO, despite covering 79 classes, has an average of only about 32 pairs per class and solely supports removal operations (as adding objects often leads to unrealistic images in complex scenes). Extending to more categories and attribute types (e.g., material, lighting, viewpoint) is a natural path forward.
TAPAScore only validates direction correctness, not magnitude consistency: Currently, it only checks whether latents respond in the correct direction (activation vs. suppression) but does not verify whether the magnitude of the response is proportional to the extent of the attribute change. Introducing a continuous-valued version of TAPAScore could enable finer causal evaluations.
Discrepancy in output patterns between DINOv2 and CLIP backbones: On DINOv2, the negative impact of overcompleteness on TAPAScore is milder than on CLIP, indicating that the choice of the backbone model significantly influences SAE interpretability—an aspect that is not investigated deeply in this work.

vs. Bricken et al. (Towards Monosemanticity): The pioneering work on SAEs in the LLM domain, which proposed the basic framework for using SAEs to disentangle superimposed representations. However, its evaluation mainly relied on manually inspecting top images/texts that activate specific latents. This paper advances its evaluation methodology from qualitative to quantitative, specifically addressing the bottleneck of lacking intervention benchmarks in the computer vision domain.
vs. Fel et al. (Archetypal SAE): Proposed using image collage synthesis for SAE identifiability evaluation (Soft Identifiability Benchmark), which is a pioneering work on vision SAE evaluation. However, its synthetic scenes lack visual richness and rely on discrete object collaging instead of continuous attribute editing, which limits evaluation realism. This paper's synCUB/synCOCO use diffusion models to generate more natural counterfactual samples.
vs. Pach et al. (MS Score): Proposed the MS (MonoSemanticity) score to measure the semantic consistency of latents, computed via activation-weighted image similarity. The sanity check in this paper shows that MS cannot distinguish trained from untrained SAEs, indicating that activation consistency does not imply conceptual interpretability—an untrained, random latent can still be consistently activated by semantically similar inputs.
vs. SUB Benchmark (Bader et al.): SUB constructs concept replacement samples from CUB but pairs across different classes, making it unsuitable for image-level intervention experiments. In contrast, synCUB performs attribute editing on the exact same bird, making it more suitable for paired-input intervention evaluations.
vs. Härle et al. (FMS): FMS trains classifiers to map latents to semantic attributes; while it passes the sanity check on COCO, it fails on CUB, indicating that its efficacy depends heavily on the granularity and complexity of the concept set.

Rating¶

Novelty: ⭐⭐⭐⭐ Systematically migrates the dual-dimensional paradigm of "statistical alignment + causal validation" from LLM SAE evaluation to the vision domain. The binarized matching pursuit in FBMP and the perturbation alignment design in TAPAScore are both highly original, and the synthetic datasets fill the gap in visual interventional evaluation of SAEs.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Compares 4 SAE variants, 6 dictionary sizes, 4 matching strategies, 2 backbone models, 2 datasets and their synthetic versions, along with sanity checks, \(\Delta\)stay leakage validation, sparsity sensitivity analysis, and comparisons with NN-OMP. The experimental design is comprehensive and the conclusions are rock-solid.
Writing Quality: ⭐⭐⭐⭐ The motivation is clearly articulated, and the naming and classification of the three failure modes (feature splitting, absorption, and composition) help readers build intuitive understandings. The diagram of Figure 1 is highly beneficial for understanding the interactions among the three components. The appendix provides complete pseudocode and visualization of the matching process.
Value: ⭐⭐⭐⭐ Provides a reusable, standardized evaluation protocol for the vision SAE community (recommending FBMP F0.5 + TAPAScore as the default configuration) with clear practical guidelines (finding that moderate dictionary size + moderate sparsity is the optimal balance). Furthermore, synCUB/synCOCO are public datasets that can be directly used in subsequent developments.