SAEExplainer: Interpreting SAE Features with Activation-Guided Preference Optimization¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/he-jingyi/SAEExplainer
Area: Interpretability
Keywords: Sparse Autoencoders (SAEs), Feature Explanation, Direct Preference Optimization (DPO), Activation-Guided Reward, Mechanistic Interpretability
TL;DR¶
Addressing the pervasive issues of over-generalization and explanation hallucinations caused by open-loop paradigms in Sparse Autoencoder (SAE) interpretation, SAEExplainer establishes a closed-loop framework that operationalizes objective model activation scores into verifiable rewards, achieving continuous self-refinement and superior causal triggering through two rounds of iterative Direct Preference Optimization (DPO).
Background & Motivation¶
In recent years, Sparse Autoencoders (SAEs) have emerged as an indispensable instrument within mechanistic interpretability research for large language models. By decomposing dense, polysemantic internal activations into ultra-high-dimensional sparse representations, SAEs effectively isolate monosemantic conceptual vectors, unlocking unprecedented avenues for understanding and steering LLM computations. Nevertheless, an isolated SAE feature vector is essentially a continuous latent representation that lacks inherent natural language articulation. To translate these representations into actionable human knowledge, automated feature interpretation has become paramount. Current methodologies generally divide into two streams: prompting an external LLM to heuristically summarize top-activating text snippets (exemplified by OpenAI's AutoInterp and Neuronpedia), or training an explainer model to directly translate injected activation vectors into natural language descriptions (such as LatentQA and Activation Oracles).
Despite initial progress, existing systems universally operate within a vulnerable open-loop, single-pass generation paradigm. Because they lack any mechanistic feedback verification loop, models cannot distinguish between fluent, plausible-sounding descriptions and mechanistically faithful ones. Consequently, generated explanations are heavily plagued by explanation hallucinations and over-generalization—for instance, assigning a broad umbrella summary like "web development instructions" to a highly specific feature that only fires on Dart package:flutter and Zone asynchronous calls. Crucially, although downstream evaluation metrics exist, their scores have been treated merely as passive performance readouts rather than active learning signals that optimize the explainer itself. This structural absence of feedback caps the performance ceiling and prevents explainer models from continually bootstrapping their capabilities.
To break free from open-loop constraints, this work asks whether internal activation dynamics can be internalized as direct supervision for explanation refinement. The foundational premise is that an authentic explanation must reliably trigger the target feature when generating new text; failure to elicit activations indicates an invalid, hallucinated, or overly loose hypothesis. Core idea: turn the target model's objective SAE activation scores into verifiable preference rewards, construct high-contrast preference pairs by pairing high-activating candidates with plausible yet inactive hard negatives, and drive continuous self-refinement via iterative Direct Preference Optimization (DPO).
Method¶
Overall Architecture¶
The objective of SAEExplainer is to construct an end-to-end explainer model capable of directly ingesting SAE feature vectors and generating causally faithful natural language explanations. The workflow proceeds through three stages: first, feature vectors are injected into the explainer's residual stream via norm-matched scaling, followed by a lightweight Supervised Fine-Tuning (SFT) phase to establish foundational feature-decoding capabilities; second, the explainer samples multiple candidate hypotheses per feature at high temperature, synthesizes representative text samples, and calculates real activation scores on the target model to assemble high-contrast preference pairs; finally, the system executes two sequential rounds of Direct Preference Optimization (Iterative DPO), using the refined intermediate policy to synthesize a cleaner, higher-standard preference dataset that progressively sharpens causal boundaries.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Target Model SAE Decoded Feature Vector $v_i$"] --> B["Activation Injection & Explainer Initialization<br/>Residual stream norm-matching injection + SFT warm-up"]
B --> C["Mechanistic Preference Pair Construction<br/>Candidate generation + text synthesis + activation scoring & hard negative mining"]
C --> D["Iterative Direct Preference Optimization<br/>Round-1 policy alignment + Round-2 on-policy data bootstrapping"]
D --> E["Output: Causally Grounded Natural Language Explanation $e_i$"]
Key Designs¶
1. Activation Injection & Explainer Initialization: Modality Mismatch & Unaligned Latents -> Residual Norm Matching & SFT Warm-Up -> Foundational Decoding Capability
Feeding continuous high-dimensional SAE feature vectors directly as token embeddings induces catastrophic modality collapse and disrupts early self-attention computation. To enable the explainer to natively comprehend the geometric semantics of latent features, SAEExplainer adopts a residual stream injection mechanism. A dedicated placeholder token ⟨ACT⟩ is embedded in the instruction template at sequence position \(p\). During the forward pass, after the second Transformer layer (Layer 1), the original hidden state \(h_p\) is modulated by adding the \(\ell_2\)-norm-matched feature vector:
$\(h'_p = h_p + \frac{\|h_p\|}{\|v_i\|} v_i\)$
where \(v_i \in \mathbb{R}^{d_{\text{model}}}\) is the \(i\)-th column of the target SAE decoder weight matrix. This norm-matching operation ensures that the injected perturbation aligns with the host network's intrinsic dynamic range, allowing \(h'_p\) to propagate seamlessly through subsequent layers for causal decoding. Subsequently, on a subset \(V_1\) of sampled features, the model is warmed up via Supervised Fine-Tuning (SFT) using existing baseline annotations (such as Neuronpedia) as targets. This phase bridges the initial format gap and establishes essential instruction-following competence prior to preference alignment.
2. Mechanistic Preference Pair Construction: Open-Loop Hallucinations & Plausible Fakes -> Objective Activation Scoring & Relative-Margin Hard Negative Mining -> High-Contrast Mechanistic Preference Data SFT alone produces plausible but causally ungrounded explanations that models cannot self-correct. To supply discriminative gradients, SAEExplainer implements an automated feedback pipeline on a disjoint set of unseen features \(V_2\). For each feature \(v_j\), the model samples \(M\) candidate explanations \(E_j = \{e_m^j\}_{m=1}^M\) under a high temperature (\(T=0.9\)). A generator LLM then synthesizes \(W\) representative short texts \(T_m^j = \{t_1, \dots, t_W\}\) for each candidate. Feeding these texts through the target LLM, the objective mechanistic score \(s(e_m^j)\) is defined as the maximum SAE feature activation achieved across all tokens: $\(s(e_m^j) = \max_{t \in T_m^j} \max_{\text{token } \tau \in t} a_t(\tau)\)$ Candidates undergo dual quality validation: the maximum score must meet threshold \(\tau_{\text{act}}\), and the relative variance must exceed \(\tau_{\text{diff}}\). For each valid feature, the highest-scoring candidate is designated as the chosen sample \(y_w^{(j)} = \arg\max_{e \in E_j} s(e)\). Crucially, instead of pairing it with a trivial, low-quality candidate that provides minimal learning signal, the method identifies a subset of rejected candidates \(E_j^- = \{e \in E_j \mid \frac{s(y_w^{(j)}) - s(e)}{s(y_w^{(j)})} \ge \tau_{\text{margin}}\}\) whose activation drops by at least a relative margin, and selects the highest-scoring candidate within this subset as the rejected sample \(y_l^{(j)}\). This yields a challenging "hard negative"—a description that sounds deceptively reasonable and semantically adjacent to the true concept, but strictly fails to produce robust activation—compelling the model to learn sharp semantic boundaries.
3. Iterative Direct Preference Optimization: Single-Pass Optimization Ceilings -> Sequential Two-Round Alignment & On-Policy Bootstrapping -> Progressive Causal Trigger Convergence A single round of DPO remains constrained by the candidate distribution of the initial SFT checkpoint. To transcend this baseline ceiling, SAEExplainer executes two sequential rounds of preference optimization. In Round 1, the SFT model serves as both the initialization policy and the reference model \(\pi_{\text{ref}} = \pi_{\text{sft}}\), training on the first preference dataset \(D_{\text{DPO}}^{(1)}\) to produce an intermediate model \(\pi_{\text{DPO1}}\). This intermediate policy already exhibits vastly improved precision. Leveraging this heightened competence, \(\pi_{\text{DPO1}}\) is used to re-sample candidate explanations, synthesize new texts, and evaluate activations on the exact same feature pool, yielding a refreshed on-policy dataset \(D_{\text{DPO}}^{(2)}\). Round 2 DPO then sets \(\pi_{\text{ref}} = \pi_{\text{DPO1}}\) and fine-tunes on \(D_{\text{DPO}}^{(2)}\) to yield the final model \(\pi_{\text{DPO2}}\). This closed-loop interplay—where an improved model produces cleaner preference data, which in turn elevates the model further—continually drives the explainer toward exact causal triggers without collapsing into over-generalization.
Loss & Training¶
During the DPO training stages, the objective function minimizes the standard Direct Preference Optimization loss: $\(\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(X, y_w, y_l) \sim \mathcal{D}_{\text{DPO}}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid X)}{\pi_{\text{ref}}(y_w \mid X)} - \beta \log \frac{\pi_\theta(y_l \mid X)}{\pi_{\text{ref}}(y_l \mid X)} \right) \right]\)$ In hyperparameter configuration, Round 1 DPO utilizes \(\beta = 0.1\) with a learning rate of \(5\times 10^{-6}\), while Round 2 shifts to \(\beta = 0.15\) and a learning rate of \(3\times 10^{-6}\). Candidate pool size is fixed at \(M = 6\), text samples per candidate at \(W = 4\), relative variance threshold at \(\tau_{\text{diff}} = 0.1\), and relative rejection margin at \(\tau_{\text{margin}} = 0.15\). The initial SFT stage applies LoRA with rank \(r = 32\), \(\alpha = 64\), dropout \(0.05\), and learning rate \(5\times 10^{-5}\).
Key Experimental Results¶
Main Results¶
Experiments were conducted across diverse target models and their respective open SAE dictionaries: Gemma-2-9B (gemmascope-res-16k), Llama-3.1-8B (llamascope-res-32k), and Gemma-2-27B (gemmascope-res-131k). Evaluation measures Generative Accuracy (GEN, percentage of synthesized texts exceeding 50% of the historical maximum activation), Input Score (INP), Output Score (OUT), as well as high-activation (\(A_{\text{high}}\)) and low-activation (\(A_{\text{low}}\)) means along with their discriminative gap (\(\Delta A\)).
| Target Model & SAE Setup | Method | GEN (↑) | INP (↑) | OUT (↑) | \(A_{\text{high}}\) (↑) | \(A_{\text{low}}\) (↓) | \(\Delta A\) (↑) |
|---|---|---|---|---|---|---|---|
| Gemma-2-9B (Layer 20) | Activation Oracles | 20.33 | 58.67 | 61.67 | 17.77 | 0.49 | 17.28 |
| Neuronpedia (GPT-4o-mini) | 21.03 | 62.67 | 62.67 | 19.01 | 0.62 | 18.39 | |
| Neuronpedia w/ Claude 4.5 | 40.00 | 71.33 | 63.33 | 24.68 | 0.70 | 23.98 | |
| Neuronpedia w/ GPT-5 | 38.87 | 77.00 | 63.33 | 26.22 | 0.55 | 25.67 | |
| SAEExplainer (Ours) | 46.13 | 77.33 | 71.00 | 28.72 | 0.47 | 28.25 | |
| Gemma-2-9B (Layer 31) | Activation Oracles | 26.97 | 64.67 | 62.67 | 47.41 | 1.21 | 46.20 |
| Neuronpedia (GPT-4o-mini) | 27.77 | 70.33 | 57.33 | 50.13 | 1.42 | 48.71 | |
| Neuronpedia w/ Claude 4.5 | 49.43 | 86.33 | 63.00 | 69.01 | 0.99 | 68.02 | |
| Neuronpedia w/ GPT-5 | 49.33 | 86.33 | 64.00 | 72.46 | 1.42 | 71.04 | |
| SAEExplainer (Ours) | 54.50 | 84.33 | 72.33 | 75.49 | 1.19 | 74.30 | |
| Llama-3.1-8B (Layer 15) | Activation Oracles | 31.17 | 41.67 | 70.67 | 3.89 | 0.06 | 3.83 |
| Neuronpedia (GPT-4o-mini) | 33.93 | 49.00 | 69.33 | 4.56 | 0.02 | 4.54 | |
| Neuronpedia w/ Claude 4.5 | 55.67 | 61.33 | 71.66 | 6.52 | 0.03 | 6.49 | |
| Neuronpedia w/ GPT-5 | 54.47 | 63.33 | 70.00 | 6.50 | 0.02 | 6.48 | |
| SAEExplainer (Ours) | 58.40 | 68.00 | 76.00 | 7.19 | 0.01 | 7.18 | |
| Llama-3.1-8B (Layer 25) | Activation Oracles | 21.77 | 50.66 | 70.00 | 5.00 | 0.04 | 4.96 |
| Neuronpedia (GPT-4o-mini) | 27.87 | 56.00 | 69.00 | 5.81 | 0.06 | 5.75 | |
| Neuronpedia w/ Claude 4.5 | 49.67 | 75.00 | 73.33 | 8.79 | 0.10 | 8.69 | |
| Neuronpedia w/ GPT-5 | 46.70 | 77.33 | 75.67 | 8.49 | 0.04 | 8.45 | |
| SAEExplainer (Ours) | 52.33 | 77.67 | 77.00 | 9.08 | 0.09 | 8.99 | |
| Gemma-2-27B (Layer 10) | Activation Oracles | 29.67 | 55.33 | 60.67 | 113.53 | 0.29 | 113.24 |
| Neuronpedia (GPT-4o-mini) | 37.63 | 67.33 | 64.00 | 132.27 | 0.24 | 132.03 | |
| Neuronpedia w/ Claude 4.5 | 44.37 | 73.33 | 62.33 | 143.03 | 2.71 | 140.32 | |
| Neuronpedia w/ GPT-5 | 35.27 | 72.67 | 62.00 | 132.18 | 0.13 | 132.05 | |
| SAEExplainer (Ours) | 53.63 | 82.67 | 67.67 | 163.69 | 0.13 | 163.56 | |
| Gemma-2-27B (Layer 22) | Activation Oracles | 10.23 | 27.00 | 35.33 | 102.37 | 1.11 | 101.26 |
| Neuronpedia (GPT-4o-mini) | 16.17 | 38.33 | 33.00 | 144.01 | 0.61 | 143.40 | |
| Neuronpedia w/ Claude 4.5 | 18.80 | 42.00 | 39.33 | 158.19 | 5.39 | 152.80 | |
| Neuronpedia w/ GPT-5 | 18.60 | 46.67 | 40.00 | 177.21 | 4.80 | 172.41 | |
| SAEExplainer (Ours) | 26.63 | 51.00 | 38.33 | 216.60 | 0.29 | 216.31 |
Ablation Study¶
1. Progressive Training Stage Ablation (Llama-3.1-8B Layer 15) Tracking metrics across SFT baseline, Round 1 DPO (DPO1), and Round 2 DPO (DPO2) highlights the cumulative gains of the closed-loop optimization:
| Training Stage | Generative Accuracy (%) | Input Score (%) | Output Score (%) | Analysis |
|---|---|---|---|---|
| SFT (Baseline) | 26.10 | 51.30 | 70.33 | Superficial label imitation; lacks activation feedback and frequently over-generalizes |
| DPO1 | 52.40 (+26.30) | 71.63 (+20.33) | 75.00 (+4.67) | Incorporates real activation rewards; forces sharp boundaries between genuine triggers and fakes |
| DPO2 (Full) | 58.40 (+6.00) | 77.00 (+5.37) | 76.00 (+1.00) | Re-synthesizes on-policy candidates using DPO1 policy, sustaining further self-evolution |
2. Semantic Collision Analysis (Llama-3.1-8B Layer 15, 2000 sampled features) Evaluating whether distant SAE feature vectors are erroneously assigned identical or similar explanations (lower FSSR and Distant Text Similarity denote superior distinction):
| Method | Feature Semantic Similarity Rate (FSSR) (%) (↓) | Distant Text Similarity (↓) |
|---|---|---|
| Neuronpedia (GPT-4o-mini) | 15.92 | 0.475 |
| Neuronpedia w/ Claude 4.5 | 6.66 | 0.406 |
| SAEExplainer (Ours) | 3.74 | 0.383 |
Key Findings¶
- Transcending the Quality of Training Supervision: Even though SAEExplainer is initially trained on Neuronpedia labels during SFT, two rounds of activation-guided DPO allow it to surpass the training data quality. On Gemma-2-27B Layer 10, Generative Accuracy jumps from 37.63% to 53.63%, comfortably exceeding the highest-grade proprietary LLM baselines (Claude 4.5 at 44.37% and GPT-5 at 35.27%).
- Drastic Reduction in Semantic Collisions: FSSR drops from 15.92% (Neuronpedia) down to 3.74%, representing a 76.5% relative reduction. In qualitative case studies, while Neuronpedia loosely abstracts Feature 31936 as "third-person singular verbs ending in -s" (yielding zero actual activation), SAEExplainer pins down the precise trigger as titles of films containing the word "Hates" (achieving a massive activation score of 12.44).
- Candidate Pool Size (\(M\)) Trade-off: Ablating \(M \in \{2, 6, 10\}\) reveals that \(M = 6\) strikes an optimal balance. While scaling \(M\) to 10 slightly increases raw Generative Accuracy, it degrades the discriminative margin \(\Delta A\) and reduces the Output score due to excessive focus on localized peak activations over holistic semantic fidelity.
Highlights & Insights¶
- Objective Scalar Activation as an Unhackable Reward: Instead of relying on learned proxy reward models prone to reward hacking, SAEExplainer employs the model's authentic internal activations as deterministic ground truth, ensuring uncompromised mechanistic fidelity.
- Relative-Margin Hard Negative Mining: Mining hard negatives by picking the highest-scoring candidate among those suffering a substantial activation drop creates an intense contrastive signal, punishing subtle over-generalizations that appear linguistically flawless.
- Transferable Closed-Loop Paradigm: The "hypothesize \(\rightarrow\) synthesize \(\rightarrow\) verify activation \(\rightarrow\) preference optimize" pipeline offers a foundational paradigm that easily extends beyond SAE features to circuit interpretation and attention head analysis.
Limitations & Future Work¶
- High Training and Preference Data Synthesis Overhead: Generating multi-candidate hypotheses, synthesizing synthetic evaluation corpora, and executing batched forward passes to extract token activations incur substantial computational costs, which may hinder effortless scaling to hundreds of millions of features.
- Homogeneous Model Architecture Constraint: To preserve exact dimensional alignment for residual injection, the current explainer must share the architecture and hidden dimensions of the target model. Future work should investigate cross-model projection adapters to enable lightweight universal explainers.
- Deep-Layer Downstream Output Divergence: As network depth progresses toward 90%, Output Scores experience a steep drop despite high Input scores, reflecting the diffusion of semantic influence across complex late-stage nonlinear layers.
Related Work & Insights¶
- vs Neuronpedia / AutoInterp: AutoInterp methods summarize top-activating n-grams in open-loop fashion, inevitably resulting in hallucinations when token co-occurrences mislead the summary; SAEExplainer closes the loop with verifiable activation feedback, dramatically improving causal precision.
- vs Activation Oracles / LatentQA: These approaches train direct activation-to-text models using purely supervised objectives (SFT), lacking contrastive mechanisms to suppress negative hypotheses; SAEExplainer introduces iterative DPO with hard negatives to carve out sharp conceptual boundaries.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First framework to turn internal SAE activations into verifiable preference optimization rewards.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-model, multi-layer, and human evaluations paired with causal and semantic collision analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Flawless logical rigor with complete implementation and architectural transparency.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly reliable, self-improving foundation for automated mechanistic interpretability at scale.