Skip to content

Adaptive Knowledge Routing for Complex Reasoning in LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Benchmark Redundancy, Sparse Autoencoders, Evaluation-Free Diagnosis, Feature Saturation, Evaluation Suite Pruning

TL;DR

This paper proposes an evaluation-free benchmark analysis paradigm based on Sparse Autoencoder (SAE) feature coverage, transforming the computationally prohibitive multi-model performance-matrix problem into a lightweight feature-footprint diagnosis on a single indicator model, achieving a Spearman rank correlation of \(\rho \approx 0.85\) with true performance redundancy across 17 benchmarks and enabling deterministic, low-variance subset compression.

Background & Motivation

The exponential explosion of the large language model evaluation landscape has made exhaustive benchmark curation practically untenable. Model developers face ever-expanding evaluation suites spanning elementary arithmetic, advanced scientific reasoning, coding, and multilingual understanding. Every new pre-training checkpoint, fine-tuning recipe, or post-training alignment method must be verified across dozens of benchmarks, incurring immense costs in benchmark maintenance, data curation, and model inference. While prior works on efficient benchmarking—such as tinybenchmarks and SparseEval—attempt to reduce these expenses by sampling representative subsets or predicting full-dataset performance, they inherently rely on dense historical evaluation matrices gathered from hundreds or thousands of models, creating a circular dilemma where one must first run exhaustive evaluations to decide how to avoid them.

Against this backdrop, a more fundamental scientific question arises: to what extent do diverse evaluation benchmarks actually measure different underlying capabilities? Grounded in a functionalist behavioral perspective, an evaluation instance is characterized not by its surface prompt, human-assigned metadata, or intended taxonomy label, but by its operational role in distinguishing model capabilities. However, measuring this behavioral equivalence directly via empirical performance leads to a combinatorial explosion across model panels and benchmark suites. Prior to exhaustive evaluation, the community lacks a principled, evaluation-free diagnostic tool that can quantify both intra-benchmark redundancy (sample-level compression) and inter-benchmark similarity (suite-level consolidation) without relying on expensive model runs or supervised regression models.

The paper breaks this circularity through a central mechanistic insight: because processing an evaluation instance corresponds to engaging specific functional subspaces within a language model, behavioral equivalence can be effectively approximated by internal monosemantic feature overlap. The authors leverage Sparse Autoencoders (SAEs) to extract sparse, interpretable latent features from a single indicator model, translating the intractable performance-matrix estimation problem into a geometric feature-footprint analysis. Core idea: Use Sparse Autoencoder (SAE) monosemantic features as an evaluation-free proxy for behavioral equivalence in language models, transforming expensive multi-model performance matrices into single-model feature-footprint coverage to quantify intra-benchmark redundancy and inter-benchmark capability overlap, thereby enabling zero-variance benchmark compression and principled evaluation-suite curation.

Method

Overall Architecture

The proposed framework translates benchmark redundancy and capability overlap into the monosemantic feature space of a single indicator model equipped with an SAE, comprising three interconnected components: (1) Intra-benchmark Feature Redundancy: Quantifying individual benchmark information density by computing the expected SAE feature-coverage saturation curve across sample subsets, adjusted by an inverse growth-rate factor to preserve absolute capability breadth; (2) SAE-Guided Greedy Subset Selection: Formulating representative subset selection as a combinatorial distribution-matching problem that minimizes the \(L_2\) discrepancy between subset and full-benchmark feature activation frequencies via an incremental, deterministic greedy algorithm; and (3) Inter-benchmark Symmetrized Feature Overlap with Partial Correlation: Defining a min-normalized symmetric feature overlap metric to assess capability containment across benchmarks, paired with partial correlation analysis to control for general model ability and isolate benchmark-specific capability overlap.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Benchmark Datasets Input<br/>Instance Set D = {x_1, ..., x_K}"] --> B["Indicator Model + SAE Latent Extraction<br/>Active Feature Set F(x_i)"]
    B --> C["Intra-Benchmark Redundancy<br/>Feature Coverage ck and Scale Correction ˆR(D)"]
    B --> D["SAE-Guided Greedy Subset Selection<br/>Minimizing Feature Frequency L2 Loss"]
    B --> E["Inter-Benchmark Overlap & Partial Correlation<br/>overlap_min and General Ability Control via MMLU"]
    C --> F["High-Redundancy Benchmark Diagnosis<br/>ρ=0.85 Rank Correlation with True Redundancy"]
    D --> G["Deterministic Low-Variance Subsets<br/>+5-10% Kendall's Tau at Small Subset Sizes"]
    E --> H["Evaluation Suite Pruning & Selection<br/>Pearson r=75.5% with De-biased Similarity"]

Key Designs

1. Intra-Benchmark Redundancy: Feature Coverage Saturation with Growth-Rate Correction

Traditional behavioral redundancy \(R(D)\) is measured by the area under the rank-correlation curve (AUC) tracing the expected Kendall's \(\tau_k\) between subsets and the full benchmark across a model panel. The authors map this into feature space: for a sample \(x_i\), its active feature set is \(F(x_i) = \{j \in \{1, \dots, D\} : z_j(x_i) > 0\}\), and the full benchmark footprint is \(F(D) = \bigcup_{i=1}^K F(x_i)\). The expected feature coverage \(c_k\) at subset size \(k\) mirrors the behavioral curve:

\[c_k = \mathbb{E}_{S \subseteq D, |S|=k} \left[ \frac{|F(S)|}{|F(D)|} \right]\]

However, simply computing the area under the coverage curve \(\text{AUC}(c_k)\) discards the absolute capability breadth: two benchmarks of size \(K\) with identical linear coverage trajectories would receive the same AUC, even if one activates 2,000 distinct features and the other activates only 1,000. To preserve absolute dimensionality, the authors scale \(\text{AUC}(c_k)\) by the reciprocal of the per-sample feature growth rate, yielding the scale-corrected feature redundancy metric:

\[\hat{R}(D) = \text{AUC}(c_k) \cdot \frac{K}{|F(D)|} = \frac{\sum_{k=1}^K c_k}{|F(D)|}\]

This formulation penalizes datasets that generate fewer novel features per instance, allowing direct, evaluation-free redundancy comparisons across benchmarks of vastly different sizes and domains.

2. SAE-Guided Greedy Subset Selection: Deterministic Feature Distribution Matching

While uniform random sampling preserves expected distributions on average, it suffers from severe variance when the sample budget is tight (e.g., \(k/K < 0.3\)), risking arbitrary rank distortions. The authors propose selecting the smallest subset that faithfully matches the full benchmark's feature activation distribution. Defining the feature frequency vector \(f(D) \in \mathbb{R}^D\) as:

\[[f(D)]_i = \frac{1}{|D|} \sum_{x \in D} \mathbf{1}(i \in F(x))\]

The optimization objective minimizes the \(L_2\) discrepancy between the subset and full-benchmark frequency profiles:

\[\hat{S} = \arg\min_{S} \| f(S) - f(D) \|_2^2\]

To solve this combinatorial problem efficiently, an incremental greedy algorithm is employed: beginning with an empty set, each iteration selects the single sample that produces the maximal reduction in \(L_2\) loss. Because this procedure yields a nested sequence of subsets \(S_1 \subset S_2 \subset \dots \subset S_K = D\), a single \(O(K^2)\) pass generates compressed subsets of all intermediate sizes simultaneously. Furthermore, the selection exhibits near-zero variance across random tie-breaking shuffles, providing an entirely stable, one-shot compression mechanism.

3. Inter-Benchmark Symmetrized Feature Overlap and General Ability De-biasing

To identify capability containment and redundancy across benchmarks, the asymmetric overlap \(\text{overlap}(D_1, D_2) = \frac{|F(D_1) \cap F(D_2)|}{|F(D_1)|}\) quantifies whether the capabilities probed by \(D_1\) are subsumed by \(D_2\). To align with symmetric empirical correlation \(\rho(D_1, D_2)\), the authors formulate a min-normalized symmetric overlap metric:

\[\text{overlap}_{\min}(D_1, D_2) = \frac{|F(D_1) \cap F(D_2)|}{\min(|F(D_1)|, |F(D_2)|)}\]

The min-denominator ensures that the score remains high whenever a smaller benchmark's feature footprint is predominantly enclosed within a broader benchmark. However, empirical performance correlations are heavily confounded by general model ability (\(g\) factor)—capable models achieve superior scores across all tasks, inflating pairwise correlations between entirely disjoint capabilities. To disentangle this confound, partial correlation analysis is applied using MMLU as the control variable \(G\):

\[\rho_{\text{partial}}(D_1, D_2 \mid G) = \frac{\rho_{12} - \rho_{1g}\rho_{2g}}{\sqrt{1 - \rho_{1g}^2}\sqrt{1 - \rho_{2g}^2}}\]

Residualizing against general capability removes spurious cross-domain correlations while preserving authentic within-domain alignment, unlocking a faithful comparison with SAE feature overlap.

Benchmark & Evaluation Protocol

Experiments are executed using Qwen3-30B-A3B-Base as the indicator model, with its corresponding SAE inserted at layer 36 using Top-k activation (\(k=100\)). Ground-truth performance metrics are computed over a panel of 26 Qwen-3.5-35B-A3B checkpoints varying in training steps and domain data mixtures (math, code, multilingual, general text) across 17 diverse benchmarks. Robustness is further verified using a Gemma3-12B-it SAE and an expanded 34-model heterogeneous panel spanning Gemma3, Llama-3.1, Qwen2.5, and Qwen3 across 4B to 14B parameter scales.

Key Experimental Results

Main Results: Intra-Benchmark Redundancy and Inter-Benchmark Similarity Prediction

Diagnostic Dimension Evaluation Scale / Scope Metric Evaluated SAE Feature Proxy Baseline / Comparison Key Takeaway & Empirical Insight
Intra-Benchmark Redundancy 17 Diverse Benchmarks Spearman \(\rho\) 0.85 Random/Uninformed (0.00) Predicts true performance redundancy without evaluating any models
Intra-Benchmark Redundancy 17 Diverse Benchmarks Pearson \(r\) (Log-space) 0.78 Raw Coverage AUC (uncalibrated) Scale-corrected metric successfully decouples size from redundancy
Inter-Benchmark Similarity 8 Benchmarks (28 Pairs) Raw Pearson \(r\) 68.4% - Symmetric min-normalized overlap reflects empirical model alignment
Inter-Benchmark Similarity 8 Benchmarks (28 Pairs) Partial Pearson \(r\) 75.5% Raw Correlation 68.4% (+7.1%) Controlling for MMLU general ability eliminates spurious cross-domain correlation
Inter-Benchmark Similarity 8 Benchmarks (28 Pairs) Partial Spearman \(\rho\) 71.3% Raw Correlation 60.7% (+10.6%) Robustly identifies candidate benchmarks for suite consolidation
Model Panel Robustness 34 Heterogeneous Models Redundancy Spearman \(\rho\) 0.85 Original 26 Models (0.85) Benchmark rank order remains invariant under diverse external model families

Ablation Study: SAE-Guided Sampling vs. Random Sampling and Cross-Model Generalization

Configuration / Benchmark Sampling Ratio / Setting Random Baseline (Kendall's \(\tau\)) SAE-Guided Sampling (Ours) Variance & Relative Advantage
GSM8K (\(K=1319\)) Full-Curve AUC 0.900 (\(\pm\) high std) 0.910 (near-zero std) AUC gain +0.010; eliminates wide sampling fluctuations
MATH (\(K=5000\)) Full-Curve AUC 0.935 (\(\pm\) high std) 0.951 (near-zero std) AUC gain +0.016; at \(k=200\), prevents \(\pm 0.05\) random draw noise
MMLU-Pro (\(K=12032\)) Full-Curve AUC 0.924 (\(\pm\) high std) 0.933 (near-zero std) Highly faithful representative rank preservation on complex reasoning
CMMLU (\(K=11528\)) Full-Curve AUC 0.969 (\(\pm\) high std) 0.972 (near-zero std) Robust compression on large-scale Chinese multidisciplinary tasks
Small-Budget Regime (\(k/K < 0.3\)) Relative Improvement Baseline Reference +5% to +10% higher \(\tau\) Greatest ranking preservation advantage when evaluation budget is minimal
Alternative SAE (Gemma3-12B) Intra-Redundancy Spearman \(\rho\) - 0.87 (vs. Qwen3-30B 0.85) Confirms proxy robustness across distinct SAE indicator architectures
Alternative SAE (Gemma3-12B) Partial Pearson Similarity - 75.9% (vs. Qwen3-30B 75.5%) Consistent discovery of domain containment across independent SAE models

Key Findings

  • Sample count does not dictate redundancy: Dataset size \(K\) is not an indicator of informational diversity. For instance, GSM8K (1,319 samples) exhibits substantially higher performance and feature redundancy than MMLU-Redux (3,000 samples). When subsampling GSM8K to 198 items to match GPQA-Diamond, its redundancy remains high (\(R = 0.794 \pm 0.010\)) compared to GPQA-Diamond's \(0.542\), proving that advanced multi-step STEM reasoning possesses intrinsic low redundancy.
  • Early general-purpose benchmarks exhibit high redundancy: MMLU, MMLU-Pro, and CMMLU cluster in the high-redundancy quadrant, whereas the recent SuperGPQA benchmark (26K samples across 285 graduate disciplines) achieves significantly lower redundancy despite its large volume, indicating that intentional capability diversity translates directly into lower feature overlap.
  • SAE-guided sampling drives variance to zero: The primary practical utility of SAE-guided selection lies in variance elimination. While random sampling approaches comparable mean correlation given dozens of repeated draws, a single random subset at small \(k\) can fluctuate widely (e.g., \(\tau = 0.79 \pm 0.05\) on MATH at \(k=200\)). SAE-guided selection is essentially deterministic, enabling reliable one-shot evaluation compression.
  • Unveiling structural containment hierarchies: Asymmetric overlap reveals that \(63\%\) of GSM8K features are subsumed by MATH, proving that evaluation suites containing MATH can safely discard GSM8K without sacrificing diagnostic fidelity. Similarly, code benchmarks (EvalPlus, MBPP, MultiPL-E) cluster tightly together, while broad knowledge benchmarks subsume specialized tests.

Highlights & Insights

  • Reframing the performance-matrix bottleneck as feature geometry: Evaluating \(N\) benchmarks across \(M\) models conventionally requires \(O(M \times N)\) expensive inference runs. By extracting feature footprints via a single forward pass of one indicator model, the methodology computes redundancy in \(O(N)\) passes without evaluating any target models.
  • Growth-rate calibration solves the normalization paradox: Normalizing coverage curves to \([0, 1]\) typically discards absolute feature space dimensions; multiplying by \(K/|F(D)|\) restores sensitivity to raw capability breadth, allowing valid comparisons across vastly different benchmark sizes.
  • Partial correlation debiasing isolates true capability overlap: Recognizing that general intelligence (\(g\) factor) inflates all pairwise benchmark correlations, the paper uses partial correlation conditioned on MMLU to strip out shared capability variance, increasing the alignment between feature overlap and empirical similarity from 68.4% to 75.5%.

Limitations & Future Work

  • Ground truth assumes benchmark validity: The framework treats existing benchmarks as ground truth populations without evaluating whether underlying questions are poorly phrased, whether answer keys contain annotation noise, or whether difficulty distributions are balanced.
  • Dependence on indicator model capability horizon: The extracted feature representations depend on the dictionary capacity of the indicator model's SAE; tasks testing capabilities far outside the base model's pre-training horizon may fail to trigger meaningful monosemantic activations.
  • Future directions: Extending evaluation-free diagnostics to multi-layer SAE feature intersections, dynamic interactive agent environments, and multi-turn open-ended dialogue suites.
  • vs. Item Response Theory and Fisher Information (tinybenchmarks, Metabench): Prior efficient benchmarking methods require pre-computed evaluation scores from hundreds of historical models or train predictive MLPs; this work is entirely evaluation-free, requires no external model results, and offers monosemantic interpretability.
  • vs. Surface Text Embedding Clustering: Semantic clustering using sentence embeddings (e.g., text-embedding-3) only captures lexical and topical similarity; SAE features reflect internal functional mechanisms engaged during model computation, detecting computational redundancy even across divergent prompt formats.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering application of mechanistic interpretability tools (SAE) to macroscopic evaluation benchmark analysis and suite design.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 17 benchmarks, 26 pre-training checkpoints, and 34 cross-family heterogeneous models with rigorous ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity in formalizing functionalist behavioral equivalence and motivating scale-corrected metrics.
  • Value: ⭐⭐⭐⭐⭐ Provides an immediately actionable, compute-efficient framework for pruning redundant datasets and curating scalable evaluation suites.