Skip to content

Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cat-sk/AdaRAS
Area: LLM Reasoning
Keywords: Activation Steering, Reasoning-Critical Neurons, Test-Time Intervention, Polarity Filtering, Adaptive Gating

TL;DR

AdaRAS reveals that a tiny subset of MLP neurons in LLMs strongly correlates with multi-step reasoning validity; by contrasting trajectory-level activation differences with polarity-aware sign-flip filtering and gating interventions via a lightweight failure predictor, it rectifies erroneous reasoning traces at test time with negligible computational overhead.

Background & Motivation

Recent large language models (LLMs) have achieved unprecedented performance on complex mathematical problem solving and code generation. However, achieving reliable multi-step deduction remains inherently challenging, as models frequently suffer from domain constraint hallucinations, algebraic miscalculations, and logical path collapse. To mitigate these failures, current paradigms predominantly rely on heavy post-training (such as reinforcement learning via PPO, GRPO, or distillation from advanced teacher models) and costly test-time compute scaling (such as Self-Consistency majority voting, speculative tree search, or verifier-guided reranking). While effective, these approaches universally treat LLMs as opaque black boxes, incurring steep training expenditures or multiplying inference latency while offering zero interpretability into internal failure mechanisms.

Representation engineering and activation steering provide an appealing white-box alternative, demonstrating that targeted activation perturbations on selected attention heads or feed-forward neurons can steer behavioral attributes such as honesty, persona, and refusal. Nevertheless, reasoning reliability fundamentally diverges from surface-level style or safety modulation: it represents a temporally coherent, trajectory-level property spanning hundreds of tokens with intricate long-range dependencies. Conventional probing-based interventions that depend on the final token hidden states or naive classifier weights fail to generalize across diverse problem instances, often distorting foundational semantic representations and inadvertently degrading performance on prompts that the base model could already solve correctly.

To overcome this dilemma, this work investigates whether internal neuron activations can directly signal and regulate reasoning correctness across complete generation trajectories. Empirical probing reveals that only a very sparse subset of MLP neurons exhibits distinct activation polarization between correct and failed reasoning paths. Motivated by this insight, the authors introduce AdaRAS (Adaptive Reasoning Activation Steering), a parameter-free, test-time framework. Core idea: isolate Reasoning-Critical Neurons (RCNs) by contrasting trajectory-level mean activation differences with polarity-reversal filtering, and adaptively steer their activations during inference via a lightweight prompt-based failure gate to rectify flawed derivations without harming already-correct trajectories.

Method

Overall Architecture

The AdaRAS framework comprises two complementary phases: offline identification and sparse selection of Reasoning-Critical Neurons (RCNs), followed by online adaptive activation intervention during test-time decoding. In the offline phase, contrastive reasoning trajectory pairs (sharing identical prompts but yielding opposite correctness labels) are generated via self-sampling. The framework measures global sequence-averaged activation differences across all MLP neurons, applies a polarity-based sign-flip filter, and selects the top-\(K\) most discriminative neurons to construct sparse steering vectors for each layer. In the online inference phase, an attention-based failure predictor evaluates early hidden states induced by the input prompt. If and only if a reasoning failure is predicted, AdaRAS dynamically injects the precomputed sparse steering vector into the residual stream of the corresponding MLP layers at every decoding step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query & Contrastive Trajectories<br/>Paired correct and incorrect traces"] --> B["Global Trajectory Mean Difference<br/>Sequence-averaged token activation subtraction"]
    B --> C["Polarity-Aware Filtering & Sparse Selection<br/>Sign-flip screening & Top-K neuron isolation"]
    C --> D["Lightweight Adaptive Failure Gate<br/>Early prompt state classification"]
    D -->|Predicted failure: activate steering| E["MLP Activation Steering Intervention<br/>h_l = h_l + α S'_l"]
    D -->|Predicted success: direct bypass| F["Standard Base Model Decoding"]
    E --> G["Reliable Output Reasoning Trace"]
    F --> G

Key Designs

1. Global Trajectory Mean Difference: Local Probing Suffers from Feature Noise -> Sequence-Averaged Activation Subtraction -> Robust Extraction of Reasoning Vectors

Traditional activation steering typically trains linear probes solely on the hidden state of the final answer token or computes token-level differences at the answer boundary. However, in multi-step analytical reasoning, flawed logic often originates early in the intermediate derivations, making single-token states vulnerable to local lexical noise and feature overfitting. To capture sustained reasoning momentum, AdaRAS introduces a trajectory-level mean difference criterion. Given a dataset of paired positive and negative reasoning traces \(\mathcal{P} = \{(p_k^+, p_k^-)\}_{k=1}^N\), the reasoning-critical score for neuron \((l, i)\) is formulated as the expected difference of full-sequence mean activations:

\[S(l, i) = \mathbb{E}_{k \sim [N]} \left[ \mu(a_i^l, p_k^+) - \mu(a_i^l, p_k^-) \right], \quad \text{where} \quad \mu(a, p) \triangleq \frac{1}{|p|} \sum_{t \in p} a(t)\]

Concatenating \(S(l, i)\) across all neurons in layer \(l\) forms a raw layer-wise steering vector \(S_l = [S(l, 1), \dots, S(l, d)]^\top\). This probing-free, global averaging smooths out token-level fluctuations and reliably isolates neurons that systematically foster valid reasoning steps.

2. Polarity-Aware Filtering & Sparse Selection: Dense Interventions Introduce Representational Drift -> Sign-Flip Screening & Top-K Truncation -> Isolation of High-Precision RCNs

Modern open-weight LLMs (such as Qwen3 and LLaMA) widely employ the SwiGLU activation function, whose signed output is essential for feature representation learning. Empirical investigation demonstrates that intervening across entire layers indiscriminately introduces widespread representational noise. Crucially, the neurons most predictive of correctness exhibit an activation polarity reversal between successful and failed traces. AdaRAS therefore filters the candidate pool by enforcing a strict sign-flip constraint:

\[\mathbb{E}_k \left[ \mu(a_i^l, p_k^+) \right] \cdot \mathbb{E}_k \left[ \mu(a_i^l, p_k^-) \right] < 0\]

This polarity test ensures that retained neurons actively switch from positive activation to negative suppression (or vice versa) depending on reasoning validity. After filtering, neurons are ranked by absolute magnitude \(|S(l, i)|\), retaining only the top-\(K\) neurons (with optimal \(K=50\), representing merely ~0.03% of total parameters) to build the sparse steering vector \(S'_l\). At inference time, steering is added to the MLP output:

\[h_l = h_{l-1} + W_l^{down} \phi(W_l^{up} h_{l-1}) + \alpha S'_l\]

where \(\alpha\) is a positive scalar (typically set between 0.1 and 0.3), concentrating interventions in middle-to-late transformer layers.

3. Lightweight Adaptive Failure Gate: Unconditional Steering Harms Valid Traces -> Prompt-Induced Early State Prediction -> Targeted On-Demand Intervention

A common failure mode in activation addition is the degradation of initially correct trajectories: applying a rigid steering bias to high-confidence inputs frequently perturbs delicate, correct logical representations. To eliminate this trade-off, AdaRAS incorporates a lightweight adaptive intervention module that serves as an inference gate. Using ANOVA F-statistics, the system selects 256 salient features from early hidden states triggered by the input prompt. An attention-based binary classifier (consisting of a 2-layer MLP with ReLU and dropout) is trained on these features, achieving an AUROC of 0.8347 on AIME. During decoding, if the prompt is classified as likely to succeed under the base model, the steering mechanism remains inactive; if a reasoning failure is predicted, the RCN steering vector is applied. This selective execution ensures maximum gains on hard instances while entirely preventing performance regression on easy cases.

Loss & Training

The adaptive gating module is trained offline using standard binary cross-entropy loss:

\[\mathcal{L}_{gate} = - \left( y \log \sigma(f_\theta(x)) + (1-y) \log (1 - \sigma(f_\theta(x))) \right)\]

Trained on prompt-level representations with the Adam optimizer for 100 epochs, this gating module requires only tens of seconds to a few minutes of one-time training. In contrast, the steering vectors \(S'_l\) are extracted analytically without backpropagation or parameter updates. At inference time, AdaRAS introduces a single vector addition per decoding step, resulting in an almost imperceptible latency overhead of ~90 ms per problem.

Key Experimental Results

Main Results

AdaRAS was evaluated across 10 competitive mathematics and coding benchmarks against greedy CoT, three prominent open-source post-trained reasoning models (DeepSeek-R1-Distill-Qwen-1.5B, OpenThinker3-1.5B, and OpenReasoning-Nemotron-1.5B), and a vanilla probing-based steering baseline (Probing). All evaluations utilized Qwen3-1.7B as the base model under greedy decoding.

Dataset Benchmark Category CoT R1-Distill Nemotron OpenThinker Probing AdaRAS (Ours) Absolute Gain
AIME-24 Olympiad Math 47.83% 34.78% 39.13% 26.08% 43.48% 60.87% +13.04%
AIME-25 Olympiad Math 40.91% 22.73% 40.91% 50.00% 50.00% 54.55% +13.64%
AIME-Extend Olympiad Math 47.33% 21.33% 47.33% 42.67% 48.00% 52.67% +5.34%
MATH-500 Multi-domain Math 84.80% 66.20% 85.40% 82.00% 85.40% 86.40% +1.60%
GSM8K Grade-school Math 88.32% 72.93% 87.57% 67.55% 88.32% 89.08% +0.76%
AMC-12 High-school Math 65.93% 41.76% 65.93% 53.85% 73.63% 70.33% +4.40%
HumanEval Code Synthesis 77.18% 45.64% 57.72% 59.06% 77.85% 79.19% +2.01%
HumanEval+ Rigorous Code 69.80% 42.95% 51.68% 53.69% 72.48% 73.15% +3.35%
MBPP Code Synthesis 68.78% 42.59% 53.97% 35.71% 70.37% 72.22% +3.44%
MBPP+ Rigorous Code 58.20% 37.30% 43.92% 30.95% 59.52% 60.58% +2.38%

Ablation Study

Table 1 outlines component ablations on the demanding AIME-24 and AIME-25 datasets; Table 2 isolates the impact of the polarity-aware filtering mechanism against an unconstrained magnitude-only baseline.

Table 1: Ablation of key components in AdaRAS (Base: Qwen3-1.7B)

Variant / Configuration AIME-24 Accuracy AIME-25 Accuracy Empirical Observation
Qwen3-1.7B Base (CoT) 47.83% 40.91% Default unsteered baseline
Random Steering 34.78% 27.27% Indiscriminate neuron modification causes catastrophic drop (>13%)
w/o Mean Difference (Probe Weights) 43.48% 50.00% Probing weights suffer from overfitting; drops 17.39% on AIME-24
w/o Activation Selection (No Polarity) 52.17% 45.45% Lacks sign-flip constraint; drops ~9-10% on both datasets
w/o Adaptive Intervention (Always-On) 56.52% 45.45% Perturbs initially correct trajectories, leading to degraded accuracy
AdaRAS (Full Framework) 60.87% 54.55% Combines MD, polarity filtering, and adaptive gating for peak reliability

Table 2: Validation of polarity-aware selection over magnitude-only selection

Dataset Base Model Magnitude Only (Top-50) Polarity-Aware (AdaRAS)
AIME-24 47.83% 56.52% 60.87% (+4.35%)
AIME-25 40.91% 45.45% 54.55% (+9.10%)
AIME-Extend 47.33% 50.67% 52.67% (+2.00%)

Key Findings

  • Disproportionate Gains on Complex Tasks: AdaRAS delivers massive improvements on hard mathematical competitions (AIME-24: +13.04%, AIME-25: +13.64%), while showing smaller improvements on saturated elementary benchmarks like GSM8K (+0.76%). This proves that RCN intervention primarily prevents multi-step error propagation rather than simply aiding memorization.
  • Cross-Domain and Cross-Dataset Transferability: RCNs discovered on AIME generalize seamlessly to coding benchmarks without re-identification, lifting HumanEval from 79.19% to 80.54% and MBPP from 72.22% to 72.75%. This indicates that RCNs capture fundamental, domain-agnostic meta-reasoning behaviors (e.g., condition verification and algebraic discipline).
  • Stabilization of Latent Trajectory Dynamics: Layer-wise geometric analysis revealed that AdaRAS substantially diminishes sequence-averaged latent tortuosity (Magnitude) while preserving the semantic direction (Angle). This shows that the steering vector straightens erratic hidden state paths into direct derivations without corrupting the model's linguistic representation space.
  • Zero Length-Bias and Minimal Compute Overhead: Textual metrics (character count, word count, sentence count, and syntactic dependency depth) confirm that AdaRAS does not inflate generation verbosity. Furthermore, the single-pass greedy decoding requires only ~90 ms additional runtime per sample, vastly outperforming multi-sample verification or best-of-\(N\) searches.

Highlights & Insights

  • Surpassing Post-Training via Test-Time Internal Steering: The paper demonstrates that carefully steering merely 50 critical neurons in the MLP layers outperforms resource-intensive post-trained reasoning models (such as DeepSeek-R1-Distill and Nemotron) by substantial margins without altering a single weight.
  • Exploiting the Signed Dynamics of SwiGLU: Rather than relying purely on absolute magnitude sorting, AdaRAS leverages the polarity sign-flip across paired correct and incorrect trajectories as a strong natural filter, successfully eliminating spurious background neurons.
  • Mechanistic Grounding of Reasoning Rectification: Through detailed geometric case studies, the authors show that RCN steering explicitly restores boundary checks in geometry problems and enforces symbolic identity precision in algebraic operations, providing tangible mechanistic validation.

Limitations & Future Work

  • Architectural Scope: The empirical validation centers primarily on the Qwen3 family (1.7B, 4B, and 14B). Evaluating cross-architecture transferability to LLaMA-3, Mistral, and Mixture-of-Experts (MoE) models remains to be explored.
  • Dependence on Contrastive Trace Generation: Extracting RCNs requires a balanced pool of correct and incorrect traces generated from the same prompt. For ultra-capable frontier models (which rarely fail on moderate tasks) or weak base models (which rarely succeed on hard tasks), collecting balanced contrastive pairs poses a cold-start bottleneck.
  • Static Linear Intervention Profile: Currently, the steering vector is injected uniformly across all generation steps. Developing an adaptive step-level gating mechanism that modulates steering intensity based on real-time token entropy or attention dispersion represents an exciting next step.
  • vs Test-Time Compute Scaling (Self-Consistency & Verifiers): Standard inference scaling trades multiplied compute budgets (e.g., \(N=4\) or \(N=64\) sample generation followed by majority voting or reward reranking) for reliability. AdaRAS operates strictly in single-pass greedy decoding, delivering superior accuracy over Best-of-4 verifiers at a tiny fraction of the computational footprint.
  • vs Activation Addition & CAA: Methods like Contrastive Activation Addition (Rimsky et al.) target prompt-level stylistic attributes (such as sycophancy or sentiment) via simple endpoint differences. AdaRAS introduces sequence-averaged trajectory accumulation and polarity filtering, tailoring activation steering to the temporal fragility of mathematical reasoning.
  • vs Probing-Based Inference-Time Intervention (ITI): Prior ITI frameworks steer attention heads using linear probing classifier coefficients, which frequently overfit and degrade multi-hop reasoning (e.g., causing a 4.35% drop on AIME-24). AdaRAS relies on non-parametric mean difference and sparse polarity selection, demonstrating vastly superior robustness.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the link between sparse polarized neurons and multi-step reasoning validity; provides an elegant parameter-free steering framework]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluation across 10 diverse math and code benchmarks, scaling from 1.7B to 14B parameters with geometric latent dynamics analysis]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear argumentation, mathematically rigorous formulation, and insightful qualitative case studies]
  • Value: ⭐⭐⭐⭐⭐ [Establishes a highly efficient, plug-and-play inference reliability paradigm that circumvents expensive post-training and sampling costs]