Skip to content

IdioSteer: Controlling Figurative-vs-Literal Interpretation via Residual-Stream Activation Steering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Intellexus-DSI/IdioSteer
Area: LLM Reasoning
Keywords: Idiom Interpretation, Activation Steering, Residual Stream, Cross-Layer Intervention, Non-Compositional Semantics

TL;DR

Addressing the pervasive bias where large language models default to figurative readings of idiomatic expressions, this paper introduces the IdioSteer benchmark and establishes a white-box residual-stream activation steering framework that leverages an asymmetric cross-layer regime to systematically boost literal interpretation rates by 2.4x to 4.5x without degrading generation fluency.

Background & Motivation

Natural language abounds with non-compositional semantic constructions whose meaning cannot be assembled directly from constituent words, with idiomatic expressions serving as a quintessential exemplar. While modern large language models have acquired the capacity to recognize idiomatic phrases across diverse benchmarks, their next-token generation dynamics remain heavily encumbered by conventional defaults. In unsteered open-ended generations where a sentence prefix ends in an idiom that genuinely admits either a figurative or a literal reading, LLMs display an overwhelming propensity to pursue the conventional idiomatic sense, largely ignoring compositional readings even when physical context makes them equally or more plausible.

This rigid interpretative bias highlights an unresolved tension in representation engineering. Prior activation steering methodologies have predominantly targeted coarse, low-level behavioral traits such as sentiment polarity, compliance versus refusal, or factual association strength. In contrast, higher-order semantic phenomena—specifically the dynamic disambiguation between compositional literal meaning and conventional non-compositional meaning—have remained largely unexplored. Standard same-layer intervention techniques often face an acute dilemma: intervening in shallow layers fails because abstract figurative directions are not yet linearly decodable, whereas intervening in late layers leaves insufficient network depth for downstream self-attention layers to propagate and structure the desired continuation.

To break this impasse, the authors explore whether idiomatic interpretation is encoded as a stable, manipulable direction in the residual stream and how representation extraction can be decoupled from causal steering leverage. Core idea: construct an idiom-general direction via class-conditional mean differences in mid-to-late residual layers and inject it as a single-position intervention at the earliest layers (\(L_i \le 2\)), establishing an asymmetric cross-layer steering regime that reliably shifts LLM continuations toward literal interpretations while preserving generation coherence.

Method

Overall Architecture

The IdioSteer framework establishes an inference-time intervention pipeline that guides frozen autoregressive language models away from their default figurative bias and toward compositional literal interpretations. The workflow comprises three primary phases: First, residual activations at the final token of idiomatic phrases are harvested from a training corpus (IdioLink) across figurative and literal usages to estimate an idiom-general steering vector. Second, the vector is normalized and scaled via a model-dependent multiplier calibrated to residual-stream norms, configured for either additive modification or norm-preserving angular rotation. Third, during evaluation prompt processing, a forward hook fires at the final idiom token at a shallow injection layer, updating the KV cache and causing subsequent autoregressive decoding to adhere to the literal semantic trajectory.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input sentence prefix ending at target idiom<br/>Prefix ending at idiom"] --> B["Forward pass to extract mid-to-late activations<br/>Source Layer Ls Activation"]
    B --> C["Estimate Mean-Difference Steering Vector<br/>Mean-Difference Vector (vMD)"]
    C --> D["Calibrate Multiplier Scaling & Select Operator<br/>Multiplier Scaling & Operator"]
    D --> E["Single-position Cross-Layer Injection at Li<br/>Cross-Layer Injection at Li"]
    E --> F["Upward KV-cache Propagation<br/>Propagate to Upper Layers"]
    F --> G["Autoregressive Literal Continuation Generation<br/>Literal Continuation"]

Key Designs

1. Mean-Difference Vector Construction: Contrastive Centroid Estimation Overcoming Probe Non-Identifiability To isolate a robust directional axis in residual space, the authors contrast empirical class-conditional centroids rather than relying on logistic regression probe weights. While standard probes achieve high training and cross-validation accuracy, over-parameterization causes probe weight vectors to be non-unique and highly unstable across models and idiom subsets. By evaluating the training split of IdioLink (comprising 1,683 balanced sentences across 22 idioms), activations \(h \in \mathbb{R}^d\) are recorded at the idiom's final token at source layer \(L_s\). The normalized mean-difference steering vector is computed directly as: $\(v_{\text{MD}} = \frac{\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}}{\|\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}\|}\)$ where positive scaling pushes toward figurative interpretations and negative scaling pushes toward literal interpretations. PCA-based dimensionality reduction demonstrates that as unconstrained optimization freedom is pruned, logistic probe weights converge geometrically toward \(v_{\text{MD}}\) (cosine similarity increasing from 0.484 to 0.717), confirming that the contrastive mean difference uncovers the true principal axis of semantic distinction while resisting probe non-identifiability.

2. Multiplier Scaling and Dual Steering Operators: Norm-Aware Cross-Model Calibration and Angular Rotation Because raw mean-difference norms \(s_{\text{MD}} = \|\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}\|\) vary by over three orders of magnitude across architectures (from 4.18 in Llama-3.2-3B to 7,134.37 in Gemma-3-12B), unnormalized steering coefficients cannot transfer across models. The framework parameterizes additive intervention as \(\alpha = \alpha_{\text{factor}} \cdot s_{\text{MD}}\), bounding effective multiplier grids within \([-4.78, +4.78]\) across all evaluated models: $\(\mathcal{O}_+(h, v; \alpha) = h + \alpha v\)$ To further prevent activation norm distortion from impairing generation fluency, the paper introduces a norm-preserving angular rotation operator (\(\mathcal{O}_\theta\)). Target activation \(h\) is decomposed into components parallel and orthogonal to \(v\): \(c_1 = v^\top h\), \(r = h - c_1 v\), \(c_2 = \|r\|\), yielding an orthonormal 2D basis \(\{v, b_2(h)\}\) with \(b_2(h) = r / \|r\|\). Rotating coordinates by angle \(\theta \in [-\pi, \pi]\) preserves \(\|h\|\) identically: $\(c'_1 = c_1 \cos \theta - c_2 \sin \theta, \quad c'_2 = c_1 \sin \theta + c_2 \cos \theta\)$ $\(\mathcal{O}_\theta(h, v; \theta) = c'_1 v + c'_2 b_2(h)\)$ The angular operator ensures constant activation norms, offering a resilient alternative in model families that are acutely sensitive to magnitude scaling.

3. Cross-Layer Steering Regime: Decoupling Representation Decodability from Downstream Causal Leverage Conventional activation addition enforces same-layer intervention (\(L_s = L_i\)), which inherently couples the layer of semantic extraction with the layer of steering execution. The authors identify a structural spatial separation: linear decodability of idiom interpretation peaks and plateaus in the mid-to-late network (50%–80% of model depth, e.g., layer 14 of 28 in Llama-3.2-3B or layer 20 of 26 in Gemma-2-2B), whereas downstream causal control is vastly most potent at the earliest layers (\(L_i \in \{0, 2\}\)). By estimating \(v_{\text{MD}}\) at \(L_s\) and injecting it at \(L_i \le 2\) via a forward hook on the final prompt token, the modified hidden state propagates upward through all subsequent transformer blocks. This single-position modification updates the key-value caches that future autoregressive tokens attend to, outperforming same-layer steering by 3.0x to 3.5x while keeping perplexity growth within ~10%.

Benchmark & Evaluation Protocol

To rigorously evaluate the method under genuine semantic ambiguity, the authors introduce IdioSteer, a curated benchmark of 100 sentence prefixes spanning 20 English idioms (5 variants per idiom) selected from MAGPIE and completely disjoint from the 22 training idioms. Each prefix terminates precisely at the target idiom with zero post-idiom tokens. Crucially, candidate sentences were validated under a strict dual-completion protocol: two independent human annotators were required to produce both a fluent figurative continuation and a fluent literal continuation; any prefix where either reading felt forced was discarded and rewritten.

During evaluation, five base models (Llama-3.2-3B, Llama-3.1-8B, Gemma-2-2B, Gemma-3-4B, Gemma-3-12B) generate 5 completions per sentence (500 completions per condition) under temperature 0.7 and top-p 0.9. Evaluation is scaled using an LLM-as-a-judge (Gemma-4-31B-it), which was calibrated on 280 human-annotated generations (human-human Cohen's \(\kappa = 0.867\), judge accuracy 90.0%, \(\kappa = 0.821\)) and validated across 795 independent multi-model samples (91.9% accuracy, \(\kappa = 0.868\)). The evaluation tracks literal generation rate, coherence rate, and token micro-perplexity relative to baseline.

Key Experimental Results

Main Results

The primary benchmark evaluation tests the highest literal rate attained by each steering method across the calibrated coefficient grid on IdioSteer, contrasted with the unsteered baseline and random perturbation controls:

Model Params Baseline Lit. Rate MD-Additive (Ours) MD-Angular Probe-Additive Random-Additive MD-Additive Coherence
Llama-3.2-3B 3.2B 0.116 [0.070, 0.170] 0.518 [0.442, 0.594] 0.275 0.370 0.259 0.956 [0.934, 0.976]
Llama-3.1-8B 8.0B 0.124 [0.072, 0.182] 0.512 [0.440, 0.586] 0.251 0.310 0.213 0.910 [0.848, 0.958]
Gemma-2-2B 2.6B 0.134 [0.088, 0.184] 0.574 [0.490, 0.662] 0.320 0.135 0.249 0.988 [0.974, 0.998]
Gemma-3-4B 4.0B 0.130 [0.074, 0.188] 0.362 [0.280, 0.444] 0.370 0.129 0.218 0.962 [0.940, 0.980]
Gemma-3-12B 12.0B 0.144 [0.086, 0.210] 0.342 [0.268, 0.418] 0.260 0.145 0.193 0.996 [0.990, 1.000]

Note: Square brackets indicate 95% bootstrap confidence intervals over 1,000 resamples. Baseline literal rates of 0.12–0.14 demonstrate the intense default figurative bias across all models. MD-Additive yields 2.4x–4.5x gains in literal rate while keeping coherence above 0.91.

Ablation Study

Ablations and mechanistic diagnostics conducted on Llama-3.2-3B dissect the impact of layer pairings, directional asymmetry, and semantic decomposition:

Configuration / Condition Parameter Setting Literal Rate Human-Corrected Lit. Coherence Rate Perplexity Ratio Description & Mechanism
Unsteered Baseline \(\alpha = 0\) 0.116 — 0.998 1.000 Heavy default preference for figurative sense
Cross-Layer MD-Additive \(L_s=14, L_i=2, \alpha_{\text{factor}}=-2.0\) 0.527 0.535 0.980 1.100 Optimal cross-layer design, maximizing leverage
Same-Layer Diagonal MD \(L_s=14, L_i=14, \alpha_{\text{factor}}=-2.0\) 0.168 — 0.998 1.020 Same-layer intervention yields minimal effect
Same-Layer Norm-Matched \(L_s=14, L_i=14, \alpha_{\text{factor}}=-7.5\) 0.182 — 0.978 1.080 Matched relative perturbation still fails to steer
Pure Concreteness Vector (\(v_{\text{conc}}\)) \(L_s=14, L_i=2, \alpha=+10\) 0.432 0.370 0.928 1.070 Captures physical scene bias; fails on abstract literal
Concreteness-Ablated Vector (\(v_\perp\)) \(L_s=14, L_i=2, \alpha=-10\) 0.344 0.320 0.898 1.110 Retains non-compositional syntactic control
Figurative Direction (Sign Flip) \(\alpha_{\text{factor}}=+2.39\) 0.130 — 0.980 1.090 Asymmetry: cannot increase already saturated figurative bias
Random Vector Control \(\alpha_{\text{factor}}=-4.78\) 0.250 — 0.860 1.170 Symmetric, undirected degradation of representations

Key Findings

  • Pronounced Sign Asymmetry Confirming Directionality: Pushing activations along the literal direction (negative \(\alpha_{\text{factor}}\)) creates a large, monotonic surge in literal generation rate, whereas pushing along the figurative direction leaves literal rate near baseline. Crucially, generation perplexity under the unsteered model increases symmetrically with intervention magnitude (1.27 at \(-4.78\) vs. 1.14 at \(+4.78\)). This double dissociation between sign-asymmetric semantics and sign-symmetric perplexity proves that steering effects stem from a true directional vector rather than generic activation corruption.
  • Cross-Layer Superiority (3.0x–3.5x Advantage): Across all evaluated models, pairing a mid-to-late source layer with an early injection layer (\(L_i \le 2\)) outperforms same-layer steering by more than threefold. Even when the injection-site relative strength \(r_{\text{rel}} = |\alpha| / \|h(L_i)\|\) is matched to within 0.2% on Llama-3.2-3B, the diagonal intervention achieves only 0.182 literal rate versus 0.527 for cross-layer steering, showing that early injection allows downstream self-attention layers to restructure the entire generative sequence.
  • Decomposition of Concreteness vs. Idiomaticity: A control vector constructed purely from 250 concrete versus 250 abstract nouns accounts for a notable fraction of the average steering effect (0.432 literal rate). However, an orthogonalized vector with the concreteness subspace completely projected out (\(v_\perp\)) still attains a 0.344 literal rate. Crucially, the two components dissociate across idioms: \(v_{\text{conc}}\) fails on idioms whose literal reading is not a physical scene (e.g., out of the blue, reaching only 0.07), whereas \(v_\perp\) excels on them (reaching 0.68), demonstrating that the learned vector captures genuine compositional semantics beyond lexical concreteness.

Highlights & Insights

  • Cross-Layer Activation Steering as a General Paradigm: Demonstrates that the layer with peak linear decodability is spatially disconnected from the layer with maximum causal steering leverage. Decoupling source and injection layers resolves a major bottleneck in representation engineering.
  • IdioSteer's Bidirectional Plausibility Benchmark: By enforcing that human annotators must successfully produce natural continuations in both figurative and literal directions, IdioSteer guarantees that model generation choices reflect internal semantic preferences rather than preceding context constraints.
  • Mechanistic Clarification of Concreteness Bias: Disentangles the long-standing linguistic debate on whether idiom literalization is merely an artifact of scene concreteness, demonstrating through orthogonal projection that residual space maintains an independent axis for compositional syntactic re-parsing.

Limitations & Future Work

  • Restriction to English Base Models: Experiments focus strictly on pre-instruction-tuned English foundation models. Evaluating whether directional interpretative axes persist through instruction tuning, RLHF, and cross-lingual transfer remains an open challenge.
  • Scale of the Benchmark Sample: IdioSteer deliberately curates 100 tightly controlled prefixes across 20 idioms to satisfy bidirectional plausibility. Expanding to larger lexicons, syntactic inversions, and varied sentence positions represents a natural next step.
  • Necessity vs. Sufficiency in Representation: While cross-layer steering demonstrates that the mean-difference vector is causally sufficient to flip model interpretations, establishing whether this exact residual direction is strictly necessary during natural, unsteered disambiguation requires deeper circuit analysis.
  • vs. Contrastive Activation Addition (CAA; Rimsky et al., 2024): Whereas CAA injects steering vectors at every decoding step across the full generation horizon, IdioSteer intervenes strictly once at the final prompt token, leveraging standard KV-cache propagation to guide downstream generation with minimal perplexity disruption.
  • vs. Concreteness Steering (Ghosh & Jiang, 2026): While prior work examined lexical concreteness as a broad stylistic dimension, IdioSteer isolates phrasal interpretation selection in multi-word expressions, demonstrating via orthogonal decomposition that compositional idiom parsing operates independently of descriptive concreteness.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Extends activation steering to high-order non-compositional semantic disambiguation and formalizes the cross-layer steering paradigm]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans five base models across two architectural families, with rigorous human-validated judging, perplexity dissociation, and orthogonal ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear argumentation, transparent disclosure of failure modes, and elegant mathematical formulation of operators]
  • Value: ⭐⭐⭐⭐⭐ [Provides foundational methodologies for controlling semantic bias in LLMs and deepens mechanistic understanding of non-compositional representation]