IdioSteer: Controlling Figurative-vs-Literal Interpretation via Residual-Stream Activation Steering¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Intellexus-DSI/IdioSteer
Area: Information Extraction
Keywords: Idiom Understanding, Activation Steering, Residual Stream Representation, Figurative-vs-Literal Control, Non-compositional Semantics
TL;DR¶
Addressing large language models' deep-seated bias toward figurative interpretations of idioms during open-ended generation, this paper proposes residual-stream activation steering to achieve precise inference-time control: by injecting a cross-layer mean-difference vector at the prompt's final idiom token, models are systematically shifted toward literal continuations while maintaining fluency, with the steering direction transferring seamlessly to unseen idioms.
Background & Motivation¶
Natural language abounds with non-compositional semantic constructions, among which idiomatic expressions represent a foundational challenge: their phrasal meanings cannot be derived compositionally from individual lexical items. While modern large language models (LLMs) demonstrate strong capabilities on standard idiom identification and classification benchmarks, recent studies reveal that they harbor a persistent, systemic bias toward figurative interpretations. In unconstrained generation contexts, when presented with an ambiguous prefix such as "During the party, Sarah tried to break the ice", LLMs overwhelmingly generate conventional figurative continuations (e.g., chatting with guests to relieve tension), completely ignoring the equally natural and physically plausible literal interpretation (e.g., crushing ice cubes into pieces). This rigid default preference restricts model flexibility in pun interpretation, physical commonsense reasoning, and contexts requiring compositional semantic re-analysis.
From the perspective of interpretability and controllable generation, modulating this non-compositional preference without supervised fine-tuning (SFT) or reinforcement learning remains a major unresolved challenge. Existing residual-stream activation steering techniques have primarily targeted coarse or surface-level attributes such as sentiment polarity, refusal behavior, or factual associations, leaving higher-order linguistic-semantic disambiguation largely unexplored. Crucially, mechanistic studies indicate that figurative meanings are retrieved in parallel with literal compositional pathways during internal processing; standard same-layer activation interventions struggle to alter this balance, typically trading off generation coherence against insufficient semantic steering.
To overcome these barriers, this work investigates whether the figurative-vs-literal distinction is encoded as a causally manipulable linear direction within the residual stream. The authors construct a tightly controlled benchmark ensuring bidirectional plausibility and establish an intervention protocol that decouples feature extraction from injection depth. Core idea: by estimating a mean-difference vector between figurative and literal activations at mid-to-late transformer layers and injecting it into the earliest layers at the final idiom token, single-pass KV-cache propagation systematically steers language model generation toward literal interpretations without sacrificing fluency.
Method¶
Overall Architecture¶
The IdioSteer framework enables training-free, inference-time causal steering between figurative and literal generation across frozen LLMs. The pipeline operates in three sequential phases: First, using an independent training pool of idiom-bearing sentences (IdioLink), residual-stream hidden states at the final token of idiomatic expressions are collected at a designated source layer \(L_s\) to construct a normalized mean-difference direction vector. Second, during inference on a prefix ending in an unresolved idiom, a forward hook injects this vector at a shallow injection layer \(L_i\) (\(L_i = 2\) by default) strictly at the final idiom token. Third, the perturbed activation propagates through higher layers during the prompt forward pass, updating key-value (KV) caches such that all subsequent autoregressive tokens attend to this modified representation and generate literal continuations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Ambiguous Prefix<br/>Prompt ending at target idiom"] --> B["Stage 1: Mean-Difference Vector Construction<br/>Extract class mean difference at mid-to-late layer"]
B --> C["Stage 2: Cross-Layer Single-Token Injection<br/>Perturb residual stream at shallow layer on final token"]
C --> D["Stage 3: KV Cache Causal Propagation<br/>Single-pass prompt update biases autoregressive decoding"]
D --> E["Output Coherent and Diverse Literal Continuations"]
Key Designs¶
1. Mean-Difference Vector Construction: Addressing Probe Non-Identifiability in Over-Parameterized Spaces -> Extracting Class Centroid Differences with Global Multiplier Scaling -> Providing Robust Cross-Model Directional Control While prior steering work frequently derives steering directions from the weight vector of a logistic regression probe, linear probes in over-parameterized regimes (\(d/n > 1\)) can achieve 100% training accuracy across diverse arbitrary hyperplanes, leading to catastrophic failure when transferring to downstream steering (as observed on the Gemma model family). IdioSteer instead adopts Contrastive Activation Addition via the normalized mean-difference (MD) vector. Given sets of figurative (\(H_{\text{fig}}\)) and literal (\(H_{\text{lit}}\)) activations extracted at source layer \(L_s\) on the training pool, the unit-norm steering direction is computed as: $\(v_{\text{MD}} = \frac{\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}}{\|\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}\|}\)$ By convention, positive scaling points toward figurative interpretation and negative scaling toward literal interpretation. Geometric PCA analysis confirms that constraining the optimization subspace of probes causes their weights to converge directly toward \(v_{\text{MD}}\), demonstrating that the mean-difference vector uniquely captures the primary axis of class-conditional semantic variation.
2. Cross-Layer Single-Token Injection: Resolving the Spatial Mismatch Between Representation Decodability and Intervention Leverage -> Estimating at Mid-to-Late Layers and Injecting at Shallow Layers -> Maximizing Downstream Causal Influence Conventional activation steering assumes same-layer intervention (\(L_s = L_i\)). However, systematic probing reveals that linear decodability of the figurative-vs-literal distinction only plateaus (exceeding 95%) in mid-to-late transformer layers (50%–80% depth), whereas intervention leverage is highest at the earliest layers (\(L_i \in \{0, 2\}\)), where perturbations can shape all subsequent transformer computation. IdioSteer resolves this tension via a cross-layer steering regime: the vector \(v_{\text{MD}}\) is constructed at the optimal decodability layer \(L_s\) and injected into residual activations at layer \(L_i = 2\): $\(h' = h + \alpha v_{\text{MD}}, \quad \alpha = \alpha_{\text{factor}} \cdot s_{\text{MD}}\)$ Here, \(s_{\text{MD}} = \|\bar{h}_{\text{fig}} - \bar{h}_{\text{lit}}\|\) normalizes the intervention strength across models, successfully overcoming the three-orders-of-magnitude difference in raw residual stream norms across model families (ranging from 4.18 in Llama-3.2-3B to 7,134 in Gemma-3-12B).
3. KV Cache Causal Propagation: Preventing Multi-Step Cumulative Distortion -> Firing Once at the Completed Phrasal Boundary via Standard Attention Caches -> Preserving Fluency and Eliciting Diverse Literal Strategies Unlike methods such as CAA that continuously add perturbations across all generated tokens, IdioSteer respects the causal attention structure of autoregressive transformers. Because the final token of the idiom integrates all preceding sentence context and earlier idiom tokens under causal attention, steering is applied exactly once at this terminal position \(p\) during the prompt pass. The modified activation \(h'\) ascends through the upper transformer layers, writing altered key and value states into the KV cache. When generating subsequent completion tokens, the model's standard attention naturally conditions on this literal-biased semantic state. This single-position intervention maintains high coherence rates (\(\ge 0.91\)) and prevents mode collapse, unlocking diverse compositional strategies such as reinterpreting the head noun as a physical object, forming unexpected noun compounds (e.g., "out of the blue room"), or constructing novel literal narratives.
Loss & Training¶
The proposed method operates strictly at inference time without requiring any gradient updates or parameter fine-tuning of the base models. In the offline calibration stage, hidden states are recorded during a single forward pass over 1,683 sentences from the IdioLink training pool (covering 22 idioms across 10 subject categories). Model-specific source layers \(L_s\) and additive multiplier grids \(\alpha_{\text{factor}} \in [-4.78, +4.78]\) are determined using training-pool sentences only, leaving the evaluation benchmark untouched. In addition to additive steering, an angular operator rotating activations within the \(\text{span}\{h, v\}\) plane is evaluated across angles \(\theta \in \{\pm 60^\circ, \pm 70^\circ, \pm 90^\circ, \pm 120^\circ\}\) to test norm-preserving geometric transformations.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on the IdioSteer benchmark (100 sentences across 20 unseen idioms with bidirectional plausibility) spanning five base LLMs across two architectural families (2B to 12B parameters). Evaluation is scaled using Gemma-4-31B-it as a calibrated LLM judge (achieving 91.9% agreement and Cohen's \(\kappa = 0.868\) against gold human annotations).
| Model | Baseline Literal Rate | MD-add Literal Rate (Ours) | Literal Rate Gain | MD-add Coherence Rate | Optimal \(\alpha_{\text{factor}}\) |
|---|---|---|---|---|---|
| Llama-3.2-3B | 0.116 [0.070, 0.170] | 0.518 [0.442, 0.594] | 4.47x | 0.956 [0.934, 0.976] | -2.39 |
| Llama-3.1-8B | 0.124 [0.072, 0.182] | 0.512 [0.440, 0.586] | 4.13x | 0.910 [0.848, 0.958] | -2.00 |
| Gemma-2-2B | 0.134 [0.088, 0.184] | 0.574 [0.490, 0.662] | 4.28x | 0.988 [0.974, 0.998] | -2.60 |
| Gemma-3-4B | 0.130 [0.074, 0.188] | 0.362 [0.280, 0.444] | 2.78x | 0.962 [0.940, 0.980] | -1.50 |
| Gemma-3-12B | 0.144 [0.086, 0.210] | 0.342 [0.268, 0.418] | 2.38x | 0.996 [0.990, 1.000] | -1.00 |
Ablation Study¶
Ablations examine layer configurations (cross-layer vs same-layer), operator and vector variants, and semantic component decomposition on Llama-3.2-3B.
| Configuration / Variant | Injection Layer / Operator | Literal Rate | Coherence Rate | Notes / Comparison vs Baseline |
|---|---|---|---|---|
| Unsteered Baseline | — | 0.116 | 1.000 | Pronounced figurative default |
| Same-Layer Steering (Diagonal \(L_s=14, L_i=14\)) | Additive (\(\alpha_{\text{factor}}=-2.0\)) | 0.168 | 0.998 | Weak gain (+0.052); low leverage at deep layers |
| Cross-Layer Steering (\(L_s=14, L_i=2\)) | Additive (\(\alpha_{\text{factor}}=-2.0\)) | 0.527 | 0.980 | 3.14x over same-layer; substantial causal leverage |
| Probe Vector Steering (PROBE-add) | Additive (\(\alpha_{\text{factor}}=-2.39\)) | 0.370 | 0.950 | Inferior to MD; collapses to baseline on Gemma (0.135) |
| Angular Steering (MD-ang) | Angular Rotation (\(\theta=+90^\circ\)) | 0.275 | 0.942 | Preserves activation norm but restricted dynamic range |
| Random Vector Control (avg. 3 unit vectors) | Additive (\(\alpha_{\text{factor}}=-4.78\)) | 0.259 | 0.938 | Non-directional perturbation; symmetric across sign |
| Concreteness Control (\(v_{\text{conc}}\), Brysbaert) | Additive (\(\alpha=+10\)) | 0.432 (corr. 0.370) | 0.928 | Captures physical scene concreteness |
| Concreteness-Ablated Idiom Vector (\(v_\perp\)) | Additive (\(\alpha=-10\)) | 0.344 (corr. 0.320) | 0.898 | Sustains control on non-scene idioms ("behind bars") |
Key Findings¶
- Cross-layer intervention yields a 3.0x–3.5x performance multiplier: Decoupling the source layer (where representation decodability peaks at 50%–80% depth) from the injection layer (\(L_i \le 2\)) outperforms same-layer steering by more than threefold. A matched relative-strength diagnostic confirms that this advantage is genuine rather than an artifact of activation norm differences.
- Directional asymmetry dissociates from generic perturbation: Steered generation exhibits strong sign asymmetry in literal rate (negative multipliers yield 3.0x–4.3x higher literal rates than positive multipliers), while token perplexity ratios remain almost perfectly symmetric with respect to sign (\(1.27\) at \(-4.78\) vs \(1.14\) at \(+4.78\)). This cleanly rules out random perturbation hypotheses.
- Robust idiom-general transfer and semantic decomposition: Steering vectors derived from 22 training idioms generalize to 20 unseen evaluation idioms, outperforming baselines in 97 of 100 (model, idiom) cells. Concreteness ablation reveals that the steering direction decomposes into a general scene concreteness component and an idiom-specific non-compositional component (\(v_\perp\)), which independently governs re-parsing when concreteness and literalness diverge.
Highlights & Insights¶
- Decoupled Cross-Layer Steering Protocol: Establishes that the optimal layer for extracting a semantic direction does not coincide with the optimal layer for applying it. This spatial decoupling provides a generalizable paradigm for residual-stream representation engineering.
- Single-Token KV-Cache Propagation: Demonstrates that modifying a single residual state at the phrasal boundary during the prompt forward pass suffices to govern entire autoregressive sequences, avoiding the degradation associated with multi-token steering.
- Subspace Orthogonal Decomposition: Elegantly dissects the learned steering vector into general concreteness and pure idiomaticity components via geometric projection, resolving debates regarding whether figurative-to-literal steering merely reflects concrete vocabulary biasing.
Limitations & Future Work¶
- Scope Limited to English Base Models: Experiments focus on English idioms and pre-instruction-tuning base models. Investigating whether similar linear directions emerge across typologically diverse languages and persist after RLHF/SFT alignment remains an open problem.
- Broader Figurative Constructions: The study focuses on fixed multi-word idiomatic expressions. Extending representation steering to metaphor, irony, hyperbole, and similes requires further exploration.
- Controlled Benchmark Scale: IdioSteer contains 100 carefully curated prefixes across 20 idioms to enforce strict bidirectional plausibility. Expanding the benchmark scale while retaining bidirectional quality is an important future direction.
Related Work & Insights¶
- vs CAA (Rimsky et al., 2024) & Activation Addition (Turner et al., 2024): Whereas prior contrastive activation addition applies continuous perturbations across all generation steps within the same layer, IdioSteer introduces a single-token prompt-pass hook combined with cross-layer injection, preserving higher fluency and achieving superior semantic leverage.
- vs Idiom Probing and Mechanistic Analyses (Hefetz et al., 2025; Oh et al., 2026): Prior work primarily traced idiom representations observationally; IdioSteer operationalizes these insights into an active, training-free causal steering mechanism.
- vs Concreteness Steering (Ghosh & Jiang, 2026): While Ghosh & Jiang steer broad lexical concreteness styles, IdioSteer targets phrasal non-compositional semantic disambiguation and demonstrates through orthogonal decomposition that idiom steering encodes substantial non-compositional structure beyond concreteness.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First demonstration of causal residual-stream steering over high-order figurative-vs-literal interpretation, introducing the cross-layer intervention regime.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 5 models from 2 families, supported by rigorous directional asymmetry checks, layer sweeps, and human-in-the-loop judge validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative progression, self-contained mathematical formulations, and thorough reporting of experimental controls and negative findings.
- Value: ⭐⭐⭐⭐☆ Highly impactful for mechanistic interpretability, representation steering, and training-free behavioral control of foundation models.