Unlocking the Spoken Style: Mechanistic Localization and Lightweight Control of Orality in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/tjunlp-lab/llm-spoken
Area: Audio & Speech
Keywords: Orality Control, Mechanistic Interpretability, Causal Activation Patching, Lightweight Activation Steering, Voice Agents
TL;DR¶
Addressing the issue where voice agents produce overly written responses and lack audience-adaptive intensity control, this paper constructs WSPairs—the first parallel corpus of 4,709 written-to-spoken pairs annotated with 0–5 Orality Levels—mechanistically localizes orality to late Transformer layers and sparse attention heads via causal patching, and proposes two lightweight controllers: weight-free Calibrated Activation Steering (CAS) and Learned Activation Steering (LAS) requiring only 0.004%–0.006% trainable parameters to achieve fine-grained, monotonic style rewriting.
Background & Motivation¶
For safety and regulatory auditability, modern voice agents overwhelmingly employ a cascaded "text-first" architecture: a large language model (LLM) first drafts the conversational reply in text, which a neural text-to-speech (TTS) engine subsequently vocalizes. However, written and spoken registers serve divergent sociolinguistic functions. Written text is compact, syntactically complex, and information-dense, whereas natural spoken dialogue is characterized by discourse markers, fillers, hedges, loose clause fragmentation, and real-time self-repairs. This fundamental mismatch makes LLM outputs sound unnatural and robotic, even when read by expressive, state-of-the-art TTS models. Crucially, real-world conversational applications—such as healthcare consultations, customer service, and elderly companionship—require graded, audience-tailored orality: a brisk, mildly conversational response suffices for working adults, whereas cognitive-support or elderly interactions demand frequent pauses, lexical reductions, and fragmented clause structures to ease cognitive load.
Despite this necessity, graded written-to-spoken control faces three critical hurdles. First, style-transfer benchmarks predominantly focus on the reverse spoken-to-written direction (e.g., ASR transcript clean-up) or binary formal/informal classifications, completely lacking parallel corpora in the deployment-aligned written-to-spoken direction with fine-grained ordinal levels. Second, while mechanistic interpretability has successfully localized discrete retrieval tasks—such as factual recall, refusal behaviors, and emotional representations—it remains unclear whether surface-level stylistic attributes like orality are diffusely scattered across the network or localized to specific circuits. Third, full supervised fine-tuning (SFT) or parameter-efficient fine-tuning (PEFT) per target level is computationally prohibitive and transfers poorly across domains, while existing activation-side steering mechanisms are engineered for binary switches rather than monotonic multi-level intensity dials.
The central insight addressing these challenges is that if spoken style is manifested through specific discourse markers and syntactic looseness, language models must maintain an internal causal circuit responsible for this stylistic rendering. Core idea: By constructing a 0–5 graded spoken-style parallel corpus grounded in sociolinguistic features, causally localizing orality to late Transformer layers and a sparse set of attention heads, and intervening on this identified sub-circuit via weight-free calibrated steering (CAS) and ultra-lightweight element-wise affine steering (LAS with ~0.004% parameters), achieve parameter-efficient, semantics-preserving, and cross-domain transferable graded orality control.
Method¶
Overall Architecture¶
The proposed framework comprises three sequential phases: dataset construction and level quantification, mechanistic causal localization, and lightweight intervention control. The system takes a formal written text \(x_w\) and a desired target orality level \(t \in \{0, 1, \dots, 5\}\) as input, and outputs the rewritten spoken text \(x_s\). First, the authors construct WSPairs (4,709 parallel pairs from authentic task-oriented dialogue transcripts) and establish a feature-based estimator \(g(\cdot)\) using seven sociolinguistic features combined via Entropy Weight Method (EWM) and k-means clustering, eliminating the noise of single-text human subjective scoring. Next, causal activation patching and contrastive influence scoring (CIS) localize orality rendering to a critical band of late layers \(L_{\text{oral}}\) and a sparse set of high-influence attention heads \(A_{\text{oral}}\). Finally, two complementary controllers—weight-free CAS and learned affine LAS—are implemented directly on this localized circuit.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Input: Written Text $x_w$ + Target Orality Level $t \in \{0..5\}$"] --> B["Stage 1: Corpus & 7-Feature Orality Estimator<br/>WSPairs 4,709 pairs + EWM/KMeans level partition"]
B --> C["Stage 2: Causal Patching & Head Attribution<br/>Locate critical layers $L_{oral}$ and sparse heads $A_{oral}$"]
C --> D{"Stage 3: Lightweight Control Interventions"}
D -->|Training-Free Dial| E["CAS: Calibrated Activation Steering<br/>Multi-layer vector injection + Polynomial calibration"]
D -->|Learned Adapter| F["LAS: Learned Activation Steering<br/>Affine transform on residual stream & heads (0.004% params)"]
E --> G["Output: Target Spoken-Style Text $x_s$"]
F --> G
Key Designs¶
1. Corpus Construction & 7-Feature Orality Level Estimator: Solving the Subjectivity and Lack of Graded Baselines in Spoken Register
Orality is psychometrically ordinal and relative: human raters achieve high consistency when comparing two texts pairwise, but assigning an absolute 0–5 integer to an isolated text produces substantial inter-annotator noise. To construct WSPairs, authentic human agent utterances \(x_s\) preserving natural fillers and disfluencies were mined from SpokenWoZ, MediTOD, and Bargaining ASR transcripts. Written counterparts \(x_w\) were initially drafted by five LLMs and strictly filtered (semantic BERTScore \(\ge 0.85\), Jaccard similarity \(< 0.7\), and clean written verification), followed by rigorous human manual verification across 5 annotators (achieving Cohen's \(\kappa = 0.92\) on independent double-checks). To quantify orality objectively, seven sociolinguistic features \(f(x) \in \mathbb{R}^7\) were introduced: five spoken-oriented features (fillers \(f_1\), interaction markers \(f_2\), stance/hedges \(f_3\), lexical reduction \(f_4\), and repetition \(f_5\)) and two written-oriented structural features (propositional idea density \(f_6\) and noun-verb ratio \(f_7\)). Applying the Entropy Weight Method (EWM) objectively assigns higher weights to high-variance features. Weighted PCA followed by k-means clustering (\(k=6\), optimal by silhouette score) maps any text deterministically to discrete levels \(g(x) \in \{0, \dots, 5\}\). On 200 held-out human pairwise anchor judgments, \(g(\cdot)\) achieves 91.0% pair accuracy, surpassing both an LLM judge ensemble (88.5%) and trained Bradley-Terry reward models (85.5%).
2. Layer-wise Causal Patching & Head-level Contrastive Attribution: Revealing Late-Layer and Sparse-Head Localization
To determine whether orality is diffusely encoded or structurally localized, the authors performed activation patching across layers on contrastive prompts requesting Level 0 versus Level 5 rewriting for the same input. By patching oral hidden states \(h_{\text{oral}}^\ell\) into the written run and recording the Average Indirect Effect (AIE) on the predicted orality level, a sharp "hockey-stick" profile emerged: the indirect effect on Qwen-3B remains virtually zero through layer 19 before sharply rising at layer 20 and saturating near the top, while semantic preservation (BERTScore) remains steady. This establishes the critical oral layer set \(L_{\text{oral}} = \{20, \dots, 35\}\) (with the elbow occurring at layer 15 in Mistral-7B and layer 5 in Llama-1B). Moving to fine-grained head attribution within \(L_{\text{oral}}\), zero ablation was paired with a Contrastive Influence Score (CIS) targeting a spoken vocabulary \(V_{\text{oral}}\) versus non-oral vocabulary \(V_{\text{non-oral}}\): $\(CIS_a = \mathbb{E}_{y \in V_{\text{oral}}}[I_a(y)] - \mathbb{E}_{y \in V_{\text{non-oral}}}[I_a(y)]\)$ where \(I_a(y) = P_M(y|x, y_{<t}) - P_{M \setminus a}(y|x, y_{<t})\). Filtering by \(|CIS_a - \mu| > 2\sigma\) identified only 35 orality heads \(A_{\text{oral}}\) (34 positive, 1 negative) out of 448 heads in Qwen-3B, concentrated in layers 20–30. Logit lens inspections of the top-ranking head (L20–H8) confirmed that it specifically promotes spoken discourse tokens like well, I guess, so, and you know, operating as dedicated discourse-rendering components rather than general style units.
3. CAS (Calibrated Activation Steering): Training-Free Multi-Layer Graded Control
Extending classic single-layer Activation Addition, CAS operates purely at inference time without any weight updates. To overcome the failure of single-layer fixed scalars in multi-level control, CAS introduces two innovations: multi-layer injection across all identified layers \(L_{\text{oral}}\) and target-conditioned polynomial calibration. For each \(\ell \in L_{\text{oral}}\), a steering vector is derived by taking the normalized difference of mean hidden states between Level 5 and Level 0 at the assistant role-tag token position: \(\hat{v}_\ell = v_\ell / \|v_\ell\|_2\). During generation, the vector is broadcast across decoding positions: \(h_\ell' = h_\ell + \alpha \hat{v}_\ell\). Because the empirical response curve \(\alpha \mapsto \hat{\ell}(\alpha)\) exhibits a rise-then-plateau trend, CAS sweeps \(\alpha \in [0, 4]\) on a validation split to find optimal per-level coefficients \(\alpha^*_t = \arg\min_\alpha |\hat{\ell}(\alpha) - t|\) and fits a quadratic control curve \(\alpha^*(t) \approx w_2 t^2 + w_1 t + w_0\). This quadratic calibration produces a unified steering knob that dramatically improves level-matching accuracy over baseline prompting without updating model weights.
4. LAS (Learned Activation Steering): Ultra-Lightweight Affine Steering Matching Full SFT with 0.004% Parameters
While CAS enables weight-free control, its static direction cannot adapt to varying syntactic contexts, leading to text expansion at extreme levels. LAS freezes the entire base LLM and learns component-specific element-wise affine transformations strictly on the localized causal circuit \(L_{\text{oral}} \cup A_{\text{oral}}\). For each localized layer's residual stream and each identified head's output projection input, LAS learns scale and bias vectors \(d_c, b_c \in \mathbb{R}^d\):
$\(h_c' = h_c + (d_c \odot h_c + b_c)\)$
Parameters \(d_c\) and \(b_c\) are initialized to zero, ensuring training begins smoothly from an exact identity mapping. The total parameter count \(|\phi| = 2 d_{\text{model}} (|L_{\text{oral}}| + |L_{\text{oral}}^{\text{head}}|)\) accounts for merely 0.004%–0.006% of the backbone (e.g., 119K parameters on Qwen-3B). Under Grouped-Query Attention (GQA), the head intervention is cleanly applied to the reshaped [B, T, nhead, dhead] tensor immediately before o_proj, fully preserving shared KV projection memory efficiency. Optimized end-to-end with language modeling cross-entropy loss on WSPairs, LAS adaptively modulates activations to achieve precise stylistic intensity while rigorously anchoring semantic facts.
Loss & Training¶
LAS leaves all backbone model weights frozen and optimizes only the sparse affine parameters \(\phi = \{d_c, b_c\}\) using standard autoregressive cross-entropy loss over the 3,183 WSPairs training pairs: $\(\mathcal{L}(\phi) = - \sum_{i=1}^N \sum_{t=1}^{|x_s^{(i)}|} \log P_{\theta, \phi}\left(x_{s, t}^{(i)} \mid x_w^{(i)}, \text{prompt}(t^{(i)}), x_{s, <t}^{(i)}\right)\)$ Training is executed on a cluster of 4 NVIDIA A800 80GB GPUs in bf16 precision for 10 epochs, with an effective batch size of 64 using AdamW and a cosine learning rate schedule with 10% warm-up. Peak learning rates are set to \(5 \times 10^{-3}\) for Qwen-3B and Mistral-7B, and \(1 \times 10^{-3}\) for Llama-1B. Training completes within 0.4–1.5 GPU hours per backbone, producing adapter checkpoints under 1MB.
Key Experimental Results¶
Main Results¶
The authors evaluated open-source models across weight-free baselines (Few-shot, single-layer fixed ActAdd, multi-layer CAS) and supervised parameter-efficient methods (Full SFT, LoRA, \(\text{IA}^3\), LAS). Metrics include decoding throughput (Tokens/s), content preservation (BERTScore F1), discrete Level Accuracy with signed bias (\(\text{L-Acc}_{\text{bias}}\)), and reward-score MAE with Pearson correlation (\(\Delta R_r\)) under an independent Mistral-7B Bradley-Terry reward model.
In the weight-free setting, CAS dramatically outperforms Few-shot and ActAdd in level-hit accuracy while sharply reducing signed bias:
| Model | Method | Tokens/s (↑) | BERTScore (↑) | \(\text{L-Acc}_{\text{bias}}\) (↑) | \(\Delta R_r\) (↓) |
|---|---|---|---|---|---|
| Qwen-3B | Few-shot | 420.5 | 0.900 | \(0.246_{-0.98}\) | \(0.52_{0.43}\) |
| ActAdd | 418.1 | 0.893 | \(0.149_{-1.87}\) | \(0.46_{0.51}\) | |
| CAS (Ours) | 415.8 | 0.894 | \(0.390_{-0.26}\) | \(0.30_{0.81}\) | |
| Mistral-7B | Few-shot | 185.1 | 0.892 | \(0.210_{-1.36}\) | \(0.49_{0.46}\) |
| ActAdd | 183.6 | 0.875 | \(0.324_{-0.17}\) | \(0.39_{0.62}\) | |
| CAS (Ours) | 181.5 | 0.852 | \(0.378_{-0.49}\) | \(0.32_{0.74}\) | |
| Qwen-7B | Few-shot | 175.4 | 0.902 | \(0.169_{-1.63}\) | \(0.58_{0.40}\) |
| ActAdd | 173.6 | 0.899 | \(0.053_{-2.56}\) | \(0.51_{0.49}\) | |
| CAS (Ours) | 171.2 | 0.878 | \(0.245_{-0.63}\) | \(0.33_{0.78}\) |
In the supervised setting, LAS tunes only ~0.004% of parameters yet achieves superior level accuracy over scope-matched LoRA and \(\text{IA}^3\), even outperforming 100% parameter Full SFT on Qwen-3B and Mistral-7B:
| Model | Method | Trainable Params | Tokens/s (↑) | BERTScore (↑) | \(\text{L-Acc}_{\text{bias}}\) (↑) | \(\Delta R_r\) (↓) |
|---|---|---|---|---|---|---|
| Llama-1B | Full SFT | 1.2B (100%) | 1150.2 | 0.904 | \(0.514_{-0.46}\) | \(0.27_{0.81}\) |
| LoRA | 0.6M (0.053%) | 1145.5 | 0.902 | \(0.378_{-0.08}\) | \(0.46_{0.66}\) | |
| \(\text{IA}^3\) | 0.1M (0.011%) | 1152.1 | 0.899 | \(0.363_{-0.52}\) | \(0.49_{0.61}\) | |
| LAS (Ours) | 78K (0.006%) | 1151.8 | 0.901 | \(0.413_{-0.50}\) | \(0.31_{0.78}\) | |
| Qwen-3B | Full SFT | 3.0B (100%) | 416.6 | 0.913 | \(0.573_{-0.42}\) | \(0.24_{0.83}\) |
| LoRA | 1.9M (0.061%) | 410.5 | 0.910 | \(0.464_{-0.41}\) | \(0.42_{0.71}\) | |
| \(\text{IA}^3\) | 0.4M (0.013%) | 417.2 | 0.907 | \(0.451_{-0.39}\) | \(0.46_{0.65}\) | |
| LAS (Ours) | 119K (0.004%) | 418.5 | 0.911 | \(0.584_{-0.34}\) | \(0.21_{0.86}\) | |
| Mistral-7B | Full SFT | 7.2B (100%) | 182.4 | 0.915 | \(0.572_{-0.29}\) | \(0.23_{0.84}\) |
| LoRA | 2.4M (0.033%) | 178.9 | 0.918 | \(0.575_{-0.46}\) | \(0.28_{0.79}\) | |
| \(\text{IA}^3\) | 0.5M (0.006%) | 183.5 | 0.905 | \(0.479_{-0.31}\) | \(0.45_{0.66}\) | |
| LAS (Ours) | 295K (0.004%) | 175.2 | 0.915 | \(0.592_{-0.30}\) | \(0.19_{0.88}\) |
Ablation Study¶
Ablations on Qwen-3B systematically evaluate the structural sites and calibration formulations for CAS, as well as the intervention sites and parameterization types for LAS:
CAS Ablation Analysis (WSPairs Test Set): | Configuration | \(\bar{\alpha}\) | L-Acc (↑) | BERTScore (↑) | Note | | :--- | :--- | :--- | :--- | :--- | | CAS Full Model | 2.56 | 0.390 | 0.894 | Full \(L_{\text{oral}}\) injection + quadratic calibration | | w/o Calibration (fixed scalar) | 3.50 | 0.267 | 0.883 | -12.3 points drop without level conditioning | | w/o Multi-layer (single layer) | 2.49 | 0.141 | 0.893 | -24.9 points drop, degrading to ActAdd baseline | | Isotonic Regression (Iso) | 2.56 | 0.293 | 0.888 | -9.7 points drop due to saturation at high levels | | Random Layers (Rnd L) | 1.65 | 0.204 | 0.894 | -18.6 points drop from mislocated intervention sites | | Random Direction (Rnd v) | 3.31 | 0.163 | 0.893 | -22.7 points drop using random Gaussian vectors |
LAS Component & Parameterization Ablation (WSPairs Test Set): | Configuration | Params | L-Acc (↑) | BERTScore (↑) | Note | | :--- | :--- | :--- | :--- | :--- | | LAS Full Model | 119K | 0.584 | 0.911 | Oral layers + oral heads affine (\(d_c \odot h + b_c\)) | | Matched Random Sites (Rnd) | 119K | 0.507 | 0.912 | -7.7 points drop, confirming site specificity | | w/o Layer Intervention (-Lay.) | 53K | 0.397 | 0.900 | Severe drop (-18.7 points) without residual steering | | w/o Head Intervention (-Hd.) | 66K | 0.448 | 0.908 | -13.6 points drop, showing heads are critical | | w/o Scale Term (-Sc.) | 59K | 0.433 | 0.906 | -15.1 points drop without multiplicative scaling | | w/o Bias Term (-Bi.) | 59K | 0.457 | 0.906 | -12.7 points drop without additive translation | | Matched Scope LoRA (r=4) | 475K | 0.568 | 0.909 | 4× more parameters, trails LAS by 1.6 points | | Matched Scope \(\text{IA}^3\) | 59K | 0.248 | 0.900 | Severe under-control at high target levels | | Matched Scope ReFT (r=4) | 475K | 0.252 | 0.901 | Low-rank loop underperforms on stylistic rewriting |
Key Findings¶
- Dual-Site Affine Synergy is Indispensable: In LAS, disabling layer interventions causes an 18.7-point drop in L-Acc, while disabling head interventions degrades accuracy by 13.6 points. Dropping either the multiplicative scale \(d_c\) or additive bias \(b_c\) drops performance by 13–15 points, demonstrating that both baseline shifting and gain control are required simultaneously.
- Robust Out-of-Domain Generalization: In a leave-one-domain-out setting on Mistral-7B (training on 2 domains among MediTOD, SpokenWoZ, and Bargaining and testing on the third), LAS consistently achieves the highest L-Acc across all three folds, beating Full SFT by 0.9–3.9 points and outperforming the strongest PEFT baseline by 2.2–7.0 points. This confirms that the localized \(L_{\text{oral}} \cup A_{\text{oral}}\) circuit captures a universal linguistic rendering mechanism rather than domain-specific lexical artifacts.
- Superiority in Blind Human Evaluation: Across 900 test outputs evaluated by three NLP experts, LAS achieves the highest Spoken Naturalness (3.81/5) and the highest Overall Preference (25.7% win rate), while matching SFT in Semantic Preservation (4.82/5) and achieving the highest Named Entity Recognition F1 score (0.851 vs. 0.835 for SFT and 0.773 for \(\text{IA}^3\)). Crucially, LAS avoids the catastrophic repetition loops observed in SFT.
Highlights & Insights¶
- Surface-Level Styles Concentrate in Sparse Causal Circuits: Contrary to the common belief that stylistic register is diffusely distributed across LLM parameters, this work proves that orality rendering is causally concentrated in late layers (final ~40% of depth) and less than 8% of attention heads, expanding mechanistic interpretability from discrete lookup targets to continuous stylistic dimensions.
- Affine Modulation Trumps Low-Rank Projection: Rather than using complex orthogonalized low-rank projection loops (as in ReFT) or global weight rank updates (LoRA), zero-initialized element-wise affine scaling and translation (\(d_c \odot h + b_c\)) on causally verified components matches or outperforms full fine-tuning with 20,000× fewer parameters.
- Data-Driven Clustering Overcomes Psychometric Noise: By grounding orality in seven sociolinguistic features combined with Entropy Weighting and KMeans clustering, the authors translate a subjective, fuzzy concept into a reproducible, highly consistent 0–5 discrete ladder, bypassing the noise inherent to single-text Likert annotations.
Limitations & Future Work¶
- Language and Evaluator Scope: The 7-feature estimator \(g(\cdot)\) is tailored to English sociolinguistics. While it exhibits 91% agreement with human judges, it may fail to capture subtle pragmatic nuances, sociolects, non-native English varieties, or transfer to other languages without dedicated re-tuning.
- Absence of End-to-End Speech Synthesis Integration: Evaluation is strictly text-based. Acoustic properties (speech prosody, speech rate, pause durations, pitch contour) and the end-to-end interactive latency of real voice agents were not empirically examined in a live audio testbed.
- High-Level Non-Monotonicity in CAS: In weight-free CAS, the mean predicted level reaches 2.37 at Target 4 but dips slightly to 2.27 at Target 5 on Qwen-3B, indicating that static steering vectors suffer from non-linear saturation at extreme intensities.
Related Work & Insights¶
- vs. ActAdd (Turner et al., 2024): ActAdd introduces a fixed scalar difference vector at a single grid-searched layer for binary concept switching. CAS generalizes this by injecting across all localized oral layers and fitting a quadratic calibration curve across multiple discrete intensity levels.
- vs. ReFT (Wu et al., 2024): ReFT inserts low-rank read-write loops into hidden representations but struggles with high-level stylistic saturation. LAS restricts interventions to causally localized heads and layers using element-wise affine transforms, slashing parameter counts while achieving superior cross-domain generalization.
- vs. CS2W (Guo et al., 2023) / GYAFC (Rao & Tetreault, 2018): Earlier style-transfer benchmarks focused on spoken-to-written cleanup or binary formality classification on web crawl data. WSPairs establishes the reverse, deployment-aligned written-to-spoken benchmark with multi-level ordinal supervision based on authentic ASR transcripts.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering mechanistic localization of continuous spoken style in LLMs, introducing elegant weight-free (CAS) and ultra-lightweight affine (LAS) steering.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous causal layer/head attribution across 3 model backbones, comprehensive ablations, out-of-domain transfer, and blind expert human evaluations.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured and clear, bridging sociolinguistic theory with mechanistic interpretability and practical voice-agent engineering.
- Value: ⭐⭐⭐⭐⭐ Highly impactful for deploying responsive, natural, audience-adaptive voice agents with minimal serving overhead.