The Anatomy of an ASR Hallucination¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/nvidia/stt_en_conformer_ctc_large
Area: LLM Reasoning
Keywords: ASR Hallucination, Mechanistic Interpretability, Grounding Failure, Conformer Encoder, 4-Gram Divergence Rate
TL;DR¶
This paper provides a mechanistic interpretability analysis of speech recognition hallucinations under environmental and accent shifts across Conformer CTC and RNN-T architectures, revealing that the terminal encoder stage (final 1–2 blocks) serves as a critical boundary where acoustic signals transition into decoder-readable symbolic tokens, establishing that grounding failure is a causal prerequisite rather than a sufficient cause for fluent fabrication.
Background & Motivation¶
Automatic speech recognition (ASR) systems deployed in real-world environments occasionally emit syntactically fluent, coherent transcriptions that bear no semantic relationship to the actual acoustic stream, particularly under environmental degradations such as codec distortion, reverberation, and noise gaps. In speech processing, these ungrounded transcriptions are conventionally classified as ASR hallucinations. Unlike hallucination in text-based large language models where generation drifts across discrete token spaces, speech hallucinations occur across a continuous-to-discrete cross-modal mapping. Dominant production architectures—stateless Connectionist Temporal Classification (CTC) models and autoregressive Recurrent Neural Network Transducers (RNN-T)—have traditionally been assumed to suffer from decoder-side internal language model over-reliance, yet a rigorous causal explanation of where along the continuous acoustic processing pipeline grounding is actually severed has remained elusive.
Existing mitigation techniques concentrate almost exclusively on the decoder, relying on internal language model estimation (ILME), hybrid autoregressive transducers (HAT), lookahead decoding, or adaptive language model discounting to suppress implicit linguistic bias. While intuitive, treating the predictor or joint network as the sole culprit overlooks the representation dynamics within the acoustic encoder. Furthermore, prior diagnostic benchmarks frequently adopt rule-based divergence metrics—such as 4-gram divergence rate (DR) or token insertion thresholds—as direct proxies for semantic hallucinations, failing to separate non-fluent degradation phenotypes (e.g., repetition collapse or garbled acoustic echoes) from genuine, fluent off-topic fabrications.
Addressing this fundamental gap, this work reframes ASR hallucinations as one particular downstream manifestation of a broader mechanistic phenomenon: acoustic grounding failure. By investigating two independently trained, industrial-grade Conformer-Large models (one CTC with 18 blocks and one RNN-T with 17 blocks) across both environmental distortions (WildASR) and native-language accent shifts (L2-ARCTIC), the authors construct an integrated diagnostic framework combining causal skip-layer interventions, encoder logit lens decoding, representation rank geometry, and strict LLM-judge output taxonomy. Core idea: deconstruct ASR hallucinations into grounding failure phenotypes and establish through causal skip-layer ablation and streaming effective rank geometry that the terminal one to two encoder blocks constitute the critical boundary for acoustic-to-symbolic conversion and head-readability, demonstrating that loss of acoustic grounding is a necessary mechanistic precondition rather than a sufficient trigger for fluent fabrication.
Method¶
Overall Architecture¶
The methodological pipeline aims to causally locate where acoustic grounding fails within the encoder and determine the exact semantic nature of the resulting decoding output. The study analyzes two independently trained NVIDIA NeMo Conformer-Large checkpoints (hidden dimension 512, 8 attention heads, 4× convolutional subsampling with a 40 ms frame stride, ~120M parameters each): one with a 1×1 convolutional projection to a 129-class vocabulary (18 encoder blocks, L=0..17), and one with an LSTM predictor, joint network, and linear projection to a 1,025-class vocabulary (17 encoder blocks, L=0..16). The evaluation spans the 7 conditions of WildASR and the 6 native-language backgrounds of L2-ARCTIC.
The analysis is structured into four complementary methodological phases: first, causal skip-layer intervention is performed across all encoder depths to identify essential grounding boundaries; second, the encoder lens probes the point at which intermediate hidden representations become interpretable to the trained frozen decoder head; third, layer-wise linear probing tracks the decay of continuous acoustic traits against the rise of discrete symbolic grapheme identity; finally, streaming effective rank geometry and a five-way LLM-judge taxonomy categorize the output phenotypes to verify whether causal intervention directly induces semantic fabrications.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Acoustic Signal<br/>16 kHz waveform / 4x conv subsampling"] --> B["Skip-Layer Causal Ablation<br/>SkipL: hL ← hL-1 isolates critical blocks"]
B --> C["Encoder Lens Decoding<br/>Frozen decoder head probes intermediate hL"]
C --> D["Acoustic-to-Symbolic Probes<br/>Acoustic noise decay vs grapheme accuracy surge"]
D --> E["Streaming Effective Rank Geometry<br/>Spectral entropy drop: -18% to -44%"]
E --> F["Five-Way Semantic Phenotype Judge<br/>Disentangles fluent fabrication from collapse"]
Key Designs¶
1. Skip-Layer Causal Ablation: Identifying the Causal Boundary of Grounding Failure
To establish the functional necessity of individual encoder blocks without inducing dimension incompatibility or global attention breakdown, the authors reject uninformative zero or mean replacement and deploy residual skip-layer interventions:
$\(\mathbf{h}_L \leftarrow \mathbf{h}_{L-1}\)$
By attaching paired hooks around block \(L\), the transformation of that layer is bypassed while downstream layers receive properly normalized inputs within the expected residual scale. Systematic scanning over all encoder layers exposes three distinct operational regimes: early blocks ($L=0\text}1$) degrade acoustic feature fidelity under severe noise (e.g., far-field \(\Delta\text{WER}\) jumps +69 to +72 pp) but barely affect divergence rate (\(\Delta\text{DR}\) mean +6 to +10 pp); the middle band ($L=2\text{13$) exhibits deep functional redundancy, where bypassing any individual block changes WER by only 1–7 pp and keeps \(\Delta\text{DR} \le 8.5\text{ pp}\); in stark contrast, bypassing the final block (\(L=17\) for CTC, \(L=16\) for RNN-T) triggers a catastrophic collapse, causing divergence rates to explode by +94 to +99 pp in CTC and +42 to +88 pp in RNN-T, accompanied by average WER increases exceeding 90 pp. This identifies the terminal stage as causally indispensable for maintaining acoustic grounding.
2. Encoder Lens Decoding: Tracing the Emergence of Decoder-Readable Representations
To determine the exact network depth at which representations become compatible with the model's output decoder, the authors adapt the logit lens paradigm to ASR architectures. Denoting the post-LayerNorm hidden state at layer \(L\) as \(\mathbf{h}_L(x)\) and the decoder head mapping as \(\text{head}(\cdot)\), intermediate representations are fed directly into the decoding pipeline: $\(\text{LensWER}(L) = \text{WER}(\text{head}(\mathbf{h}_L(x)), r)\)$ In CTC, the classification head operates directly on hidden states, whereas in RNN-T, NeMo's CUDA-graph decoding is deactivated to hook representations directly into the predictor-joint interface. The lens unveils an abrupt operational discontinuity: prior to the final block, the CTC lens outputs degenerate, severely over-emitted text with WERs ranging from 108% to 515% and DR exceeding 97%, collapsing precisely to the production baseline decode WER at \(L=17\) (matching 7.85% on clean audio to two decimal places); RNN-T outputs near-total deletion errors across early layers before rapidly transitioning to usable transcriptions exclusively at \(L=16\) (WER 9.3% to 25.6%). This confirms that head-readable textual structure is not progressively formed across the backbone, but crystallizes abruptly at the final encoder stage.
3. Acoustic-to-Symbolic Probing: Decoupling Physical Distortions from Symbolic Identities
To characterize the informational transformation executed by the terminal stage, single linear probe layers are trained on frozen activations across all depths. Six-way environmental shift-type classification accuracy peaks immediately at \(L=1\) (reaching 0.978 in CTC and 0.981 in RNN-T) before decaying steadily to 0.38–0.42 at the final layer, mirroring a continuous drop in SNR regression \(R^2\) from ~0.58 to ~0.10. Conversely, frame-level grapheme identity probing (using the CTC model's final-layer argmax as a frame-aligned proxy) rises across depth and undergoes a pronounced surge across the final layers: in CTC, grapheme probe accuracy jumps from 0.811 at \(L=15\) to 0.919 at \(L=16\) and 0.990 at \(L=17\). The terminal stage thus completes the acoustic-to-symbolic hand-off, discarding raw channel distortions in favor of discrete linguistic targets.
4. Effective Rank Geometry and Five-Way Semantic Phenotyping: Separating Preconditions from Fabrications
To evaluate representational geometry, the authors compute the spectral entropy-based dimensionless effective rank (ER) over a streaming Gram matrix of all audio frames:
$\(\text{ER} = \exp\left(-\sum_{i} p_i \log p_i\right), \quad p_i = \frac{\lambda_i}{\sum_j \lambda_j}\)$
Both architectures exhibit an identical geometric trajectory: a gradual ramp over $L=0\text}4$, a rise across the middle band peaking at \(L=15\) (CTC 186, RNN-T 246 on clean audio), and an abrupt terminal drop of 18% to 44% across the final 1–2 blocks. To determine whether the resulting divergence corresponds to semantic hallucinations, 354 outputs under skip-layer and predictor interventions were evaluated by Gemini 2.5 Pro across five categories (fluent off-topic fabrication, on-topic recognition error, repetition/collapse, empty output, garbled non-language). Crucially, 0 out of 354 intervention outputs were judged as fluent fabrications: CTC final-block skip produced 87% garbled non-language fragments, while RNN-T final-block skip produced 98% repetition loops. Consequently, terminal grounding failure is established as a strictly necessary precondition for hallucination, rather than a sufficient cause of fluent verbal fabrication.
Benchmark & Evaluation Protocol¶
All models were evaluated on the complete 10,058 utterances of WildASR (clean, clipping, phone codec, noise gap, far field, reverberation, accent) and the 3,599 utterances of L2-ARCTIC across 6 native language backgrounds. Text was normalized using the standard Whisper EnglishTextNormalizer. Causal skip-layer and encoder lens evaluations were benchmarked on controlled 200-utterance subsets per condition, with baseline metrics strictly recomputed on identical subsets to ensure internal validity.
Key Experimental Results¶
Main Results¶
Baseline decoding across the full splits of WildASR highlights error distributions, rule-based 4-gram divergence rates (DR), and inter-model hypothesis disagreement between CTC and RNN-T.
| Condition | N | Duration (min) | CTC WER (%) | RNN-T WER (%) | CTC DR (%) | RNN-T DR (%) | Disagreement (%) |
|---|---|---|---|---|---|---|---|
| clean | 647 | 107.5 | 7.86 | 7.14 | 0.31 | 0.46 | 66.9 |
| clipping | 647 | 107.5 | 17.96 | 15.76 | 1.55 | 0.77 | 92.4 |
| phone_codec | 1294 | 215.1 | 9.19 | 7.41 | 0.46 | 0.46 | 73.7 |
| noise_gap | 2588 | 481.9 | 17.26 | 11.90 | 2.20 | 1.24 | 95.3 |
| far_field | 1941 | 405.1 | 16.24 | 14.48 | 2.01 | 0.88 | 90.4 |
| reverberation | 1941 | 403.6 | 14.82 | 13.00 | 2.58 | 1.44 | 84.8 |
| accent | 1000 | 57.9 | 5.65 | 7.16 | 2.10 | 2.50 | 40.8 |
Ablation & Causal Intervention Studies¶
In the RNN-T predictor batch-mean ablation, the authors uncover how corpus ordering in evaluation batches creates significant content leakage.
| Condition | Unique Ref/Batch | Baseline DR (%) | Baseline WER (%) | Contiguous DR (%) | Contiguous WER (%) | Shuffled DR (%) | Shuffled WER (%) |
|---|---|---|---|---|---|---|---|
| clean | 4 | 0.5 | 7.0 | 86.5 | 440.0 | 86.0 | 433.0 |
| accent | 4 | 1.0 | 6.1 | 85.0 | 470.0 | 83.0 | 458.0 |
| phone_codec | 4 | 0.5 | 7.2 | 90.5 | 463.0 | 88.0 | 456.0 |
| clipping | 4 | 1.0 | 17.4 | 91.0 | 381.0 | 91.5 | 417.0 |
| noise_gap | 1 | 0.0 | 12.1 | 6.0 | 40.0 | 89.0 | 512.0 |
| far_field | 2 | 0.0 | 15.8 | 32.0 | 113.0 | 84.0 | 367.0 |
| reverberation | 2 | 0.5 | 13.1 | 37.0 | 104.0 | 86.5 | 376.0 |
Across dimensionality estimators at peak (\(L=15\)) versus final layers on clean audio, CTC ER drops from 186 to 137 while TwoNN intrinsic dimension increases from 9.0 to 12.3; RNN-T ER drops from 246 to 200 while TwoNN rises from 9.1 to 11.0. This dissociation confirms that terminal compression is a linear variance-anisotropic event rather than an intrinsic manifold collapse.
Key Findings¶
- The Terminal Encoder Stage is a Functional Cliff: Middle encoder blocks ($L=2\text{13\() tolerate single-block skips with negligible impact (\)\Delta\text{DR} \le 8.5\text{ pp}$), whereas skipping the final block induces near-complete divergence (+94 to +99 pp in CTC, +42 to +88 pp in RNN-T), identifying it as the critical locus for acoustic grounding.
- Rule-Based Divergence Conflates Error Phenotypes: Deterministic 4-gram divergence rules exhibit extreme false-positive rates when evaluated against strict semantic definitions; only 0.5% (CTC 1/185) and 6.9% (RNN-T 8/116) of flagged baseline utterances represent genuine fluent off-topic fabrications.
- Batch Composition Confound in Predictor Interventions: Under corpus-order batching, consecutive degraded variants of the same reference sentence leak the target content into the batch mean, masking predictor ablation severity (noise_gap DR appeared as 6% in corpus order vs. 89% when shuffled).
Highlights & Insights¶
- Conceptual Disentanglement of Grounding Failure and Hallucination: Demonstrates that the breakdown of acoustic grounding within the encoder yields degenerate acoustic echoes or repetition collapse; fluent off-topic fabrication requires active linguistic generation from downstream autoregressive components.
- Divergent Geometric Fingerprints of Environmental vs. Accent Shifts: Uncovers that spectral distortions (reverberation, far-field) sharply contract representation rank (ER gap of 37–86) and elevate cross-model CKA similarity, whereas L1 accent shifts leave effective rank virtually unchanged despite remaining linearly decodable deep into the network.
- Methodological Auditing of Batch-Averaged Ablations: Illustrates how naive batch-level interventions inherit hidden data dependencies from benchmark layout, providing an essential cautionary precedent for representation engineering in sequence models.
Limitations & Future Work¶
- Restricted Model Architecture Diversity: Experiments are conducted on two Conformer-Large checkpoints from NVIDIA NeMo, leaving empirical verification across Whisper-style encoder-decoder models or emerging speech LLMs for future study.
- Dependence on Commercial LLM Judges: Semantic output taxonomy relies entirely on Gemini 2.5 Pro; while prompt templates and self-reported confidences are documented, large-scale blinded human validation remains unperformed.
- Coarse-Grained Layer-Level Interventions: Causal attribution relies on full-block residual bypassing, leaving sub-component attention head pruning and targeted activation patching within the final block unexplored.
Related Work & Insights¶
- vs Whisper Mechanistic Interpretability (Glazer et al., 2026): While prior work analyzes attention drift in autoregressive encoder-decoder Whisper models, this paper delivers the first layer-resolved causal account for streaming CTC and transducer models dominating production ASR.
- vs Decoder-Side LM Mitigation (Meng et al., 2021; Unni et al., 2022): Conventional strategies focus solely on dampening predictor language model bias; this work demonstrates that the terminal encoder bottleneck is equally decisive in preventing downstream collapse.
- vs Diagnostic Hallucination Metrics (Atwany et al., 2025): Demonstrates through empirical LLM-judge validation that surface-level 4-gram divergence rules misclassify minor substitutions and repetition loops as hallucinations, calling for metric refinement.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes a rigorous mechanistic and geometric foundation for acoustic grounding in speech recognition.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive synthesis of causal ablation, logit lens probing, streaming rank metrics, and semantic taxonomy.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally precise terminology, meticulous disclosure of experimental caveats, and transparent reporting of statistical bounds.
- Value: ⭐⭐⭐⭐⭐ Provides critical insights for ASR robustness engineering, efficient model pruning, and trustworthy hallucination benchmarking.