Energy-Based Transformers as Predictors of Reading Difficulty¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/jakdot/energy-transformers-reading-difficulty
Area: LLM Reasoning
Keywords: Energy-Based Transformers, Reading Difficulty, Surprisal Theory, Attention Entropy, Associative Memory
TL;DR¶
This paper introduces energy-based transformers (NRGPT) grounded in dense associative memory to computational psycholinguistics, demonstrating that a single scalar energy metric naturally unifies discrete next-token predictability and continuous memory integration effort, subsuming both surprisal and attention entropy across controlled relative-clause processing and three reading-time corpora.
Background & Motivation¶
Autoregressive transformer language models have become foundational computational tools for modeling human sentence processing and predicting behavioral reading times. Historically, the literature has relied on two distinct, complementary theoretical accounts to quantify cognitive processing load: surprisal theory, which posits that processing difficulty is proportional to the negative log-probability of the incoming word given its preceding context, and attention-based metrics such as attention entropy, which operationalize memory retrieval interference under cue-based memory architectures. However, these two metrics frequently account for disjoint aspects of cognitive difficulty, and as standard transformer models scale up in size, their surprisal estimates exhibit a well-documented empirical degradation in fitting human reading-time data.
This divergence presents two fundamental obstacles to developing unified mechanistic accounts of human language processing. First, the psycholinguistics community lacks an overarching, singular metric capable of capturing both predictive processing load and syntactic memory integration costs within a single formal framework. In standard relative-clause processing paradigms, for example, surprisal explains the processing penalty at the embedded subject noun phrase in object-relative clauses (ORCs) but completely fails at the embedded verb; conversely, attention entropy successfully captures retrieval difficulty at the embedded verb but cannot account for the slowdown at the embedded noun phrase. Second, while the conceptual link between cue-based retrieval memory and transformer self-attention is intuitively appealing, standard transformers lack a rigorous, formal connection to established associative memory architectures in cognitive science, compelling researchers to rely on heuristic interventions such as memory masking or artificial context truncation to induce human-like constraints.
To resolve this dichotomy, recent breakthroughs at the intersection of statistical physics and deep learning demonstrate that modern Hopfield networks and dense associative memory are mathematically equivalent to continuous energy-based formulation of self-attention. Energy-based transformers, exemplified by NRGPT, reformulate the forward pass as gradient descent on a scalar energy landscape defined over token states, where the scalar energy serves as the continuous analog of negative log transition probability while naturally embedding expected attention matching scores and attention entropy into its analytic decomposition. Core idea: formalize human reading difficulty directly as the scalar energy state of an energy-based transformer (NRGPT) along its gradient descent trajectory, thereby unifying prediction surprisal and associative memory integration into a single physical metric that provides the first principled formal bridge between modern associative memory models and human sentence processing.
Method¶
Overall Architecture¶
NRGPT recasts autoregressive sequence generation from a feedforward layer stack into iterative dynamical optimization along a scalar energy landscape \(E_A(g)\). Rather than passing token representations through heterogeneous layers with distinct parameters, NRGPT recurrently applies a single parameter-tied parallel transformer block across \(k\) descent iterations. The per-token scalar energy \(E_A(g)\) decomposes into an attention energy term \(E^{\text{AT}}_A(g)\) and a feedforward energy term \(E^{\text{FF}}_A(g)\), designed such that their negative partial gradients directly recover standard multi-head causal attention and multi-layer perceptron operations. For cognitive modeling, the token state starts at the preceding context embedding and proceeds downhill via gradient descent for \(k=6\) steps toward a low-energy attractor; the scalar energy value observed at the convergence plateau (specifically layer iterations 5 and 6) serves as the unified predictor of cognitive integration effort and human reading time.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Context Sequence<br/>Token Positions 1 ... A-1"] --> B["Parallel Energy Function Formulation<br/>Attention Energy E_AT + Feedforward Energy E_FF"]
B --> C["Dynamical Gradient Descent Iteration<br/>x^(t+1) = x^(t) - η (∂E/∂g)"]
C --> D["Unified Decomposition of Cognitive Metrics<br/>Expected Score -⟨s⟩_p Offsetting -H(p)"]
D --> E["Layer 5/6 Convergence Plateau Energy E_A<br/>Linear Mixed-Effects Reading Time Prediction"]
Key Designs¶
1. Parallel Energy Function Formulation: Exact Equivalence with Dense Associative Memory
Conventional transformers execute self-attention and feedforward layers sequentially with distinct layer normalizations, an architecture that prevents the derivation of a global scalar potential function. To establish an exact formal correspondence with modern Hopfield networks and dense associative memory, NRGPT adopts a parallel block structure where attention and MLP components operate concurrently on the normalized token state \(g_A = \text{LayerNorm}(x_A)\). The per-token energy is explicitly formulated as \(E_A(g) = E^{\text{AT}}_A(g) + E^{\text{FF}}_A(g)\), where the attention energy \(E^{\text{AT}}_A(g)\) represents a negated Log-Sum-Exp (LSE) over causal attention scores across all heads: $\(E^{\text{AT}}_A(g) = - \frac{1}{\beta} \sum_{h=1}^H \alpha_h \log \sum_{B < A} \exp \left( \beta g_B^\top (W_K^h)^\top W_Q^h g_A \right)\)$ The feedforward energy \(E^{\text{FF}}_A(g)\) is defined as the anti-derivative integral of the activation function over projected hidden states. Differentiating \(E_A(g)\) with respect to \(g_A\) precisely recovers standard causal multi-head attention and two-layer MLP transformations. This construction provides a rigorous physical energy foundation where processing a word corresponds directly to resolving associative memory patterns within high-dimensional cognitive basins.
2. Dynamical Gradient Descent Iteration: Layer Depth as Cognitive Integration Trajectory
Rather than assigning distinct representational duties to disparate physical layers, NRGPT iteratively updates the token representation through repeated gradient steps along the energy landscape: $\(x_A^{(t+1)} = x_A^{(t)} - \eta^{(t)} \frac{\partial E_A}{\partial g_A^{(t)}}\)$ where \(\eta^{(t)}\) denotes a learnable inference-rate matrix. Over \(k=6\) layer iterations, the token representation is pulled toward a low-energy attractor configuration, causing the vocabulary distribution projected by the unembedding language model head to settle into a coherent prediction. In computational psycholinguistics, this energy descent mirrors cognitive integration effort: words that integrate seamlessly into existing syntactico-semantic context lie in shallow, accessible energy basins, whereas incongruent, syntactically complex, or non-local dependencies reside on steep, high-energy slopes. Empirical trajectory analysis reveals that token energy stabilizes into a convergence plateau at layer 5, yielding the most diagnostic and theoretically sound predictor of human reading slowdowns.
3. Unified Decomposition of Cognitive Metrics: Reconciling Expected Matching and Attention Entropy
Surprisal theory (predictive probability) and attention entropy (retrieval dispersion) have historically been treated as unrelated phenomenological metrics. NRGPT's mathematical formulation reveals their deep formal connection. Since the attention distribution \(p\) is the causal softmax over raw dot-product scores \(s\), the Log-Sum-Exp function for an attention head can be decomposed into the expected attention score plus the Shannon entropy: $\(\text{LSE}(s) = \langle s \rangle_p + H(p) = \sum_{B < A} p_B s_B - \sum_{B < A} p_B \log p_B\)$ Consequently, the attention energy \(E^{\text{AT}}_A\) decomposes into two competing terms: the negative expected matching score \(-\langle s \rangle_p\) (measuring how well attended context keys match the current query) and the negated attention entropy \(-H(p)\) (measuring the dispersion of retrieval cues). When syntactic integration is difficult, the lack of well-matching antecedent keys yields a low expected score, driving \(-\langle s \rangle_p\) high and increasing overall energy; concurrently, broad attention dispersion increases \(H(p)\), which acts with a negative sign. Quantitative component breakdowns demonstrate that the magnitude of the expected score penalty heavily dominates and offsets the negative entropy term, enabling scalar energy to naturally subsume both expectation-based surprisal and memory retrieval costs into a single cohesive metric.
Loss & Training¶
The underlying model utilizes the NRGPT_H_FF2W variant with 128M parameters, 12 attention heads, hidden embedding dimension 1,536, context window length 1,024 tokens, and 6 recurrent layer iterations. It was pretrained on OpenWebText using the AdamW optimizer (\(\beta_1=0.9, \beta_2=0.99\), learning rate \(3 \times 10^{-5}\) decaying via cosine schedule to \(3 \times 10^{-6}\), warmup 2,000 steps, weight decay 0.01). The authors evaluate both the publicly released 500,000-iteration checkpoint and an in-house checkpoint trained for 100,000 iterations (requiring ~60 hours on 4× NVIDIA H100 GPUs), confirming that all key psycholinguistic predictive advantages replicate fully under moderate, accessible training budgets.
Key Experimental Results¶
Main Results¶
The authors evaluate linear mixed-effects models predicting log-transformed reading times across three major psycholinguistic benchmarks: the Natural Stories Corpus (NSC, self-paced reading, 704k observations), the UCL eye-tracking corpus (UCL et, go-past reading time, 37k observations), and the UCL self-paced reading corpus (UCL spr, 218k observations). Performance is measured by the change in log-likelihood (\(\Delta\text{LL}\)) over a baseline model containing word length, unigram/bigram frequency, sentence position, and subject/story random intercepts:
| Model Predictors | NSC (spr) \(\Delta\text{LL}\) | NSC Sign | UCL (et) \(\Delta\text{LL}\) | UCL (et) Sign | UCL (spr) \(\Delta\text{LL}\) | UCL (spr) Sign |
|---|---|---|---|---|---|---|
| Surprisal (NRGPT) | 552 | + | 155 | + | 1 | 0 |
| Surprisal-GPT2 | 561 | + | 153 | + | 4 | - |
| Energy (L5) | 203 | + | 8 | + | 28 | + |
| Energy (L6) | 6 | - | 386 | + | 5 | - |
| Surprisal + Energy (L5) | 687 | +, + | 166 | +, + | 28 | 0, + |
| Surprisal + Energy (L6) | 557 | +, - | 487 | +, + | 5 | 0, - |
| Surprisal-GPT2 + Energy (L5) | 695 | +, + | 166 | +, + | 30 | -, + |
| Surprisal-GPT2 + Energy (L6) | 568 | +, - | 458 | +, + | 7 | -, - |
| Entropy (L5) | 2 | + | 82 | - | 39 | + |
| Surprisal + Entropy (L5) | 553 | +, 0 | 246 | +, - | 40 | 0, + |
| Entropy (L5) + Energy (L5) | 213 | -, + | 136 | -, + | 47 | +, + |
| Entropy-GPT2 (all layers) | 90 | - | 421 | - | 18 | + |
| Surprisal + Entropy-GPT2 (all) | 673 | +, - | 540 | +, - | 18 | 0, + |
Note: Higher \(\Delta\text{LL}\) indicates superior empirical fit; +/-/0 indicates whether the fixed-effect coefficient is significantly positive, negative, or non-significant (\(|t| > 1.96\)). Cognitive theories anticipate positive correlations (+) with reading difficulty. While Entropy-GPT2 displays large \(\Delta\text{LL}\) on UCL (et), its negative sign violates theoretical tenets; models incorporating Energy achieve the best fit with theoretically consistent positive signs.
Relative Clause Processing & Component Decomposition¶
In controlled experiments using 24 sentence pairs from Staub (2010) contrasting subject-extracted relative clauses (SRC) and object-extracted relative clauses (ORC), the authors evaluate reading difficulty across the embedded verb (e.g., called) and the embedded noun phrase onset (e.g., the):
| Critical Evaluation Locus | Metric / Predictor | Difference (ORC - SRC) | Statistic (\(t\)-value) | Significance (\(p\)-value) | Theoretical Consistency |
|---|---|---|---|---|---|
| Embedded Verb (called) | Surprisal (NRGPT) | \(\approx 0\) | \(t = 0.1\) | \(p > 0.1\) | ❌ Fails to detect verb difficulty |
| Embedded Verb (called) | Attention Entropy (L5) | +3.9 | \(t = 12.8\) | \(p \ll 0.001\) | ✅ Predicts verb retrieval load |
| Embedded Verb (called) | Energy (L5) | +55.0 units | \(t = 15.3\) | \(p \ll 0.001\) | ✅ Highly significant predicted difficulty |
| Verb Decomposition | \(-\langle s \rangle_p\) (neg. expected score) | -50.6 | \(t = -9.3\) | \(p \ll 0.001\) | Primary driver of positive energy |
| Verb Decomposition | \(E^{\text{FF}}\) (feedforward energy) | +8.7 | \(t = 2.7\) | \(p < 0.01\) | Statistically significant co-driver |
| Noun Phrase Onset (the) | Surprisal (NRGPT) | +4.5 nats | \(t = 15.8\) | \(p \ll 0.001\) | ✅ Predicts structural disconfirmation |
| Noun Phrase Onset (the) | Attention Entropy (L5) | -0.06 | \(t = -0.3\) | \(p > 0.1\) | ❌ Completely misses subject cost |
| Noun Phrase Onset (the) | Energy (L5) | +16.7 units | \(t = 4.7\) | \(p < 0.001\) | ✅ Successfully detects subject cost |
| Noun Onset Decomposition | \(-\langle s \rangle_p\) (neg. expected score) | -15.5 | \(t = -2.8\) | \(p < 0.01\) | Drives positive energy difference |
Key Findings¶
- Dual-Locus Cognitive Subsumption: In relative-clause constructions, surprisal accounts exclusively for the noun phrase onset (\(t=15.8\)) and is blind to the embedded verb (\(t=0.1\)); conversely, attention entropy detects only the embedded verb (\(t=12.8\)) and is insensitive to the noun phrase (\(t=-0.3\)). Layer 5 energy is statistically significant at both critical loci (\(t=4.7\) and \(t=15.3\), respectively), proving that energy unifies expectation-based and memory-based cognitive bottlenecks within a single metric.
- Superior Broad-Coverage Fit Across Corpora: Adding NRGPT energy to baseline surprisal yields massive increases in log-likelihood across all three naturalistic reading corpora (\(\Delta\text{LL}\) surges from 552 to 687 in NSC, and from 155 to 487 in UCL et). Crucially, energy maintains the theoretically expected positive sign across paradigms where attention entropy frequently collapses into uninterpretable negative coefficients.
- Dynamical Convergence and Layer Sensitivity: Energy descent exhibits a pronounced deceleration and stabilization at layer 5, where the cognitive alignment with human reading times reaches its peak. At layer 6, unconstrained descent dynamics occasionally introduce erratic fluctuations, underscoring that intermediate convergence plateaus offer the most faithful representation of human online processing.
Highlights & Insights¶
- First Formal Link Between Modern Associative Memory and Psycholinguistic Processing: Moves computational psycholinguistics beyond treating transformer attention as a heuristic cue-retrieval proxy, grounding sentence processing directly in the physical energy minimization principles of modern Hopfield networks.
- Mathematical Resolution of the Attention Entropy Sign Paradox: Unpacks the Log-Sum-Exp attention energy into expected score and negative entropy terms, providing empirical proof that expected attention matching dominates negative entropy in magnitude, thereby resolving why higher cognitive difficulty translates to elevated scalar energy.
- High Resource Efficiency and Reproducibility: Demonstrates that all core psycholinguistic findings replicate robustly on a model trained for only 100k steps (~60 hours on 4× H100s), establishing an accessible and reproducible protocol for researchers exploring energy-based cognitive architectures.
Limitations & Future Work¶
- Architectural Scope and Scaling Constraints: Evaluated exclusively on a single 128M parameter NRGPT configuration; future work must establish whether the observed energy predictive power persists across billions-scale architectures and alternative energy-based formulations.
- Behavioral Modality Boundaries: Confined strictly to behavioral self-paced reading and eye-tracking fixations in English; extensive cross-lingual validation and evaluation against continuous neural time-series data (e.g., EEG event-related potentials like N400/P600 and fMRI hemodynamic response) are essential next steps.
- Layer Selection Sensitivity: Optimal predictive performance alternates between iteration 5 (for self-paced reading) and iteration 6 (for eye-tracking); developing dynamic, adaptive stopping criteria such as Time-to-Convergence could provide a more flexible, continuous measure of processing effort.
Related Work & Insights¶
- vs. Surprisal Theory (Hale 2001, Levy 2008): Surprisal scores discrete lexical prediction probabilities under an output softmax, overlooking the continuous representational work required to resolve non-local syntactic dependencies; NRGPT energy captures continuous trajectory integration effort, explaining why predictable tokens can still incur severe reading delays.
- vs. Cue-Based Retrieval & Attention Entropy (Lewis & Vasishth 2005, Ryu & Lewis 2025): Attention entropy measures retrieval dispersion but often yields theoretically inconsistent negative coefficients in eye-tracking corpora; NRGPT energy integrates expected dot-product scores, enforcing correct directional alignment with cognitive reading times.
- vs. Lossy-Context Surprisal (Futrell & Levy 2017, Hahn et al. 2022): Prior attempts to capture memory limits artificially inject noise or memory decay into the context window; NRGPT generates human-like memory bottleneck phenomena endogenously from the attractor dynamics of dense associative memory without degrading model representations.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering application of energy-based transformers and dense associative memory to computational psycholinguistics]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluation spanning three naturalistic reading corpora, controlled relative-clause experiments, and detailed mathematical decompositions]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity in synthesizing machine learning theory, mathematical proofs, and psycholinguistic hypotheses]
- Value: ⭐⭐⭐⭐⭐ [Establishes a transformative, unified framework linking statistical physics, memory architectures, and human language processing]