Skip to content

Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Interpretability
Keywords: Agreement Attraction, Surprisal Theory, Cognitive Modeling, Syntactic Processing, Transformer Interpretability

TL;DR

Evaluating eleven autoregressive Transformers across comprehensive agreement attraction paradigms using surprisal theory reveals that while models align with human reading times in prepositional phrase modifiers, their predictions diverge sharply and contradict human cognitive asymmetries in object-extracted relative clauses, challenging the view that Transformers are faithful cognitive models of human morphosyntax.

Background & Motivation

Autoregressive large language models have established undisputed dominance across computational linguistics and practical NLP benchmarks, sparking intense interest in cognitive science and psycholinguistics: to what extent do deep neural networks trained strictly on next-token prediction acquire the inductive biases and cognitive mechanisms of human sentence processing? Historically, recurrent neural architectures (LSTMs) showed notable success in mirroring human reading time dynamics on agreement attractionโ€”a classic psycholinguistic phenomenon where processing an ungrammatical verb is paradoxically facilitated by an intervening grammatically unrelated noun bearing the required number feature. In human sentence comprehension, this effect is widely explained by cue-based retrieval over a capacity-limited working memory, exhibiting crisp cognitive asymmetries (e.g., robust facilitation for ungrammatical singular subjects, but absence of attraction in grammatical sentences or plural subjects).

However, existing psycholinguistic evaluations of Transformer-based models suffer from critical methodological bottlenecks. Prior studies evaluated almost exclusively prepositional phrase (PP) modifier constructions while focusing narrowly on configurations where attraction is empirically observed, leaving untested whether models properly refrain from predicting attraction where humans exhibit null effects. Furthermore, most evaluations investigated isolated single models (typically GPT-2) or psychologically implausible bidirectional architectures (BERT), preventing general conclusions regarding whether observed effects stem from general Transformer inductive biases or isolated idiosyncratic artifacts.

To resolve these tensions, this paper undertakes a systematic, fully balanced empirical sweep across syntactic constructions, model families, and theoretical configurations. Core idea: using negative log-probability surprisal as a cognitive linking function, evaluate eleven autoregressive Transformers across four model families on complete 2ร—2ร—2 factorial designs for both prepositional phrase modifiers and object-extracted relative clauses, revealing structural prediction divergences and cognitive breakdowns in complex syntax.

Method

Overall Architecture

The experimental methodology establishes an end-to-end psycholinguistic evaluation pipeline. Using controlled linguistic stimuli drawn from seminal human self-paced reading (SPR) benchmarks (Wagers et al., 2009), sentences are processed strictly left-to-right by autoregressive language models without downstream task fine-tuning. Model-derived token surprisals at the critical verb position serve as linear predictors of human reading times under surprisal theory. The resulting surprisal scores are subsequently evaluated across three levels of statistical modeling: pairwise condition t-tests for each model, per-configuration linear regressions testing cognitive asymmetries at the population level, and global-level mixed-effects models estimating parameter scaling interactions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Psycholinguistic Stimuli Input<br/>PP Modifiers (192 sents) / ORC (384 sents)"] --> B["Autoregressive Causal Inference<br/>11 Pre-trained Models (124M-4B)"]
    B --> C["Critical Verb Surprisal Measurement<br/>Negative Log2 Probability Aggregation"]
    C --> D["Three-Tier Statistical Modeling<br/>Paired t-tests + Fine Regressions + Global LMM"]
    D --> E["Cognitive Fidelity Assessment<br/>PP Alignment vs. ORC Divergence"]

Key Designs

1. Surprisal Linking Mechanism: Bridging Autoregressive Probability to Processing Effort

To bridge model token probabilities with human reading times in self-paced reading, the study relies on Hale and Levy's surprisal theory. Under this framework, processing effort at token \(w_i\) is linearly proportional to its negative log probability conditioned on the preceding causal context \(w_0 \dots w_{i-1}\):

\[\text{surprisal}(w_i) = -\log_2 P(w_i \mid w_0 \dots w_{i-1})\]

When tokenizers split the critical verb into multiple sub-word units, the total surprisal is computed by summing the constituent token surprisals. Because the evaluation stimuli utilize common, high-frequency verbs, tokenization segmentation penalties remain minimal and do not confound cross-model comparisons.

2. 2ร—2ร—2 Full Factorial Design: Orthogonal Coverage of Syntactic Configurations

Rather than examining isolated attraction configurations, the study executes a full \(2 \times 2 \times 2\) factorial manipulation across two foundational syntactic constructions: - Prepositional Modifier Construction (Experiment 1: PP): The subject-verb dependency resides in the main clause, with the attractor noun embedded in an intervening prepositional phrase (e.g., The slogan on the poster(s) unsurprisingly was/were designed...). - Object-Extracted Relative Clauses (Experiment 2: ORC): The subject-verb dependency resides inside the relative clause, and the main clause subject acts as the preceding attractor noun (e.g., The marine(s) who the officer(s) want(s) to promote...).

Across both experiments, factors manipulate subject number (singular vs. plural), grammaticality (grammatical vs. ungrammatical), and attractor number mismatch (no attraction vs. attraction). This enables testing whether models reproduce human behavioral asymmetries: (a) singular-plural asymmetry (attraction present for singular subjects, attenuated/absent for plurals) and (b) grammaticality asymmetry (facilitation in ungrammatical sentences, absent in grammatical sentences).

3. Multi-Family Population Modeling and Mixed-Effects Regression: Eliminating Model Idiosyncrasy

To prevent spurious generalizations from a single architecture, the evaluation spans 11 foundation models from four families: monolingual models (OpenAI-GPT, GPT-2 base/large) and multilingual models (Bloom 560M/1.7B/3B, XGLM 564M/1.7B/2.9B, Gemma-3 1B/4B), ranging from 124M to 4B parameters.

The analytical framework operates across three distinct statistical tiers: 1. Paired t-tests: Conducted per model across all theoretically plausible contrast pairs to detect significant facilitatory (negative) or inhibitory (positive) surprisal shifts; 2. Per-configuration Regressions: Linear models with crossed random intercepts for items and models, testing population-level presence of singular-plural and grammaticality asymmetries; 3. Global-level Linear Mixed Models: A unified regression model incorporating centered \(\log_{10}\) parameter count (\(n_{par}\)), condition factors, and full higher-order interactions to examine parameter scaling dynamics.

Loss & Training

The evaluation solely inspects off-the-shelf pre-trained foundation models trained on causal next-token language modeling objectives without instruction fine-tuning or task-specific supervised training.

Key Experimental Results

Main Results

Mean critical verb surprisal and estimated attraction effects across syntactic constructions:

Syntactic Construction Stimuli / Observations Grammatical Mean Surprisal Ungrammatical Mean Surprisal Singular Ungrammatical Attraction (\(\hat{\beta}\)) Human Alignment
PP-Modifier (Exp 1) 192 sents (24 sets) / 1056 obs 5.5 bits 11.3 bits -2.38 bits (\(t=-22.92^{***}\)) High Alignment
ORC (Exp 2) 384 sents (48 sets) / 2112 obs 11.2 bits 16.5 bits -1.03 bits (\(t=-5.27^{***}\)) Severe Divergence / Failure

Detailed Regression Analysis on Cognitive Asymmetries

Experiment & Configuration Subset Predictor / Interaction Term Estimate (\(\hat{\beta}\)) Std. Error \(t\) value Finding & Psycholinguistic Interpretation
Exp 1 (PP) Singular Subject ungram (ungrammatical base) +7.2202 0.1076 67.08 Ungrammaticality incurs massive surprisal penalty
Exp 1 (PP) Singular Subject ungram \(\times\) attr -2.2620 0.1522 -14.86 Robust human-like attraction facilitation effect
Exp 1 (PP) Plural Subject ungram \(\times\) attr -0.3453 0.1249 -2.76 Attraction effect sharply attenuated for plural subjects
Exp 2 (ORC) Singular RC Subject ungram \(\times\) attr -1.0312 0.1958 -5.27 Modest ungrammatical attraction effect
Exp 2 (ORC) Plural RC Subject attr (grammatical attraction) -0.8139 0.1140 -7.14 Spurious facilitation: unexplained attraction in grammatical sentences
Exp 2 (ORC) Plural RC Subject ungram \(\times\) attr +0.2673 0.1613 +1.66 Interaction non-significant; fails human grammaticality asymmetry

Key Findings

  • High Consistency in Prepositional Modifiers: In PP modifiers, all 11 models faithfully produce the signature human-observed agreement attraction effect in ungrammatical singular contexts (average 2.4 bits surprisal reduction) while correctly refraining from attraction in grammatical conditions.
  • Severe Prediction Divergence in Relative Clauses: In object-extracted relative clauses, model predictions diverge wildly from human data and across model families. Almost all models predict a spurious facilitatory effect in grammatical plural contexts where human comprehension shows no facilitation or slight inhibition. Furthermore, models from the XGLM family fail entirely to predict attraction even in the classic singular ungrammatical condition.
  • Parameter Scaling Does Not Cure Cognitive Divergence: Global mixed-effects regressions confirm that while increasing parameter size (\(n_{par}\)) universally lowers average surprisal and mildly sharpens grammaticality discrimination, it does not alleviate the spurious predictions in ORCs. Larger multilingual models (Gemma-3 4B) exhibit even larger divergences from human baselines than smaller monolingual models.

Highlights & Insights

  • Debunking Single-Construction Generalizations: The stark contrast between PP modifiers and relative clauses demonstrates that validating a computational model on an isolated, simple syntactic frame risks severe confirmation bias.
  • Marr's Levels of Analysis Disconnect: While humans and autoregressive Transformers share a computational-level goal (incremental next-word prediction), their algorithmic and hardware levels diverge fundamentally. Human attraction stems from cue-based retrieval over capacity-constrained memory, whereas Transformers utilize unrestricted self-attention matrices over full context windows.
  • Intra-Family Coherence vs. Cross-Family Divergence: Predictions correlate much more strongly within architectural families (0.83โ€“0.98) than across families (dropping to 0.68), indicating that pre-training corpora, tokenization designs, and architectural nuances dominate fine-grained psycholinguistic behaviors.

Limitations & Future Work

  • Typological Breadth: Stimuli and self-paced reading references are restricted to English; cross-linguistic testing in morphologically rich languages (e.g., German, Russian, Spanish) is essential.
  • Training Corpus Noise vs. Architectural Bias: The degree to which observed attraction effects reflect true structural representations versus superficial imitation of typographical agreement errors in web crawl corpora remains an open question requiring pre-training data auditing.
  • Architecture Exploration: The study examines standard dense autoregressive Transformers up to 4B parameters, leaving modern linear attention, state space models (Mamba/SSM), and mixture-of-experts open for future cognitive benchmarks.
  • vs Wagers et al. (2009): The empirical psycholinguistic baseline establishing that human readers exhibit agreement attraction selectively in ungrammatical, singular-subject conditions. This paper proves that Transformers fail to replicate this selective asymmetry in complex syntactic frames.
  • vs Linzen et al. (2016) & Arehalli and Linzen (2020): Pioneering studies showed that LSTMs capture agreement attraction patterns across intervening distractors. This work shows that the newer Transformer paradigm does not universally resolve cognitive fidelity, instead exhibiting fragile structural boundaries when transitioning from simple PP modifiers to relative clauses.

Rating

  • Novelty: โญโญโญโญโ˜† First multi-family, multi-construction, full-factorial cognitive evaluation of Transformers on agreement attraction across both PP and relative clauses.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluates 11 models across 4 families using rigorous 2ร—2ร—2 designs, paired tests, and crossed random-effects regressions.
  • Writing Quality: โญโญโญโญโญ Precise theoretical framing connecting psycholinguistic memory retrieval models with formal surprisal mechanics.
  • Value: โญโญโญโญโญ Delivers a vital methodological warning against overinterpreting local psycholinguistic successes in LLM cognitive modeling.