Skip to content

Reading Time Modeling with Pixel-Based Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM (Other)
Keywords: Pixel-Based Language Models, Reading Time Modeling, Parafoveal Vision, Surprisal Theory, Word Skipping

TL;DR

This paper introduces tokenizer-free pixel-based language models (PLMs) into computational psycholinguistics for the first time, formulating a novel Visual Word Prediction Difficulty (VWPD) metric with Gaussian-blurred future context to simulate human parafoveal vision, achieving superior and complementary predictive power over conventional token-based surprisal across diverse multilingual eye-tracking corpora.

Background & Motivation

In computational psycholinguistics and cognitive science, neural language models have established themselves as an indispensable vehicle for quantitatively formalizing and testing expectation-based theories of human sentence comprehension. Under this paradigm, human readers are hypothesized to incrementally anticipate upcoming linguistic inputs from preceding syntactic and semantic context, where information-theoretic measures such as surprisal quantify the cognitive load during word recognition and demonstrate strong logarithmic linear fits with fixation durations. Nevertheless, conventional language models operate over predetermined vocabularies of discrete token IDs (TLMs), which inevitably introduces non-trivial methodological confounds: tokenizers are constructed purely for engineering convenience without cognitive plausibility, routinely creating artificial probability dislocation across subword boundaries and entirely excluding bottom-up visual orthographic decoding from cognitive modeling.

Natural human reading is intrinsically a multi-level sensorimotor process coordinating low-level visual perception with high-level cognitive integration. A substantial fraction of words in natural text reading is processed without receiving any direct foveal fixation; instead, these words are successfully skipped because partial visual features—such as approximate word length, coarse exterior shape, and salient stroke configurations—are already perceived pre-attentively in the parafoveal visual field. Discrete token-based language models, completely detached from visual appearances, are fundamentally incapable of simulating how degraded parafoveal visibility guides saccadic skipping decisions, nor can they probe how font, layout, or script-specific orthographic complexity modulate cognitive processing times.

This study breaks away from the discrete tokenization bottleneck by introducing pixel-based language models (PLMs) as a foundational cognitive modeling infrastructure. By rendering continuous text directly onto two-dimensional pixel arrays and corrupting the right-side upcoming context with a Gaussian blurring operator to mimic human parafoveal visual degradation, the authors formulate novel metrics of Visual Word Prediction Difficulty (VWPD) and visual prediction entropy. Core idea: leverage tokenizer-free pixel-based language models to reconstruct masked text patches under degraded future visual context, simulating human parafoveal sight and demonstrating that visual word prediction difficulty provides powerful, complementary predictive capability alongside discrete token surprisal for both gaze duration and word skipping behavior.

Method

Overall Architecture

The proposed framework establishes an end-to-end cognitive modeling pipeline that translates raw textual stimuli into pixelated representations, simulates human parafoveal vision degradation, quantifies visual reconstruction difficulty in pixel space, and models human eye-tracking behavior via generalized additive mixed models (GAMM).

Given an input sentence, a tokenizer-free renderer constructs a two-dimensional image, which is partitioned into non-overlapping spatial patches and mapped to continuous latent embeddings via a linear projection. To faithfully reflect human reading where the fovea focuses on the target word while the parafovea previews degraded upcoming words, all patches spanning the target word are masked out, and the right-side future context patches are subjected to two-dimensional spatial Gaussian blurring. A vision Transformer (ViT) encoder-decoder architecture reconstructs the masked target patches conditioned on the uncorrupted past context and blurred future preview. The cognitive load is quantified through instance-wise spatial reconstruction discrepancy (either Structural Similarity, SSIM, or Mean Squared Error, MSE) and pixel-level prediction entropy, which subsequently serve as key predictors in mixed-effects regressions against human reading time and saccadic skipping measures.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural Reading Text Sequence"] --> B["Text Image Rendering & Patch Embedding<br/>Tokenizer-free 2D patch projection"]
    B --> C["Parafoveal Gaussian Blurring Simulation<br/>Degraded visual acuity for future context"]
    C --> D["Target Word Patch Masking & Reconstruction<br/>Bidirectional ViT patch reconstruction"]
    D --> E["Visual Word Prediction Difficulty (VWPD)<br/>SSIM structural similarity & MSE error"]
    E --> F["Visual Prediction Entropy Calculation<br/>Pixel-level binary stroke uncertainty"]
    F --> G["GAMM Psychometric Predictive Power Evaluation<br/>Fitting gaze durations (FFD/FPT/TRT) & skipping (FSKP/SKP)"]

Key Designs

1. Text Image Rendering & Patch Embedding: Bypassing Tokenizer Bias via Continuous Visual Encodings

Conventional language models enforce discrete symbol lookup tables that obliterate fine-grained topological and sub-character structures. In contrast, this approach deploys the multilingual Pixel-m4 architecture. An input sentence \(w = [w_1, \dots, w_n]\) is directly rendered into an uncompressed image \(x \in \mathbb{R}^{H \times W \times C}\), where \(C, H, W\) represent color channels, height, and width respectively. The rendered canvas is subdivided into \(N = \frac{HW}{P^2}\) non-overlapping patches \(X = [x_1, \dots, x_N] \in \mathbb{R}^{N \times P^2 \times C}\) with patch size \(P \times P\). Each flattened patch is linearly projected into a \(d\)-dimensional embedding space with learnable positional encodings: $\(h_i = \text{Flatten}(x_i) W_E + b_E + e_{pos_i}\)$ where \(W_E \in \mathbb{R}^{P^2C \times d}\) and \(b_E \in \mathbb{R}^d\). This continuous visual embedding operates independently of character inventories, vocabularies, or subword segmenters, naturally preserving bottom-up visual-orthographic features across typologically diverse writing systems such as Arabic-based scripts, Hangul syllable blocks, and Hanzi ideograms.

2. Parafoveal Gaussian Blurring Simulation: Biologically Plausible Future Context Degradation

Completely masking out all future patches \(X_{>k}\) induces a catastrophic distribution mismatch in bidirectional vision Transformers, while leaving future text fully unmasked violates human physiological reading constraints where parafoveal acuity is severely attenuated. To bridge this gap, the authors design a spatial Gaussian blurring filter \(g(\cdot)\) applied strictly to the right-side upcoming patches: $\(X'(i, j) = \sum_{k=-m}^{m} \sum_{l=-m}^{m} X(i-k, j-l) \cdot G(k, l), \quad G(k, l) = \frac{1}{2\pi\sigma^2} e^{-\frac{k^2+l^2}{2\sigma^2}}\)$ with discrete convolution radius \(m = \frac{\text{kernel\_size}-1}{2}\), instantiated with kernel size 5 and standard deviation \(\sigma = 3\). This transformation preserves macro-level visual features such as overall word boundaries, word length, and baseline text density, while deliberately removing high-frequency edge information required for exact character identification, thereby mimicking the human parafoveal visual field during forward reading.

3. Visual Word Prediction Difficulty (VWPD): Multi-Perspective Spatial Reconstruction Error

For a target word \(w_t\) spanning from the \(s_t\)-th to the \(e_t\)-th patch, the corresponding patches are iteratively masked and reconstructed by the PLM: \(\tilde{x}_k = F_k(X_{<k} \circ m(x_k) \circ g(X_{>k}))\). Rather than relying solely on standard Mean Squared Error (MSE): $\(\overleftrightarrow{s}_{\text{MSE}}(x_k | X_{<k}, g(X_{>k})) = \|\tilde{x}_k - x_k\|_2^2\)$ the framework also implements a perception-oriented metric using Structural Similarity (SSIM), which models structural distortions rather than raw pixel-level differences: $\(1 - \text{SSIM}(\tilde{x}_k, x_k) = 1 - \frac{(2\mu_{\tilde{x}}\mu_x + C_1)(2\sigma_{\tilde{x}x} + C_2)}{(\mu_{\tilde{x}}^2 + \mu_x^2 + C_1)(\sigma_{\tilde{x}}^2 + \sigma_x^2 + C_2)}\)$ The aggregate Visual Word Prediction Difficulty \(\overleftrightarrow{I}(w_t | w_{\backslash t}) = \sum_{k=s_t}^{e_t} \overleftrightarrow{s}(x_k | X_{<k}, g(X_{>k}))\) accumulates the patch-wise reconstruction discrepancies, directly quantifying the cognitive processing difficulty required to resolve the visual identity of the upcoming word.

4. Visual Prediction Entropy Calculation: Quantifying Stroke-Level Uncertainty in Skipping Decisions

In addition to deterministic reconstruction error, the authors formulate visual prediction entropy to capture the model's distributional confidence over binary ink presence versus blank canvas. Each predicted pixel intensity within patch \(k\) is normalized via a logistic sigmoid \(\hat{p}_k = \sigma(\overleftrightarrow{\hat{x}}_k) \in [0, 1]^{P^2 \times C}\), where 0 denotes white paper background and 1 represents black text strokes. The entropy across all patches occupied by word \(w_t\) is defined as: $\(H(w_t | w_{\backslash t}) = \sum_{k=s_t}^{e_t} \sum_{l=1}^{P} \sum_{m=1}^{P} \sum_{c=1}^{C} \left[ -\hat{p}_{k,l,m,c} \log_2 \hat{p}_{k,l,m,c} - (1 - \hat{p}_{k,l,m,c}) \log_2 (1 - \hat{p}_{k,l,m,c}) \right]\)$ Low visual entropy indicates that the surrounding context strongly constrains the visual form of the upcoming word, enabling human readers to execute an early saccadic skip without initiating a full foveal fixation. Conversely, elevated visual entropy signifies divergent orthographic hypotheses, necessitating an immediate direct fixation.

Loss & Training

The underlying Pixel-m4 backbone utilizes a Vision Transformer encoder-decoder trained on patch reconstruction. For non-Chinese languages, the model is finetuned on 100,000 text sequences from the multilingual C4 corpus for 150,000 steps with a batch size of 32, minimizing the masked patch reconstruction MSE loss \(\mathcal{L} = \frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} \|\hat{x}_i - x_i\|_2^2\) until validation loss drops below 0.020. In psycholinguistic regression, Generalized Additive Mixed Models (GAMM) are fit using R's mgcv package. Reading durations (FFD, FPT, TRT) are log-transformed, while skipping outcomes (FSKP, SKP) are modeled under a binomial logistic link. All regressions rigorously control for baseline covariates including current and preceding word length (\(l_n, l_{n-1}\)), Zipf word frequency (\(f_n, f_{n-1}\)), and word sentence position (\(idx_n\)), alongside crossed random intercepts and slopes for subject IDs and stimuli IDs. Statistical significance and Psychometric Predictive Power (PPP, measured by log-likelihood difference \(\Delta \text{LLK}\)) are derived via 100 iterations of crossed random-effects bootstrapping with replacement.

Key Experimental Results

Main Results

The empirical evaluation was conducted across 459 human readers spanning 8 languages and 4 diverse scripts (German, English, Spanish, Turkish in Latin script; Persian in Abjad; Korean in Hangul; Simplified and Traditional Chinese in Hanzi) from MECO Wave 1, MECO Wave 2, and the Persian PSR corpus. The table below presents the mean paired log-likelihood difference (\(\Delta \text{LLK}\)) of the interactive model (\(\text{pixel\_uni} \times \text{gpt}\)) against the baseline monolingual GPT-2 surprisal model (positive values indicate superior predictive power from integrating the pixel language model, extracted from Table 3 in the paper):

Language & Dataset Reading Measure Monolingual GPT Baseline pixel_uni \(\times\) gpt (SSIM) Significance pixel_uni \(\times\) gpt (MSE)
Persian Far (PSR) FFD (First Fixation Duration) Baseline ref +103.8 \(p < .001\) +56.2 (\(p < .001\))
Persian Far (PSR) FSKP (First-pass Skipping) Baseline ref +328.7 \(p < .001\) +56.2 (\(p < .001\))
Persian Far (PSR) SKP (Overall Skipping) Baseline ref +337.6 \(p < .001\) +63.0 (\(p < .001\))
Korean Kor (MECO-1) FSKP (First-pass Skipping) Baseline ref +114.5 \(p < .001\) +44.2 (\(p < .001\))
Korean Kor (MECO-1) SKP (Overall Skipping) Baseline ref +167.4 \(p < .001\) +31.1 (\(p < .001\))
Simplified Chinese Simp (MECO-2) SKP (Overall Skipping) Baseline ref +19.2 \(p < .001\) +119.1 (\(p < .001\))
Traditional Chinese Trad (MECO-2) SKP (Overall Skipping) Baseline ref +23.7 \(p < .001\) +97.4 (\(p < .001\))
German Deu (meco po) FSKP (First-pass Skipping) Baseline ref +76.1 \(p < .001\) +66.1 (\(p < .001\))
Spanish Esp (meco es) FSKP (First-pass Skipping) Baseline ref +134.7 \(p < .001\) +132.7 (\(p < .001\))
English Eng (meco uk) TRT (Total Reading Time) Baseline ref +100.3 \(p < .001\) +51.8 (\(p < .001\))

Ablation Study

To evaluate the standalone predictive capacity of visual prediction difficulty, the table below highlights the direct log-likelihood difference (\(\Delta \text{LLK}\)) between the isolated pixel model with unidirectional blurring (pixel_uni) and isolated GPT-2 surprisal (negative numbers indicate GPT-2 superiority, positive numbers indicate PLM superiority, extracted from Table 3 in the paper):

Dataset Error Metric \(\Delta\) FFD \(\Delta\) FPT \(\Delta\) TRT \(\Delta\) FSKP \(\Delta\) SKP
Persian Far (PSR) SSIM -126.6*** -321.2*** -129.1*** +177.0*** +214.9***
Korean Kor (ko) SSIM -5.6* -44.8*** -68.1*** +49.2*** +107.6***
Simplified Chinese Simp (ch-s) MSE -13.9*** -49.9*** -52.9*** +14.2*** +30.0***
Traditional Chinese Trad (ch-t) MSE -0.9 (n.s.) -1.5 (n.s.) +39.2*** +37.9*** +53.0***
German Deu (ge_po) SSIM -32.7*** -60.1*** -214.0*** +16.4*** -8.7***
English Eng (en_uk) SSIM -68.9*** -100.4*** -194.3*** +41.8*** -6.2 (n.s.)
Turkish Tur (tur) SSIM -4.1* -13.2*** -15.5*** -6.3*** +7.8***

Key Findings

  • Decisive Superiority in Word Skipping Prediction: While discrete token-based GPT-2 surprisal accounts better for durative fixation measures (FFD, FPT, TRT) reflecting higher-level semantic access, standalone PLM visual difficulty (VWPD) significantly outperforms GPT-2 on word skipping metrics (FSKP and SKP) in non-Latin scripts (e.g., +214.9 on Persian SKP, +107.6 on Korean SKP). This proves that word skipping decisions are predominantly governed by early parafoveal visual identifiability rather than late symbolic surprisal.
  • Divergent Error Metric Preferences Across Scripts: Across alphabetical scripts (German, English, Spanish, Turkish, and Persian), the structural similarity metric (SSIM) consistently provides superior predictive power over pixel-wise MSE. However, for Chinese (both Simplified and Traditional), MSE decisively outperforms SSIM (e.g., Simplifed Chinese SKP interaction gains of +119.1 for MSE vs. +19.2 for SSIM). This reflects that dense logographic characters require exact stroke-level intensity fidelity, which spatial-window averaging in SSIM tends to dilute.
  • Surprisal-Mediated Visual Re-examination: Interaction analyses reveal that under high token-level surprisal (+1 SD), the predictive slope of VWPD on First Fixation Duration (FFD) is flattened or reversed. Conversely, high surprisal drastically amplifies the positive effect of VWPD on first-pass refixation duration (\(\text{FPREF} = \text{FPT} - \text{FFD}\)). This indicates that when high lexical difficulty impedes immediate lexical access, the human reading oculomotor system falls back onto detailed, bottom-up visual orthographic re-inspection.

Highlights & Insights

  • Pioneering Modality Shift in Psycholinguistic Modeling: Successfully demonstrates that tokenizer-free pixel-based language models eliminate decades-old tokenization artifacts in surprisal estimation, establishing raw visual input as a cognitively grounded foundation for computational linguistics.
  • Elegantly Simple Parafoveal Modeling: The application of a localized 2D Gaussian blur filter over upcoming future patches provides an intuitive, differentiable, and effective simulation of human retinal eccentricity and parafoveal acuity loss.
  • Dual-Stream Reading Account: Computationally demonstrates that human naturalistic reading integrates two complementary channels: a discrete symbolic channel governing lexical-semantic integration, and a continuous visual-perceptual channel orchestrating saccadic targeting and skipping.

Limitations & Future Work

  • Reliance on Fixed Bigram Text Rendering: The experiments exclusively employ the Bigram text rendering pipeline bundled with pretrained Pixel-m4. Future work must investigate the sensitivity of psychometric fits across different font families, font sizes, line heights, and character spacings.
  • Uniform vs. Continuous Parafoveal Acuity: The current implementation applies a static Gaussian blur kernel to all future context, whereas human biological vision features a continuous drop in visual acuity as a logarithmic function of retinal eccentricity. Implementing eccentricity-dependent graded blurring represents an essential next step.
  • Absence of Generative Eye Movement Control: The current evaluation is restricted to offline GAMM regressions on isolated fixation measures. A compelling future direction is to interface PLM visual representations directly with computational saccadic control architectures (such as E-Z Reader or SWIFT) to generate full synthetic scanpaths.
  • vs. Token-Based Surprisal Theory (Hale, 2001; Levy, 2008; Wilcox et al., 2023): Conventional surprisal theory relies on autoregressive token models \(-\log_2 p(w_t | w_{<t})\), capturing deep semantic expectations but remaining blind to pre-attentive visual processing. This paper shows that PLMs provide orthogonal, complementary explanatory power, particularly for saccadic skipping.
  • vs. Tokenization Artifact Mitigations (Nair & Resnik, 2023; Oh & Schuler, 2024; Giulianelli et al., 2024): Previous studies attempted to resolve tokenizer biases by reallocating subword probabilities or correcting whitespace conventions. This work resolves the problem at its architectural root by operating entirely in continuous pixel space without discrete token boundaries.
  • vs. Classical Computational Eye-Movement Models (Reichle et al., 2003; Engbert et al., 2005): Traditional models (E-Z Reader, SWIFT) rely on hand-crafted lexical frequency and word length heuristics. This study demonstrates that self-supervised vision Transformers can naturally induce human-like reading and skipping dynamics directly from raw visual text.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering introduction of tokenizer-free pixel language models to human reading time modeling and psycholinguistics.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 459 readers, 8 languages, 4 diverse scripts, accompanied by bootstrapping and 10-fold cross-validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, tightly coupling cognitive psychology motivation with computer vision architecture.
  • Value: ⭐⭐⭐⭐⭐ Provides a transformative paradigm for bridging natural language processing, computer vision, and cognitive psycholinguistics.