Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention¶

Conference: ECCV 2026
Paper: CVF / ECCV Poster
Code: https://github.com/cvlab-stonybrook/Gazette
Area: Human Understanding
Keywords: Gaze Decoding, Multimodal Large Language Model, Instruction Tuning, Top-down Attention, Cognitive Context

TL;DR¶

Gazette is the first generative gaze-to-text decoding framework that formulates human attention inference as open-ended natural language generation, employing LLM-synthesized think-aloud transcripts across multiple observers during training to isolate goal-specific dynamics from individual scanpath idiosyncrasies.

Background & Motivation¶

Inferring and understanding human attention and intent underpins critical applications ranging from human-robot collaboration and hands-free assistive technologies to driver monitoring and psychological diagnostics. Eye-tracking offers a non-intrusive, scalable, and cost-effective window into cognitive processes compared to neural decoding modalities like EEG, fMRI, or ECoG. However, conventional gaze decoding approaches predominantly frame this problem as a discriminative classification task over fixed category sets—such as distinguishing free-viewing from visual search, or selecting which object category from a predefined pool the user is seeking. When the category space is narrow, the predictions are overly coarse and fail to provide downstream systems with fine-grained spatial or contextual attributes; when expanded, the models suffer from severe data sparsity, fundamentally lacking the capacity to articulate rich, open-ended human intentions.

Formulating gaze decoding as open-ended natural language generation introduces a severe ill-posed inverse problem. While a user's underlying goal (e.g., finding a "red car on the right") reflects a clean, task-driven objective, the recorded gaze scanpath is heavily confounded by participant-specific idiosyncrasies, exploratory saccades, and bottom-up salience biases. Consequently, different observers pursuing the identical goal display divergent scanpaths, creating noisy and weak supervision signals. Furthermore, off-the-shelf multimodal large language models (MLLMs) lack domain-specific eye-movement priors, leading standard instruction tuning to fit spurious, observer-specific correlations rather than invariant cognitive dynamics.

Grounded in Bayesian ideal-observer models and Cognitive Relevance Theory, top-down attention allocation under a shared visual stimulus and task objective is theoretically invariant across observers, whereas individual variations act as approximately orthogonal noise. This paper leverages this insight: by pooling scanpaths from multiple participants sharing the same objective during training, an external language model can extract this shared spatiotemporal attentional logic as a cognitive scaffold. Core idea: formulate gaze decoding as an autoregressive vision-language generation task and introduce auxiliary think-aloud transcript generation during training to isolate goal-driven attentional invariants from participant-specific noise, enabling single-scanpath open-vocabulary intention decoding at inference.

Method¶

Overall Architecture¶

Gazette is built upon the LLaVA-1.5-7B architecture. It receives an input image \(I\) along with a textually serialized gaze scanpath \(S\), and autoregressively generates text \(D\) characterizing the user's cognitive context. The decoded context is structured hierarchically into two tiers: a coarse-level behavior type \(D_{\text{type}}\) (e.g., Target-Present Visual Search, Target-Absent Visual Search, Object Referral, or VQA) and a fine-level stimulus \(D_{\text{goal}}\) (ranging from a search category label to free-form referring expressions or questions).

During training, Gazette is fine-tuned with Low-Rank Adaptation (LoRA) jointly on the primary gaze decoding task (GazeDec) and an auxiliary think-aloud transcript generation task (ThinkAloud). At inference time, the model takes only the image and a single observed scanpath to decode the cognitive context directly via greedy decoding, requiring no group scanpaths or auxiliary transcripts. The processing flow and task interaction are outlined below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I + Gaze Scanpath S"] --> B["Prompt Formulation<br/>(GazeDec / ThinkAloud)"]
    B --> C["Vision Encoder + Linear Projection Layer"]
    C --> D["Autoregressive LLM Backbone"]
    D --> E["Attention Allocation Explanation<br/>(Auxiliary Think-aloud Supervision)"]
    D --> F["Hierarchical Cognitive Context Decoding<br/>(Behavior Type & Goal Description)"]

Key Designs¶

1. Textual Scanpath Serialization: Preserving Spatiotemporal Structure for MLLMs Raw gaze tracking sequences \(S = \{f_i\}_{i=0}^{N-1}\) consist of \(N\) sequential fixations \(f_i = (x_i, y_i, t_i)\), where \((x_i, y_i) \in [0, 1]^2\) represents normalized 2D coordinates on the image canvas and \(t_i\) denotes the fixation duration in milliseconds. To exploit the rich spatial reasoning and language understanding priors of pre-trained MLLMs without introducing specialized graph or temporal encoder subnets, Gazette serializes the scanpath directly into a structured text prompt \(T_{\text{GazeDec}}\) (e.g., Given scanpath [(0.4, 0.4, 20), ..., (0.1, 0.3, 60)], decode the behavior type and stimulus). The image is embedded via the pre-trained CLIP vision encoder and projected by a linear layer into the LLM token space, where it is concatenated with the serialized scanpath tokens. This enables the model's self-attention mechanism to jointly attend to visual patch tokens and sequential fixation points.

2. Think-Aloud Transcript Generation: Disentangling Noise via Multi-Observer Invariance To overcome the underdetermined inverse problem where individual scanpaths \(S_i = f(I_i, D_i) + U_i\) entangle the latent goal \(D_i\) with unobserved participant nuisance variables \(U_i\), the framework groups training instances sharing the identical stimulus and objective: \((I, D, \{S_i \mid i \in G\})\). While individual trajectories vary, the shared attentional allocation strategy remains invariant across observers. Gazette queries GPT-4 with normalized scene object bounding boxes, category tags, and the multi-observer scanpaths, prompting it to synthesize a structured "think-aloud transcript" consisting of three idea units: - Top-down attention allocation explanation (e.g., narrative describing how gaze traverses salient distractors before fixating and verifying the actual target); - Target spatial localization bounding box (normalized coordinates \([x, y, w, h]\), or predicting null when the target is absent); - Scanpath length estimation (reflecting the degree of exploratory versus focused visual search). These transcripts serve as pseudo-supervision for the auxiliary instruction task \(T_{\text{ThinkAloud}}\), training the MLLM to ground temporal gaze shifts in high-level cognitive intent. During inference, this auxiliary objective is discarded, allowing single-scanpath decoding.

Loss & Training¶

The framework is built on LLaVA-1.5-7B, applying LoRA to the attention projection weights to prevent overfitting on scarce eye-tracking datasets. The training uses standard autoregressive cross-entropy language modeling loss across both \(T_{\text{GazeDec}}\) and \(T_{\text{ThinkAloud}}\) prompts. Training is conducted on two NVIDIA RTX A6000 GPUs with a total batch size of 32 and a learning rate of \(2 \times 10^{-5}\).

At inference time, the model processes \(T_{\text{GazeDec}}\) via greedy decoding. The generated text is parsed into \(D_{\text{type}}\) and \(D_{\text{goal}}\). \(D_{\text{type}}\) is mapped to the four target tasks via MiniLM embedding cosine similarity. \(D_{\text{goal}}\) is directly evaluated as open-ended text for Object Referral and VQA, or mapped via nearest neighbor matching to the 18 predefined classes in COCO-Search18.

Key Experimental Results¶

Main Results¶

On open-ended generation benchmarks—Object Referral (RefCOCO-Gaze, Task A) and Visual Question Answering (AiR-D, Task B)—Gazette was evaluated against general LLaVA-1.5, a heuristic LLaVA-last baseline (describing the object at the final fixation), and an ablated model without auxiliary think-aloud training (w/o ThinkAloud).

Task	Method	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr
Noise Ceiling	RefCOCO-IAC	0.500	0.057	0.244	0.452	1.115
Object Referral (Task A)	LLaVA-1.5 [39]	0.066	0.000	0.077	0.105	0.062
	LLaVA-last	0.170	0.014	0.113	0.221	0.113
	Gazette (w/o ThinkAloud)	0.479	0.070	0.232	0.443	0.872
	Gazette (Full)	0.519	0.098	0.248	0.480	0.974
VQA (Task B)	LLaVA-1.5 [39]	0.124	0.006	0.039	0.103	0.047
	Gazette (w/o ThinkAloud)	0.329	0.095	0.147	0.286	0.263
	Gazette (Full)	0.364	0.159	0.160	0.324	0.367

For categorical visual search on COCO-Search18, Gazette was benchmarked against leading discriminative methods:

Task	Method	Precision	Recall	F1	Accuracy
Target-Present (Task C)	GazeGNN [68]	0.338	0.345	0.319	0.335
	GST [48]	-	-	-	0.544
	Mondal et al. [45]	-	-	-	0.776
	Gazette (w/o ThinkAloud)	0.768	0.746	0.742	0.742
	Gazette (Full)	0.786	0.775	0.775	0.773
Target-Absent (Task D)	GazeGNN [68]	0.241	0.251	0.218	0.270
	GST [48]	-	-	-	0.385
	Mondal et al. [45]	-	-	-	0.387
	Gazette (w/o ThinkAloud)	0.437	0.424	0.420	0.445
	Gazette (Full)	0.438	0.426	0.424	0.443

Ablation Study & Qualitative Evaluation¶

To evaluate semantic accuracy beyond surface n-gram overlap, GPT-4 was utilized as an external judge (1–10 scale) on grounded referential and question rubrics:

Task	Rubric	LLaVA-last	w/o ThinkAloud	Gazette (Full)
Object Referral (Task A)	Expression Overlap	3.515	5.019	5.980
	Referential Equivalence	4.100	5.771	6.638
	Category Correctness	7.577	8.376	8.743
VQA (Task B)	Question Overlap	-	2.103	2.792
	Answer Equivalence	-	1.642	2.135

Key Findings¶

Scanpath dynamics are essential beyond final landing points: LLaVA-last scored only 0.014 BLEU-4 and 0.113 CIDEr on Object Referral. This indicates that relying solely on where the eye ultimately stops is insufficient for distinguishing ambiguous objects; understanding the full temporal scanpath provides indispensable relational context.
Think-aloud instruction tuning yields massive generative gains: Incorporating think-aloud supervision boosted BLEU-4 by +40.00% (0.070 to 0.098) in Object Referral—even outperforming the human Inter-Annotator Consistency noise ceiling (RefCOCO-IAC of 0.057). In VQA, BLEU-4 improved by +67.37% (0.095 to 0.159) and CIDEr by +39.54% (0.263 to 0.367).
Behavioral collapse in target-absent trials: For Target-Absent visual search, ThinkAloud yielded modest gains (F1 0.420 vs. 0.424). As confirmed by psychological literature, observers searching for a non-existent target gradually transition into unconstrained free-viewing, causing cross-observer fixation agreement to collapse and weakening the shared patterns extracted by GPT-4.

Highlights & Insights¶

From Closed-world Classifiers to Cognitive Translators: Gazette redefines gaze decoding from narrow, categorical classification into open-ended natural language generation, bridging the semantic gap between implicit physiological signals and expressive descriptions.
Solving Undetermined Inversion via Group Invariance: By exploiting the invariant goal structure shared across multi-observer scanpaths while discarding individual idiosyncrasies via offline LLM summarization, the method provides an elegant paradigm for regularizing weak supervision without adding inference computational overhead.

Limitations & Future Work¶

Reliance on Observer Concordance: The framework assumes high inter-observer consistency. In open-ended exploratory viewing or target-absent scenarios with severe scanpath divergence, the supervisory quality of extracted transcripts drops significantly.
LLM Prior Bias in Pseudo-annotations: The think-aloud transcripts reflect GPT-4's statistical world knowledge and reasoning biases rather than genuine sub-conscious human cognitive states, potentially over-rationalizing human fixation strategies.
Textual Serialization Coarseness: Representing gaze as a text list of coordinates may underutilize fine-grained continuous spatiotemporal dynamics that could be captured through direct heatmap embeddings or spatio-temporal graph tokens.

vs. Discriminative Gaze Decoding (GST / GazeGNN / Mondal et al. 2025): Prior methods rely on fixed-set classifiers or predefined vocabulary alignment. In contrast, Gazette employs an autoregressive vision-language backbone capable of generating rich open-vocabulary descriptions while matching or exceeding classification accuracy on standard benchmarks.
vs. Attention-based Personalization (Xue et al. CVPR 2026): Whereas Xue et al. absorb user attention into an observer embedding during training to personalize image captioning without requiring gaze at test time, Gazette directly processes test-time scanpaths to decipher immediate, dynamic cognitive goals.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates open-ended generative gaze decoding and proposes a cross-observer invariance framework using think-aloud transcripts.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Rigorous evaluation spanning object referral, VQA, and target-present/absent search with both lexical and LLM-judge metrics.
Writing Quality: ⭐⭐⭐⭐⭐ Strong theoretical foundation linking Bayesian ideal-observer theory and cognitive relevance theory with clear presentation.
Value: ⭐⭐⭐⭐⭐ Substantial potential for AR glasses, assistive robotics, and human-computer interfaces requiring nuanced intent understanding.