Do Vision-Language Models Understand Human Engagement in Games?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zoewang0213/vlm-game-engagement
Area: LLM Reasoning
Keywords: Vision-Language Models, Player Engagement Prediction, Affective Computing, Prompting Strategies, Theory-Guided Probing
TL;DR¶
This paper presents the first systematic evaluation of vision-language models on inferring human engagement from gameplay video across 9 FPS games, discovering that VLMs achieve only ~57% zero-shot accuracy (lagging behind majority baselines) due to a fundamental perception-understanding gap and reliance on visual spectacle shortcuts.
Background & Motivation¶
Player engagement is a foundational construct across game design, esports broadcasting, automated highlight generation, and interactive media analytics. Uncovering when and why a player enters an immersive state or becomes detached offers critical guidance for dynamic difficulty adjustment and player retention modeling. Recent vision-language models (VLMs) have exhibited remarkable proficiencies in basic visual recognition, object localization, and visual question answering; however, these perceptual capabilities on observable physical scenes do not inherently translate into an understanding of human internal psychological states. Engagement is a multidimensional, subjective, and temporally dynamic latent state that cannot be directly inspected from raw surface pixels, requiring nuanced contextual reasoning over challenge, progress, and narrative tension.
Video games serve as a uniquely rigorous diagnostic testbed for evaluating latent psychological inference because they explicitly decouple visual intensity from human engagement. A scoreboard screen displaying a razor-thin 15–14 match point requires multi-step contextual deduction—recognizing numerical parity, projecting competitive stakes, and inferring peak mental tension—yet visually presents a completely static interface without dynamic movement. Conversely, a chaotic explosion-filled scene loaded with particle visual effects may indicate trivial combat with zero survival pressure, producing minimal affective investment. Existing affective computing literature has predominantly centered on text sentiment analysis or facial emotion recognition from still portraits, while game-centric VLM evaluations have remained confined to perceptual task completion such as agentic navigation, leaving visual engagement inference entirely unaddressed.
To bridge this critical divide, this work investigates whether current frontier VLMs truly understand human engagement in video games, and whether structured prompt scaffolding—such as theory-grounded cognitive probes or cross-game few-shot demonstrations—can overcome inherent model limitations. Core idea: by establishing the first systematic benchmark across 3 architecturally distinct VLMs, 9 diverse first-person shooter titles, and 6 progressive prompting strategies, the authors expose a severe perception-understanding gap where zero-shot VLMs fail against simple majority-class baselines, uncover five systematic failure modes, and demonstrate that theoretical scaffolding without concrete demonstrations actively reinforces superficial shortcuts.
Method¶
Overall Architecture¶
The evaluation framework is constructed upon the GameVibe Few-Shot (GVFS) affective gameplay benchmark, covering 9 target FPS games and using DOOM as the cross-game demonstration source domain. The prediction task takes a sequence of 16 frames \(\{f_1, \dots, f_{16}\}\) sampled from a 1-second gameplay window and predicts a binary engagement label \(y \in \{\text{High}, \text{Low}\}\) derived from aggregated human observer traces. The pipeline operates under two complementary configurations: pointwise engagement estimation for isolated windows and pairwise engagement change prediction between adjacent windows. Three architecturally distinct VLMs—InternVL3.5-8B-Instruct, Qwen3-VL-8B-Instruct, and GPT-4o—are systematically probed under six prompting strategies of increasing sophistication (S1–S6).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Gameplay Video Stream<br/>16 frames per 1-second window"] --> B["Pointwise and Pairwise Task Formulation<br/>Reaction-lag shift and trimmed-mean thresholding"]
B --> C["Six-Stage Progressive Prompting Hierarchy<br/>S1 zero-shot to S6 theory-retrieval hybrid"]
C --> D["Four-Pillar Psychological Theory Probing<br/>Flow/GameFlow/SDT/MDA visual operationalization"]
D --> E["Dual-Space Cross-Domain Exemplar Retrieval<br/>VLM hidden representations and CLIP vision embeddings"]
E --> F["Cognitive Failure Attribution and Miscalibration Audit<br/>Visual intensity bias and temporal incoherence evaluation"]
Key Designs¶
1. Pointwise and Pairwise Task Formulation: Aligning reaction latency and isolating meaningful transitions Human affective annotations naturally carry physiological reaction latency and inter-annotator variance. To rigorously account for reaction lag, the authors split each 61-second gameplay clip into 1-second non-overlapping windows and introduce a 1-second forward temporal shift, pairing video window \(t\) with human annotation trace \(t+1\), yielding 59 valid evaluation windows per clip. For pointwise evaluation, continuous traces from 5 trained observers are aggregated via a trimmed mean—discarding the highest and lowest scores before averaging the remaining three—and binarized at a threshold of \(0.5\). For pairwise evaluation, consecutive window pairs with absolute engagement change \(|\Delta e| > 0.05\) are categorized into Increase versus Decrease, strictly filtering out near-stable transitions to insulate the benchmark from subjective boundary noise.
2. Six-Stage Progressive Prompting Hierarchy: Disentangling perceptual priors, demonstration retrieval, and theory guidance To systematically chart the causal impact of prompt scaffolding, the authors design six structured prompting strategies of ascending complexity (S1 to S6). Strategy S1 establishes the zero-shot baseline, providing behavioral definitions of high and low engagement based purely on observable visual cues such as active gunfights or idle pause menus. S2 implements theory-guided zero-shot prompting, requiring the model to assess game scenes against established psychological dimensions. Strategies S3 and S4 introduce retrieval-augmented few-shot demonstrations utilizing internal VLM embeddings and CLIP vision encoder representations, respectively. Finally, S5 and S6 synthesize theoretical scaffolding with retrieved few-shot exemplars (VLM-based and CLIP-based), enabling controlled orthogonal isolation of conceptual scaffolding versus empirical in-context grounding.
3. Four-Pillar Psychological Theory Probing: Operationalizing latent constructs into observable visual indicators Rather than treating engagement as an uninterpretable holistic impression, the authors translate four canonical game engagement frameworks into concrete visual markers within structured prompts. Flow Theory is operationalized via intermediate health pools and close scoreboards (challenge-skill balance) alongside damage numbers and hit markers (immediate feedback). GameFlow is mapped to crosshair alignment, active character movement (control), and uninterrupted full-screen viewport (immersion). Self-Determination Theory (SDT) is operationalized through kill feeds, achievements (competence), and visible teammate coordination (relatedness). The Mechanics-Dynamics-Aesthetics (MDA) framework probes narrative tension during clutch match situations and aesthetic sensation driven by particle effects. Models must explicitly tally support across these four theories before outputting a grounded prediction.
4. Dual-Space Cross-Domain Exemplar Retrieval: Error-driven corrective memory and task-agnostic semantic alignment For pointwise prediction, strategies S3 and S5 construct an in-game error-driven case memory by evaluating the model on the initial 70% of windows, storing mispredicted failure cases, and retrieving the top-\(k\) nearest neighbors via cosine similarity during inference on the remaining 30% held-out test windows. For pairwise prediction, difference vectors between consecutive frame embeddings are computed to retrieve DOOM demonstration pairs exhibiting similar dynamical shift patterns. The retrieval space compares task-specific representations extracted from the VLM's final hidden state against task-agnostic representations extracted from a pretrained CLIP ViT encoder, evaluating whether high-level visual-semantic embeddings transfer more reliably across heterogeneous visual art styles than internal language-projected representations.
Benchmark & Evaluation Protocol¶
All model evaluations are conducted strictly zero-shot or few-shot without parameter fine-tuning, preserving the frozen pre-trained weights of InternVL3.5-8B-Instruct, Qwen3-VL-8B-Instruct, and GPT-4o. Model performance is gauged through raw accuracy and relative accuracy gain over the per-game majority-class baseline \(\Delta A\): $\(\Delta A = \frac{A_{\text{model}} - A_{\text{baseline}}}{A_{\text{baseline}}}\)$ where positive \(\Delta A\) indicates performance surpassing the trivial constant classifier. To probe whether models are seduced by low-level visual spectacle, visual intensity \(I(f)\) is quantified as a composite score: $\(I(f) = \frac{1}{4} \left( \hat{S}(f) + \hat{E}(f) + \hat{V}(f) + \hat{R}(f) \right)\)$ integrating min-max normalized color saturation \(\hat{S}\), Canny edge density \(\hat{E}\), luminance variance \(\hat{V}\), and red channel intensity \(\hat{R}\). Model temporal stability across adjacent 1-second windows is quantified through flip rate (frequency of adjacent prediction toggles) and lag-1 autocorrelation.
Key Experimental Results¶
Main Results¶
The pointwise engagement prediction accuracy across 9 evaluation titles demonstrates that raw zero-shot VLMs consistently fall short of the per-game majority-class baseline, with retrieval-augmented strategies providing substantial yet model-dependent gains.
| Model | Majority Baseline (M) | S1 (Zero-Shot) | S2 (TG Zero-Shot) | S3 (Few-Shot VLM) | S4 (Few-Shot CLIP) | S5 (TG + FS VLM) | S6 (TG + FS CLIP) | |---|---|---|---|---|---|---| | InternVL3.5-8B | 67.2% | 56.9% | 52.9% | 71.3% | 56.9% | 68.3% | 51.4% | | Qwen3-VL-8B | 67.2% | 56.7% | 53.1% | 75.0% | 67.8% | 74.1% | 69.7% | | GPT-4o | 67.2% | 56.6% | 56.6% | — | — | — | — |
In the pairwise engagement change prediction setting, models struggle severely across virtually all prompting strategies, hovering near chance level:
| Model | Majority Baseline (M) | S1 (Zero-Shot) | S2 (TG Zero-Shot) | S3 (Few-Shot VLM) | S4 (Few-Shot CLIP) | S5 (TG + FS VLM) | S6 (TG + FS CLIP) | |---|---|---|---|---|---|---| | InternVL3.5-8B | 56.5% | 53.1% | 49.8% | 51.8% | 53.1% | 55.9% | 49.8% | | Qwen3-VL-8B | 56.5% | 54.7% | 54.0% | 47.7% | 52.1% | 58.8% | 54.9% | | GPT-4o | 56.5% | 51.9% | 54.6% | — | — | — | — |
Failure Mode Taxonomy & Qualitative Attribution¶
Manual analysis of all 78 prediction errors generated by strategy S2 on CSGO18 and CS:GO Office reveals that challenge misassessment and superficial relatedness heuristics constitute over two-thirds of all cognitive failures.
| Failure Type Code & Name | CSGO18 Errors | CS:GO Office Errors | Total Errors | Proportion (%) | Primary Failure Mechanism |
|---|---|---|---|---|---|
| Type D: Challenge Misassessment | 16 | 14 | 30 | 38.5% | Buy menus and voting dismissed as non-interactive low engagement |
| Type B: Relatedness Shortcut | 19 | 5 | 24 | 30.8% | Visible teammate models or radio chatter trigger automatic High labels |
| Type C: Sensation Shortcut | 0 | 10 | 10 | 12.8% | Particle effects and muzzle flashes trigger High regardless of true stakes |
| Type A: Static Scene Misclass. | 3 | 4 | 7 | 9.0% | Post-match scoreboards showing close match points labeled Low due to stillness |
| Type E: Other / Ambiguous | 6 | 1 | 7 | 9.0% | Unclear visual cues or multi-source conflicting information |
Key Findings¶
- Zero-shot models consistently trail trivial heuristics: Across all 9 evaluation games, zero-shot accuracy lands around ~57% for all three models, lagging more than 10 percentage points behind the 67.2% majority baseline. GPT-4o delivers performance identical to open-source 8B models (56.6%), confirming that parameter scaling alone does not resolve latent psychological inference limitations.
- Theory scaffolding acts as a shortcut amplifier: Introducing theoretical guidance alone (S2) degrades accuracy across open-source models (InternVL: 56.9% \(\to\) 52.9%; Qwen: 56.7% \(\to\) 53.1%). Rather than promoting multi-step causal deduction, theoretical terminology provides the model with rich vocabulary to post-hoc rationalize surface-level pattern matching (e.g., attributing High engagement to SDT relatedness simply because an ally avatar is present).
- Severe temporal inconsistency across consecutive windows: VLMs display an erratic flip rate of 0.310 on CSGO18—18 times higher than the human label flip rate of 0.017. Similarly, the VLM lag-1 autocorrelation is only 0.275 compared to 0.950 for human annotators, showing that models treat consecutive 1-second video slices as isolated uncorrelated frames.
- Alarming inverse confidence-accuracy correlation: Text-derived rationale confidence analysis exposes severe model miscalibration: incorrect predictions exhibit higher average linguistic confidence (0.597) than correct ones (0.453). Predictions made with high confidence achieve only 38.5% accuracy, whereas low-confidence predictions attain 64.3%, indicating that fluent, confident model explanations are systematically deceptive.
Highlights & Insights¶
- Defining the capability hierarchy of multimodal affective reasoning: The paper conceptualizes a four-stage capability hierarchy: Perception \(\to\) Description \(\to\) Interpretation \(\to\) Inference. State-of-the-art VLMs succeed at perceiving HUD numbers and describing environmental assets, but systematically break down when attempting to interpret game context or infer human affective immersion.
- Exposing the "rationalization trap" of prompt engineering: Elaborate theory-guided prompts can induce counterproductive shortcut amplification. Without grounded training, instructing models with conceptual frameworks encourages superficial stimulus-response heuristics wrapped in plausible-sounding academic jargon.
- Blueprint for hybrid neuro-symbolic engagement systems: To address these failure modes, the authors propose an architectural decoupling roadmap: leveraging VLMs solely as structured game-state extractors (health, score diffs, entity counts), followed by lightweight temporal models (HMMs or LSTMs) and calibration modules (Platt scaling) to produce stable, smoothed predictions.
Limitations & Future Work¶
- Constrained benchmark scale and genre coverage: The GVFS dataset supplies only 61 seconds of gameplay per game (59 evaluation windows) and is exclusively restricted to first-person shooter titles, leaving broader genres such as turn-based strategy, MOBAs, or rhythm games unexplored.
- Observer annotation bias vs. firsthand player experience: Ground truth labels originate from 20 third-party gaming-familiar observers rather than the active players themselves, potentially introducing observer-specific visual spectacle biases that may conflate with model tendencies.
- Near-chance pairwise transition modeling: Pairwise engagement trend prediction remains fundamentally unsolved across all tested strategies, indicating that VLMs cannot reliably serve as automated preference annotators or reward rankers without task-specific fine-tuning.
Related Work & Insights¶
- vs. Melhart et al. (2025) Text-Only Game Engagement Evaluation: Melhart et al. explored LLMs conditioned on textual game telemetry and action logs; in contrast, this paper performs the first end-to-end multimodal video benchmark directly from raw pixels, establishing the baseline for visual-to-psychological inference.
- vs. Pinitas et al. (2025) Supervised Few-Shot Engagement Modeling: Prior efforts relied on specialized small convolutional and recurrent networks trained with supervised game-specific objectives; this work assesses the zero-shot and in-context learning boundaries of general-purpose foundation VLMs, demonstrating that foundation models still trail specialized supervised models.
- vs. Zhang et al. (2025) VideoGameBench: Previous game benchmarks probed task completion rates and functional instruction-following; this work pioneers the evaluation of subjective affective alignment, opening a crucial direction for human-centered multimodal evaluation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering benchmark introducing latent psychological engagement inference from gameplay video with rigorous theory-guided probes.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage of 3 frontier VLMs, 9 titles, 6 progressive prompting strategies, temporal autocorrelation, bootstrap CIs, and manual error categorization.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptional prose clarity, rigorous formalization, and deeply insightful failure analysis with compelling qualitative case studies.
- Value: ⭐⭐⭐⭐⭐ Essential reality check and architectural roadmap for multimodal affective computing, LLM-as-a-judge evaluation, and adaptive game design.