Skip to content

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Vision-Language Models, Information Structure, Discourse Pragmatics, Hungarian Syntax, Mode Collapse

TL;DR

By exploiting Hungarian as a discourse-configurational language where Topic and Focus overtly map onto word order, this work compares six contemporary VLMs against human speakers in visual question answering, revealing that VLMs over-regularise competing discourse and grammatical pressures by collapsing onto rigid structural templates.

Background & Motivation

Vision-language models (VLMs) are increasingly evaluated as general-purpose systems for generating situated text grounded in visual inputs. However, contemporary linguistic benchmarking of these models remains disproportionately focused on syntax, coreference, and lexical-semantic identificationโ€”asking whether a model can detect objects, spatial relations, actions, or retrieve factual answers from an image. In human communication, natural language generation is fundamentally more than retrieving accurate perceptual content; speakers must package information appropriately relative to the common ground established with their interlocutors. This packaging is governed by information structure (IS), which partitions an utterance into a Topic (linking to discourse-old, established entities) and a Focus (asserting discourse-new or contrastive information). Although IS facilitates cognitive comprehension speed and directly modulates assertion truth conditions in human communication, it remains virtually absent from the evaluation landscape of multimodal foundation models.

Evaluating IS in written English is notoriously challenging because English encodes Topic and Focus primarily through prosodic intonation and pitch accent rather than word order, making information packaging largely unobservable in standard text corpora. Consequently, it remains unknown whether VLMs possess a principled sensitivity to discourse status, or how such sensitivity interacts with other pervasive cognitive and grammatical biases documented in human sentence production. Specifically, human language production exhibits an independent grammatical-role bias that prioritises subjects as natural Topics, as well as a definiteness bias that favours definite articles for established entities and indefinite articles for newly introduced referents. When these pressures come into direct conflictโ€”such as when a question renders an object discourse-old while eliciting a subject as new informationโ€”it is entirely unexplored whether VLMs exhibit flexible communicative adaptation or fall back on rigid syntactic shortcuts.

To bridge this gap, the authors leverage Hungarian, a classic discourse-configurational language where Topic and Focus occupy dedicated preverbal and postverbal structural positions, and where Focus placement overtly triggers the displacement of verbal modifiers (igekรถtล‘) in written text. Drawing upon an ecologically validated visual question answering paradigm from psycholinguistics, the authors present paired transitive cartoons accompanied by wh-questions that systematically flip discourse status between subjects and objects. Core idea: evaluate the linguistic pragmatics of VLMs through Hungarian word order as an overt diagnostic of information structure, exposing that while models engage with discourse cues, they over-regularise competing linguistic pressures into severe structural template collapse.

Method

Overall Architecture

The evaluation framework establishes an end-to-end pipeline connecting psycholinguistic experimental design with automated dependency parsing and statistical convergence testing. The architecture comprises four sequential phases: controlled visual-question stimulus formulation, multi-run sampling across human speakers and diverse VLMs, rule-enhanced dependency parsing with huSpaCy, and quantitative information-structural mapping across aligned versus conflicting linguistic pressures.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 47 Transitive Cartoon Images + Wh-Question Pairs<br/>(Subject-Focus vs. Object-Focus)"] --> B["Controlled VQA Stimuli & Prompt Formulation"]
    B --> C["Bimodal Multi-Run Sampling & Bootstrap Convergence"]
    C --> D["Automated Dependency Parsing & Rule-Based Enhancement"]
    D --> E["Information-Structural Mapping & Pressure-Conflict Analysis"]
    E -->|Statistical Verification| F["Output: IS-Type Distributions, Delta P(Topic), and Focus Definiteness Asymmetry"]

In the stimulus generation phase, 47 black-and-white cartoon drawings illustrating reversible transitive interactions between two human characters are paired with wh-questions that establish either the subject or the object as discourse-old. Human native speakers and six instruction-tuned VLMs generate complete, concise responses describing the scene. The generated textual outputs are then parsed into verbal arguments (Subject, Verb, Verb Modifier, Object) via an automated Hungarian NLP pipeline, converted into distinct IS categories according to question condition, and subjected to omnibus non-parametric significance testing against competing cognitive pressures.

Key Designs

1. Controlled VQA Stimuli & Prompt Formulation: Isolating Discourse Status with Overt Modifiers To ensure that syntactic word order directly reflects communicative intent rather than extraneous lexical or perceptual artifacts, the experimental design strictly neutralises semantic reversibility, lexical frequency, and animacy confounds. All 47 visual stimuli depict interactions between two human animate characters engaged in semantically reversible actions, eliminating animacy hierarchies as an alternative driver of constituent order. Lemma frequencies for character names were verified against the Hungarian Gigaword Corpus, demonstrating uniformly high frequencies (mean log10 frequency of 4.35 with a standard deviation of 0.61). Critically, every action verb is paired with a separable verb modifier (igekรถtล‘, such as le- or meg-). In Hungarian grammar, an identificational Focus constituent immediately preceding the verb systematically displaces this prefix into a postverbal position (yielding Focus-Verb-Modifier), whereas neutral or topic-initial sentences retain the prefix before the verb (Modifier-Verb). Human participants received spoken prompts simulating a collaborative image-disambiguation game to ensure ecological validity, while VLMs received standardized system instructions requesting a concise, complete single-sentence response identifying both participants and their event.

2. Bimodal Multi-Run Sampling & Bootstrap Convergence: Rigorous Stochastic Control Because generative multimodal models exhibit non-deterministic decoding variability across random seeds, drawing reliable conclusions about probability distributions requires principled sample size estimation. The authors evaluated six models spanning proprietary APIs (Claude Opus 4, Gemini 2.5 Pro, GPT-4.1) and open-weight architectures (Gemma 3 12B/27B, Mistral Small 3.1 24B). Using a power analysis on contingency tables with Cramรฉr's \(V\), the authors established that while 11 to 13 responses achieve statistical power (\(\pi = 0.8, \alpha = 0.05\)) for robust models, noisy models like Mistral Small 3.1 require at least 55. To eliminate small-sample noise, the authors executed 30 complete runs per model (31 for Gemini 2.5 Pro), comprising 94 queries per run (47 items \(\times\) 2 conditions). A bootstrap convergence protocol with 300 resamples monitored the standard deviation of Cramรฉr's \(V\), enforcing a stopping criterion where variance fell below a tight threshold of 0.02, guaranteeing that reported distributional medians represent stable model characteristics.

3. Automated Dependency Parsing & Rule-Based Enhancement: High-Fidelity Syntactic Recovery Annotating over 14,000 model-generated utterances manually is infeasible, while standard Hungarian parsers frequently stumble over split verbal prefixes and irregular clause structures. The authors deployed huSpaCy (hu_core_news_lg v3.8.1) to recover the root verb (ROOT), nominal subject (nsubj), direct object (obj), and verbal modifier (compound:preverb). The raw parses are refined via four dedicated structural heuristics: first, restricting extraction exclusively to immediate dependents of the root verb to eliminate subordinate clause interference; second, implementing a preverb insertion heuristic that inspects whether an unseparated prefix from the question is morphologically fused to the verb stem when huSpaCy misses split tokens; third, collapsing consecutive duplicate role labels; and fourth, enforcing a strict four-role filter (requiring S, O, V, and I) that discards unclassifiable sentences. Benchmarked against 658 expert-annotated model outputs, this pipeline achieved 90.7% exact agreement, with genuine misclassifications occurring in only 1.5% of cases.

4. Information-Structural Mapping & Pressure-Conflict Analysis: Disentangling Competing Drivers The foundational analytical contribution lies in decoupling pure discourse conditioning from grammatical role and definiteness biases. Under the object-focus condition (e.g., answering Who is the boy painting?), discourse pressure (marking the given subject as Topic) aligns perfectly with universal grammatical-role priors (preferring subjects in topic position), yielding canonical SOVI word order (Top-preVF). Conversely, under the subject-focus condition (answering Who is painting the girl?), discourse pressure dictates fronting the given object as Topic (OSVI / OIVS), which directly conflicts with the subject-first grammatical prior. Furthermore, discourse theory predicts that Focus introduces new entities and thus favours indefinite determiners, whereas subjects independently attract definite marking. By tracking both the drop in topicalisation probability \(\Delta P(\text{Topic})\) between conditions and the proportion of indefinite noun phrases occupying the preverbal Focus slot, the protocol precisely diagnoses whether models track discourse context or rely on superficial grammatical heuristics.

Benchmark & Evaluation Protocol

The empirical evaluation compares 48 valid native Hungarian participants (1,413 analyzed utterances) against six instruction-tuned VLMs across 14,294 analyzed utterances. Performance is tracked along four primary dimensions: 1. Sentence-Type Distribution: Relative proportions of preverbal focus with topic (Top-preVF), postverbal focus with topic (Top-postVF), bare preverbal focus without topic (preVF), canonical order without focus (default), and IS-incongruent errors (error); 2. Topicalisation Shift (\(\Delta P(\text{Topic})\)): Magnitude of topicalisation probability drop when transitioning from subject-topic alignment to object-topic conflict; 3. Focus Definiteness Rate: Median proportion of indefinite NPs in the preverbal Focus position across conditions; 4. Human Expert Linguistic Judgments: Double-blind Likert ratings (1-5) evaluating lexical choice, morphosyntactic well-formedness, semantic fidelity, and holistic naturalness.

Key Experimental Results

Main Results

The table below presents the full distribution (%) of syntactic sentence types and corresponding information-structural functions across models and focus conditions (quoted directly from Table 4 of the paper):

Model / Group Condition default (%) preVF (%) Top-preVF (%) Top-postVF (%) error (%) Total (n)
Human OBJ_FOC 0.0 18.4 81.0 0.6 0.0 705
SUBJ_FOC 0.1 31.6 68.1 0.1 0.0 708
Total 0.1 25.1 74.5 0.4 0.0 1413
Claude Opus 4 OBJ_FOC 0.0 0.0 84.8 15.0 0.2 1298
SUBJ_FOC 0.2 99.7 0.1 0.0 0.1 1221
Total 0.1 48.3 43.7 7.7 0.1 2519
Gemini 2.5 Pro OBJ_FOC 0.0 0.0 96.3 2.0 1.8 1371
SUBJ_FOC 6.5 90.0 3.5 0.0 0.1 1404
Total 3.3 45.5 49.3 1.0 0.9 2775
Gemma 3 12B OBJ_FOC 0.0 0.0 98.3 1.0 0.7 841
SUBJ_FOC 7.1 90.1 0.5 0.0 2.3 1005
Total 3.8 49.1 45.1 0.4 1.6 1846
Gemma 3 27B OBJ_FOC 0.0 0.0 92.3 0.4 7.3 1132
SUBJ_FOC 0.0 99.6 0.4 0.0 0.0 1343
Total 0.0 54.0 42.5 0.2 3.4 2475
GPT-4.1 OBJ_FOC 0.0 0.0 76.5 23.4 0.1 1348
SUBJ_FOC 2.3 62.8 35.0 0.0 0.0 1356
Total 1.1 31.5 55.7 11.7 0.0 2704
Mistral Small 3.1 OBJ_FOC 0.0 3.5 18.9 27.3 50.3 715
SUBJ_FOC 6.3 73.0 20.2 0.0 0.4 1260
Total 4.1 47.8 19.7 9.9 18.5 1975
All VLM (Aggregated) OBJ_FOC 0.0 0.4 81.4 11.1 7.1 6705
SUBJ_FOC 3.6 85.6 10.4 0.0 0.4 7589
Total 1.9 45.6 43.7 5.2 3.5 14294

Ablation & Conflict Analysis: Focus Definiteness Across Conditions

The table below details median indefiniteness rates for preverbal Focus NPs across focus conditions and statistical significance under Bonferroni-corrected Wilcoxon tests (quoted directly from Table 7 and Section 4):

Model / Participant Group SUBJ_FOC Median OBJ_FOC Median Shift Direction Statistical Notes (Bonferroni Corrected)
Human ~0.43 ~0.43 None \(p_{bonf} = 0.834\) (n.s.); maintains balanced, high indefiniteness (~43%) across grammatical roles
Claude Opus 4 0.05 0.12 OBJ > SUBJ Moderate and statistically significant (\(p < 0.0001\)); tracks grammatical object
Gemini 2.5 Pro 0.03 0.10 OBJ > SUBJ Statistically significant (\(p < 0.0001\)); comparable magnitude to Claude Opus 4
Gemma 3 12B 0.00 0.41 OBJ > SUBJ Largest observed disparity; subject focus is categorically definite (0.00)
Gemma 3 27B 0.02 0.12 OBJ > SUBJ Statistically significant; consistent with 12B pattern, slightly attenuated
Mistral Small 3.1 0.02 0.00 SUBJ > OBJ Reversed direction; artifact of limited language competence and 50.3% error rate
GPT-4.1 0.00 0.00 None \(p_{bonf} > 0.05\) (n.s.); near-categorical definiteness across all conditions (extreme regularisation)

Key Findings

  • Alignment Convergence vs. Conflict Collapse: Under aligned discourse and grammatical conditions (object-focus), VLMs match human preferences closely, producing 81.4% Top-preVF responses compared to humans' 81.0%. However, when discourse and grammatical priors conflict (subject-focus requiring object topicalisation), human speakers preserve 68.1% Top-preVF topicalisation, whereas VLMs experience a severe collapse to 10.4% (an 82.1-point plunge), retreating into bare preverbal focus without a topic (preVF surging to 85.6%).
  • Definiteness Tracks Grammatical Role Rather Than Discourse Focus: Humans apply indefinite marking to newly introduced focus entities at roughly equivalent rates (~43%) regardless of whether the focus is a subject or an object. In contrast, VLMs exhibit a sharp grammatical-role asymmetry (Kruskal-Wallis omnibus \(H = 233.29, p < 0.0001\)): Claude, Gemini, and Gemma models mark focus entities as indefinite significantly more often when they are direct objects than when they are subjects. This demonstrates that models rely on surface grammatical-role correlations rather than true discourse-level novelty.
  • Divergent Error Modalities (GPT-4.1 vs. Mistral Small 3.1): While Mistral Small 3.1 fails due to linguistic incompetence in Hungarian (generating 50.3% ungrammatical or infelicitous errors under object focus), GPT-4.1 produces fully grammatical outputs that exhibit an extreme form of post-training over-regularisation, collapsing onto 0.00% indefinite focus expressions and displaying an artificially rigid template structure.

Highlights & Insights

  • First Systematic Probe of Information Structure in Multimodal Models: The paper pioneer a novel evaluation frontier by showing that identifying correct visual facts is insufficient; models must also package facts in accordance with discourse flow.
  • Connecting Linguistic Over-Regularisation to Generative Mode Collapse: The authors provide compelling empirical evidence that post-training (RLHF and instruction tuning) causes models to discard the diverse, valid communicative strategies present in human language distributions, collapsing onto narrow, high-probability syntactic modes.
  • Exploiting Non-English Syntactic Properties as Diagnostic Tools: By turning to Hungarian word order and verbal modifier inversion, the study demonstrates how typologically diverse languages can serve as crystalline, non-invasive diagnostic lenses into LLM internal representations.

Limitations & Future Work

  • Single-Language and Constrained Semantic Domain: Findings are situated entirely within Hungarian and tested on simple, semantically reversible transitive cartoons between animate humans, leaving open whether similar collapse occurs across intransitive, ditransitive, or passive constructions in other discourse-configurational languages.
  • Pronounced Character Gender Skew in Visual Stimuli: Out of 24 unique cartoon characters, 20 are male and only 4 are female. While Hungarian lacks grammatical gender, lexical gender associations prevented evaluating how gender biases might interact with topicalisation choices.
  • Uncontrolled Closed-Model Hyperparameters and Training Corpora: Proprietary models accessed via API (Claude Opus 4, Gemini 2.5 Pro, GPT-4.1) prevent inspectability into pretraining data distributions, temperature variability, or latent representation probing.
  • vs. Cuneo et al. (2025): While Cuneo et al. probed acceptability judgments on English long-distance dependencies, this study establishes the first generative benchmark directly evaluating naturalistic, visually grounded information packaging.
  • vs. MultiBLiMP & Syntactic Minimal Pairs (Warstadt et al., 2020; Jumelet et al., 2026): Unlike minimal-pair perplexity scoring that evaluates isolated sentence probabilities, this framework evaluates open generation under discourse pragmatics and multi-agent common ground.
  • Implications for Foundation Model Alignment: Standard RLHF rewards modal, unambiguous responses, directly driving the mode collapse identified here. Future alignment protocols must incorporate distributional entropy constraints and discourse-level pragmatic diversity to preserve human-like communicative richness.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering exploration of information structure in multimodal models using a discourse-configurational language.]
  • Experimental Thoroughness: โญโญโญโญโญ [Exemplary methodology combining 48 human participants, 6 VLMs across 30 bootstrap-converged runs, and expert double-blind validation.]
  • Writing Quality: โญโญโญโญโญ [Impeccably written, rigorous theoretical grounding, and crystal-clear argumentation.]
  • Value: โญโญโญโญโญ [Sets a transformative standard for linguistic evaluation beyond surface content accuracy toward true pragmatic competence.]