Skip to content

DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Lanni-ni/DiscoTrack
Area: NLP Understanding
Keywords: discourse tracking, discourse relations, bridging inference, entity salience, multilingual evaluation

TL;DR

DiscoTrack establishes a standardized question-answering benchmark spanning 12 typologically diverse languages and 11 subtasks across four discourse tiers (salience, entity tracking, discourse relations, bridging inference), demonstrating that even frontier LLMs struggle with implicit document-level discourse reasoning.

Background & Motivation

Large language models have demonstrated impressive proficiency across standard benchmarks such as GLUE, SuperGLUE, and MMLU; however, the vast majority of existing natural language understanding suites primarily target explicit factual extraction from single sentences or localized passages. Models can often succeed by performing keyword matching or pattern recognition over isolated sentences without engaging in genuine document-level reasoning. While recent efforts such as Disco-Bench attempted to address discourse-aware evaluation, they were restricted to Chinese and English and relied on high-level proxy tasks such as reading comprehension and text expansion rather than directly probing underlying discourse processing capabilities. Concurrently, benchmarks such as BeDiscovER target temporal reasoning and sentence ordering, yet leave out critical document-spanning phenomena including whole-document entity tracking, graded salience ranking, and associative bridging inference.

True discourse understanding in computational linguistics hinges upon resolving implicit structural relations, priority hierarchies, and unexpressed semantic ties distributed across multi-sentence paragraphs and multi-party dialogues. Models must continuously maintain discourse referents, track non-identical associative antecedents in bridging anaphora, and infer unstated causal or temporal relations between propositions. Prior linguistic resourcesโ€”such as GUM-SAGE, CorefUD, DISRPT, and Czech-PDTโ€”contain rich expert annotations for these phenomena; however, their disparate task formulations, specialized graph or tree representations, and divergent tagsets have prevented their direct adoption as a unified, accessible LLM evaluation testbed.

To close this diagnostic gap, this paper reformulates high-quality, heterogeneous multilingual discourse corpora into an aligned, instruction-following question-answering suite. Core idea: DiscoTrack unifies heterogeneous discourse resources across 12 typologically diverse languages into a standardized 11-subtask benchmark covering salience recognition, entity tracking, discourse relations, and bridging inference, directly diagnosing the failure modes of large language models on document-spanning implicit discourse structures.

Method

Overall Architecture

The DiscoTrack benchmark pipeline is organized into four core functional modules reflecting essential layers of computational discourse processing: Salience Recognition, Entity Tracking, Discourse Relations, and Bridging Inference. Long-document texts and task-specific instruction prompts are fed into target language models under greedy decoding, producing structured text outputs that are strictly parsed against gold-standard references and verified against human performance baselines to generate multi-tier diagnostic metrics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Multilingual Input Corpora<br/>GUM / CorefUD / DISRPT / PDT"] --> B["Standardized Reformulation of Discourse Corpora<br/>QA prompt conversion & unified tagsets"]
    B --> C["Four-Tier Decoupled Task Formulation<br/>11 subtasks across salience/tracking/relations/bridging"]
    C --> D["Adversarial Hard Negatives & Balanced Sampling<br/>Coreference system false positives & frequency strata"]
    D --> E["Cross-Lingual & Human-Aligned Evaluation<br/>12 languages in zero/few-shot & dual-pass human baseline"]
    E --> F["Diagnostic Evaluation & Error Analysis Output"]

Key Designs

1. Standardized Reformulation of Discourse Corpora: Transforming specialized linguistic annotations into leak-free generative QA prompts Traditional computational discourse resources typically rely on specialized graph-based, CoNLL, or XML span annotations that do not natively align with text-in, text-out generative LLM interfaces. DiscoTrack resolves this friction by translating multi-source corpora (GUM-SAGE, CorefUD 1.3, DISRPT, GUMBridge, and Czech-PDT) into clean natural language prompts. For categorical relation prediction, it standardizes label inventories into 17 unified discourse relations based on ISO DR-Core and PDTB, 11 fine-grained English bridging types, and 5 coarse-grained Czech bridging types, complete with definitions. For mention extraction tasks, the prompt imposes unambiguous extraction constraints (requiring exact verbatim substrings while filtering pronouns, speaker tags, and ambiguous truncated mentions), ensuring rigorous evaluation across both open-weight and proprietary models.

2. Four-Tier Decoupled Task Formulation: Isolating discourse phenomena from global priority to implicit propositional links To prevent macro-level task performance from conflating disparate capabilities, DiscoTrack explicitly decouples discourse tracking into 11 specialized subtasks across four tiers: In the Salience Recognition tier, models must extract the top-3 most prominent entities aligned with five independent human summaries (SALTOP) or perform binary relative importance comparisons between entities with graded salience scores from 1 to 5 (SALREL); In the Entity Tracking tier, models must identify the single most mentioned entity across the full document (ENTMAX), compare mention frequencies between two candidate entities across balanced frequency bins (ENTMORE), and resolve coreference under adversarial distractors (ENTCOREF); In the Discourse Relations tier, models perform 17-way relation classification between argument spans (RELCLF), mark explicit discourse connectives within sentences (RELEXP), and supply human-appropriate implicit connectives connecting propositions lacking lexical markers (RELIMP); In the Bridging Inference tier, models detect associative bridging anaphors (BRIDGDET), classify fine/coarse semantic bridging categories (BRIDGTYPE), and identify associative antecedents in preceding discourse (BRIDGANTE).

3. Adversarial Hard Negatives & Balanced Sampling: Eliminating lexical shortcuts and frequency heuristics Binary and multi-choice discourse tasks are vulnerable to models exploiting superficial word frequency or simple lexical co-occurrence. To prevent shortcut learning, DiscoTrack constructs stratified and adversarial test sets. In ENTMORE, candidate pairs are sampled across five distinct frequency bins (1โ€“4, 5โ€“8, 9โ€“12, 13โ€“16, 17โ€“20 mentions) with controlled frequency differences of 1 to 5 mentions, preventing models from relying on crude contrast heuristics. In ENTCOREF, instead of random negative pairs, negative samples are mined using a state-of-the-art multilingual neural coreference system (MSCAW-coref), selecting its highest-probability false-positive antecedent and verifying that its mention span exists in the gold annotations. Furthermore, in BRIDGDET and RELIMP, realistic negative examples requiring "no explicit connective" or "no bridging anaphor" are injected to test resistance to spurious association.

4. Cross-Lingual & Human-Aligned Evaluation: Providing balanced coverage across 12 languages and statistically verified human baselines Discourse annotated data is notoriously scarce outside English. DiscoTrack covers 12 typologically diverse languages (English, Chinese, Czech, French, German, Italian, Polish, Portuguese, Russian, Spanish, Thai, and Turkish), balancing subtask allocations according to resource availability (e.g., RELCLF features 500 instances across all 12 languages). Crucially, the benchmark establishes a rigorous human evaluation baseline across English subtasks via two independent annotation passes with an inter-annotator agreement of 78.9% (achieving 74.1% and 68.5% accuracy). This human reference statistically outperforms the best LLM setting, GPT-5 few-shot at 51.1% (\(\chi^2 = 30.40, p = 3.51 \times 10^{-8}\) for Annotator 1; \(\chi^2 = 17.02, p = 3.70 \times 10^{-5}\) for Annotator 2), providing an empirical performance ceiling for the community.

Key Experimental Results

Main Results

The benchmark evaluates open-source models (mt0-XL, Qwen-1.5-7B, Aya-Expanse 8B) alongside leading proprietary frontier models (GPT-4o mini, GPT-5) under both zero-shot and few-shot conditions, benchmarked against human annotator performance.

The following table presents micro-average performance scores (accuracy / F1, %) across the core evaluation suite:

Model / Setting BRIDGDET BRIDGTYPE (Fine / Coarse) BRIDGANTE SALREL SALTOP ENTMORE ENTCOREF RELCLF RELEXP RELIMP
mt0-XL (zero) 45.49 6.85 / 12.33 2.05 54.20 23.30 44.30 38.18 4.52 0.00 19.90
mt0-XL (few) 50.21 1.37 / 47.03 5.13 13.80 4.80 45.58 42.87 3.38 0.13 17.29
Qwen-1.5-7B (zero) 48.28 30.14 / 47.95 0.51 52.80 35.10 37.60 54.43 6.48 7.09 14.17
Qwen-1.5-7B (few) 52.15 15.98 / 49.40 2.56 59.20 41.22 33.45 45.06 5.28 18.07 6.83
Aya-Expanse 8B (zero) 50.21 29.68 / 47.03 8.21 52.20 38.65 31.13 55.37 3.38 0.00 13.16
Aya-Expanse 8B (few) 62.23 29.69 / 47.03 1.03 49.60 39.45 29.13 51.49 14.72 9.09 4.01
GPT-4o mini (zero) 57.30 32.42 / 54.34 18.97 52.60 39.61 64.63 69.70 28.00 48.60 13.97
GPT-4o mini (few) 49.14 30.59 / 52.97 42.05 67.00 43.16 66.38 72.97 28.57 50.37 19.47
GPT-5 (zero) 53.65 21.92 / 46.56 38.46 54.20 56.20 64.58 60.71 48.02 39.59 11.74
GPT-5 (few) 54.29 31.51 / 58.90 38.97 67.80 59.26 64.65 73.22 38.37 42.94 23.33
Human Reference 80.00 50.00 / 83.30 60.00 76.70 87.80 93.30 93.30 80.00 86.70 40.00

Overall macro scores across language subsets and evaluation partitions are reported below:

Model English Task Avg Multilingual: Eng Multilingual: Non-Eng Multilingual: Overall All Tasks Avg
MT0-XL (zero) 24.04 21.54 21.55 21.56 22.80
MT0-XL (few) 20.39 17.00 18.89 18.66 19.53
Qwen-1.5-7B (zero) 35.80 26.36 22.22 22.21 29.01
Qwen-1.5-7B (few) 36.75 24.23 21.17 20.09 28.42
Aya-Expanse 8B (zero) 37.66 23.67 18.20 18.86 28.26
Aya-Expanse 8B (few) 38.17 24.73 18.48 20.09 29.13
GPT-4o mini (zero) 42.54 45.50 43.50 43.93 43.24
GPT-4o mini (few) 47.49 50.25 46.25 46.54 47.02
GPT-5 (zero) 41.17 42.25 43.74 44.61 42.89
GPT-5 (few) 51.79 47.27 46.35 46.33 49.06

Ablation & Cross-Lingual Analysis: Multilingual Entity Tracking

To analyze whether models consistently perform better on English due to pretraining volume, the authors break down ENTMAX (identifying the single most frequently mentioned entity, accuracy %) across 8 languages:

Model / Setting English (EN) German (DE) Polish (PL) Turkish (TR) Czech (CS) French (FR) Russian (RU) Spanish (ES)
MT0-XL (zero) 28.21 9.50 18.60 13.64 23.00 16.78 34.64 35.40
MT0-XL (few) 2.99 3.17 10.20 4.55 0.40 0.00 0.00 0.20
Qwen-1.5-7B (zero) 22.22 3.62 10.00 4.55 12.80 11.89 15.64 27.20
Qwen-1.5-7B (few) 23.50 6.33 16.20 13.64 8.60 0.00 0.00 26.40
Aya-Expanse 8B (zero) 25.21 1.81 1.40 0.00 15.20 2.10 34.08 0.80
Aya-Expanse 8B (few) 29.91 1.81 1.00 0.00 22.00 5.59 35.20 1.20
GPT-4o mini (zero) 31.62 23.52 29.20 18.20 63.40 34.27 56.42 53.20
GPT-4o mini (few) 35.04 19.46 31.80 31.82 66.20 32.17 59.78 55.60
GPT-5 (zero) 37.18 26.24 28.20 22.73 70.40 32.17 58.66 57.20
GPT-5 (few) 32.91 25.79 31.80 27.27 45.80 27.97 50.84 45.40

Key Findings

  • LLMs substantially lag behind human competence on discourse tracking: Even GPT-5 few-shot, the strongest evaluated configuration, attains an overall average of only 49.06%, while human baseline annotators achieve 74.1%โ€“68.5% on corresponding subsets and exceed 93% accuracy on coreference comparison tasks (ENTMORE and ENTCOREF). In bridging antecedent resolution (BRIDGANTE), models top out at 42.05% versus 60.00% for humans.
  • Few-shot prompting can degrade discourse reasoning: For smaller open-source models such as MT0-XL and Qwen-1.5-7B, few-shot demonstrations often act as distractors within the context window, causing steep performance drops (e.g., MT0-XL dropping from 28.21% to 2.99% on ENTMAX). Even for GPT-4o mini, few-shot prompts reduce performance on BRIDGDET from 57.30% to 49.14%.
  • Non-English performance can exceed English due to genre composition: Contrary to the common assumption that English always achieves higher accuracy due to pretraining representation, ENTMAX scores for GPT-4o mini and GPT-5 on Czech (63%โ€“70%), Russian (56%โ€“59%), and Spanish (53%โ€“57%) are markedly higher than on English (31%โ€“37%). This discrepancy stems from corpus genre: English GUM incorporates complex spoken dialogues and informal genres, whereas East-Slavic and Romance CorefUD subsets consist mostly of well-formed written news text.
  • Persistent confusion between causal and explanatory relations: Confusion matrix analysis indicates that models struggle to differentiate between objective causal relations (CAUSAL, e.g., physical consequence) and epistemic/justification claims (EXPLANATION, e.g., providing reasons for an assertion), with 222 causal instances erroneously classified as explanation by zero-shot GPT-5.

Highlights & Insights

  • First comprehensive standardization of multilingual computational discourse: Bridges the gap between specialized linguistic resources (GUM, CorefUD, DISRPT, PDT) and generative LLM benchmarking by mapping divergent annotations into standardized QA prompts.
  • Model-in-the-loop adversarial coreference pair generation: Employs an external multilingual neural coreference system (MSCAW-coref) to select high-confidence system errors as hard negatives, forcing models to resolve subtle category distinctions rather than relying on trivial entity boundaries.
  • Disentangling tracking logic from aggregate numerical reasoning: Highlights that poor performance on ENTMAX stems from a dual bottleneckโ€”both anaphoric chain tracking and long-context frequency tallyingโ€”which is successfully isolated via the pairwise ENTMORE formulation.

Limitations & Future Work

  • License-restricted language and task availability: Due to varying academic licenses across DISRPT and CorefUD, several high-quality datasets could not be incorporated; moreover, salience tasks remain English-only due to the absence of graded salience datasets in other languages.
  • Absence of specialized prompting and reasoning scaffolds: The benchmark was evaluated using minimal direct prompting without exploring chain-of-thought (CoT), scratchpads for tracking entity mention lists, or multi-step verification agents, which could offer performance improvements on long documents.
  • vs Disco-Bench (Wang et al., 2023): Disco-Bench covers only English and Chinese and focuses heavily on text expansion, MT, and reading comprehension; DiscoTrack expands to 12 languages and directly targets four foundational linguistic tiers (salience, coreference tracking, relation classification, bridging anaphora) in a strictly decoupled fashion.
  • vs BeDiscovER (Li and Carenini, 2026): BeDiscovER targets temporal reasoning, discourse markers, and sentence ordering, but omits entity salience ranking, long-range coreference tracking, and bridging resolution; DiscoTrack fills this critical gap in holistic document-level implicit reasoning.

Rating

  • Novelty: โญโญโญโญโญ Formulates the first 12-language, four-tier benchmark targeting implicit discourse-level tracking for LLMs.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous coverage of 5 models across zero- and few-shot settings, complemented by a dual-pass human evaluation and deep error analysis.
  • Writing Quality: โญโญโญโญโญ Precise linguistic formulations, well-motivated task structures, and clear presentation of data distributions.
  • Value: โญโญโญโญโญ Provides an indispensable diagnostic instrument for pushing LLM capabilities beyond sentence-level extraction toward document-level cohesion.