Skip to content

Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cesare-spinoso/ImplicatureX
Area: NLP Understanding
Keywords: implicature cancellation, communicative belief updates, pragmatic reasoning, common ground benchmark, LLM evaluation

TL;DR

This paper introduces IMPLICATUREX, the first expert-annotated and crowdsourced benchmark formalizing communicative belief updates through implicature recognition and cancellation, revealing that leading LLMs lag substantially behind humans in naturally-occurring dialogues and suffer from severe belief inertia and prior memorization.

Background & Motivation

Human natural language communication is fundamentally driven by threads of unspoken beliefs dynamically negotiated within an evolving conversational common ground. Rather than spelling out every factual presupposition, speakers frequently introduce provisional commitments implicitly through pragmatic implicatures, which can subsequently be retracted or weakened when new communicative evidence arisesโ€”a phenomenon known in linguistics as implicature cancellation. For example, if a host remarks that "Cai was looking for someone to buy drinks," an interlocutor readily infers that Cai needs assistance. However, if the speaker promptly follows up with "Though they must have gotten around to it by now," the listener must update their internal representation of the common ground and cancel that belief. Tracking such non-monotonic communicative belief trajectories is essential if large language models (LLMs) are to become collaborative and pragmatically competent dialogue partners.

Despite its centrality in human discourse, non-monotonic reasoning and belief update evaluations in NLP have remained largely confined to formal logic puzzles and static world-knowledge editing. Prior pragmatic studies investigating conversational or scalar implicatures treat them primarily as static, single-turn classification benchmarks, overlooking whether an LLM can flexibly revoke an implicated proposition upon observing subsequent conversational turns. Consequently, the research community lacks systematic empirical insight into whether current language models engage in genuine pragmatic reasoning or merely rely on superficial statistical associations and static priors memorized during pretraining.

To bridge this gap, this paper formalizes communicative belief updates as probability shifts over propositions in the conversational common ground and constructs a rigorous linguistic benchmark. The core idea is to formalize dynamic communicative belief updates via implicature recognition and cancellation under a probabilistic common ground framework, diagnosing LLM pragmatic reasoning through the expert-verified IMPLICATUREX benchmark and counterfactual control experiments.

Method

Overall Architecture

The proposed methodology operationalizes communicative beliefs and updates within a probabilistic formulation of common ground, spanning benchmark creation, expert/crowdsourced validation, and counterfactual diagnostic controls. Given situational context \(c\), an utterance history \(u\), and an implicated proposition \(b\), the evaluation probes the model's posterior probability assignments over whether \(b\) holds. The framework quantitatively decomposes pragmatic understanding into implicature recognition accuracy, cancellation recognition accuracy, and joint belief update accuracy across four distinct linguistic genres.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Context c, Triggering Utterance u,<br/>and Candidate Belief b"] --> B["Formal Operationalization of Pragmatic Beliefs<br/>Probability shifts in common ground G"]
    B --> C["IMPLICATUREX Multi-genre Benchmark<br/>Expert double-annotation + Crowdsourced Likert z-scores"]
    C --> D["Counterfactual Diagnostic Controls<br/>Prior control / explicit negation / update types"]
    D --> E["Model Probability Extraction & Evaluation<br/>Joint recognition & non-monotonic update assessment"]

Key Designs

1. Formal Operationalization of Pragmatic Beliefs: addressing the unformalized tracking of conversational common ground

Classical linguistic pragmatics describes common ground qualitatively, making computational evaluation elusive. This work formalizes communication as an utterance sequence and introduces a common ground function \(G : B \times C \times U^* \to [0, 1]\) mapping a proposition \(b \in B\), context \(c \in C\), and utterance history \(u \in U^*\) to a subjective probability. An implicature introduces belief \(b\) into the common ground whenever utterance \(u\) makes \(b\) more probable than its negation: $\(G(b \mid c, \langle u \rangle) > G(\neg b \mid c, \langle u \rangle)\)$ Subsequently, a cancelling utterance \(u^\times\) extends the communicative history and accomplishes implicature cancellation if and only if it strictly reduces the mutually held probability: $\(G(b \mid c, \langle u, u^\times \rangle) < G(b \mid c, \langle u \rangle)\)$ In the experimental setup, \(G(b \mid c, u)\) is approximated by model probability \(P_M(b \mid t_b(c, u))\) using normalized next-token logits over "True" and "False" options (for open-weight models) or frequency sampling (for API models), with option ordering shuffled to prevent positional bias.

2. IMPLICATUREX Multi-genre Benchmark: addressing the scarcity of multi-genre implicature cancellation data

The authors construct IMPLICATUREX, a curated dataset of 271 items across four linguistic categories: 1. Scalar Implicatures (46 items): Built upon lexical entailment pairs \(\langle w_1, w_2 \rangle\) (e.g., \(\langle \text{some}, \text{all} \rangle\)) drawn from the Switchboard Corpus and augmented with expert-authored cancellations; 2. Discourse Implicatures (31 items): Mined from implicit causal relations in Wall Street Journal articles from the Penn Discourse Treebank (PDTB 3.0); 3. Synthetic Conversational Implicatures (144 items): Multi-turn pragmatic exchanges adapted from existing pragmatics datasets (George & Mamidi, Louis et al., Wilson & Bishop, Hu et al.); 4. Naturally-Occurring Conversational Implicatures (50 items): Extracted directly from spontaneous telephone dialogues in Switchboard, containing natural hesitation, complex conversational backgrounds, and realistic colloquial disfluencies.

To ensure pristine dataset quality, two professional linguists performed double-blind expert review, attaining 85% raw agreement on implicature plausibility (PABAK 0.70) and 86% on cancellation correctness (PABAK 0.72). Furthermore, 90 native English speakers were recruited on Prolific to provide 7-point Likert ratings with attention checks, producing z-scored human judgment baselines.

3. Counterfactual Diagnostic Controls: disentangling genuine pragmatic inference from memorized priors and syntax

To ascertain whether high model performance reflects true contextual reasoning or memorization artifacts, the authors design three targeted control conditions: - Prior Common Ground Control: Context \(c\) and utterance \(u\) are omitted entirely (\(t_b(\emptyset, \emptyset)\)), probing the LLM's unconditional prior belief \(P_M(b \mid t_b(\emptyset, \emptyset))\) to check if models score "some implies not all" purely by rote habit; - Form Control (IMPLICATURE\(^\perp\)): Replaces natural conversational cancellations with explicit syntactic negation \(u^\perp = \text{โ€œ}d + \neg b\text{โ€}\) featuring discourse markers like "actually" or "I don't mean to imply that," testing whether failures stem from subtle pragmatics or generic negation incomprehension; - Update Type Control (IMPLICATURE\(^+\) and IMPLICATURE\(^\approx\)): Introduces reinforcing utterances \(u^+\) (strengthening \(b\)) and neutral utterances \(u^\approx\) (irrelevant assertions leaving \(b\) unchanged), testing whether models can discriminate update directions or simply exhibit indiscriminate bias toward maintaining initial commitments.

A Worked Example

Consider a naturally-occurring dialogue item from Switchboard: - Context \(c\): S1 and S2 are chatting about golfing as a leisure activity. S2 mentions recently getting married and buying a house. - Triggering Utterance \(u\): "We're about fifty yards from the golf course --" - Implicated Belief \(b\): "S2 goes to play golf at the golf course near their house." - Cancelling Utterance \(u^\times\): "-- but unfortunately the house is taking up my time."

When exposed to \(u\), human annotators readily infer \(b\) (yielding positive z-scores). Upon reading \(u^\times\), human listeners swiftly revise their common ground and suppress \(b\) (z-scores dropping to negative values). However, state-of-the-art LLMs consistently fail on this example: the semantic salient tokens "fifty yards" and "golf course" persistently anchor the model's probability distribution, preventing it from executing the pragmatic belief revocation.

Key Experimental Results

Main Results

The benchmark assesses contemporary open models (Gemma 3 4B-27B, Llama 3.1/3.2/3.3 3B-70B, Qwen 2.5 3B-72B, Qwen 3 0.6B-32B with/without Thinking) alongside proprietary frontiers (GPT-5.2 and GPT-5.4 with/without Thinking). The primary quantitative results across all four categories are detailed below:

Model Task Scalar Imp. (46) Discourse Imp. (31) Synthetic Conv. (144) Natural Conv. (50)
Human (avg.) Implicature Recognition 1.000 0.903 0.910 0.780
Cancellation Recognition 1.000 0.903 0.972 0.920
Belief Update 1.000 0.903 0.903 0.720
GPT-5.4 Thinking Implicature Recognition 0.826 0.839 0.806 0.440
Cancellation Recognition 0.935 0.871 0.819 0.360
Belief Update 0.804 0.806 0.764 0.340
GPT-5.4 Implicature Recognition 0.891 0.903 0.764 0.440
Cancellation Recognition 0.870 0.645 0.750 0.380
Belief Update 0.826 0.645 0.708 0.280
Qwen 3 32B Thinking Implicature Recognition 0.935 0.903 0.771 0.420
Cancellation Recognition 0.978 0.903 0.951 0.840
Belief Update 0.913 0.839 0.764 0.400
Llama 3.3 70B Implicature Recognition 0.957 0.935 0.771 0.600
Cancellation Recognition 0.761 0.677 0.812 0.600
Belief Update 0.717 0.613 0.660 0.360
Gemma 3 27B Implicature Recognition 1.000 0.806 0.674 0.460
Cancellation Recognition 0.717 0.742 0.833 0.640
Belief Update 0.717 0.581 0.569 0.240

Ablation Study & Diagnostic Controls

The table below contrasts performance under zero-context prior probing, explicit negation cancellation (IMPLICATURE\(^\perp\)), and update type interventions (unchanging vs. strengthening):

Model Prior Control (Scalar) Prior Control (Natural) Explicit Negation (Natural) Belief Unchanging Accuracy (Natural) Belief Strengthening Accuracy (Natural)
Human (avg.) 1.000 0.780 0.920 - -
GPT-5.4 Thinking 0.957 0.200 0.500 0.960 0.320
GPT-5.4 0.935 0.260 0.500 0.940 0.200
Qwen 3 32B Thinking 0.957 0.420 0.960 0.840 0.680
Llama 3.3 70B 1.000 0.640 0.840 0.900 0.460
Gemma 3 27B 0.957 0.560 0.900 0.960 0.500

Key Findings

  • Severe Degradation on Naturally-Occurring Dialogues: While models achieve competitive accuracies on synthetic and scalar items, their ability to recognize implicatures collapses dramatically on spontaneous conversations (Switchboard). Except for Llama 3.3 70B (0.600), virtually all leading modelsโ€”including GPT-5.4 (0.440), GPT-5.4 Thinking (0.440), and Qwen 3 32B Thinking (0.420)โ€”perform worse than random chance (\(< 0.500\)), lagging far behind humans (0.780).
  • Scalar Success Driven by Pretraining Priors: The prior common ground control reveals that Llama 3.3 70B and Qwen 2.5 72B achieve perfect 1.000 implicature recognition without observing the context or triggering utterance at all. The apparent mastery of scalar reasoning is heavily confounded by lexicalized prior memorization rather than true pragmatic deduction.
  • Pronounced "Belief Inertia" in Language Models: The update type diagnostic reveals that LLMs strongly favor maintaining existing beliefs over modifying them. GPT-5.4 achieves 0.940-0.960 accuracy when presented with irrelevant neutral utterances, but drops to 0.280-0.340 when required to cancel beliefs and 0.200-0.320 when required to strengthen them.
  • Explicit Negation Fails to Eliminate Failure Modes: Supplying formulaic explicit negations (\(u^\perp\)) improves cancellation recognition on simple splits, yet GPT-5.4 still plateaus at 0.500 on naturally-occurring dialogues, demonstrating that communicative belief failures reflect deep pragmatic misunderstanding rather than simple negation misparsing.

Highlights & Insights

  • Operationalizing Pragmatic Defeasibility: Instead of static classification, the authors formulate implicature cancellation as a non-monotonic probability transition in conversational common ground, providing a mathematically grounded framework for dynamic dialogue evaluation.
  • Unmasking Prior Exploitation via Counterfactual Controls: By evaluating models on prompts stripped of all conversational context, the authors effectively debunked the assumption that scalar implicature mastery reflects situational pragmatic reasoning, setting a new rigorous control standard.
  • Broadly Reusable Diagnostic Suite: The contrasting triad of cancellation, strengthening, and unchanging follow-ups provides an adaptable probing paradigm for detecting belief inertia across conversational agents, dialogue state tracking, and long-horizon tool-use planning.

Limitations & Future Work

  • Subjectivity in Cancellation Annotations: Pragmatic interpretations inevitably exhibit inter-annotator variance; while five crowdsourced workers per item provided reliable consensus, certain borderline conversational implicatures retain subtle ambiguities.
  • Logit-Based Probability Elicitation vs. Open Generation: Extracting probabilities via next-token logits over multiple-choice options sidesteps the challenges of free-form generation evaluation, but does not directly assess an agent's ability to maintain behavioral consistency during prolonged autonomous conversations.
  • Monolingual Focus: The study exclusively evaluates English, leaving cross-linguistic pragmatic variations and cultural norms in implicature cancellation unexplored.
  • vs. IMPPRES & GLUE NLI Probes (Jeretic et al., 2020): Conventional NLI benchmarks treat presupposition and implicature as static sentence-pair entailment, lacking conversational history and dynamic revocation.
  • vs. Indirect Answer Benchmarks (Louis et al., 2020; Ruis et al., 2023): Prior datasets explore isolated question-indirect answer pairs, whereas this paper extends the evaluation to subsequent cancellations, evaluating belief lifecycles across turns.
  • vs. Defeasible NLI (Rudinger et al., 2020): While Defeasible NLI studies the non-monotonicity of general commonsense reasoning, this work targets interpersonal conversational common ground and communicative pragmatic commitments.

Rating

  • Novelty: โญโญโญโญโญ Pioneering formalization of implicature cancellation as dynamic common ground updates.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 20+ models, supported by expert double-annotation, crowdsourced z-scores, and rigorous counterfactual controls.
  • Writing Quality: โญโญโญโญโญ Eloquent narrative, mathematically rigorous formulations, and insightful critical discussion.
  • Value: โญโญโญโญโญ Exposes critical pragmatic blind spots and pervasive belief inertia in modern LLMs, setting a vital research agenda for communicative agent architectures.