Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Discourse Particles, Pragmatics, Colloquial Malay, Pragmatic Scaffolding, LLM Evaluation
TL;DR¶
To tackle the struggle of LLMs in capturing subtle pragmatic nuances of discourse particles in low-resource Southeast Asian languages, this paper introduces the MALAYPRAG benchmark along with a theoretically grounded five-attribute scaffolding framework, substantially elevating model accuracy across pragmatic function inference and particle slot-filling.
Background & Motivation¶
As the development of large language models rapidly shifts toward human-like, natural conversational interaction, researchers have increasingly recognized the pivotal role of discourse particles—such as English well, like, and kind of—in orchestrating pragmatic communication. Discourse particles typically do not alter the propositional truth conditions of an utterance, yet they fundamentally convey epistemic stances, negotiate common ground, signal interpersonal politeness, and structure discursive flow. Despite their communicative ubiquity, contemporary pragmatic evaluation suites (such as PUB) overwhelmingly concentrate on speech acts and implicatures in high-resource Western languages, largely ignoring low-resource Southeast Asian languages that possess intricate, highly polysemous particle systems.
The core tension underlying the poor performance of LLMs lies in the prevalent direct annotation paradigm: traditional NLP pragmatics projects map utterances straight onto coarse-grained, top-level functional labels, naively expecting end-to-end models to acquire abstract conversational intentions from raw text. In spoken interaction, however, the interpretation of particles is governed by latent first-order pragmatic drivers, such as speaker epistemic authority, assumed listener knowledge, and emotional arousal. Directly predicting macro-level pragmatic categories leads to severe data sparseness, poor inter-rater reliability among human annotators, and leaves language models unable to build reliable causal mappings between surface syntax and deeper communicative intent.
To break this black-box impasse, this study grounds abstract functional categories in discrete, computationally measurable linguistic variables. Core idea: Construct MALAYPRAG as a rigorously verified colloquial Malay benchmark around contrastive particles kan and ke, establishing a theory-grounded five-attribute framework (Epistemic Stance, Listener Agreement, Emotion, Question Type, and Particle Position) that acts as an explicit cognitive scaffold to bridge discourse particle generation and pragmatic function interpretation in LLMs.
Method¶
Overall Architecture¶
The proposed framework encompasses an end-to-end research methodology spanning raw social media data collection, controlled counterfactual synthesis, linguistic attribute annotation, unsupervised functional clustering, and progressive LLM probing tasks. In the data construction phase, informal Malay utterances from Reddit and Twitter/X were curated and augmented with balanced counterfactual substitutions before being labeled with five theoretical attributes by trained native linguists. In the extraction phase, K-means clustering over the annotated attributes was coupled with qualitative linguistic review to establish an overarching taxonomy of 7 pragmatic functions. Finally, ten general-purpose and regional LLMs were systematically benchmarked on MALAYPRAG across attribute prediction, function prediction, and masked particle generation to quantify the diagnostic impact of structured scaffolding.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Corpus Curation & Counterfactual Synthesis<br/>Informal Reddit / Twitter Utterances"] --> B["Five-Attribute Linguistic Annotation<br/>Epistemic Stance / Listener Agreement / Emotion / Question / Position"]
B --> C["Attribute Clustering & Pragmatic Taxonomy Extraction<br/>K-means Clustering and Qualitative Inductive Review"]
C --> D["Three-Tier Pragmatic Probing Benchmark<br/>Attribute / Function / Particle Prediction"]
D --> E["Explicit Scaffolding-Driven Inference<br/>Attribute-Enhanced Prompting & CoT Interaction"]
Key Designs¶
1. Five-Attribute Pragmatic Schema: Abstract pragmatic notions resist direct computation -> Grounding in first-order linguistic dimensions -> Minimizing annotation subjectivity and establishing causal inference bridges To resolve the low inter-rater reliability and subjective variance inherent in coarse-grained pragmatic labeling, the authors operationalize particle function into five theoretically grounded, discrete attributes: (1) Epistemic Stance: Certain, Uncertain, or Neutral, tracking the speaker's degree of knowledge commitment and epistemic authority; (2) Listener Agreement: Assumed Agreement, Confirmation Seeking, or Neutral, defining whether the proposition is treated as pre-existing common ground or an open proposition subject to dispute; (3) Emotion / Affect: Positive, Negative, or Neutral, capturing the affective payload; (4) Question Type: Declarative/Statement, Yes/No Interrogative, or Rhetorical Interrogative, encoding syntactic surface force; and (5) Particle Position: Front, Middle/End, or N/A. Chi-square tests of independence confirm that all five attributes are significantly correlated with the overarching pragmatic functions (\(p < 2.2 \times 10^{-16}\)), with Question Type (\(V = 0.893\)) and Listener Agreement (\(V = 0.863\)) displaying the strongest discriminative power.
2. Attribute-Driven Pragmatic Function Induction: Unsupervised cluster discovery lacks functional semantics -> K-means grouping combined with expert qualitative induction -> Constructing a robust 7-function pragmatic taxonomy Rather than arbitrarily imposing prescriptive functional categories, the pipeline performs K-means clustering (\(K=16\)) across the combined GOLD (\(N = 187\), triple-annotated with majority adjudication) and SILVER (\(N = 950\), single-annotated) datasets (\(N = 1,137\)). Native Malay linguists then conducted qualitative induction over utterances within the 16 clusters, consolidating them into 7 distinct, overarching pragmatic functions: Assumed-Agreement Rhetorical Stance, Neutral Declaratives, Information-Seeking Verification, Affective Confirmation-Seeking Question, Emphatic / Discourse Marking, Null Forms Retaining Pragmatic Meaning, and Negative Rhetorical Challenge / Evaluation. This bottom-up grouping ensures empirical fidelity while mitigating individual annotator bias.
3. Controlled Orthogonal Probing Protocol: Surface lexical cues confound true pragmatic disambiguation -> Balanced synthetic substitution paired with multi-condition prompting -> Disentangling syntactic memorization from deep communicative reasoning
In raw colloquial Malay corpora, occurrences of kan drastically outnumber ke. Direct training or evaluation risks models exploiting frequency heuristics rather than pragmatic reasoning. The authors synthesized neutral variants (removing kan) and counterfactual substituted utterances (replacing kan with ke, \(N = 113\)), ensuring that kan and ke appear in identical syntactic slots while communicating sharply diverging pragmatic intentions (e.g., claiming common ground vs. expressing genuine uncertainty or rhetorical challenge). Building upon this controlled setup, the evaluation introduces three progressive tasks: Task 1 (attribute prediction under raw and function-provided settings), Task 2 (pragmatic function prediction comparing direct prompting, CoT, and attribute scaffolding), and Task 3 (masked particle slot-filling [___] under five conditions evaluating raw, attribute-only, function-only, and combined attribute-function context).
Benchmark & Evaluation Protocol¶
The benchmark was administered to ten frontier LLMs: eight global general-purpose models (GPT-5, GPT-5.4-mini, Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.1 Flash, DeepSeek-v4-Pro, and DeepSeek-v4-Flash) and two open-weight Southeast Asia-specialized models (Llama-SEA-LION-70B and Gemma-SEA-LION-27B). All evaluations were executed on the verified GOLD split (\(N = 187\)) using greedy decoding (temperature = 0) constrained to closed label sets. The baseline random-guess performance corresponds to ~33% or 50% for attribute prediction, 14.3% for the 7-class pragmatic function prediction, and 33.3% for 3-class particle selection.
Key Experimental Results¶
Main Results¶
The primary diagnostic experiments in Task 2 (Pragmatic Function Prediction) and Task 3 (Masked Particle Prediction) demonstrate the decisive superiority of explicit attribute scaffolding over raw and chain-of-thought baselines:
| Model | Task 2 Direct | Task 2 CoT | Task 2 +Attributes | Task 2 \(\Delta\) (Attr - CoT) | Task 3 Direct | Task 3 +Attr. & Func. |
|---|---|---|---|---|---|---|
| GPT-5 | 0.300 | 0.396 | 0.529 | +0.133 | 0.513 | 0.690 |
| GPT-5.4-mini | 0.294 | 0.316 | 0.503 | +0.187 | 0.385 | 0.690 |
| Claude Sonnet 4.6 | 0.257 | 0.321 | 0.524 | +0.267 | 0.471 | 0.706 |
| Claude Haiku 4.5 | 0.283 | 0.348 | 0.524 | +0.241 | 0.471 | 0.711 |
| Gemini 3.1 Pro | 0.305 | 0.348 | 0.519 | +0.214 | 0.428 | 0.690 |
| Gemini 3.1 Flash | 0.337 | 0.342 | 0.524 | +0.187 | 0.433 | 0.679 |
| DeepSeek-v4-Pro | 0.316 | 0.326 | 0.588 | +0.272 | 0.476 | 0.754 |
| DeepSeek-v4-Flash | 0.353 | 0.364 | 0.546 | +0.193 | 0.449 | 0.717 |
| Llama-SEA-LION-70B | 0.123 | 0.219 | 0.401 | +0.182 | 0.332 | 0.711 |
| Gemma-SEA-LION-27B | 0.230 | 0.262 | 0.588 | +0.358 | 0.337 | 0.562 |
| Average | 0.2796 | 0.324 | 0.5246 | +0.201 | 0.430 | 0.691 |
Ablation Study¶
A fine-grained ablation over contextual prompt scaffolding in Task 3 (Masked-Slot Particle Prediction: kan, ke, or neutral) isolates the contribution of each linguistic tier:
| Context Prompt Condition | Mean Accuracy | Gain over Direct | Mechanism & Insight |
|---|---|---|---|
| (3a) Direct Prompting from Raw Data | 0.430 | — | Models lack pragmatic grounding, performing near chance |
| (3c) Function-provided (+ Pragmatic Function) | 0.531 | +0.101 | Top-level macro-intent imposes moderate functional constraints |
| (3b) Attribute-provided (+ Attributes) | 0.615 | +0.185 | Fine-grained attributes offer more concrete guidance than macro-labels |
| (3d) Attribute + Function Jointly Provided | 0.691 | +0.261 | Dual grounding achieves comprehensive pragmatic disambiguation |
| (3e) CoT + Function Provided | 0.689 | +0.259 | Elicited reasoning aids token selection, matching attribute scaffolding |
In Task 1 (individual attribute identification), Particle Position proved the easiest for models to predict (EN average: 84.54%), whereas Listener Agreement was the most difficult (EN average: 47.00%), paralleling human annotation disagreements. Furthermore, models evaluated with English prompts consistently outperformed those evaluated with Malay prompts (69.26% vs. 65.16%), exposing a meta-linguistic reasoning gap across prompt languages.
Key Findings¶
- Severe Deficiency in Direct Pragmatic Inference: Without external grounding, frontier models like GPT-5 and Claude Sonnet 4.6 achieve merely 25.7%–35.3% accuracy on pragmatic function prediction (averaging 27.96%), barely beating random guessing (14.3%). This proves that standard pretraining fails to impart implicit particle pragmatics in low-resource vernaculars.
- Structured Scaffolding Drastically Outperforms CoT: Providing the five attributes drives pragmatic function accuracy from 27.96% up to 52.46% (a diagnostic jump of +24.49%), markedly outpacing CoT prompting (32.4%). Because pragmatic intentions represent implicit interactional effects rather than deductive formal proofs, eliciting token-level reasoning steps alone fails to close the pragmatic gap.
- The Regional Model Paradox: Gemma-SEA-LION-27B scored poorly under zero-shot direct inference (23.0%), yet with attribute scaffolding its accuracy surged by +35.8% to 58.8%—matching DeepSeek-v4-Pro and surpassing GPT-5. While regional models suffer from smaller pretraining scales, their culturally aligned tokenizers and latent regional knowledge can be powerfully unlocked once explicit scaffolding bridges the pragmatic divide.
Highlights & Insights¶
- First Theory-Grounded Multi-Attribute Scaffold for Discourse Pragmatics: This paper moves beyond crude end-to-end classification by constructing a bidirectional, interpretable bridge: \(\text{Raw Utterance} \leftrightarrow \text{Five Theoretical Attributes} \leftrightarrow \text{Macro Pragmatic Functions}\).
- Unveiling the Asymmetric Utility of CoT in Semantics vs. Pragmatics: The findings sharply demonstrate that while CoT reasoning is reasonably effective for surface-level particle generation (Task 3), it falters in decoding latent interpersonal stances (Task 2), confirming that implicit pragmatics requires structural symbolic framing rather than unguided auto-regressive generation.
- Broad Portability across Regional Vernaculars: The proposed five-dimensional taxonomy (epistemic stance, listener orientation, affect, sentence type, and position) offers an off-the-shelf blueprint for analyzing discourse particles in Singlish (lah, leh, meh), Cantonese (aa3, wo3), and Thai.
Limitations & Future Work¶
- Lack of Multimodal Prosodic Signals: The benchmark is derived strictly from text-only social media data, stripping away vital acoustic cues such as pitch contours and intonation. Given that Listener Agreement exhibited the highest human and machine disagreement, integrating audio-speech modalities represents an essential next step.
- Prompting-Only Probing without Parameter Internalization: The current work relies exclusively on in-context prompting; investigating whether supervised fine-tuning (SFT) or preference optimization (DPO) can internalize attribute-level reasoning directly into weights remains open.
- Distributional Shift in Synthetic Balancing: The dataset synthesized 113 counterfactual
kesentences to offset the overwhelming natural predominance ofkan. Practitioners utilizing this corpus for model training must account for this synthetic distribution to avoid distorting real-world colloquial frequency priors.
Related Work & Insights¶
- vs Sheffield et al. (ACL 2025) English Discourse Particles: Sheffield et al. demonstrated that LLMs fail to disambiguate English just, where adding conversational context unexpectedly degrades performance. This paper shows that this failure is even more pronounced in low-resource vernaculars and presents a principled solution: structured attribute scaffolding actively restores and boosts pragmatic reasoning accuracy.
- vs Sravanthi et al. (2024) PUB Pragmatic Benchmark: While PUB benchmarks broad speech acts and conversational implicatures in high-resource regimes, this paper tackles sub-sentential grammaticalized particles in colloquial Southeast Asian language, establishing a pioneer diagnostic standard for low-resource pragmatics.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering multi-attribute scaffolding framework addressing the longstanding neglect of colloquial discourse particles]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous benchmarking of 10 frontier and regional LLMs across three progressive tasks with rich ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Flawlessly articulated linguistics grounding paired with dense, logically coherent empirical exposition]
- Value: ⭐⭐⭐⭐⭐ [Provides a foundational benchmark and an actionable methodology for aligning LLMs with human-like pragmatic competence in diverse languages]