When Humans and LLMs Disagree: Understanding Human-LLM Disagreement in Learner Intent Annotation for Educational Chatbot Dialogues¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/d-isasterhub/studychat-automatic-labeling
Area: Dialogue Systems
Keywords: Educational Chatbots, Intent Annotation, Human-LLM Disagreement, Weak Supervision, Learning-to-Learn
TL;DR¶
This paper introduces a Learning-to-Learn (L2L) theory-grounded hierarchical decision-tree annotation scheme for learner intent in educational chatbot interactions, revealing that while zero-shot LLMs exhibit agreement approaching human reliability (LLM \(\kappa = 0.67\) vs. human \(\kappa = 0.73\)), their alignment with human reference labels drops sharply (\(\kappa \le 0.52\)), and qualitative error auditing demonstrates that over half of disagreements stem from data ambiguity and human variability rather than model failure alone.
Background & Motivation¶
As generative large language model (LLM) chatbots become deeply embedded in educational ecosystems, educational researchers and learning analytics practitioners urgently require scalable methods to capture and understand authentic learner interactions. However, conventional dialogue act annotation frameworksโsuch as DAMSL, the Switchboard Dialogue Act Corpus (SwDA), and ISO 24617-2โwere established primarily for human-human everyday conversations. They categorize utterances according to broad communicative functions like questions, statements, or acknowledgments. In educational settings, this coarse-grained surface taxonomy completely obscures critical differences in learning behavior: asking "Can you give me the solution?" and "Does a for-loop repeat until the condition becomes false?" are both syntactically tagged as questions, even though the former seeks direct task offloading while the latter actively pursues conceptual understanding.
Bridging this gap requires grounding dialogue annotation in established pedagogical paradigms such as Learning-to-Learn (L2L) and Self-Regulated Learning (SRL). However, operationalizing psychological constructs into fine-grained discourse schemas poses severe methodological challenges. Deep cognitive and metacognitive operationsโsuch as internal reflection, mental model restructuring, and self-monitoringโare not directly observable in raw, multi-turn conversational logs. Early attempts to assign multi-label cognitive tags directly led to severe inter-annotator disagreement (with Krippendorff's \(\alpha\) dropping between -0.02 and 0.67). Furthermore, while automated LLM annotation offers a promising pathway to bypass the prohibitive cost of manual double-coding across thousands of interactions, existing NLP studies typically evaluate LLMs by treating human annotations as unassailable ground truth, relying solely on aggregate macro-metrics like accuracy or Cohen's \(\kappa\). This oversimplified setup ignores human annotator drift, personal bias, and inherent linguistic ambiguity.
To overcome these limitations, this paper investigates the nature of annotation disagreement rather than merely chasing higher automated benchmark scores. The authors combine deductive, theory-driven codebook development with inductive, empirical refinements on authentic interaction logs from a university-level computer science course (the StudyChat dataset). By restructuring overlapping cognitive constructs into a deterministic, single-label hierarchical decision tree centered on a prioritized solution-seeking versus knowledge-seeking distinction, the authors establish a rigorous human double-coded benchmark and evaluate two zero-shot LLMs across both intermediate sub-decisions and final learner intent categories. Core idea: transform implicit cognitive constructs into an observable, prioritized hierarchical decision flow for dialogue annotation, and leverage multi-agent human-LLM disagreement auditing to reframe divergent labels not as algorithmic failures, but as complementary weak supervision signals that expose data ambiguity and guide collaborative human-in-the-loop workflows.
Method¶
Overall Architecture¶
The proposed methodology encompasses three core stages: an L2L theory-grounded hierarchical decision tree scheme, a modular zero-shot automated annotation pipeline driven by structured DSPy signatures and AST-based retrieval, and an exhaustive qualitative disagreement auditing protocol. Given an authentic learner-chatbot dialogue exchange contextualized within course assignment files, the pipeline determines assignment mappings, task relation, and a single leaf intent label through sequential decisions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Learner-Chatbot Multi-turn Utterance<br/>Course Context & Assignment Materials"] --> B["1. Theory-Grounded Hierarchical Decision Tree<br/>Prioritized Solution-Seeking Routing"]
B --> C["2. Modular DSPy Prompting & AST Retrieval<br/>Dynamic Context Injection via ChromaDB"]
C --> D["3. Zero-Shot Hierarchical LLM Inference<br/>Sequential Decision Execution via GPT-5.4 & OSS-120B"]
D --> E["4. Multi-Pairing Agreement Discrepancy Auditing<br/>Human-Human / LLM-LLM / Human-LLM Matrix"]
E --> F["5. Disagreement Auditing & Weak Supervision Protocol<br/>Tri-Factor Root-Cause Taxonomy"]
Key Designs¶
1. Theory-Grounded Hierarchical Decision Tree: Unobservable Mental States -> Prioritized Solution-Seeking Routing -> Stable and Replicable Coding
Directly evaluating multi-label cognitive engagement in open-domain dialogues inevitably produces noisy annotations due to the frequent functional overlap between acquiring knowledge and obtaining code solutions. To eliminate this ambiguity, the framework imposes a strict priority rule: whenever an utterance could reasonably be interpreted as both solution seeking and knowledge seeking, it is strictly assigned to solution seeking if the chatbot's response would directly generate, complete, or require only trivial adaptation to solve an assignment subtask. The resulting hierarchical decision tree routes each prompt systematically: first evaluating whether it constitutes solution seeking; if yes, branching into solution type ("Create" vs. "Verify/Fix/Adapt"); if no, branching into whether it is knowledge seeking. For knowledge-seeking utterances, two orthogonal binary attributes are assessed: "Cognitive Engagement" (clear behavioral evidence of reacting to or building upon preceding chatbot responses) and "Mental Model" (explicitly testing, hypothesizing, or validating personal conceptual understanding).
2. Modular DSPy Prompting & AST Retrieval: Context Length Explosion and Noise -> Modular Signatures with AST Chunking -> Targeted Context Injection
To prevent prompt length bloat and reduce reasoning distraction, the automated pipeline rejects monolithic zero-shot prompts in favor of modular DSPy signatures tailored to each specific decision node. Assignment context is preprocessed and indexed into ChromaDB using all-MiniLM-L6-v2 embeddings: Python scripts are parsed via Abstract Syntax Tree (AST) to isolate function and class blocks, Jupyter notebooks are chunked cell-by-cell, and Markdown files are split along structural headings into segments of at most 2,000 characters. For solution-seeking decisions, the system retrieves the top-5 most similar task chunks alongside truncated chat histories (with past assistant responses collapsed into ellipses and student prompts truncated to 250 characters at both ends). Conversely, for intrinsic mental model classification, extraneous task excerpts and dialogue histories are omitted, passing only the immediate learner prompt to prevent external contextual bias from inducing hallucinations.
3. Zero-Shot Hierarchical LLM Inference: Multi-Turn Annotation Bottlenecks -> Dual-Model Sequential Routing -> Cross-Architecture Bias Profiling
The inference pipeline evaluates two distinct LLM architectures under a zero-shot regime at temperature 0.0: the proprietary frontier model GPT-5.4 accessed via the OpenAI API, and the open-weights MoE model GPT-OSS:120B deployed locally on a single NVIDIA H200 NVL GPU (141 GB VRAM, MXFP4 quantization, 32,768 context length). Instead of predicting flat classification labels in a single forward pass, each model sequentially navigates the decision tree, producing structured intermediate rationales alongside binary/categorical decisions for assignment mapping (a1โa7), assignment relation (related, unrelated, setting-up), and the 2โ4 sequential branch decisions required to reach a terminal main label.
4. Multi-Pairing Agreement Discrepancy Auditing: Ground-Truth Fallacy -> Matrix Cross-Validation -> Uncovering Latent Semantic Alignment Gaps
To avoid evaluating model outputs against flawed unilateral reference standards, the evaluation establishes a comprehensive tripartite comparison matrix across 408 double-coded interaction instances: measuring Human-Human inter-rater agreement, LLM-LLM inter-model consistency, and Human-LLM alignment against consolidated human consensus labels. While humans achieve substantial agreement on the final main label (\(\kappa = 0.73\), accuracy 0.80) and high consistency on solution seeking (\(\kappa = 0.79\)), the two LLMs also achieve high mutual agreement (\(\kappa = 0.67\), accuracy 0.78) with notable consensus on mental model tracking (\(\kappa = 0.74\)). However, cross-pairing agreement between humans and LLMs plummets (\(\kappa = 0.52\) for GPT-5.4, \(\kappa = 0.41\) for GPT-OSS:120B), empirically confirming that high inter-model reliability does not imply fidelity to human expert pedagogical interpretations.
Disagreement Auditing and Human-LLM Collaboration Protocol¶
To understand the substantive drivers behind this agreement gap, the authors conducted an iterative secondary qualitative coding over all instances where the consolidated human label and GPT-5.4 predictions diverged, classifying discrepancies into three overarching dimensions comprising 11 distinct root causes:
- Model Limitations: Including ignored context (overlooking task excerpts in solution-type selection), reasoning failures (e.g., misclassifying explicit "just read this" instructions as solution-seeking), missed implicit intent (over-relying on surface-level syntax and missing hidden requests for assignment completion), hallucinations (inventing non-existent parameters like hidden_layer_sizes=(10, 2) from earlier turns), and task relation misses (misattributing out-of-scope errors to assignment tasks).
- Task & Annotation Design Issues: Including label misinterpretation (discrepancies in operationalizing whether text reformatting or stylistic rewriting constitutes solution seeking), missing context (unavailability of raw .csv/.txt data files during retrieval), and prompting misreads (treating meta-prompt instructions as student inputs).
- Data Ambiguity & Human Factors: Including interpretive label ambiguity (instances where both human and LLM choices are defensible under valid competing interpretations, accounting for 23.7% of disagreement codes), unclear student intent (students pasting raw code snippets with ambiguous commentary like "That part should be pretty easy tho", accounting for 17.8%), and genuine human annotation drift or errors (accounting for 13.2%).
Critically, non-model factors account for more than 54% of all disagreement labels. This empirical reality invalidates the premise of treating automated annotation as an autonomous, single-pass replacement for human supervision. Instead, the authors advocate for a human-in-the-loop weak-supervision protocol: decision nodes exhibiting high interpretive ambiguity or low class support (such as mental model verification) should incorporate explicit "abstain / ambiguous" flags, automatically routing unconfident or split cases to human educators while treating both LLM and human outputs as complementary, probabilistic supervisory signals.
Key Experimental Results¶
Main Results¶
Quantitative evaluations were conducted on the 408 double-coded learner-chatbot exchanges from the StudyChat dataset (Spring 2025 cohort). The table below details agreement scores (Cohen's \(\kappa\) and Accuracy) across human annotators, between the two LLMs, and between each LLM and the consolidated human consensus labels across all decision tree nodes.
| Variable / Decision Node | Sample Size \(N\) | Human-Human \(\kappa\) (Acc.) | LLM-LLM \(\kappa\) (Acc.) | Human-GPT-5.4 \(\kappa\) (Acc.) | Human-GPT-120B \(\kappa\) (Acc.) |
|---|---|---|---|---|---|
| Assignment Mapping (a1โa7) | 408 | 0.91 (0.93) | 0.99 (1.00) | 0.96 (0.97) | 0.97 (0.98) |
| Assignment Relation (related/unrelated/setup) | 408 | 0.70 (0.98) | 0.71 (0.97) | 0.47 (0.96) | 0.48 (0.95) |
| Solution Seeking (yes / no) | 408 | 0.79 (0.91) | 0.53 (0.78) | 0.63 (0.87) | 0.50 (0.77) |
| Solution Type (create vs. verify/fix/adapt) | 272 / 217 / 248 / 221 | 0.85 (0.93) | 0.52 (0.76) | 0.67 (0.83) | 0.56 (0.79) |
| Knowledge Seeking (yes / no) | 100 / 103 / 95 / 94 | 0.85 (0.96) | 0.89 (0.97) | 0.38 (0.87) | 0.55 (0.87) |
| Cognitive Engagement (yes / no) | 82 / 85 / 78 / 72 | 0.73 (0.87) | 0.78 (0.93) | 0.53 (0.79) | 0.39 (0.75) |
| Mental Model (yes / no) | 82 / 85 / 78 / 72 | 0.52 (0.89) | 0.74 (0.91) | 0.48 (0.85) | 0.37 (0.82) |
| Final Main Label | 408 | 0.73 (0.80) | 0.67 (0.78) | 0.52 (0.64) | 0.41 (0.56) |
Per-Class Performance and Distribution¶
To dissect performance variations across frequent versus long-tail learner intents, the table below outlines the class distribution within the 408 double-coded reference interactions and presents per-class Precision (\(P\)), Recall (\(R\)), and \(F_1\) scores for both automated models evaluated against consolidated human labels.
| Main Label Intent Category | Support (%) | \(P_{5.4}\) | \(R_{5.4}\) | \(F1_{5.4}\) | \(P_{120B}\) | \(R_{120B}\) | \(F1_{120B}\) |
|---|---|---|---|---|---|---|---|
| Solution seeking: create | 147 (36.03%) | 0.87 | 0.71 | 0.78 | 0.66 | 0.80 | 0.72 |
| Solution seeking: verify/fix/adapt | 140 (34.31%) | 0.67 | 0.74 | 0.70 | 0.83 | 0.41 | 0.55 |
| Knowledge seeking + cognitive engagement + no mental model | 45 (11.03%) | 0.52 | 0.62 | 0.57 | 0.37 | 0.60 | 0.46 |
| Knowledge seeking + no cognitive engagement + no mental model | 39 (9.56%) | 0.56 | 0.38 | 0.45 | 0.45 | 0.26 | 0.33 |
| Knowledge seeking + cognitive engagement + mental model | 13 (3.19%) | 0.28 | 0.54 | 0.37 | 0.17 | 0.38 | 0.24 |
| Knowledge seeking + no cognitive engagement + mental model | 1 (0.25%) | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Other / non-knowledge seeking | 23 (5.64%) | 0.18 | 0.22 | 0.20 | 0.28 | 0.43 | 0.34 |
| Macro Average | - | 0.44 | 0.46 | 0.44 | 0.39 | 0.41 | 0.38 |
| Weighted Average | 408 (100.0%) | 0.67 | 0.64 | 0.65 | 0.63 | 0.56 | 0.56 |
Key Findings¶
- Complementary Human vs. LLM Strengths: Human annotators achieve their highest consistency on observable, action-oriented categories such as solution seeking (\(\kappa = 0.79\)) and solution type (\(\kappa = 0.85\)), but struggle on subtle internal reflections like mental models (\(\kappa = 0.52\)). In sharp contrast, LLMs display strong mutual consistency on mental model presence (\(\kappa = 0.74\)) while diverging markedly on solution seeking (\(\kappa = 0.53\)), demonstrating that human annotators and language models possess fundamentally distinct semantic and pragmatic biases.
- Catastrophic Drop on Rare Metacognitive Intents: The dataset exhibits heavy class imbalance: solution-seeking behaviors dominate 70.34% of the corpus, whereas explicit mental model expressions account for only 3.44% (14 instances). Under zero-shot prompting, GPT-5.4 achieves an \(F_1\) of only 0.37 on "Knowledge seeking + cognitive engagement + mental model", while GPT-OSS:120B drops to 0.24; both models score 0.00 on the isolated single instance of mental model without cognitive engagement.
- Validation Across Single-Coded Subsets: On the remaining 1,985 single-coded interaction instances annotated without joint deliberation, GPT-OSS:120B maintains virtually identical agreement with human labels (\(\kappa = 0.43\)) compared to the double-coded subset (\(\kappa = 0.41\)). This verifies that the observed human-machine divergence is an inherent characteristic of the task and models, rather than an artifact of consensual human negotiation.
Highlights & Insights¶
- Deconstructing the Ground-Truth Illusion: By rigorously categorizing human-LLM disagreements, this study empirically proves that 54.7% of classification divergences are driven by natural linguistic ambiguity (23.7%), underspecified student intent (17.8%), and human coding noise (13.2%). Evaluating automated educational NLP solely through the lens of model accuracy against human reference sets is fundamentally misleading.
- Operationalizing Cognitive Offloading via Priority Trees: Translating abstract educational psychology theories (L2L / SRL) into a prioritized decision tree provides a robust, reusable protocol for distinguishing passive code offloading from active conceptual scaffolding in educational human-computer interaction.
- Weak Supervision for Educational Discourse: Framing LLM and human annotations as complementary, noisy weak-supervision sources unlocks a viable blueprint for scalable learning analytics, directing expensive pedagogical expert attention specifically to low-support, high-disagreement decision nodes.
Limitations & Future Work¶
- Single-Domain Academic Context: The corpus is drawn exclusively from an undergraduate artificial intelligence programming course conducted in Spring 2025. The generalizability of this decision hierarchy to humanities writing, open-ended conceptual tutoring, or K-12 educational environments requires further empirical validation.
- Restricted Retrieval Modalities: To manage context limits, the pipeline excluded non-code data files (.csv and .txt), which directly caused retrieval failures and misclassifications whenever learners inquired about dataset properties.
- Zero-Shot Boundary: The study deliberately focused on zero-shot inference to probe intrinsic model behavior; evaluating dynamic few-shot in-context exemplars, active chain-of-thought verification, or interactive agentic clarification remains an open avenue for future research.
Related Work & Insights¶
- vs. Classical Dialogue Act Standards (SwDA / ISO 24617-2): Classical taxonomies emphasize communicative mechanics (interrogative vs. assertive) but fail to capture pedagogical intent, treating code-seeking and concept-learning questions identically; this work explicitly separates task completion from knowledge construction.
- vs. Coarse SRL Mapping (Lyu & Ding 2025; Lai 2024): Existing self-regulated learning studies classify student prompts into broad lifecycle phases (forethought, performance, reflection) via keyword dictionaries, often lumping all implementation queries into "performance control"; this paper operates at utterance-level cognitive granularity.
- vs. Standard LLM Annotation Benchmarks (Ahmed et al. 2025; Wang et al. 2024): Where prior studies benchmark LLMs against presumed ground-truth human annotations, this work provides a two-way qualitative auditing framework that treats both parties as fallible observers of complex educational discourse.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering empirical audit of human-LLM disagreement in educational dialogue, introducing a theoretically sound priority tree for intent classification.
- Experimental Thoroughness: โญโญโญโญโ Rigorous analysis of 2,393 interactions, 408 double-coded instances, and granular multi-pairing evaluation across two distinct LLM families.
- Writing Quality: โญโญโญโญโญ Exemplary clarity in conceptual definitions, robust qualitative taxonomy, and transparent reporting of agreement metrics.
- Value: โญโญโญโญโญ Highly impactful blueprint for learning analytics, educational chatbot auditing, and human-in-the-loop weak-supervision pipelines.