Skip to content

Evaluating LLM-generated Explanatory Dialogue Turns through Dialogue Completion

Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.72/
Code: https://github.com/doyouwantsometea/dialog_completion
Area: LLM Evaluation
Keywords: dialogue completion, explanatory dialogue, evaluation metric bias, FED metric suite, iterative self-refinement

TL;DR

This paper uses dialogue completion as a dual lens across human-spoken, human-written, and synthetic modalities to investigate explanatory dialogue quality and metric behavior, uncovering a systematic written-text bias in the widely used reference-free FED metric that penalizes spoken interactivity, while showing that metric-driven zero-shot self-refinement reliably closes the model interactivity gap.

Background & Motivation

Interactive explanatory dialogues are vital for applications ranging from conversational tutoring systems to AI model rationalization. However, real-world explanations are fundamentally collaborative and dynamic rather than unidirectional lectures: an effective explainer must continuously adapt their phrasing, pacing, and pedagogical moves based on the interlocutor's evolving comprehension and communicative feedback. While modern large language models can generate fluent, coherent, and informative explanations, their conversational turns often lack pragmatic engagement, responsiveness, and genuine interactive flow, leaving a noticeable gap between synthetic explanations and authentic human interactions.

The challenge of understanding this gap is further compounded by the limitations of automated dialogue evaluation suites. Traditional n-gram matching metrics (such as BLEU or token-level F1) notoriously fail to capture interactive dynamics, driving the research community toward reference-free evaluation metrics powered by pretrained language models, most notably FED and USR. Yet, whether these metrics evaluate spoken conversations fairly remains under-explored: because models like DialoGPT were pretrained predominantly on written web dialogues, their likelihood-based scoring might conflate formal written polish with conversational quality, systematically penalizing the fragmented, colloquial, and highly dynamic nature of spoken speech.

To disentangle model generation limitations from automated metric biases, this paper formulates the dialogue completion task as an analytical probe: removing a central explainer turn from an ongoing conversation and instructing LLMs to fill it in within its immediate context. Core idea: use dialogue completion across spoken, written, and synthetic modalities as a dual probe to expose and mathematically disentangle the written-language bias in pretrained dialogue metrics like FED, while utilizing automated metric weaknesses as targeted feedback for zero-shot iterative self-refinement to enhance conversational interactivity.

Method

Overall Architecture

The workflow comprises four tightly coupled stages: dataset processing and dialogue slicing, controlled dialogue completion, fine-grained multidimensional evaluation, and metric-driven iterative self-refinement. The pipeline extracts target explainer turns \(T_i\) of length at least \(l=30\) tokens from three corpora spanning different modalities (spoken ReWIRED, synthetic WikiDialog, and written ELI5-dialogues), paired with a symmetric surrounding context window of \(w=2\) turns on each side. Three representative open and proprietary LLMs (Mistral-0.3 7B, Llama-3.1 8B, and Claude-3 Haiku) are prompted under varied conditions to fill in the missing turn. The resulting completions and original human baselines are scored across 18 pragmatic aspects via FED and 6 numerical linguistic features via IXQUISITE. Finally, normalized score differentials guide an automated self-refinement prompt to correct specific model deficiencies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Explanatory Dialogue Corpora<br/>Spoken ReWIRED / Synthetic WikiDialog / Written ELI5"] --> B["Dialogue Completion Task Framing<br/>Filter length lโ‰ฅ30 explainer turns, window w=2"]
    B --> C["Prompt-Controlled Completion<br/>Vanilla / Topic & Speakers / Open-end"]
    C --> D["Multidimensional Dual-Suite Evaluation<br/>FED 18 pragmatic aspects + IXQUISITE 6 linguistic features"]
    D --> E["Metric Bias Diagnostics & Surface Regression<br/>Quantify written-text bias & penalization of spoken speech"]
    D --> F["Metric-Driven Iterative Self-Refinement<br/>Normalized deficit ฮ”f, targeted natural language feedback"]
    F --> G["Refined Interactive Dialogue Turn"]

Key Designs

1. Dialogue Completion Task Framing: Context-infilled turns as a behavioral and metric probe

Full dialogue generation and open-ended conversation synthesis frequently suffer from topic drift, making rigorous, controlled comparisons between models and authentic human interactions difficult. This work formulates the task as bidirectional dialogue inpainting, governed by minimum target length \(l=30\) tokens and context window \(w=2\). Filtering by \(l\) eliminates conversational filler (e.g., "Could you say that again?", "That makes sense") so that every target turn \(T_i\) carries substantive explanatory content or comprehension checks. A window of \(w=2\) captures the most recent complete exchange between the explainer and explainee, providing immediate conversational grounding without triggering input length penalties or distant distractors.

2. Prompt-Controlled Completion: Disentangling context requirements and metadata sensitivity

To evaluate what types of contextual information actually govern model choices, the authors systematically test prompt variants: the baseline context setup (Vanilla), explicit specification of the topic and speaker expertise level (Topic & Speakers), and a forward-truncated variant (Open-end) where subsequent turns are removed. Empirical results show that LLMs rely heavily on the immediate surrounding conversational context: omitting future turns (Open-end) sharply degrades completion coherence and increases semantic misalignment with the interlocutor. In contrast, appending explicit meta-descriptions about the domain topic or the interlocutor's background (e.g., child, undergraduate, expert colleague) produces only marginal variations in metric scores.

3. Multidimensional Dual-Suite Evaluation: Joint pragmatic and linguistic feature profiling

To overcome the blind spots of single-score metrics, the evaluation couples DialoGPT-based reference-free FED scoring with count-based linguistic analysis from IXQUISITE. FED evaluates 8 turn-level aspects (e.g., Relevant, Fluent, Specific) and 10 dialogue-level aspects (e.g., Engaging, Likeable, Inquisitive, Coherent) by measuring the likelihood that DialoGPT responds to the target turn with predefined positive or negative reaction utterances. In parallel, IXQUISITE extracts reference-free surface and structural statistics, including lexical complexity (difficult word frequency), syntactic coherence (conjunction density), and readability grade level (Flesch-Kincaid). This pairing maps high-level pragmatic quality onto concrete syntactic and lexical choices.

4. Metric Bias Diagnostics & Surface Regression: Formalizing and disentangling FED's written bias

Baseline scoring across the human reference turns reveals an unexpected paradox: the written ELI5 corpus consistently outperforms the spoken ReWIRED corpus across nearly every FED aspect, including interactive qualities such as "engaging" and "likeable". Because spoken conversation is inherently more interactive, this exposes a structural artifact in DialoGPT's likelihood scoring toward formal written grammar. To isolate this bias, the authors fit an ordinary least squares regression predicting raw FED scores from four surface features (turn length, mean word length, punctuation density, and type-token ratio): $\(\text{Score}_{\text{corrected}} = \text{Score}_{\text{raw}} - \hat{f}(\text{len}, \text{word\_len}, \text{punct}, \text{TTR})\)$ This regression confirms that engagement scores are inflated by longer turns and repeated tokens, while fluency scores reward shorter, denser texts. Independent validation via LLM-as-a-Judge (Gemini 2.5 Pro and Claude Haiku 4.5 blind ranking over 300 sampled turns) reveals a decisive preference for spoken human turns (65.0%โ€“85.0% win rate over model outputs), demonstrating that raw FED scores penalize genuine spoken dynamics.

5. Metric-Driven Iterative Self-Refinement: Targeted feedback from normalized feature deficits

To test whether models can actively narrow the interactivity gap, the framework converts automated evaluation outputs into recursive feedback. For each completion, feature differences relative to human references are normalized by the pooled mean and standard deviation: $\(\Delta f = \frac{f_m - \mu_F}{\sigma_F} - \frac{f_o - \mu_F}{\sigma_F}\)$ where \(f_m\) and \(f_o\) are the model and original reference scores for feature \(F\), and \(\mu_F, \sigma_F\) denote pooled statistics. The system selects the worst \(n=3\) features with the lowest \(\Delta f\) (e.g., lacking specificity, flexibility, or inquisitiveness) and maps them to descriptive natural language instructions. Prompting the model to revise its draft according to these targeted critiques produces substantial improvements in conversational engagement while naturally paring back sterile, textbook-like fluency.

Key Experimental Results

Main Results

The main experiments evaluate task accomplishment rates (valid, non-truncated JSON adherence) and normalized performance profiles across the three model families (Mistral-0.3 7B, Llama-3.1 8B, and Claude-3 Haiku) and three corpora. Table 1 reports accomplishment rates, while Table 4 reports normalized aspect percentage scores under the Vanilla prompt configuration.

Dataset / Modality Data Points Built Mistral-0.3 7B Accomplishment Llama-3.1 8B Accomplishment Claude-3 Haiku Accomplishment
ReWIRED (Human-spoken) 743 53.6% (398) 70.1% (521) 70.5% (524)
WikiDialog (Synthetic) 2,613 74.3% (1,941) 82.2% (2,149) 78.6% (2,053)
ELI5-dialogues (Human-written) 2,407 62.4% (1,503) 69.6% (1,675) 38.1% (916)

Normalized evaluation profiles across representative aspects under the Vanilla prompt illustrate the systematic trade-off between surface fluency and conversational engagement:

Evaluated Aspect Mistral-0.3 7B Llama-3.1 8B Claude-3 Haiku
Fluent 47.9 60.4 37.7
Understandable 46.3 59.1 35.9
Semantically appropriate 42.2 57.4 30.0
Engaging -38.5 -53.7 -15.5
Likeable -38.1 -53.9 -16.3
Inquisitive -45.4 -57.9 -31.4
Specific -44.7 -56.6 -28.9
Lexical complexity 28.9 2.2 19.1

Ablation Study

To verify that refinement gains reflect genuine behavioral adjustments rather than overfitting to FED feedback, the authors conducted an ablation on Claude-3 ร— ELI5 restricting feedback exclusively to the 6 non-FED IXQUISITE metrics. Additionally, surface-feature regressions were applied across model-dataset cells to isolate surface artifacts.

Feedback Ablation (Claude-3 ร— ELI5, N=1,225) Engagement (7 metrics) Mean Cohen's d (Eng.) Fluency (3 metrics) Mean Cohen's d (Flu.) Other (8 metrics)
FED + IXQUISITE Combined Feedback 6 / 7 sig. improved +0.41 3 / 3 sig. decreased -0.41 0 / 8 sig. changed (ยฑ0.00)
IXQUISITE-only Feedback (No FED) 7 / 7 sig. improved +0.22 3 / 3 sig. decreased -0.24 8 / 8 sig. changed (+0.77)

The table below summarizes effect sizes (Cohen's d) for iterative refinement before and after removing the four surface features via linear regression for Llama-3.1:

Dataset N Raw Engagement d Corrected Engagement d Raw Fluency d Corrected Fluency d
ReWIRED (Spoken) 482 +0.51 +0.31 -0.49 -0.27
WikiDialog (Synthetic) 1,861 +0.64 +0.39 -0.80 -0.55
ELI5 (Written) 1,583 +0.42 -0.14 +0.18 -0.45

Key Findings

  • The Fluency-Interactivity Dichotomy: Across all models and datasets, LLMs consistently excel at surface fluency, grammatical correctness, and semantic plausibility, but score significantly below human baselines on pragmatic engagement, inquisitive checks, and topic-specific tailoring (\(p < 0.001\)).
  • Severe Written-Text Bias in FED: Pretrained on DialoGPT, the FED test suite systematically scores human-written dialogues (ELI5) higher than spoken dialogues (ReWIRED) across interactive dimensions; surface regression proves this bias stems directly from turn length and token repetition.
  • LLM Judges Validate Human Spoken Superiority: Independent blind rankings by Gemini 2.5 Pro and Claude Haiku 4.5 overturn FED's preference, awarding 65.0% and 85.0% win rates to authentic human spoken turns on ReWIRED, confirming that automated metrics penalize colloquial interactivity.
  • Metric-Driven Refinement Trades Fluency for Interactivity: Incorporating targeted natural language feedback based on normalized metric deficits reliably improves engagement across all nine dataset \(\times\) model cells. Notably, this gain coincides with a controlled decline in sterile fluency, demonstrating a shift toward natural, conversational pragmatics.

Highlights & Insights

  • Framing dialogue completion as an inpainting task offers a highly controlled, bidirectional lens that cleanly decouples conversational interaction from open-ended thematic drift.
  • Uncovering and mathematically isolating the written-text bias in DialoGPT-based metrics provides a crucial cautionary lesson for automated dialogue evaluation across spoken and multimodal domains.
  • The zero-shot self-refinement framework uses normalized feature deficits to construct actionable natural language critiques, yielding targeted improvements without requiring model fine-tuning.

Limitations & Future Work

  • Sample size constraints: The human-spoken ReWIRED corpus is relatively small, leading to batches under 400 instances after strict length and formatting filtering, which introduces potential sampling variance.
  • Reliance on automated judge proxies: While the paper triangulates findings using FED, IXQUISITE, and two independent frontier LLM judges, large-scale fine-grained human perceptual evaluations on the completed turns were not conducted.
  • Truncation under strict JSON constraints: Claude-3 Haiku encountered elevated failure rates on the verbose ELI5 dataset due to token limits causing unclosed JSON formatting.
  • vs FED / USR: Pretrained reference-free dialogue evaluators have long been treated as reliable benchmarks; this study demonstrates their systematic vulnerability to surface-level length and written formality biases, providing regression-based debiasing methods.
  • vs IXQUISITE: Whereas the original suite was developed primarily for descriptive didactic tracking, this work repurposes its numerical linguistic indicators into normalized feedback vectors for automated self-correction.
  • vs Self-Refine: Standard self-refinement frameworks rely on heuristic or generic prompt prompts; this work demonstrates the value of grounding feedback in standardized, multidimensional evaluation metrics to target specific pragmatic deficits.

Rating

  • Novelty: โญโญโญโญ [Innovative use of dialogue completion as a dual probe for model behavior and metric bias]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive cross-modal evaluation spanning three LLMs, two metric suites, surface regressions, and independent LLM judges]
  • Writing Quality: โญโญโญโญโญ [Clear motivation, rigorous methodology, disciplined statistical validation, and well-structured exposition]
  • Value: โญโญโญโญ [Essential insights for conversational AI development, multimodal evaluation, and interactive pedagogical agents]