Skip to content

DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/nehasrikn/discotrace
Area: NLP Understanding
Keywords: question answering, discourse analysis, rhetorical structure theory, large language models, answering strategies

TL;DR

DiscoTrace represents long-form answers as structured sequences of question-related discourse acts paired with explicit question interpretations, revealing that human communities exhibit diverse rhetorical answering policies while large language models remain rhetorically homogeneous and systematically over-answer low-probability interpretations.

Background & Motivation

As human-AI interactions transition from simple factoid retrieval to complex, open-ended, subjective, and ambiguous information needs, long-form question answering (LFQA) has become the dominant paradigm for modern language systems. When faced with imperfect questionsโ€”queries that are underspecified, vague, or embedded with false presuppositionsโ€”human respondents naturally deploy diverse communicative strategies: some ask clarifying questions, others challenge false premises, while domain experts systematically contextualize claims with authoritative evidence. However, modern alignment frameworks based on reinforcement learning from human feedback (RLHF) and pairwise preference modeling implicitly collapse these multi-objective rhetorical choices into a single monolithic score, optimizing for a generic prototypical persona and conflating factuality, length, and style with underlying answering strategy.

The fundamental tension lies in the lack of formal, interpretable representations that decouple how an answer is rhetorically structured from what underlying interpretation of the query is being resolved. Prior LFQA benchmarks, such as ELI5, heavily center on layperson conceptual simplification, discarding ambiguous or presupposition-laden queries and reducing discourse structures to coarse sentence-level organizational roles (e.g., summary, example). These schemas fail to capture the pragmatics of question-answeringโ€”specifically, the communicative moves answerers use to engage, negotiate, or redirect user inquiries across heterogeneous knowledge communities.

This paper tackles this gap by unifying speech act theory and rhetorical structure theory (RST) within an explicit question interpretation space. The core idea is to represent an answer as an ordered sequence of question-oriented discourse acts paired with question interpretations, DiscoTrace, revealing high-level communicative policies through boundary-driven rhetorical segmentation and demonstrating that LLMs suffer from strategic homogeneity and over-answering.

Method

Overall Architecture

DiscoTrace takes as input an information-seeking question \(Q\) and an answer \(A\) (along with optional community context \(C\)), and maps them into an interpretable rhetorical trajectory \(S_{DT} = \{(a_1, i_1), (a_2, i_2), \dots, (a_n, i_n)\}\). The pipeline operates across three modular stages: rhetorical structure theory parsing and boundary-based coarsening into coherent action segments, question-related discourse act classification, and candidate interpretation space construction followed by eligible act-interpretation pairing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Question Q and Answer A"] --> B["Stage 1: RST-Based Parsing & Segmentation<br/>Merge EDUs into action segments via boundary relations"]
    B --> C["Stage 2: Discourse Act Labeling<br/>Tag segments across 5 families and 21 functional acts"]
    C --> D["Stage 3: Interpretation Generation & Pairing<br/>Deduplicate interpretation space SI and pair eligible acts"]
    D --> E["Output: Strategy Sequence SDT<br/>Ordered tuples of discourse acts and interpretations"]

Key Designs

1. Boundary-Driven RST Action Segmentation: Bridging Atomic Clauses and Discourse Actions Standard rhetorical structure theory (RST) represents text as a binary constituent tree of elementary discourse units (EDUs). However, EDUs typically correspond to sub-clause fragments that are far too fine-grained to carry high-level communicative intent relative to an external question. DiscoTrace addresses this by identifying a curated subset of RST relations called "boundary relations" that reliably mark rhetorical shifts between text segments. These boundary relation-nuclearity pairs include Contrast (NN), Comparison (NN), Topic-Change (NN, NS, SN), Evaluation (NS, SN, NN), Summary (NN, NS, SN), and Background (NS, SN). Traversing the RST tree produced by an end-to-end neural RST parser, the algorithm recursively splits nodes at boundary relations while enforcing an EDU threshold (\(k \ge 3\)) on Background branches to prevent over-fragmentation from minor contextual qualifiers. Subtrees without boundary relations are collapsed into unified spans, yielding coherent "action segments" that serve as the semantic units for subsequent strategy tagging.

2. Fine-Grained Question-Related Discourse Act Ontology: Formalizing Answering Moves Moving beyond general dialogue acts or document-internal text organization schemas, the authors iteratively develop a domain-agnostic ontology of 21 discourse acts grouped into five pragmatic families: - Answer Question (AQ): Provide Background, Provide Example, Assert Answer, Provide Reasoning or Justification, Provide Recommendation, Answer Question outside of Interpretation Space, and Cite External Source. - Seek Information from Asker (SI): Clarification, Probe Context, and Counter-Question. - Comment on the Question (CQ): Reject Presupposition, Correct Fact/Term, Comment on Question, Redirect Formulation, and Disambiguate or Surface Interpretations. - Redirect the Question (RQ): Direct to Resource and Recommend Expert. - No-Op (NO): Non-Answer, Presentational, Aside/Other Remark, and Offer.

Using contextual prompts that expose preceding discourse states, a sequence tagger assigns each segment (and individual EDUs where intra-segment shifts occur) an action from this ontology. Human validation demonstrates substantial agreement (\(\kappa = 0.75\) at the family level, \(\kappa = 0.62\) at the individual act level), with over \(71\%\) of remaining disagreements confined strictly within the same functional family.

3. Eligibility-Constrained Interpretation Pairing: Grounding Communicative Acts in Query Intent Answering complex questions requires pragmatically adopting or distinguishing among multiple valid readings of the inquiry. DiscoTrace builds an explicit interpretation space \(S_I\) for question \(Q\) by prompting two frontier LLMs (GPT-4.1 and Claude Sonnet 4.6) conditioned on community context, followed by dense embedding-based deduplication. Crucially, the authors recognize that not all discourse acts correspond to a specific reading of the question: meta-commentary, conversational fillers, or presupposition rejections contest or bypass the query framing rather than resolving an interpretation. DiscoTrace formalizes an Act Eligibility criterion, gating interpretation assignment exclusively to 10 eligible communicative acts (marked with stars in the ontology, such as Assert Answer, Provide Background, and Clarification). Eligible acts are then mapped to specific candidate interpretations \(i \in S_I\) or marked as NONE, completing the full \((a_j, i_j)\) trajectory.

A Worked Example

Consider a question from the linguistics community (r/asklinguistics): "Can language be described by math?" 1. Interpretation Space Generation: Produces three candidate readings: \(i_1\): Can math usefully describe a range of linguistic phenomena? \(i_2\): Can math describe the totality of linguistic phenomena? \(i_3\): Can math accurately represent the causal processes of human language production and comprehension? 2. Human Answer Parsing into DiscoTrace: - Segment 1: "What exactly do you mean?" \(\to\) Act: Clarification, Interpretation: NONE. - Segment 2: "Yes, mathematical logic and discrete mathematics has often been used in describing and explaining language at various different levels." \(\to\) Act: Assert Answer, Interpretation: \(i_1\). - Segment 3: "Two major areas, syntax and semantics, have treated natural languages as being formal languages..." \(\to\) Act: Provide Background, Interpretation: \(i_1\). - Segment 4: "Is the question 'can math usefully describe...' or is it 'can math describe the totality...'?" \(\to\) Act: Surface Interpretations / Disambiguate, Interpretation: NONE. - Segment 5: "The answer could be 'yes' for all of those, but it's most transparently a yes for the combo of 'a range' and 'similar output'." \(\to\) Act: Assert Answer, Interpretation: \(i_1, i_3\).

This trace clearly displays how an expert answerer navigates ambiguity: first probing the question, committing to a substantiated reading, explicitly enumerating competing interpretations, and finally synthesizing a nuanced resolution.

Key Experimental Results

Main Results

The authors evaluate answers across 9 diverse Reddit communities comprising 2,700 questions and 18,968 responses. Bigram language models \(p(a_{\text{next}} | a_{\text{prev}})\) are trained over discourse act sequences to capture community answering "policies." Cross-perplexity measures the divergence of discourse patterns across communities and between humans and LLMs.

Model / Evaluated Setting Evaluation Condition Bigram Perplexity Description & Notes
Human within-community Trained & evaluated on same community 5.5 - 7.9 (median 7.9) Baseline self-perplexity across 9 subreddits
Human cross-community 72 cross-community pairs 6.1 - 14.9 Expert forums predict each other well; generic forums are easily modeled
Claude Haiku 4.5 Default zero-shot QA 19.9 Significant rhetorical divergence from all human communities
Qwen3-32B Default zero-shot QA 21.4 Uniform, rigid discourse act distribution across topics
Claude Sonnet 4.5 Default zero-shot QA 28.2 Highest divergence from human pragmatic policies
Claude Sonnet 4.5 Mimic community guidelines 16.4 Detailed community rules reduce divergence but fail to close the gap

In community adaptation experiments, prompting Claude Sonnet 4.5 with comprehensive rules from r/AskHistorians lowers cross-perplexity from 26.2 to 16.3, yet human self-perplexity within that community remains at 7.9. On r/ScienceBasedParenting, mimicking reduces perplexity from 21.8 to 14.0, but human self-perplexity is similarly 7.9. Human answers remain unpredictable to LLM-trained bigrams (perplexity 14.0 - 16.3), while LLM answers are readily predicted by human models (perplexity 6.4 - 10.4), demonstrating that LLMs occupy a restricted, narrow subspace of human rhetorical behavior.

Interpretation Selection & Depth of Dedication

To analyze content selection, the paper introduces the Dedication metric: the proportion of eligible segments in an answer allocated to an addressed interpretation. Higher dedication reflects deep, focused answering rather than superficial multi-interpretation skimming.

Answerer Group Prompt Mode Eligible Segments / Answer Matched Segments / Answer Dedication (Mean) Dedication (Median)
Human Answerers โ€” 3.3 2.5 0.62 0.53
Claude Haiku 4.5 Default 5.7 4.6 0.46 0.40
Qwen3-32B Default 12.1 10.4 0.40 0.31
Claude Sonnet 4.5 Default 5.8 4.8 0.48 0.40
Claude Sonnet 4.5 Mimic guidelines 9.8 8.0 0.41 0.33

Across the 9 communities, the average interpretation space size \(|S_I|\) ranges from \(3.1 \pm 1.4\) to \(3.9 \pm 1.9\), while humans address only 34%โ€“45% of available interpretations per response. In contrast, LLMs exhibit severe "over-answering": when fewer than 10% of human answerers choose to address a marginal interpretation, Claude Haiku addresses it over 20% of the time, and Qwen3-32B addresses it 40% of the time.

Key Findings

  • LLMs systematically trade depth for breadth: While humans commit deeply to 1โ€“2 plausible interpretations (dedication median 0.53), LLMs deluge the user with excessive discourse segments (up to 12.1 segments in Qwen) spread thinly across peripheral interpretations (dedication median 0.31โ€“0.40).
  • Rhetorical norms cannot be recovered by surface prompting alone: Providing multi-paragraph community guidelines (e.g., forbidding speculation, requiring evidence) fails to bridge the structural perplexity gap, indicating that pragmatic reasoning about information needs operates deeper than textual rule-following.
  • Asymmetric predictability across expert and lay communities: Expert forums (AskHistorians, AskEconomics) exhibit constrained, evidence-backed discourse policies that generalize across disciplinary boundaries, whereas generic QA communities deploy loose, conversational policies that fail to account for rigorous argumentative structures.

Highlights & Insights

  • Speech acts grounded in RST and interpretation spaces: By coupling rhetorical constituent trees with external query interpretation spaces, DiscoTrace transforms abstract pragmatic strategy from an unobservable latent quality into an explicit, manipulable symbolic plan.
  • Exposing the structural cost of default RLHF alignment: The empirical findings confirm that standard alignment encourages models to hedge, over-answer, and produce verbose, low-diversity responses across fundamentally distinct question contexts.
  • Enabling scaffolded planning for persona-sensitive QA: DiscoTrace provides an actionable framework for controllable generation. A system serving clinicians can be constrained to plan [Disambiguate -> Assert Answer -> Cite External Source], while a patient-facing system can instantiate [Assert Answer -> Provide Reasoning -> Recommend Expert].

Limitations & Future Work

  • Domain confinement to English Reddit: All analyzed discussions originate from Reddit, which carries platform-specific norms; findings may not fully extrapolate to professional forums like StackOverflow or diverse cultural and multilingual settings.
  • Reliance on proprietary LLMs for labeling: The annotation and interpretation generation stages utilize commercial LLMs (GPT-4.1, Claude 3.5/Sonnet). While backed by substantial human agreement, subtle model biases could influence fine-grained classification.
  • Descriptive rather than generative execution: Current experiments focus on diagnostic analysis and post-hoc evaluation; future work should integrate DiscoTrace directly into decoding-time planners and RL reward modeling.
  • vs. LFQA sentence roles (Xu et al., 2022; Fan et al., 2019): Previous work evaluated ELI5 using six static organizational roles and filtered out premise-violating queries. DiscoTrace expands to 21 active speech acts across 9 communities and explicitly embraces conversational maneuvers like premise rejection and context probing.
  • vs. Questions Under Discussion (Roberts, 2012; Namuduri et al., 2025): QUD reconstructs implicit questions internal to monologue discourse. DiscoTrace specifically models how an agent selects and navigates the explicit interpretations of an external user's question.

Rating

  • Novelty: โญโญโญโญโญ Pioneering discourse-act and interpretation pairing framework for LFQA strategy representation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous cross-perplexity modeling and human agreement evaluation across 2,700 questions and nearly 19k answers.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear framing, rich visual walkthroughs, and insightful pragmatic analysis.
  • Value: โญโญโญโญโญ Offers a transformative diagnostic tool for evaluating LLM rhetorical capabilities and designing personalized AI assistants.