Adaptive Knowledge Routing for Complex Reasoning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ysufsimsek/Not-All-Evidence-Is-Equal-Source-Sensitive-Calibration-in-Large-Language-Models
Area: LLM Reasoning
Keywords: Source-sensitive calibration, Epistemic vigilance, Turkish evidentiality, Hearer-side uptake, Action readiness
TL;DR¶
This work establishes a human-grounded benchmark for source-sensitive calibration in Turkish and English by holding propositional content constant while manipulating source trustworthiness and evidential markers, discovering that LLMs possess only weak, additive source awareness and severely fail to translate source credibility into appropriately cautious, human-like action-readiness judgments.
Background & Motivation¶
As large language models (LLMs) transition from open-domain text generation to high-stakes applications such as automated decision support, agentic workflow orchestration, and evidence summarization, evaluating their reasoning under uncertainty becomes critical. In natural human communication, hearers never evaluate a communicated proposition in an epistemic vacuum: when faced with an emergency bulletin issued by an official institution versus an informal rumor circulated on an unverified social media account, rational actors assign drastically different degrees of epistemic commitment and action readiness to the claim, even when the underlying assertion is identical. This foundational human capacity—grounded in epistemic vigilance—ensures that belief, recommendation, and physical action are strictly conditioned on the trustworthiness of the source.
Despite rapid advancements, existing benchmark suites overwhelmingly assess whether language models can retrieve the correct answer, follow complex chains of thought, or filter out factual inaccuracies, yet fail to examine whether models condition downstream behavioral judgments on the epistemic quality of evidence. In agentic execution or decision-support pipelines, treating a dubious unverified rumor as nearly as action-worthy as an authenticated institutional report is an alarming breakdown of evidence-based calibration rather than a mere semantic misunderstanding. Evaluating source-grounded reasoning must therefore be treated as an action-grounding and calibration challenge: models must be rigorously tested on how cautiously or decisively they act upon information given its provenance.
To address this evaluation gap, this work introduces a human-grounded experimental framework operationalizing hearer-side uptake across two typologically distinct languages: Turkish, which morphologically encodes evidential contrasts in the past domain via -DI (direct evidence) and -mI¸s (indirect/inferential/reportative evidence), and English, which conveys evidential commitment lexically through markers such as apparently. Core idea: formalize evidence-conditioned reasoning as an action-readiness calibration problem using a controlled 2×2 factorial design crossing source reliability with evidential forms, revealing that while human hearers strongly modulate action readiness based on source trust, LLMs exhibit weak cue sensitivity, lack interactive pragmatic integration, and suffer from severe response-scale compression.
Method¶
Overall Architecture¶
The evaluation paradigm formalizes hearer-side uptake through four-part trial items: a constant propositional assertion (e.g., "classes were cancelled"), an attributed source (an authoritative official entity versus an unverified social-media account), an evidential marker (the morphological -DI vs. -mI¸s contrast in Turkish; the unmarked simple past vs. lexical apparently in English), and an action-oriented probe (e.g., "According to this news, how likely are you to go to class?"). Both human participants and language models respond on a 1–7 Likert scale. Items are directionally recoded so that higher scores consistently reflect stronger behavioral uptake aligned with the reported proposition.
The experimental pipeline operates across three interconnected tiers: first, establishing a controlled human baseline via an IRB-approved online study involving 74 native Turkish speakers across 40 factorial trials (2,960 total observations); second, conducting LLM evaluations on Turkish using an initial 10-scenario set followed by an expanded 50-scenario dataset (200 critical items, 3 repeated runs yielding 600 trials per model) evaluated across 8 specialized and multilingual LLMs; third, porting the 50 scenarios into English with review by native American English speakers to test 11 diverse open-weight models under a lexical evidential contrast.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Constant Proposition & Action Question<br/>Hold propositional content fixed; evaluate on 1-7 Likert scale"] --> B["2×2 Factorial Evidence Manipulation<br/>Source Reliability (High vs. Low) × Evidential Form (Direct vs. Indirect)"]
B --> C["Cross-Linguistic Epistemic Probes<br/>Turkish morphological contrast -DI/-mI¸s ∥ English lexical marker apparently"]
C --> D["Human Grounding Baseline<br/>74 native Turkish participants, 2960 trials, response times & ratings"]
C --> E["LLM Hearer-Side Uptake Evaluation<br/>200-item Turkish expanded suite + 11-model English cross-lingual transfer"]
D --> F["Source-Sensitive Calibration Analysis<br/>Compute High-Low effect size, evidential main effects & scale compression"]
E --> F
Key Designs¶
1. 2×2 Factorial Evidence Manipulation: Disentangling Content from Epistemic Standing
To eliminate confounding between propositional plausibility and source credibility, every critical scenario holds the core proposition strictly invariant across conditions. Source Reliability is manipulated across two binary tiers: High Trust, represented by institutional or expert sources possessing a direct epistemic basis and formal authority over the domain, versus Low Trust, represented by informal or unverified entities lacking verifiable access. For language model evaluation, generation parameters are held strictly constant across runs (temperature = 0.1, top_p = 1.0, max_tokens = 50, do_sample = true, repeated 3 times) and prompts explicitly instruct models to output only a single integer between 1 and 7 without explanation, preventing verbose output generation from obscuring numerical decision calibration.
2. Cross-Linguistic Evidential Framing: Morphological Contrasts vs. Lexical Adverbial Marking
The benchmark exploits typological divergence in evidential systems. In Turkish, the past tense obligatorily distinguishes between -DI (the default unmarked past form associated with direct witnessing) and -mI¸s (the evidential past associated with indirect inference or reportative hearsay). In English, where morphological evidentials are absent, the design introduces a functional parallel by comparing regular-tense sentences with those containing the modal adverb apparently, which signals reduced speaker commitment and indirect evidence. This cross-linguistic contrast enables probing whether models and humans treat evidential meaning as an independent decision-relevant factor or whether overt source attribution overrides grammaticalized evidential cues.
3. Polarity Alignment and Scale Compression Auditing: Unmasking Pseudocalibration
To prevent habitual response bias, scenarios incorporate both action-promoting items and action-inhibiting items. In data post-processing, action-inhibiting responses are systematically reverse-coded via: $\(S_{\text{aligned}} = 8 - S_{\text{raw}}\)$ ensuring that higher transformed scores consistently indicate stronger action in accordance with the reported claim. Crucially, the benchmark implements a scale compression audit to distinguish superficial cue recognition from genuine behavioral calibration. By quantifying the High–Low source effect: $\(\Delta_{\text{source}} = \bar{S}_{\text{High}} - \bar{S}_{\text{Low}}\)$ alongside ceiling saturation frequencies (e.g., percentage of trials rated at 7), the evaluation diagnoses whether models compress their decision range and excessively trust low-credibility rumors.
Benchmark & Evaluation Protocol¶
The empirical evaluation was conducted in Google Colab utilizing G4 GPU runtimes across all model executions. The Turkish evaluation evaluated 8 models: Gemma-2-9B-IT-TR, Gemma-3-27B-IT, GPT-OSS-20B-Astro, Kizagan-E4B, Magibu-11B-v0.8, Novus-7B-TR, Qwen3-32B, and WiroAI-TR-8B. The English evaluation evaluated 11 models, including language-adapted models and broad general-purpose variants (such as Gemma-2-9B-IT, Gemma-4-E4B-IT, and Llama-3.1-8B-Instruct). Execution latency ranged from 33 seconds (Llama-3.1-8B-Instruct) to 21 minutes and 35 seconds (Qwen3-32B).
Key Experimental Results¶
Main Results¶
The table below contrasts Turkish human participants with 8 evaluated language models on the expanded 200-item Turkish uptake benchmark (scores directionally aligned to a 1–7 scale):
| Evaluation Subject / Model | High + -DI | High + -mI¸s | Low + -DI | Low + -mI¸s | Source Effect (H - L) | Source p-value | Morphology Effect (-mI¸s - -DI) |
|---|---|---|---|---|---|---|---|
| Turkish Human Baseline | 5.91 | 5.93 | 4.22 | 4.25 | +1.69 | < 1e-14 | +0.02 |
| GPT-OSS-20B-Astro | 6.08 | 6.06 | 5.43 | 5.55 | +0.58 | .0011 | +0.05 |
| Gemma-2-9B-IT-TR | 4.74 | 4.75 | 4.27 | 4.24 | +0.49 | .0045 | -0.01 |
| Qwen3-32B | 5.69 | 5.71 | 5.27 | 5.15 | +0.49 | < .001 | -0.05 |
| Gemma-3-27B-IT | 5.27 | 5.39 | 5.01 | 5.02 | +0.31 | .0228 | +0.06 |
| Kizagan-E4B | 6.03 | 5.73 | 5.73 | 5.63 | +0.20 | .0260 | -0.20 |
| Novus-7B-TR | 6.37 | 6.43 | 6.21 | 6.33 | +0.13 | .5930 | +0.09 |
| WiroAI-TR-8B | 3.91 | 3.95 | 3.79 | 3.81 | +0.13 | .0800 | +0.03 |
| Magibu-11B-v0.8 | 6.00 | 6.28 | 6.12 | 6.15 | +0.01 | .9650 | +0.15 |
| 8-Model Macro Mean | 5.51 | 5.54 | 5.23 | 5.24 | +0.29 | .0056 | +0.02 |
The table below summarizes performance across 11 language models on the 50-scenario English uptake evaluation crossing source reliability and the lexical marker apparently:
| Model | H-Reg. | H-App. | L-Reg. | L-App. | Source Effect (H - L) | Lexical Effect (Reg > App) | Interaction (Diff-in-Diff) |
|---|---|---|---|---|---|---|---|
| Gemma-3-27B-IT | 5.30 | 5.13 | 4.85 | 4.60 | +0.49 | +0.21 | -0.08 |
| Gemma-4-E4B-IT | 5.27 | 5.24 | 4.90 | 4.77 | +0.42 | +0.08 | -0.10 |
| Gemma-2-9B-IT-TR | 4.59 | 4.45 | 4.31 | 4.07 | +0.33 | +0.19 | -0.09 |
| Qwen3-32B | 5.52 | 5.14 | 5.26 | 4.77 | +0.32 | +0.44 | -0.11 |
| Gemma-2-9B-IT | 4.45 | 4.26 | 4.17 | 4.01 | +0.27 | +0.18 | +0.03 |
| Llama-3.1-8B-Instruct | 4.61 | 4.55 | 4.33 | 4.31 | +0.26 | +0.04 | +0.05 |
| WiroAI-TR-8B | 4.29 | 4.03 | 4.03 | 3.82 | +0.23 | +0.23 | +0.05 |
| GPT-OSS-20B-Astro | 6.01 | 5.27 | 5.51 | 5.41 | +0.18 | +0.42 | +0.63 |
| Kizagan-E4B | 6.42 | 6.37 | 6.35 | 6.26 | +0.09 | +0.07 | -0.05 |
| Magibu-11B-v0.8 | 3.50 | 3.15 | 3.52 | 2.95 | +0.09 | +0.46 | -0.21 |
| Novus-7B-TR | 5.03 | 5.17 | 4.91 | 5.10 | +0.09 | -0.17 | +0.05 |
| 11-Model Macro Mean | 5.00 | 4.80 | 4.74 | 4.55 | +0.25 | +0.20 | +0.02 |
Key Findings¶
- Humans exhibit robust epistemic vigilance dominated by source trust: Native Turkish speakers assign +1.69 higher uptake to High-Trust sources than Low-Trust sources (\(p = 9.15 \times 10^{-15}\)). However, evidential morphology exhibits no independent effect once the source is overtly named: the difference between
-DIand-mI¸sis an insignificant +0.02 points (\(p = 0.514\)), with no interaction (\(p = 0.913\)). Hearers prioritize epistemic standing over grammatical morphology. - LLMs show weak, uneven calibration and severe scale saturation: Across 8 models on the 200-item Turkish set, the average High–Low source effect is only +0.29 points (\(p = 0.0056\)), roughly one-sixth of the human effect size. Furthermore, several models suffer from pathological ceiling saturation: transformed ratings of 7 account for 85.3% of Magibu-11B-v0.8 responses, 83.7% of Novus-7B-TR responses, and 73.7% of Kizagan-E4B responses, demonstrating an inability to moderate action readiness for unverified rumors.
- Lexical evidentials reduce uptake additively without pragmatic synergy: In English,
apparentlyinduces a statistically reliable reduction in action readiness (+0.195 to +0.20 points, \(p = 0.0068\)). However, the Source × Evidential interaction remains negligible (+0.016 to +0.02 points, \(p = 0.819\)). Models treat lexical uncertainty markers and source trustworthiness as independent, additive surface cues rather than synthesizing them into a contextualized pragmatic inference.
Highlights & Insights¶
- Reframing source reasoning as action-readiness calibration: Demonstrates that epistemic reasoning should not be evaluated purely as text classification or factuality detection, but as an action-grounding problem measuring the willingness to act on information given its epistemic backing.
- Pragmatic division of labor between grammar and attribution: Clarifies that grammatical evidentials (
-DI/-mI¸s) define the type of epistemic relation to a proposition, whereas discourse attribution specifies source credibility; in the presence of explicit sources, credibility naturally dominates behavioral uptake. - Diagnosing the gap between surface recognition and calibrated execution: Uncovers that LLMs often correctly parse attribution tags in generation yet fail to translate that recognition into calibrated downstream decision thresholds, exposing critical vulnerabilities in autonomous agent planning.
Limitations & Future Work¶
- Overt attribution framing constraint: All critical stimuli explicitly utilized attribution frames (e.g., "According to X"), which may overshadow the independent communicative contribution of evidential morphology; future studies should probe contexts where source identity is unstated or inferred.
- Coarse-grained offline measurement: Uptake was quantified using static 1–7 Likert ratings, capturing final decision outcomes without probing internal cognitive processing, token-level confidence dynamics, or eye-tracking/reaction-time equivalents.
- Absence of English human comparison group: While the English stimuli underwent rigorous native-speaker validation, collecting empirical human uptake data in English remains an important avenue to confirm whether human hearers also treat lexical evidentials additively.
Related Work & Insights¶
- vs. Standard QA Benchmarks (e.g., TruthfulQA, MMLU): Standard benchmarks evaluate whether an LLM outputs the truth given static prompts; this work holds truth value constant and evaluates how models modulate behavioral readiness across varying source trustworthiness.
- vs. Credibility-Aware RAG (e.g., Pan et al., 2024; Hwang et al., 2025): While credibility-aware retrieval focuses on selecting authenticated documents to minimize generation hallucination, this study investigates hearer-side pragmatic uptake to evaluate how LLMs act when exposed to varying evidence quality.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an innovative human-grounded evaluation benchmark linking linguistic evidentiality, source trustworthiness, and action readiness.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Combines an IRB-approved 74-participant human baseline, a 200-item expanded Turkish evaluation, and cross-linguistic English transfer across 11 open-weight LLMs.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous integration of formal semantics, cognitive pragmatics, and empirical statistical methodology with crystal-clear reporting.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable evaluation framework and diagnostic tool for epistemic safety in LLM-based decision support and autonomous agent deployment.