Adaptive Knowledge Routing for Complex Reasoning in LLMs (KNOWSIM)¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yjo2lee/knowsim
Area: LLM Reasoning
Keywords: Knowledge State Modeling, Information Calibration, User Simulation, Cognitive Load Theory, Complex Reasoning
TL;DR¶
KNOWSIM introduces a user simulator grounded in cognitive learning theory that tracks explicit knowledge state trajectories over a prerequisite Information Unit (IU) graph under cognitive load constraints, computing mechanistic metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) and uncovering aptitude–treatment interactions across frontier LLMs.
Background & Motivation¶
In multi-turn collaborative problem solving, users increasingly rely on Large Language Models (LLMs) as interactive partners for knowledge-intensive reasoning tasks such as clinical diagnostics and competition mathematics. However, standard post-training and alignment paradigms like RLHF heavily favor information maximization within a single message, incentivizing assistants to produce dense, comprehensive encyclopedic answers. Such approaches provide no training signal regarding how complex conceptual dependencies should be scaffolded and sequenced over multi-turn conversational trajectories according to the user's evolving comprehension and cognitive capacity.
Existing LLM-based user simulators deployed to evaluate and train conversational agents suffer from critical architectural limitations. They operate merely as behavioral proxies via zero-shot prompt role-playing, without explicitly maintaining or updating an internal mental model of the user. Lacking explicit knowledge state representations, these simulators cannot evaluate information calibration—the ability to deliver content pitched at the user's current prerequisite boundary across heterogeneous expertise levels—only measuring coarse aggregate utility. Furthermore, without tracking actual understanding dynamics, evaluation relies on surface-level response fluency and subjective LLM-as-a-judge scoring, which often rewards superficial politeness and verbose explanations that fail to foster genuine conceptual mastery.
This paper bridges this fundamental gap by operationalizing classic educational psychology and cognitive learning theories—specifically Ausubel's subsumption theory of prerequisite anchoring, Sweller's cognitive load theory, and Vygotsky's Zone of Proximal Development (ZPD)—into a computable, multi-turn conversational simulator. Core idea: represent full domain understanding as a directed acyclic graph of Information Units (IUs) with prerequisite constraints, update discrete user knowledge states per turn based on assistant teaching quality, prerequisite ceilings, and non-linear cognitive load damping, and directly derive mechanistic metrics (Knowledge Gain, Delivery Calibration, and Cognitive Overload) from state trajectories.
Method¶
Overall Architecture¶
The KNOWSIM evaluation pipeline consists of three core phases: prerequisite IU graph extraction from reference solutions, multi-turn state-constrained cognitive dialogue simulation, and deterministic calibration metric computation over the resulting state trajectory. The framework first parses the task question and reference answer into an Information Unit DAG \(G_q=(V, E)\), initializing user knowledge states according to target expertise strata (Novice, Intermediate, Advanced). In each conversation turn, the user simulator generates an authentic query strictly constrained by its accessible concept horizon; a dual-signal extraction module parses both the assistant's pedagogical depth and the user's reasoning engagement; and a deterministic state transition engine updates concept mastery subject to prerequisite gating and capacity-bounded non-linear load damping until cognitive termination conditions are triggered.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Reference Solution & Task Input<br/>(q, r)"] --> B["Prerequisite IU Graph Construction<br/>IU Graph Gq=(V, E)"]
B --> C["Knowledge-Stratified Initialization<br/>s0(ℓ): knows_well / partial / struggling / unaware"]
C --> D["Knowledge-Constrained User Message Generation<br/>Conditioned on Local Density & Connectable Unknowns"]
D --> E["Bidirectional Interaction Signal Extraction<br/>Assistant Teaching Quality + User Attempt Type"]
E --> F["Learning-Theoretic State Update & Load Damping<br/>Prerequisite Ceiling + Cognitive Load Lt + Sigmoid ηt"]
F -->|Mastery Reached or Persistent Overload| G["Dynamic Cognitive Termination Check"]
F -->|Active Conversation Continues| D
G --> H["Mechanistic Calibration Metric Computation<br/>Knowledge Gain / Delivery Calibration / Cognitive Overload"]
Key Designs¶
1. Prerequisite IU Graph Construction: Fine-Grained Conceptual Abstraction with Bounded Difficulty Gaps
Prior conversational benchmarks rely on unstructured textual prompts that fail to capture the hierarchical dependencies inherent in complex reasoning. KNOWSIM models the conceptual domain as a directed acyclic graph \(G_q = (V, E)\), where each node \(v \in V\) represents a self-contained Information Unit (IU) explainable and assessable in 2–4 sentences. Nodes are assigned a continuous abstraction level \(l(v) \in [0, 1]\), spanning foundational cross-domain principles (\(l \approx 1.0\)), intermediate domain-specific connecting mechanisms (\(l \approx 0.5\)), down to context-bound calculations and final solutions (\(l \approx 0.0\)). To prevent discontinuous conceptual leaps, every prerequisite edge \((v_i, v_j) \in E\) enforces an abstraction gap bound \(\Delta l = |l(v_i) - l(v_j)| \le 0.35\); larger gaps require intermediate bridging IUs. Concept states are discretized into four ordinal levels: unaware (0), struggling (1), partial_understanding (2), and knows_well (3), initialized via topological sampling to guarantee that advanced concepts are never assumed without their foundational roots.
2. Knowledge-Constrained User Message Generation: Cognition-Grounded Natural Language Utterances
To prevent artificial role-play artifacts such as omniscient prompting, user generation is strictly bounded by the agent's current mental model. At turn \(t\), the simulator receives only the conversational history and a restricted concept partition, highlighting "connectable unknowns"—IUs whose immediate prerequisites have reached partial understanding but which remain unmastered. The user's articulation mode is governed by the local neighborhood density \(d\) of familiar concepts within \(G_q\): high density (\(d \ge 0.6\)) produces explicit, confident inquiries; medium density (\(0.3 \le d < 0.6\)) yields vague inquiries; and low density (\(d < 0.3\)) triggers deferential, passive follow-ups. For concepts in the struggling state, generation prompts explicitly induce realistic misconceptions (e.g., misremembered formulas or sign errors), while unaware concepts are strictly forbidden from spontaneous invocation.
3. Bidirectional Interaction Signal Extraction: Decoupling Teaching Depth from Active Reasoning
Accurate state updating requires isolating genuine knowledge transfer from conversational noise. In each turn pair \((u_t, a_t)\), an extraction LLM call classifies per-IU signals relative to \(s_{t-1}\). For the assistant turn \(a_t\), teaching quality is classified as well_explained (providing derivations, causal mechanisms, or worked examples sufficient for an uninitiated learner), shallow (naming, confirming, or passing mentions without explanation), or not_mentioned, under a conservative rule favoring shallow on borderline cases. For the user turn \(u_t\), the system identifies engagement modes: reasoning (generating new inferences, calculations, or problem framing, even if flawed), articulation (paraphrasing or repeating prior assistant output), or none. This distinction ensures that passive agreement is not conflated with active schema construction.
4. Learning-Theoretic State Update & Load Damping: Non-Linear Absorption Under Cognitive Capacity Limits
Knowledge acquisition is formalized under three cognitive constraints. First, the Prerequisite Ceiling caps the reachable state of any IU by the minimum proficiency of its direct prerequisites, preventing learners from mastering complex synthesis while foundational concepts remain ungrasped. Second, Cognitive Overload computes a state-weighted load \(L_t\) across all concepts active in the turn:
where unfamiliar concepts incur the highest cognitive cost (\(W_S = 1.75\) for unaware vs. \(0.35\) for knows_well), and active reasoning contributes additional load (\(W_R \in [0.35, 1.75]\)). When \(L_t\) exceeds the graph-scaled capacity \(\tau = \max(2, \text{round}(0.55 \cdot |G_q|))\), a continuous sigmoid damping factor attenuates upward state transitions:
Partial fractional credit carries forward across turns, capturing the cumulative benefits of distributed exposure. Third, Monotonicity guarantees that mastered concepts do not regress within a single session.
5. Dynamic Cognitive Termination Check: Progress-Driven Natural Conversational Bounds
Rather than enforcing arbitrary fixed turn limits, KNOWSIM terminates conversations according to the Learning Progress Hypothesis. The session stops at the earliest occurrence of either: (1) Mastery, where \(\ge 80\%\) of all IUs in \(G_q\) achieve knows_well; or (2) Persistent Overload and Stagnation, where over a five-turn sliding window at least two of the following conditions persist: (i) \(\ge 4\) turns flagged with \(L_t > \tau\); (ii) zero upward state transitions across the graph; or (iii) complete teaching silence (all IUs classified as not_mentioned). Conversations that hit neither condition are capped at 15 turns.
Benchmark & Evaluation Protocol¶
To validate KNOWSIM, the authors collected KNOWCHAT, comprising 705 human–AI sessions across two distinct task domains: - Domains: Competition mathematics from MATH Level 5 (15 problems spanning 5 subject categories, paired with objective pre/post-test knowledge gain) and open-ended multidisciplinary expert QA from ExpertQA (150 questions across professional fields); - Study Arms: Strategy arms (holding the model fixed while comparing Adaptive, Comprehensive, and Socratic prompting strategies) and Model arms (holding the delivery strategy constant while evaluating distinct foundational LLMs); - Baselines: Three progressively capable behavioral simulators: Zero-shot (ZS), Zero-shot CoT (ZS-CoT), and Zero-shot CoT with synthetic user profiles (ZS-CoT-Prof); - Core Internal Metrics: Knowledge Gain (\(\text{KG} = \sum_{v \in V} \max(0, s_T(v) - s_0(v))\)), Delivery Calibration (\(\text{DC} = \frac{2PR}{P+R}\) combining precision over explained IUs and recall over teachable IUs), and Cognitive Overload (\(\text{CO} = \frac{1}{T}\sum_t \min(1, L_t / \tau)\)).
Key Experimental Results¶
Main Results: Simulator Alignment with Human Judgment¶
On high-signal human contrasts (\(|\text{Cliff's } \delta| \ge 0.15\)), KNOWSIM achieves high directional sign agreement with human outcome rankings, substantially outperforming existing user simulators.
| Evaluation Setting | Task & Comparison Arm | High-Signal Pairs (HS) | Sign Agreement | Agreement Rate | Binomial \(p\)-value |
|---|---|---|---|---|---|
| KNOWSIM | MathQA–Strategy | 17 | 13 | 76% | \(p = .025^*\) |
| KNOWSIM | MathQA–Model | 20 | 14 | 70% | \(p = .058^\dagger\) |
| KNOWSIM | ExpertQA–Strategy | 20 | 16 | 80% | \(p = .006^{**}\) |
| KNOWSIM | ExpertQA–Model | 15 | 10 | 67% | \(p = .151\) |
| KNOWSIM (Pooled) | MathQA (All Arms) | 37 | 27 | 73% | \(\mathbf{p = .003^{**}}\) |
| KNOWSIM (Pooled) | ExpertQA (All Arms) | 35 | 26 | 74% | \(\mathbf{p = .003^{**}}\) |
| Baseline ZS-CoT | All Arms Pooled (IQ Metric) | 22 | 14 | 64% | \(p = .143\) |
| Baseline ZS | All Arms Pooled (IQ Metric) | 22 | 15 | 68% | \(p = .067^\dagger\) |
| Baseline ZS-CoT-Prof | All Arms Pooled (IQ Metric) | 22 | 16 | 73% | \(p = .026^*\) |
| KNOWSIM (Ours) | All Arms Pooled (IQ Metric) | 22 | 17 | 77% | \(\mathbf{p = .008^{**}}\) |
Frontier LLM Benchmarking & Aptitude–Treatment Interactions (ATI)¶
Benchmarking 9 frontier, mid-tier, and open-weight LLMs with KNOWSIM demonstrates that level-agnostic aggregate rankings mask substantial aptitude-treatment trade-offs (\(\bar{R}\) denotes mean rank across metrics within level; \(\text{CO}\downarrow\) indicates lower overload is better):
| Model | Novice KG | Novice CO↓ | Novice DC | Novice IQ | Novice ¯R | Advanced KG | Advanced CO↓ | Advanced DC | Advanced IQ | Advanced ¯R | Level-Marginalized Mean Rank |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7 | 14.3 | 0.882 | 0.090 | 7.80 | 3.25 | 0.6 | 0.879 | 0.107 | 8.30 | 3.00 | 2.75 (Rank 1) |
| Gemini 3.1 Pro | 13.2 | 0.875 | 0.086 | 8.39 | 3.25 | 0.7 | 0.873 | 0.122 | 7.96 | 1.75 | 3.50 (Rank 2) |
| DeepSeek V4 | 15.5 | 0.945 | 0.092 | 6.50 | 4.00 | 0.5 | 0.890 | 0.109 | 7.74 | 4.25 | 4.00 (Rank 3) |
| Gemini 3.1 Flash | 15.1 | 0.936 | 0.093 | 6.83 | 3.25 | 0.5 | 0.912 | 0.109 | 7.64 | 6.00 | 4.50 (Rank 4) |
| Claude Sonnet 4.6 | 12.9 | 0.877 | 0.079 | 7.06 | 4.75 | 0.6 | 0.866 | 0.100 | 8.00 | 3.50 | 5.25 (Rank 5) |
| GPT-5.4 | 14.9 | 0.940 | 0.088 | 5.24 | 5.50 | 0.4 | 0.903 | 0.102 | 7.05 | 7.75 | 5.25 (Rank 6) |
| Qwen-3.6-35B | 14.4 | 0.934 | 0.082 | 5.68 | 5.25 | 0.5 | 0.892 | 0.081 | 7.39 | 7.00 | 6.00 (Rank 7) |
| GPT-5.4 mini | 12.6 | 0.946 | 0.075 | 5.41 | 8.00 | 0.5 | 0.901 | 0.114 | 7.24 | 5.75 | 6.50 (Rank 8) |
| Llama-4-Maverick | 11.8 | 0.927 | 0.071 | 4.58 | 7.75 | 0.3 | 0.882 | 0.117 | 6.07 | 6.00 | 7.25 (Rank 9) |
Key Findings¶
- Novice Alignment Dominates Behavioral Fidelity: KNOWSIM aligns most strongly with human judgment at the novice level across both tasks (100% sign agreement on MathQA, 13/13, \(p < .001\); 83% on ExpertQA, 10/12, \(p = .019\)). For novices, comprehension progress and cognitive bandwidth constitute the primary drivers of satisfaction, which KNOWSIM captures with high mechanistic fidelity.
- Divergence of Perceived Satisfaction and Objective Gain: In human MathQA dialogues, subjective Interaction Quality (IQ) exhibits a statistically significant negative correlation with objective test score gains (Spearman \(\rho = -0.268, p < .01\)). Users frequently reward assistants that provide smooth, effortless answers even when little genuine learning occurs, highlighting the hazards of optimizing conversational assistants solely via preference-based LLM judges.
- Polarized Breakdown of Calibration Failures: Categorizing 214,642 explained IUs across the benchmark via an ordered decision tree revealed directional failure patterns: novice failures are dominated by over-reaching (PREREQ-BLOCKED at 37% and OVERLOAD-WASTED at 28%, wasting 65% of explained content), whereas advanced user failures are overwhelmed by over-repetition (REDUNDANT explanations accounting for 89%).
- Aptitude–Treatment Inversion Across Models: The optimal assistant shifts dramatically by user level. DeepSeek V4 achieves the highest novice Knowledge Gain (\(\text{KG} = 15.5\)) but inflicts severe cognitive overload (\(\text{CO} = 0.945\)). Conversely, Gemini 3.1 Pro is the best-calibrated model for advanced users (\(\bar{R} = 1.75\), \(\text{DC} = 0.122\)), yet drops to 6th place in novice knowledge gain.
Highlights & Insights¶
- First-Principles Grounding in Cognitive Science: Instead of treating conversational quality as a black-box text score, KNOWSIM maps established learning principles (Ausubel's advance organizers, Sweller's cognitive load, Vygotsky's ZPD) onto explicit graph traversal algorithms and non-linear damping functions.
- Unmasking the "Pleasant but Ineffective" Evaluation Trap: By demonstrating empirical negative correlations between perceived interaction quality and measured knowledge acquisition, the paper exposes a blind spot in current LLM alignment, proving that conversational satisfaction cannot serve as a reliable proxy for cognitive scaffolding.
- Fine-Grained Diagnostic Failure Taxonomy: By tracking concepts at the IU level, the framework decomposes why information delivery fails into five mutually exclusive failure categories, providing model builders with actionable diagnostic insights into whether an assistant is over-reaching or over-repeating.
Limitations & Future Work¶
- Omission of Motivational and Affective Dynamics: The simulator models cognitive capacity and prerequisite gating but neglects emotional factors such as frustration, boredom, and self-efficacy, causing simulated knowledge gain to plateau prematurely for advanced users.
- Generalization to Unstructured Domains: Prerequisite IU extraction is well-defined for structured domains like mathematics and professional question answering, but establishing rigorous prerequisite DAGs in divergent tasks (e.g., creative writing or brainstorming) remains challenging.
- Inference Latency and API Overhead: Evaluating long multi-turn sessions requires per-turn LLM calls for signal extraction and response generation. Distilling these structured components into compact, specialized reward models offers a promising avenue to scale throughput.
Related Work & Insights¶
- vs. Behavioral Simulators (SimulatorArena, KAUCUS): Prior simulators rely on prompt-conditioned personas without state tracking, evaluating assistants purely on conversational surface quality. KNOWSIM introduces an evolving cognitive state, providing mechanistic metrics that explain why an interaction succeeds or overwhelms.
- vs. Deep Knowledge Tracing (DKT, ITS): Traditional student modeling infers binary problem-solving mastery within discrete educational platforms. KNOWSIM generalizes state tracing to unconstrained multi-turn natural language dialogues, capturing instructional granularity, cognitive overload, and prerequisite gating.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates a learning-theoretic user simulator with explicit prerequisite graph state tracking and non-linear cognitive load damping.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Validated against 705 human sessions across two domains with extensive benchmarking of 9 frontier LLMs.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formalization, elegant conceptual illustrations, and compelling narrative exposition.
- Value: ⭐⭐⭐⭐⭐ Highly valuable for interactive alignment, multi-turn conversational evaluation, and automated pedagogical agent development.