Skip to content

Teaching by Learning: Harnessing LLMs Collaboration for Adaptive Data Synthesis via Cognitive Load Feedback

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Data Synthesis, Multi-Model Collaboration, Cognitive Load, Inference Stress Index, Direct Preference Optimization

TL;DR

This work proposes CoFAG, a self-evolving multi-model collaborative data synthesis framework that operationalizes the target model's internal cognitive load via the dilution-resistant Inference Stress Index (ISI), iteratively training the synthesizer via DPO to target the learner's cognitive frontier and significantly outperforming prior synthesis baselines on complex reasoning benchmarks.

Background & Motivation

In recent years, breakthroughs in Large Language Models (LLMs) across complex reasoning, mathematical problem solving, code generation, and domain-specific challenges have been heavily propelled by supervised fine-tuning on large-scale instruction datasets. However, manual curation and domain-expert annotation of such datasets are prohibitively expensive and inherently bottlenecked in terms of scalability. Consequently, the research community has increasingly embraced LLM-driven synthetic data generation. Early paradigms primarily operated in a single-model self-instruct regime, relying on iterative prompting to bootstrap synthetic corpora. Yet single models inevitably suffer from self-delusion and mode collapse, as a model rarely produces training data that breaks through its own intrinsic knowledge boundaries.

To overcome these structural limitations, recent inquiries have pivoted toward multi-model collaborative data synthesis, where distinct LLMs assume specialized roles such as instruction generators and target solvers. Nonetheless, existing feedback mechanisms driving these collaborations suffer from either coarse granularity or severe signal dilution. Outcome-level feedback relies solely on binary correctness signals, rendering it blind to the subtle logical rigor, intermediate conceptual transformations, and density of reasoning steps. Conversely, sequence-level metrics—such as token perplexity, instruction-following difficulty (IFD), and influence functions—routinely average scores across the entire generated sequence. Because lengthy chain-of-thought outputs contain disproportionately large numbers of straightforward connective words and trivial algebraic manipulations, critical reasoning tokens with high computational stress are effectively washed out by the vast sea of uninformative tokens.

Drawing inspiration from human cognitive load theory in educational psychology, the computational strain imposed on working memory surges when an agent executes multi-step logical deductions, confronts unfamiliar distributional shifts, or reconciles multi-source knowledge. The core idea is to shift from external, result-oriented evaluations to an internal, process-aware feedback paradigm that quantifies the target model's computational effort via the Inference Stress Index (ISI)—a Pareto-style aggregation focused on high-uncertainty tokens—and to orchestrate a closed-loop symbiotic collaboration (CoFAG) where the synthesizer is iteratively refined via DPO to generate instructions that systematically push the learner's cognitive frontier.

Method

Overall Architecture

CoFAG (Cognitive-Feedback Adaptive Generation) establishes a closed-loop multi-model collaborative ecosystem comprising three specialized agents: the Synthesizer model (\(S\)), which proposes candidate instructions; the Fixer model (\(F\)), which acts as the learner executing the tasks while exposing its internal decoding state as cognitive feedback; and the Collaborator model (\(C\)), which serves as an authoritative expert providing gold-standard reference responses. Following an initial model preparation phase that aligns basic capabilities and output formats, the framework iteratively executes three sequential stages over multiple rounds: Data Synthesis, Cognitive Feedback via ISI computation, and Model Refinement via DPO and cumulative SFT.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task Prompt P + Seed Dataset Dseed"] --> B["Stage 1: Model Preparation Warm-up<br/>Synthesizer domain SFT + Fixer format alignment"]
    B --> C["Stage 2: Collaborative Data Synthesis<br/>Synthesizer samples instructions + Collaborator generates references"]
    C --> D["Stage 3: Inference Stress Index ISI Computation<br/>Fixer decoding entropy + Pareto Top-ρ aggregation"]
    D --> E["Stage 4: Cognitive Preference Dataset Construction<br/>Top 50% high-load iw vs Bottom 50% low-load il"]
    E --> F["Stage 5: Dual Strategy Refinement<br/>Synthesizer DPO optimization + Fixer cumulative SFT"]
    F -->|Iterative shift of the cognitive frontier| C
    F --> G["Final Deliverables: Enhanced Fixer F and High-Quality Dataset Dfinal"]

Key Designs

1. Model Preparation Warm-up: Eliminating Stylistic Confounders to Isolate True Reasoning Stress

A critical vulnerability in bootstrapping internal state feedback lies in format discrepancy: if a target model is prompted out-of-distribution, elevated output entropy often merely reflects unfamiliarity with syntactic framing rather than genuine logical difficulty. CoFAG decouples syntactic confusion from reasoning stress via a symmetric SFT warm-up protocol. First, the synthesizer model \(S\) is fine-tuned on seed dataset instructions \(\mathcal{D}_{seed}\) using fixed prompt templates \(P\) (e.g., "Generate a clear, solvable, challenging math problem"):

\[\max_{\theta} \sum_{(i) \in \mathcal{D}_{seed}} \log P(i \mid P; \theta)\]

Next, this warmed-up synthesizer generates a small preliminary instruction pool \(\mathcal{D}_{warmup}\) to perform SFT on the fixer model \(F\). Crucially, this aligns the fixer's response formatting and reasoning cadence with standard templates, ensuring that subsequent token-wise entropy measures strictly reflect the intrinsic cognitive load of problem solving rather than lexical or structural unfamiliarity.

2. Inference Stress Index (ISI): Dilution-Resistant Cognitive Load Quantification via Pareto Truncation

Standard sequence-level metrics fail on lengthy mathematical derivations because straightforward connective tokens (such as "Therefore", "We have", simple additions) dominate sequence length, averaging down the score of the critical few tokens where pivotal theorems or conceptual transformations are decided. CoFAG resolves this through token-level entropy and Pareto truncation. For a synthesized instruction \(i_j\) and context tokens \(t_{<k}\), the fixer model's predictive distribution over vocabulary \(\mathcal{V}\) yields the instantaneous Cognitive Bifurcation \(B_k\):

\[B_k = - \sum_{v \in \mathcal{V}} P(v \mid i_j, t_{<k}; \phi) \log P(v \mid i_j, t_{<k}; \phi)\]

A high \(B_k\) indicates acute model hesitation across multiple plausible reasoning branches or non-trivial parametric knowledge retrieval. To aggregate these values into a sample-level reward without signal dilution, CoFAG implements a Pareto-inspired truncation (\(\rho = 0.2\), adhering to the 80/20 rule), considering exclusively the top \(\rho\)-fraction of highest-entropy tokens:

\[\text{ISI}(i_j; \phi, \rho) = \frac{1}{m_j} \sum_{k=1}^{m_j} B_{\pi_j(k)}, \quad m_j = \max(1, \lceil \rho L_j \rceil)\]

where \(\pi_j\) is the permutation sorting \(\{B_k\}_{k=1}^{L_j}\) in descending order. By isolating this upper 20% critical decision junction, ISI accurately discriminates between rote mechanical calculations and profound multi-step conceptual breakthroughs.

3. Dual Strategy Refinement: DPO-Driven Policy Evolution Coupled with Cumulative Fixer Distillation

Passive filtering of static candidate pools cannot adaptively guide generation policies toward newly emergent model weaknesses. CoFAG structures the synthetic candidates into preference pairs: candidate instructions are sorted by their ISI scores, with the top 50% designated as winning samples \(i_w\) (high cognitive load) and the bottom 50% as losing samples \(i_l\) (low cognitive load), assembling \(\mathcal{D}^{(round)}_{pref}\). The synthesizer is then optimized via Direct Preference Optimization (DPO):

\[\mathcal{L}(S; S_{ref}) = - \mathbb{E}_{(i_w, i_l) \sim \mathcal{D}_{pref}} \left[ \log \sigma \left( \beta \log \frac{P_S(i_w)}{P_{S_{ref}}(i_w)} - \beta \log \frac{P_S(i_l)}{P_{S_{ref}}(i_l)} \right) \right]\]

This encourages the synthesizer to master the generative distributions that trigger high cognitive tension in the learner. Concurrently, the fixer model is fine-tuned on the accumulated high-quality instruction-response dataset \(\mathcal{D}^{(round)}_{accumulated} = \bigcup_{j=1}^{round} \mathcal{D}^{(j)}_{high\_quality}\), where responses are provided by the expert collaborator. As the fixer improves, its cognitive baseline shifts upward, dynamically forcing the synthesizer in subsequent iterations to craft even more challenging, multi-layered problem spaces.

Loss & Training

  • Synthesizer Optimization: LoRA fine-tuning (Rank 16) with cosine learning rate schedule (\(2.0 \times 10^{-6}\) for warm-up, \(5.0 \times 10^{-7}\) for DPO), \(\beta = 0.1\), and FTX coefficient of 0.2.
  • Fixer Optimization: Full-parameter fine-tuning under DeepSpeed ZeRO-3 (\(1.0 \times 10^{-5}\) for warm-up, \(1.0 \times 10^{-6}\) across iterative accumulation).
  • Inference & Rollout: VLLM acceleration with temperature 1.2 and top-p 0.9 for diverse instruction synthesis; greedy deterministic decoding (temperature 0) for expert collaborator response generation.

Key Experimental Results

Main Results

Main benchmarks assess accuracy (%) on mathematical reasoning (GSM8K, MATH, Infinity-Instruct), legal reasoning (O1aw), and complex clinical reasoning (medical-o1-reasoning-SFT). All iterative baselines adhere to the standard two-round iteration setup.

Fixer Model / Method GSM8K MATH Infinity-Inst Math Avg. O1aw (Law) Med Domain Avg. Overall Avg. Gain (\(\Delta\))
Qwen2.5-3B (Zero-shot) 79.52 59.62 77.50 72.21 39.84 34.02 36.93 58.10 —
Forewarned is Forearmed (It.2) 80.97 61.32 79.30 73.86 38.97 33.72 36.34 58.86 +0.76
Montessori-Instruct (It.2) 80.81 60.94 78.60 73.45 39.41 34.06 36.73 58.76 +0.66
LLM2LLM (It.2) 80.36 60.22 78.30 72.96 38.65 33.24 35.95 58.15 +0.05
sRecycled 80.66 60.80 78.60 73.35 39.53 34.18 36.86 58.75 +0.65
CoFAG (Ours, It.1) 81.12 60.76 79.10 73.66 38.59 33.78 36.19 58.67 +0.57
CoFAG (Ours, It.2) 81.50 62.30 80.70 74.83 40.16 34.90 37.53 59.91 +1.81
Llama3.2-3B-Instruct (Zero-shot) 78.08 51.70 75.60 68.46 37.03 32.14 34.59 54.91 —
Forewarned is Forearmed (It.2) 79.60 51.84 76.30 69.25 37.59 32.90 35.25 55.65 +0.74
sRecycled 78.92 51.82 76.50 69.08 38.15 33.48 35.81 55.77 +0.86
CoFAG (Ours, It.2) 80.43 52.68 77.30 70.14 38.03 33.26 35.64 56.34 +1.43

Across parameter scaling on Qwen2.5 (3B, 7B, and 14B) evaluated on MATH and AIME (2024 + 2025 combined): - Qwen2.5-3B: MATH reaches 62.30% (+0.98% over Forewarned), AIME reaches 15.00% (+3.33% over Forewarned); - Qwen2.5-7B: MATH reaches 66.16% (+0.78% over Forewarned), AIME reaches 16.67% (+3.34% over Forewarned); - Qwen2.5-14B: MATH reaches 69.14% (+0.74% over Forewarned), AIME reaches 18.33% (+1.66% over Forewarned).

Ablation Study

Ablation experiments conducted on MATH and GSM8K using Qwen2.5-3B as the fixer model:

Configuration / Variant MATH (%) GSM8K (%) Observations & Key Notes
CoFAG (Full Model) 62.30 81.50 Full two-round collaborative adaptive feedback
Warm-up Component Ablations
- w/o Synthesizer's warm-up 60.76 (-1.54) 79.37 (-2.13) Unaligned synthesizer introduces massive semantic noise
- w/o Fixer's warm-up 61.40 (-0.90) 80.97 (-0.53) Formatting misalignment creates spurious entropy noise
Feedback Mechanism Variants
- Forewarned is Forearmed (Binary) 61.32 (-0.98) 81.04 (-0.46) Result-oriented binary feedback misses fine-grained reasoning quality
- Montessori-Instruct (Influence) 60.94 (-1.36) 80.81 (-0.69) Seed-test loss reduction is computationally expensive and less targeted
- IFD (Sequence-averaged PPL) 61.24 (-1.06) 80.89 (-0.61) Full-sequence averaging severely dilutes critical token stress
Synthesizer Refinement Variants
- Original Synthesizer (One-shot) 59.88 (-2.42) 79.68 (-1.82) Static generation fails to steer toward learner deficiencies
- LLM2LLM (Error Rewriting) 60.22 (-2.08) 80.36 (-1.14) Confined to rewriting past mistakes without systemic generalization
- Static Filtering (High ISI Top-50%) 61.08 (-1.22) 80.66 (-0.84) Filtering without DPO optimization misses policy distribution shifts

Key Findings

  • Internal Cognitive Feedback Vastly Outperforms External Outcome Signals: CoFAG consistently eclipses binary correctness baselines (Forewarned is Forearmed, 61.32% vs. 62.30%) and sequence-averaged difficulty metrics (IFD, 61.24%), proving that the top-20% high-uncertainty tokens are the true determinants of instructional quality.
  • Dynamic Policy Optimization Trumps Static Data Filtering: Simply using ISI as a static filter achieves 61.08% on MATH, which is significantly lower than the 62.30% achieved by DPO-based synthesizer refinement. Qualitative analysis shows that DPO shifts the instruction distribution from early basic algebra dominance toward intricate spatial geometry and multi-source logical puzzles, raising the proportion of complex multi-attribute instructions from 28.8% to 52.6%.
  • Superlinear Data Efficiency: Across global iteration analysis, CoFAG's second iteration achieves peak accuracy (~62.0%) with only 13.1K–15.5K training instances, cutting the data volume needed in the first iteration (30K) by nearly half while establishing a higher performance ceiling.

Highlights & Insights

  • Grounding Cognitive Load Theory in Token-Level Decoding Dynamics: By mapping psychological cognitive load directly to the Shannon entropy of token prediction distributions and enforcing a Pareto 80/20 truncation, the authors provide an elegant, mathematically sound solution to the long-standing signal dilution problem in LLM data synthesis.
  • Symbiotic Closed-Loop Curriculum Scheduling: Rather than relying on external heuristics, the synthesizer and fixer form a natural curriculum: as the fixer masters simpler concepts, its internal cognitive stress on them drops, naturally pushing the synthesizer via DPO to explore deeper knowledge frontiers and more complex structural compositions.
  • Negligible Training Overhead with Zero Inference Cost: A thorough first-order computational analysis confirms that CoFAG's training-side overhead is only \(\approx 1.67\times\) that of a standard SFT run (requiring 50,000 forward passes vs. 30,000 for standard SFT), entirely confined to offline data preparation without modifying the target model's inference architecture.

Limitations & Future Work

  • Dependence on Collaborator Response Fidelity: CoFAG primarily governs instruction difficulty and complexity; the accuracy of the final supervision signal remains bounded by the collaborator model. Any reasoning hallucinations or errors from the collaborator may propagate into the fine-tuning corpus.
  • Vulnerability to Cold-Start Format Drift: The ablation studies demonstrate that if the fixer model is insufficiently warmed up, syntactic unfamiliarity pollutes the ISI metric with spurious uncertainty, degrading the quality of the DPO preference pairs.
  • Exploration at Frontier Scales and Long-Thinking Regimes: The framework has been validated up to 14B models. Investigating how internal cognitive load manifests in frontier models (70B+) and reasoning-specialized architectures utilizing test-time rollouts (e.g., DeepSeek-R1, OpenAI o1) remains an open challenge.
  • vs Forewarned is Forearmed / LLM2LLM: These methods rely on model failure as a binary trigger to generate or rewrite data. In contrast, CoFAG demonstrates that even if a model eventually answers correctly, high internal cognitive bifurcation flags critical reasoning hurdles that offer rich pedagogical value.
  • vs Sequence-Level Quality Selection (IFD, sRecycled): Sequence-averaged perplexity and instruction difficulty fail in extended multi-step reasoning due to length dilution. CoFAG's Pareto-style Top-\(\rho\) aggregation specifically isolates high-leverage decision junctions, providing a significantly sharper selection signal.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant mapping of cognitive load theory into dilution-resistant decoding entropy for data synthesis.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain (math, law, medicine), multi-scale (3B to 14B), ablation, and detailed computational FLOP/forward-pass accounting.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough appendix documentation, and insightful qualitative case studies.
  • Value: ⭐⭐⭐⭐⭐ Highly practical and scalable methodology for aligning reasoning LLMs without manual dataset curation.