Skip to content

Teaching by Learning: Harnessing LLMs Collaboration for Adaptive Data Synthesis via Cognitive Load Feedback

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Multi-Agent Collaboration, Data Synthesis, Cognitive Load, Inference Stress Index, Direct Preference Optimization

TL;DR

Addressing the issues of coarse outcome feedback and sequence-level averaging dilution in multi-model collaborative data synthesis, this paper proposes CoFAG, a framework that quantifies internal cognitive load via a dilution-resistant Inference Stress Index (ISI) based on token bifurcation entropy and iteratively optimizes instruction synthesis through a three-agent closed loop with Direct Preference Optimization (DPO).

Background & Motivation

The rapid advancement of Large Language Models (LLMs) heavily relies on massive, high-quality instruction-tuning corpora. However, manual annotation is prohibitively expensive and inherently difficult to scale, driving the community toward synthetic data generated by LLMs themselves. Early approaches leveraging single-model self-improvement commonly suffer from mode collapse and self-delusion, as a model rarely generates novel concepts or complex structures beyond its own intrinsic knowledge boundaries. Consequently, recent paradigms have transitioned toward multi-model collaborative data synthesis, where distinct models assume specialized roles such as instruction generators and student target executors. In this interactive setup, the core research challenge lies in designing an effective feedback mechanism that enables the target model to reliably signal the generator on how to synthesize genuinely high-quality data.

Existing feedback mechanisms exhibit fundamental flaws in identifying high-value instructions that require complex reasoning, deviate from standard distributions, and demand multi-source knowledge integration. Outcome-level feedback methods rely on binary correctness signals, judging solely whether the target model's final answer is right or wrong; such coarse indicators are completely blind to intermediate reasoning steps and cognitive bottlenecks. Conversely, sequence-level feedback methods utilize perplexity, instruction-following difficulty (IFD), or influence functions, but they uniformly aggregate token-level metrics by averaging across the entire sequence. While sequence averaging offers length invariance, it severely dilutes the impact of critical reasoning tokens: a genuinely challenging competition problem can easily be assigned a mediocre score because a handful of crucial logical crossroads are overwhelmed by dozens of trivial function words and standard procedural tokens.

In cognitive psychology, Cognitive Load Theory posits that truly high-quality educational challenges impose substantial mental effort on working memory during execution. In language models, when an instruction compels deep multi-step deduction, unfamiliar knowledge structures, or cross-domain synthesis, the model's internal computational process becomes significantly more strained. Rather than relying on external outcome verification or diluted global metrics, the feedback mechanism should be an internal, process-aware measure based directly on computational effort. Core idea: quantify the target model's internal cognitive load via the conditional entropy of high-uncertainty decoding steps (Cognitive Bifurcation) aggregated through a dilution-resistant Pareto cutoff (Inference Stress Index, ISI), and steer a three-model self-evolving loop using DPO to adaptively synthesize instructions that target the model's cognitive frontiers.

Method

Overall Architecture

The Cognitive-Feedback Adaptive Generation (CoFAG) framework orchestrates a closed-loop multi-agent system comprising three specialized roles: a Synthesizer model (\(S\)), a Fixer model (\(F\)), and a Collaborator model (\(C\)). The operational pipeline begins with an initial model preparation stage to eliminate semantic mismatch and format drift, followed by iterative cycles across multiple rounds. Within each round, the Synthesizer generates candidate instructions given domain prompts; the Collaborator provides high-fidelity reference responses with chain-of-thought derivations; the Fixer processes each instruction-response pair under teacher forcing and exposes token-level entropy; the framework aggregates these scores into dilution-resistant Inference Stress Index (ISI) values to rank instructions and assemble preference pairs; finally, Direct Preference Optimization (DPO) updates the Synthesizer's generation policy, while the Fixer is supervised fine-tuned on the accumulated high-load synthetic dataset.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Initial Seed Dataset Dseed"] --> B["Stage 1: Model Preparation & Alignment<br/>SFT for task priming and decoding style alignment"]
    B --> C["Stage 2: Collaborative Data Synthesis<br/>Synthesizer proposes instructions + Collaborator provides reference responses"]
    C --> D["Stage 3: Cognitive Load Quantification via ISI<br/>Top-ρ Pareto aggregation of bifurcation entropy"]
    D --> E["Stage 4: Cognitive Feedback Strategy Refinement<br/>DPO optimizes Synthesizer + SFT enhances Fixer"]
    E -->|Proceed to next round R| C
    E --> F["Enhanced Target Model & High-Quality Synthetic Dataset"]

Key Designs

1. Model Preparation & Alignment: Eliminating Semantic Noise and Formatting Artifacts Before engaging in dynamic collaborative feedback, baseline competency alignment between the Synthesizer and Fixer is essential. If an unaligned Synthesizer is prompted directly, it frequently outputs out-of-domain gibberish or semantic noise; conversely, if the Fixer has not learned standard output formatting, its decoding uncertainty will be dominated by superficial syntax rather than genuine reasoning difficulty. To address this, the Synthesizer is first fine-tuned on seed dataset \(\mathcal{D}_{\text{seed}} = \{(i_j, r_j)\}_{j=1}^N\) with fixed task prompts \(P\) (e.g., "Generate a mathematical problem..."): $\(\max_{\theta} \sum_{i \in \mathcal{D}_{\text{seed}}} \log P(i \mid P; \theta)\)$ The preliminarily fine-tuned Synthesizer then produces a warmup instruction set used to train the Fixer. This guarantees that the Fixer adheres to consistent solution conventions, ensuring that subsequent cognitive stress measurements purely capture intrinsic multi-step deduction effort rather than stylistic confusion.

2. Cognitive Bifurcation & Dilution-Resistant Inference Stress Index (ISI): Fine-Grained Process-Aware Difficulty To resolve the coarseness of binary correctness and the signal dilution of sequence-averaged perplexity, the framework formalizes internal cognitive load through token-level information entropy and Pareto truncation. For a candidate pair \((i_j, r_j)\) where \(r_j = (t_1, \dots, t_{L_j})\), the Fixer's vocabulary probability distribution \(P(\cdot \mid i_j, t_{<k}; \phi)\) at decoding step \(k\) yields the instantaneous Cognitive Bifurcation \(B_k\): $\(B_k = - \sum_{v \in \mathcal{V}} P(v \mid i_j, t_{<k}; \phi) \log P(v \mid i_j, t_{<k}; \phi)\)$ A elevated \(B_k\) signifies that the model faces multiple viable reasoning branches or must retrieve non-trivial knowledge. To prevent lengthy, formulaic derivations from diluting the impact of pivotal cognitive hurdles, CoFAG applies an 80/20 Pareto principle, sorting bifurcations in descending order \(\pi_j\) and averaging exclusively over the top fraction \(\rho = 0.2\) of tokens: $\(\text{ISI}(i_j; \phi, \rho) = \frac{1}{m_j} \sum_{k=1}^{m_j} B_{\pi_j(k)}, \quad \text{where } m_j = \max(1, \lceil \rho L_j \rceil)\)$ This formulation maintains sequence-length invariance while displaying acute sensitivity to pivotal structural reasoning steps (such as unfolding 3D geometries or setting up multi-variable constraints), serving as a precise proxy for instruction quality.

3. Cognitive Preference-Driven Co-Evolution: Breaking Generation Boundaries via DPO Merely filtering out instructions with high ISI operates only as a static dataset curator without shifting the generative model's underlying distribution. CoFAG leverages ISI as an intrinsic reward signal: candidate instructions are ranked by ISI, and the top 50% cognitively demanding instructions \(i_w\) are paired with the bottom 50% trivial instructions \(i_l\) to construct preference dataset \(\mathcal{D}_{\text{pref}}^{(\text{round})}\). The Synthesizer is then updated via Direct Preference Optimization (DPO): $\(\mathcal{L}(S; S_{\text{ref}}) = - \mathbb{E}_{(i_w, i_l) \sim \mathcal{D}_{\text{pref}}} \left[ \log \sigma \left( \beta \log \frac{P_S(i_w)}{P_{S_{\text{ref}}}(i_w)} - \beta \log \frac{P_S(i_l)}{P_{S_{\text{ref}}}(i_l)} \right) \right]\)$ This forces the Synthesizer to actively internalize the generative distribution of high-cognitive-load problems. The refined Synthesizer then synthesizes fresh training data, the Collaborator supplies golden answers, and the Fixer is fine-tuned on the accumulated high-quality corpus. As the Fixer becomes more capable, it raises the cognitive bar for subsequent iterations, driving a reciprocal self-improving curriculum.

Loss & Training

The framework executes a staged training regime. During the preparation phase, the Synthesizer is fine-tuned with LoRA (rank 16, learning rate \(2.0 \times 10^{-6}\), cosine scheduler), while the Fixer undergoes full-parameter tuning with DeepSpeed ZeRO-3 (learning rate \(1.0 \times 10^{-5}\)). In iterative optimization, the Synthesizer is updated using DPO with LoRA (rank 16, learning rate \(5.0 \times 10^{-7}\), policy deviation parameter \(\beta = 0.1\)), and the Fixer is updated on accumulated datasets via full-parameter tuning (learning rate \(1.0 \times 10^{-6}\)). For generation with VLLM, creative instruction sampling uses temperature 1.2 with top-p 0.9, while reference response generation by the Collaborator uses greedy decoding (temperature 0) to guarantee factual consistency and deterministic execution.

Key Experimental Results

Main Results

Evaluation was conducted across five benchmarks spanning mathematics (GSM8K, MATH, Infinity-Instruct), law (O1aw), and medicine (medical-o1-reasoning-SFT), with Qwen-plus-latest serving as the semantic judge. With Qwen2.5-3B and Llama3.2-3B-Instruct as Fixers, CoFAG consistently surpassed all competitive baselines.

Fixer Backbone Method / Iteration GSM8K (Acc) MATH (Acc) Inf-Inst (Acc) Math Avg. O1aw (Law) Med (Clinical) Global Avg. Relative Gain (\(\Delta\))
Qwen2.5-3B Zero-shot 79.52 59.62 77.50 72.21 39.84 34.02 58.10 -
Forewarned (Iter 2) 80.97 61.32 79.30 73.86 38.97 33.72 58.86 +0.76
Montessori-Inst (Iter 2) 80.81 60.94 78.60 73.45 39.41 34.06 58.76 +0.66
LLM2LLM (Iter 2) 80.36 60.22 78.30 72.96 38.65 33.24 58.15 +0.05
sRecycled 80.66 60.80 78.60 73.35 39.53 34.18 58.75 +0.65
CoFAG (Iter 1) 81.12 60.76 79.10 73.66 38.59 33.78 58.67 +0.57
CoFAG (Iter 2) 81.50 62.30 80.70 74.83 40.16 34.90 59.91 +1.81
Llama3.2-3B Zero-shot 78.08 51.70 75.60 68.46 37.03 32.14 54.91 -
Forewarned (Iter 2) 79.60 51.84 76.30 69.25 37.59 32.90 55.65 +0.74
Montessori-Inst (Iter 2) 79.15 51.94 76.60 69.23 37.40 32.76 55.57 +0.66
LLM2LLM (Iter 2) 78.62 51.76 76.00 68.79 37.15 32.30 55.17 +0.26
sRecycled 78.92 51.82 76.50 69.08 38.15 33.48 55.77 +0.86
CoFAG (Iter 1) 79.45 52.04 76.50 69.33 37.34 32.54 55.57 +0.66
CoFAG (Iter 2) 80.43 52.68 77.30 70.14 38.03 33.26 56.34 +1.43

On competition-level mathematics (MATH and AIME 2024/2025 combined), evaluating scalability across Qwen2.5 parameter sizes (3B, 7B, 14B) demonstrates that CoFAG consistently achieves the highest reasoning accuracy:

Fixer Model Scale Method MATH (Acc) AIME (Acc) Avg. Acc \(\Delta\) vs. Forewarned
Qwen2.5-3B Base Model 59.62 10.00 34.81 -
Forewarned is Forearmed 61.32 11.67 36.50 -
CoFAG (Ours) 62.30 15.00 38.65 +2.15 (+3.33 on AIME)
Qwen2.5-7B Base Model 64.22 13.33 38.78 -
Forewarned is Forearmed 65.38 13.33 39.36 -
CoFAG (Ours) 66.16 16.67 41.42 +2.06 (+3.34 on AIME)
Qwen2.5-14B Base Model 67.86 15.00 41.43 -
Forewarned is Forearmed 68.40 16.67 42.54 -
CoFAG (Ours) 69.14 18.33 43.74 +1.20 (+1.66 on AIME)

Ablation Study

Ablations on Qwen2.5-3B dissect the individual contributions of warmup alignment, feedback modalities, and synthesizer optimization paradigms.

Module Group Configuration Variant MATH (Acc) GSM8K (Acc) Impact Note
Full Framework CoFAG (Full System) 62.30 81.50 Standard 2-round dynamic collaboration loop
Warmup Ablations w/o Synthesizer Warm-up 60.76 79.37 -1.54 / -2.13 (Unaligned generator produces off-topic semantic noise)
w/o Fixer Warm-up 61.40 80.97 -0.90 / -0.53 (Format mismatch contaminates intrinsic reasoning entropy)
Feedback Mechanism Replace with Forewarned (Binary failure) 61.32 81.04 -0.98 / -0.46 (Coarse binary outcome ignores intermediate thought steps)
Replace with Montessori-Instruct (Influence) 60.94 80.81 -1.36 / -0.69 (Loss gradient approximation tends to reward in-distribution fitting)
Replace with IFD (Sequence-averaged difficulty) 61.24 80.89 -1.06 / -0.61 (Uniform sequence averaging severely dilutes critical token signals)
Synthesizer Optimization Original Synthesizer (Single-pass static) 59.88 79.68 -2.42 / -1.82 (Zero feedback loop; restricted to initial generator distribution)
LLM2LLM (Rewriting failed queries) 60.22 80.36 -2.08 / -1.14 (Heuristic rewriting fails to broaden complex reasoning coverage)
High-ISI Static Filtering (w/o DPO) 61.08 80.66 -1.22 / -0.84 (Filtering alone lacks policy adaptation to generate novel frontiers)

Key Findings

  • Synthesizer warmup serves as the indispensable anchor: Omitting the initial SFT for the synthesizer causes the steepest performance collapse (-1.54% on MATH), as an unprimed synthesizer injects severe semantic drift into the collaborative pipeline.
  • Process-aware cognitive signals decisively outshine outcome-level and averaged metrics: Substituting ISI with binary failure feedback (Forewarned, 61.32%), influence functions (Montessori-Instruct, 60.94%), or sequence-averaged perplexity (IFD, 61.24%) validates that top-20% bifurcation entropy captures cognitive strain far more effectively.
  • DPO policy optimization fundamentally expands synthesis frontiers: Analysis of 1,000 synthesized instructions reveals that instances possessing two or three high-quality attributes (complex reasoning, distribution deviation, multi-source knowledge) surged from 28.8% to 52.6% (vs. 34.1% for Forewarned). The margin between passive filtering (61.08%) and active DPO optimization (62.30%) highlights the necessity of policy refinement.
  • The global iterative cycle exhibits rapid data-efficiency scaling: In MATH, round 1 requires 30K training samples to peak at 61.82%, whereas round 2 achieves 62.00% with only 13.1K–15K samples, while raising the absolute ceiling to 62.30%.

Highlights & Insights

  • Internalizing Cognitive Load Theory into generative machine learning: Grounding difficulty in the target model's internal computational stress rather than external black-box verdicts marks a paradigm shift toward process-aware data synthesis.
  • Dilution-resistant Pareto aggregation (Top-\(\rho\)): Computing the mean entropy over the top 20% high-uncertainty tokens elegantly overcomes the classic sequence-averaging trap, preserving high-impact signals at critical conceptual junctures.
  • Modular three-role decoupling: Segregating instruction generation, cognitive load sensing, and reference response generation allows smaller, specialized models to drive high-end reasoning curricula without requiring omnipotent single-agent capabilities.

Limitations & Future Work

  • Additional offline computational overhead: While inference cost for the final fixer model remains unchanged, offline data construction incurs extra overhead (collaborator response sampling, fixer teacher-forcing ISI scoring, and synthesizer DPO training), measuring approximately 1.67x the forward-pass cost of standard SFT.
  • Reliance on collaborator response fidelity: The fixer's final training depends on golden answers provided by the collaborator model; if the collaborator hallucinates in prolonged deductions, noise will propagate into the SFT data.
  • Scaling behavior on frontier models: Evaluations focus on 3B to 14B parameter models; the calibration of ISI and entropy distribution shifts on frontier-scale models (e.g., 70B+) remains to be investigated.
  • vs Forewarned is Forearmed / LLM2LLM: These systems rely on binary correctness or error-induced rewriting, which cannot differentiate between an easy mistake and a profound reasoning obstacle; CoFAG inspects the internal entropy landscape, detecting cognitive struggle even when the model stumbles upon the right answer.
  • vs IFD / Perplexity Filtering: These frameworks compute sequence-wide average cross-entropy, causing critical token signals to vanish in long outputs; CoFAG's top-20% Pareto aggregation isolates key conceptual pivots.
  • vs Montessori-Instruct: Montessori-Instruct relies on costly influence function calculations on validation sets; CoFAG leverages straightforward forward-pass entropy, providing fine-grained feedback with high practical efficiency.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates target model cognitive load as token-level bifurcation entropy with Pareto dilution resistance, introducing a novel process-aware synthesis paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain and cross-architecture evaluations (math, AIME competition, law, medicine; Qwen2.5 and Llama3.2 backbones) with rigorous ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual framing, clear mathematical formulations, intuitive cognitive psychology grounding, and comprehensive appendix details.
  • Value: ⭐⭐⭐⭐⭐ Offers a highly effective, open-source-friendly methodology for synthesizing high-complexity reasoning datasets without reliance on closed-source API distillation.