On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Zzzz-166/Guided-OPD
Area: LLM Reasoning
Keywords: On-Policy Distillation, Multi-turn Agents, Curriculum Learning, Turn-level Guidance, Trajectory Distribution Drift
TL;DR¶
Addressing the failure mode where compounding errors in multi-turn On-Policy Distillation (OPD) drive rollouts out of the teacher's familiar distribution, this paper proposes Guided-OPD, which injects teacher turns via a cosine curriculum schedule, pairs asymmetric KL divergences according to the acting role, and eliminates training-inference mismatch.
Background & Motivation¶
Multi-turn autonomous agents that formulate multi-step plans, invoke tools, and iteratively engage with dynamic environments have emerged as a dominant paradigm for solving intricate real-world problems spanning embodied navigation, scientific experimentation, and web interaction. However, sustaining robust long-horizon reasoning and error recovery has historically demanded massive foundation models comprising tens or hundreds of billions of parameters. Deploying such enormous models in latency-sensitive, on-device, or cost-constrained environments remains economically and computationally prohibitive. On-Policy Distillation (OPD) has recently gained prominence as an appealing recipe for transferring agentic reasoning capabilities to compact student models, leveraging student-generated exploration rollouts while providing dense token-level divergence supervision from a capable teacher.
Despite its success in single-turn mathematical reasoning and code generation, applying vanilla OPD to multi-turn interactive agents reveals a characteristic failure mode rooted in the compounding nature of environment interactions. Within an interactive episode, an agent's actions directly govern state transitions under partial observability; consequently, minute behavioral deviations committed by the student in early turns trigger unexpected environment feedback that compounds across subsequent interaction rounds. Empirical analysis of the teacher model's perplexity (PPL) evaluated over student-generated tokens reveals a steady, monotonic upward climb across the interaction horizon that fails to subside even after extended optimization. In effect, the trajectory quickly drifts outside the state distribution familiar to the teacher, leaving the teacher profoundly uncertain and rendering its token-level supervision least dependable precisely when the struggling student requires accurate guidance most.
Attempting to resolve this failure mode using existing distillation variants exposes a dilemma between trajectory coherence and training-inference consistency. Interleaving teacher and student actions at the fine-grained token level yields chimera text fragments that violate grammatical syntax and destroy decision coherence, whereas coarse-grained sequence handover mechanisms—such as AdaSwitch, which permanently delegates all remaining generation to the teacher upon exceeding a divergence threshold—monopolize the trajectory tail with teacher tokens, leaving the student untrained for autonomous error recovery under pure inference conditions. The fundamental objective is therefore to introduce stabilizing expert anchors without inheriting persistent distribution mismatches. Core idea: Anchor the rollout at the discrete granularity of full environment interaction turns by stochastically mixing teacher and student steps, optimize them via role-aligned asymmetric KL divergences, and employ a cosine curriculum schedule that decays teacher intervention to zero to guarantee convergence back to an unguided on-policy regime.
Method¶
Overall Architecture¶
Guided-OPD formulates an interactive on-policy distillation pipeline designed to prevent rollout-level distribution drift during training while smoothly transitioning to a fully unassisted, inference-consistent policy. The methodology comprises two tightly integrated components: Turn-level Guidance and Curriculum Guidance.
During training step \(t\), the agent interacts with the environment across an interaction horizon to sample rollout trajectory \(\tau\). At each interaction turn \(u\)—which encapsulates the current state history \(h_u\), the thought-action response \(a_u\), and the subsequent environment observation \(o_{u+1}\)—a stochastic role indicator \(z_u \sim \text{Bernoulli}(\beta_t)\) assigns the driving role to either the teacher \(\pi_\phi\) with probability \(\beta_t\) or the student \(\pi_\theta\) with probability \(1 - \beta_t\). Regardless of which policy produces \(a_u\), the action is executed in the environment and appended to the shared trajectory history. Turns sampled by the student are optimized via mode-seeking reverse KL divergence, whereas turns generated by the teacher are learned via mass-covering forward KL divergence. As training advances, the intervention probability \(\beta_t\) decays according to a cosine schedule from \(\beta_{\text{start}} = 1.0\) down to \(\beta_{\text{end}} = 0\), ensuring that the final training phase operates strictly under vanilla on-policy dynamics.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Initial Observation o0 and Context h0"] --> B["Training Progress & Curriculum Computation<br/>ξt = min(t / Tdecay, 1)"]
B --> C["Cosine Curriculum Scheduling<br/>Dynamically compute teacher mixing probability βt"]
C --> D{"Turn-level Role Sampling<br/>zu ~ Bernoulli(βt)"}
D -->|zu = 1 Teacher Turn Prob βt| E["Turn-level Guidance: Teacher Demonstration<br/>au ~ πϕ(· | hu), environment step hu+1"]
D -->|zu = 0 Student Turn Prob 1-βt| F["Turn-level Guidance: Student Exploration<br/>au ~ πθ(· | hu), environment step hu+1"]
E --> G["Asymmetric Loss: Forward KL Divergence<br/>DKL(πϕ ∥ πθ) enforces mass covering"]
F --> H["Asymmetric Loss: Reverse KL Divergence<br/>DKL(πθ ∥ πϕ) enforces mode seeking"]
G --> I["Gradient Backpropagation & Model Update<br/>Smooth convergence to pure OPD as βt → 0"]
H --> I
Key Designs¶
1. Turn-level Guidance: Aligning intervention granularity with environment state transitions
In multi-turn interactive environments, the driving force behind environment state transitions is not an isolated individual token, but an executable decision unit comprising complete reasoning thoughts followed by concrete action commands. To prevent the syntactic fragmentation and chimeric artifacts caused by token-level switching, Guided-OPD establishes the complete turn \(a_u = (\text{thought}, \text{action})\) as the fundamental unit of teacher intervention. Conditioned on interaction history \(h_u\), each turn is independently drawn from the teacher with probability \(\beta\) or the student with probability \(1 - \beta\):
The generated response is incorporated into the persistent history \(h_{u+1} = (h_u, a_u, o_{u+1})\), ensuring that subsequent environment states remain structurally coherent while intermittently resetting drifting trajectories back toward the teacher's high-confidence domain.
2. Asymmetric Loss Design: Matching divergence objectives to generative roles
Applying a uniform divergence across both student- and teacher-generated turns misaligns the optimization objectives with the underlying role of each turn. Guided-OPD introduces a role-specific asymmetric loss formulation. On a student-generated turn, the objective is to constrain the student's sampled actions to modes endorsed by the teacher without forcing it to cover every alternative expert path; hence, it utilizes the mode-seeking reverse KL divergence:
Conversely, on a teacher-generated turn, the teacher injects an expert demonstration to recover from potential deadlocks, and the student must absorb this demonstration into its behavioral repertoire; hence, it employs the mass-covering forward KL divergence:
This prevents the student from using its current incomplete policy to discount valid expert actions, successfully harmonizing exploration sharpening with demonstration coverage.
3. Cosine Curriculum Scheduling: Eliminating persistent training-inference mismatch
Maintaining a fixed non-zero teacher intervention rate throughout training inadvertently conditions the student to rely on periodic expert rescues, leaving it vulnerable when forced to navigate long trajectories entirely on its own at test time. Guided-OPD incorporates a progressive curriculum schedule over the training timeline. Across \(T_{\text{total}}\) optimization steps, the first fraction \(\rho\) (default \(\rho = 0.8\)) is designated as the curriculum stage with \(T_{\text{decay}} = \rho T_{\text{total}}\). Defining the progress ratio \(\xi_t = \min(t / T_{\text{decay}}, 1)\), the intervention probability \(\beta_t\) decays along a cosine trajectory:
Initialized at \(\beta_{\text{start}} = 1.0\) and ending at \(\beta_{\text{end}} = 0\), the cosine curve exhibits gentle gradients near both extremities with rapid middle-phase transition. This architecture provides heavy expert scaffolding during early exploration when student drift is acute, and subsequently phases out external guidance so the final \(20\%\) of training replicates unassisted test-time inference.
Loss & Training¶
At each training step \(t\), complete trajectories \(\tau\) are sampled under the turn-level mixed policy distribution \(\pi_{\theta, \phi}^{\beta_t}\). The cumulative objective aggregates the role-specific losses across all visited turns:
In expectation, this decomposes into the convex combination of reverse-KL distillation on student-induced states and forward-KL distillation on teacher-induced states:
All models are implemented on the Trinity-RFT framework and trained across an 8-GPU node of NVIDIA H20 accelerators (configured as 4 actor GPUs, 2 teacher GPUs, and 2 learner GPUs). Training employs the AdamW optimizer with a learning rate of \(1 \times 10^{-6}\), gradient clipping of 1.0, micro-batch size of 16, and total batch size of 64 over 250 steps.
Key Experimental Results¶
Main Results¶
Experiments distill a Qwen3-30B-A3B teacher into Qwen3 student models of scale 0.6B, 1.7B, and 4B across three interactive benchmarks: ALFWorld (evaluating both IID seen rooms and OOD unseen rooms), ScienceWorld, and WebShop. Evaluation metrics encompass task Score (continuous \([0, 100]\)), Success Rate (SR % binary), and average interaction rounds (Round \(\downarrow\)).
| Student Size | Method | ALFWorld IID SR | ALFWorld OOD SR | ScienceWorld Score | ScienceWorld SR | WebShop Score | WebShop SR | Average Score | Average SR | Average Round |
|---|---|---|---|---|---|---|---|---|---|---|
| 30B-A3B | Teacher Reference (Zero-Shot) | 43.57 | 40.80 | 27.73 | 7.25 | 55.07 | 19.25 | 41.40 | 27.72 | 17.96 |
| 0.6B | Zero-Shot | 0.24 | 0.25 | 5.19 | 2.25 | 10.37 | 3.75 | 7.78 | 1.62 | 23.51 |
| Vanilla OPD | 19.29 | 16.42 | 10.42 | 1.50 | 39.89 | 8.00 | 25.16 | 11.30 | 21.68 | |
| AdaSwitch | 25.11 | 17.91 | 9.56 | 1.25 | 43.41 | 7.75 | 26.49 | 13.01 | 21.43 | |
| TCOD | 24.29 | 14.93 | 15.71 | 3.00 | 46.37 | 8.25 | 31.04 | 12.62 | 20.94 | |
| Guided-OPD (Ours) | 27.14 | 19.40 | 19.63 | 3.75 | 48.44 | 9.75 | 34.04 | 15.01 | 19.96 | |
| 1.7B | Zero-Shot | 8.10 | 7.10 | 6.48 | 0.75 | 28.71 | 10.25 | 17.60 | 6.55 | 22.37 |
| Vanilla OPD | 23.57 | 26.87 | 16.94 | 3.00 | 47.59 | 11.00 | 32.27 | 16.11 | 20.20 | |
| AdaSwitch | 25.71 | 23.13 | 17.02 | 2.75 | 46.79 | 10.50 | 31.91 | 15.52 | 20.59 | |
| TCOD | 27.86 | 24.63 | 19.08 | 3.25 | 49.47 | 14.25 | 34.28 | 17.50 | 19.63 | |
| Guided-OPD (Ours) | 32.86 | 29.85 | 23.24 | 5.25 | 52.63 | 11.75 | 37.94 | 19.93 | 18.73 | |
| 4B | Zero-Shot | 28.57 | 27.86 | 16.21 | 3.75 | 20.59 | 8.25 | 18.40 | 17.11 | 19.30 |
| Vanilla OPD | 31.43 | 32.09 | 26.45 | 7.50 | 50.87 | 13.50 | 38.66 | 21.13 | 18.78 | |
| AdaSwitch | 29.29 | 29.10 | 25.24 | 6.25 | 48.91 | 12.35 | 37.08 | 19.25 | 19.40 | |
| TCOD | 32.86 | 31.34 | 25.86 | 9.25 | 53.13 | 19.00 | 39.50 | 23.11 | 18.71 | |
| Guided-OPD (Ours) | 37.14 | 35.08 | 29.50 | 11.50 | 55.78 | 17.75 | 42.64 | 25.37 | 18.26 |
Ablation Study¶
Ablation experiments conducted on Qwen3-1.7B evaluate the asymmetric divergence pairing, curriculum necessity, and schedule curvature (metrics averaged across ALFWorld, ScienceWorld, and WebShop):
| Group | Variant Configuration | Score ↑ | SR (%) ↑ | Round ↓ | Description and Analysis |
|---|---|---|---|---|---|
| Full Model | Guided-OPD (Ours) | 37.94 | 19.93 | 18.73 | Default setup: asymmetric loss (student rKL, teacher fKL) + cosine curriculum |
| (a) Loss Assignment | Inverted (student fKL, teacher rKL) | 33.32 | 15.47 | 23.30 | Misaligns divergences: student turns drift while teacher turns are discarded (-4.62 Score) |
| Symmetric rKL on both roles | 36.58 | 17.83 | 20.77 | Student fails to cover broad demonstrations provided during teacher interventions | |
| Symmetric fKL on both roles | 34.56 | 17.02 | 21.13 | Forces mass-covering on student turns, aggravating exploratory divergence | |
| (b) Curriculum Schedule | Fixed mixing \(\beta = 0.25\) | 34.41 | 16.09 | 19.97 | Insufficient early anchoring allows severe early compounding drift |
| Fixed mixing \(\beta = 0.50\) | 35.17 | 15.62 | 20.05 | Intermediate guidance prevents student from developing unassisted recovery skills | |
| Fixed mixing \(\beta = 0.75\) | 35.85 | 14.82 | 19.36 | Heavy reliance inflates partial-credit Score but crashes binary SR (-5.11%) | |
| (c) Decay Curvature | Linear schedule | 36.12 | 17.01 | 20.45 | Lacks early plateau for high-drift phase and gentle exit for pure OPD |
| Exponential schedule | 34.44 | 15.79 | 19.58 | Aggressive early decay cuts teacher guidance prematurely, causing sharp drop |
Key Findings¶
- Pronounced scaling benefits on smaller students: Guided-OPD delivers relative gains over vanilla OPD of +35.3% in Score and +32.8% in SR on the 0.6B student, compared to +10.3% in Score and +20.1% in SR on the 4B student. Smaller students suffer far more acute rollout drift, making turn-level anchoring disproportionately effective.
- 4B student surpasses 30B teacher: Distilled via Guided-OPD, the Qwen3-4B student achieves an overall average Score of 42.64, surpassing its 7.5x larger Qwen3-30B-A3B teacher (41.40) and outperforming it on ScienceWorld (29.50 vs 27.73) and WebShop (55.78 vs 55.07).
- Fixed mixing creates a divergence between partial credit and true task success: Under fixed \(\beta\), increasing intervention from 0.25 to 0.75 raises Score from 34.41 to 35.85 but drops binary SR from 16.09% to 14.82%. Constant external support teaches the model to imitate intermediate sub-actions without learning autonomous recovery, a dilemma resolved by the decaying curriculum.
Highlights & Insights¶
- Teacher PPL as an empirical diagnostic: Demonstrates that the breakdown of multi-turn OPD is driven by rollout-level distribution drift rather than insufficient training duration, providing a quantifiable metric for agent distillation stability.
- Role-matched asymmetric divergence pairing: Capitalizes on the mode-seeking property of reverse KL on self-generated turns and the mass-covering property of forward KL on expert demonstrations, establishing an optimal balance between trajectory sharpening and skill acquisition.
- Purely algorithmic, critic-free design: Eliminates the overhead of training external process reward models or designing complex state-rollback heuristics, providing an easily implementable mechanism across diverse agent architectures.
Limitations & Future Work¶
- Restricted environment modalities: Validation is currently focused on text-based environments with discrete action vocabularies, leaving open questions regarding continuous control, multi-modal GUI agents, and complex coding environments with open-ended API calls.
- Intra-family distillation constraint: Experiments exclusively evaluate the Qwen3 series sharing identical tokenizers and inductive biases; cross-family distillation (e.g., Qwen to Llama) with divergent token spaces warrants further investigation.
- Heuristic schedule tuning: Hyperparameters such as curriculum ratio \(\rho=0.8\) and initial mixing \(\beta=1.0\) are fixed statically; adaptive curriculum mechanisms modulated by real-time teacher PPL or policy entropy represent promising extensions.
Related Work & Insights¶
- vs Vanilla On-Policy Distillation (Agarwal et al., 2024): Standard OPD relies solely on student rollouts, which leads to compounding trajectory drift in multi-turn environments; Guided-OPD stabilizes exploration via turn-level teacher interventions, reducing interaction rounds by 6.0% while boosting average Success Rate by 25.5%.
- vs Adaptive One-Shot Handover (AdaSwitch, Peng et al., 2025): AdaSwitch irreversibly delegates generation to the teacher once token divergence spikes, introducing an intractable training-inference mismatch; Guided-OPD alternates turns dynamically and anneals the intervention probability to zero.
- vs Macro-Horizon Curriculum Distillation (TCOD, Wang et al., 2026b): TCOD progressively expands the truncation depth \(k\) but leaves deep rollouts unguided; Guided-OPD intervenes microscopically inside the rollout, achieving superior task scores across all model sizes.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Pinpoints the core mechanism of multi-turn rollout drift via teacher PPL and introduces an elegant turn-level asymmetric curriculum formulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across three agent benchmarks, three student scales, comprehensive ablations on loss and scheduling, and detailed teacher takeover diagnostics.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative flow, strong theoretical grounding, and clean presentation of algorithmic schematics.
- Value: ⭐⭐⭐⭐⭐ Delivers a practical, high-performance training recipe for distilling agentic reasoning into lightweight models without external reward modeling overhead.