MENTOR: Reinforcement Learning via Flexible Teacher-Optimized Rewards for Tool-Use Distillation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/choics2623/MENTOR-RL
Area: LLM Reasoning
Keywords: tool-integrated reasoning, knowledge distillation, reinforcement learning, flexible process rewards, policy alignment
TL;DR¶
MENTOR resolves the exploration dilemma in distilling tool-use capabilities to small language models by replacing rigid trajectory imitation with a flexible process-aware reward (correctness, unordered set-level teacher alignment, and runtime validation) under GRPO, substantially boosting out-of-domain and unseen-tool reasoning performance.
Background & Motivation¶
Augmenting large language models (LLMs) with external computational environments—such as code interpreters and web retrieval APIs—has emerged as a foundational paradigm known as tool-integrated reasoning (TIR). Despite their formidable multi-step problem-solving capabilities, the substantial latency and exorbitant deployment costs of hundreds-of-billions-parameter models render them impractical for broad real-time and on-device deployment. Consequently, transferring tool-assisted reasoning expertise to lightweight small language models (SLMs) has become an urgent research focus. To date, this capability transfer has overwhelmingly relied on supervised fine-tuning (SFT) over static teacher demonstrations. However, offline behavioral cloning confines capacity-limited student models to memorizing rigid token distributions, suffocating exploratory flexibility and causing catastrophic performance degradation when confronted with out-of-domain scenarios or novel tools.
While on-policy reinforcement learning (RL) presents an appealing pathway by enabling student models to discover and refine tool-use logic through interactive trial and error, designing reward signals for capacity-limited SLMs exposes a crippling optimization dilemma. Relying strictly on sparse outcome-only rewards fails because the uninformative reinforcement signal leaves the student unable to navigate the vast action space, prompting it to drift toward complete tool avoidance. Conversely, enforcing strict sequential trajectory matching imposes an excessively restrictive optimization burden on smaller architectures, suffocating valid alternative problem-solving routes and triggering a progressive accumulation of invalid, unexecutable tool calls throughout training.
Empirical comparative diagnostics between the teacher (Qwen3-235B) and student (Qwen3-8B) unearth an illuminating paradox: the student model actually invokes tools at a higher frequency than the teacher, yet achieves drastically lower task accuracy. This performance gap does not stem from insufficient tool engagement, but from severe strategic misalignment: the smaller model relies heavily on redundant functions, falls into repetitive invocation loops, and suffers an invalid call rate 19 times higher than that of the teacher. Crucially, while the teacher exhibits near-flawless error resilience (a mere 0.5 percentage point drop under execution faults), a single malformed tool call triggers a devastating 29.8 percentage point accuracy plunge in the SLM. This systemic fragility underscores that distillation must impart high-level strategic tool-selection planning rather than mechanical token imitation. Core idea: abandon rigid sequential trajectory cloning in favor of MENTOR, an on-policy reinforcement learning framework driven by a composite reward combining task correctness \(R_c\), unordered set-level teacher tool alignment \(R_a\), and execution validity \(R_v\) under GRPO, granting capacity-limited SLMs the flexibility to internalize teacher strategies while exploring valid reasoning paths.
Method¶
Overall Architecture¶
The MENTOR framework is constructed upon Group Relative Policy Optimization (GRPO), operating across four tightly coupled stages: reference trajectory curation, multi-candidate on-policy exploration, multi-faceted flexible reward evaluation, and policy advantage updates. For each query in the training corpus, a verified reference trajectory is generated by the teacher (Qwen3-235B-Thinking) and filtered to retain only demonstrably correct solutions. The student policy then samples a cohort of \(G\) independent rollouts for each question. Each rollout trajectory is assessed against the corresponding teacher reference across three decoupled axes: final answer correctness (\(R_c\)), presence of the essential teacher-selected tool set (\(R_a\)), and full-trajectory runtime executability (\(R_v\)). Finally, group-relative advantages are computed to update the student policy parameters using PPO-clipped objectives with loss masking over external environment feedback tokens, ensuring that the model's update concentrates exclusively on internal thought tokens and tool-invocation syntax.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query x<br/>Retrieve Verified Teacher Trajectory O(t)"] --> B["Student On-Policy Rollout Exploration<br/>Sample G Candidate Trajectories via π_old"]
B --> C["Multi-Faceted Flexible Reward Evaluation<br/>Compute Scalar Reward R(O(s), O(t))"]
C --> D["Final Correctness Assessment<br/>Verify Output Match to Ground Truth Rc"]
C --> E["Unordered Set-Level Teacher Tool Alignment<br/>Match Invocations to Teacher Tool Set Ra"]
C --> F["Full-Trajectory Runtime Validity Check<br/>Verify Zero Execution Exceptions Rv"]
D --> G["Weighted Reward Aggregation<br/>R = wc*Rc + wa*Ra + wv*Rv"]
E --> G
F --> G
G --> H["Group-Relative Advantage Update<br/>Compute Aj,k with Observation Masking"]
Key Designs¶
1. Final Correctness Assessment: Preventing Exploitative Trajectory Drift
Reinforcement learning over intermediate reasoning steps carries the constant hazard of policy gaming, where an agent learns to emit superficial tool calls merely to harvest process incentives without actually solving the problem. To anchor exploration to genuine task mastery, MENTOR integrates the binary Final Correctness Reward \(R_c\). The student's final prediction \(\hat{y}^{(s)}\) is evaluated against the teacher's ground-truth-verified solution \(\hat{y}^{(t)}\):
This outcome signal forms the foundational pillar of the optimization objective, guaranteeing that no matter how well intermediate behaviors mimic teacher choices, no policy update is positively reinforced unless it culminates in a verifiably correct answer.
2. Unordered Set-Level Teacher Tool Alignment: Liberating Trajectory Search from Rigid Sequences
Conventional trajectory distillation frameworks force strict step-by-step sequential imitation (e.g., ToolRL), requiring student actions to strictly match the exact positions, argument structures, and execution order of the teacher. Given the limited capacity of SLMs, such over-constrained targets bottleneck exploration and induce optimization collapse. MENTOR resolves this by abstracting alignment to the tool-set level. Letting \(S(\tau)\) denote the set of distinct tool functions invoked in trajectory \(\tau\), the Teacher Alignment Reward \(R_a\) grants full credit whenever the student invokes the exact set of essential tools utilized by the teacher, completely agnostic to invocation order or frequency:
By decoupling tool selection from rigid execution order, this set-matching design accommodates legitimate computational variations (e.g., executing arithmetic operations in alternative valid orders) while supplying clear guidance that directs the student toward the teacher's strategic choice of instruments.
3. Full-Trajectory Runtime Validity Check: Eliminating Hallucinated and Malformed Calls
Because set-level matching relaxes sequence and argument constraints, unguided exploration could inadvertently reward malformed JSON calls, missing arguments, or runtime crashes that happen to mention the correct tool names. Considering that smaller models exhibit near-zero self-correction ability once an error occurs (crashing task accuracy by 29.8 percentage points), MENTOR enforces the Tool Validation Reward \(R_v\). This reward acts as an all-or-nothing execution gate, awarding credit only when every single tool call in the rollout executes without raising a syntax or runtime exception:
A single runtime fault within a multi-turn trajectory instantly zeroes out \(R_v\). This uncompromising execution filter penalizes speculative, erroneous invocations and drives the invalid tool-call rate down to near zero during training.
Loss & Training¶
The overall scalar reward optimizes a balance across all three dimensions:
Hyperparameter sensitivity sweeps establish the robust default configuration \((w_c, w_a, w_v) = (0.7, 0.3, 0.1)\). Policy optimization is driven by the GRPO objective:
Here, \(\hat{A}_{j,k}\) denotes the group-normalized relative advantage computed across the \(G=10\) sampled rollouts, with clipping threshold \(\epsilon = 0.2\) and KL regularization weight \(\beta = 0.001\) against the frozen base student model \(\pi_{\text{ref}}\). Crucially, the indicator function \(I(\tau_{j,k})\) acts as a loss mask that evaluates to 1 exclusively for model-generated reasoning and tool-action tokens, and 0 for environment observation tokens returned by the remote execution sandbox. This guarantees that external environment outputs do not contaminate policy gradients.
Key Experimental Results¶
Main Results¶
MENTOR is trained on 1.27k verified tool-integrated trajectories derived from AceReason-Math using a remote Python sandbox. Evaluation spans 6 in-domain mathematical reasoning benchmarks (Math-Forge, Omni-MATH, AIME24, AIME25, AMC23, MinervaMath) alongside 4 zero-shot out-of-domain benchmarks testing unseen tools and capabilities: BFCL-v4 (comprising Non-Live, Multi-Turn, Live, and Agentic sub-tracks) and multi-hop retrieval-augmented QA (HotpotQA, 2WikiMultiHopQA, Bamboogle) powered by FlashRAG.
| Model | Method | In-Domain Math (Macro Acc %) | OOD BFCL-v4 (Official Acc %) | OOD RAG (Macro EM %) | Alignment Score AS (%) |
|---|---|---|---|---|---|
| Qwen3-235B | Teacher Baseline | 52.06 | 50.13 | 39.57 | 100.0 |
| Qwen2.5-1.5B | Vanilla | 1.70 | 24.00 | 1.66 | - |
| SFT | 3.17 | 24.59 | 2.43 | 68.20 | |
| Outcome-based RL | 6.69 | 25.12 | 7.59 | 80.12 | |
| Strict RL | 6.95 | 24.85 | 6.39 | 84.50 | |
| MENTOR (Ours) | 9.89 | 25.70 | 8.39 | 87.24 | |
| Qwen2.5-7B | Vanilla | 21.32 | 26.38 | 14.05 | 56.40 |
| SFT | 22.48 | 26.50 | 14.57 | 62.80 | |
| Outcome-based RL | 26.13 | 32.94 | 18.89 | 82.51 | |
| Strict RL | 26.64 | 33.15 | 19.10 | 85.87 | |
| MENTOR (Ours) | 28.63 | 33.52 | 20.73 | 88.63 | |
| Qwen3-1.7B | Vanilla | 33.37 | 29.81 | 16.53 | - |
| SFT | 36.34 | 30.13 | 17.45 | 74.30 | |
| Outcome-based RL | 38.62 | 30.65 | 18.18 | 83.10 | |
| Strict RL | 38.94 | 30.95 | 18.42 | 86.40 | |
| MENTOR (Ours) | 40.90 | 31.20 | 19.23 | 88.90 | |
| Qwen3-8B | Vanilla | 39.12 | 41.35 | 31.83 | 64.10 |
| SFT | 41.47 | 42.16 | 34.14 | 71.80 | |
| Outcome-based RL | 45.19 | 42.78 | 34.81 | 84.20 | |
| Strict RL | 45.77 | 42.95 | 34.66 | 87.10 | |
| MENTOR (Ours) | 48.26 | 43.39 | 35.44 | 89.50 |
Ablation Study¶
On the Qwen2.5-7B backbone, systemic ablations dissect the individual reward components alongside alternative flexible alignment formulations:
| Configuration / Alignment Variant | Math (Acc %) | BFCL-v4 (Acc %) | RAG (EM %) | Alignment Score AS (%) | Mechanism Description |
|---|---|---|---|---|---|
| (1) \(R_c\) (Outcome-based RL) | 25.94 | 30.71 | 18.22 | 82.51 | Sparse outcome signal; suffers from tool avoidance |
| (2) \(R_c + R_v\) | 26.71 | 30.58 | 8.61 | 81.70 | Enforces valid syntax but lacks strategic guidance |
| (3) \(R_a + R_v\) | 24.63 | 26.91 | 14.02 | 78.66 | Lacks correctness anchor; encourages reward gaming |
| (4) \(R_c + R_{\text{strict format}}\) (Strict RL) | 26.37 | 29.92 | 15.61 | 86.28 | Enforces position-wise sequence & argument match |
| (5) \(R_c + R_{\text{strict format}} + R_v\) | 26.82 | 30.47 | 18.19 | 85.87 | Strict matching with validity; constrained generalization |
| (6) \(R_c + R_a + R_v\) (MENTOR) | 27.95 | 31.52 | 21.08 | 88.63 | Full flexible set-matching framework (optimal) |
| Variant: Ordered matching | 26.98 | 30.21 | 19.48 | 87.12 | Positional order matched, arguments ignored |
| Variant: Multiset matching | 27.21 | 30.74 | 20.18 | 87.80 | Order-agnostic but enforces identical call counts |
| Variant: Recall-only | 27.05 | 30.49 | 19.96 | 87.45 | Rewards teacher tool coverage, ignores extra calls |
| Variant: F1 (Graded set overlap) | 27.43 | 31.04 | 20.61 | 88.15 | Continuous partial credit weakens full-recovery push |
Key Findings¶
- Strategic Policy Alignment Correlates with Generalization: Across all configurations, downstream reasoning accuracy exhibits a robust linear correlation with the Jensen-Shannon divergence-based alignment score (Pearson \(r = 0.85\)). MENTOR achieves the tightest alignment with the teacher's strategic tool-selection distribution while maintaining high execution flexibility.
- Validation Penalty Decouples Call Volume from Execution Errors: Without \(R_v\), error rates remain volatile and high throughout training. Injecting \(R_v\) rapidly drives invalid calls to near zero, eliminating the catastrophic cascading degradation observed in vanilla SLMs.
- Diminishing Returns of Strict Trajectory Mimicry: While Strict RL achieves high alignment scores on paper, its rigid sequential constraints restrict the student's ability to recover from unexpected environment states, causing it to fall noticeably short of MENTOR on zero-shot OOD tool benchmarks (e.g., 21.08% vs. 18.19% on RAG for Qwen2.5-7B).
Highlights & Insights¶
- Abstracting Alignment from Tokens to Sets: Traditional distillation assumes knowledge transfer requires fine-grained token-level logit mimicry or step-level state alignment; MENTOR demonstrates that for capacity-limited SLMs, unordered set-level matching provides the ideal level of abstraction—transferring strategic intent without over-constraining execution.
- Targeting Execution Fragility Over Call Frequency: The paper proves that SLMs fail not from tool avoidance but from catastrophic error cascades (29.8% drop after one failure). Incorporating a binary trajectory-level validity check provides a minimal yet highly effective safeguard that stabilizes RL rollouts.
- Cross-Architecture Extensibility: Supplementary evaluations verify that when paired with a brief SFT warm-up, MENTOR successfully transfers reasoning expertise to distinct model families (LLaMA-3.1-8B and LLaMA-3.2-3B) and integrates seamlessly across modern policy optimizers including DAPO and GDPO.
Limitations & Future Work¶
- Reliance on Initial Tool-Grounded Capability: MENTOR presupposes a base model that possesses rudimentary syntax-level tool-calling capabilities. When applying the framework to raw base models with zero tool priors, a preliminary SFT cold-start phase is mandatory.
- Scope Confined to Verifiable Sandboxes: The empirical validation focuses on sandboxed code and structured retrieval where syntax and outputs can be deterministically verified. Extending flexible reward structures to open-ended, non-deterministic web and GUI environments with noisy observations remains an open challenge.
- Teacher Policy Suboptimality: Because reference trajectories are drawn from the teacher, any redundant or suboptimal tool combinations present in successful teacher rollouts are inadvertently adopted as alignment targets.
Related Work & Insights¶
- vs. SFT Imitation Distillation (e.g., MobileLLM, TACTIC): SFT forces rigid imitation of static teacher tokens, suffering from severe out-of-domain degradation; MENTOR adopts an on-policy RL paradigm where the student explores diverse paths under flexible reward guidance.
- vs. Outcome-Only RL (e.g., ReTool): Outcome-only reinforcement learning provides sparse, delayed signals that cause capacity-limited students to abandon tool use; MENTOR introduces process-aware set-matching that maintains active, purposeful tool engagement.
- vs. Rigid Format RL (e.g., ToolRL): ToolRL enforces strict step-by-step position and argument alignment, imposing excessive optimization constraints on SLMs; MENTOR relaxes sequential constraints to unordered tool sets, yielding superior policy stability and OOD generalization.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Elegantly identifies the exploratory dilemma of SLMs and addresses it via flexible set-level reward design rather than mechanical trajectory cloning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 10 in-domain and out-of-domain benchmarks, multiple model scales, ablation variants, and cross-family validations.
- Writing Quality: ⭐⭐⭐⭐⭐ Outstanding narrative clarity, supported by rigorous empirical motivation, transparent mathematical definitions, and detailed failure-case analyses.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly practical and reproducible training blueprint for building resilient, tool-augmented lightweight agents.