Rubric-Aware Multi-Expert Online Policy Distillation for Chinese Elementary School Essay Writing¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/TW-NLP/MOPD
Area: Multimodal VLM
Keywords: Rubric-Aware Alignment, Online Policy Distillation, Multi-Expert Fusion, Reverse-KL Divergence, Elementary Essay Generation
TL;DR¶
Addressing the triple failure modes of frontier LLMsโgenre misrouting, adult register mismatch, and conflicting multi-objective gradientsโin Chinese elementary essay writing, this work introduces RA-MOPD to post-train a 4B open model via rubric-weighted reverse-KL distillation and parameter-free cross-genre expert mixing, surpassing 3-shot GPT-5.3 in rubric compliance while Pareto-dominating on child register authenticity.
Background & Motivation¶
Writing instruction in Chinese primary education is inherently rubric-driven, with classroom teachers assessing student essays across structured, genre-specific criteria including format context, narrative completeness, descriptive nuance, and novelty. In instructional support settings, teachers increasingly require high-quality, pedagogue-aligned demonstration essays that accurately reflect these curricula requirements. However, generating suitable classroom model essays imposes strict, multi-faceted constraints: the generation system must select and adhere to the target genre (letters/notices in applied writing, imaginative narratives, chronological accounts, or object descriptions), achieve high scores across genre-specific rubric dimensions, respect precise length boundaries, and crucially, maintain an authentic, age-appropriate student voice. An essay that appears superficially fluent fails pedagogically if it slips into an adult register, violates the required genre format, or misses the core evaluation rubrics.
Directly prompting frontier closed-source large language models reveals three chronic failure modes across standardized benchmarks. First, format non-compliance and genre misrouting occur frequently under unconstrained prompts; for instance, Qwen3-max attains only 64.4% routing accuracy in a 0-shot setting and 75.3% with 3-shot demonstrations. Second, frontier models exhibit acute register mismatch: models such as GPT-5.3 consistently default to polished, highly analytical adult prose that lacks childlike perspective, leading primary school educators in blind human studies to prefer GPT-5.3 outputs in only 28.7% of pairwise comparisons against rubric-tuned student models. Third, jointly optimizing multiple per-genre rubrics within a single reinforcement learning model (Mix-GRPO) triggers severe gradient interference between conflicting objectives; empirical gradient analysis demonstrates that the gradients of Applied Format and Narrative Completeness anti-correlate with a negative cosine similarity (\(\cos \approx -0.13\)), preventing unified multi-task RL from matching the performance of dedicated per-genre specialists.
These challenges demand a unified training paradigm that decouples single-genre reinforcement learning from multi-genre student consolidation without incurring the steep inference costs of serving multiple parallel experts. The core idea is to train modular, per-genre GRPO specialists and distill them into a single 4B student model via RA-MOPD, augmenting on-policy reverse-KL distillation with reward-model perturbation attribution (RW-KL) to upweight rubric-critical positions and utilizing an empirical cross-genre transfer matrix (CGM) as a parameter-free expert blending prior.
Method¶
Overall Architecture¶
The RA-MOPD framework comprises a coherent four-stage post-training pipeline. In Stage 1, a Qwen3-4B base model is fine-tuned on a filtered subset of 1,885 high-quality student essays (scoring \(\ge 4\) on both rubric dimensions) to establish a standardized format router ("Genre: g / Essay: y"). In Stage 2, eight dimension-specific reward models (Qwen3-4B backbones) are trained on doubly-annotated student corpora across four genres (Applied, Imagination, Narrative, Object), achieving an average QWK of 0.864 against adjudicated gold standards. In Stage 3, four dedicated genre specialists are trained independently using Group Relative Policy Optimization (GRPO) with a composite reward encompassing genre routing, two RM rubric metrics, and a Gaussian length penalty. Finally, in Stage 4 (RA-MOPD), a single dense 4B student model is trained on its own rollouts through an on-policy reverse-KL objective without any cross-entropy loss, accelerated and refined by two specialized extensions: Rubric-Weighted KL (RW-KL) and Cross-Genre Mixing (CGM).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Student Essay Corpus<br/>6,000 double-annotated essays"] --> B["Stage 1: Genre-Router SFT<br/>High-quality subset format alignment"]
A --> C["Stage 2: 8 Specialized Reward Models<br/>Train Qwen3-4B RMs on rubric dimensions"]
B --> D["Stage 3: Per-Genre GRPO Experts<br/>Independent policy optimization per genre"]
C --> D
D --> E["Stage 4: Online Policy Distillation (RA-MOPD)<br/>Pure on-policy reverse-KL distillation"]
C -->|Perturbation attribution| F["RW-KL Rubric-Weighted KL<br/>Localize and up-weight rubric tokens"]
D -->|Empirical transfer matrix| G["CGM Cross-Genre Expert Mixing<br/>Parameter-free Top-2 expert blending"]
F --> E
G --> E
E --> H["Single Deployable 4B Generator<br/>Joint rubric mastery & child register"]
Key Designs¶
1. Rubric-Weighted Token-Level KL (RW-KL): Exploiting Reward Model Attribution for Focused Distillation Standard online policy distillation minimizes token-uniform reverse-KL divergence, treating all generated positions equally. However, pedagogical essay rubrics are governed by highly localized structural elements: in applied letters, formal salutations and sign-offs account for less than 10% of total tokens yet dictate over 40% of the format rubric score, while narrative completeness hinges on key connectives and climactic action verbs. To inject fine-grained scoring awareness directly into the distillation gradient, RW-KL partitions each on-policy student rollout \(\hat{y}\) into \(K=10\) contiguous segments \(\{c_1, \ldots, c_K\}\) and computes zeroth-order leave-one-out perturbation attribution scores for each relevant dimension \(d \in \{d_a, d_b\}\): $\(\Delta_d(c_i) = \text{RM}_d(x, \hat{y}) - \text{RM}_d(x, \hat{y} \setminus c_i)\)$ where \(\hat{y} \setminus c_i\) substitutes chunk \(c_i\) with a length-matched neutral placeholder. Following non-negative rectification, rollout-level normalization, and maximum rescaling, chunk scores are expanded to token-level weights \(\hat{a}_g(t)\). The final token weight is defined as \(w_t = 1 + \lambda \hat{a}_g(t)\) (with sharpness parameter \(\lambda=2\)), selectively magnifying reverse-KL gradients at rubric-critical junctures and preventing subtle structural flaws from being smoothed over by dominant sequence-level losses.
2. Cross-Genre Expert Mixing (CGM): Parameter-Free Distillation Prior from Empirical Transfer Matrices Conventional multi-expert distillation exclusively distills knowledge from the single matched specialist \(\pi_{\phi_g}\), discarding the substantial cross-genre competencies acquired by other experts (validation analysis reveals off-diagonal transfer scores exceeding 3.51 across all expert pairs). Rather than adding a complex, learned routing network that introduces inference latency, CGM computes a static empirical transfer matrix \(H \in \mathbb{R}^{4 \times 4}\) on the validation split. For any target genre \(g\), CGM extracts the Top-2 performing experts from row \(H_{g,:}\) and normalizes their relative weights via a temperature-scaled softmax (\(\tau=0.2\)): $\(\pi_{g'|g} = \frac{\exp(H_{g,g'}/\tau) \mathbb{I}[g' \in \text{Top2}(H_{g,:})]}{\sum_{g''} \exp(H_{g,g''}/\tau) \mathbb{I}[g'' \in \text{Top2}(H_{g,:})]}\)$ The effective teacher distribution is formed dynamically at the probability level as \(Q_{\text{mix}}^{(t)} = \sum_{g' \in \text{Top2}} \pi_{g'|g} \text{Softmax}(\ell_{\phi_{g'}}^{(t)})\). This parameter-free formulation channels valuable cross-genre lexical diversity and rhetoric into the student (e.g., enriching object descriptions with dynamic narrative verbs) at zero parameter cost, matching the performance of complex learned-gating oracles.
3. Pure On-Policy Reverse-KL Optimization with Composite Group Rewards RA-MOPD adheres strictly to a pure on-policy reverse-KL distillation objective (\(D_{\text{KL}}(P_\theta^{(t)} \parallel Q_{\text{mix}}^{(t)})\)) evaluated solely on the student's self-generated rollouts \(\hat{y} \sim \pi_\theta(\cdot \mid x)\), omitting any offline cross-entropy (CE) stream. This mode-seeking orientation prevents the student from wasting probability mass on low-probability teacher outputs that it cannot reliably execute, avoiding the severe exposure bias documented in offline distillation baselines. Furthermore, this distillation objective directly consolidates the multi-objective reward structure established during Stage 3 GRPO, which unifies binary format compliance \(R_{\text{genre}}\), bounded rubric rewards \(R_d \in [0, 1]\), and Gaussian length-tolerance penalties \(R_{\text{len}} = \exp\left(-\frac{(L(y)-L^\star)^2}{2\sigma_L^2 (L^\star)^2}\right)\) into a stable, single-model parameterization.
Key Experimental Results¶
Main Results¶
Evaluations were conducted on a held-out benchmark of 1,200 prompts strictly disjoint from the training data. The table below reports genre-routing accuracy (Acc), the 8-dimension mean reward model score (8-Avg), external GPT-4.1 holistic scoring, BERTScore-F1 against human references, and teacher preference win rates.
| System | Routing Acc (%) | RM 8-Avg (1-5) | GPT-4.1 Holistic | BERTScore-F1 | Teacher Pref (%) |
|---|---|---|---|---|---|
| Gold (Human Essays) | โ | 3.48 | 3.71 | โ | โ |
| Qwen3-4B Base | 39.3 | 3.30 | โ | โ | โ |
| Qwen3-max (3-shot) | 75.3 | 3.65 | โ | โ | โ |
| DeepSeek-V4-pro (3-shot) | 58.3 | 3.66 | โ | โ | โ |
| GPT-5.3 (3-shot) | 94.8 | 4.12 | 3.96 | 0.641 | 28.7 |
| Stage 1 SFT | 94.1 | 3.78 | 3.62 | 0.623 | 17.1 |
| Mix-GRPO (Joint Multi-Genre) | 91.0 | 3.84 | โ | โ | โ |
| Best Per-Genre GRPO Expert | 90.8 | 3.91 | โ | โ | โ |
| 4-Expert Logit-Average (4ร Cost) | 100.0 | 4.18 | โ | โ | โ |
| Base MOPD (Single-Expert Reverse-KL) | 97.5 | 4.09 | โ | โ | โ |
| RA-MOPD (Ours) | 97.8 | 4.21 | 4.04 | 0.679 | 62.8 |
Ablation Study¶
Ablation investigations isolate the incremental gains of the proposed components against Base MOPD, alongside alternative distillation and mixing protocols on the held-out test set.
| Variant / Distillation Protocol | 8-Avg (RM) | \(\Delta\) over Base | Routing Acc (%) | Core Mechanism / Analysis |
|---|---|---|---|---|
| Base MOPD (Single Expert) | 4.09 | โ | 97.5 | Pure on-policy reverse-KL baseline |
| + RW-KL only | 4.15 | +0.06 | 97.6 | Localized token-level perturbation reweighting |
| + CGM only | 4.15 | +0.06 | 97.7 | Empirical transfer-prior Top-2 teacher blending |
| RA-MOPD (+ RW-KL + CGM) | 4.21 | +0.12 | 97.8 | Perfectly complementary combined extensions |
| Offline CE Distillation (Seq-level KD) | 3.93 | -0.16 | 94.8 | High exposure bias on pre-generated rollouts |
| Online Forward-KL | 4.02 | -0.07 | 96.3 | Mean-seeking KL spreads mass to low-quality modes |
| Random 2nd Expert (CGM Control) | 4.10 | +0.01 | 97.5 | Disproves "any secondary teacher helps" hypothesis |
| Uniform Top-2 (0.50 / 0.50) | 4.12 | +0.03 | 97.6 | Sub-optimal without empirical matrix weighting |
| Learned Gating Oracle (CGM Control) | 4.16 | +0.07 | 97.7 | CGM matches learned oracle at zero parameter cost |
Key Findings¶
- Orthogonal Complementarity of RW-KL and CGM: Decomposing RA-MOPD demonstrates that RW-KL (+0.06) and CGM (+0.06) provide exactly additive improvements (+0.12). RW-KL achieves its most pronounced gains on structurally localized rubrics (Applied Format +0.07, Narrative Completeness +0.07), while CGM delivers maximum lift on dimensions enriched by cross-genre vocabulary and rhetoric transfer (Object Multisense & Rhetoric +0.08, Narrative Detail +0.07).
- Pareto Dominance on the Rubric-Register Plane: When plotting rubric compliance (RM 8-Avg) against Child Register Authenticity (CRS, scored by external teachers), GPT-5.3 collapses to a low register score of 2.91 despite achieving 4.12 in rubric compliance. Human gold essays hold the register ceiling (4.28) but lag in rubric rigor (3.48). RA-MOPD achieves 4.21 in rubric compliance and 3.74 in CRS, strictly Pareto-dominating all other deployable generators.
- Robust Transfer Across Model Scales: Replicating the entire pipeline on Qwen3-1.7B without hyperparameter retuning yields a stable +0.11 gain for RA-MOPD over Base MOPD (3.93 \(\to\) 4.04, 96.3% routing), demonstrating that the framework's algorithmic components generalize across parameter scales.
Highlights & Insights¶
- Transforming Black-Box Reward Models into Token-Level Attention Oracles: Rather than relying exclusively on scalar, sequence-level RL rewards, the paper cleverly extracts leave-one-out chunk sensitivities from trained reward models to guide the token-level reverse-KL distillation loss, directing optimization gradients directly to pedagogical pivot points.
- Replacing Complex Learned Routing with Empirical Task Transfer Matrices: In contrast to parameter-heavy Mixture-of-Experts architectures, the framework demonstrates that a static \(4 \times 4\) transfer matrix computed on validation prompts provides an optimal mixing prior for Top-2 expert teachers, matching the performance of learned gating networks at zero additional parameter cost.
- De-biasing Asymmetric Frontier Comparisons: The authors explicitly frame their results not as general 4B superiority over massive foundation models, but as evidence of the immense marginal utility of narrow-domain post-training, format-explicit supervision, and targeted RLHF alignment in overcoming the adult-register biases of frontier LLMs.
Limitations & Future Work¶
- Curricular and Geographic Bias: The underlying corpus is restricted to grades 3โ6 within the Shanghai municipal educational standard (with grade 6 comprising only 10.1% and concentrated in applied writing). Generalization across diverse regional curricula, unobserved rubrics, or unseen genres remains unverified.
- Coarse Attribution on Diffuse Dimensions: For holistic criteria such as Imagination Novelty, chunk-substitution attribution achieves lower rank consistency (Kendall \(\tau=0.52\)), likely due to non-linear token interactions and conflation between content semantics and token lengths.
- Future Directions: Exploring gradient-based or integrated-gradient continuous attribution methods within the distillation loop, alongside evaluating modular single-expert preference optimization (e.g., per-genre DPO/ORPO) distilled via RA-MOPD.
Related Work & Insights¶
- vs. Automated Essay Scoring (AES): Traditional AES systems (e.g., Taghipour & Ng, iflyea, BERT-AES) treat rubric evaluation as an end-of-pipe grading or feedback generation task; RA-MOPD directly internalizes eight fine-grained rubric scoring models into the inner loop of generative policy distillation.
- vs. Standard Online Policy Distillation (MiniLLM / DeepSeek-V4 / MiMo-V2): Prior OPD methods rely on token-uniform reverse-KL matching against a single monolithic teacher; RA-MOPD advances the state of the art by introducing token-importance weighting guided by reward-model attribution and parameter-free cross-genre expert blending.
Rating¶
- Novelty: โญโญโญโญโ Elegantly bridges reward model perturbation attribution with token-weighted reverse-KL distillation and static empirical expert mixing.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive 6,000-essay benchmark, 8 validated reward models, 6-teacher blinded evaluations, and multi-scale validation.
- Writing Quality: โญโญโญโญโญ Outstanding clarity, thorough ablation coverage, rigorous error analysis, and transparent pedagogical problem framing.
- Value: โญโญโญโญโญ Sets a high benchmark for localized, multi-attribute policy alignment, instructional AI support, and lightweight on-device generative modeling.