Skip to content

H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/buptyqx/H-OPD
Area: Multimodal VLM
Keywords: Knowledge Distillation, Multimodal Reasoning, On-Policy Distillation, Heterogeneous Multi-Teacher, Confidence-Aware Arbitration

TL;DR

Targeting the limitations of static instance-level teacher assignment in multimodal on-policy distillation, H-OPD introduces offline vision-to-language description transfer alongside an entropy-based confidence-aware token-level teacher arbitration mechanism, dynamically combining vision-language and text-only teachers across student rollout trajectories to achieve superior reasoning accuracy and training efficiency over GRPO and single-teacher OPD across 11 benchmarks.

Background & Motivation

Complex multimodal reasoningโ€”spanning mathematical geometry, scientific chart comprehension, and visual logic puzzlesโ€”represents a pivotal frontier in post-training vision-language models (MLLMs). Existing post-training recipes predominantly follow two paths: supervised fine-tuning (SFT) and reinforcement learning (RL, notably Group Relative Policy Optimization as popularized by DeepSeek-R1). SFT relies strictly on static target distributions, rendering models susceptible to exposure bias when encountering self-generated errors during inference. Conversely, while RL alleviates distributional drift via online policy exploration, it remains constrained by sparse sequence-level outcome rewards, exorbitant sampling overhead, and training instability. Recently, on-policy distillation (OPD) has emerged as an appealing alternative by providing dense token-level distribution supervision directly over student-generated rollout trajectories, combining the training stability of supervised learning with the exploration benefits of online RL.

Nevertheless, current OPD frameworks in multimodal contexts invariably adopt coarse-grained teacher allocation schemes. They either establish separate teacher-student pipelines for disjoint modalities or enforce static instance-level routing, where an entire sample is rigidly assigned to a single teacher (e.g., multimodal inputs to a vision-language model and text queries to a language model). This paradigm hinges on the flawed assumption that an entire autoregressive generation sequence is governed by a singular, invariant capability. In reality, multimodal reasoning is a dynamic, multi-phase cognitive trajectory where the requirements for fine-grained perceptual grounding, abstract algebraic deduction, and linguistic formatting alternate continuously across decoding steps. Prior empirical studies reveal that MLLMs often extract accurate perceptual primitives early on, only to misinterpret or overwrite them during downstream multi-step deduction, confirming that teacher reliability fluctuates significantly along the response trajectory.

Furthermore, vision-language (VL) teachers and text-only LLM teachers exhibit asymmetric and complementary strengths. While a VL teacher provides indispensable grounding on raw image features, text-only LLMs often demonstrate superior depth and precision in purely abstract logical chains. Static sample-level routing forces an artificial trade-off between perception and abstract logic. Core idea: shift from static sample-level teacher routing to dynamic token-level teacher arbitration along shared student rollouts, bridging the modality gap via offline vision-to-language description transfer and arbitrating supervisory authority token by token using predictive entropy.

Method

Overall Architecture

The workflow of H-OPD orchestrates two core stages: offline modality bridging and online token-level collaborative distillation. In the offline preprocessing stage, a vision-language teacher transcribes the key visual and geometric properties of each image into a structured, task-aligned textual proxy, filtered via a strong language evaluator (GPT-4.1-mini) to provide the text-only teacher with complete visual context while completely removing serial visual encoding overhead during training. During online distillation, the student model samples rollouts from its own policy using the raw multimodal input. At each autoregressive step, both the VL teacher and the text-only teacher asynchronously evaluate the identical student prefix; their top-\(k\) predictive distributions are unified over a dynamic merged support and adaptively weighted via predictive entropy, guiding the student policy via restricted-support reverse KL divergence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input instance (Question text x, Image I)"] --> B["Vision-to-Language Description Transfer<br/>Offline transcription + GPT-4.1-mini filtering"]
    B --> C["Student On-Policy Rollout<br/>Autoregressive sampling y ~ ฯ€ฮธ(ยท|x, I)"]
    C --> D["Candidate Set Merging & Truncated Inference<br/>Top-k union support ฮฉt = KV โˆช KT with zero-filling"]
    D --> E["Confidence-Aware Token-Level Arbitration<br/>Entropy-based dynamic weight ฮปt fusing target distribution"]
    E --> F["Student Policy Parameter Update<br/>Restricted-support reverse KL divergence optimization"]

Key Designs

1. Vision-to-Language Description Transfer: Offloading Perceptual Grounding into Offline Text Proxies

Directly incorporating a text-only teacher into multimodal distillation is impeded by its inability to digest visual inputs, which causes severe hallucinations if prompted on text questions alone. H-OPD resolves this modality asymmetry by transforming raw image semantics into an explicit textual proxy during offline preprocessing. Using a structured prompt tailored to the visual domain (e.g., exacting coordinate axes, line slopes, and numeric labels for geometry, or verbatim LaTeX code for tables and charts), a multimodal teacher generates an exhaustive description \(d\). This proxy is validated and filtered by GPT-4.1-mini to eliminate inaccurate transcriptions before being concatenated with the original stripped text query: $\(\tilde{x} = x_{txt} \oplus \text{[IMAGE DESCRIPTION]} \oplus d\)$ This decoupling enables the text-only teacher \(\tau_T\) to access critical visual evidence in native token form, allowing it to focus exclusively on multi-step symbolic deduction without introducing serial vision-encoder latency into the online rollout loop.

2. Candidate Set Merging & Truncated Inference: Bounding Multi-Teacher Alignment to O(k) Complexity

Conditioned on the identical student trajectory prefix \(y_{<t}\), the VL teacher and text teacher evaluate the state independently via parallel forward paths: \(q_V^{(t)}(v) = \tau_V(v \mid x, I, y_{<t})\) and \(q_T^{(t)}(v) = \tau_T(v \mid \tilde{x}, y_{<t})\). Because full-vocabulary probability evaluation across multiple heterogeneous models incurs prohibitive computation and admits noisy tail distributions, H-OPD extracts only the top-\(k\) candidate tokens from each teacher, forming a dynamic union support: $\(\Omega_t = K_V^{(t)} \cup K_T^{(t)}, \quad |\Omega_t| \le 2k\)$ Each teacher's distribution is renormalized over its own top-\(k\) tokens and extended to \(\Omega_t\) via zero-filling for tokens outside its respective support: \(\bar{\pi}_j^{(t)}(v) = \bar{q}_j^{(t)}(v)\) for \(v \in K_j^{(t)}\) and \(0\) otherwise (\(j \in \{V, T\}\)). This formulation bounds the per-step loss computation to \(O(k)\) complexity while preserving genuine competition among the high-confidence candidates prioritized by either teacher.

3. Confidence-Aware Token-Level Arbitration: Dynamically Allocating Authority via Predictive Entropy

Because individual tokens along a reasoning sequence demand varying degrees of visual grounding versus abstract symbolic manipulation, static weighting yields compromised supervision. H-OPD measures teacher reliability at each step using the predictive Shannon entropy of the truncated distributions over \(\Omega_t\), establishing the confidence score as the negative entropy \(s_j^{(t)} = -H(\bar{\pi}_j^{(t)})\). A temperature-controlled softmax function converts these confidence scores into an adaptive token-level arbitration coefficient \(\lambda_t\): $\(\lambda_t = \frac{\exp(s_V^{(t)} / \tau_f)}{\exp(s_V^{(t)} / \tau_f) + \exp(s_T^{(t)} / \tau_f)}\)$ where \(\tau_f\) is the fusion temperature. The resulting arbitrated supervisory target distribution is: $\(q_A^{(t)}(v) = \lambda_t \bar{\pi}_V^{(t)}(v) + (1 - \lambda_t) \bar{\pi}_T^{(t)}(v)\)$ When generating visual tokens, the VL teacher displays lower entropy, driving \(\lambda_t \to 1\); during symbolic calculations or equation solving, the text teacher exhibits higher confidence, shifting authority to the language specialist. The student policy is optimized across valid response tokens \(m_t\) via the reverse KL divergence: $\(\mathcal{L}(\theta) = \mathbb{E}_{y \sim \pi_\theta} \left[ \sum_{t=1}^{|y|} m_t D_{\text{KL}}^{\Omega_t}(p_\theta^{(t)} \parallel q_A^{(t)}) \right]\)$

Key Experimental Results

Main Results

Experiments were conducted using the verl framework and vLLM on a single node equipped with 8 NVIDIA B200 180GB GPUs, trained on 55K filtered samples from MMFineReason. Evaluation spans 11 benchmarks across mathematical reasoning, document/chart interpretation, visual logic, and general visual QA. The table below presents the performance of the Qwen3-VL-2B-Instruct student under various post-training paradigms.

Benchmark Category Dataset Student Baseline (2B) GRPO SFT (4B VL) OPD (4B VL) ExOPD (4B VL) H-OPD (4B VL + 4B Text)
Multimodal Math MathVista 61.3 63.6 61.6 64.2 64.4 66.8
MathVision 28.6 30.3 26.8 30.9 31.6 33.5
DynaMath 54.2 55.7 53.9 55.9 56.3 57.9
WeMath 35.8 37.1 35.6 38.9 39.1 39.8
Doc & Chart CharXiv (RQ) 26.8 28.0 26.7 27.9 28.3 32.0
ChartQA 79.1 80.0 79.2 81.6 82.4 83.2
Visual Logic VisuLogic 11.5 13.6 10.5 17.8 18.1 21.1
LogicVista 35.8 40.3 32.4 41.4 39.6 49.4
General VQA OCRBench (/10) 85.8 87.2 82.4 88.2 88.4 88.7
MMStar 58.3 60.7 58.5 61.3 61.2 62.3
HallusionBench 51.4 54.9 51.1 54.0 54.5 55.0
Overall Average Average 48.1 50.1 47.2 51.1 51.3 53.5

Ablation Study

Ablations on LogicVista isolate the mechanism of teacher arbitration against routing baselines (Table A), while sensitivity tests on mathematical benchmarks assess fusion temperature \(\tau_f\) and support truncation \(k\) (Tables B and C).

Table A: Controlled Teacher Arbitration Ablations on LogicVista (Student: Qwen3-VL-2B-Instruct)

Configuration Accuracy (%) \(\Delta\) vs Baseline Description / Insight
Student baseline 35.8 - Un-tuned base instruct model
Uniform two-teacher averaging 31.3 -4.5 Unconditional averaging blurs distinct high-probability modes
Text teacher only + description 26.6 -9.2 Textual proxies cannot replace direct visual grounding
Sample-level teacher routing 42.5 +6.7 Static routing helps but fails to handle intra-trajectory drift
H-OPD (Token-level arbitration) 49.4 +13.6 Dynamic token weighting outperforms sample routing by +6.9%

Tables B & C: Hyperparameter Sensitivity on Fusion Temperature and Support Truncation

Temperature \(\tau_f\) MathVista MathVision Support Size \(k\) MathVista Step Wall-Time
0.1 (sharp selection) 66.1 31.0 4 66.2 62 s
0.5 (optimal balance) 66.8 33.5 8 (default setting) 66.8 92 s
1.0 (smooth weighting) 66.2 32.9 32 67.1 718 s
1.5 (approaching uniform) 66.2 32.3 - - -

Key Findings

  • Token-Level Arbitration is Essential: On LogicVista, uniform teacher averaging degrades performance to 31.3% (below the 35.8% student baseline), proving that naive multi-teacher ensembling generates incoherent supervision. Token-level arbitration achieves 49.4%, outperforming static sample-level routing (42.5%) by 6.9 percentage points.
  • Superior Token & Compute Efficiency: By providing dense, high-utility supervision on on-policy states, H-OPD converges substantially faster than GRPO. On MathVerse-200, reaching 45.0% accuracy requires only 2.45 wall-clock hours (19.6 GPU-hours) with H-OPD compared to 23.75 hours (190.0 GPU-hours) with GRPOโ€”a compute reduction of nearly 90%.
  • Scaling across Teachers and Students: Paired with an RL-enhanced teacher (Qwen3-VL-4B-RL), H-OPD scores 69.9% on MathVista and 51.4% on LogicVista, outpacing single-teacher OPD (68.8% and 42.5%). Scaling the student to Qwen3-VL-4B yields an average score of 61.7%, closely matching the 8B teacher baseline (62.1%).

Highlights & Insights

  • Decoupled Perception and Reasoning Pipeline: Translating visual evidence into structured textual summaries offline enables non-visual language specialists to supervise multimodal reasoning tasks without imposing additional visual forward latency during online training.
  • Predictive Negative Entropy as Dynamic Arbitrator: The entropy-driven weighting mechanism naturally implements functional specialization without explicit rule engineering: empirical heatmaps confirm that perceptual tokens draw predominantly from the VL teacher (\(\lambda_t \to 1\)), while algebraic and symbolic reasoning steps shift authority to the text-only model (\(\lambda_t \to 0\)).
  • Generalizable Heterogeneous Distillation Blueprint: The unified frameworkโ€”comprising top-\(k\) union support truncation, zero-filling, and state-dependent entropy arbitrationโ€”provides an extensible foundation for distilling mixtures of specialized domain teachers (e.g., code, formal theorem provers, and vision specialists) into compact models.

Limitations & Future Work

  • Dependence on Visual Description Quality: The efficacy of the text-only teacher is conditioned on the fidelity of the transferred visual descriptions; omissions, geometric inaccuracies, or spatial ambiguities in the offline descriptions propagate directly into the language teacher's supervision.
  • Vulnerability to Overconfident Calibration Errors: Negative entropy serves as a heuristic for certainty rather than factual correctness; poorly calibrated teachers that emit erroneous tokens with low entropy can deceive the arbitration mechanism.
  • Scale Constraints: Constrained by compute, experiments were evaluated primarily on 2B to 8B dense architectures; evaluating heterogeneous token-level arbitration on larger MoE models and 30B+ foundation models remains open for exploration.
  • vs GRPO (DeepSeek-R1): GRPO relies on sequence-level scalar rewards and group-relative advantages across multiple sampled trajectories, necessitating heavy sampling exploration; H-OPD provides dense token-level distribution matching, dramatically accelerating training convergence.
  • vs Single-Teacher OPD (OPD / ExOPD): Prior OPD methods mandate a single vision-language teacher to oversee the entire trajectory, forcing a compromise between visual perception and complex multi-step reasoning; H-OPD dynamically synergizes heterogeneous specialists.
  • vs Sample-Level Routing: Traditional mixture-of-teachers setups assign each problem instance wholly to one expert; H-OPD dynamically reallocates teacher authority token by token within the same autoregressive trajectory.

Rating

  • Novelty: โญโญโญโญโญ First framework to replace static sample routing with dynamic token-level confidence-aware arbitration for multimodal on-policy distillation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously evaluated across 11 benchmarks, cost-matched compute budgets, extensive ablations, and token-level heatmaps.
  • Writing Quality: โญโญโญโญโญ Lucid problem formulation, solid theoretical motivation, and transparent implementation specifics.
  • Value: โญโญโญโญโญ Offers a practical and token-efficient post-training blueprint for distilling multimodal capabilities into edge-scale MLLMs.