Skip to content

RIGEL: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Image and Video Captioning Evaluation, LLM-as-a-Judge, Self-Distilled Scoring Head, Ordinal Score Adaptation, Vid-Lepus

TL;DR

Addressing the vocabulary-label mismatch where standard LM heads allocate probability mass over \(\sim 10^5\) tokens instead of a small ordinal set (\(1\sim 5\)), RIGEL introduces a two-phase self-distilled score adaptation framework with an EMD-distilled scoring head and human-guided backbone adaptation, outperforming prior metrics across extensive image and video captioning benchmarks.

Background & Motivation

Automatic evaluation of image and video captioning serves as an indispensable pillar for benchmarking multimodal foundation models. Traditional evaluation metrics, such as BLEU, ROUGE, METEOR, CIDEr, and SPICE, fundamentally rely on surface-level n-gram overlap or graph-based syntactic matching. Consequently, they correlate poorly with human judgments whenever generated candidate captions are semantically accurate yet lexically diverse. While recent data-driven metrics built upon multimodal embeddings (e.g., Polos, PAC-S++) improve correlation, they remain constrained to coarse holistic similarities rather than fine-grained descriptive, relevant, and fluent dimensions.

The emergence of multimodal large language models (MLLMs) has ushered in the LLM-as-a-Judge paradigm (exemplified by FLEUR and G-VEval), offering unprecedented qualitative interpretability and alignment. However, standard LLM-based evaluation metrics suffer from a fundamental architectural mismatch. Language modeling (LM) heads are optimized for autoregressive next-token prediction across vast vocabularies (\(|V|\sim 10^5\)), whereas caption quality assessment is an ordinal judgment task over a constrained, discrete scale (\(\mathcal{M}=\{1,2,3,4,5\}\), where \(|\mathcal{M}|\ll |V|\)). Empirical inspections reveal that standard LM heads allocate significant logit mass to non-score tokens, which act as task-irrelevant noise and degrade evaluation robustness. Furthermore, for video captioning, existing evaluation benchmarks provide test suites but entirely lack supervised training sets paired with multi-dimensional human judgments.

To resolve this vocabulary-label discrepancy while overcoming the training data deficit, the authors decouple the evaluation process from next-token language modeling. Rather than predicting scores within a noisy vocabulary space, the internal representation of the LLM is leveraged to distill an evaluation-specific ordinal head, which is subsequently adapted to human judgment data. Core idea: Propose RIGEL, a self-distilled score adaptation metric that first distills a lightweight ordinal scoring head from a frozen LLM using Earth Mover's Distance (EMD) to filter vocabulary noise, and second adapts the LLM backbone to human judgments via LoRA; paired with Vid-Lepus, a new benchmark training set of 14,802 human judgments for video caption evaluation.

Method

Overall Architecture

The input to RIGEL is formulated as a triplet \((X_{\text{vis}}, X_{\text{ref}}, x_{\text{cand}})\), where \(X_{\text{vis}} \in \mathbb{R}^{T\times 3\times H\times W}\) represents visual inputs across \(T\) frames (\(T=1\) for still images), \(X_{\text{ref}} = \{x_{\text{ref}}^{(i)}\}_{i=1}^N\) denotes a set of \(N\) reference captions (\(X_{\text{ref}} = \emptyset\) in reference-free settings), and \(x_{\text{cand}}\) is the candidate caption to be evaluated. RIGEL executes across two distinct training phases: Phase (i) Denoised Head Self-Distillation, which trains an evaluation-specific scoring head on the frozen backbone, and Phase (ii) Human-Guided Score Adaptation, which tunes the multimodal backbone with human scores while keeping the scoring head frozen. At inference, RIGEL predicts continuous scores in a single forward pass via expectation-based score smoothing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>(Visual frames Xvis, References Xref, Candidate xcand)"] --> B["Frozen VLM Backbone extracts hidden representation h"]
    B --> C["Phase 1: Denoised Head Self-Distillation<br/>Freeze backbone; distill 5-score ordinal head g_theta via EMD loss"]
    C --> D["Phase 2: Human-Guided Score Adaptation<br/>Freeze scoring head; fine-tune backbone via LoRA on human gold labels"]
    D --> E["Single-Pass Expectation Smoothing<br/>Compute continuous expectation score y_hat normalized to range [0, 1]"]

Key Designs

1. Denoised Head Self-Distillation: Eliminating vocabulary noise while preserving ordinal manifold structure To address the vulnerability of standard LM heads where logit mass disperses across non-score tokens in \(|V|\), Phase 1 replaces the entire language head with a compact two-layer MLP scoring head \(g_\theta: \mathbb{R}^d \to \mathbb{R}^{|\mathcal{M}|}\) (2.63M parameters) mapped directly onto \(\mathcal{M}=\{1,2,3,4,5\}\). Given multimodal input processed by the frozen LLM backbone, the final-layer hidden state \(h \in \mathbb{R}^d\) is routed into \(g_\theta\) to generate denoised logits \(\hat{z} = g_\theta(h)\). Concurrently, from the frozen LM head's vocabulary logits, score logits \(z_{\text{score}} \in \mathbb{R}^{|\mathcal{M}|}\) are extracted and scaled by temperature \(\tau\) to yield a soft pseudo-label distribution: $\(q = \text{softmax}(z_{\text{score}} / \tau)\)$ Because scores \(1\sim 5\) possess an intrinsic ordinal distance structure, standard KL divergence fails to penalize errors proportionally to their metric distance along the ordinal axis. Therefore, Phase 1 employs the one-dimensional Earth Mover's Distance (EMD) as the self-distillation objective: $\(\mathcal{L}_{\text{P1}} = \tau \cdot \text{EMD}\left(q, \text{softmax}(\hat{z} / \tau)\right)\)$ where for two discrete probability distributions \(a\) and \(b\) over \(K\) ordered bins, the 1D EMD is defined as: $\(\text{EMD}(a, b) = \sum_{i=1}^{K-1} \left| \sum_{j=1}^i a_j - \sum_{j=1}^i b_j \right|\)$ This transfers the implicit evaluation capability from the expansive vocabulary into an isolated, clean ordinal prediction space.

2. Human-Guided Score Adaptation: Re-aligning backbone representations under fixed ordinal geometry While Phase 1 creates a noise-free ordinal projection head, its initial knowledge remains bounded by the teacher model's pre-existing biases and uncalibrated perceptual judgments. Phase 2 adapts the multimodal backbone to ground-truth human judgments under the frozen calibrated scoring head \(g_{\theta^*}\). Freezing \(\theta^*\) establishes a rigid geometric coordinate system for ordinal scoring, preventing representation drift and training instability. Low-Rank Adaptation (LoRA, rank \(r=8\), \(\alpha=16\), with only 1.61M trainable parameters) is applied strictly to the backbone attention blocks. Given human gold labels \(y_{\text{gold}} \in \{0, 1\}^{|\mathcal{M}|}\) derived from averaged human judgments, Phase 2 minimizes cross-entropy loss: $\(\mathcal{L}_{\text{P2}} = - \sum_{k=1}^{|\mathcal{M}|} y_{\text{gold}, k} \log p_k, \quad \text{where } p_k = \left[\text{softmax}(g_{\theta^*}(h))\right]_k\)$ The backpropagated gradients update only the LoRA weights, steering the latent multimodal feature \(h\) to align with human evaluation criteria.

3. Expectation-Based Score Smoothing: Continuous scalar evaluation via single forward pass At test time, RIGEL bypasses autoregressive token sampling and chain-of-thought generation entirely. Input samples undergo a single forward pass through the adapted backbone and frozen scoring head to produce the probability vector \(p = \text{softmax}(g_{\theta^*}(h))\). Rather than taking a discrete argmax, the final scalar judgment score \(\hat{y}\) is computed via expectation smoothing: $\(\hat{y} = \sum_{k=1}^{|\mathcal{M}|} k \cdot p_k\)$ The continuous score \(\hat{y}\) is subsequently linearly normalized to the \([0, 1]\) interval. This expectation formulation provides continuous, sub-integer sensitivity, mitigating boundary discretization artifacts while mirroring fine-grained human ranking granularity.

4. Vid-Lepus Dataset: Supervised human-aligned dataset for video captioning evaluation To resolve the chronic absence of human-annotated training data in video captioning evaluation, the authors construct Vid-Lepus. Sourced from the validation split of VATEX, Vid-Lepus comprises 3,338 video clips, 33,380 reference captions (10 fixed references per clip), and 5,637 candidate captions generated across varied models. A cohort of 241 crowdworkers annotated candidate captions on a 5-point Likert scale across three orthogonal dimensions: descriptiveness, relevance, and fluency, generating 14,802 human judgments. Stringent quality filtering removed rapid or unvarying submissions, and samples exhibiting a rating spread \(\ge 3\) underwent expert arbitration. Vid-Lepus achieves an inter-annotator agreement of Krippendorff's \(\alpha = 0.45\), markedly exceeding prior benchmarks (CapEval1K at 0.37 and Spica at 0.44).

Loss & Training

Phase 1 trains the scoring head over 50 epochs using the Adam optimizer (\(\beta_1=0.9, \beta_2=0.999\)), learning rate \(5 \times 10^{-4}\), batch size 64, and weight decay 0.01, with early stopping triggered if validation Kendall's \(\tau_c\) does not improve for 5 consecutive epochs. Phase 2 tunes the backbone LoRA for 1 epoch on image and video data using Adam with learning rate \(1 \times 10^{-4}\), batch size 8, and weight decay 0.01. Training is executed on eight NVIDIA H200 SXM GPUs (141 GB VRAM) in approximately 11.6 hours. At inference, single-sample evaluation on a single H200 GPU requires only \(\sim 610\) ms and 770 GFLOPs.

Key Experimental Results

Main Results

RIGEL was evaluated across image captioning benchmarks (Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, FOIL) and video captioning benchmarks (VATEX-EVAL, ActivityNet-Fact, YouCook2-Fact, ActivityNet-FOIL) in both reference-based and reference-free settings.

Quantitative Comparison on Image Captioning Benchmarks

Setting Metric Composite (\(\tau_b / \tau_c\)) Flickr8K-EX (\(\tau_b / \tau_c\)) Flickr8K-CF (\(\tau_b / \tau_c\)) Nebula (\(\tau_b / \tau_c\)) FOIL (1-ref Acc %)
Ref-based BLEU 28.3 / 30.6 30.6 / 30.8 16.4 / 8.7 46.5 / 44.1 66.5
Ref-based CIDEr 34.9 / 37.7 43.6 / 43.9 24.6 / 12.7 51.5 / 48.8 82.5
Ref-based RefPAC-S++ 54.7 / 59.1 55.3 / 55.7 37.9 / 19.6 53.3 / 50.6 93.5
Ref-based Pearl 55.8 / 60.4 58.2 / 58.6 38.6 / 20.0 58.4 / 55.4 96.5
Ref-based G-VEval (GPT-4o) - / - 60.5 / 58.7 38.2 / 19.9 - / - 97.8
Ref-based RIGEL (Ours) 61.2 / 66.1 58.6 / 59.0 40.4 / 20.9 60.6 / 57.5 99.1
Ref-free PAC-S++ 53.9 / 58.3 54.1 / 54.5 37.0 / 19.1 50.5 / 47.9 90.2
Ref-free FLEUR - / 63.5 - / 53.0 38.6 / - - / - 96.8
Ref-free DISCODE (InternVL) 60.5 / 64.9 57.7 / 58.1 40.1 / 20.8 - / - 98.2
Ref-free RIGEL (Ours) 59.5 / 64.3 59.5 / 59.9 40.4 / 20.9 58.1 / 55.1 98.7

Quantitative Comparison on Video Captioning Benchmarks

Setting Metric VATEX 1-ref (\(\tau_b / \rho\)) VATEX 9-ref (\(\tau_b / \rho\)) ActNet-Fact Para (\(r\)) ActNet-Fact Sent (\(r\)) YouCook2 Para (\(r\)) ActNet-FOIL Acc (%)
Ref-based EMScore 28.6 / 37.1 36.8 / 47.2 42.7 35.2 54.3 92.4
Ref-based RefPAC-S++ 32.2 / 41.5 39.8 / 50.8 - - - 93.4
Ref-based FactVC - / - - / - 55.1 46.5 60.6 -
Ref-based G-VEval 44.9 / - 48.1 / - 56.6 48.1 72.4 -
Ref-based RIGEL (Ours) 45.6 / 57.8 50.8 / 64.0 67.8 54.0 65.4 97.1
Ref-free PAC-S++ 28.1 / 36.4 28.1 / 36.4 - - - 91.0
Ref-free FactVC - / - - / - 46.2 37.1 40.8 -
Ref-free G-VEval 39.4 / - 39.4 / - 48.8 38.6 55.8 -
Ref-free RIGEL (Ours) 42.1 / 53.7 42.1 / 53.7 63.6 51.9 55.0 96.2

Ablation Study

The ablation investigations systematically disentangle the contributions of Phase 1, Phase 2, the scoring head architecture, the backbone model, and the distillation objective in the reference-free regime.

Component and Phase Ablation (Reference-free)

Config Phase 1 Phase 2 Head Architecture Backbone Model Composite (\(\tau_b / \tau_c\)) Flickr8K-EX (\(\tau_b / \tau_c\)) Nebula (\(\tau_b / \tau_c\)) VATEX-Eval (\(\tau_b / \rho\))
(i) โœ— โœ“ Standard LM Head Qwen3-VL-2B 58.3 / 63.0 59.0 / 59.4 57.4 / 54.4 39.7 / 50.8
(ii) โœ“ โœ— Scoring Head Qwen3-VL-2B 53.1 / 57.4 22.3 / 22.6 52.2 / 55.1 43.2 / 54.3
(iii) โœ— (Random Init) โœ“ Scoring Head Qwen3-VL-2B 58.0 / 62.8 58.8 / 59.2 56.3 / 53.4 35.4 / 45.4
(iv) โœ“ โœ“ Scoring Head Qwen2.5-VL-3B 59.4 / 64.2 59.2 / 59.7 57.3 / 54.3 39.8 / 50.8
(v) Full RIGEL โœ“ โœ“ Scoring Head Qwen3-VL-2B 59.5 / 64.3 59.5 / 59.9 58.1 / 55.1 42.1 / 53.7

Phase 1 Objective Ablation (KL Divergence vs. EMD)

Distillation Loss Composite (\(\tau_b / \tau_c\)) Flickr8K-EX (\(\tau_b / \tau_c\)) Nebula (\(\tau_b / \tau_c\)) VATEX-Eval (\(\tau_b / \rho\))
KL Divergence 59.0 / 63.8 59.3 / 59.7 57.7 / 54.7 41.9 / 53.4
EMD (Earth Mover's Distance) 59.5 / 64.3 59.5 / 59.9 58.1 / 55.1 42.1 / 53.7

Key Findings

  • Crucial synergy of both training phases: Comparing Config (i) with (v) demonstrates that a task-specific scoring head consistently outperforms standard LM heads even under identical Phase 2 tuning (+1.3 points on Composite \(\tau_c\), +2.4 points on VATEX \(\tau_b\)). Comparing Config (iii) with (v) reveals that omitting Phase 1 self-distillation severely impairs video performance (VATEX-Eval \(\tau_b\) collapses from 42.1 to 35.4), proving that pre-training priors must be properly distilled before human adaptation. Conversely, omitting Phase 2 (Config ii) leads to severe performance drops on image benchmarks (Flickr8K-EX \(\tau_b\) collapses by 37.2 points).
  • Superiority of ordinal geometry with EMD: Incorporating Earth Mover's Distance in Phase 1 outperforms standard KL divergence across all evaluation datasets (+0.5 on Composite \(\tau_b/\tau_c\), +0.4 on Nebula \(\tau_b/\tau_c\)). EMD penalizes probability discrepancies relative to their geometric distance along the ordinal score axis, directly harmonizing with expectation-based inference.
  • Controlled backbone comparison confirms architectural advantage: Under identical Qwen3-VL-2B backbone and data environments, RIGEL scores 64.3 on Composite \(\tau_c\) vs. 53.1 for G-VEval and 60.3 for FLEUR, proving that performance gains stem from the self-distillation paradigm rather than model scale or prompt tricks.

Highlights & Insights

  • Formulating vocabulary mismatch as an SNR challenge: The paper cogently identifies the tension between \(10^5\)-token vocabulary modeling and 5-point ordinal evaluation, demonstrating that non-score logits contribute substantial noise to judgment calibration.
  • Harmonized ordinal training and continuous inference: By aligning EMD loss in Phase 1 with expectation smoothing at test time, the method ensures theoretical self-consistency across discrete training and continuous evaluation.
  • Cost-effective single-pass evaluation: With only 2.63M head parameters and 1.61M LoRA parameters, RIGEL scores samples in 610 ms via a single forward pass, providing a practical alternative to expensive autoregressive CoT evaluation.

Limitations & Future Work

  • Dependence on teacher pseudo-label quality: Phase 1 distillation relies on soft pseudo-labels extracted from the uncalibrated frozen LM head; errors or inherent biases present in the base model's initial score token logits may be transferred to the head.
  • Requires white-box model weights: Computing the evaluation score necessitates access to internal hidden representations \(h\), restricting the framework from being applied to closed-source commercial APIs (e.g., GPT-4o, Claude 3.5 Sonnet).
  • Insensitivity to fine-grained scene text errors: Qualitative error categorization indicates that RIGEL occasionally overestimates captions with subtle OCR discrepancies (e.g., scoring a caption that misread "crepes" as "cremes" at 0.78 when human annotators scored it 0.42).
  • vs G-VEval / FLEUR: While conventional LLM judges prompt MLLMs to generate verbal explanations or parse LM head token probabilities, they incur heavy inference latency and suffer from vocabulary-induced logit dispersion. RIGEL decouples scoring into a dedicated ordinal head with single-pass expectation inference, achieving both higher efficiency and superior correlation.
  • vs Polos / PAC-S++ / Pearl: Prior supervised metrics rely on dual-stream contrastive CLIP encoders that lack generative spatial-temporal reasoning. RIGEL harnesses generative MLLM backbones and refines them via self-distillation and human preference tuning, delivering state-of-the-art alignment across image and video modalities.

Rating

  • Novelty: โญโญโญโญ [Insightful formulation of the LM vocabulary mismatch in LLM-as-a-Judge, resolved via ordinal self-distillation]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive validation across 9 benchmarks, same-backbone, same-data, and objective ablations with detailed error taxonomy]
  • Writing Quality: โญโญโญโญโญ [Clear mathematical formulation, fluent prose narrative, and well-structured tables]
  • Value: โญโญโญโญโญ [Delivers an efficient, robust, open-source evaluation standard and a much-needed supervised video caption dataset]