No frontmatter or enclosing backticks will be used. Here is the translated paper note body:

A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretability¶

Conference	arXiv	Code	Area	Keywords
ACL 2025	2502.12052	GitHub	interpretability	NLG Meta-Evaluation, LLM-as-a-Judge, Dual-Perspective, Automatic Benchmark, Ordinal Classification

TL;DR¶

This work proposes a dual-perspective NLG meta-evaluation framework that decomposes traditional human-metric correlation into a global perspective (ordinal classification to judge coarse-grained quality levels) and a local perspective (adjacent pairwise comparison to distinguish fine-grained quality differences). By employing an automatic benchmark construction method, it avoids manual annotation and data contamination. Experiments on 16 LLM evaluators reveal that Qwen-2.5-72B achieves global optimality, while DeepSeek-V3 performs best locally.

Background & Motivation¶

Background: The performance of NLG evaluation metrics (such as BLEU, BERTScore, and LLM-as-a-Judge) is usually measured via "meta-evaluation", which calculates the consistency between metrics and human ratings. A common practice is to average the scores from multiple annotators and then compute Spearman/Pearson correlation coefficients. While LLM-as-a-Judge has been widely adopted in scenarios like AlpacaEval, a reliable meta-evaluation to measure the capabilities of these judges themselves is still lacking.

Limitations of Prior Work: Traditional NLG meta-evaluation suffers from three critical flaws: (1) Unjustified score averaging—the quality gaps between adjacent steps on a Likert scale are uneven; for instance, averages of (2,3,4) and (3,3,3) both yield 3 but represent totally different qualities. Furthermore, aggregated fractional scores (e.g., 4/3) do not truly represent more fine-grained quality levels (human re-evaluation agreement is only 42% vs. 88% for original adjacent levels). (2) Ambiguous metric selection—rankings derived from Spearman and Pearson correlations differ significantly, and there is no theoretical basis for choosing which one to use. (3) Outdated and contaminated datasets—the generation systems in commonly used benchmarks are obsolete and face data leakage risks.

Key Challenge: Traditional meta-evaluations measure metric performance using a single correlation coefficient, which conflates two fundamentally different evaluation capabilities: coarse-grained quality judgment ("is this text good, medium, or bad?") and fine-grained quality distinction ("is A slightly better than B?"). Different application scenarios require different capabilities (e.g., the former is needed for training data selection, while the latter is required for preference optimization), which a single correlation coefficient cannot distinguish.

Goal: (1) Decompose the ambiguous "human-metric correlation" into two interpretable and independent dimensions; (2) design an automatic benchmark construction method that requires no new human annotations and avoids data contamination; (3) comprehensively evaluate the performance differences of 16 mainstream LLMs across the two evaluation capabilities.

Key Insight: Starting from the statistical properties of Likert scales, the authors prove that both score averaging and fractional scores are unreasonable (validated via human re-evaluation experiments), thereby advocating for a new meta-evaluation paradigm. Decomposing the problem into ordinal classification (global) and adjacent comparison (local) serves as a natural and elegant entryway.

Core Idea: Replace the traditional single correlation coefficient with a dual perspective of "global ordinal classification + local adjacent comparison" to provide a more interpretable NLG meta-evaluation.

Method¶

Overall Architecture¶

The input is an NLG evaluation benchmark (incorporating source text, target text, and human ratings), and the output is the performance scores of various evaluation metrics/LLMs from both global and local perspectives. The framework consists of two parallel paths: (1) global perspective—modeling evaluation as an ordinal classification task; and (2) local perspective—modeling evaluation as an adjacent quality comparison task. Each perspective is paired with its corresponding automatic benchmark construction method.

Key Designs¶

Global Perspective—Ordinal Classification Meta-Evaluation:
- Function: Assessing the capability of evaluation metrics to judge coarse-grained text quality levels.
- Mechanism: The original human evaluation scale levels (e.g., 1-5 points) are preserved, and meta-evaluation is modeled as an ordinal classification problem. The Closeness Evaluation Measure (CEM) is employed as the evaluation metric. CEM does not assume equal steps between adjacent categories (aligning with the nature of Likert scales), and penalizes misclassification to adjacent categories less than misclassification to distant ones. Only target texts with consensus among annotators are retained to avoid noise from score divergence.
- Design Motivation: This directly addresses the two core problems of unreasonable score averaging and invalid fractional scores. Ordinal classification naturally handles unequal category intervals.
Local Perspective—Adjacent Comparison Meta-Evaluation:
- Function: Evaluating the ability of evaluation metrics to distinguish fine-grained quality differences.
- Mechanism: For each source text, a quality-decreasing target sequence \(t_{i1}, t_{i2}, \cdots, t_{ik}\) is constructed, and the evaluation metric is required to correctly distinguish the quality relationship between adjacent target pairs. Performance is measured by adjacent comparison accuracy: \(\frac{1}{n(k-1)}\sum_{i=1}^{n}\sum_{1 \leq j < k}\mathbb{1}(x_{ij} < x_{i,j+1})\). The number of targets in a sequence can significantly exceed the number of original scale levels (e.g., a 10+ target sequence can be constructed for a 5-point scale), enabling a much finer-grained capability assessment than the original scale.
- Design Motivation: To complement the blind spot of the global perspective—which does not reward fine-grained discrimination within the same category—and fill this gap through the local perspective.
Automatic Benchmark Construction—Controllable Error Injection:
- Function: Scaffolding the automatic generation of evaluation benchmarks required for both perspectives without new human annotations.
- Mechanism:
  - Global Benchmark: Different numbers of errors are simultaneously injected into high-quality reference texts (implemented using OpenAI o1), with each error corresponding to a randomly selected sub-aspect of evaluation. An "anchoring method" is used to estimate the quality levels of candidate targets—consensual samples between human and strong LLM ratings from original benchmarks are selected as "anchors", and candidate targets are compared pairwise with anchors of each level to determine their grades.
  - Local Benchmark: Starting from the reference text, one error is iteratively injected in each step without altering other contents. Cumulative errors guarantee a monotonic decrease in quality. The number of iterations is customizable, enabling quality sequences of arbitrary granularity.
- Design Motivation: Avoid the high cost of new human annotation, mitigate the risk of data leakage associated with existing benchmarks, and leverage the vulnerability of error injection to precisely construct the desired quality distribution.

Loss & Training¶

This work does not involve model training as it is an evaluation framework. The core evaluation formulas are the CEM metric for the global perspective and the adjacent comparison accuracy for the local perspective.

Key Experimental Results¶

Main Results—Global Perspective (CEM Metric)¶

LLM	SummEval Avg	Topical-Chat Avg	Overall
Qwen-2.5-72B	0.830	0.908	0.869 (1)
CompassJudger-32B	0.855	0.869	0.862 (2)
Themis-8B	0.845	0.835	0.840 (3)
Phi-4-14B	0.752	0.882	0.817 (4)
GPT-4o	0.744	0.870	0.807 (5)
GPT-4 Turbo	0.724	0.865	0.795 (7)
Auto-J-13B	0.610	0.617	0.613 (16)

Main Results—Local Perspective (Adjacent Comparison Accuracy)¶

LLM	SummEval Avg	Topical-Chat Avg	Overall
DeepSeek-V3	0.662	0.728	0.695 (1)
GPT-4o	0.669	0.719	0.694 (2)
GPT-4 Turbo	0.673	0.705	0.689 (3)
GPT-4o mini	0.641	0.727	0.684 (4)
Qwen-2.5-72B	0.657	0.691	0.674 (5)
Themis-8B	0.355	0.479	0.417 (16)

Ablation Study—Scoring vs. Direct Comparison¶

LLM	Scoring then Comparing	Direct Pairwise Comparison	Winner
GPT-4o	0.669	0.401	Scoring
GPT-4 Turbo	0.673	0.533	Scoring
DeepSeek-V3	0.662	0.654	Scoring (marginal)
Prometheus-2-8x7B	0.575	0.694	Direct comparison

Key Findings¶

Distinct rankings across the two capabilities: Qwen-2.5-72B, which is globally optimal, ranks 5th locally, whereas DeepSeek-V3, which is locally optimal, ranks only 10th globally. This demonstrates the necessity of the dual-perspective decomposition.
Saturation of global capability in smaller models: Phi-4-14B (14B parameters) and GPT-4o mini exhibit near-GPT-4o performance on the global perspective, suggesting that the coarse-grained quality judgment task is not highly sensitive to model scale.
Individual scoring outperforms direct comparison: Under fine-grained variations (I(1)), the strategy of individual scoring then comparing for general LLMs generally outperforms direct pairwise comparison, overturning conclusions from prior work. Only specifically fine-tuned evaluation models perform better on direct comparison.
GPT-4o behaves strictly in scoring: Confusion matrices indicate that GPT-4o tends to assign lower scores to medium-quality texts, potentially because its higher linguistic capability sets a loftier standard.
The optimal interval exists for scoring ranges: Extending the scoring scale beyond 1-10 yields no further improvement, and the optimal range appears model-dependent.

Highlights & Insights¶

Persuasiveness of human re-evaluation experiments: The human re-annotation experiments on SummEval prove that "fractional scores generated via score averaging are unreliable" (with agreement at only 42%). This simple yet compelling experimental design stands as the crucial support for motivating this study.
Level estimation via anchoring: Borrowing the concept of anchoring from psychometrics, known reliable samples are used as a reference frame to estimate new samples' quality levels, mitigating the unreliability of direct scoring by LLMs.
Simplicity of iterative error injection: The local benchmark construction method (injecting one error at a time) provides a theoretical guarantee of monotonically decreasing quality while allowing arbitrary granularity control, rendering the design exceptionally clean.
Direct guidance for LLM-as-a-Judge practices: It provides clear guidance for practitioners—if the scenario involves filtering high-quality data, focus on global capability (favoring Qwen-2.5-72B); if the goal is preference data annotation, focus on local capability (favoring DeepSeek-V3).

Limitations & Future Work¶

Error injection relies on LLM quality: Benchmark construction employs OpenAI o1 and GPT-4o, meaning the quality of the benchmark is bound by the capabilities of these models. Furthermore, the distribution of injected errors does not necessarily mirror the quality deterioration patterns in real-world scenarios.
Limited coverage of NLG tasks: The experiments are validated only on text summarization (SummEval) and dialogue generation (Topical-Chat), without extension to other typical NLG tasks such as machine translation or story generation.
Sub-aspect decomposition relies on manual auditing: Although o1 generates candidate sub-aspects, human selection and refinement are still required, falling short of full automation.
A monotonic quality decrease is assumed in the local perspective: While cumulative error injection guarantees a monotonically decreasing quality theoretically, real-world errors across multiple dimensions might interact, complicating their actual quality relationship.
Known LLM-as-a-Judge flaws like positional bias are unconsidered: The framework prioritizes capability decomposition, leaving unanalyzed whether these ability evaluations are susceptible to biases like position or length.

vs. Perrella et al. (2024) Redefining MT Meta-Evaluation: They redefine meta-evaluation as binary classification and re-ranking in machine translation, emphasizing interpretability. This work conducts similar efforts across broader NLG tasks but from a distinct angle—focusing on the decomposition of evaluation capabilities rather than interpreting metric scales.
vs. Wang et al. (2024) Perturbation Attack: They employ perturbation to quantify LLM evaluator biases (e.g., positional bias). This work also leverages perturbations (error injection) but for a different purpose—to construct quality-controlled evaluation benchmarks, rather than to analyze bias.
vs. Kim et al. (2024) Prometheus-2: A model specifically fine-tuned for evaluation, performing better than general LLMs on the local perspective (direct comparison mode) of this work, but falling behind Qwen-2.5-72B on the global perspective. This highlights that the effectiveness of evaluation fine-tuning depends on the application scenario.
This framework can be extended to multimodal evaluation (e.g., MLLM-as-a-Judge), representing a promising direction for future research.

Rating¶

Novelty: ⭐⭐⭐⭐ The idea of dual-perspective decomposition is creative, though ordinal classification and pairwise comparisons are not novel in other fields.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely thorough, encompassing 16 LLMs, 2 NLG tasks, and multi-dimensional comparative analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Clear structure with rigorous motivational arguments supported by human re-evaluation experiments.
Value: ⭐⭐⭐⭐ Offers direct guidelines for NLG evaluation and LLM-as-a-Judge practices.