Skip to content

๐Ÿ“Š LLM Evaluation

๐Ÿ’ฌ EMNLP2026 ยท 29 paper notes

๐Ÿ“Œ Same area in other venues: ๐Ÿง  NeurIPS2026 (83) ยท ๐ŸŽž๏ธ ECCV2026 (3) ยท ๐Ÿ”ฌ ICLR2026 (131) ยท ๐Ÿ’ฌ ACL2026 (96) ยท ๐Ÿงช ICML2026 (40) ยท ๐Ÿค– AAAI2026 (16)

๐Ÿ”ฅ Top topics: LLM ร—13 ยท Dialogue ร—2

Auditing MCQA Benchmarks through Probability Landscapes

Proposes a two-level MCQA benchmark auditing framework using model output probability distributions, characterizing macro benchmark structure via top prediction probability and normalized residual entropy summarized by Mean Pairwise Distance, and filtering distractor competition via noise injection to effectively prioritize flawed items for targeted human review.

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

Tsinghua University and collaborating institutions introduce LongJudgeBench, the first meta-evaluation benchmark tailored for LLM-as-a-judge on long-form outputs averaging 9,249.7 tokens across five realistic scenarios, exposing a severe reliability gap, position biases, and superficial coverage illusions in current judges.

Check The Scoreboard: An Analysis of Scoring Schemes on Multiple Choice Evaluation

By systematically adapting six educational measurement scoring schemes to multiple-choice QA, this paper penalizes guessing, differentiates model capabilities beyond standard accuracy, and demonstrates that Answer Until Correct scoring strongly predicts real-world human preferences in LLM Arena.

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

Addressing systemic scoring biases in LLM-as-a-judge evaluation that cannot be averaged away by increasing test scenarios, this work establishes a generalizability theory variance decomposition and demonstrates that a round-robin judge assignment strategy (CyclicJudge) cancels panel-relative judge offsets and maximizes generation diversity at zero additional cost, cutting top-1 ranking flips by 61%.

docfollow a hierarchical fine grained constraints following benchmark in long co

Addressing the critical gap in evaluating Long-In-Long-Out text generation and the unreliability of LLM-as-a-judge scoring, DocFollow introduces a benchmark of 2,436 expert-validated instances across 5 domains and 20 subdomains with hierarchical constraints and a deterministic rule-based evaluation protocol.

Does task decomposition improve automatic NLG evaluation?

A systematic benchmark comparing decomposition-based and direct prediction LLM-as-a-Judge frameworks reveals that task decomposition yields no intrinsic performance gains, that historical advantages were artifacts of human label supervision and continuous float tie-breaking, and that direct prediction reaches human-level consensus when fairly calibrated.

Evaluating and Improving LLM Self-Modeling

This paper operationalizes LLM self-modeling as an empirical behavioral prediction capability over verifiable input-output distributions, introduces an unbiased skill benchmark spanning 10 task formats across four domains, and demonstrates that multi-task RL on closed-loop synthetic perturbation data reliably enhances self-modeling skill without requiring privileged internal introspection.

Evaluating LLM-generated Explanatory Dialogue Turns through Dialogue Completion

This paper uses dialogue completion as a dual lens across human-spoken, human-written, and synthetic modalities to investigate explanatory dialogue quality and metric behavior, uncovering a systematic written-text bias in the widely used reference-free FED metric that penalizes spoken interactivity, while showing that metric-driven zero-shot self-refinement reliably closes the model interactivity gap.

Exposรญa: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback

This paper introduces Exposรญa, the first publicly available multi-stage dataset linking student research project proposal writing and peer review feedback in higher education, paired with fine-grained pedagogically grounded rubrics across 36 proposal criteria and 24 review criteria, and benchmarks leading open-weight and closed-source LLMs on automated multi-aspect scoring.

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

RULERS introduces a criteria-transfer framework for frozen LLM judges that locks human rubrics into structured task specifications, extracts mechanically verifiable textual evidence via typed operators, and calibrates model signals through monotone quantile distribution mapping, substantially outperforming direct scoring baselines in Quadratic Weighted Kappa (QWK) across four diverse benchmarks.

From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing

Addressing the fragility and bias of conventional single-shot supervision in LLM routing, this paper proposes DARS (Distribution-Aware Routing Supervision), which models query-level capability distributions via semantics-preserving rewrites and repeated stochastic decoding to construct a risk-aware utility that markedly boosts router effectiveness across architectures.

From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

This paper introduces ESTBOOK, a formalized pedagogical diagnostic benchmark spanning 10,576 multimodal questions across five major English standardized exams (SAT, GRE, GMAT, TOEFL, IELTS), which models test-solving as structured cognitive trajectories to pinpoint LLMs' "Integration Bottleneck" and validates lightweight, diagnostic-guided cognitive scaffolds that achieve dramatic performance gains.

Gold vs. Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

This paper introduces an incremental proxy validity framework with nested regression controls to test whether machine-translated MMLU benchmarks validly proxy native gold benchmarks across five Nordic and Baltic languages, proving that proxy validity follows a content-overlap gradient rather than simple ranking agreement.

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

This paper introduces Granuscore, the first reference-free measure of semantic granularity that leverages the radial depth and anchor manifold distances in a hyperbolic hierarchical embedding space, demonstrating that LLMs underperform on question answering partly due to pathological over-granular guessing when uncertain.

How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

A novel register-aware linguistic evaluation framework is introduced based on 67 Biber lexico-grammatical features and Maximum Mean Discrepancy (MMD) two-sample testing, revealing that prominent open-source LLMs deviate systematically from the human baseline across registers, and human-likeness depends heavily on communicative register rather than model parameter scale.

Is my model perplexed for the right reason? Contrasting LLMsโ€™ Benchmark Behavior with Token-Level Perplexity

This paper introduces a principled interpretability framework grounded in token-level perplexity (PPL) across minimal sentence pairs, demonstrating that even when large language models achieve near-perfect benchmark accuracy, the linguistically crucial tokens driving sentence validity account for less than half of the perplexity shift, exposing a severe disconnect between behavioral accuracy and underlying linguistic competence.

JuDi: Revisiting Judge Decoding from First Principles via Training-Free Distributional Divergence

Uncovering that token criticality learned by costly supervised judges is fundamentally encoded in draft-target distributional divergence, JuDi establishes a training-free verification mechanism via token-level KL divergence with high-confidence fallback, matching or outperforming learned judges while eliminating supervision bottlenecks.

LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertainty

This paper demonstrates that LLM abstention on True/False questions can be heavily inflated (dropping accuracy by ~20 percentage points) simply by adding an extra "Unknown" optionโ€”a prompt artifact induced by supervised instruction tuning where later network layers override intact mid-layer reasoning representations.

Measuring Source-Resolution Distortion in LLM Output Arbitration: A Source-Bounded Evaluation Framework

Addressing the irreversible loss of critical source structure caused by information compression when LLMs arbitrate among multiple candidate answers, this paper formalizes a source-bounded evaluation benchmark and introduces A2R, a decision-ledger-based arbitration method that markedly cuts conflict erasure and boosts provenance recoverability while maintaining competitive preference.

Multi-Domain Latent Concepts Evaluation: Auditing Interpretability and Robustness

To overcome the prohibitive cost of frontier judges and the brittleness of single-pass small models against subtle jailbreaks, this paper introduces a multi-agent debate framework combining refusal screening, dynamic 5-aspect value alignment, and bounded critic-defender deliberation, alongside FSโ€“HAJailBench (an 11,100-instance human-annotated jailbreak benchmark), achieving near-GPT-4o agreement with a 54% reduction in inference cost.

Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue

This paper proposes the first formal multi-expert Conformal Risk Control (CRC) framework for pairwise LLM-as-a-Judge evaluation in open-ended dialogue, introducing Marginal-Calibrated Conformal Consensus (MC3) to bridge calibration and testing via a unified decision function, provably bounding error risk below target thresholds while significantly boosting coverage.

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Auditing

PaperAudit-Bench presents the first document-level benchmark for scientific error detection, comprising 220 top-tier AI papers with 2,200+ controlled synthetic flaws and a tiered auditing framework that disciplines AI peer reviewers into grounded, critical evaluators.

Post-hoc Alignment of LLM-judges to Human Judgment Distribution

Addressing the severe misalignment of LLM-judges when predicting unaggregated soft-labels despite human-level hard-label accuracy, this paper proposes NAPHA (eNtropy-Aware Post-Hoc Alignment), which stratifies instances by prediction Shannon entropy and routes them to specialized lightweight MLPs to accurately match human judgment distributions with minimal training overhead.

Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

In a controlled double-blind study of 162 participants interacting with six LLMs across two model families, pre-interaction capability framing persistently biased subjective ratings and altered prompting styles, while post-interaction impression change was driven exclusively by met expectations and self-efficacy gains rather than objective task performance.

Self-Calibrating Language Models via Test-Time Discriminative Distillation

This paper introduces SECL, the first test-time training pipeline for language model confidence calibration that leverages the model's inherently better-calibrated discriminative probabilities as a self-supervision signal, triggering bounded directional LoRA updates via adaptive entropy gating to reduce Expected Calibration Error by 56%โ€“78% without ground-truth labels while preserving task accuracy.

The Evaluation Scaling Law: Toward More Reliable LLM Assessment via Multi-Prompt Diversity

Addressing the vulnerability of LLM evaluation to template-induced formatting and phrasing fluctuations, this paper establishes an Evaluation Scaling Law showing that aggregating diverse prompts systematically reduces measurement noise and improves ranking monotonicity in a log-linear fashion, enabled by an automated Quality-Diversity (QD) prompt selection algorithm.

The Evaluation Scaling Law: Toward More Reliable LLM Assessment via Multi-Prompt Diversity

To resolve substantial prompt sensitivity and ranking volatility in standard LLM benchmarking, this paper formulates the Evaluation Scaling Law showing that evaluation noise decreases smoothly while ranking monotonicity scales log-linearly with prompt template diversity, operationalized via a novel Quality-Diversity prompt selection strategy.

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

This paper formally deconstructs the self-bias in automated LLM benchmarking into compounding test set generation and evaluator biases, proves that low-proficiency linguistic collapse drives model-specific degeneration artifacts, and demonstrates that within-model diversity filtering effectively mitigates benchmark distortions.

When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR

This paper uncovers a pervasive failure mode across 15 modern VLMsโ€”termed "over-correction"โ€”where models spontaneously repair student mistakes during multi-line handwritten math OCR, and introduces PINK, a rubric-based semantic metric with stepped penalties that restores transcription faithfulness and inverts BLEU-based model rankings with 55.0% human expert preference.