Skip to content

📊 LLM Evaluation

🧠 NeurIPS2026 · 83 paper notes

📌 Same area in other venues: 🎞️ ECCV2026 (3) · 🔬 ICLR2026 (131) · 💬 ACL2026 (96) · 🧪 ICML2026 (40) · 🤖 AAAI2026 (16) · 🧠 NeurIPS2025 (38)

🔥 Top topics: LLM ×32 · Reasoning ×6 · Adversarial Robustness ×3 · Personalized Generation ×2 · Layout & Composition ×2

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

Addressing the lack of internal logits and the failure of traditional confidence metrics in open-ended question answering, this paper presents Sem-ECE, a semantic-sampling calibration evaluation framework, along with a held-out estimator Sem2 that eliminates the winner's curse and provides an observable diagnostic for question difficulty.

A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation

A unified influence-based perturbation framework for Bradley-Terry leaderboards formalizes match additions, drops, flips, and player removals with one-step Newton corrections, enabling quantitative stability audits and targeted manipulation across Top-\(k\) ranks, Kendall's \(\tau\), and confidence intervals.

AI Evaluation Should Require Standardized Item-Level Data Releases

Addressing the crisis where aggregate benchmark scores conceal construct misalignment and item degradation, this paper proposes standardizing item-level data releases as essential AI evaluation infrastructure and introduces OpenEval, archiving 10.7M+ responses across 444 models.

AI GAMESTORE: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games

To overcome the rapid saturation and narrow scope of static AI benchmarks, this paper proposes evaluating machine general intelligence across the open-ended "Multiverse of Human Games" via AI GAMESTORE—a platform that sources, synthesizes, and adapts commercial games using LLMs with humans in the loop, showing that frontier VLMs achieve less than 10% of median human performance on 100 games.

Aligning Language Model Benchmarks with Pairwise Preferences

Addressing the predictive validity gap where static benchmarks fail to predict downstream utility, this paper introduces benchmark alignment and the BenchAlign diagnostic tool, showing that learning-to-rank item weights enables small-model evaluations to zero-shot rank unseen 70B+ models according to target preferences.

APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings

The paper proposes the Arbitrary Preference Mapping (APM) benchmark, which decouples semantic priors and cancels LLM judge biases via a randomized zero-mean mapping between user prompt traits and response principles, demonstrating that routing-based adaptation is the most reliable approach for implicit style personalization.

Ask-E: An Environment for Calibrated Question Generation

Presents the Ask-E evaluation and training environment, where an asker LM interacts with two boundary solver models to generate a question solvable by exactly one of them, inverting the conventional need for stronger supervisors and improving downstream math reasoning through RL with no new data or correctness rewards.

AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents

AssayBench unifies 1,920 human CRISPR screens into a free-text conditioned phenotypic endpoint gene ranking benchmark and introduces Adjusted nDCG (AnDCG) to correct for assay-specific baselines, demonstrating that while frontier generalist LLMs outperform specialized biology models, a substantial gap remains relative to empirical performance ceilings.

BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments

Addressing the intertwined challenges of human annotation disagreement and systematic LLM judge biases in pairwise evaluation, BayesJudge introduces an online streaming Bayesian meta-evaluation layer that uses tie-open labels and paired order swaps to disentangle item ambiguity from evaluator biases, producing full posterior verdict distributions and calibrated uncertainty.

BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems

Introduces Black-Box Optimization Word Problems (BBOWP) and the BBOWP-Bench suite, evaluating how well LLMs acting as meta-solvers can infer design variable search spaces and select suitable optimization algorithms directly from natural-language task descriptions.

Benchmark for Assessing Olfactory Perception of Large Language Models

The authors introduce the Olfactory Perception (OP) benchmark comprising 1,010 objective questions across eight task categories, revealing through dual prompting (compound names vs. isomeric SMILES) that current frontier LLMs rely predominantly on lexical recall rather than genuine molecular structure reasoning.

Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of Large Language Models

Proposes the Benchmark Health Index (BHI), a meta-evaluation framework auditing LLM benchmarks across Capability Discrimination, Anti-Saturation, and Impact, using Leave-One-Benchmark-Out calibration and CRITIC objective weighting to diagnose benchmark obsolescence and ecological misalignment.

Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

To address the severe generalization drop of LLM safety guard models on unforeseen out-of-distribution (OOD) alignment failures, this paper introduces the MOOD benchmark covering seven distinct OOD failure modes and demonstrates that linearly combining guard models with Mahalanobis distance and perplexity-based OOD detectors significantly improves misalignment recall under strict false positive rate constraints.

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

Introduces BenchHub, a composition-aware framework that treats LLM benchmarks as distributions over multi-dimensional instance-level attributes across 54 datasets (839K questions in 10 languages), showing that aggregate scores mask critical category variations and ranking shifts.

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

Addressing ceiling and floor saturation in fixed benchmarks as well as the uncomparable nature of worst-case adversarial attacks, Dynamic Boundary Evaluation (DBE) integrates the 1PL Rasch model with Skill-Guided Boundary Search (SGBS) to actively locate a model's \(p \approx 0.5\) decision boundary, establishing and adaptively extending a globally comparable difficulty scale.

BioXArena: Benchmarking LLM Agents on Multi-modal Biomedical Machine Learning Tasks

Addressing the gap where existing biomedical AI benchmarks emphasize question answering or narrow pipelines without evaluating end-to-end predictive modeling, BioXArena introduces a standardized benchmark comprising 76 end-to-end multi-modal machine learning tasks across 9 biomedical domains, systematically evaluating 18 agent-backbone configurations under controlled sandbox environments.

BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

To tackle high environment isolation overhead and exploration instability in mutable database debugging, BIRD-RL proposes a four-stage progressive curriculum and trajectory-scoped persistent execution infrastructure, cutting online I/O by 12.8x and rivaling frontier proprietary agents with open-source SLMs.

Calibrating LLMs with Semantic-Level Reward

Addressing the issues of overconfident guessing in RLVR and brittle surface-token dependency in verbalized confidence methods, CSR calibrates language models directly in semantic space using pairwise semantic consistency across sampled rollouts as a training reward, optimized via curriculum learning and GRPO without any verbalized confidence interface.

Can We Trust Item Response Theory for AI Evaluation?

This paper investigates the fundamental regime mismatch between psychometric item response theory (IRT) and LLM benchmarking through 18,000 simulation runs across six benchmarks, demonstrating that while aggregate scores remain stable, reliable item diagnostics strictly require \(N \ge 100\) models and model ranking recovery is governed primarily by item difficulty coverage.

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

To resolve the fundamental misalignment between surface-level semantic clustering and latent capability demands in LLMs, this paper proposes Evidence-Calibrated Clustering (ECC), which integrates posterior pairwise model comparisons to calibrate prior embeddings and performs probe-informed mixture inference for accurate query-specific capability ranking.

CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning

CAREBench establishes the first benchmark grounded in cognitive appraisal theory that captures complete inferential chains—spanning situations, open-ended appraisal reasoning, multi-dimensional rating vectors, and multi-label emotions—from both first- and third-person perspectives, revealing that LLMs' strong downstream emotion label prediction masks severe deficits in appraisal reasoning and subjective heterogeneity modeling.

Celeus: Certifiable and Efficient LLM Evaluation via e-Processes

Addressing the issue where sequential testing and adaptive early stopping violate nominal coverage guarantees (double dipping), Celeus introduces an anytime-valid confidence sequence framework based on e-processes combined with active uncertainty-guided sampling and lightweight surrogate scoring, preserving nominal coverage while reducing evaluated samples by 54%–62%.

Conditional Evaluation of Language Models with Cheap Auxiliary Signals

Addressing the high cost of gold labels and the bias of cheap proxy signals (e.g., LLM-as-a-judge, confidence scores) in fine-grained conditional evaluation of large language models, this paper proposes LACE (Local Augmented Control-Variate Evaluation), which applies local centering within profiling covariate neighborhoods and regularized control-variate augmentation to achieve calibration-free, asymptotically unbiased, and variance-optimal performance profile estimation.

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

CL-Bench introduces the first expert-validated continual learning benchmark across six diverse real-world domains, employing an explicit stateful-versus-stateless gain metric to rigorously isolate genuine online experience accumulation from static base model capabilities.

Corrective Diffusion Language Models

Standard masked diffusion language models (MDLMs) fail to refine errors because unsupervised visible tokens receive overconfident estimates; this paper proposes Corrective Diffusion Language Models (CDLM) combining absorbing masks and uniform replacement to induce error-aware confidence, verified on a new executable Code Revision Benchmark (CRB).

Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well

Addressing the validation crisis in machine learning caused by limited test sample sizes, this paper introduces the "variance-equivalent test sample gain" to quantify the virtual test augmentation effect of cross-validation, proving that multi-split CV yields up to 10–15× variance reductions while providing an early-stopping redundancy diagnostic based on the first few folds.

CruxBench: A Benchmark of Information Discovery

Introduces CruxBench, the first open-ended benchmark evaluating LLMs' capability in "information discovery" by measuring the Value of Information (VOI) of model-generated subquestions (cruxes) against real-world belief updates in prediction markets.

diffscore text evaluation beyond autoregressive likelihood

DiffScore models natural language generation evaluation as masked reconstruction using Masked Large Diffusion Language Models (MDLLMs), eliminating autoregressive positional bias and providing multi-timestep quality profiles alongside bidirectional PMI decomposition.

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking

DiscoverPhysics is an interactive discovery benchmark featuring 22 simulated worlds with non-canonical physics (screened forces, fractional Laplacians, dark matter, and extra dimensions), evaluating LLMs through multi-round active experimentation, trajectory MSE, and expert-rubric-judged conceptual explanations.

Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection

Addressing the failure of the conventional "Top-k by accuracy" heuristic in LLM ensembling due to strong error correlations among models, this paper establishes an information-theoretic saturation error floor via a Gaussian-copula framework and proposes a Greedy Mutual Information (Greedy MI) subset selection algorithm that consistently outperforms strong accuracy-based baselines under identical query budgets.

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability — Diverse Evaluation Is Required

This paper challenges the prevailing practice of treating gains on narrow benchmarks like SWE-bench as evidence of general coding capability, demonstrating via a controlled Django benchmark suite that post-trained models fail to transfer across core software engineering modalities.

EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies

Presents EcoGym, an open-source benchmark for long-horizon plan-and-execute decision making across three interactive economic environments (Vending, Freelance, Operation) over an unbounded horizon, uncovering systematic disconnects between high-level macro strategies and fine-grained action execution across frontier LLMs.

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

This paper reframes efficient LLM benchmarking as a classic two-stage problem of statistical feature selection and multiple regression, using minimum redundancy maximum relevance (mRMR) to extract informative question subsets and polynomial kernel ridge regression to model non-linear score interactions, substantially cutting prediction error and boosting ranking fidelity.

Efficient Evaluation of LLM Performance with Statistical Guarantees

This paper introduces Factorized Active Querying (FAQ) and Pro-Active Inference (PAI), combining historical Bayesian factor modeling with hybrid variance-reduction and active-learning sampling to achieve valid frequentist confidence intervals for LLM benchmark accuracy using up to 5× fewer queries.

Evaluation Awareness in Language Models Has Limited Effect on Behaviour

Through paired causal interventions and natural distributional analyses across 8 frontier open-weight reasoning models and 4 diverse benchmarks, this work demonstrates that verbalised evaluation awareness in chain-of-thought reasoning has a negligible causal effect on downstream model behavior (\(\omega \le 0.12\)), refuting the common assumption that evaluation awareness directly implies evaluation gaming or alignment faking.

Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

To address the fragmentation and lack of contextual metadata in AI evaluation reporting, this paper introduces Evaluation Cards, an operational reporting framework and pipeline that unifies benchmark, run, and model metadata through a five-level rollout hierarchy and four interpretive signals, accompanied by an empirical audit of 5,816 models revealing widespread reproducibility and comparability gaps.

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

This paper reinterprets delta-rule linear attention as an explicit Euler discretization of a zero-order-hold continuous-time ODE and leverages the rank-1 algebraic structure of the dynamics matrix to derive Exact Flow Linear Attention (EFLA), a parameter-free exact closed-form flow that eliminates discretization error while strictly preserving chunkwise parallelism and linear time complexity.

Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth

This paper constructs BonaFide, the first benchmark with step- and trajectory-level ground-truth faithfulness labels, and conducts a systematic meta-evaluation of eight prominent faithfulness metrics, revealing that most perform near random chance, exhibit extreme prediction skews, and incur prohibitive computational cost.

FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence

Addressing the lack of localized verifiable evidence in natural-language proof evaluation and the dual challenges of locality and semantic drift in formal theorem provers, this paper proposes FaithSieve, a framework combining proof-state tree decomposition, typed obligation extraction, Lean verification, and semantic alignment gating for accurate first-error localization in mathematical proofs.

Feedback Forensics: A Toolkit to Measure AI Personality

Feedback Forensics introduces an open-source Python toolkit and web platform to explicitly quantify and audit AI personality traits in pairwise model responses, leveraging LLM annotators with a relevance-weighted chance-corrected agreement metric (Strength) to uncover hidden biases in RLHF benchmarks and detect model personality drift.

FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis

FindStatBench introduces an execution benchmark constructed from FindStat with 2,329 closed-book tasks and 5.52M hidden instances, revealing that frontier open- and closed-source models tightly converge while few-shot examples can unexpectedly derail canonical mathematical bijection implementations.

Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models

Introduces Flame, an automated benchmark generation framework that leverages multi-agent cooperation and solution-graph-driven problem construction grounded in authoritative textbooks to synthesize comprehensive, fine-grained, and contamination-resistant multiple-choice evaluation benchmarks.

FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting

This paper introduces FinReasoning, a hierarchical evaluation benchmark designed for reliable financial research reporting that decomposes analyst workflows into semantic consistency, data alignment, and deep insight across 4,800 expert-verified samples, uncovering pronounced capability stratification across model architectures.

Forecasting Downstream Performance of LLMs With Proxy Metrics

The paper constructs a library of 80 token-level proxy metrics from a single forward pass of candidate language models over expert solution trajectories, reliably ranking heterogeneous reasoning models, selecting pretraining data at 10,000x less compute, and halving downstream training extrapolation error.

From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

Addressing the severe Elo scale distortion and bloated uncertainty intervals caused by collapsing continuous LLM-judge score differences into ternary win/loss labels, this paper introduces Soft-Elo—a framework that propagates temperature-calibrated soft probabilities into Bradley-Terry estimation and wraps ratings with split conformal prediction intervals, cutting Elo MAE to 17.9 without requiring human annotations.

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

Addressing the gap where existing benchmarks only assess toy-scale modeling and direct solver calls, FrontierOR introduces 180 literature-grounded large-scale optimization tasks with up to \(10^7\) variables and constraints, evaluating frontier LLMs and self-evolving agents on feasibility, solution quality, and quality-time efficiency against verified Gurobi baselines.

Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence

Heterogeneous Judge-Aware (HJA) ranking canonically decomposes multi-judge pairwise comparison score matrices into a shared consensus direction, judge-specific consensus sensitivity, and an orthogonal low-rank residual disagreement component, enabling identifiability, geometric alternating estimation, and calibrated asymptotic confidence intervals under fixed judge panels.

Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

Addressing how standard LLM evaluations ignore pipeline variance across judge models, prompt phrasings, and temperatures—causing severe confidence interval under-coverage and leaderboard gaming—this paper introduces the Total Evaluation Error (TEE) framework grounded in generalizability theory to decompose multi-source variance and guide decision study (D-study) allocations, halving estimation error at equal cost and cutting the benchmark gaming surface by 43%.

How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Evolution

This paper introduces RPEval, a benchmark measuring rational preference utilization in LLM personalized assistants, and develops RP-Reasoner, a pragmatic Bayesian reasoning framework that models query formulation to eliminate irrelevant memory interference, improving multi-preference intent accuracy by 258% and fixing 80% of commercial assistant failures.

How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustness

This paper formalizes benchmark manipulation as shift bribery in social choice theory, proves that benchmark-specific training is NP-hard under mean win rate and Borda count, and demonstrates that mean win rate requires manipulating significantly more tasks (78% on MMLU, 92% on BBH) than arithmetic mean to rig a leaderboard.

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

Addressing the budget-constrained evaluation bottleneck with multiple heterogeneous, unknown-reliability LLM judges, this paper establishes the instance-optimal adaptive algorithm Est-IVWE via optimistic variance regularization and proves matching local minimax lower bounds.

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

JudgmentBench introduces a benchmark of 30 real-world legal tasks with paired rubric and pairwise preference annotations from 51 experienced practicing attorneys, demonstrating that comparative judgments recover constructed quality orderings far more reliably than rubrics (Spearman's rank correlation 0.908 vs. 0.150) in less than half the time.

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

Presents JuICE, a multilingual multicultural benchmark featuring 7,470 span-level cultural and linguistic error annotations across four countries and languages validated by native speakers, demonstrating that frontier LLM-judges severely struggle to detect deep, situated "thick" cultural errors (achieving at most 0.52 F1 and <50% recall on human-detected errors).

L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education

Oxford University Press and partners introduce L2-Bench, the first open-source benchmark for second language learning experience design, combining a 12-competency taxonomy, 1,000 diverse tasks parameterized by 33 context factors, and three-layer binary rubrics to rigorously measure LLM pedagogical capabilities.

Language Model Goal Selection Differs from Humans in a Self-Directed Learning Task

Evaluating five leading LLMs against humans on an open-ended cognitive learning task reveals that models diverge fundamentally from human goal selection, often succumbing to extreme single-goal reward hacking or low-performing exploration traps without capturing human behavioral diversity.

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

ListQA introduces a benchmark of 9,045 cross-validated, evergreen questions grounded in Wikipedia tables to assess multi-element factual recall and structured list formatting, framing evaluation as a linear sum assignment problem paired with LLM grading to prove that frontier models fail to exceed 37% F1 and that parametric recall is the primary bottleneck.

LLM Judge Validation Under Sparse Overlap: From Inference to Design

This paper treats the quantity and allocation of overlapping annotations for LLM-judge validation as a statistical design problem: variance analysis guides overlap planning, while stratified allocation improves representativeness; low overlap substantially changes deployment decisions and judge rankings on real benchmarks, but stratification can increase false approval even as it reduces false rejection.

LLMs Show No Signs Of Individuated Metacognition

Decomposing binary confidence judgments from 20 frontier LLMs across six benchmarks via tetrachoric factor analysis and pairwise calibration demonstrates that stated confidence reflects a shared item-difficulty heuristic rather than model-specific individuated self-knowledge.

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Introduces MLS-Bench, an atomic benchmark spanning 140 tasks across 12 machine learning domains to evaluate whether AI agents can invent generalizable, scalable algorithms rather than just tune code, revealing substantial gaps in frontier models' scientific discovery and experimental judgment.

Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness

Introduces ProofRank, the first benchmark evaluating LLM mathematical proof quality beyond binary correctness through five scalable proxies—conciseness, computational ease, cognitive simplicity, diversity, and adaptivity—paired with Bradley-Terry ranking, revealing pronounced trade-offs between accuracy, verbosity, and conceptual elegance.

OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

Addressing the inability of traditional endpoint QA to inspect latent mental-state representations, OmniToM introduces an explicit belief-structure modeling benchmark spanning 895 stories and 22,343 seven-dimensional labeled belief propositions, exposing a severe information-tracking bottleneck when LLMs convert narrative facts into actor-specific beliefs.

OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models

OTROPE uses optimal transport in semantic embedding space to reweight human-labeled residuals from an older model and correct a newer model's mean proxy evaluation, reducing cross-model preference-win-rate estimation error without response likelihoods, but its effectiveness depends on representation coverage and residual transferability, and it does not outperform PPI under every shift.

PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models

PCEval introduces the first fully automated physical computing benchmark decomposing evaluation into 4 tasks across logical circuits, physical breadboard layouts, and bidirectional code generation, revealing that LLMs struggle severely with physical wiring mechanics and pin conflicts.

PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?

This paper introduces PhysElite, a large-scale bilingual multimodal benchmark comprising 11,586 Olympiad-tier physics problems paired with visual diagrams, step-by-step bilingual derivations, and open-ended answers; evaluation across 18 frontier models reveals that even the top system achieves only 33.7% answer accuracy, exposing critical bottlenecks in physical perception and physical law formulation.

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

Addressing the limitation of standard ARC evaluation which only tests single test-grid output generation, PotARCin introduces a programmatic generator- and verifier-backed five-dimensional evaluation protocol (Definition, Classification, Constrained Generation, Editing, and Inversion), revealing a 25–52 percentage-point performance plunge in frontier models and exposing widespread Potemkin understanding where models contradict their own correct rules.

Provable Joint Decontamination for Benchmarking Multiple Large Language Models

To eliminate benchmark data contamination across multiple audited LLMs without yielding fragmented model-specific subsets, this paper proposes Joint Envelope Conformal Selection (JECS), a provable conformal framework that aggregates per-model evidence via max-p statistics and reconstructs a conservative null envelope to rigorously control the Global Contamination Rate (GCR) while drastically recovering selection power.

Real In, Real Out: What If We Only Use Real Data for Scene Text Editing?

Addressing the synthetic-to-real domain gap and visual artifacts caused by heavy reliance on synthetic training pairs in scene text editing (STE), TextRIRO introduces a self-paired training paradigm using only real data via character-column shuffling and cropping (Shuffle&Crop), achieving state-of-the-art realistic text editing and delivering a 3-million-scale dataset (TextRIRO-3M) that markedly boosts downstream text recognition.

RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

Constructing RxnOptBench from 552 real wet-lab condition-screening tables published in 2025 flagship chemistry journals, this paper benchmarks 12 frontier and chemistry-specialized LLMs, uncovering substantial performance headroom and severe combinatorial brittleness when models transition from single-factor to joint full-condition selection.

SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?

Addressing the struggle of leading LLMs in anticipating human next-utterance behavior during multimodal dialogues, this paper introduces SayNext-Bench with a comprehensive multi-level evaluation suite and proposes SayNext-Chat, a cognitively inspired dual-route MLLM featuring learnable priming tokens.

SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

SCDBench introduces the first standardized benchmark for LLM-based smart contract decompilers, containing 600 real-world Ethereum contracts and 227k semantic test cases evaluated across a four-stage cumulative pipeline (format, compilation, ABI recovery, and semantic replay), revealing that even frontier models achieve perfect semantic consistency on only 7% of contracts.

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

SciR generates verifiable deduction, induction, and causal-discovery problems before rewriting their premises as multi-document scientific discourse, separately varying inference complexity and premise obfuscation to diagnose LLMs; document formalisation remains a substantial bottleneck even with symbolic solvers.

Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning

OracleLadder fixes teacher-written mathematical sub-goals and crosses isolated milestone tests with increasing roadmap and answer assistance on the parent problem: across six models, 33–48% of a screened 354-problem NuminaMath set falls into a composition gap, where all milestones are solvable but the parent remains unrecovered; the share is 24–37% after excluding grading issues flagged by automated review, but this is not a pure causal measurement of an internal composition ability.

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

SpanUQ distills claim consistency across offline sampled responses into a probe over frozen LLM hidden states, using set prediction to locate semantic spans and assign continuous uncertainty; five separately trained backbone probes achieve 0.908–0.944 AUROC and support finer-grained risk filtering than rejecting entire responses.

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

This paper introduces Skill Entropy, a directed pairwise metric quantifying the cognitive difficulty of switching between reasoning skills, establishes Skill2-Bench spanning 558 skills across 9 domains, and develops Skill-Entropy RL to supervise skill planning and transitions via an entropy-aligned reward.

Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

Constructed from 2.12 million authentic interaction logs of 200 representative users over three months on the Kuaishou platform, OmniBehavior establishes the first user simulation benchmark spanning long horizons, multiple business scenarios, and heterogeneous actions, systematically benchmarking 11 state-of-the-art LLMs and uncovering their intrinsic "positivity-and-average" structural bias.

Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking

Addressing the "winner's curse" and optimistic evaluation bias in adaptive prompt and program search, SIREN introduces a rigorous reporting protocol that freezes candidate shortlists, decouples splitwise selection from held-out evaluation, and deploys an item-level Gaussian multiplier bootstrap to perform unbiased estimation and simultaneous confidence interval inference over finite tuning budgets.

Training Optimal Large Diffusion Language Models (Quokka Scaling Law)

Triage: Evaluating Prospective Metacognitive Control in Large Language Models

Introduces TRIAGE, an evaluation framework that measures prospective metacognitive control (selection, sequencing, and per-problem token allocation) in LLMs under finite budgets, finding that extended reasoning improves accuracy without improving triage quality, and reasoning models exhibit severe degradation in recognizing unsolvable tasks.

TSNBench: Benchmarking LLM Proficiency in Time-Sensitive Networking

Presents TSNBench, the first comprehensive benchmark evaluating LLM proficiency in Time-Sensitive Networking (TSN), comprising 939 expert-validated MCQs and 200 open-ended Worst-Case Delay (WCD) computation tasks backed by verified Network Calculus solvers and closed-form equations, exposing a stark capability divergence between declarative terminology and deterministic numerical reasoning.

VEHBench: A Stage-Local Diagnostic Benchmark for LLM-Assisted Vibration Energy Harvester Design

Addressing tightly coupled physical constraints in battery-free IoT cantilever vibration energy harvester (VEH) design, VEHBench introduces an engineering-native diagnostic benchmark comprising 763 literature-grounded tasks evaluated by a deterministic analytical physical oracle, demonstrating that LLM capabilities across specification triage, verifier-guided search, corrupted-state recovery, and policy selection are heavily stage-dependent.

When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice

Introduces GDB, a benchmark evaluating "goal displacement" where conversational LLMs substitute persona-maintenance warmth for actionable guidance, establishing an operationalized deletion-test rubric and revealing that preference optimization (DPO) rather than SFT suppresses this failure mode.

XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

Presents XL-SafetyBench, a dual-track cross-cultural safety benchmark spanning 10 country-language pairs with 5,500 test cases, decoupling adversarial jailbreak robustness from culturally embedded sensitivity recognition to expose capability fractures across 37 global and local LLMs.

ProactBench: Beyond What The User Asked For

ProactBench introduces the first benchmark to decompose conversational proactivity into three phase-tied trigger types (Emergent inference, Critical synthesis, and post-completion Recovery) across an asymmetric three-agent architecture, uncovering that post-task Recovery is severely decorrelated from existing benchmarks and reveals a cross-model policy bottleneck.