Skip to content

Tampered ML: Evaluating LLM Judgment of Machine Learning Research

Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1536/
Code: https://github.com/tamperedml/tamperedml
Area: LLM Reasoning
Keywords: Automated scientific discovery, Methodological soundness, Counterfactual tampering, Position bias, Test-time compute

TL;DR

This paper introduces TAMPERED ML, a benchmark comprising 1,858 pairwise experiment proposals from top-tier ML papers paired with minimally edited counterfactuals harboring targeted methodological flaws; empirical results show that frontier reasoning models reach only 77.9% both-orders accuracy, exhibit substantial presentation-order vulnerability, and consistently mistake descriptive transparency and conventional practice for methodological rigor.

Background & Motivation

As automated AI research systems and autonomous agents increasingly handle end-to-end scientific workflows—ranging from paper literature synthesis and code implementation to running training sweeps and compiling experimental logs—ensuring that research proceeds reliably becomes an acute safety and oversight challenge. In empirical machine learning and AI alignment, the pivotal obstacle is not the mechanical execution of scripts or the reporting of metric gains on standard benchmarks. Instead, it lies in high-level scientific judgment: determining which hypotheses warrant investigation, how to model potential failure modes, and whether an empirical setup genuinely provides valid, confound-free causal evidence for its stated claims. When autonomous systems cannot reliably critique proposed experiments before committing substantial computational and human resources, the entire research cycle is vulnerable to generating specious conclusions supported by invalid methodologies.

Evaluating whether language models possess sound scientific discernment is inherently difficult under standard experimental setups. Directly comparing two independently written research proposals introduces numerous confounding variables—such as disparate tasks, datasets, tuning budgets, baselines, and prose quality—making it virtually impossible to establish uncontroversial, objective ground-truth labels. Conversely, evaluating models by asking them to audit entire published papers frequently gets sidetracked by trivial typographic defects, stylistic quirks, or long-context retrieval noise rather than the core validity of the experiment. Furthermore, existing research agent benchmarks predominantly measure code-writing and engineering execution rather than critical methodological discrimination.

To circumvent these limitations while retaining genuine scientific complexity, this paper proposes a controlled counterfactual tampering methodology. The authors reconstruct faithful experimental proposals from oral and spotlight papers published at top-tier venues (NeurIPS, ICML, and ICLR) and inject a single, targeted methodological flaw into each while preserving virtually all other textual and contextual elements. Core idea: construct minimally edited counterfactual proposal pairs—contrasting faithful published experiments against versions with a single injected validity flaw—to convert open-ended scientific critique into a controlled pairwise judgment task that directly measures whether models discern genuine methodological soundness.

Method

Overall Architecture

The TAMPERED ML pipeline consists of four serial stages designed to ensure rigorous variable control: source paper selection and structured fact extraction, two-stage faithfulness auditing and prose proposal reconstruction, targeted flaw planning and minimal editing, and five multi-faceted admission gates coupled with difficulty filtering. Each resulting benchmark item is a quadruple \((C, O, T, f)\) comprising the research context \(C\) (phenomenon, hypothesis, and research question), a faithful original proposal \(O\), a tampered proposal \(T\), and a held-out ground-truth flaw description \(f\). Models are presented with \((C, O, T)\) across both possible presentation orders (\(O\)-first and \(O\)-second) and must consistently identify the more methodologically sound design to receive credit.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Top-Tier Conference Papers (NeurIPS/ICML/ICLR)"] --> B["Paper Suitability Screening & Fact Extraction<br/>Sonnet 4.6 6-criterion audit + Opus 4.6 extracts E_raw"]
    B --> C["Faithfulness Verification & Proposal Rewriting<br/>Majority-vote G1,a/G1,b/G1,sc ensure faithful self-containment"]
    C --> D["Targeted Flaw Planning & Minimal Editing<br/>Plan flaw across 8 categories; median text edit only 2.0%"]
    D --> E["Admission Gating & Difficulty Filtering<br/>G2-G5 gates reject leaks, equivalence, or surface artifacts"]
    E --> F["Bilingual Order-Permuted Pairwise Evaluation<br/>Strict both-orders accuracy penalizes position bias + Fluke Test"]

Key Designs

1. Paper Suitability Screening & Fact Extraction: Isolating Load-Bearing Empirical Setups To prevent hallucinated assumptions and conflation across multiple experiments within a single manuscript, the pipeline sources papers exclusively from NeurIPS, ICML, and ICLR (2024–2026) oral and spotlight tracks. Sonnet 4.6 scores papers on six explicit criteria (clarity of phenomenon, testable hypothesis, specification of baseline and metrics, external researcher accessibility, non-trivial design, and anonymity preservation), requiring an 8/12 score for entry. Next, Opus 4.6 extracts a structured fact set \(E_{\text{raw}}\) recording conditions, datasets, procedures, tuning protocols, metrics, and decision rules, strictly marking any unstated variables as explicitly unspecified to eliminate downstream confabulation.

2. Faithfulness Verification & Proposal Rewriting: Ensuring Ground-Truth Validity of the Original Design To eliminate semantic drift between the raw academic text and the narrative proposal, the pipeline implements three distinct verification gates. Gate \(G_{1,a}\) deploys multiple independent LLM checkers via majority vote to verify that \(E_{\text{raw}}\) strictly reflects the source paper. Upon rewriting \(E_{\text{raw}}\) into an autonomous proposal \(O\), Gate \(G_{1,b}\) audits \(O\) against \(E_{\text{raw}}\). Candidates failing a gate receive feedback and are granted a single repair attempt before discard. Finally, Gate \(G_{1,sc}\) verifies that proposal \(O\) is fully understandable and internally consistent as a standalone experimental blueprint.

3. Targeted Flaw Planning & Minimal Editing: Disentangling Methodology from Stylistic Confounders Directly rewriting an experiment often introduces subtle lexical, length, or stylistic artifacts that models can exploit as heuristic shortcuts. TAMPERED ML decouples this by first planning a specific methodological flaw \(f\) from a taxonomy of eight failure categories (confounded comparisons, blocked mechanisms, measurement mismatches, unfair comparisons, degenerate setups, unmet theoretical prerequisites, circular evaluations, and data leakage). The generator then applies this plan as a surgical edit to produce \(T = O + f\). Across the 1,858 leaderboard items, the median proposal is approximately 1,050 words, while the median edit modifies only 2.0% of the text (and never exceeds 10%), ensuring that differences between \(O\) and \(T\) are strictly methodological.

4. Admission Gating & Difficulty Filtering: Verifying Non-Equivalence and Contextual Self-Sufficiency Candidate pairs must pass four rigorous downstream gates: \(G_2\) (tamper unfaithfulness) verifies that \(T\) genuinely diverges from the source paper rather than accidentally recreating a documented baseline or ablation; \(G_3\) (experimental non-equivalence) confirms that no single software implementation could satisfy both \(O\) and \(T\); \(G_4\) (item self-containedness) guarantees that the flaw is diagnosable purely from the provided context \(C\) and proposal \(T\) without external knowledge; and \(G_5\) (leakage and artifact filtering) strips out meta-commentary, formatting tells, or unnatural phrasing. A difficulty filter subsequently discards items that a panel of base models trivially solves in both presentation orders, isolating a rigorous benchmark.

5. Order-Permuted Pairwise Evaluation & Fluke Audit: Exposing Position Inconsistency and Reasoning Authenticity To counter the pervasive first-token position bias exhibited by autoregressive models, every item is evaluated under both permutations: \(O\)-first (\(O\) before \(T\)) and \(O\)-second (\(T\) before \(O\)). The evaluation protocol defines two core metrics: $\(\text{Acc}_{\text{both-orders}} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[y^{(i)}_{O\text{-first}} \wedge y^{(i)}_{O\text{-second}}\right], \quad \text{Acc}_{\text{per-order}} = \frac{1}{2N}\sum_{i=1}^{N}\left(y^{(i)}_{O\text{-first}} + y^{(i)}_{O\text{-second}}\right)\)$ Random guessing scores 25% on \(\text{Acc}_{\text{both-orders}}\) and 50% on \(\text{Acc}_{\text{per-order}}\). For instances where a model succeeds in only one order, an automated Fluke Test powered by Opus 4.7 (at maximum reasoning effort) audits the reasoning trace against the held-out ground truth \(f\) to determine whether the model identified the actual causal mechanism (\(85.5\%\) exact match across all audited items) or simply guessed favorably.

Benchmark & Evaluation Protocol

  • Evaluation Dataset: 1,858 validated, difficult item-pairs derived from 981 oral and spotlight papers (ICLR: 324, ICML: 145, NeurIPS: 512).
  • Evaluated Models: 19 frontier and representative reasoning systems, including the Claude family (Opus 5, Fable 5, Opus 4.7, Opus 4.6, Sonnet 4.6, Haiku 4.5), OpenAI GPT family (GPT-5.5, GPT-5.4, GPT-5.2), Google Gemini family (Gemini 3.1 Pro, Gemini 3 Flash), and open-weight/thinking architectures (DeepSeek V4 Pro, DeepSeek V3.1, Kimi K2.6, GLM-5.1, Qwen3-235B/30B-Thinking, MiniMax M2.5, Grok 4.3).
  • Human Expert Validation: Post-stratified audit conducted by CS/ML graduate and PhD validators under a label-blind two-stage protocol confirms an estimated benchmark validity rate of 86% (\(95\%\) CI: [81%, 91%]).

Key Experimental Results

Main Results

Pairwise accuracy on the 1,858 TAMPERED ML leaderboard items, highlighting the performance drop between per-order accuracy and strict both-orders accuracy:

Model Configuration Both-Orders Accuracy (\(\text{Acc}_{\text{both-orders}}\)) Per-Order Accuracy (\(\text{Acc}_{\text{per-order}}\)) Order Drop (\(\Delta\)) Gain vs. Random (25%)
Fable 5 (with Opus 5 fallback) 78.4% 87.2% -8.8% +53.4%
Opus 5 (effort: max) 77.9% 86.8% -8.9% +52.9%
GPT-5.5 (reasoning: high) 73.1% 84.6% -11.5% +48.1%
Gemini 3.1 Pro (thinking: high) 70.8% 83.2% -12.4% +45.8%
Opus 4.7 (effort: max) 69.4% 82.1% -12.7% +44.4%
GPT-5.4 (reasoning: xhigh) 68.2% 81.5% -13.3% +43.2%
Kimi K2.6 (default reasoning) 64.9% 78.7% -13.8% +39.9%
Sonnet 4.6 (effort: max) 61.2% 76.5% -15.3% +36.2%
DeepSeek V4 Pro (thinking) 59.5% 75.1% -15.6% +34.5%
Grok 4.3 (default) 58.7% 74.3% -15.6% +33.7%
GLM-5.1 (default reasoning) 57.3% 73.4% -16.1% +32.3%
Qwen3-235B-Thinking 56.4% 72.8% -16.4% +31.4%
GPT-5.2 (reasoning: high) 55.6% 72.1% -16.5% +30.6%
Gemini 3 Flash 53.8% 71.0% -17.2% +28.8%
Opus 4.6 (effort: max) 43.1% 66.2% -23.1% +18.1%
Haiku 4.5 (budget: 60k) 42.5% 64.9% -22.4% +17.5%
Qwen3-30B-Thinking 41.2% 63.8% -22.6% +16.2%
MiniMax M2.5 (default) 39.8% 62.7% -22.9% +14.8%

Ablation & Diagnostic Scaffolding Study

On a challenging subset of 896 items where at least one top model failed both orders, diagnostic prompts test whether errors can be recovered when models are guided toward the methodological flaw:

Scaffolding Configuration Opus 4.7 \(\text{Acc}_{\text{both}}\) Gemini 3.1 Pro \(\text{Acc}_{\text{both}}\) Avg. Gain vs. Baseline Diagnostic Purpose
Baseline (Standard open-ended prompt) 44.5% 42.7% — Autonomous pairwise comparison without hints
+ Flaw Disclosure (Informs that one has a flaw) 57.5% 54.9% +12.6% Reframes task from open critique to defect hunting
+ Category Disclosure (+ specific taxonomy tag) 71.6% 66.3% +25.3% Constrains the conceptual search space
+ Review Decomposition (Tiered peer critiques) 53.7% 50.8% +8.6% Provides structured critiques without revealing tamper status
+ Full Scaffold (Flaw + Category + Reviews) 71.5% 66.2% +25.2% Saturated performance; category hint drives the gain

Key Findings

  • Substantial Frontier Capability Ceiling: Even premier reasoning models like Opus 5 and Fable 5 fail on over 22% of items when held to the rigorous both-orders criterion, indicating that automated scientific discrimination remains far from dependable.
  • Pervasive Position Bias Distorts Deliberation: Models exhibit a strong bias toward the first-presented proposal (\(O\)-first). When the original proposal appears second (\(O\)-second), models require substantially longer thinking traces (geometric mean thinking length ratio up to 3.42× for Opus 5) and report degraded confidence, even on items they ultimately answer correctly in both permutations.
  • Pointwise Absolute Scoring Collapses: When evaluated under isolated 0–100 rubric scoring (evaluating one proposal at a time without pairwise comparison), performance drops precipitously across every model by 4 to 19 percentage points (Opus 5 drops from 86.8% to 68.8%; Opus 4.7 drops to 52.5%), with near-zero item-level rank correlation (\(\text{Spearman } \rho \in [0.17, 0.26]\)) against pairwise accuracy.

Highlights & Insights

  • Surgical Counterfactual Probing of Scientific Judgment: Rather than attempting messy whole-paper evaluations, TAMPERED ML isolates the causal impact of a single methodological choice by perturbing an average of only 2.0% of the proposal text, establishing a clean, interpretable evaluation standard.
  • Substituted Heuristics as a Primary Failure Mode: Qualitative error analysis reveals that models frequently confuse superficial diligence with scientific rigor. They consistently praise tampered proposals that include superfluous parameter disclosures ("specificity as rigor") or adhere to common but invalid community shortcuts ("convention as validity"), ignoring the fatal validity leaks those choices introduce.
  • Fluke Test Validates Single-Order Rationales: The Opus 4.7-based audit reveals that 85.5% of correct single-order choices accurately identify the exact modified element, changed state, and causal mechanism of the injected flaw, demonstrating that position inconsistency stems from context-induced cognitive interference rather than random guessing.

Limitations & Future Work

  • Pipeline Dependence on Proprietary LLMs: The generation, filtering, and checking phases rely heavily on Claude Opus and Sonnet, which may inherit latent model biases or systematic blind spots concerning certain nuances of experimental design.
  • Selection Bias from High-Impact Source Papers: Using oral and spotlight papers ensures clear prose and well-formulated baselines, but this might make injected flaws easier to discern than in typical, ambiguously drafted submissions.
  • Discriminator vs. Generator Gap: The benchmark strictly measures discriminative competence (detecting flaws in proposed designs); it does not evaluate whether models introduce similar methodological defects when generating end-to-end research pipelines from scratch.
  • vs FLAWS / PAPERAUDIT-BENCH: These benchmarks insert synthetic errors across entire multi-page manuscripts, confounding methodological critique with needle-in-a-haystack retrieval and formatting noise; TAMPERED ML focuses on concise, side-by-side experimental proposals where error localization is controlled.
  • vs SPOT: SPOT compiles real errata and retractions from published literature; while ecologically valid, such historical errors are sparse and vary wildly in scope. TAMPERED ML provides a scalable, balanced, and taxidermized counterfactual test suite.
  • vs MLE-Bench / The AI Scientist: These agentic benchmarks measure code execution, engineering output, and iterative hill-climbing; TAMPERED ML targets the complementary, hard-to-supervise capability of validating whether an experimental design actually supports its empirical claims.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates a rigorous, minimally invasive counterfactual benchmark specifically targeting LLM scientific judgment.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 1,858-pair evaluation across 19 frontier models, complemented by extensive diagnostic scaffolding, test-time compute curves, and post-stratified human validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate taxonomy, precise statistical confidence intervals, and lucid qualitative failure mode categorizations.
  • Value: ⭐⭐⭐⭐⭐ Critical contribution to scalable oversight, automated science, and mitigating deceptive reasoning in autonomous AI research agents.