Skip to content

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/AI4Engi/RePro
Area: LLM Reasoning
Keywords: Dynamic benchmark generation, Data contamination, Formal theorem proving, Neural automated theorem provers (ATPs), Lean 4 verification

TL;DR

This paper proposes RePro, the first verification-driven benchmark rewriting framework integrating Lean 4 neural automated theorem provers (ATPs) to filter candidate rewrites through staged feasibility screening, executable formalization, and kernel-checked proof verification, achieving 100% well-definedness, feasibility, and answer correctness on retained instances while exposing significant model sensitivity and memorization effects.

Background & Motivation

Evaluating mathematical problem-solving capabilities is foundational for assessing reasoning in frontier large language models (LLMs). However, benchmark reliability is severely compromised by widespread data contamination resulting from public overlaps between web-scale pretraining corpora and static evaluation suites such as GSM8K and MATH. This data leakage makes it difficult to discern whether high benchmark scores reflect genuine deductive mathematical reasoning or superficial memorization of surface linguistic patterns, formula structures, and specific numerical values. To alleviate contamination without constantly constructing manual benchmarks from scratch, recent research has explored dynamic benchmark generation and rewriting techniques, including synonym substitution, numerical perturbation, and structural paraphrasing.

Nonetheless, existing rewriting approaches predominantly rely on heuristic mechanisms such as rule-based perturbers or "LLM-as-a-Judge," which lack deterministic guarantees of problem validity and ground-truth answer correctness. In complex multi-step mathematical domains, unconstrained rewriting frequently introduces severe failure modes: questions become ill-defined due to semantic drift or dropped constraints; numerical alterations yield negative counts or physically impossible premises; and, most critically, reference answers derived from unverified model execution or solver runs suffer from persistent calculation hallucinations. Recent audits of expert benchmarks such as Humanity's Last Exam (HLE) reveal that while problem validity exceeds 92%, reference answer validity within its mathematical subset drops to 59.6%, underscoring that unverified rewriting simply swaps data contamination for benchmark noise and erroneous ground truth.

To break this impasse between generation diversity and mathematical truth guarantees, this work advocates replacing heuristic validation with machine-checkable formal logic. Core idea: integrate Lean-oriented neural automated theorem provers (ATPs) into benchmark rewriting, where LLMs generate diverse problem rewrites with conservative semantic screening, neural ATPs synthesize formal proofs, and Lean kernel-level checking verifies the reference answers, retaining exclusively instances backed by machine-verifiable proof certificates to guarantee 100% correctness.

Method

Overall Architecture

RePro establishes a progressive, decoupled verification paradigm that combines the flexible generative diversity of LLMs with the deterministic logical rigor of interactive theorem provers. The complete pipeline operates through four progressive stages: first, Feasibility Screening, where an instruction-tuned LLM generates candidate problem rewrites under a strict "do-not-solve" constraint and filters out semantic ambiguities and real-world constraint violations; second, Executable Formalization, translating validated natural-language rewrites into Lean 4 theorem specifications, evaluated by syntactic compiler checks and a three-way consensus semantic alignment filter; third, Proof-level Verification, deploying neural ATPs to search for formal proofs, accepted solely if they pass Lean 4 kernel-level checking without sorry; and fourth, Proof-Grounded Target-Answer Matching, where reference answers are extracted purely verbatim from verified proof spans and certified via a final-target consistency classifier. Candidate instances failing any stage are rejected and regenerated under bounded retry limits.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original Instances<br/>GSM8K / MATH"] --> B["Stage 1: Feasibility Screening<br/>Rewrite Generation without Solving + Semantic Constraint Filter"]
    B -->|Ambiguous / Infeasible / Invalid| R1["Discard & Retry (Max Retries)"]
    B -->|Passes Screening| C["Stage 2: Executable Formalization<br/>Lean 4 Translation + Compilation Check + 3-Pass Semantic Alignment"]
    C -->|Compilation Error / Inconsistent| R2["Discard & Retry (Max Retries)"]
    C -->|Passes Formalization| D["Stage 3: Proof-level Verification<br/>Neural ATP Proof Search + Lean Kernel-Level Checking"]
    D -->|Proof Search Failure / sorry| R3["Discard & Retry (Pass@k)"]
    D -->|Kernel Verified| E["Stage 4: Target-Answer Matching<br/>Read-Only Verbatim Extraction + Target Consistency Classifier"]
    E -->|Intermediate / Wrong Target / Unknown| R4["Discard Instance"]
    E -->|Fully Verified & Aligned| F["RePro Output Benchmark<br/>Well-defined · Feasible · 100% Correct Verified Answers"]

Key Designs

1. Feasibility Screening: Decoupled Problem Rewriting and Semantic Constraint Filtering

To eliminate the tendency of rewriting models to introduce hallucinated calculations early in the pipeline, RePro explicitly forbids the rewriter from solving the problem or proposing a candidate answer. Driven by Qwen3-MAX, the model applies numerical reparameterization, logical restructuring, constraint modification, and contextual story transformation while structurally enforcing the preservation of the target answer type (e.g., integer, rational number, interval, or finite set). The generated rewrite is then classified into either a pure mathematical or real-world applied problem. A dedicated feasibility screening agent rigorously eliminates ambiguous conditions, underspecified questions, internal premise contradictions, and physical impossibilities (such as negative human counts or fractional indivisible items). This stage exclusively secures problem validity, cleanly separating question legitimacy from answer verification.

2. Executable Formalization: Lean 4 Compilation and Conservative Three-Pass Semantic Alignment

To bridge the gap between unstructured natural language and machine-verifiable specifications without introducing semantic drift, candidate rewrites are formalized into Lean 4 code using Goedel-Formalizer-V2-8B. The pipeline enforces a two-tier gate: first, a format compilation check directly submits the Lean code to the Lean 4 compiler, rejecting any syntax errors, unresolved symbols, or typing mismatches; second, a conservative semantic alignment screen evaluates whether the compiled Lean theorem semantically mirrors the rewritten English problem across quantities, conditions, logical connectives, variable domains (e.g., discrete vs. continuous), and target questions. To minimize false acceptances, RePro enforces a strict "Conservative All-Pass Policy": the checker (Qwen3-MAX at temperature 0) is independently queried three times, and the instance is accepted only if all three passes return Consistent. A single mismatch, parse failure, or hesitation leads to instant rejection and regeneration.

3. Proof-level Verification: Neural ATP Search and Lean Kernel-Level Checking

To replace fallible model-based judges with sound mathematical certainty, the formalized Lean theorems are passed to neural automated theorem provers (defaulting to Goedel-Prover-V2-8B under a pass@k search budget). Rather than relying on heuristic confidence scores, the candidate Lean proof script must be checked by the Lean 4 proof assistant's kernel. Every tactic invocation must resolve without error, all proof states must be closed, and the presence of any unproven placeholder (sorry) results in immediate failure. Because the Lean kernel validates proofs according to the foundational axioms of dependent type theory (Calculus of Inductive Constructions), any proof successfully accepted by the kernel provides a deterministic mathematical certificate that the stated proposition is logically sound.

4. Proof-Grounded Target-Answer Matching: Read-Only Verbatim Extraction and Target Classification

Because a verified Lean proof frequently contains intermediate variables, witness constructions, and auxiliary lemmas, allowing an LLM to freely summarize the final answer risks introducing a post-hoc "second solving step" vulnerable to hallucination. RePro designs a strictly constrained two-step extraction protocol. First, literal answer extraction operates in a read-only mode, extracting exclusively strings and numbers that appear verbatim in the verified proof text; all arithmetic computation, fraction reduction, algebraic simplification, or calls to interactive commands (such as #eval or #reduce) are strictly forbidden. Second, a final-answer classifier categorizes the extracted candidate into final, intermediate, wrong_target, or unknown. Only instances where the candidate is unambiguously classified as final and precisely matches the exact target quantity queried by the rewritten problem are retained in the released benchmark.

Benchmark & Evaluation Protocol

RePro establishes three complementary reliability criteria to govern benchmark quality. Let \(N\) be the total generated candidates and \(N_c\) denote instances meeting criterion \(c \in \{\text{well-definedness, feasibility, answer correctness}\}\), quantified via \(N_c / N\): 1. Well-definedness: The problem statement must be complete, unambiguous, and provide sufficient constraints to delineate a unique target objective; 2. Feasibility: The problem must be logically consistent and admit a valid solution under basic real-world and mathematical domain constraints; 3. Answer Correctness: The reference answer must be backed by a kernel-verified Lean 4 proof certificate matching the requested target.

Experiments are conducted on GSM8K (grade-school arithmetic) and MATH (AoPS competition math spanning difficulty levels LV1–LV5). RePro utilizes Qwen3-MAX for rewriting/screening, Goedel-Formalizer-V2-8B for translation, and Goedel-Prover-V2-8B (pass@3) for theorem proving. Baselines include Auto-Dataset (synonym/context rewrite), ITD (inference-time semantic rewrite), and VarBench (parameterized template resampling). Tested models encompass 13 open-source architectures spanning 0.5B to 14B parameters across Qwen2, Qwen3, Llama-3.2, DeepSeek-R1, and Gemma 3 families.

Key Experimental Results

Main Results: Rewriting Quality Comparison

The table below contrasts rewriting quality across methods on the Full Dataset (all sampled candidate instances) and the RePro Successful Generation Subset (the subset where RePro successfully verifies a rewritten instance):

Dataset Subset Rewriting Method GSM8K Well-def. ↑ GSM8K Feasible ↑ GSM8K Correct ↑ GSM8K Gen. Rate ↑ MATH Well-def. ↑ MATH Feasible ↑ MATH Correct ↑ MATH Gen. Rate ↑
Full Dataset Auto-Dataset 99.60 99.19 87.10 100.00 98.43 96.23 79.99 100.00
ITD 100.00 99.19 89.11 100.00 98.26 97.85 81.15 100.00
VarBench 97.98 96.76 95.14 99.60 96.54 93.58 87.36 58.76
RePro (Ours) 100.00 100.00 100.00 88.31 100.00 100.00 100.00 59.16
RePro Subset Auto-Dataset 98.39 97.18 80.65 100.00 99.53 99.53 86.88 100.00
ITD 97.58 96.77 81.05 100.00 100.00 99.07 87.66 100.00
VarBench 98.79 98.39 96.77 100.00 95.49 93.73 89.23 60.53
RePro (Ours) 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00

Ablation & Impact Analysis: Prover Capabilities and Search Budgets

Evaluating the effect of underlying ATP capabilities (7B–8B parameter regime) and ATP call limits (pass@k) on RePro's generation success rates across difficulty levels (%):

Experimental Dimension Configuration GSM8K MATH LV1 MATH LV2 MATH LV3 MATH LV4 MATH LV5 Overall MATH
ATP Capability DeepSeek-Prover-V2-7B 61.9 38.4 33.7 16.4 6.9 6.5 ~20.6
(miniF2F p@32: 68.0%) Kimina-Prover-Distill-7B 81.9 75.8 73.9 58.0 4.0 4.6 ~43.3
(miniF2F p@32: 84.6%) Goedel-Prover-V2-8B 88.3 92.2 85.3 57.1 8.2 2.3 59.16
Rewriter Model Qwen3-8B 58.87 71.53 69.57 48.19 23.30 7.37 44.0
(with Goedel-Prover) Qwen3-32B 68.55 80.28 71.56 56.23 31.53 9.72 49.9
Qwen3-MAX 88.31 92.20 88.53 57.10 33.67 27.30 59.16
ATP Call Limit Pass@1 76.21 81.65 76.21 43.48 27.30 18.50 50.17
(Qwen3-MAX + Goedel) Pass@2 83.87 88.53 80.28 52.84 30.99 23.10 55.49
Pass@3 88.31 92.20 88.53 57.10 33.67 27.30 59.16

Key Findings

  1. 100% Deterministic Guarantee on Retained Benchmarks: RePro achieves perfect 100% well-definedness, feasibility, and answer correctness across all retained instances. In contrast, existing rewriting methods suffer significant degradation: on MATH, Auto-Dataset, ITD, and VarBench produce 20.01%, 18.85%, and 12.64% incorrect reference answers, with error rates on LV4–LV5 climbing beyond 20%–25%, proving that heuristic rewriting introduces severe ground-truth noise.
  2. Independent of Subset Selection Bias: When re-evaluating baseline methods solely on the RePro-successful subset, Auto-Dataset and ITD still generate ~19% incorrect answers on GSM8K and over 12% on MATH. This demonstrates that RePro's superiority stems from its formal proof verification filter rather than an easier data subset. Furthermore, an independent double-blind human audit of 2,400 instances yielded 100.00% agreement with RePro's automatic verdicts.
  3. Severe Model Sensitivity and Memorization Signals: On proof-verified rewritten MATH instances, smaller models suffer steep accuracy collapses. Qwen2-1.5B drops by 21.1, 12.0, and 10.5 percentage points on LV1–LV3 (relative drops of 26%–42%), while Llama-3.2-3B loses 9.8–15.1 points across all levels. Comprehensive confound analysis shows Spearman correlations between accuracy drops and surface length change (\(\rho = -0.000\)) or numeric range shift (\(\rho = 0.014\)) are essentially zero, confirming that drops are driven by sensitivity to benchmark reformulation and underlying memorization rather than cosmetic text length.

Highlights & Insights

  • Formal Verification as an Antidote to Benchmark Leakage: While previous decontamination efforts relied on LLM heuristics or unverified Python scripts, RePro establishes the first end-to-end framework embedding interactive theorem proving (Lean 4 + Mathlib) into dynamic evaluation, providing immutable proof certificates for benchmark veracity.
  • Read-Only Verbatim Extraction to Prevent Post-Hoc Drift: By enforcing read-only literal string extraction from Lean proofs followed by a discrete target classifier, RePro prevents models from re-solving or approximating answers post-verification, shutting down a subtle but pervasive source of calculation hallucination.
  • Diagnostic Insights into Neural Prover Failure Modes: An in-depth audit of DeepSeek-Prover-V2-7B failures revealed that over 94% of failed proofs were caused not by mathematical deficiency, but by output boundary corruption (e.g., generating unexpected <|Assistant|> markers that broke Lean theorem syntax) and loops of mechanical tactic repetition (norm_num, omega), providing crucial empirical guidance for improving neural provers.

Limitations & Future Work

  • Bottlenecked by Frontier ATP Proving Capacity: RePro's generation coverage is strictly bounded by neural ATP performance. While it achieves 88.3% retention on GSM8K and 59.2% on MATH, retention drops sharply on complex competition problems (33.7% on MATH LV5 and 0% on AIME 2025 under pass@32), highlighting that current provers struggle with olympiad-level formalization and search.
  • Scope Constrained to Formally Expressible Domains: The framework is inherently limited to mathematical problems formalizable within Lean 4 dependent type theory, making it inapplicable to open-ended natural language reasoning, common-sense QA, or subjective tasks.
  • Future Directions: Integrating Monte Carlo Tree Search (MCTS) and formal premise retrieval to scale up retention on olympiad-tier benchmarks (e.g., AIME and Omni-MATH), while exploring adaptations of formal verification to program synthesis and agentic planning pipelines.
  • vs Dynamic & Agentic Benchmark Rewriting (Auto-Dataset / ITD / VarBench): Existing methods rely on lexical perturbations, prompt heuristics, or parameterized Python execution, frequently generating infeasible constraints and unverified answers; RePro introduces formal Lean 4 kernel verification, ensuring 100% mathematical validity.
  • vs High-Effort Expert Benchmarks (GPQA / HLE / HLE-Verified): Static expert benchmarks incur enormous manual authoring costs and remain vulnerable to web contamination once published (with audits revealing 40.4% mathematical answer errors in HLE); RePro provides an automated, verifiable rewriting engine that renews benchmarks with guaranteed correctness.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of neural theorem provers (ATPs) and Lean 4 formal verification into dynamic LLM benchmark rewriting.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across GSM8K and MATH difficulty tiers, extensive ablation of ATPs and rewriters, 2,400-instance human audit, and comprehensive confound correlation analysis.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, mathematically precise definitions of validity criteria, and transparent reporting of prover failure modes.
  • Value: ⭐⭐⭐⭐⭐ Resolves the twin dilemmas of evaluation contamination and benchmark rewriting drift, offering a foundational blueprint for verifiable AI evaluation.