Skip to content

From Atomic to Agentic: Towards Interpretable Evaluation of LLMsโ€™ Agentic Mathematical Capabilities

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Eternity-gaga/Agentic-Math-Bench
Area: LLM Agent
Keywords: mathematical agent, process-level evaluation, atomic capability decoupling, planning and feedback, LLM-as-a-judge

TL;DR

This paper introduces AgenticMathBench (AMB), the first process-level benchmark that systematically aligns a taxonomy of reusable mathematical atomic capabilities with core agentic behaviors (Planning, Action, Feedback), demonstrating that models with comparable end-to-end accuracy often exhibit starkly disparate and fragmented intrinsic agentic profiles.

Background & Motivation

In recent years, Large Language Models (LLMs) have achieved substantial breakthroughs in solving complex mathematical problems, rapidly evolving from single-turn chain-of-thought (CoT) prompting toward autonomous agentic workflows that integrate dynamic planning, tool invocation, and iterative self-reflection. However, the vast majority of existing mathematical reasoning benchmarksโ€”such as GSM8K, MATH, and OlympiadBenchโ€”remain strictly outcome-centric, assessing only the discrete accuracy of the final answer. This outcome-oriented evaluation offers little diagnostic value for analyzing multi-step reasoning processes, masking whether a model succeeds through rigorous logical derivation or merely stumbles upon correct answers via stochastic shortcuts and surface pattern matching.

The fundamental tension stems from a natural structural and cognitive alignment: both mathematical reasoning and autonomous agentic workflows can be recursively decomposed into discrete, reusable atomic cognitive modulesโ€”such as symbol recognition, spatial perception, modeling, formalization, and deductive steps corresponding to goal planning, single-step execution, and state monitoring. Yet current evaluation frameworks remain deeply fractured. On one hand, existing mathematical benchmarks group problems by traditional mathematical disciplines (algebra, number theory, geometry) without capturing interactive decision-making dynamics. On the other hand, contemporary agent benchmarks place models directly into end-to-end software scaffolds, where the model's intrinsic capabilities are heavily entangled with complex system engineering, external tool latencies, and framework-level error recovery.

To bridge this critical evaluation gap, this work decouples intrinsic model capabilities from deployment scaffolding by mapping mathematical atomic abilities onto the core lifecycle of autonomous agents. The core idea is to establish AgenticMathBench (AMB), a dual-axis aligned process-level benchmark that decouples mathematical atomic capabilities across Planning, Action, and Feedback, enabling granular, interpretable diagnosis of decision-making, single-skill execution, and causal error localization across textual and multimodal contexts.

Method

Overall Architecture

AgenticMathBench (AMB) targets the intrinsic agentic intelligence of foundational LLMs rather than the engineering performance of fully deployed systems. The benchmark is structured across two orthogonally aligned axes: the mathematical atomic-capability axis (spanning 9 capabilities categorized into 3 hierarchical levels) and the agentic-function axis (Planning, Action, and Feedback). Rather than evaluating each axis in isolation, AMB evaluates their intersection, diagnosing how models formulate roadmaps, execute isolated actions, and critique reasoning steps.

The data curation and evaluation pipeline starts with extensive data collection and filtering across 150+ math benchmarks, synthesizing 27 vetted source datasets spanning elementary problems to Olympiad competitions. High-quality multi-step trajectories are synthesized using a plan-action-feedback Math Agent and subjected to multi-stage filtering (answer correctness, two-annotator logical review, and structural diversity). These are formalized into three decoupled subtask suites: Planning (capability selection, solution roadmaps, and next-step prediction), Action (isolated execution across 8 atomic capabilities), and Feedback (binary correctness validation, first-error localization, and fix suggestions), judged through a calibrated combination of symbolic checks, rule parsers, and deterministic LLM judges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Raw Dataset Collection & Filtering<br/>150+ math benchmarks screened โ†’ 27 source datasets"] --> B["Stage 1: Dual-Axis Math-to-Agentic Taxonomy<br/>3 levels / 9 atomic capabilities โ†” Planning, Action, Feedback"]
    B --> C["Stage 2: Trajectory Synthesis & Multi-Stage Filtering<br/>Math-agent synthesis โ†’ Answer verification โ†’ Human review โ†’ Diversity filtering (17.3% kept)"]
    C --> D["Stage 3: Decoupled Three-Tier Evaluation Tasks<br/>Evaluation across Planning / Action / Feedback suites"]
    D --> E["Hybrid Verification & Interpretable Diagnosis<br/>Exact matching / Symbolic equivalence / LLM-as-a-Judge scoring"]

Key Designs

1. Dual-Axis Math-to-Agentic Taxonomy: Resolving the disconnect between academic disciplines and agent behaviors

Traditional benchmarks categorize problems into static mathematical domains that fail to reflect agentic interaction loops. Developed in close consultation with mathematics researchers and university educators, AMB defines a 3-level, 9-dimension hierarchy of reusable atomic capabilities aligned directly with agent operations: - Level 1 (Foundational Concept and Calculation): Symbol Recognition, Concept Understanding, and Calculation, serving as fundamental execution primitives for single-step actions; - Level 2 (Advanced Reasoning and Application): Spatial Perception, Formal Mathematical Language (Lean4), Deductive and Inductive Reasoning, and Mathematics Modeling, supporting structured multi-stage action execution; - Level 3 (Mathematical Meta-Cognitive): Theorem Application, Self-Reflection, and New Knowledge Learning, naturally aligning with global strategic planning and trajectory feedback mechanisms.

In this taxonomy, Level 1 and Level 2 skills provide the functional basis for the Action module, while Level 3 meta-cognitive abilities govern top-down coordination in Planning and bottom-up error monitoring and state auditing in Feedback.

2. Decoupled Three-Tier Evaluation Tasks: Overcoming the uninterpretable error accumulation of end-to-end testing

To isolate foundational LLM capabilities from confounding multi-step error propagation, AMB constructs three distinct evaluation suites: - Planning Evaluation: Formulated as ordered decision sequence generation. It spans three progressive tasks: โ‘  Capability Planning, predicting the set of required atomic capabilities \(A_{sub} \subseteq \mathcal{A}\) for a given problem; โ‘ก Solution Planning, generating a structured, step-by-step roadmap \(\pi = \{(a_1, g_1), \dots, (a_T, g_T)\}\) detailing atomic actions and subgoals; โ‘ข Next-Step Planning, where ground-truth trajectories are truncated at 20%, 50%, or 80% completion ratios, requiring the model to predict the immediate next action \(a_{t+1}\) and subgoal \(g_{t+1}\) conditioned on historical context \(s_t\). - Action Evaluation: Measures the model's ability to execute isolated, decoupled atomic mathematical actions without planning overhead across 8 capabilities. Custom evaluation protocols are designed for each: LaTeX transcription is evaluated via expression rate ExpRate and symbol rate SymRate; Concept Understanding measures key concept extraction via Set-F1; Calculation tests exact match EM (numeric) and CAS algebraic equivalence CAS-Eq; Spatial Perception assesses geometric relation predicates via Set-F1 and ValueAcc; Formal Language tests Lean4 compilation pass@k and semantic theorem alignment; Deductive Reasoning and Theorem Application measure proof outline coverage and consistency; and Modeling parses extracted variables, algebraic constraints, and objective functions. - Feedback Evaluation: Formalized as a trajectory-level state evaluation function \(f: (\text{problem}, \tau_{\le t}) \to \{\text{status}, \text{type}, \text{sugg}\}\). It tests three diagnostic tiers: โ‘  Correctness Judgment, performing binary verification on mixed correct and erroneous trajectories; โ‘ก Error Localization, requiring models to pinpoint the earliest erroneous step index \(t_{err}\) and classify its error type without gold references; โ‘ข Fix Suggestion, where a trajectory is truncated right before the earliest failure point, requiring the model to formulate a valid repair rationale and propose the corrective next step \((a_{t+1}, g_{t+1})\).

3. Rigorous Trajectory Synthesis and Hybrid Evaluation Pipeline: Ensuring high-fidelity ground truth and robust scoring

Because raw math benchmarks lack process-level step annotations, AMB synthesizes authoritative reference trajectories from competition problems (including OlympiadBench, Omni-MATH, and AMO-Bench). Using a Math Agent guided by explicit atomic capability sequences, trajectories are generated and passed through three rigorous filtration stages: โ‘  Final-answer correctness filtering, which discards ~76.8% of incorrect candidate runs; โ‘ก Human reasoning review, where two annotators independently verify step completeness, logical consistency, and mathematical validity (discarding any trajectory with a non-perfect score); โ‘ข Structural diversity filtering, removing trajectories with fewer than two steps or involving fewer than two distinct atomic abilities. Ultimately, only 17.3% of candidates (217 pure-text and 20 multimodal trajectories) are retained.

To evaluate open-ended textual plans and proof steps fairly without penalizing semantically equivalent variations, AMB deploys DeepSeek-V3 at temperature 0 as an automated judge guided by 3 to 5 human-authored criteria per task. Human validation on a 10% random sample shows over 96% agreement (overall F1 of 0.86) between LLM judge scores and human expert evaluations. Furthermore, a second-judge robustness check using GPT-5-mini yields a 0.96 cross-judge agreement, confirming that the benchmark's process-level conclusions are resilient to judge variance.

A Worked Example

The paper illustrates a diagnostic case study from the Feedback (Error Localization) task, demonstrating how process-level evaluation detects causal misattributions invisible to outcome-oriented metrics:

  • Input Problem: Calculate the limit \(\lim_{x \to 0} \frac{e^x - 1 - x}{x^2}\).
  • Trajectory Shown to the Model:
  • Step 1: Direct substitution gives \(\frac{0}{0}\), so Lโ€™Hรดpitalโ€™s rule applies (correct).
  • Step 2: Differentiate numerator: \((e^x - 1 - x)' = e^x - 1\) (correct).
  • Step 3: Differentiate denominator: \((x^2)' = 2x\) (correct).
  • Step 4: Apply Lโ€™Hรดpitalโ€™s rule again, but differentiate denominator as \((2x)' \to 1\) (incorrect; earliest causal error).
  • Step 5: Substitute \(x = 0\) to conclude the limit is 1 (direct algebraic consequence of Step 4).
  • Ground-Truth Expected Output: Earliest error is Step 4; error type is differentiation / algebraic computation error; repair action requires re-differentiating \(2x\) as 2, yielding \(\lim_{x \to 0} \frac{e^x}{2} = \frac{1}{2}\).
  • Actual Output from a Representative Model: "The first error occurs in Step 5. The previous applications of Lโ€™Hรดpitalโ€™s rule are valid, but the final substitution gives an incorrect final value. The solution should recompute the final limit instead of concluding 1."
  • Diagnostic Finding: An outcome-centric evaluation only notes that the model flags an incorrect trajectory, but AMB reveals that the model misattributes the failure entirely to the final evaluation in Step 5, missing the true causal error in Step 4. In a real-world agent workflow, this leads to repeated, fruitless recalculation of a fatally flawed intermediate state.

Key Experimental Results

Main Results

The authors evaluated leading models across general open-source, math-specialized open-source, and frontier commercial categories. The table below summarizes model performance across Planning and Feedback subtasks.

Model Category Model Name Capability Plan. F1 Solution Plan. Overall Next-Step Plan. Overall Correctness Acc Error Local. Overall Fix Suggestion Overall
General Open Llama-4-Scout-17B 60.8 36.2 33.5 50.0 30.8 28.0
General Open Llama-4-Maverick-17B 64.1 44.3 46.1 44.4 19.2 24.3
General Open DeepSeek-V3.2 52.1 76.1 46.3 49.0 60.5 31.5
General Open Qwen3-32B 41.5 40.2 36.3 56.1 51.4 24.7
General Open Qwen3-235B-A22B-Thinking 54.1 7.2 28.7 56.5 22.3 2.4
General Open GLM-4.7 56.0 8.3 29.0 47.3 16.3 13.0
Math Open Qwen2.5-Math-72B-Instruct 65.4 11.2 30.2 51.7 15.9 11.4
Math Open DeepSeek-Math-V2 57.9 13.6 29.5 48.8 34.6 9.5
Commercial GPT-5.2 74.0 85.1 41.0 49.3 47.2 34.7
Commercial Claude-4.5-Sonnet-Thinking 78.8 84.0 49.4 46.3 46.3 32.8
Commercial Gemini-3-Pro 72.9 66.2 34.2 52.8 43.2 28.6

Core Comparison: End-to-End Math Accuracy vs. Agentic Capability Decoupling

The table below contrasts standard end-to-end benchmark scores (MATH, AIME 2025) with decoupled performance across AMB's core agentic dimensions (Planning, Feedback, Action).

Model Name MATH (E2E) AIME25 (E2E) Planning (Plan.) Feedback (Feed.) Action (Act.) Diagnostic Profile & Breakdown
Llama-4-Scout-17B 82.6 10.0 33.5 46.7 53.7 Modest overall; balanced lower-tier agent profile
Llama-4-Maverick-17B 90.6 15.9 46.1 40.9 67.4 Stronger dynamic next-step planning; weak error localization
Qwen3-235B-A22B 98.0 81.5 28.7 21.4 84.5 High E2E score, but severe collapse in solution planning & fixes
GLM-4.7 98.8 95.7 29.0 29.7 64.5 SOTA on AIME25; solution planning falls below 10%
Qwen2.5-Math-72B 85.9 20.0 30.2 27.2 30.5 Severe degradation when decoupled from canned templates
GPT-5.2 100.0 99.0 41.0 53.7 89.3 Balanced leader across Action and Feedback; slight dynamic drop
Gemini-3-Pro 100.0 95.7 34.2 48.3 80.6 Top-tier E2E matching GPT-5.2, but significantly lower Planning

Key Findings

  • High end-to-end accuracy conceals severe agentic collapse: Models such as GLM-4.7 and Qwen3-235B-A22B achieve remarkable end-to-end scores on AIME 2025 (95.7% and 81.5%), yet exhibit near-total failure in Solution Planning (8.3% and 7.2%) and Fix Suggestion (13.0% and 2.4%). Optimizing models solely on final-answer correctness encourages shortcut behaviors where intermediate logical roadmaps are skipped or conflated.
  • Divergence between static roadmap planning and dynamic next-step adaptation: While leading models (DeepSeek-V3.2, Claude-4.5, GPT-5.2) achieve 76%โ€“85% overall scores in full solution planning, their performance sharply degrades to 34%โ€“49% in Next-Step Planning when evaluating partially executed trajectories. Models struggle to maintain accurate trajectory state tracking when required to adapt mid-course.
  • Causal error localization remains a formidable bottleneck: Across all models, binary correctness verification remains surprisingly low (hovering between 44% and 56%), and overall error localization rarely exceeds 50% (commercial models range between 43% and 47%). When errors occur, models frequently misattribute the failure to the final numerical result rather than the earliest faulty premise, producing generic or ungrounded fix recommendations.
  • Concept understanding is the primary bottleneck in action execution: While models score exceptionally high in raw symbolic calculation and objective formulation (often >80%โ€“95%), performance drops substantially on Concept Understanding Set-F1 (13%โ€“56%). This reveals that action failures in mathematical execution stem primarily from conceptual misalignment rather than arithmetic incompetence.

Highlights & Insights

  • Cognitive Isomorphism for Agentic Evaluation: The benchmark elegantly recognizes that mathematical problem-solving and agentic loops share identical atomic substructures, pioneering a dual-axis taxonomy that provides transparent diagnostics for agent designers.
  • Exposing the "Outcome-Reward Shortcut Illusion": The empirical findings provide concrete evidence that outcome-supervised reinforcement learning incentivizes models to take heuristic shortcuts, compromising their structural coherence, state tracking, and error-recovering capabilities in interactive environments.
  • High-Standard Trajectory Synthesis: By enforcing a stringent 17.3% retention threshold with dual-annotator blinded validation and diversity filtering, AMB sets a robust methodological standard for constructing trustworthy, unpolluted process-level datasets.

Limitations & Future Work

  • Absence of Long-Horizon Memory and Continuous Learning: Due to constraints in controlled benchmarking, AMB currently factors out long-term memory updates and dynamic in-context learning of unfamiliar theorems, leaving tool interaction loops and memory perturbation tests for future iterations.
  • Limited Scale in Multimodal Trajectories: High-difficulty competition problems featuring intricate geometric diagrams are inherently scarce, resulting in a smaller multimodal trajectory pool (20 retained trajectories) compared to the textual split.
  • Omission of Token and Latency Cost Accounting: The benchmark assesses intrinsic capability scores independently of token consumption or inference latency; incorporating Pareto frontiers that balance capability against computational budgets remains an important open direction.
  • vs. Outcome-Centric Math Benchmarks (GSM8K, MATH, OlympiadBench): Prior benchmarks evaluate multi-step reasoning as an impenetrable black box; AMB unrolls the trajectory into decoupled planning, execution, and feedback tasks, enabling fine-grained process-level interpretability.
  • vs. Domain-Taxonomy Benchmarks (GAUSS, MathEval): Existing skill-based benchmarks organize problems strictly along mathematical subject areas (algebra, geometry, calculus); AMB establishes an orthogonal mapping between mathematical capabilities and agentic operational functions.
  • vs. End-to-End Agent Framework Benchmarks (ToRA, MathAgent): Deployed agent benchmarks conflate model reasoning with scaffolding engineering and tool latency; AMB evaluates the foundational model's intrinsic agentic intelligence in isolation, providing a pure metric for backbone selection.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering dual-axis alignment of mathematical atomic capabilities and agentic behaviors with process-level task decoupling]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluation across 11 frontier models, rigorous 17.3% trajectory filtration, and validated human/second-judge agreement]
  • Writing Quality: โญโญโญโญโญ [Clear mathematical formalizations, well-structured taxonomy, and insightful case study diagnostics]
  • Value: โญโญโญโญโญ [Crucial diagnostic framework for understanding failure modes in mathematical reasoning agents and guiding process-level reinforcement learning]