AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering¶
Conference: NeurIPS 2026
arXiv: 2609.34428
Code: https://github.com/JohnnyNLP/agenthop
Area: Others
Keywords: Agent Benchmark, Multi-Hop Literature QA, Diagnostic Evaluation, Resource-Constrained Sandbox, Attribution Decomposition
TL;DR¶
AgentHop introduces a diagnostic benchmark of 1,011 multi-hop scientific questions grounded in citation networks across 7,205 arXiv papers and a 7-tool resource-constrained sandbox, dissecting aggregate accuracy into retrieval, synthesis, tool-call signatures, and resource management across 19 frontier LLMs.
Background & Motivation¶
Language agents built on large language models interact with open external environments through iterative tool invocations and multi-turn observation gathering. In scientific literature exploration and academic multi-hop question answering, an agent must autonomously navigate extensive citation topologies, retrieve candidate papers, inspect fine-grained sections, and synthesize deep logical conclusions across multiple documents. However, the execution trajectories of these multi-step interactive tasks are inherently complex; system failures can stem from navigation dead-ends, breakdowns in cross-document deductive reasoning, cyclic tool execution loops, or pre-mature budget exhaustion. Each failure stage demands fundamentally different post-training interventions or architectural remedies, yet existing evaluation protocols fail to provide developers with clear diagnostic insights regarding which specific sub-ability constitutes the primary bottleneck for a given model.
Prevailing agent benchmarks across scientific research (e.g., OpenScholar), software engineering (SWE-bench), and broad web environments (WebArena, GAIA) have largely inherited the legacy single-scalar accuracy convention from traditional NLP benchmarks, prioritizing leaderboard comparability over diagnostic granularity. This monolithic scoring convention obscures the underlying mechanics of failure: it cannot distinguish whether a model located the ground-truth literature but failed to synthesize the conclusion (a synthesis bottleneck), possessed superior reasoning capabilities but became trapped in a retrieval loop (a retrieval bottleneck), or was prematurely terminated by system resource limits due to undisciplined tool use. Concurrent decomposition efforts either isolate only chain geometry (such as AgenticRAGTracer) or inspect atomic tool syntax (such as TRAJECT-Bench), failing to expose cross-axis behavioral interactions.
To resolve this diagnostic ambiguity and systematically characterize agentic sub-abilities in long-horizon scientific navigation, this paper introduces a four-axis decomposition evaluation framework. Core idea: construct a rigorous diagnostic benchmark of 1,011 multi-hop scientific questions derived from 7,205 arXiv papers and over 450K citation edges via an eight-stage construction pipeline, pairing it with a 7-tool sandbox bounded by a 20-turn, 200K-token, and 30-point budget constraint, to decompose aggregate accuracy into search recall, evidence conversion, tool-use fingerprints, and resource termination attribution across 19 frontier language models.
Method¶
Overall Architecture¶
The AgentHop evaluation framework integrates an eight-stage benchmark curation pipeline, an asymmetric cost-constrained seven-tool interaction sandbox, and a four-axis diagnostic evaluation suite. During evaluation, the agent receives a seed paper identifier, a scientific research question, and four candidate options labeled AโD. The gold papers containing the verifiable evidence are withheld; the agent must navigate starting from the seed across a two-hop citation neighborhood (averaging 311 candidate papers per query). Under strict resource constraints (a 20-turn limit, a 200K-token aggregate context ceiling, and a 30-point tool budget), the agent invokes literature search, reference graph traversal, section header listing, full-text reading, and internal reasoning tools to assemble evidence and commit to a final answer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Seed Paper ID + Research Question + MCQ Options"] --> B["Eight-stage Citation Pipeline & Dual Adversarial Filtering<br/>4 Generation Stages + 4 Ensemble & Audit Stages"]
B --> C["Asymmetric Resource-Constrained Sandbox<br/>7 Tools + 20 Turns / 200K Tokens / 30-Point Budget Caps"]
C --> D["Four-Axis Decomposed Evaluation Suite<br/>Search Recall + Synthesis Conversion + Action Trigrams + Resource Failures"]
D --> E["Output: Option Commitments & Model Capability Profiles"]
Key Designs¶
1. Eight-stage Citation Pipeline & Dual Adversarial Filtering: Rigorous curation of 1,011 multi-hop questions from 7,205 papers To prevent data contamination, shortcut exploitation, and factual drift, AgentHop implements a symmetric eight-stage construction pipeline comprising four generation stages and four filtering stages. The generation phase samples 953 seed papers across nine top CS venues (NeurIPS, ICML, ICLR, ACL, EMNLP, NAACL, CVPR, ECCV, SIGIR from 2022 to 2025), expanding outgoing citations across depth-1 (BFS single-target) and depth-2 (DFS multi-target) graphs; chains are retained only if they satisfy Jaccard reference overlap (\(\ge 0.08\)) and pass an LLM curiosity-gap rating (\(\ge 4/5\)). GPT-5.4 drafts evidence-grounded questions while explicitly barring target paper titles or distinctive methodology terms in the prompt to prevent shortcut retrieval; three adversarial distractors are subsequently generated (seed-only, sibling paper, and parametric hallucinations). The filtering phase employs a 3-model ensemble (GPT-4.1, Claude Sonnet 4.6, DeepSeek-chat) to eliminate 68 shortcut leakage items and 434 weak-evidence consensus failures, followed by structural sanity checks (dropping 71 broken chains), 7-auditor expert verification via a dedicated Gradio UI (retiring 35 flawed extraction items), and section length filtering (retiring 27 parser collapse items), yielding 1,011 verified items.
2. Asymmetric Resource-Constrained Sandbox: Fine-grained point budgeting and triple hard ceilings
To simulate real-world deployment constraints where API latency and execution budgets are bounded, the sandbox exposes seven distinct tools with calibrated cost weights. Five tools interact with the academic corpus: keyword_search (retrieves paper IDs and abstracts matching a query, cost 1 point), get_paper_info (retrieves title, authors, abstract, cost 1 point), get_references (exposes citation lists with surrounding sentence context, cost 1 point), list_sections (enumerates section headers and character counts, cost 1 point), and read_section (returns full section text, cost 5 points). Two unmetered control tools manage execution: think (logs internal model deliberation notes) and submit_answer (records the option choice and terminates the run). By assigning a heavy 5-point cost to read_section, the sandbox penalizes brute-force reading, compelling agents to navigate via cheaper metadata tools first. The environment simultaneously enforces three rigid ceilings: a 20-turn exchange limit, a 200K aggregate token limit, and a 30-point budget. Calls overdrawing points are rejected with warnings; consecutive violations or reaching any boundary trigger immediate termination and failure scoring.
3. Four-Axis Decomposed Evaluation Suite: Isolating retrieval, reasoning synthesis, and behavioral fingerprints
To move beyond monolithic headline accuracy, the benchmark formalizes four orthogonal diagnostic axes:
- Search Axis: Evaluates paper recall (\(r_i \in \{0, 1\}\)), where \(r_i = 1\) if and only if the agent successfully invoked read_section on at least one section of every gold paper for that item; it also reports a stricter section recall requiring delivered text to match direct-labeled evidence sections.
- Synthesis Axis: Formulates the conditional conversion rate, measuring the fraction of correct answers among samples where paper recall was satisfied:
$\(P(c_i = 1 \mid r_i = 1) = \frac{\sum_{i=1}^N c_i \cdot r_i}{\sum_{i=1}^N r_i}\)$
This metric decouples reasoning quality from retrieval luck, isolating whether an agent can overcome engineered distractors once the necessary evidence is in context.
- Tool-use Pattern Axis: Measures average calls per turn to delineate strictly sequential (\(=1.0\)) from parallel (\(>1.0\)) execution regimes, supplemented by transition bigram associations and per-family trigram fingerprints.
- Resource Axis: Tracks token, turn, and point expenditures while decomposing non-submission terminations into Max Tokens, Max Turns, and Max Budgets failure rates to identify binding architectural bottlenecks.
Loss & Training¶
As an evaluation benchmark and diagnostic sandbox, AgentHop involves no direct gradient optimization of model parameters. However, the evaluation harness enforces formal constraint-checking and statistical significance testing:
1. Budget Rejection and Penalization Policy: If a tool call exceeds the remaining point budget, it is rejected without deducting points, returning: "REJECTED: Insufficient budget". If the agent issues two consecutive budget-rejected calls, the trajectory terminates under terminated_by = "budget".
2. Hard Termination Enforcement: Cumulative tokens are tracked after every assistant turn; exceeding 200,000 tokens immediately terminates the run under terminated_by = "token_limit". Reaching 20 turns without submitting triggers terminated_by = "max_turns". All non-submissions are scored as incorrect (\(c_i = 0\)).
3. Paired Significance Testing: Conversion rate differences across models are validated via exact two-sided McNemar tests on common recalled items, evaluating discordant answer pairs to ensure conversion leadership is statistically significant (\(p < 0.05\)).
Quantitative Experiments¶
Main Results¶
The 1,011 benchmark items were evaluated across 19 frontier and open-weight models categorized into closed frontier, open frontier, and compute-efficient tiers:
| Tier & Model Name | Search Recall (ST/MT) | Section Recall (ST/MT) | Conversion (ST/MT) | Calls/Turn | Avg Tokens (k) | Failure Tok/Turn/Bud (%) | Accuracy (%) |
|---|---|---|---|---|---|---|---|
| Closed Frontier | |||||||
| Gemini-3 Pro | 0.868 / 0.752 | 0.859 / 0.655 | 0.957 / 0.970 | 1.05 | 92.4 | 4.4 / 0.1 / 0.0 | 89.1 |
| GPT-5.3 codex | 0.812 / 0.623 | 0.775 / 0.470 | 0.896 / 0.902 | 1.44 | 69.5 | 0.3 / 0.0 / 0.7 | 82.5 |
| GPT-5.4 | 0.808 / 0.643 | 0.752 / 0.454 | 0.878 / 0.846 | 1.74 | 67.5 | 0.3 / 0.0 / 4.9 | 79.0 |
| Claude Opus 4.6 | 0.873 / 0.777 | 0.854 / 0.639 | 0.871 / 0.855 | 1.25 | 110.5 | 6.7 / 0.0 / 0.0 | 77.5 |
| Claude Sonnet 4.6 | 0.798 / 0.600 | 0.773 / 0.479 | 0.936 / 0.891 | 1.32 | 95.8 | 3.5 / 0.9 / 0.0 | 76.6 |
| Gemini-3 Flash | 0.724 / 0.501 | 0.680 / 0.386 | 0.625 / 0.644 | 1.01 | 180.3 | 27.7 / 2.1 / 0.2 | 54.7 |
| Open Frontier | |||||||
| GLM-5.1 | 0.768 / 0.673 | 0.734 / 0.506 | 0.830 / 0.768 | 1.20 | 102.0 | 3.2 / 1.6 / 3.2 | 69.6 |
| DeepSeek-reasoner | 0.720 / 0.558 | 0.667 / 0.436 | 0.760 / 0.717 | 1.00 | 128.6 | 5.2 / 1.2 / 0.0 | 59.6 |
| DeepSeek-chat | 0.773 / 0.623 | 0.743 / 0.506 | 0.706 / 0.670 | 1.00 | 156.7 | 12.2 / 2.2 / 0.0 | 56.4 |
| Kimi-K2.5 | 0.764 / 0.569 | 0.720 / 0.463 | 0.613 / 0.504 | 1.08 | 115.5 | 3.6 / 3.9 / 2.5 | 44.1 |
| MiniMax-M2.7 | 0.667 / 0.553 | 0.604 / 0.334 | 0.636 / 0.465 | 1.33 | 84.3 | 1.0 / 1.4 / 4.8 | 41.3 |
| Compute-Efficient | |||||||
| Gemma-4-31B | 0.621 / 0.521 | 0.579 / 0.348 | 0.788 / 0.861 | 1.01 | 87.6 | 2.4 / 0.1 / 0.0 | 64.8 |
| Qwen3.5-27B | 0.678 / 0.589 | 0.623 / 0.404 | 0.691 / 0.617 | 1.00 | 100.6 | 7.1 / 2.1 / 0.6 | 48.2 |
| Qwen3.6-35B-A3B | 0.505 / 0.302 | 0.456 / 0.174 | 0.672 / 0.657 | 1.00 | 109.5 | 11.2 / 3.7 / 0.8 | 38.1 |
| Qwen3.5-35B-A3B | 0.576 / 0.472 | 0.514 / 0.291 | 0.517 / 0.450 | 1.00 | 110.4 | 9.3 / 2.4 / 1.5 | 32.0 |
| Gemma-4-26B-A4B | 0.352 / 0.158 | 0.236 / 0.093 | 0.425 / 0.557 | 1.01 | 163.7 | 37.0 / 6.9 / 0.3 | 23.6 |
| Gemma-3-27B | 0.139 / 0.054 | 0.106 / 0.025 | 0.342 / 0.250 | 1.00 | 33.6 | 0.1 / 0.0 / 0.0 | 18.6 |
| Qwen3-30B-A3B | 0.470 / 0.460 | 0.423 / 0.309 | 0.318 / 0.225 | 1.00 | 101.7 | 4.8 / 6.4 / 0.3 | 16.0 |
| Qwen3-32B | 0.153 / 0.090 | 0.130 / 0.059 | 0.517 / 0.225 | 1.02 | 37.6 | 0.6 / 0.2 / 0.0 | 13.3 |
Ablation Studies¶
To isolate whether multi-hop reasoning relies on keyword searches or reference graph traversal, tool ablation experiments were conducted across three representative checkpoints:
| Model | Zero-tool Closed-book (%) | Sandbox Lift \(\Delta_{\text{tools}}\) (pp) | No-search Topology (%) | Keyword Lift \(\Delta_{\text{kw}}\) (pp) | Full Sandbox (%) |
|---|---|---|---|---|---|
| DeepSeek-chat | 18.3 | +35.0 | 53.3 | +3.1 | 56.4 |
| Gemma-4-31B | 31.7 | +23.1 | 54.8 | +10.0 | 64.8 |
| GPT-5.3 codex | 48.6 | +30.7 | 79.3 | +3.2 | 82.5 |
Key Findings¶
- Model family stopping signatures diverge markedly: When navigating multi-target citation graphs, GPT models exhibit premature commitment (GPT-5.3 codex immediately commits on 47% of ST and 67% of MT items, committing prematurely on 34โ44% of MT failures). Anthropic models (Opus and Sonnet 4.6) and GLM-5.1 follow a verify-before-commit policy, almost never raw-submitting (\(\le 2\%\)) without invoking
think. Conversely, DeepSeek, Kimi, and Flash over-search, reading extensively even after finding gold papers. Gemini-3 Pro achieves the most balanced navigation strategy. - Decoupled axes uncover distinct sub-ability bottlenecks: MiniMax-M2.7 and Kimi-K2.5 achieve competitive retrieval recall but suffer sharp conversion drops of 11โ17 pp on multi-target questions (synthesis bottleneck). Conversely, Gemma-4-31B achieves an exceptional 0.861 multi-target conversion rate despite a low 0.521 recall ceiling, demonstrating that its primary weakness lies in graph search rather than reasoning.
- Architectural upgrades reveal hidden capability trade-offs: In the Qwen MoE lineage, upgrading from 3.5-35B-A3B to 3.6-35B-A3B lifted headline accuracy from 32.0% to 38.1%; however, decomposition reveals this lift was entirely driven by single-paper synthesis, while multi-target paper recall regressed dramatically from 0.472 to 0.302, quietly sacrificing multi-paper navigation.
- Reference-following navigation carries the primary retrieval burden: Ablations show that removing keyword search while retaining reference traversal (No-search) captures the vast majority of tool gains (+23 to +35 pp over closed-book), whereas adding keyword search provides only marginal gains (+3.1 to +10.0 pp), confirming that structural citation links serve as highly effective navigation scaffolding.
Highlights & Insights¶
- Decomposed diagnostics replace opaque leaderboards: Dissecting end-to-end performance into search recall, evidence conversion, execution patterns, and resource management allows practitioners to pinpoint whether an agent needs better retrieval scaffolding, cross-document reasoning training, or stricter token budgeting.
- ReAct-style verification correlates directly with accuracy gains: Bigram trajectory analysis reveals that invoking
read_section -> thinkconfers a +13.6 pp accuracy advantage, andthink -> submit_answeryields a +12.6 pp lift; in contrast, unreflective consecutive reading (read_section -> read_section) incurs an 11.5 pp penalty. - Strict resource budgeting exposes brittle trajectory pathologies: Hard token and turn caps unmask inefficiencies that unconstrained benchmarks hide, such as Gemma-4-26B-A4B suffering a 37% non-submission rate due to verbose MoE generation exhausting the token budget.
Limitations & Future Directions¶
- Acknowledged limitations: The benchmark corpus is restricted to nine computer-science venues (2022โ2025) and formatted as four-option MCQs; questions were synthesized predominantly using GPT-5.4, introducing possible generator-evaluator stylistic bias.
- Real-world exploration gap: The sandbox relies on text-based section parsing and does not currently support inspecting embedded figures, tables, or companion software repositories, which are integral to full-scale scientific investigation.
- Future extensions: Expanding the sandbox to include multimodal PDF perception, code execution environments, and open-ended scientific synthesis tasks (such as literature review generation and hypothesis falsification).
Related Work¶
- vs OpenScholar: OpenScholar evaluates scientific literature synthesis in retrieval-augmented LM settings where relevant documents are provided upfront; AgentHop requires agents to autonomously search an unannotated 2-hop citation graph and balance exploration against strict budgets.
- vs AgenticRAGTracer: AgenticRAGTracer diagnoses multi-step RAG purely across chain hop geometry; AgentHop fully decouples retrieval recall, reasoning conversion, tool-use fingerprints, and resource management failure modes.
- vs GAIA & WebArena: General web agent benchmarks assess task completion end-to-end without fine-grained cost structures; AgentHop employs an asymmetric point budget and strict resource bounds to capture execution efficiency and stopping behaviors.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering four-axis decomposed diagnostic framework that separates search, synthesis, tool-use signatures, and resource constraints for scientific agents.
- Soundness: โญโญโญโญโญ Rigorous eight-stage construction with 3-model filtering and human expert audits, paired with comprehensive evaluation across 19 models.
- Clarity: โญโญโญโญโญ Exceptionally clear narrative, structured diagnostic questions, and rich empirical visualizations.
- Significance: โญโญโญโญโญ Provides critical diagnostic benchmarks and actionable behavioral insights for language agent training and deployment.