EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/RetroDnix/EPIBench
Area: Multimodal VLM
Keywords: multi-turn research workflow, proactive search, cross-paper evidence fusion, evidence memory, agent benchmark

TL;DR¶

EpiBench instantiates a short literature study as a multi-turn question episode: the task provides no target paper and only an image snippet or a short bibliographic cue, so the agent must search out several papers on its own, extract figures and tables as evidence, append that evidence to an episode-level memory pool, and perform cross-paper multi-evidence fusion in a final turn where all tools are disabled; beyond final correctness, the protocol scores evidence correctness (EC) and the minimality gap (MG) so that "did it reach the right evidence" and "did it get there efficiently" are measured separately.

Background & Motivation¶

Scientific-literature benchmarks have become dense in recent years: SciVQA, CharXiv, and SPIQA all target question answering over scholarly figures, and LiveXiv pushes this paradigm further with large-scale, continuously updated VQA tasks generated from papers. Their common shape, however, is single-document: a question lands on one figure, one table, or one text span inside one paper, and the answer is often available from that single artifact. This bypasses two things that any real literature review necessarily involves — aligning information across several figures and tables before a conclusion can be drawn, and assembling evidence from multiple papers (cross-paper reasoning). The multi-turn and long-horizon memory line has MMDU and MULTIVERSE as multi-image multi-turn dialogue benchmarks, and memory QA datasets such as Memory-QA and EMP stretch the reasoning horizon, but their objective is to retrieve an answer to an individual query from previously stored memories; they are not paper-centric and rarely require cross-paper multimodal evidence integration. The closest prior work is PaperArena, a tool-augmented multi-hop navigation benchmark over academic papers, yet its tasks rarely require jointly integrating several figures and tables to derive an answer. The evaluation protocol is the second mismatch: because repeated re-browsing and re-parsing of PDFs is typically allowed, a benchmark can hardly answer "did the agent reuse evidence it had already acquired?", and it seldom checks whether intermediate evidence is correctly grounded — so evidence-attribution errors (answering correctly but from the wrong figure) and integration failures (holding all the evidence yet failing to combine it) both collapse into a single accuracy number.

The paper turns these two mismatches into two explicit gaps: no benchmark systematically covers the human-like workflow of proactive search plus multi-turn interaction plus cross-paper multimodal evidence accumulating and fusing over time, and no protocol provides the process-level metrics for evidence reuse and evidence correctness that fine-grained diagnosis needs. EpiBench is designed to close both. On the task side, its prompts provide neither a target paper nor direct identifiers — only an image snippet or a short bibliographic hint as a seed, which mirrors how real users actually ask, rarely arriving with precise citations. Answers are objective questions whose comparisons, selection rules, and set-style constraints force alignment across multiple figures and tables in multiple papers, and whose evidence must be read off visual legends and plotted values, which leaves little room for answering from world knowledge or a partial visual context. On the protocol side, the paper introduces an access-budget evaluation setting: required evidence units are annotated at the granularity of a paper identifier plus a figure or table identifier, so evidence reuse and tool efficiency can be computed automatically from the agent's interaction trace.

The core idea is to make the reusability of a research process measurable by pairing episodic multi-turn tasks with a memory-restricted final turn: each episode carries a minimal tool-call checklist and a set of required evidence units, the final turn disables all tools so that only multimodal evidence already in the memory pool can be used for fusion, and two process metrics, EC and MG, decompose a coarse "answered correctly" into "did it reach the right evidence" and "did it get there efficiently."

Method¶

Overall Architecture¶

EpiBench is not a model method but a package of task formulation, a data-construction pipeline, and a process-level evaluation protocol. Its problem is to compress a small real literature study into an automatically scorable episode while keeping "did the agent actually follow a research workflow" measurable. The chain runs as follows: first formalize the episodic multi-turn task (who answers what under which constraints), then use a human-curated pipeline to turn real papers into 102 episodes, then give the agent a literature-specific toolset and a chronologically appended memory pool, and finally cut the tools in the final turn, allow only memory reads, and settle the score with success rates plus two process metrics. The three pieces are coupled: without per-turn tool-call checklists and evidence-unit annotations the two process metrics cannot be computed, and without the tool-disabled final turn, evidence reuse cannot be separated from "just looking it up again."

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Research task<br/>question chain + seed cue"] --> B["Episodic multi-turn research task<br/>no target paper, cues only"]
    B --> C["Expert-curated construction pipeline<br/>68 → 485 → 102 episodes"]
    C --> D["Literature tool environment and episodic memory<br/>four tools + chronological memory pool"]
    D --> E["Budgeted process-level evaluation protocol<br/>memory-only final turn + EC / MG"]

Key Designs¶

1. Episodic multi-turn research tasks: no target paper, cues only, cross-paper multi-evidence fusion enforced

Most paper-QA benchmarks hand the answer's coordinates to the model in the prompt — either the target paper is named or an arXiv ID is given, so the agent only needs to open and read. EpiBench removes those coordinates: each episode is a sequence of turns \(\{(q_t, I_t)\}_{t=1}^{T}\), where \(q_t\) is an objective question and \(I_t\) is only an optional seed cue (an image snippet or a short bibliographic hint). The target papers must be discovered by the agent from citation cues and indirect hints, and the answers are deliberately built so that several papers each contribute a piece of the required evidence: questions use comparisons, selection rules, and set-style constraints that demand alignment between legends and plotted values in several figures, entries in several tables, and the accompanying prose, as well as reconciliation of experimental settings across sources. The design works in both directions. It closes the "world knowledge guesses it right" path, because the answer depends on specific figures and tables inside specific papers, and it also closes the "one figure is enough" path, because the correct option depends on joint constraints across sources. Later turns within an episode depend on earlier ones, which creates a cross-turn consistency requirement: if an experimental setting established in an early turn is misremembered, the conclusion in a later turn contradicts itself.

2. Expert-curated construction pipeline: model-drafted question chains, human-verified evidence targets

Construction proceeds in four steps: seeds, pool expansion, drafting, and human finalization. The seed set is collected from public venues and preprint repositories (OpenReview, arXiv) and covers six broad areas of computer vision and machine learning, comprising 68 representative papers. The candidate pool is then expanded at scale around each seed with Connected Papers, a citation-graph exploration tool; after deduplication and basic quality filtering this yields a source corpus of 485 papers. The third step uses GPT-5.2 to draft the episodic question chains, and crucially the drafting also produces a per-turn checklist of tool calls, each specifying the target paper and, when applicable, the target figure or table — this checklist later serves both as an evaluation reference and as the source of the minimal viable path. Finally, five master-level computer-science candidates refine, validate, and curate the generated episodes: they remove ambiguous cases, correct evidence targets, resolve annotation inconsistencies in a last consolidation pass, and ensure that every paper the agent is required to read contributes indispensable evidence to the final answer. The human pass matters because automatic drafting readily produces pseudo multi-evidence questions that appear to need three papers while the first one already suffices; only human review removes them.

The result is 102 episodes split into Easy (37) and Hard (65). Difficulty is not assigned by prompt length but by three quantities at once: the number of turns required, the amount of cross-paper and multimodal evidence integration, and how much evidence reuse later turns demand. The reported dataset statistics (average papers, turns, and images per difficulty, plus the distribution over subject areas) are shown in the paper's Figure 3, which is not expanded into textual numbers here; ⚠️ refer to the original Figure 3 for the exact values.

3. Literature tool environment and episodic memory: four tools for retrieval and visual grounding, memory appended in access order

The model itself is not modified; only the interface is provided. There are four tools, all serving the chain from a cue to a paper and then from a paper to a figure or table. Web Search queries a search backend and returns ranked results with titles, snippets, and URLs (the paper uses both Google and DuckDuckGo backends); it locates candidate papers, resolves paper identities from indirect cues, and follows citation leads. PDF Extractor takes a paper identifier (title or arXiv ID) and downloads and parses the PDF into a normalized markdown representation (the paper cites MinerU for this). PDF Extractor RAG performs retrieval over that cached markdown: given a text query it returns the most relevant spans with lightweight provenance such as section context and figure or table references, so the agent can look up specific evidence without rereading the whole paper. Content Extractor extracts a specified figure or table from a cached paper — for figures it returns the image plus caption context, and for tables a structured representation plus surrounding text when available; this is the only channel through which pixel-level evidence can be obtained when a question demands visual grounding.

Memory is implemented as an episode-level chronological store: retrieved PDFs, extracted figures and tables, and tool observations are all appended in the order they were accessed. The design looks plain but is in fact the precondition for process-level evaluation — a transparent state is what allows reuse and efficiency to be computed post hoc from the interaction trace. Paired with memory is a hard constraint: for episodes that require reuse, the final turn runs in memory-only mode with all tools disabled, so the agent must answer from evidence already in memory and cannot reopen or re-download documents. This isolates memory utilization from retrieval ability and makes reuse behavior comparable across models. The loop itself is ReAct-like: the agent first produces a plan of intended actions, then generates executable code for the tool invocation. Each turn is bounded by a step budget (10 steps by default), and if no final answer arrives within budget the agent gets one extra opportunity to summarize its exploration and deliver an alternative answer in a single response.

4. Budgeted process-level evaluation protocol: memory-restricted final turn, EC / MG process metrics, and error attribution

Reporting only final accuracy conflates very different failures — an episode may be lost because the paper was never found, or because it was found and then not reused. The protocol separates them. On the success side, besides episode success rate (ESR, where an episode counts as correct only if every turn is correct), the paper separately reports final-turn accuracy \(\mathrm{Acc_{final}}\) and the average accuracy over non-final turns \(\mathrm{Acc_{pre}}\), since the final turn is deliberately the hardest and carries multi-evidence fusion and memory reuse. On the evidence side, each turn \(t\) is annotated with a set of required evidence units \(E_{i,t}\) indexed by paper identifier and figure-or-table identifier; the evidence the agent actually accessed, \(A_{i,t}\), is recovered from the tool trace and compared only on correctly answered turns:

\[\mathrm{EC}=\frac{\sum_{i=1}^{N}\sum_{t=1}^{T_i} y_{i,t}\,\lvert E_{i,t}\cap A_{i,t}\rvert}{\sum_{i=1}^{N}\sum_{t=1}^{T_i} y_{i,t}\,\lvert E_{i,t}\rvert}\]

where \(y_{i,t}\) indicates whether that turn was answered correctly. EC measures the fraction of required evidence units actually accessed on the turns that were answered correctly, and it exists specifically to expose the "answered correctly but from a different figure or paper" case. On the efficiency side, every episode carries a minimal viable path checklist of \(c_i^{\star}\) tool calls; with \(c_i\) the number of calls the agent actually executed and \(s_i\in\{0,1\}\) indicating episode success, the minimality gap is reported on successful episodes:

\[\mathrm{MG_{succ}}=\frac{1}{\sum_{i=1}^{N}s_i}\sum_{i=1}^{N}s_i\cdot\frac{c_i}{c_i^{\star}}\]

A larger MG means more redundancy relative to the minimal viable solution. The last dimension is error attribution: each failed episode receives a single label based on the earliest point of deviation in the tool trace and evidence alignment, in five categories — Retrieve (the intended papers were never located), Perception (relevant evidence was accessed but the figure or table was misread or misinterpreted), Reasoning (the necessary evidence was available yet integration or constraint satisfaction failed), Reading Memory (evidence retrieved earlier was not correctly reused in later turns, including conflation across sources and missing evidence selection during final fusion), and Others (residual failures such as max-step termination, runtime errors, or PDF-parsing errors). The emphasis on the earliest deviation matters because one failed episode usually cascades into further errors downstream, and only attribution to the source can guide improvement. Together these metrics make "high scores, low competence" (high Acc_pre but low ESR) interpretable.

A Worked Example¶

⚠️ The paper does not provide a fully numeric walkthrough (its Figure 2 only sketches the flow), so the following reconstructs the skeleton of a typical episode from the original description without inventing numbers. In a representative episode, the early turns start from an image snippet or a short bibliographic cue: the agent uses Web Search to surface candidate papers, opens them with PDF Extractor, and pulls a specific figure or table with Content Extractor, reading an experimental setting off a legend and a plotted value. Middle turns move to a second paper, extract a table entry under the comparison condition, and align the two settings. The final turn asks for a comparison or set-style judgment that requires joining several evidence units accumulated earlier — and at that point the tools are gone, so the agent can only select and fuse the figures and tables it stored in the memory pool. With a 10-step budget, the early turns have to be efficient, because redundant re-browsing directly eats into the steps available for later evidence extraction.

Key Experimental Results¶

Main Results¶

The evaluation covers six open-source and four closed-source LMMs as agent backbones (model names are given as written in the original paper; ⚠️ refer to the original), plus a human baseline produced by two master-level computer-science candidates solving the same episodes under the same tool interface. Turn-level correctness is assigned by LLM-as-a-Judge (GPT-5.2) as a binary label under a fixed rubric.

Backbone LMM	Easy (ESR / Acc_final / Acc_pre)	Hard (ESR / Acc_final / Acc_pre)	Average (ESR / Acc_final / Acc_pre)
Qwen3.5-9B	5.26 / 55.26 / 41.11	0.00 / 36.07 / 24.07	2.02 / 43.43 / 28.33
Qwen3.5-35B-A3B	32.43 / 77.78 / 57.61	10.77 / 46.88 / 52.30	18.63 / 58.00 / 53.60
Qwen3-VL-235B-Instr.	16.98 / 41.51 / 70.67	8.16 / 51.02 / 49.77	12.75 / 46.08 / 55.17
Qwen3-VL-235B-Think.	30.00 / 65.00 / 76.09	3.23 / 43.55 / 43.84	13.73 / 51.96 / 51.90
GLM-4.5V	8.11 / 51.35 / 38.95	0.00 / 30.77 / 29.17	2.94 / 38.24 / 31.59
Kimi-K2.5	52.94 / 73.53 / 89.77	26.15 / 49.23 / 79.51	35.35 / 57.58 / 81.91
GPT-5-Mini	29.73 / 62.16 / 84.21	21.54 / 50.77 / 74.65	24.51 / 54.90 / 77.02
GPT-5.2	58.33 / 77.78 / 91.30	29.23 / 49.23 / 86.46	39.60 / 59.41 / 87.63
Grok-4.1	15.38 / 66.67 / 63.10	6.35 / 39.68 / 51.07	9.80 / 50.00 / 53.85
Gemini-2.5-Pro	27.03 / 64.86 / 75.79	7.69 / 46.15 / 57.29	14.71 / 52.94 / 61.88
Human	91.43 / 97.14 / 96.67	81.36 / 94.42 / 95.79	85.11 / 95.74 / 96.01

Ablation Study¶

The first group is the process-level diagnostic (Table 3 of the paper): evidence correctness EC (higher is better) and minimality gap MG (lower is better), restricted to models with complete tool traces.

Model	Easy EC / MG	Hard EC / MG	Average EC / MG
Qwen3-VL-235B-Instr.	64.63 / 1.41	63.48 / 1.03	63.85 / 1.23
Qwen3-VL-235B-Think.	54.55 / 1.02	45.66 / 1.00	48.63 / 1.01
GLM-4.5V	24.49 / 2.60	31.20 / 2.60	29.31 / 2.60
Kimi-K2.5	81.30 / 1.49	84.91 / 1.20	84.13 / 1.35
GPT-5-Mini	75.40 / 3.00	82.43 / 1.94	80.75 / 2.32
GPT-5.2	85.93 / 1.34	86.15 / 1.27	86.10 / 1.30
Grok-4.1	85.71 / 3.33	78.33 / 2.88	80.12 / 3.07
Gemini-2.5-Pro	71.30 / 2.17	68.71 / 1.41	69.39 / 1.80
Human	83.45 / 1.00	88.35 / 1.02	86.88 / 1.01

The second group is the protocol-side ablations. The closed-book no-tool ablation has fairly complete numbers: GPT-5.2 drops from 39.6% to 6.9% average ESR, Kimi-K2.5 from 35.4% to 6.9%, and Qwen3-VL-235B-Thinking from 13.7% to 2.9%, while GLM-4.5V stays low in both settings; on the Hard split the no-tool ESR is at most 3.1% for every model in this ablation, against 29.2% for GPT-5.2 and 26.2% for Kimi-K2.5 with tools enabled. The other two ablations (enabling tools on the final turn, and removing PDF text RAG) are reported in the paper as trends in Figure 5 without concrete numbers, so the table below records directions only.

Config	Key metric	Note
Main protocol (tools disabled on the final turn)	GPT-5.2 average ESR 39.60	final turn may only use evidence in the memory pool
Tools enabled on the final turn (upper-bound reference)	large recovery in ESR and Acc_final	multimodal evidence cached across turns is insufficient for reliable final fusion
Removing PDF text RAG	smaller, model-dependent effect	text retrieval is not the main bottleneck for these episodes
Closed-book, no tools (GPT-5.2)	39.6 → 6.9 ESR	a 32.7-point drop; parametric knowledge cannot solve the benchmark
Closed-book, no tools (Kimi-K2.5)	35.4 → 6.9 ESR	a 28.5-point drop
Closed-book, no tools (Qwen3-VL-235B-Think.)	13.7 → 2.9 ESR	a 10.8-point drop
Step budget 5 → 10	consistent ESR improvement	a tighter budget often truncates before evidence acquisition and recovery finish
Step budget 10 → 15	ESR nearly unchanged, max-steps errors barely fall	remaining failures are not budget-induced but evidence grounding, alignment, and reuse problems

Key Findings¶

High turn-level accuracy does not mean an episode can be finished. Several closed-source models already reach 84%–91% Acc_pre, yet Acc_final falls to 50%–59% and ESR sits at only 10%–40%. Losses concentrate after evidence acquisition — organizing, recalling, and aligning evidence under constraints. GPT-5.2 is the only model near 30% ESR on Hard (29.23%), but the gap to the human baseline at 81.36% remains enormous; humans reach 91.43% ESR on Easy and 85.11% on average with an MG near 1.01, showing that the minimal viable path is attainable.
EC and ESR do not rank models identically, so some correct answers are lucky. GPT-5.2 (average EC 86.10) and Kimi-K2.5 (84.13) more often ground their correct answers in sufficient evidence, whereas GLM-4.5V has an average EC of only 29.31 while still scoring 2.94 ESR, a clear sign of reliance on priors or guessing. Grok-4.1 is a counterexample in the other direction: its average EC of 80.12 is not low (85.71 on Easy, close to GPT-5.2), yet its average ESR is only 9.80 — it reads the right evidence but does not convert it, with failures falling later, in integration and reuse.
Redundancy does not track difficulty. MG varies widely (GLM-4.5V 2.60, Grok-4.1 3.07 versus GPT-5.2 1.30 and Kimi-K2.5 1.35) and correlates only weakly with EC; several models are actually more redundant on Easy than on Hard (GPT-5-Mini 3.00 → 1.94, Grok-4.1 3.33 → 2.88), indicating over-exploration on simpler tasks. The human MG of 1.01 shows the metric has a solid ceiling.
Where agents break down shifts with capability. Weaker models (Gemini-2.5-Pro, Qwen3-VL-Thinking) are dominated by Retrieve errors — they are often out before multi-evidence fusion even becomes decisive. Stronger reasoning-oriented models push the error mass later: GPT-5.2 and Kimi-K2.5 carry larger shares of Perception and Reasoning errors. GPT-5.2 also has a higher Reading Memory error rate, which the authors explain by its stronger early acquisition bringing it to memory-reuse turns more often — in other words it is tested at that step more frequently rather than being worse, and the result still highlights substantial room for improvement in reusing multimodal evidence from memory.
Tools are necessary. With tools disabled throughout, ESR collapses for every model (at most 3.1% on Hard), while enabling tools on the final turn produces a large recovery. The two ablations point in opposite directions and jointly localize the bottleneck to the selection, indexing, and alignment of already-cached multimodal evidence across turns, rather than to retrieval itself or text RAG.

Highlights & Insights¶

Memory reuse is turned into an enforceable constraint rather than a post-hoc statistic. Disabling every tool on the final turn and allowing only memory reads converts "reuse" from a vague behavioral notion into a protocol-level, forcibly triggered test condition. Any long-horizon agent evaluation can borrow this move: instead of counting whether earlier information was used, simply cut off the way back.
Annotating evidence units at paper-plus-figure/table granularity makes lucky guesses visible. EC counts only correctly answered turns in its denominator and only genuinely accessed required evidence in its numerator, which naturally separates "right answer, wrong basis" from accuracy. The idea — splitting correctness into an outcome metric and a provenance metric — transfers to retrieval-augmented QA and code-generation attribution: label each correct answer with a set of required sources and check whether the model's trace covers them.
A single minimal-path checklist plays three roles at once. The tool-call checklist from drafting is simultaneously the handle for human verification, the reference for MG, and part of the difficulty definition. Reusing one annotation to support both quality control and an efficiency metric avoids building separate annotations per metric.
"Earliest deviation" attribution makes failure diagnosable. Giving each failed episode exactly one label at the earliest point of departure in the trace prevents several error chains from contaminating the statistics, and it is what yields the finding that weak models fail at retrieval while strong models fail at perception and reasoning — a distribution that accuracy numbers alone cannot reveal.

Limitations & Future Work¶

The scale is modest: 102 episodes (Easy 37 / Hard 65) drawn from a corpus pool of 485 papers, with only 68 seeds. That is enough as a diagnostic instrument, but for model selection or as a training signal, 37 Easy episodes put the ESR resolution at roughly 2.7 percentage points, so small gaps should not be over-interpreted.
Scoring relies on a single LLM judge (GPT-5.2) producing binary labels under a fixed rubric; the paper reports no agreement between the judge and human labels, no prompt-sensitivity analysis, and no cross-judge robustness, even though Acc_final, Acc_pre, and ESR all rest on that judgment.
For error labels the paper only states that attribution follows the earliest point of deviation in the trace, without saying whether it is rule-based, human, or model-assisted, and without reporting annotator agreement; the boundaries between the five categories (notably Reasoning versus Reading Memory during final fusion) leave room for subjectivity.
The construction used GPT-5.2 to draft question chains. Human curation removes obvious problems, but the drafting model is the same family as one of the evaluated systems, creating a potential style bias in phrasing and evidence selection that the paper neither analyzes nor cross-validates.
Difficulty has only two levels, Easy and Hard, with no intermediate tier and no fine-grained decay curve across turns, which makes it hard to localize which turn a model's performance actually degrades at.
Directions for improvement: first, refine EC from "was the figure or table accessed" to "was the right region accessed," so perception errors also become visible to a process metric; second, extend to more disciplines and longer episodes (from single-digit to a dozen-plus turns) to test memory decay; third, model MG jointly with correctness to give a Pareto view of efficiency versus accuracy instead of reporting the two numbers separately.

vs SPIQA / CharXiv / SciVQA / LiveXiv: they pose figure-and-table questions inside a single paper, often answerable from one artifact and never touching the citation network between papers; EpiBench requires proactively searching out several papers from a cue and aligning evidence across multiple figures and tables before an answer is possible, and it evaluates whether the whole research process was carried through rather than whether one question was answered.
vs MMDU / MULTIVERSE / VisChainBench: these are multi-image multi-turn dialogue benchmarks probing multi-image understanding and contextual reasoning within a conversation, but images are handed over up front, no proactive search is required, and there is no tool use or evidence-grounding check; EpiBench makes "finding out which figure to look at" part of the test and has the agent extract the figure from the paper itself.
vs Memory-QA / EMP: they focus on recalling an answer from an already stored memory, with memory given as a premise; EpiBench lets the agent fill memory through its own tool calls and then forces it to use that self-built memory via the tool-disabled final turn, so memory quality and retrieval behavior are evaluated together.
vs PaperArena: the closest work, also doing tool-augmented multi-hop navigation over academic papers; the difference is that its tasks less often require jointly integrating multiple figures and tables, and it lacks the process-level evidence-reuse constraint and metrics such as EC and MG. EpiBench's increment lies mainly in the multimodal coupling of its tasks and the process diagnosability of its protocol.
vs MMDeepResearch / MMSearch-Plus / SIN-Bench: the first covers cross-paper and multimodal settings without evidence memory or multi-turn dialogue; SIN-Bench targets evidence chains in long multimodal documents; EpiBench is the only benchmark in the paper-centric setting that opens all five dimensions — cross-paper, multimodal, proactive search, evidence memory, and multi-turn dialogue — at once.

Rating¶

Novelty: ⭐⭐⭐⭐ Binding proactive search, cross-paper multimodal evidence fusion, and a memory-restricted final turn into one episodic task, with automatically computable process metrics for evidence reuse, is a new combination among paper-centric benchmarks.
Experimental Thoroughness: ⭐⭐⭐⭐ Ten open- and closed-source backbones plus a human baseline, with main results, process diagnostics, error attribution, and three ablations; however the ablations are mostly reported as trends (no concrete numbers in Figures 5 and 7) and there is no judge-agreement or scale-sensitivity analysis.
Writing Quality: ⭐⭐⭐⭐ The problem is sharply localized (the capability comparison in Table 1 is immediately legible) and the metrics are fully defined with formulas; weaker points are that several statistics appear only as figures without textual expansion, and reproducibility details (error-attribution procedure, judging rubric) are thin.
Value: ⭐⭐⭐⭐ The benchmark is small, but the protocol design — tool-disabled final turn, evidence-unit annotation, minimal-path checklist — transfers directly to other long-horizon agent evaluations, and its diagnostic value exceeds its leaderboard value.