AnaBench for Enhancing Scientific Table & Figure Analysis¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: scientific table & figure analysis, multi-agent collaboration, ANABENCH benchmark, modular reinforcement learning, multimodal long-context reasoning
TL;DR¶
This paper introduces ANABENCH, a large-scale benchmark of 63,178 instances across 9 scientific domains and 170 subdisciplines categorized along seven complexity dimensions, and proposes ANAGENT, a four-agent collaborative framework (PLANNER, EXPERT, SOLVER, CRITIC) with modular SFT and GRPO training that outperforms baselines by up to +42.12%.
Background & Motivation¶
In modern scientific research, tables and figures serve as high-density visual and structural vehicles that carry vital empirical comparisons, system architectures, parametric curves, and evidence supporting hypothetical claims. With multimodal large language models (MLLMs) rapidly adopted in research assistance, hypothesis exploration, and automated literature synthesis, there is growing momentum toward developing "AI Co-Scientists" capable of independently digesting scientific literature and generating sound analytical prose. However, real-world scientific figures and tables exhibit extreme heterogeneityโranging from raw LaTeX tables and XML encodings to statistical bar charts, t-SNE scatterplots, and intricate architectural flowchartsโwhile sound interpretation frequently hinges upon synthesizing long-context narratives spanning multiple sections and external citations. Current MLLMs struggle severely with such compound inputs, frequently stumbling over perceptual misalignments, layout parsing failures, specialized terminology misinterpretations, and cascading factual hallucinations.
A primary bottleneck impeding research progress is the critical lack of both comprehensive benchmarks mirroring real scientific rigor and dedicated agentic reasoning paradigms. Existing benchmarks for scientific visual understanding (such as SCITAB, SPIQA, WildSci, and S2abEL) typically reduce chart and table comprehension to isolated, narrow downstream tasks like single-hop visual question answering, binary claim verification, or superficial caption generation. Consequently, they fail to evaluate key cognitive dimensions intrinsic to scientific analysis writing: the spectrum of analytical depth (superficial descriptive reporting versus deep deductive reasoning), analysis width (self-contained artifact inspection versus complex synthesis integrating internal narrative and external literature), and analytical objectives (methodological architecture explanation versus empirical benchmark trend interpretation). Concurrently, deploying monolithic MLLMs to execute end-to-end scientific analysis forces a single model to juggle perception, tool-augmented retrieval, deep deduction, and editorial synthesis simultaneously, routinely precipitating cognitive overload, ungrounded tool invocations, and missed factual verification.
Addressing this fundamental tension between document heterogeneity and analytical depth, the authors draw inspiration from how human researchers deliberately examine scientific artifacts, recasting table and figure analysis as a context-aware open-ended generation task. Rather than relying on fragile monolithic end-to-end models, the authors decouple the cognitive process into modular roles equipped with specialized scientific toolkits, establishing a structured "plan-retrieve-reason-critique" feedback loop. Core idea: construct ANABENCH, a 63k-scale benchmark spanning 9 broad domains, 170 subdisciplines, and 7 complexity dimensions, and introduce ANAGENT, a collaborative multi-agent architecture (PLANNER, EXPERT, SOLVER, CRITIC) optimized via modular supervised finetuning and tailored GRPO reinforcement learning that substantially outperforms test-time compute scaling.
Method¶
Overall Architecture¶
ANAGENT formalizes scientific table and figure analysis as a context-aware generation problem: given multimodal inputs \(x \in \{\{x_t\}, \{x_f\}, \{x_t, x_f\}\}\) comprising one or multiple tables \(x_t\) and/or figures \(x_f\), source document context \(s\), and a task query \(q\) specifying analytical objectives and depth requirements, the goal is to synthesize an evidence-grounded, coherent, and domain-accurate analytical narrative \(y\).
To mirror the workflow of human scientists, ANAGENT operates across four interactive stages: in Stage 1, PLANNER decomposes the high-level challenge into a structured sequence of executable subtasks; in Stage 2, EXPERT executes task-oriented multi-turn information gathering across 5 scientific toolkits (16 specialized tools) to retrieve relevant multimodal context; in Stage 3, SOLVER synthesizes the accumulated knowledge, raw visual/tabular inputs, and source context into a structured candidate analysis; in Stage 4, CRITIC formally evaluates the candidate text against a five-dimensional scientific assessment protocol and provides targeted corrective critique, prompting iterative refinement until convergence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Scientific Tables & Figures with Context<br/>x, s, q"] --> B["Task Decomposition & Strategic Planning<br/>PLANNER parses scope and generates subtask sequence"]
B --> C["Tool-Assisted Knowledge Retrieval<br/>EXPERT invokes 16 tools across 5 scientific toolkits"]
C --> D["Context-Aware Evidence Synthesis<br/>SOLVER fuses retrieved data into candidate analysis"]
E["Five-Dimensional Reflective Refinement<br/>CRITIC audits draft against 5 quality dimensions"]
D --> E
E -->|Critique feedback loop for refinement| D
E -->|Meets quality protocol or max turns| F["Final Scientific Analysis Narrative y"]
Key Designs¶
1. Task Decomposition & Strategic Planning: Aligning Analytical Scope and Execution Trajectories Monolithic MLLMs frequently suffer from tunnel vision or cognitive divergence when presented with dense scientific tables, resulting in incomplete coverage, missed methodological nuances, or superficial descriptive restatements. PLANNER resolves this by establishing a top-down semantic blueprint before any draft generation occurs. It explicitly evaluates the required analysis width (self-contained, internal context, or external references), depth (shallow restatement vs. in-depth causal inference), and objective (methodology interpretation vs. empirical performance analysis), decomposing the query \(q\) into an ordered sequence of \(M_p\) actionable subtasks \(\tau_1, \tau_2, \dots, \tau_{M_p}\). During training, PLANNER is framed as a multi-choice preference selection problem, optimized to select decomposition strategies empirically validated to maximize downstream analytical quality over baseline plans.
2. Tool-Assisted Knowledge Retrieval: Active Exploration of Heterogeneous Scientific Modalities Interpreting scientific tables and figures requires resolving domain-specific nomenclature, math symbols, nuanced measurement units, and ambiguous visual detailsโchallenges that static parameters cannot reliably address. EXPERT tackles this knowledge bottleneck by orchestrating 5 dedicated toolkits encompassing 16 tools: Document Toolkits (PDF/XML hierarchical parsers), Knowledge Toolkits (bidirectional citation context finders, section extractors), Search Toolkits (arXiv, PubMed, Semantic Scholar, web, and Wikipedia interfaces), Vision Toolkits (OCR extractors, figure parsers, interactive image analyzers), and a Sandbox Toolkit (isolated Python execution). For each subtask \(\tau_e\), EXPERT reads the current knowledge state \(K_{e-1}\) and invokes tools with tailored parameters to dynamically expand the evidence repository: \(K_e = K_{e-1} \cup \text{EXPERT}(\tau_e, K_{e-1})\). Decoupling retrieval from narrative generation insulates the agent from context dilution and anchors subsequent reasoning in verifiable facts.
3. Context-Aware Evidence Synthesis: Cross-Modal Alignment and Structured Analysis Writing Scientific analysis requires more than enumerating numbers; it demands rigorous narrative synthesis that connects empirical shifts to underlying algorithmic principles. SOLVER receives the raw inputs \(x\), source text \(s\), query \(q\), accumulated evidence \(K_n\), and feedback \(f_{i-1}\), generating structured analysis drafts: \(y_i = \text{SOLVER}(x, s, q, K_n, f_{i-1})\). SOLVER actively establishes relational bridges between tabular variances (such as an ablation drop) and textual methodologies (such as the removal of a specific loss term). In modular training, SOLVER is optimized with token-level SciBERT semantic similarity rewards and an explicit length-compliance reward window, preventing models from defaulting to brief, trivial summaries and encouraging thorough scientific argumentation.
4. Five-Dimensional Reflective Refinement: Rigorous Self-Correction and Feedback Loops Raw drafts produced by generative models routinely contain subtle hallucinations, flipped empirical comparisons, or ungrounded speculative assertions. CRITIC addresses this vulnerability by conducting post-hoc verification decoupled from the generation process. Operating under a rigorous five-dimensional protocolโevaluating content accuracy, analytical completeness, format compliance, clarity and coherence, and grounding faithfulnessโCRITIC systematically audits draft \(y_i\) against input data \(x\), context \(s\), and evidence \(K_n\). Rather than returning a coarse scalar, it formulates concrete, paragraph-specific revision feedback \(f_i = \text{CRITIC}(y_i, x, s, q, K_n)\). Through iterative refinement loops between SOLVER and CRITIC, analytical defects are progressively excised until the output satisfies peer-review standards of rigor.
Loss & Training¶
ANAGENT develops a two-stage modular training strategy to prevent role collapse and loss of specialization common in end-to-end multi-agent tuning:
-
Shared Supervised Finetuning (SFT): All agents are initialized via SFT on a downsampled training corpus of 31,350 instances from ANABENCH, training for 1 epoch with learning rate \(1 \times 10^{-4}\) to minimize negative log-likelihood against human author-written ground-truth analyses \(y^*\): $\(L_{\text{SFT}}(\theta) = \mathbb{E}_{(x, s, q, y^*)} \left[ -\sum_{t=1}^{|y^*|} \log p_\theta(y^*_t \mid y^*_{<t}, x, s, q) \right]\)$
-
Agent-Specific Reinforcement Learning (GRPO Optimization): Initialized from SFT policies, each agent \(a \in \{\text{PLANNER}, \text{EXPERT}, \text{SOLVER}, \text{CRITIC}\}\) is tuned via Group Relative Policy Optimization (GRPO) on dedicated datasets using learning rate \(1 \times 10^{-6}\): $\(L_{\text{RL}}(\theta_a) = -\mathbb{E}_{\xi_a} \left[ \frac{1}{K} \sum_{k=1}^K \hat{A}_a^{(k)} \log \pi_{\theta_a}(z_a^{(k)} \mid \xi_a) \right]\)$
Specialized reward functions \(R_a = \sum_m \lambda_{a,m} r_{a,m}\) are engineered to reward agent-specific competence: - PLANNER: \(R_{\text{PLANNER}} = \lambda_{Pf} r_{Pf}(z_a) + \lambda_{Pacc} r_{Pacc}(z_a, z_a^*)\), verifying JSON schema format and rewarding selection accuracy against performance-validated plans. - EXPERT: \(R_{\text{EXPERT}} = \lambda_{Ef} r_{Ef}(z_a) + \lambda_{Eacc} r_{Eacc}(z_a, z_a^*)\), assessing syntactic tool formatting and enforcing exact tool type matches and parameter validity: \(I[T(z_a) = T(z_a^*)] \cdot \rho(z_a, z_a^*)\). - SOLVER: \(R_{\text{SOLVER}} = \lambda_{Sf} r_{Sf} + \lambda_{Slen} r_{Slen} + \lambda_{Sacc} r_{Sacc}\), incorporating length window compliance \(r_{Slen} = I[0.5|z_a^*| \le |z_a| \le 1.5|z_a^*|]\) and SciBERT token-level cosine harmonic mean similarity \(r_{Sacc}\). - CRITIC: \(R_{\text{CRITIC}} = \lambda_{Cf} r_{Cf} + \lambda_{Cacc} r_{Cacc}\), rewarding correct identification of analytical flaws matching reference critiques.
Key Experimental Results¶
Main Results¶
ANABENCH evaluation is performed on 7,319 instances filtered exclusively from 2025 publications across 9 scientific domains, 170 subdisciplines, and all 7 complexity dimensions. Evaluation reports lexical metrics (\(S_{\text{LEX}}\): average of ROUGE-L, BLEU, Word Overlap), semantic metrics (\(S_{\text{SEM}}\): average of Cosine, SciBERT, METEOR), and the global aggregate \(S_{\text{AVG}}\).
The table below contrasts direct generation baselines against training-free ANAGENT and modularly finetuned variants (excerpted from Table 1 and Table 2 of the paper):
| Backbone Model | Scale | Configuration Stage | Semantic \(S_{\text{SEM}}\) (%) | Lexical \(S_{\text{LEX}}\) (%) | Overall \(S_{\text{AVG}}\) (%) | Relative Gain \(\Delta_{\text{rel}}\) |
|---|---|---|---|---|---|---|
| GPT-4.1-mini | API | Direct Baseline | 45.18 | 10.54 | 27.86 | โ |
| GPT-4.1-mini | API | ANAGENT (Zero-Shot) | 48.11 | 11.75 | 29.93 | +7.43% |
| GPT-4.1-mini | API | ANAGENT (One-Shot) | 49.47 | 12.98 | 31.22 | +12.06% |
| Gemini-2.5-Flash | API | Direct Baseline | 42.47 | 9.20 | 25.84 | โ |
| Gemini-2.5-Flash | API | ANAGENT (One-Shot) | 47.64 | 10.98 | 29.31 | +13.43% |
| Qwen2.5-VL | 3B | Direct Baseline | 43.68 | 9.49 | 26.59 | โ |
| Qwen2.5-VL | 3B | ANAGENT (One-Shot) | 47.26 | 11.31 | 29.29 | +10.15% |
| Qwen2.5-VL | 3B | ANAGENT (SFT, One-Shot) | 51.02 | 17.69 | 34.35 | +29.18% |
| Qwen2.5-VL | 3B | ANAGENT (SFT+RL, One-Shot) | 52.40 | 20.29 | 36.34 | +36.67% |
| Qwen3-VL | 4B | Direct Baseline | 43.99 | 9.53 | 26.76 | โ |
| Qwen3-VL | 4B | ANAGENT (Zero-Shot) | 46.95 | 10.50 | 28.73 | +7.36% |
| Qwen3-VL | 4B | ANAGENT (One-Shot) | 47.56 | 11.20 | 29.38 | +9.79% |
| Qwen3-VL | 4B | ANAGENT (SFT, Zero-Shot) | 52.19 | 19.85 | 36.02 | +34.60% |
| Qwen3-VL | 4B | ANAGENT (SFT+RL, Zero-Shot) | 53.08 | 20.81 | 36.95 | +38.08% |
| Qwen3-VL | 4B | ANAGENT (SFT+RL, One-Shot) | 54.18 | 21.91 | 38.03 | +42.12% |
| Qwen3-VL | 8B | Direct Baseline | 44.73 | 10.16 | 27.44 | โ |
| Qwen3-VL | 8B | ANAGENT (One-Shot) | 49.15 | 12.98 | 31.07 | +13.23% |
| InternVL-3.5 | 8B | Direct Baseline | 44.71 | 9.98 | 27.34 | โ |
| InternVL-3.5 | 8B | ANAGENT (One-Shot) | 48.52 | 12.22 | 30.37 | +11.08% |
Ablation Study¶
To investigate whether ANAGENT's improvements stem from collaborative division of labor rather than brute inference compute scaling, and to validate the modular training recipe over end-to-end tuning, the authors conducted controlled compute-accounting and architectural ablations (excerpted from Table 3 and Table 15):
| Experiment Group | Method / Configuration | Backbone | MLLM Calls | Tokens/Call Ratio | Latency Ratio | Overall \(S_{\text{AVG}}\) (%) | Relative Gain \(\Delta_{\text{rel}}\) |
|---|---|---|---|---|---|---|---|
| Compute Controls | Direct Single-Pass | Qwen3-VL-4B | 1.00 | 1.00ร | 1.00ร | 26.76 | โ |
| Compute Controls | Self-Consistency (SC@10) | Qwen3-VL-4B | 10.00 | 1.18ร | 7.09ร | 26.82 | +0.22% |
| Compute Controls | Best-of-N (BoN@10, Oracle) | Qwen3-VL-4B | 10.00 | 1.07ร | 7.11ร | 27.03 | +1.01% |
| Compute Controls | Self-Refine (SR@10) | Qwen3-VL-4B | 10.00 | 4.06ร | 7.47ร | 26.79 | +0.11% |
| Agent Ablations | Omnion (Flat Tools, Training-Free) | Qwen3-VL-4B | 5.48 | 4.87ร | 7.26ร | 22.87 | -14.54% |
| Agent Ablations | ANAGENT (Complete 4-Agent, Training-Free) | Qwen3-VL-4B | 6.64 | 4.52ร | 6.24ร | 28.73 | +7.36% |
| Agent Ablations | ANAGENT (Modular SFT) | Qwen3-VL-4B | 6.79 | 4.38ร | 6.36ร | 36.02 | +34.60% |
| Training Recipe | ANAGENT (End-to-End Joint Tuning) | Qwen3-VL-4B | 6.79 | 4.41ร | 6.38ร | 32.02 | +19.66% |
| Training Recipe | ANAGENT (Modular SFT + RL) | Qwen3-VL-4B | 6.79 | 4.38ร | 6.36ร | 38.03 | +38.08% |
Key Findings¶
- Unstructured Tool Exposure Induces Catastrophic Confusion: In training-free settings, granting a standalone SOLVER access to all 16 tools (the Omnion variant) degrades performance by 14.54% relative to direct baseline generation (\(S_{\text{AVG}}\) plunges from 26.76% to 22.87%). Without high-level planning, expansive toolspaces overwhelm MLLMs with redundant context and invalid execution calls.
- Multi-Agent Orchestration Outperforms Pure Test-Time Compute Scaling: Allocating equal or greater inference compute to single-model scaling strategies yields negligible benefits (SC@10 and SR@10 gain only +0.22% and +0.11% across 10 calls; even oracle-selected BoN@10 reaches only +1.01%). In contrast, ANAGENT consumes an average of 6.64 calls to achieve a +7.36% boost training-free, surging to +34.60% to +42.12% post-finetuning. This confirms that structural role decomposition and active retrieval surpass blind sampling.
- Modular Training Rescues Role Specialization: Modular training significantly outpaces end-to-end tuning (+38.08% vs. +19.66% on Qwen3-VL-4B). Qualitative audits show that end-to-end optimization leads to role erosion, where upstream PLANNER agents begin prematurely generating incomplete answers, degrading the downstream division of labor.
- Scientific Tables Are Harder Than Figures: Across all models, performance on tables is consistently 1.47% to 4.83% lower than on figures. The discrete, multi-axis, and non-continuous relational topology of scientific tables presents greater comprehension hurdles than continuous visual graphics.
Highlights & Insights¶
- Formulates Scientific Analysis as Open-Ended Academic Writing: Transcends conventional benchmarks that reduce multimodal understanding to synthetic QA or classification, extracting authentic author-written discussion prose to establish an ecologically valid evaluation suite for scientific discourse.
- Rigorous Compute Accounting Demystifies Multi-Agent Value: Provides an empirical refutation of the hypothesis that multi-agent gains merely reflect higher token consumption, showing that structured collaborative trajectories unlock capabilities unreachable by self-consistency or iterative self-refinement.
- Finetuning Optimizes Strategic Tool Selection: After modular alignment, ANAGENT increases its invocation of vision tools in figure-centric tasks by 16.39% while substantially reducing failed API calls, proving that specialized RL fosters goal-directed tool utilization.
Limitations & Future Work¶
- Inference Latency and Resource Footprint: While more compute-efficient than brute-force sampling, multi-agent collaboration requires approximately 6.6 to 6.8 MLLM calls and over 6ร latency compared to direct single-pass generation, constraining deployment in ultra-low-latency real-time applications.
- Platform and Disciplinary Distributional Skew: ANABENCH draws primarily from open repositories (arXiv and PubMed), resulting in heavier representation of computer science, physics, and biomedicine, with limited coverage of non-English scholarship, patents, or humanities.
- Multi-Hop External Citation Bottlenecks: Error analysis reveals that tasks requiring synthesis of external cross-document citations remain the most prone to logical gaps, indicating the need for graph-augmented scientific literature retrievers and specialized symbolic table engines.
Related Work & Insights¶
- vs SCITAB / SCITAT / SPIQA: While SCITAB tackles binary claim verification and SPIQA focuses on isolated multimodal QA, ANABENCH covers 9 broad disciplines across 170 subdomains and formalizes chart analysis as multi-paragraph deductive synthesis across 7 complexity tiers.
- vs Monolithic Tool-Augmented Agents (Omnion): Traditional methods prompt a single agent with all tools, inducing catastrophic attention interference and execution failures; ANAGENT strictly isolates high-level planning from low-level tool invocation, demonstrating that role separation resolves cognitive overload.
- Insights: For sophisticated multimodal domain intelligence, combining modular agent specialization with targeted GRPO alignment offers a generalizable paradigm readily extensible to automated experimental design, code-to-paper verification, and autonomous peer review.
Rating¶
- Novelty: โญโญโญโญโญ (Pioneers a 7-dimensional complexity curriculum for scientific chart analysis alongside a fully decoupled multi-agent collaborative framework)
- Experimental Thoroughness: โญโญโญโญโญ (Encompasses 8 major commercial/open MLLMs, 63k-scale data, five-dimensional LLM-as-judge, multi-discipline human expert verification, and rigorous compute-controlled baselines)
- Writing Quality: โญโญโญโญโญ (Exceptionally well-structured paper featuring lucid problem formulation, comprehensive ablation studies, and insightful failure taxonomy)
- Value: โญโญโญโญโญ (Sets an enduring benchmark and design blueprint for next-generation AI Co-Scientists handling complex multimodal literature)