Skip to content

ABDATABENCH: Evaluating AI Agents for Evidence-Grounded Scientific Data Analysis

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/GAIR-NLP/ABDataBench
Area: LLM Agent
Keywords: Scientific Data Construction, Antibody Benchmark, Multi-Agent Collaboration, Evidence Provenance, Skill Accumulation

TL;DR

Introduces ABDATABENCH, the first benchmark assessing AI agents on constructing complete, evidence-grounded antibody records across scientific literature and patents (44 documents, 279 records, 22-field tiered schema with Hungarian alignment and omission penalties), alongside ABCURATOR, a record-centric multi-agent framework achieving 78.3 (proprietary) and 76.0 (open-source) through deterministic tool integration and iterative skill accumulation.

Background & Motivation

AI agents are increasingly envisioned as accelerators for autonomous scientific discovery, yet their practical utility depends far less on open-ended hypothesis generation than on resolving a foundational bottleneck: transforming dispersed, unstructured evidence across scientific literature into high-fidelity, machine-actionable datasets. In therapeutic antibody development, this bottleneck is particularly acute. Modern biomolecular foundation models, deep-learning structure prediction, de novo protein design, and language-model-guided affinity maturation all require rigorous supervision that precisely pairs primary amino acid sequences with experimental binding kinetics. However, vital antibody attributesโ€”including variable heavy and light chains (VH, VL), complementarity-determining regions (CDRH3), chain pairing, target epitopes, binding constants (\(K_D\), \(k_{on}\), \(k_{off}\)), cross-reactivity, and in vivo efficacyโ€”remain scattered across heterogeneous prose, multi-column patent claims, supplementary tables, sequence alignment figures, and external structural repositories.

Curation of antibody evidence demands zero-tolerance precision. A single amino acid mutation often determines whether a candidate antibody is functional or non-viable, while a misattributed equilibrium dissociation constant (\(K_D\)) or \(EC_{50}\) value binds incorrect kinetic properties to an otherwise valid biological entity. Conventional biomedical relation extraction benchmarks typically operate at the sentence or abstract level over flat, coarse schemas, completely bypassing the complex challenge of synthesizing multi-modal evidence across dozens of pages. Furthermore, real-world literature is riddled with shifting nomenclature aliases, non-standard unit scales (spanning picomolar to micromolar ranges), and fragmented image-based sequence alignments. Existing agent benchmarks evaluate open-ended coding or task reproduction, offering neither tiered schema scoring nor calibrated penalties for hallucination and omission.

The core insight of this work is to discard document- or mention-centric extraction and instead establish the individual antibody record as the atomic unit of coordination, while routing closed-form operations (regular expressions, schema validation, and PDB/GenBank parsing) to auditable deterministic tools. Core idea: Establish ABDATABENCH as the first benchmark evaluating complete, multi-field antibody records via Hungarian matching, tiered weights, and asymmetric omission penalties, and develop ABCURATOR, a record-centric multi-agent framework paired with expert-in-the-loop skill accumulation to resolve long-horizon scientific evidence binding.

Method

Overall Architecture

ABCURATOR ingests OCR Markdown, supplementary files, and linked figure images, outputting structured records formatted for downstream antibody engineering. The entire pipeline operates around the antibody record as the central unit of coordination across document conditioning, paginated skeleton extraction, parallel multi-channel evidence enrichment, biological constraint validation, and bounded review loops. Crucially, the system features a dual-loop design: an inner operational loop handles automated extraction, rule validation, and targeted retry, while an outer co-evolution loop distills expert review of residual failures into versioned prompt skill modules loaded into agent instructions for subsequent curation rounds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Document Input<br/>OCR Markdown + Patents + Figures"] --> B["Document Conditioning & Guidance<br/>Layout repair & regex beacon scanning"]
    B --> C["Paginated Skeleton Construction<br/>Candidate antibody entity generation"]
    C --> D["Parallel Evidence Enrichment<br/>Tables + Image VLM + PDB database"]
    D --> E["Biological Rule Validation & Bounded Review<br/>Hard constraint checks & targeted retry"]
    E --> F["Expert Skill Accumulation Workflow<br/>Failure distillation & versioned skill prompts"]
    F --> G["Standardized Antibody Records<br/>22-field tiered structured output"]

Key Designs

1. Record-Centric Paginated Skeleton Construction: Context Window Degradation and Entity Duplication -> Incremental Chunk Generation with State Tracking -> Establishing Atomic Units and Suppressing Redundancy When processing long-context patent filings exceeding 50 pages, generic LLMs easily suffer from entity boundary drifting and memory saturation. ABCURATOR strictly defines an extraction unit as an evidence-grounded record pairing a single antibody entity with its target, assay, condition, and verbatim quotes. The Skeleton Agent utilizes a paginated invocation protocol, generating at most \(k\) candidate records per step while maintaining a dynamic ledger of processed antibody names. Pre-extraction focus analysis categorizes prospective entities into core study antibodies versus reference comparators, preventing cross-record pollution and deduplicating recurring mentions across different experimental sections.

2. Determinism-First Parallel Evidence Enrichment: Dispersed Multi-Modal Evidence -> Dedicated Multi-Channel Retrieval and Priority Merging -> Decoupling Reasoning from Computation to Prevent Hallucination Critical antibody measurements are distributed across diverse formats: sequences often reside in alignment figures, kinetic curves in tables, and structures in external repositories. ABCURATOR decouples semantic interpretation from deterministic data processing. Tables are parsed via native Python tools; external databases (PDB, GenBank) are queried via dedicated API fetching modules with automated linker/tag filtering; sequence figures are processed by a specialized Vision-Language Model. Extracted evidence is merged into the record skeleton under strict local-attribution priority rules: evidence is bound only when explicitly anchored to the local antibody identifier, strictly preventing family-level properties from broadcasting across sibling records.

3. Biological Constraint Validation and Bounded Review: Stochastic Output Flaws -> Rule-Driven Hard Verification and Targeted Repair -> Establishing a Self-Correcting Closed Loop To guarantee physical validity, ABCURATOR integrates a rule-based Validator that operates downstream of LLM generation. The validator enforces strict biomedical checks, including standard 20-amino-acid alphabet legality, physiological heavy/light chain length boundaries, CDRH3 motif anchor consistency, and cross-record duplicate detection, emitting pass, warn, or fail flags. Upon failure, a Reviewer Agent generates granular, field-specific corrective instructions, prompting the Orchestrator to rerun extraction under localized constraints. The retry loop operates within a bounded step budget, ensuring convergence while preventing runaway API overhead.

4. Expert-Guided Skill Accumulation Workflow: Brittleness of Static System Prompts Across Heterogeneous Literature -> Failure Distillation into Versioned SOP Modules -> Monotonic Capability Expansion Without Retraining Standard prompt engineering relies on manual trial and error, making systematic improvement challenging across complex scientific domains. ABCURATOR implements an outer human-in-the-loop co-evolution mechanism. After each evaluation cycle, domain scientists analyze residual failure modes (such as scFv linker misattribution, OCR broken table lines, or row-order mismatches in alignment blocks) and codify actionable fixes into versioned skill modules (e.g., ocr-format-repair, sequence-image-extraction, pdb-postprocess). These modular skills are dynamically loaded into agent system prompts in subsequent rounds, enabling steady, reproducible performance gains without altering underlying model weights.

Benchmark & Evaluation Protocol

ABDATABENCH compiles 44 challenging scientific documents (27 peer-reviewed papers from PubMed and 17 patents from USPTO/EPO) covering oncology, virology, and parasitic diseases, yielding 279 high-density antibody records annotated independently by domain-expert immunologists.

The benchmark establishes a unified 22-field schema categorized into three weighted tiers reflecting downstream computational utility: - Core Tier (Weight 2.0): CDRH3 amino acid sequence, full VH sequence, full VL sequence, equilibrium dissociation constant (\(K_D\)). - Standard Tier (Weight 1.0): Target_Name, Epitope, Antibody_Type, Mechanism_of_Action, Experiment, \(k_{on}\), \(k_{off}\), \(EC_{50}\), Structure. - Auxiliary Tier (Weight 0.5): source organism, Reference_Source, Target_Type, Antibody_Isotype, Cross_Reactivity, Quantitative_Metric, In_Vivo_Half_Life, In_Vivo_Efficacy, Thermal_Stability_Tm.

Record alignment between predictions and ground-truth uses the Hungarian algorithm based on a composite similarity function: $\(\text{sim}(\hat{a}, a) = 0.85 \bar{N}(\hat{a}, a) + 0.15 C(\hat{a}, a)\)$ where \(\bar{N}\) represents normalized longest common subsequence name similarity, and \(C\) denotes a weighted context similarity across five shared fields (Target_Name: 0.45, Antibody_Type: 0.25, Experiment: 0.15, Structure: 0.10, Reference_Source: 0.05), rejecting pairs below \(\tau=0.5\). Matched fields undergo deterministic checks (unit conversion, relative numerical tolerance \(\le 10\%\) for full credit, \(\le 3\times\) fold for partial credit; string edit distance for sequences), while semantic fields are adjudicated by a calibrated LLM-as-Judge (Gemini-3.1-Pro). The document-level score applies an asymmetric penalty: $\(S_d = \frac{\sum_{a} R(a)}{|A_d|} \times 100 \times \max\left(0, 1 - 0.01 \lfloor n_{FP}/5 \rfloor - 0.05 n_{FN}\right)\)$ Penalizing omissions (\(\lambda_{FN}=0.05\)) substantially more than over-generation (\(\lambda_{FP}=0.01\) per five spurious records) reflects the reality of scientific data curation, where missing verified data creates permanent information loss, whereas candidate false positives can be filtered during secondary review.

Key Experimental Results

Main Results

Performance of 4 proprietary and 4 open-source backbone models integrated into the full ABCURATOR pipeline across all 44 ABDATABENCH documents (279 records):

Model Ab. Precision (%) โ†‘ Ab. Recall (%) โ†‘ Seq. Hit (%) โ†‘ \(K_D\) Hit (%) โ†‘ Overall Score โ†‘
Proprietary Models
Claude-4.6-Opus 45.5 86.4 69.9 79.6 78.3
Claude-4.6-Sonnet 36.8 87.1 67.4 76.1 76.4
GPT-5.5 29.6 84.6 62.7 66.4 72.7
Gemini-3.1-Pro 29.4 75.3 41.0 44.2 63.1
Open-Source Models
GLM-5.1 31.7 97.8 66.8 55.8 76.0
DeepSeek-V4-Pro 49.6 84.9 64.2 67.3 75.5
Qwen3.5-Plus 35.3 84.6 46.4 47.8 70.8
MiniMax-M2.7 32.5 80.3 41.8 34.5 59.5

Ablation Study

Evaluation of individual ABCURATOR pipeline components across all 8 backbone models (scores averaged over 44 documents):

Configuration Proprietary Avg Open-Source Avg Overall Mean Absolute Drop Description
Full ABCURATOR Pipeline 72.6 70.7 71.6 - Complete multi-agent pipeline with all skills
Skeleton-only Extraction 63.3 61.8 62.6 -9.0 Single-pass generation without enrichment
w/o Parallel Enrichment 66.7 64.2 65.4 -6.1 Excluding tables, image VLM, and PDB lookups
w/o Validation & Retry 71.8 69.4 70.7 -0.9 Removing rule validator and reviewer loop
w/o Accumulated Skills 66.1 64.2 65.1 -6.4 Omitting expert-distilled prompt modules

Key Findings

  • Enrichment and Skill Accumulation Drive Major Gains: Ablating skeleton-only extraction leads to the sharpest performance collapse (-9.0 points), while removing accumulated domain skills and parallel enrichment drops scores by 6.4 and 6.1 points respectively. This demonstrates that neither raw LLM reasoning nor generic document prompting suffices without multi-modal tools and expert-derived procedural SOPs.
  • Asymmetric Precision-Recall Trade-off: Across all models, antibody recall remains high (75.3%โ€“97.8%), while precision is notably lower (29.4%โ€“49.6%). Models consistently over-extract background controls, parental clones, and wild-type reference strains, indicating that distinguishing focal scientific contributions from background experimental scaffolding remains a core agent weakness.
  • Competitive Open-Source Performance: GLM-5.1 achieves an outstanding 97.8% antibody recall and an overall score of 76.0, closely trailing Claude-4.6-Sonnet. Meanwhile, DeepSeek-V4-Pro delivers the highest antibody precision (49.6%) and superior \(K_D\) grounding (67.3%) at an exceptionally low operational cost ($0.15 per document versus $2.95 for Claude-4.6-Opus).

Highlights & Insights

  • Record-Centric Evaluation Paradigm: By shifting from unstructured generation or entity-level triples to Hungarian-aligned record matching with biological utility weighting, ABDATABENCH establishes a rigorous evaluation standard that directly mirrors the data quality requirements of molecular foundation models.
  • Hybrid Multi-Agent System Architecture: The system cleanly separates semantic ambiguity (handled by LLM agents) from deterministic execution (handled by rule parsers, regex scanners, and database lookup tools), ensuring high fidelity for critical quantitative metrics and sequence integrity.
  • Co-Evolutionary Skill Accumulation: Formalizing human expert corrections into versioned, modular skill prompts creates an effective feedback loop that steadily expands agent domain competency without costly fine-tuning or retraining.

Limitations & Future Work

  • Upstream Optical Character Recognition Bottlenecks: ABCURATOR relies heavily on VLM-based OCR (MinerU). Severe layout distortions, complex multi-panel image tables, or degraded scanned patents can introduce transcription errors that propagate into sequence truncations.
  • Absence of Downstream Wet-Lab Validation: Current evaluation benchmarks accuracy against human expert annotations. Future studies should evaluate whether agent-curated antibody datasets successfully enhance downstream wet-lab affinity maturation or generative antibody design.
  • Domain Specialization Overhead: The 22-field tiered schema is tightly tailored to immunology. Extending the framework to other scientific domains (e.g., small molecules, enzyme biocatalysis, polymer chemistry) requires defining new domain schemas and calibration rubrics.
  • vs SciREX / BioRED / TDMS-IE: Prior information extraction benchmarks evaluate sentence-level or abstract-level relations using flat, coarse schemas. ABDATABENCH evaluates comprehensive, multi-field antibody records spanning text, tables, figures, supplements, and external databases.
  • vs ScienceAgentBench / PaperBench: Recent agent benchmarks emphasize broad, open-ended scientific tasks such as code reproduction or hypothesis generation. ABDATABENCH uniquely focuses on the foundational, precision-critical task of structured scientific record construction.
  • vs SAbDab / Thera-SAbDab / OAS: Existing antibody repositories provide sequence or structural catalogs but rarely curate comprehensive functional affinity and kinetic measurements at scale. ABCURATOR offers an automated pipeline capable of systematically mining functional data across scientific literature.

Rating

  • Novelty: โญโญโญโญโญ Establishes the first record-centric antibody curation benchmark with Hungarian alignment and tiered asymmetric penalties, supported by an expert-driven skill accumulation workflow.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 8 proprietary and open-source models, thorough ablation experiments, qualitative error case studies, and scaling demonstration on 48,000+ documents.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear, well-structured, and precise in technical terminology, with extensive appendices documenting prompt modules, scoring rubrics, and resource costs.
  • Value: โญโญโญโญโญ Bridges the critical gap between natural language agent systems and computational biology data pipelines, establishing a foundational infrastructure for automated scientific data curation.