Skip to content

ABDATABENCH: Evaluating AI Agents for Evidence-Grounded Antibody Record Construction

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/GAIR-NLP/ABDataBench
Area: LLM / NLP
Keywords: Scientific Information Extraction, AI Agents, Antibody Data Construction, Multimodal Evidence Fusion, Hungarian Alignment

TL;DR

Addressing the bottleneck of dispersed and unstructured scientific evidence in therapeutic antibody literature, this paper presents ABDATABENCH, the first benchmark evaluating complete 22-field antibody record construction across 44 expert-annotated documents (279 records), alongside ABCURATOR, a record-centric multi-agent framework paired with an expert-driven skill accumulation workflow.

Background & Motivation

AI agents are increasingly deployed to accelerate scientific discovery, yet their tangible real-world impact hinges less on open-ended hypothesis generation than on a quieter bottleneck: converting fragmented, multimodal, and heterogeneous scientific literature into reliable, structured, AI-ready datasets. In therapeutic antibody engineering, this bottleneck is particularly acute. Modern antibody structural modeling, de novo design, and machine learning-guided affinity maturation critically depend on vast supervision pairing sequence with binding function. However, essential experimental evidenceโ€”such as variable heavy/light chain sequences (VH/VL), complementarity-determining regions (CDRH3), chain pairing, isotype, target epitope, binding kinetics (\(K_D, k_{\text{on}}, k_{\text{off}}\)), and in vivo therapeutic efficacyโ€”remains widely scattered across main text prose, patent claims, multi-panel alignment figures, supplementary data tables, and external molecular repositories.

Curating functional antibody datasets imposes uniquely stringent constraints compared to traditional NLP tasks. A single amino acid substitution or OCR omission can completely alter binding specificity or render an engineered sequence invalid, while misattributing an equilibrium dissociation constant (\(K_D\)) to the wrong antibody clone corrupts downstream affinity modeling. Existing biomedical information extraction benchmarks (such as SciREX, BioRED, and TDMS-IE) focus predominantly on isolated sentence-level entity recognition or flat relational tuples; they fail to evaluate holistic multi-field biomolecular entity records, lack mechanisms to fuse multimodal evidence, and do not incorporate asymmetric penalty structures that reflect real-world engineering priorities where omission is far more damaging than spurious candidates.

To overcome these fundamental limitations, scientific information extraction must treat the evidence-grounded antibody record as the primary atomic unit of extraction and coordination, while establishing a tight synergy between neural reasoning and deterministic symbolic operations. Core idea: recast antibody literature curation as a long-horizon, evidence-grounded record construction task by releasing ABDATABENCHโ€”a 22-field tiered benchmark pairing Hungarian alignment with asymmetric penalty scoringโ€”and developing ABCURATOR, a record-centric multi-agent framework that combines parallel multimodal enrichment, symbolic biological validation, and an expert-in-the-loop skill accumulation workflow.

Method

Overall Architecture

The ABCURATOR architecture is orchestrated end-to-end to transform raw, uncurated scientific literature (OCR Markdown and associated figure graphics) into standardized 22-field antibody records. The pipeline proceeds through document conditioning and normalization, paginated candidate skeleton extraction, parallel multi-channel evidence enrichment, rule-governed biological validation with bounded iterative review, and an outer co-evolution loop that distills expert corrections into versioned agent competencies. The Orchestrator first cleans and normalizes OCR artifacts while running deterministic regex scans to identify key biological markers; a Skeleton Agent then outputs candidate antibody entities with strict pagination budgets. Next, concurrent retrieval channels enrich the candidate skeletons with sequence, affinity, and structural data. After record-level provenance-preserving merging, deterministic biological checks and an LLM reviewer guide bounded retries, while residual systematic errors are captured by human experts to enrich versioned prompt skill modules.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Raw Document Input<br/>MinerU OCR Markdown + Figure Images"] --> B["Document Conditioning & Normalization<br/>Table alignment repair + regex entity scanning"]
    B --> C["Paginated Skeleton Construction<br/>Extract candidate antibodies and core metadata"]
    C --> D
    subgraph E["Parallel Evidence Enrichment"]
        direction TB
        E1["External Database Channel<br/>PDB / GenBank accession lookup"]
        E2["Tables & Supplements Channel<br/>Kinetics and assay condition parsing"]
        E3["Vision Multimodal Channel<br/>VLM sequence alignments and SPR curves"]
    end
    D --> E
    E --> F["Biological Validation & Bounded Review<br/>Amino-acid legality/CDR-H3 anchor/length check/retry"]
    F --> G["Expert-Guided Skill Evolution<br/>Residual error taxonomy and versioned prompt skills"]
    G --> H["Standardized 22-Field Antibody Records"]

Key Designs

1. Record-Centric Modeling and Tiered Evaluation Schema: Resolving Disconnected Entity Tuples and Multimodal Fragmentation

Rather than evaluating isolated sentence-level relations, ABDATABENCH establishes the complete "antibody record" as the atomic evaluation unit. It defines a standardized schema of 22 fields organized into three utility tiers reflecting downstream antibody engineering priorities: Core tier (weight 2.0: CDRH3 sequence, VH full variable sequence, VL full variable sequence, and \(K_D\) binding affinity), Standard tier (weight 1.0: Target Name, Epitope, Antibody Type, Mechanism of Action, Experimental Methods, \(k_{\text{on}}, k_{\text{off}}, EC_{50}\), and Structure details), and Auxiliary tier (weight 0.5: Source species, Reference Source DOI/citation, Target Type, Antibody Isotype, Cross-Reactivity profile, Quantitative Metrics, In Vivo Half-Life, In Vivo Efficacy, and \(T_m\) thermal stability). During evaluation, predicted records and expert annotations are aligned via the Hungarian matching algorithm using a pairwise cost matrix that combines normalized antibody name similarity (weight 0.85, using longest common subsequence boosted to \(\ge 0.98\) on exact match) and a five-field contextual consistency score (weight 0.15, covering target name, antibody format, assay methods, structure, and reference), rejecting assignments below threshold \(\tau=0.5\).

2. Hybrid Deterministic and Calibrated LLM-as-Judge Scoring: Balancing Numerical Exactness with Semantic Flexibility

To accommodate heterogeneous biological data types, the evaluation pipeline enforces a three-stage tiered assessment. Seven quantitative kinetics fields are processed by deterministic parsers that normalize concentration and rate units across orders of magnitude, granting full credit (1.0) for relative differences \(\le 10\%\) and partial credit (0.5) for values within a 3-fold envelope. Six sequence, method, and formatting fields are adjudicated by exact string normalization, substring containment, and set overlap rules. The remaining 9 semantic fields are scored by a calibrated Gemini-3.1-Pro judge operating under zero temperature with field-specific rubrics. Document scores are aggregated with an asymmetric penalty function:

\[S_d = \frac{\sum_{a} R(a)}{|A_d|} \times 100 \times \max\left(0, \, 1 - 0.01 \lfloor n_{\text{FP}} / 5 \rfloor - 0.05 \, n_{\text{FN}}\right)\]

This formula penalizes omitted ground-truth records (\(\lambda_{\text{FN}}=0.05\) per missed antibody) five times more severely than spurious candidates (\(\lambda_{\text{FP}}=0.01\) per five extra records), directly reflecting downstream database requirements where candidate over-generation can be filtered by human review, but missing sequences or affinities cause irreversible information loss.

3. Neuro-Symbolic Agent Pipeline and Parallel Enrichment: Safeguarding Reproducibility While Fusing Multimodal Evidence

ABCURATOR strictly decouples stochastic neural reasoning from deterministic execution. Initial document preprocessing deploys pure Python regex scans to locate PDB accession numbers, CDR3 motifs, and multi-panel graphics. The Skeleton Agent generates initial candidate entities using strict pagination budgets (emitting at most \(k\) records per prompt invocation to prevent context truncation and duplicate hallucination). Once candidate skeletons are formed, the pipeline triggers four concurrent enrichment streams: a database channel querying PDB/GenBank APIs for verified structural and sequence data; a structured table/supplement parser; and a Vision-Language Model (VLM) channel specialized in transcribing sequence alignment images and digitizing SPR/BLI sensorgrams. All extracted evidence is merged under strict record-level priority rules, ensuring that every field retains verifiable provenance and verbatim quotes while preventing cross-clone data pollution.

4. Closed-Loop Biological Validation and Versioned Skill Evolution: Enforcing Physical Constraints and Compounding Capabilities

To guarantee that extracted sequences adhere to genuine biophysical invariants, a deterministic Validator verifies amino acid alphabet validity (purging invalid or corrupted characters), checks variable domain length envelopes (ensuring typical VH/VL bounds), verifies CDR-H3 conserved flanking motifs, and flags duplicate sequences across distinct antibody rows. When validation flags a violation, a specialized Reviewer Agent generates targeted field-level repair directives, triggering bounded re-extraction loops. Between evaluation rounds, human domain experts inspect residual failures and classify recurring error patternsโ€”such as chain pairing mismatches, incomplete alignment segment concatenation, failure to split strain-specific records, and full scFv chains assigned to VH. These insights are formalized into reusable, versioned prompt skill modules (e.g., ocr-format-repair, evidence-reduction, sequence-image-extraction, pdb-postprocess) equipped with operational guardrails, driving monotonic accuracy gains across iterations without model retraining.

Benchmark & Evaluation Protocol

ABDATABENCH encompasses 44 expert-annotated scientific documents spanning 27 peer-reviewed journal papers and 17 full patent filings sourced from PubMed, USPTO, and EPO between 2005 and 2025. Across viral targets (SARS-CoV-2, HIV-1, Influenza), oncology markers (HER2, EGFR, PD-L1), and parasitic antigens (P. falciparum), the benchmark establishes 279 densely curated antibody records. Each document underwent independent double annotation by PhD-level immunologists and structural biologists (inter-annotator agreement Cohen's \(\kappa=0.91\) for categorical fields and \(\kappa=0.95\) for numeric measurements), with disputes resolved via third-party expert adjudication.

Key Experimental Results

Main Results

The authors evaluated four frontier proprietary models and four competitive open-source LLMs under the unified ABCURATOR pipeline on all 44 benchmark documents (279 records). Evaluated metrics comprise antibody record precision (Ab. Prec.), antibody recall (Ab. Rec.), sequence field hit rate (Seq. Hit), binding affinity hit rate (KD Hit), and overall weighted benchmark score (Score).

Model Category Model Name Ab. Prec. (%) โ†‘ Ab. Rec. (%) โ†‘ Seq. Hit (%) โ†‘ KD Hit (%) โ†‘ Overall Score โ†‘
Proprietary Claude-4.6-Opus 45.5 86.4 69.9 79.6 78.3
Proprietary Claude-4.6-Sonnet 36.8 87.1 67.4 76.1 76.4
Proprietary GPT-5.5 29.6 84.6 62.7 66.4 72.7
Proprietary Gemini-3.1-Pro 29.4 75.3 41.0 44.2 63.1
Open-Source GLM-5.1 31.7 97.8 66.8 55.8 76.0
Open-Source DeepSeek-V4-Pro 49.6 84.9 64.2 67.3 75.5
Open-Source Qwen3.5-Plus 35.3 84.6 46.4 47.8 70.8
Open-Source MiniMax-M2.7 32.5 80.3 41.8 34.5 59.5

Ablation Study

To isolate the performance contribution of each design component in ABCURATOR, five configurations were systematically tested across the 44-document benchmark using the same backbone models.

Model Name Skeleton Only w/o Parallel Enrichment w/o Validation / Retry w/o Accumulated Skills Full Framework
Claude-4.6-Opus 67.9 71.8 77.6 70.3 78.3
Claude-4.6-Sonnet 66.9 71.0 75.7 68.4 76.4
GPT-5.5 65.1 68.2 71.4 65.3 72.8
Gemini-3.1-Pro 53.4 55.8 62.6 60.5 63.1
GLM-5.1 66.8 71.0 75.2 69.2 76.0
DeepSeek-V4-Pro 65.0 66.5 74.3 68.8 75.5
Qwen3.5-Plus 63.6 66.2 69.9 64.5 70.8
MiniMax-M2.7 51.8 53.1 58.8 54.2 59.5
Average 62.6 (-9.0) 65.4 (-6.1) 70.7 (-0.9) 65.1 (-6.4) 71.6

Key Findings

  • Multi-stage enrichment and skill evolution drive the largest gains: Operating on skeleton extraction alone incurs the steepest performance drop (-9.0 points on average), followed closely by the removal of accumulated expert skills (-6.4 points) and parallel multi-channel enrichment (-6.1 points). This validates that single-prompt extraction over noisy documents cannot reliably capture multimodal scientific evidence.
  • Open-source backbones compete directly with top proprietary models: GLM-5.1 attains the highest recall across all models (97.8%) and an overall score of 76.0, closely tracking Claude-4.6-Sonnet (76.4). DeepSeek-V4-Pro exhibits superior hallucination suppression, achieving the highest precision (49.6%) and an overall score of 75.5.
  • Evidence attribution and domain splitting constitute the primary error modes: Qualitative expert audits demonstrate that errors are concentrated in four structural patterns: (1) Pairing Errors, where multi-antibody sequence panels lead models to misattribute CDR-H3 loops across rows; (2) Merge Errors, where segmented alignment blocks fail to concatenate into full VH chains; (3) Split Errors, where condition-dependent assays (e.g., strain-specific \(EC_{50}\)) are collapsed into a single entity; and (4) PDB structural parsing errors, where linked scFv constructs are incorrectly assigned entirely to the VH field.
  • Scalability to wild literature: Applying ABCURATOR to 48,584 uncurated scientific publications and patent filings yielded 39,586 successful extractions (81.5% success rate). A random audit of 2,000 processed outputs revealed 15,990 validated antibody records across mAbs, scFvs, Fabs, and nanobodies (VHH), confirming industrial-scale viability.

Highlights & Insights

  • Turning expert intervention into versioned agent capabilities: Instead of continuous fine-tuning or ephemeral prompt tinkering, ABCURATOR operationalizes expert feedback into modular, versioned skill routines with clear preconditions and guardrails, creating a monotonic capability flywheel.
  • Neuro-symbolic division of labor: The framework strictly delegates closed-form sequence validation, numerical unit conversion, and database cross-referencing to deterministic Python tools, reserving LLM/VLM capacity exclusively for complex semantic disambiguation and contextual synthesis.

Limitations & Future Work

  • Domain specificity of schema and rules: The 22-field schema and biophysical validator are specifically tailored to immunoglobulin biology; extending this paradigm to small molecules, enzymes, or cell therapies will require constructing dedicated ontologies and deterministic validation tools.
  • Operational latency and inference cost: Running full OCR extraction, multi-agent coordination, multimodal VLM inspection, and iterative review loops incurs significant API cost and minutes-scale wall-clock latency per document.
  • Absence of wet-lab experimental validation: The benchmark evaluates extraction fidelity against human expert annotations from published papers, but downstream utility in training real-world affinity maturation models remains to be empirically verified in wet-lab workflows.
  • vs SciREX / BioRED: Traditional scientific information extraction datasets focus on sentence- or document-level relation extraction among entity mentions; ABDATABENCH evaluates structured multi-field records anchored across text, tables, figures, and external repositories.
  • vs ScienceAgentBench / PaperBench: General science agent benchmarks assess code synthesis, data analysis, or full-paper reproduction; ABDATABENCH targets the foundational upstream bottleneck of structured data curation with rigorous field-level matching and asymmetric penalty metrics.

Rating

  • Novelty: โญโญโญโญโญ Pioneering record-centric formulation for scientific data construction with an asymmetric, tiered evaluation protocol.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous benchmark evaluation across 8 frontier models, extensive framework ablations, failure taxonomy, and wild literature scaling.
  • Writing Quality: โญโญโญโญโญ Highly structured, precise biomedical and algorithmic descriptions with transparent scoring formulas and prompt documentation.
  • Value: โญโญโญโญโญ Delivers an essential, production-ready open-source foundation for building computational antibody engineering databases.