Skip to content

SMADE-IE: Sparse Multi-Agent Framework with Evidence-Driven Debate for Zero-Shot Information Extraction

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Cppys/SMADE-IE
Area: LLM Reasoning
Keywords: Zero-Shot Information Extraction, Sparse Multi-Agent System, Evidence-Driven Debate, Toulmin Argumentation, Bayesian Confidence Update

TL;DR

SMADE-IE addresses the high token overhead, noisy context, and unstructured disagreement in zero-shot information extraction by introducing an adaptive mode selector for complexity-guided sparse routing and an evidence-driven debate module combining Toulmin argument decomposition, external evidence scoring, and Bayesian Beta confidence updates with dual-track early stopping, reducing token consumption by over 80% on large ontologies while outperforming existing baselines across 9 benchmarks.

Background & Motivation

Zero-shot information extraction (IE) leverages the extensive parametric background knowledge of large language models (LLMs) to identify named entities, extract semantic relations, and parse joint entity-relation structures without task-specific supervised training. Standard zero-shot prompting strategies generally follow two contrasting extremes. On one hand, monolithic prompting (e.g., AEiO) extracts all entities and relations in a single unified prompt, offering high runtime efficiency but severely suffering from missed mentions, boundary misalignments, and catastrophic type confusion as the ontology size expands. On the other hand, each-type prompting (e.g., One-Step, G&O) issues independent queries for each predefined category to sharpen category-level focus, but naturally breeds widespread prediction overlaps and conflicting span labels because distinct type queries remain mutually isolated.

To reconcile these cross-type prediction conflicts, recent multi-agent debate frameworks (such as CrossAgentIE) deploy dedicated agents for different entity and relation types to reach consensus through inter-agent deliberation. However, exhaustive multi-agent interactions quickly encounter severe efficiency and reliability bottlenecks. Most critically, natural language text exhibits extreme sparsity over large ontological schemas—an individual sentence typically instantiates only one or two categories out of dozens—yet existing methods activate the entire schema's agent collective for every input. This causes prohibitive token consumption and introduces spurious candidate types that inject irrelevant, noisy context into the deliberation, easily leading agents into ungrounded consensus. Furthermore, conventional free-form debates lack explicit structural constraints to model claims, underlying factual evidence, and logical warrants, making it difficult to systematically aggregate credibility scores or establish stable mathematical stopping criteria. In joint entity-relation extraction (JERE), the absence of ontology-level bidirectional constraints further allows mismatched entity types and orphan relations to escape unchecked.

This paper addresses these challenges by introducing a fast-and-slow cognitive allocation strategy into multi-agent zero-shot extraction, matching computational depth strictly with instance complexity: routing straightforward sentences through rapid, lightweight extraction while reserving fine-grained multi-agent deliberation exclusively for complex, conflicting edge cases. Core idea: Propose SMADE-IE, a sparse and evidence-driven multi-agent framework that dynamically routes inputs to either a lightweight global extraction path or a type-centric extraction path based on sample complexity, and resolves cross-type prediction conflicts via structured Toulmin-style argumentation, external evidence scoring, and Bayesian Beta posterior updates under a dual-track early stopping criterion.

Method

Overall Architecture

The end-to-end architecture of SMADE-IE processes an input sentence \(s\) through an adaptive pipeline. The process begins with the Router Agent, which inspects the sentence alongside the target ontology to estimate sample complexity (\(c \in \{\text{low}, \text{med}, \text{high}\}\)) and identify a relevant subset of candidate types \(A\). Low-complexity inputs follow the Global Extraction Mode, where a Universal Agent executes coarse extraction and a Verification Agent refines predictions via joint addition and deletion. Medium- and high-complexity inputs are dispatched to the Type-Centric Extraction Mode, where Type-Specific Agents extract candidate spans for selected types while a Review Agent inspects residual unselected types to prevent false negatives. Any conflicting type assignments are forwarded to the Evidence-Driven Debate module, which executes Top-2 candidate ranking, Toulmin-style adversarial refutation, component validity decay, and Bayesian Beta confidence updating with dual-track early stopping. For joint extraction, an Iterative Entity–Relation Alignment (IERA) module further reconciles mutual cross-task ontological constraints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sentence s & Target Ontology Schema"] --> B["Adaptive Mode Selector<br/>Estimate complexity (low/med/high) & route type subset A"]
    B -->|Low Complexity c=low| C["Global Extraction & Refinement<br/>Universal coarse extraction + Verification filtering"]
    B -->|Medium/High Complexity c∈{med, high}| D["Sparse Type-Centric Extraction & Review<br/>Type-Specific extraction + Review residual scanning"]
    C --> G["Clean Predictions"]
    D --> E{"Cross-Type Conflicts Detected?"}
    E -->|No Conflict| G
    E -->|Has Conflict| F["Evidence-Driven Debate<br/>Toulmin decomposition + Scorer + Beta update + Early stopping"]
    F --> H["Resolved Entities & Relations"]
    G --> H
    H --> I["Iterative Entity–Relation Alignment (IERA)<br/>Relational back-fill & Blacklist consistency arbitration"]
    I --> J["Final Ontologically Consistent Output"]

Key Designs

1. Adaptive Mode Selector: Complexity-Guided Sparse Type Routing To eliminate the severe token waste and reasoning distractors caused by invoking full-schema agents on simple sentences, this module operationalizes a fast-and-slow reasoning design via a single lightweight LLM call. The Router Agent scans the sentence \(s\) against ontology \(\mathcal{T}_e\) to predict a relevant type subset \(A \subseteq \mathcal{T}_e\) and assigns a three-level complexity label \(c \in \{\text{low}, \text{med}, \text{high}\}\). Simple inputs (\(c=\text{low}\)) are routed to the Global Extraction Mode, utilizing a Universal Agent to generate candidate mentions in a single pass followed immediately by a Verification Agent that performs bidirectional refinement (inserting missed mentions and pruning spurious ones), completing extraction in just two calls. Difficult or ambiguous inputs (\(c \in \{\text{med}, \text{high}\}\)) enter the Type-Centric Extraction Mode, initializing agents strictly for the candidate subset \(A\). This trims the candidate search space from \(O(|\mathcal{T}|)\) down to \(O(|A|)\), isolating irrelevant distractor types before discourse begins.

2. Sparse Type-Centric Extraction & Review: Focused Extraction with Residual Recovery Relying solely on a single-pass router risks omitting subtle, context-dependent categories, potentially causing permanent false negatives. To safeguard recall, the Type-Centric Extraction Mode pairs focused extraction with a compensatory recovery step. First, dedicated Type-Specific Agents extract candidate spans \(E_A\) exclusively for categories in \(A\). Next, a Review Agent evaluates the residual type set \(A^* = \mathcal{T}_e \setminus A\), extracting residual candidates \(E_A^*\) and taking their union \(\tilde{E}_A = E_A \cup E_A^*\). Candidates assigned a single unique type are directly assigned to the clean set \(\tilde{E}_{\text{clean}}\), whereas spans assigned multiple conflicting types are aggregated into the conflict set \(\tilde{E}_{\text{conf}}\) to undergo structured debate. This dual-track structure guarantees that any valid category mistakenly omitted by the initial router is recovered prior to downstream conflict adjudication.

3. Evidence-Driven Debate: Toulmin-Guided Argumentation and Bayesian Posterior Dynamics To eliminate the ungrounded rhetoric, endless circularity, and subjective drift typical of free-form debates, this module recasts each competing type assignment into a formal Toulmin argument comprising Claim (\(C\)), Ground (\(G\)), Warrant (\(W\)), Backing (\(B\)), and Rebuttal (\(R\)). An external evidence scorer \(\phi\) (AlignScore-base) evaluates the factual support of each argument against the sentence context, computing confidence \(Q_n\). The system filters the candidate pool to the Top-2 qualifiers, collapsing debate complexity from \(O(N^2)\) to \(O(1)\). Opposing agents are restricted to attacking only active evidence components—Ground and Warrant. The calculated attack effectiveness \(a_{l, m}^{(t)} = \sigma(\gamma(\phi(x_l, u_{\bar{l}, m}) - \phi(x_l, z_{l, m})))\) drives an exponential decay of component validity \(v_{l, m}^{(t)}\). Simultaneously, the attack score updates the Beta posterior distribution \(\text{Beta}(\alpha_l^{(t)}, \beta_l^{(t)})\) representing candidate confidence: $$ \beta_l^{(t+1)} = \beta_l^{(t)} + \bar{v}{\bar{l}}^{(t)} \sum $$ By coupling the Beta posterior parameters directly to component validity, the system ensures that credibility increases only when an argument's factual evidence withstands adversarial refutation.} a_{l, m}^{(t)} \cdot v_{l, m}^{(t)}, \quad \alpha_l^{(t+1)} = \alpha_l^{(t)} + \sum_{m} (1 - a_{l, m}^{(t)}) \cdot v_{l, m}^{(t)

4. Dual-Track Early Stopping: Posterior Superiority and Stability Criteria Rather than executing a rigid number of debate turns, SMADE-IE adopts a dual-track stopping rule grounded in a Chebyshev-style probabilistic upper bound on decision error \(P(p_2 > p_1) \le \frac{1}{2(N+1)}(\frac{1}{\Delta^2} - 1)\): - Posterior Superiority: When the leading candidate's accumulated posterior mean dominates its opponent and the tail probability \(P(\pi_{l^*}^{(t)} > \pi_{\bar{l}^*}^{(t)}) > \delta_{\text{stop}}\), the debate terminates immediately, as additional turns cannot overturn the decisive evidence gap. - Posterior Stability: When competing candidates remain semantically close but validity decay causes consecutive posterior updates to stabilize, the squared Hellinger distance drops below threshold \(\Delta_{\text{conf}}^{(t)} = \frac{1}{2}\sum H^2(\pi_l^{(t)}, \pi_l^{(t-1)}) < \varepsilon\). This triggers early stopping because marginal information gains have depleted. If neither criterion is satisfied within the maximum budget \(T_{\max}=3\), the winner is determined by the external evidence scorer \(\phi\).

5. Iterative Entity–Relation Alignment (IERA): Cross-Task Ontology Consistency For joint entity and relation extraction (JERE), candidate entities and relations must satisfy mutual ontological consistency. IERA establishes an iterative reconciliation loop: (1) Entity Completion: If an extracted candidate relation involves an argument span missing from the entity set, a repurposed extraction call back-fills the missing entity; (2) Consistency Correction: Whenever a candidate relation violates predefined argument type constraints (e.g., work_at requiring a PER head and ORG tail), a Consistency Agent issues an authoritative decision: either marking the entity correct (blacklisting and pruning the relation) or marking the relation correct (retyping the entity to the constraint-compatible type). The loop repeats until all violations are resolved, producing fully harmonized quintuples.

Benchmark & Evaluation Protocol

  • Datasets and Tasks: Evaluated across 9 benchmark datasets under 12 zero-shot configurations. NER encompasses CoNLL03, OntoNotes5, SciERC, CrossRE, and REDFM. RE spans DocRED, SemEval-2010 Task 8, SciERC, CrossRE, and REDFM. JERE evaluates on CoNLL04 and NYT.
  • Evaluation Metrics: Evaluated using micro Partial F1 (\(F1_P\), allowing boundary overlap) and Strict F1 (\(F1_S\), requiring exact span and type matching).
  • Implementation & Hyperparameters: The default backbone is GPT-3.5-Turbo-0125 orchestrated via AutoGen, paired with a locally deployed AlignScore-base checkpoint for \(\phi\). Hyperparameters are held constant across all 12 settings: maximum debate rounds \(T_{\max} = 3\), posterior superiority threshold \(\delta_{\text{stop}} = 0.75\), posterior stability threshold \(\varepsilon = 0.02\), validity activity threshold \(\theta_v = 0.6\), validity decay coefficient \(\eta = 0.8\), revision discount \(\omega = 0.8\), sigmoid temperature \(\gamma = 8.0\), and prior strength interval \([\kappa_{\min}, \kappa_{\max}] = [0.8, 1.5]\).

Key Experimental Results

Main Results

Under the default GPT-3.5-Turbo-0125 backbone, SMADE-IE consistently achieves state-of-the-art zero-shot performance across all 12 evaluation settings, significantly outperforming AEiO, One-Step, G&O, and CrossAgentIE.

Table 1: Comprehensive performance comparison across 9 zero-shot IE benchmarks (%)

| Task | Dataset (|T|) | Metric | AEiO (Monolithic) | One-Step (Each-Type) | G&O (DeCoupled) | CrossAgentIE (Multi-Agent) | SMADE-IE (Ours) | Gain vs. CrossAgentIE | | :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | NER | CoNLL03 (3) | \(F1_P\) / \(F1_S\) | 65.24 / 61.17 | 77.29 / 73.02 | 76.36 / 72.35 | 73.15 / 70.93 | 77.33 / 74.72 | +4.18 / +3.79 | | | OntoNotes5 (18) | \(F1_P\) / \(F1_S\) | 56.88 / 43.12 | 32.38 / 25.72 | 31.65 / 23.40 | 45.30 / 37.18 | 66.08 / 52.05 | +20.78 / +14.87 | | | SciERC (5) | \(F1_P\) / \(F1_S\) | 35.36 / 23.80 | 27.89 / 21.77 | 31.83 / 24.05 | 35.96 / 25.36 | 46.56 / 37.80 | +10.60 / +12.44 | | | CrossRE (38) | \(F1_P\) / \(F1_S\) | 34.61 / 30.80 | 25.45 / 23.65 | 27.00 / 25.38 | 46.15 / 44.40 | 65.50 / 60.56 | +19.35 / +16.16 | | | REDFM (10) | \(F1_P\) / \(F1_S\) | 21.14 / 17.45 | 25.21 / 18.93 | 27.07 / 20.61 | 29.20 / 25.16 | 31.12 / 27.13 | +1.92 / +1.97 | | | NER Avg. | \(F1_P\) / \(F1_S\) | 42.65 / 35.27 | 37.64 / 32.62 | 38.78 / 33.16 | 45.95 / 40.61 | 57.32 / 50.45 | +11.37 / +9.84 | | RE | DocRED (32) | \(F1_P\) / \(F1_S\) | 30.32 / 24.84 | 22.15 / 18.41 | 19.69 / 16.71 | 13.72 / 11.49 | 30.78 / 26.99 | +17.06 / +15.50 | | | SemEval2010 (9) | \(F1_P\) / \(F1_S\) | 27.55 / 9.86 | 16.13 / 5.16 | 12.16 / 4.56 | 11.77 / 2.91 | 29.54 / 16.76 | +17.77 / +13.85 | | | SciERC (8) | \(F1_P\) / \(F1_S\) | 19.98 / 10.91 | 18.56 / 8.94 | 13.07 / 5.58 | 14.61 / 8.79 | 21.47 / 11.20 | +6.86 / +2.41 | | | CrossRE (17) | \(F1_P\) / \(F1_S\) | 4.79 / 3.56 | 12.00 / 8.31 | 8.90 / 7.01 | 11.81 / 8.57 | 16.22 / 12.25 | +4.41 / +3.68 | | | REDFM (32) | \(F1_P\) / \(F1_S\) | 10.04 / 6.24 | 7.11 / 5.34 | 5.65 / 4.12 | 5.36 / 3.44 | 13.83 / 11.34 | +8.47 / +7.90 | | | RE Avg. | \(F1_P\) / \(F1_S\) | 18.54 / 11.08 | 15.19 / 9.23 | 11.89 / 7.60 | 11.45 / 7.04 | 22.37 / 15.71 | +10.92 / +8.67 | | JERE | CoNLL04 (5) | \(F1_P\) / \(F1_S\) | — | — | — | 42.50 / 29.05 | 58.44 / 44.74 | +15.94 / +15.69 | | | NYT (7) | \(F1_P\) / \(F1_S\) | — | — | — | 17.23 / 16.77 | 30.22 / 29.22 | +12.99 / +12.45 | | | JERE Avg. | \(F1_P\) / \(F1_S\) | — | — | — | 29.87 / 22.91 | 44.33 / 36.98 | +14.46 / +14.07 |

Ablation Study & Token Efficiency Analysis

To inspect the individual contributions of the mode router, evidence-driven debate, review agent, and IERA, ablation experiments were conducted across four representative benchmarks alongside token cost auditing.

Table 2: Ablation study of core components (\(F1_P\) / \(F1_S\), %)

Configuration SciERC (NER) CrossRE (RE) CoNLL04 (JERE) NYT (JERE) Key Finding / Mechanism
Full SMADE-IE 46.56 / 37.80 16.22 / 12.25 58.44 / 44.74 30.22 / 29.22 Full system achieves highest performance
└ w/o IERA — — 59.03 / 42.73 24.94 / 22.47 NYT Strict F1 drops 6.75 points; relational alignment is vital
Type-Centric Only 46.75 / 36.81 15.73 / 11.11 59.43 / 40.33 29.65 / 28.47 Bypassing fast mode hurts CoNLL04 Strict F1 by 4.41 points
├ w/o Relevant Type Selection 44.16 / 33.71 14.00 / 10.81 48.66 / 34.18 27.43 / 26.43 Full schema activation causes heavy noise, dropping CoNLL04 by ~10 pts
├ w/o Review Agent 45.70 / 35.03 14.10 / 9.13 52.75 / 35.12 29.50 / 27.56 Omitting residual scan leads to false negatives, dropping CrossRE \(F1_S\)
└ w/o Evidence-Driven Debate 45.86 / 34.98 13.22 / 10.09 54.13 / 38.02 24.24 / 23.38 Free-form debate leads to severe degradation on NYT
Global Only 46.15 / 33.87 14.24 / 10.09 52.47 / 40.20 16.28 / 14.27 Fails on complex tasks; NYT Strict F1 plummets by 14.95 points
└ w/o Verification Agent 43.37 / 33.03 12.83 / 9.06 46.92 / 35.19 12.02 / 11.16 Lacking two-way verification harms both precision and recall

Table 3: Average token consumption per sample across datasets (Tokens / Sample)

| Method | CoNLL03 (NER, |T|=3) | REDFM (NER, |T|=10) | DocRED (RE, |T|=32) | CrossRE (RE, |T|=17) | Efficiency Impact | | :--- | :---: | :---: | :---: | :---: | :---: | | AEiO | 1,078 | 2,163 | 5,460 | 4,798 | Minimal baseline, limited capability | | One-Step | 1,407 | 7,673 | 7,615 | 6,846 | Costs scale linearly with types | | G&O | 2,304 | 14,202 | 9,271 | 14,312 | Expensive multi-stage prompts | | CrossAgentIE | 1,508 | 13,480 | 21,784 | 18,923 | Severe full-schema debate explosion | | SMADE-IE (Ours) | 1,094 | 11,514 | 3,240 | 3,765 | -85.1% on DocRED, -80.1% on CrossRE |

Key Findings

  • Sparse Routing Yields Massive Token Reductions Without Information Loss: On relation extraction datasets with large type schemas, SMADE-IE consumes only 3,240 tokens on DocRED (vs. CrossAgentIE's 21,784) and 3,765 tokens on CrossRE (vs. 18,923), slashing token overhead by 85.1% and 80.1%. Redundant debate counts (Figure 7) plunge from 4,766 to 660, demonstrating that sparse routing discards uninformative discussions rather than truncating productive deliberation.
  • Substantial Reduction in Fine-Grained Errors: Error attribution (Table 11) confirms that compared to AEiO, SMADE-IE reduces omission rates on REDFM NER from 70.1% to 22.7% and boundary errors from 14.0% to 5.8%. On OntoNotes5 and REDFM RE, type confusion drops by 7.3% and 11.1%, validating that Toulmin-based refutations suppress category ambiguity.
  • Rapid Convergence via Dual-Track Early Stopping: Early-stopping statistics indicate that posterior stability drives 60.0%–88.9% of terminations across NER and RE, keeping the average debate rounds between 1.75 and 2.76 out of the maximum budget of 3 (Figure 6). Simple datasets such as SemEval2010 terminate earliest at 1.75 rounds, whereas ambiguous benchmarks like CrossRE appropriately utilize 2.76 rounds.
  • Model-Agnostic Backbone Generalization: When evaluated across three advanced foundation models—Gemini 3 Flash Preview, DeepSeek-V4-Pro, and Claude Sonnet 4.6 (Table 10)—SMADE-IE consistently maintains the highest Strict F1 scores across OntoNotes5, SemEval2010, and CoNLL04. For instance, with DeepSeek-V4-Pro, it achieves 87.91 / 71.04 on OntoNotes5 and 53.00 Strict F1 on CoNLL04, demonstrating that its architectural gains do not rely on idiosyncratic traits of a specific LLM.

Highlights & Insights

  • Dual-System Cognitive Allocation in Multi-Agent Extraction: By integrating fast, low-cost global extraction for unambiguous inputs with targeted, high-precision debate for ambiguous ones, SMADE-IE successfully resolves the long-standing dilemma between lightweight failure and exhaustive multi-agent token explosion.
  • Anchoring Multi-Agent Consensus with Formal Argumentation and Bayesian Calculus: Replacing unstructured conversation with Toulmin-style components (restricting attacks to Ground and Warrant) and coupling external factual scores with Beta distributions provides a mathematically grounded, reproducible framework for resolving agent disagreements.
  • Modular, Reusable Conflict Resolution Blueprint: The combination of structural argument decomposition, Top-2 qualifier ranking, evidence validity decay, and dual-track Chebyshev early stopping is domain-agnostic and directly portable to other multi-agent decision tasks, such as tool-use arbitration and multi-hop fact verification.

Limitations & Future Work

  • Dependency on External Scorer Calibration: The framework relies on a frozen external scorer (AlignScore-base) to evaluate argument validity and initialize Bayesian priors. Factual misjudgments or domain shifts within this scorer could propagate errors downstream through the Bayesian update equations.
  • Sequential Multi-Agent Latency: While token consumption decreases by over 80%, end-to-end wall-clock latency remains non-trivial due to multi-agent turn dependencies, averaging 10.6–26.0s on NER/RE and 33.7s on JERE.
  • Future Directions: Exploring lightweight, trainable on-policy scorers to replace frozen checkpoints, developing speculative or asynchronous debate scheduling to cut wall-clock time, and extending sparse evidence-driven debate to event extraction and document-level cross-document information extraction.
  • vs AEiO (Li et al., 2024): AEiO uses an all-in-one prompt that suffers from boundary drift and type confusion on large schemas. SMADE-IE introduces complexity-based routing and targeted debate, matching AEiO's low cost on simple sentences while doubling Strict F1 on complex benchmarks.
  • vs CrossAgentIE (Lu et al., 2025): CrossAgentIE instantiates full-schema agents and relies on free-form debate, driving up token usage and introducing noise. SMADE-IE cuts token consumption by over 80% through sparse type routing and replaces subjective debate with Toulmin-Bayesian updates, improving average Partial F1 by 11.37 on NER and 14.46 on JERE.
  • vs G&O (Li et al., 2024): G&O employs each-type prompting with decoupled generation and clean-up, which prevents token conflicts during generation but lacks inter-type debate mechanisms. SMADE-IE combines verification with structured adversarial debate and joint relational alignment (IERA), providing a complete closed-loop solution.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant integration of sample complexity routing, Toulmin argumentation, and Bayesian Beta posterior updates with dual-track stopping for zero-shot information extraction.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 9 datasets and 12 task settings spanning NER, RE, and JERE, backed by in-depth ablations, token profiling, error categorization, and multi-backbone testing.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, rigorous architecture definitions, and cohesive motivation and experimental narratives.
  • Value: ⭐⭐⭐⭐⭐ Provides an effective, practical solution to the token explosion and noise interference bottlenecks in multi-agent LLM systems.