Skip to content

BeyondAtoms: Benchmarking and Verifying Relational Factuality Beyond Atomic Facts

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/lzy248/beyondatoms
Area: NLP Understanding
Keywords: Relational fact verification, Atomic factuality limits, Bidirectional relational alignment, Evaluation benchmark, Training-free verification

TL;DR

Addressing the vulnerability where conventional fact verifiers and atomic decomposition blindly trust entity/event overlaps while ignoring logical topologies, this paper introduces BEYONDATOMS, the first operation-centric relational factuality benchmark, and CARVE, a training-free verification framework based on claim-anchored extraction and bidirectional structural alignment.

Background & Motivation

As Large Language Models (LLMs) increasingly generate reasoning-intensive outputs such as Chain-of-Thought explanations, they naturally weave isolated factual units into complex logical topologies. Consequently, the factuality of generated text no longer hinges solely on whether isolated constituent events are true; it critically depends on the faithful preservation of the causal, temporal, or conditional relations binding them. However, current verification paradigms suffer from severe structural blind spots. Sentence-level entailment systems (such as standard NLI models) are susceptible to superficial heuristics like entity-overlap bias. Conversely, fine-grained atomic verification frameworks (such as FActScore), which decompose long-form generations into isolated subclaims, systematically sever and discard cross-event relational edges during syntactic fragmentation.

This systemic blindness creates a dangerous false sense of security: verifiers routinely endorse relationally corrupted claims simply because all constituent base events are verified as supported in the grounding evidence. Meanwhile, existing relation-aware datasets (such as DiscInfer and LoReFact) adopt a semantic-centric approach, categorizing errors by linguistic connective types and predominantly focusing on relation substitution, which captures only a small fraction of topological distortions in real-world reasoning.

To address these vulnerabilities and eliminate structural blind spots, benchmark evaluation must strictly isolate relational manipulations while holding constituent events constant, and verification frameworks should avoid costly, opaque black-box LLM reasoning by decoupling structural logic into deterministic alignment rules. Core idea: Isolate relational fact verification via BEYONDATOMS, an operation-centric benchmark manipulating four fundamental topological perturbations (Injection, Omission, Re-anchoring, Substitution) over preserved constituent events, and propose CARVE, a training-free framework that decouples structural logic and verifies claims via deterministic bidirectional relational alignment.

Method

Overall Architecture

CARVE (Claim-Anchored Relational Verification over Evidence) is a completely training-free fact verification framework designed to resolve the pervasive relation blindness caused by entity-overlap bias. Its core insight is to parse the claim and the evidence into explicit eventโ€“relation graph structures, delegating the verification of fragile logical links to an interpretable bidirectional topological alignment pipeline.

CARVE operates across four distinct stages: 1. Relation-Preserving Canonicalization: Smooths surface syntactic variations (such as fronted subordinate clauses and connective-proximate pronominal references) into standardized structural representations; 2. Claim-Anchored Asymmetric Extraction: Parses the claim into relation triples and isolated events to establish a verification anchor set, then strictly restricts evidence-side parsing to sentences exhibiting lexical overlap with these anchors, insulating the pipeline from irrelevant document noise; 3. Bidirectional Relational Alignment and Diagnosis: Applies claim-to-evidence forward alignment to detect relation Injection, Re-anchoring, and Substitution, combined with evidence-to-claim reverse scanning to detect premise Omission; 4. Auxiliary Fact Verification and Constrained LLM Fallback: Employs a lightweight model to verify the factual grounding of constituent events, and invokes a constrained LLM prompt as a localized parser to arbitrate syntactic ambiguity or unrecognized connectives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Claim and Evidence"] --> B["1. Relation-Preserving Canonicalization<br/>Smooth fronted clauses and resolve local pronouns"]
    B --> C["2. Claim-Anchored Asymmetric Extraction<br/>Extract claim anchor triples, constrain evidence candidate pool"]
    C --> D["3. Bidirectional Relational Alignment and Diagnosis<br/>Claim-to-evidence detects edge conflicts, evidence-to-claim detects omissions"]
    D --> E["4. Auxiliary Fact Verification and Constrained LLM Fallback<br/>Ground constituent events and arbitrate parsing ambiguity"]
    E --> F["Output: Decision and Fine-Grained Error Diagnosis"]

Key Designs

1. Relation-Preserving Canonicalization: Neutralizing surface syntactic noise for deterministic extraction

In natural language, identical logical relations can manifest through highly disparate surface structures, such as fronted subordinate clauses ("Because A, B"), cross-sentence discourse connectives, and proximate pronominal references. These syntactic variations lead to fragmentation artifacts and false dependency parses under deterministic rule-based systems. CARVE standardizes inputs prior to extraction by reordering fronted subordinate clauses into canonical main-clause-first structures ("B because A") and resolving pronominal references located adjacent to logical connectives. Crucially, this canonicalization is strictly relation-preserving: it standardizes existing surface syntax without introducing any external or hallucinated semantic connections, providing structural consistency for downstream extraction.

2. Claim-Anchored Asymmetric Extraction: Constraining evidence parsing scope via claim anchors

Constructing a comprehensive, exhaustive relation graph over full-length multi-sentence evidence is both computationally expensive and susceptible to overwhelming irrelevant relational noise. CARVE implements an asymmetric, claim-driven extraction paradigm: It first parses the canonicalized claim to extract a set of explicit directed relational triples \(T_c = \{(u, r_c, v)\}\) (where \(u, v\) represent constituent events and \(r_c\) denotes the directional connective) alongside isolated constituent events \(I_c\) unbound by connectives. Together, these form the claim anchor set \(\mathcal{A}(c) = T_c \cup I_c\). CARVE then performs evidence-side extraction strictly conditioned on \(\mathcal{A}(c)\), filtering the canonicalized evidence passage to retain only sentences sharing sufficient content-word overlap with at least one claim anchor. Only these candidate sentences are parsed into the compact evidence relation pool \(P_E(c)\), ensuring that downstream verification remains targeted and impervious to background relational noise.

3. Bidirectional Relational Alignment and Diagnosis: Dual-direction topology matching covering the full perturbation spectrum

CARVE converts relational verification into structural matching between \(T_c\) and \(P_E(c)\), executing deterministic diagnostic logic across two directions: - Claim-to-Evidence Alignment: For each claim triple \((u, r_c, v) \in T_c\), CARVE queries \(P_E(c)\) across both argument directions. If the evidence contains \((u, r_e, v)\) and \(r_e\) shares the same semantic family with \(r_c\) (e.g., because/since), the relation is verified as supported. If the evidence matches \((v, r_e, u)\) where \(r_e\) is an inverse pair (e.g., before/after), it is supported. Conversely, an argument direction mismatch within the same connective family triggers Re-anchoring; a mismatch in semantic families (e.g., swapping causal for concessive) triggers Substitution; and constituent events co-occurring in evidence without a connecting relation edge trigger Injection. - Evidence-to-Claim Reverse Alignment: To catch covert Omission errors where the claim completely drops a conditional connective, CARVE performs a reverse sweep over \(P_E(c)\) for unmatched conditional relations \((x, r_e, y)\) (where \(r_e\) is a conditional marker like if/when). If the consequent event \(y\) from the evidence is asserted unconditionally in the claim while the mandatory premise \(x\) is omitted, an Omission error is flagged.

4. Auxiliary Fact Verification and Constrained LLM Fallback: Decoupled structural and semantic safeguards

Following relational alignment, CARVE executes event-level factuality verification using an NLI model or constrained LLM prompt to check that all independent events in \(I_c\) and constituent arguments in verified relations are grounded in the evidence; any unsupported base event triggers a fact_unsupported label. Furthermore, when deterministic dependency parsing encounters complex sentence structures or unrecognized connectives falling outside predefined dictionaries, CARVE invokes a constrained LLM fallback (defaulting to lightweight Qwen3.5-4B). Rather than acting as an end-to-end black-box verifier, the LLM functions strictly as a localized semantic helper for syntax disambiguation, preserving system throughput and explainability while adding robust semantic flexibility.

Key Experimental Results

Main Results

The framework is evaluated on the proposed relational benchmark BEYONDATOMS (spanning Substitution Sub., Re-anchoring Re-anch., Injection Inj., and Omission Omi.), the discourse benchmark DiscInfer (Comparison Comp., Contingency Cont., and Temporal Temp.), and the standard fact verification dataset FEVER. BEYONDATOMS and FEVER report Macro-F1 (%), while DiscInfer reports Accuracy (%).

Model Category / Model BEYONDATOMS Avg. BEYONDATOMS (Sub. / Re-anch. / Inj. / Omi.) DiscInfer Avg. FEVER Macro-F1
NLI & Fact-Checking Baselines
NLI-DeBERTa-v3-Large 45.65 49.91 / 46.12 / 40.30 / 46.27 60.36 91.74
MiniCheck-DeBERTa 48.61 57.31 / 49.35 / 47.99 / 39.79 88.33 89.85
FActScore (Atomic Decomposition) 46.09 55.04 / 49.44 / 36.97 / 42.92 77.93 91.70
FDSpotter (Discourse Fine-tuned) 75.44 89.03 / 75.49 / 82.01 / 55.21 84.63 94.22
Generic-Prompt LLMs
Qwen3.5-4B 58.09 66.04 / 75.24 / 49.39 / 41.67 56.99 94.40
Qwen3.5-9B 61.96 71.63 / 78.56 / 55.11 / 42.55 69.83 94.58
Qwen3.5-4B-Thinking 72.18 81.73 / 90.37 / 64.86 / 51.76 78.78 94.32
Qwen3.5-9B-Thinking 76.70 84.75 / 90.79 / 74.37 / 56.88 81.42 94.12
GPT-5-mini 80.85 85.06 / 83.68 / 82.25 / 72.41 99.01 94.53
Our Method
CARVE (w/ Qwen3.5-4B) 90.87 91.60 / 82.88 / 96.25 / 92.77 97.07 92.63

Under the taxonomy-revealed (TR) prompt upper-bound reference, Qwen3.5-9B-Thinking-TR reaches 96.72% and Qwen3.5-4B-Thinking-TR reaches 94.89%. CARVE substantially outperforms all generic-prompt LLMs and specialized baselines without incurring prohibitive thinking costs.

Ablation Study

The paper conducts a systematic component ablation of CARVE across all four BEYONDATOMS subsets (reported in Macro-F1 %):

Configuration Average Macro-F1 Injection (Inj.) Omission (Omi.) Re-anchoring (Re-anch.) Substitution (Sub.) Note
Full CARVE 90.87 96.23 92.77 82.88 91.60 Full bidirectional pipeline
w/o Canonicalization (Norm.) 88.66 93.23 91.58 80.59 89.23 Unaligned surface syntax causes uniform drop
w/o Bidirectional Alignment (Bi-dir.) 76.38 96.23 35.70 82.34 91.23 Forward-only matching fails on dropped premises (-57.07%)
w/o Constrained LLM Fallback (LLM FB) 88.63 93.61 92.77 80.88 87.26 Degrades on unrecognized connective semantics

Key Findings

  • Bidirectional alignment is essential for catching dropped premises: Removing bidirectional alignment leads to a catastrophic collapse on Omission (dropping from 92.77% down to 35.70%). Forward claim-to-evidence matching cannot detect missing conditions because the claim contains no connective to anchor on, making reverse evidence scanning indispensable.
  • Atomic fact decomposition intrinsically shatters relational structures: FActScore achieves only 46.09% Macro-F1 on BEYONDATOMS, even underperforming simple NLI baselines. Breaking text into isolated propositional atoms severs structural edges, resulting in a dismal 36.97% Macro-F1 and 3.76% not-support recall on Injection.
  • Generic reasoning models remain structurally blind at high inference cost: Unprompted reasoning LLMs (Qwen3.5-9B-Thinking) exhibit severe blindness to existential errors like Omission (25.13% not-support recall) and incur runtimes exceeding \(10^4\) seconds. In contrast, CARVE achieves 90.87% Macro-F1 with over 40ร— faster inference.

Highlights & Insights

  • Operation-centric formulation over semantic classification: Moves beyond linguistic connective categorization by defining relational errors through graph-theoretic topological perturbations (Injection, Omission, Re-anchoring, Substitution), cleanly isolating structural factuality while preserving underlying factual events.
  • Compiling relation verification into deterministic topological alignment: Rather than relying on expensive end-to-end generative reasoning, CARVE compiles logical checking into claim-anchored pruning and bidirectional rule matching, achieving exceptional accuracy and efficiency.
  • Transferable takeaways: Fine-grained factual consistency pipelines must not treat propositions as disconnected sets; auditing covert conditional hallucinations requires asymmetric reverse scanning from evidence to claim.

Limitations & Future Work

  • Reliance on overt connectives hampers implicit relation verification: CARVE relies primarily on surface relational markers (e.g., because, if, after). It struggles when logical relations are implicit without overt lexical markers (e.g., "The road was icy. The car skidded."). On real-world summaries in FRANK-Relation, 76.47% of CARVE's errors stem from such implicit relations.
  • Vulnerability to upstream dependency parsing errors: Deterministic dependency parsers can fail on colloquial syntax, long-range dependencies, or structural ambiguity; cascading extraction errors propagate to downstream alignment, and frequent fallback degrades throughput.
  • Future directions: Integrating lightweight graph neural networks or dense relational representation models to identify implicit inter-sentential discourse relations without sacrificing inference efficiency.
  • vs FActScore / WiCE: Atomic decomposition assumes document factuality is the union of isolated atomic subclaims; this paper proves that decomposition systematically discards cross-event relational edges. CARVE restores structural dependencies via explicit graph alignment.
  • vs DiscInfer / LoReFact: DiscInfer focuses on semantic connective classification and primarily evaluates relation substitution; BEYONDATOMS provides exhaustive topological perturbation coverage, and CARVE bypasses costly LLM inference.
  • vs FDSpotter: While fine-tuned for discourse-level consistency, FDSpotter drops to 55.21% Macro-F1 on Omission due to low negative recall; CARVE's reverse evidence scanning robustly overcomes condition omission blind spots.

Rating

  • Novelty: โญโญโญโญโญ Formulates an operation-centric topological benchmark for relational factuality and devises an efficient bidirectional structural alignment framework.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluations across BEYONDATOMS, DiscInfer, FEVER, and out-of-domain FRANK with detailed class-wise precision/recall analysis.
  • Writing Quality: โญโญโญโญโญ Clear exposition, thorough error diagnosis, and vivid comparative case analyses.
  • Value: โญโญโญโญโญ Provides a crucial diagnostic benchmark and an actionable, lightweight verification pipeline for complex reasoning outputs in the LLM era.