Skip to content

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/guihuzhang/xqdt
Area: Interpretability
Keywords: Data-Text Alignment, Explainable Metric, Fine-grained Error Localisation, Closed-form Quantification, Bidirectional Generation Repair

TL;DR

XQDT introduces an end-to-end explainable and quantitative bidirectional data-text alignment metric that fine-tunes compact open-source models on semantically perturbed data to explicitly predict atomic unit labels (Correct, Omitted, Incorrect, Extra), aggregates these counts into closed-form precision, recall, and F1 scores, and provides structured feedback signals for iterative generation refinement.

Background & Motivation

Bidirectional data-text alignment is foundational across both data-to-text generation and text-to-data information extraction, requiring structured triples and natural language text to faithfully convey identical semantic contents without omissions, distortions, or hallucinations. However, existing evaluation paradigms remain trapped in an unsatisfactory dilemma between uninterpretable scalar metrics and expensive, noisy prompt-based judges. Conventional overlap-based measures such as BLEU, METEOR, and PARENT measure surface n-gram coverage rather than semantic fidelity. While learned neural evaluators—including NLI-based entailment models, regression frameworks like MonoLR, and QA-based metrics like Data-QuestEval—exhibit stronger alignment with human judgments, they inevitably condense evaluation into black-box scalar ratings. As a consequence, they offer zero actionable explanations detailing which specific facts were omitted, which entities were distorted, or where unsupported text was fabricated.

Conversely, prompting frontier large language models as judges (LLM-as-Judge, such as GPT-4 and GPT-5) can generate natural language rationales, but suffers from steep computational cost, substantial API latency, prompt sensitivity, and severe over-prediction tendencies that yield uncalibrated quantitative scores. More fundamentally, a major structural disconnect exists between localized error attribution and axiomatic metric formulation: existing factuality metrics for open-ended text (such as FActScore and SAFE) decompose generated text into atomic propositions and verify them against external knowledge sources, but they fail to support bidirectional alignment with structured inputs where correct, omitted, and distorted elements must be anchored to the structured input graph, while extraneous information must be derived and grounded from the text. Furthermore, high-quality, fine-grained error annotations at the triple level are virtually absent in natural corpora, making naive negative sampling prone to learning trivial topic-mismatch shortcuts rather than subtle factual inconsistencies.

To bridge this gap between fine-grained error diagnosis and quantitative measurement, this paper approaches alignment evaluation by formalizing data-text consistency as atomic unit-level multi-label classification over relational triples, trained through topology-preserving semantic perturbations and mapped via closed-form algebraic formulas into precision, recall, and F1 scores that directly feed downstream iterative refinement. Core idea: formalize bidirectional data-text alignment as atomic four-way error classification (Correct, Omitted, Incorrect, Extra) trained via topology-preserving adversarial perturbations and algebraically aggregated into closed-form precision, recall, and F1 scores, achieving explainable evaluation, superior human correlation, and actionable feedback for iterative generation repair.

Method

Overall Architecture

Given a data-text pair \((D, T)\), where \(D\) represents a set of structured atomic triples \(\langle s, p, o \rangle\) and \(T\) is an English text verbalizing or underlying \(D\), XQDT operates through three tightly coupled stages: an offline semantic perturbation pipeline that constructs high-fidelity multi-error synthetic training pairs; an online verification and scoring module where a fine-tuned generative Transformer identifies atomic differences and computes closed-form precision, recall, and F1 scores; and a downstream enhancement loop where predicted local errors are converted into structured feedback to guide a revision LLM across iterative correction rounds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data-Text Pair<br/>(Structured Data D, Text T)"] --> B["Topology-Preserving Semantic Perturbations<br/>Synthesize Diverse Multi-Error Negative Samples"]
    B --> C["Atomic Error Classification & Closed-Form Aggregation<br/>Predict 4-Way Labels & Compute Closed-Form P / R / F1"]
    C --> D["Structured Feedback for Iterative Repair<br/>Convert Local Errors to Prompts for Downstream Revision"]
    D --> E["Output Alignment Metrics & Refined Data / Text"]

Key Designs

1. Topology-Preserving Semantic Perturbations: Superficial Shortcut Learning & Data Scarcity -> Relation-Constrained Orthogonal Perturbation Pipeline -> High-Fidelity Multi-Error Benchmark Synthesis

Fine-grained alignment errors are rarely annotated at scale in natural corpora, and random negative sampling produces superficial topical discrepancies that models easily bypass via coarse semantic shift detection. To force the verifier to focus on genuine factual consistency, XQDT implements three relation-constrained perturbation strategies on semantically aligned gold pairs from WebNLG 3.0 and cleaned E2E: - Omitted Data: Simulates missing information by creating instances where triples exist in \(D\) but are unmentioned in \(T\). This is achieved either through related subgraph retrieval (retrieving a candidate graph \(D_2\) sharing at least one entity or predicate with \(D_1\), filtering out near-duplicates sharing \(\ge 2\) elements, and capping additions at 1–4 triples) or through field-anchored fabrication (selecting a triple from \(D_1\), fixing one field such as the predicate, and replacing the remaining elements using semantic co-occurrence pools filtered with BGE-1.5-Large embeddings below a cosine similarity threshold of 0.65). - Extra Content: Symmetrically simulates unsupported hallucinations in text. It either deletes triples \(\{d^-\}\) from the input data while holding the text unchanged, or concatenates two semantically related texts (\(T_1 \parallel T_2\)) while retaining only the first graph as input. - Incorrect Triples: Simulates factual distortion and argument misplacement. While keeping the text unchanged, a triple \(d_i \in D\) is modified by replacing one element with a semantically related alternative from the co-occurrence pool, or by swapping its subject and object. Symmetric predicates (e.g., "synonym", "shares border with") are strictly excluded to avoid generating semantically valid equivalents.

These deviations are sampled independently through Bernoulli trials to allow multi-error co-occurrence, adhering to a strict operational topology (e.g., omitted additions are excluded from subsequent extra deletions or incorrect corruptions). Finally, the corrupted text \(\tilde{T}\) is shuffled at the sentence level and data \(\tilde{D}\) at the triple level to neutralize positional cues. Manual human audits confirm a 93.9% perturbation validity and an 85.6% absence of unlabelled noise.

2. Atomic Error Classification & Closed-Form Aggregation: Black-Box Scalar Outputs & Multi-Step Overhead -> Grounded Unit-Level Categorization with Axiomatic Counting -> Fine-Grained Diagnosis & Interpretable Quantification

Existing explainable evaluators rely on multi-hop question answering or open-ended reasoning chains that are computationally demanding and challenging to quantify reliably. XQDT grounds evaluation directly in atomic semantic units: - Correct: Triple \(d_i \in D\) is faithfully, accurately, and completely verbalized in text \(T\). - Incorrect: Triple \(d_i \in D\) is verbalized in \(T\) but contains a factual distortion (e.g., one replaced element or swapped subject/object) while remaining clearly traceable to \(d_i\). - Omitted: Triple \(d_i \in D\) is absent from \(T\), or distorted across two or more elements such that it cannot be recognized. - Extra: Content expressed in \(T\) lacks any supporting representation in \(D\), extracted and verbalized as an atomic triple.

XQDT aggregates these localized discrete counts into continuous precision (\(P\)), recall (\(R\)), and harmonic mean (\(F_1\)) scores via closed-form algebraic equations:

\[P = \frac{|\text{Correct}|}{|\text{Correct}| + |\text{Extra}| + |\text{Incorrect}|}\]
\[R = \frac{|\text{Correct}|}{|\text{Correct}| + |\text{Omitted}| + |\text{Incorrect}|}\]
\[F_1 = \frac{2 \times P \times R}{P + R}\]

Crucially, an Incorrect unit penalizes both denominators simultaneously: it represents an unfulfilled input requirement (lowering recall) and an inaccurate realization in the output (lowering precision). In contrast, Omitted units penalize recall only, whereas Extra units penalize precision only. This formulation mathematically mirrors classical information retrieval axioms while preserving exact entity-level traceability.

3. Structured Feedback for Iterative Repair: Static Evaluation Scores Unable to Guide Correction -> Discrete Error Mapping to Local Edit Directives -> Significant Quality Gains in Downstream Generation

Conventional metrics produce only an evaluation score, forcing downstream generators to resample or rewrite from scratch, which frequently introduces new hallucinations. XQDT leverages its fine-grained error predictions as structured revision feedback for both generation directions: - Text-to-Data Extraction Repair: When base extractors (e.g., BT5, ReGen) yield misaligned triple sets, the verifier identifies Extra triples (flagged for removal as unsupported), Incorrect triples (flagged for targeted correction), and Omitted facts (flagged for addition). A revision LLM updates the triples within a structured Markdown table, performing localized symbolic edits without destabilizing correct facts. - Data-to-Text Generation Repair: When generators (e.g., Control Prefix) produce hallucinated or incomplete text, the verifier's feedback specifies exact missing, unsupported, and inconsistent triples. The revision model revises only the affected sentence fragments while preserving previously aligned prose under a strict minimal-edit constraint.

Iterating this revision loop across up to 3 rounds (R1 \(\to\) R3) converts the passive evaluator into an active refinement module.

Loss & Training

XQDT employs open-source autoregressive Transformers (Gemma 3, Qwen 3, Llama 3) fine-tuned on the formatted \((D, T)\) prompt pairs to generate the labeled error difference set \(E\) via standard cross-entropy loss. - LoRA Configurations: LoRA is applied across all linear attention and MLP layers with dropout 0.05. For models \(\le 1.7\text{B}\), rank \(r = 8\) and scaling \(\alpha = 16\); for models 3B–14B, \(r = 16\) and \(\alpha = 32\). - Optimization Strategy: Training utilizes a cosine-with-restarts schedule (15 cycles after 1-cycle warmup) for up to 16 epochs with early stopping after 2 non-improving epochs. Learning rates are scaled inversely with model capacity: \(1 \times 10^{-4}\) for \(<1\text{B}\), \(5 \times 10^{-5}\) for 1B–4B, and \(3 \times 10^{-5}\) for 8B–14B. - Training Balance: Training runs on NVIDIA H100 GPUs in bf16 precision. WebNLG-derived instances are weighted at approximately a 4:1 ratio over E2E instances to prevent overfitting to low-diversity schemas.

Key Experimental Results

Main Results

XQDT was evaluated on synthetic error detection benchmarks, on human correlation datasets (4L-RP-Human, WebNLG 2020/2017, E2E), and on downstream iterative text-data refinement.

Table 1: Triple-Level F1 (%) for Error Detection on WebNLG Synthetic Test Split

Model Model Class Extra F1 Omitted F1 Incorrect F1 Macro F1
Qwen3-32B-Prompt Prompted Judge 2.8 1.2 47.6 17.2
Gemma3-27B-Prompt Prompted Judge 4.6 14.4 43.2 20.7
GPT-4.1-Prompt Prompted Judge 16.8 30.8 64.1 37.2
Llama3.3-70B-Prompt Prompted Judge 13.2 51.8 52.3 39.1
GPT-5.1-Prompt Prompted Judge 18.0 81.7 78.1 59.2
Qwen3-0.6B (Ours) Fine-tuned XQDT 69.7 96.2 95.2 87.1
Gemma3-1B (Ours) Fine-tuned XQDT 66.8 94.5 92.3 84.5
Llama3.1-8B (Ours) Fine-tuned XQDT 82.2 93.8 95.6 90.5
Gemma3-4B (Ours) Fine-tuned XQDT 78.3 97.6 96.5 90.8
Qwen3-8B (Ours) Fine-tuned XQDT 79.2 96.8 96.9 90.9
Qwen3-14B (Ours) Fine-tuned XQDT 81.3 96.8 97.1 91.7

Table 2: Spearman Rank Correlation (\(\rho\)) with 4L-RP-Human Precision, Recall, and F1 Annotations

Model Evaluator Type \(P_m \text{ vs } P_h\) \(R_m \text{ vs } R_h\) \(F1_m \text{ vs } F1_h\)
Data-QuestEval (DQE) QA-based Metric 61.2 58.3 62.5
MonoLR Regression Metric 71.0 63.7 70.6
NLI-Based Entailment Metric 73.4 73.2 73.3
FactSpotter (FS) Self-supervised Recall — 86.5 —
Qwen3-32B-Prompt Prompted Judge 35.9 30.4 35.0
GPT-5.1-Prompt Prompted Judge 80.9 76.8 78.5
Qwen3-0.6B (Ours) Fine-tuned XQDT 71.8 79.3 78.4
Gemma3-4B (Ours) Fine-tuned XQDT 79.0 87.6 87.4
Llama3.2-3B (Ours) Fine-tuned XQDT 85.0 80.4 85.9
Llama3.1-8B (Ours) Fine-tuned XQDT 82.4 78.3 84.3
Qwen3-8B (Ours) Fine-tuned XQDT 88.6 88.9 89.6

Ablation Study & Downstream Refinement

Table 3: Downstream Text-to-Data Extraction Repair on GenWiki (Exact F1 %)

Base Extractor Repair Guidance Revision Stage Gemma3-12B Llama3.1-8B Qwen3-14B
BT5 Baseline (No repair) — 40.89 40.89 40.89
BT5 Self-Refine Round 1 42.13 40.26 41.49
BT5 Self-Refine Round 3 41.96 40.01 40.27
BT5 FactSpotter Round 3 36.96 38.92 33.10
BT5 NLI Round 3 49.36 41.58 40.14
BT5 PiVe Round 3 40.89 40.83 40.89
BT5 XQDT (Ours) Round 1 48.60 44.65 43.26
BT5 XQDT (Ours) Round 3 50.79 (+9.90) 47.80 (+6.91) 47.08 (+6.19)
ReGen Baseline (No repair) — 47.53 47.53 47.53
ReGen NLI Round 3 51.25 46.37 45.69
ReGen XQDT (Ours) Round 3 51.95 (+4.42) 50.48 (+2.95) 49.48 (+1.95)

Key Findings

  • Compact Fine-Tuned Verifiers Drastically Outperform Prompted Judges: Fine-tuned Qwen3-0.6B achieves 87.1% Macro F1 on synthetic error detection, substantially outperforming prompt-based Llama3.3-70B (39.1%) and GPT-5.1 (59.2%). Zero-shot prompted judges severely over-predict errors, yielding dismal Extra F1 scores of 2.8%–18.0%.
  • Extra Content Detection Presents the Highest Difficulty: While fine-tuned models achieve near-perfect detection on Omitted (up to 97.6%) and Incorrect (up to 97.1%) errors, Extra detection tops out around 82.2%. Detecting extra content requires extracting and converting unsupported free-form natural language into structured triples without relying on input copy mechanisms.
  • Model Scaling plateaus Around 8B: Qwen3-8B achieves the highest correlation with human judgments (89.6% F1 Spearman \(\rho\)), outperforming Qwen3-14B (69.5%). Beyond 8B parameters, models tend to overfit the controlled surface artifacts of synthetic perturbations, diminishing generalization to subtle, naturally occurring human errors.
  • Fine-Grained Feedback Drives Substantial Downstream Improvements: In text-to-data refinement, unguided Self-Refine degrades performance over multiple rounds (dropping from 42.13% to 41.96%), whereas XQDT-guided revision increases BT5 Exact F1 on GenWiki by up to 9.90 percentage points (reaching 50.79%).

Highlights & Insights

  • Seamless Coupling of Atomic Attribution and Axiomatic Aggregation: The framework establishes an elegant mathematical bridge between fine-grained error identification and continuous scoring, resolving the longstanding compromise between interpretability and quantitative reliability.
  • Topology-Preserving Adversarial Perturbation Design: By enforcing graph co-occurrence constraints and embedding distance filtering, the synthetic pipeline prevents trivial lexical shortcuts and constructs natural multi-error distributions.
  • Direct Repurposing of Diagnostic Evaluators into Refinement Supervisors: Because each verifier error label maps directly to a discrete edit action (Omitted \(\to\) insert, Extra \(\to\) delete, Incorrect \(\to\) substitute), XQDT functions as an off-the-shelf verification and reward engine for iterative generation pipelines.

Limitations & Future Work

  • Triple-Based Semantic Representation Constraints: The model assumes subject-predicate-object triples as atomic units. While ideal for knowledge graph and table verbalization, this formulation cannot easily represent richer semantic structures such as Abstract Meaning Representation (AMR) or discourse relation graphs.
  • Rigid Error Counting vs. Human Partial Credit: Any element mismatch marks an entire triple as Incorrect, whereas human evaluators frequently assign partial credit for partially accurate verbalizations, creating a minor systematic scaling divergence against absolute human ratings.
  • Inference Latency with Extensive Hallucinations: In extreme cases where generated text contains abundant unsupported information, generating all corresponding Extra triples autoregressively increases inference decoding time compared to scalar regression heads.
  • vs MonoLR: MonoLR directly regresses precision and recall scores without generating local error rationales; XQDT achieves both explicit error localization and superior correlation with human judgments (F1 correlation 89.6 vs 70.6).
  • vs Data-QuestEval (DQE): DQE requires a complex, multi-stage question generation and answering pipeline prone to cascading errors; XQDT performs end-to-end classification in a single unified generative pass.
  • vs FactSpotter: FactSpotter evaluates recall only and cannot detect hallucinated additions; XQDT provides fully symmetric bidirectional quantification covering both precision and recall.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the unification of localized atomic error classification with closed-form axiomatic precision/recall quantification.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across synthetic error detection, multiple human correlation benchmarks, and multi-round bidirectional downstream generation repair.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity in motivation, formal definitions, and empirical ablations.]
  • Value: ⭐⭐⭐⭐⭐ [Provides a practical, open-source verification framework and feedback signal for improving data-to-text and text-to-data systems.]