Skip to content

Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/alexmartin1722/mirage
Area: LLM Reasoning
Keywords: Multimodal RAG, Factuality Evaluation, Subclaim Decomposition, Citation Attribution, Cross-Modal Reasoning

TL;DR

This paper introduces MIRAGE, the first subclaim-centric evaluation framework for multimodal retrieval-augmented generation (RAG), which decouples information quality (INFOF1: precision and recall) from citation attribution (CITEF1: support and completeness) at the atomic subclaim level across text, audio, and video sources, resolving the fundamental inability of existing sentence-level text metrics to evaluate multimodal grounding.

Background & Motivation

Online knowledge and news consumption are increasingly dominated by audiovisual media, ranging from raw smartphone video footage of breaking events to professional investigative broadcasts. As a result, next-generation retrieval-augmented generation (RAG) systems are tasked with synthesizing long-form, knowledge-grounded text from diverse non-textual evidence while providing verifiable citation attributions back to multimodal sources. However, existing evaluation protocols for long-form generation from multimodal evidence still rely almost exclusively on surface-level text overlap metrics such as ROUGE-L and BERTScore, or adapt text-native RAG metrics like ALCE, ARGUE, and RAGAS, none of which can verify generated statements against non-text sources.

Directly extending text-centric RAG metrics to multimodal sources exposes severe conceptual and computational bottlenecks. First, text-centric metrics operate at the sentence level, forcing an evaluation judge to verify composite sentences containing multiple discrete assertions in a single step; this overpenalizes natural complex sentences containing previously supported or background claims alongside newly introduced facts. Second, multimodal inference imposes an enormous computational burden: concatenating multiple multi-minute high-resolution videos into a single multimodal context window consistently triggers GPU out-of-memory (OOM) failures or necessitates aggressive frame downsampling that obliterates critical spatio-temporal visual details. Third, metrics like RAGAS rely on an exhaustive extraction of all potential facts from the retrieved evidence, an assumption that collapses when applied to continuous video feeds where potential observations are practically infinite.

To resolve these contradictions, the authors argue that factuality and citation verification cannot treat multimodal documents as ungrounded black boxes or raw video-text concatenations. Instead, all information—regardless of whether it originates from text, audio, or video—must be projected onto a unified, discrete unit of evaluation: atomic, decontextualized subclaims. Core idea: unify multimodal RAG evaluation at the subclaim level through MIRAGE, a dual-track framework that disentangles information quality (INFOF1: factuality and coverage) from citation attribution (CITEF1: citation support and attribution completeness) and supports both compute-efficient reference-based and thorough raw-collection verification across all modalities.

Method

Overall Architecture

MIRAGE decomposes the end-to-end evaluation pipeline into atomic subclaim extraction, cross-modal premise-hypothesis scoring, and dual-track precision/recall aggregation. Given a generated response \(P\) with multimodal citations, the system first splits \(P\) into sentences and prompts an LLM to decontextualize each sentence into a set of atomic, independently verifiable subclaims \(C_P\). Evaluation then proceeds along two orthogonal axes: \(\text{INFOF1}\) evaluates overall information quality by assessing subclaim factuality (\(\text{INFOP}\)) and topic coverage (\(\text{INFOR}\)); \(\text{CITEF1}\) evaluates citation efficacy by verifying whether cited documents actually entail the attributed subclaims (\(\text{CITEP}\)) and whether recalled reference subclaims are properly cited (\(\text{CITER}\)). Crucially, MIRAGE provides two operational modes: a reference-based mode that verifies claims against curated text summaries for rapid diagnostics, and a collection-based mode that grounds subclaims directly in raw multimodal evidence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Multimodal RAG Output P<br/>(with multimodal citation markers)"] --> B["Subclaim Decomposition & Decontextualization<br/>Atomic statements CP extracted per sentence"]
    B --> C["Dual-Track Evaluation Architecture: MIRAGE"]
    subgraph INFO["INFOF1: Information Quality"]
        direction TB
        D["Information Factuality INFOP<br/>Verify CP against evidence collection/reference"]
        E["Information Coverage INFOR<br/>Assess coverage of reference subclaims in P"]
    end
    subgraph CITE["CITEF1: Citation Attribution Quality"]
        direction TB
        F["Citation Support Precision CITEP<br/>Verify CP against cited sources Di"]
        G["Proper Attribution Recall CITER<br/>Check if recalled facts cite valid sources"]
    end
    C --> D
    C --> E
    C --> F
    C --> G
    D --> H["Multimodal Scoring Backend & Decision Layer<br/>Single-source inference + importance re-weighting"]
    E --> H
    F --> H
    G --> H
    H --> I["Comprehensive Multimodal RAG Audit Report<br/>Factuality, coverage, and citation grounding"]

Key Designs

1. Subclaim Decomposition and Decontextualization: Breaking Sentence-Level Bottlenecks

Traditional RAG metrics evaluate entire sentences against citations, which punishes natural discourse containing multiple clauses or background context. MIRAGE addresses this by decomposing generated text sentence by sentence using an LLM instructed to extract decontextualized subclaims \(C_P = \{c_{P,1}, c_{P,2}, \dots, c_{P,m}\}\) where pronouns and coreferences are resolved. This transforms complex cross-modal verification into discrete, atomic premise-hypothesis checks, preventing the evaluation model from rejecting an entire paragraph merely because a single modifier is unsupported.

2. Information Quality Assessment (INFOF1): Factuality and Coverage Decoupling

To determine whether the generated text presents genuine facts while thoroughly addressing the target query, INFOF1 computes the harmonic mean of precision and recall over atomic subclaims. Information Factuality (\(\text{INFOP}\)) evaluates the proportion of predicted subclaims supported by the reference evidence \(R\):

\[\text{INFOP}(C_P, R) = \frac{1}{|C_P|} \sum_{c \in C_P} s(c, R)\]

where \(s(c, R)\) is a scoring function. Under the reference-free Collection Precision setting, \(R\) consists of the full multimodal document set, evaluated efficiently via single-source maximum support \(\max_{d \in R} s(c, d)\); under Reference Precision, \(R\) is a text proxy. Conversely, Information Coverage (\(\text{INFOR}\)) measures the proportion of gold reference subclaims \(C_R\) attested in the predicted text \(P\), computed as \(\text{INFOR}(C_R, P) = \frac{1}{|C_R|} \sum_{c \in C_R} s(c, P)\).

3. Citation Quality Assessment (CITEF1): Support Precision and Attribution Recall

To differentiate between hallucinated citations and genuine multimodal grounding, CITEF1 formalizes citation fidelity at the subclaim level. Citation Support Precision (\(\text{CITEP}\)) measures whether each subclaim \(c_i^P\) is supported by the specific documents \(D_i\) cited by its parent sentence:

\[\text{CITEP}(C_P, \mathcal{D}) = \frac{1}{|C_P|} \sum_{c_i^P \in C_P} s(c_i^P, D_i)\]

avoiding ALCE's failure-prone requirement of joint document concatenation. Proper Attribution Recall (\(\text{CITER}\)) enforces that retrieved knowledge is credited to valid evidence: it constructs a bipartite mapping between reference subclaims \(c_j^R\), the documents \(D_j\) attesting them, and the sentences \(S_k\) citing those documents:

\[\text{CITER}(C_R, P, \mathcal{D}) = \frac{1}{|C_R|} \sum_{c_j^R \in C_R} \max_{d_k \in D_j} \max_{s_l \in S_k} s(c_j^R, s_l)\]

ensuring that recalled claims are backed by an accurate citation.

4. Single-Source Inference and Weighted Variants (MIRAGE++): Solving Context Scaling

Concatenating multiple video feeds into a unified VLM context window causes prohibitive GPU memory demands and token limit exhaustion. MIRAGE++ introduces single-source inference: each subclaim is scored against individual documents in isolation, taking the maximum support score \(\max_{d \in D_i} s(c_i^P, d)\). Furthermore, MIRAGE++ incorporates information importance weights \(I_i\) to prioritize salient facts over trivia, alongside source quality weights \(Q_j\) to prioritize verified media outlets over noisy user-generated content, providing adaptable evaluation for high-stakes information retrieval.

Benchmark & Evaluation Protocol

The empirical validation spans three RAG benchmark datasets representing text and audiovisual modalities: 1. RAGTIME: A TREC 2025 multilingual text-based report generation shared task used to test text-domain alignment; 2. WIKIVIDEO: An audiovisual article-generation task requiring systems to synthesize comprehensive Wikipedia-style articles from multiple raw videos with citations; 3. WIKIVIDEO-25 (MAGMaR Shared Task): A large-scale benchmark for video-grounded article generation across diverse topic domains.

The authors collect and compare against three distinct types of human judgments evaluated using Kendall's \(\tau\) rank correlation: - Extrinsic Quality Judgments (EQJs): Holistic 1–5 Likert human ratings assessing overall factuality, adequacy, coherence, relevancy, and fluency; - Intrinsic Claim Judgments (ICJs): Granular, human-annotated subclaim-level verification against curated reference texts (human-computed INFOF1); - Grounding Judgments (GJs): Direct human verification of predicted subclaims against raw video/audio evidence (human-computed multimodal INFOP/CITEP).

Model verifiers tested include audio models (Audio Flamingo 3, Qwen2-Audio), vision-language models (Qwen2.5-VL, Qwen3.5-9B/27B), and omnimodal models (Qwen2.5-Omni, CLUE-3B).

Key Experimental Results

Main Results

System-level Kendall's \(\tau\) rank correlation across all human judgment types on WIKIVIDEO demonstrates that MIRAGE significantly outperforms conventional surface-level and text-RAG metrics, particularly on raw multimodal evidence grounding (GJ).

Metric Suite / Sub-metric Mode / Variant EQJ Avg (\(\tau\)) ICJ Avg (\(\tau\)) Grounding GJ (\(\tau\))
ROUGE-L Surface Text 56.1 68.3 -11.5
BERTScore Semantic Text 69.2 78.3 0.0
ALCE (Claim Recall / CQ) Sentence / Concat 11.6 / -0.6 28.3 / - -5.2 / -8.5
ARGUE (Nugget Cov / SS) QA / Sentence -0.1 / -5.0 11.1 / - -13.3 / -3.3
RAGAS (Faithfulness) Extracted Stmt 18.2 19.5 3.0
MIRAGE: INFOP (Qwen3.5) Reference (Ref.) 39.1 77.2 -6.7
MIRAGE: INFOP (Qwen3.5) Collection (Col.) 20.8 -3.3 73.3
MIRAGE: INFOR (Qwen3.5) Reference (Ref.) 66.6 76.2 -13.3
MIRAGE: CITEP (Qwen3.5) Collection (Col.) -10.4 7.8 26.7
MIRAGE: CITER (Qwen3.5) Reference (Ref.) 42.9 64.6 -6.7

On the text-only RAGTIME benchmark, MIRAGE similarly establishes a new state of the art in human alignment over specialized text-RAG baselines:

Evaluation Dimension ROUGE-L BERTScore Auto-ARGUE MIRAGE (Qwen3.5)
Information Coverage (\(\tau\)) -16.6 29.8 54.9 77.7
Citation Support (\(\tau\)) 45.9 -11.5 66.3 79.2

Ablation Study

Evaluating claim grounding verification accuracy across modalities reveals the specific performance characteristics of current multimodal judge architectures:

Judge Model Architecture Parameters (P) Text Only Audio Only Video Only Audiovisual
Audio Flamingo 3 7B 50.9 60.0 — —
Qwen2-Audio 7B 50.9 77.9 — —
Qwen2.5-VL 7B 64.0 — 22.1 —
Qwen3.5 9B 60.1 — 76.3 —
Qwen3.5 27B 73.3 — 79.6 —
Qwen2.5-Omni 7B 65.3 81.9 50.5 45.2
CLUE 3B 63.9 89.8 74.6 70.1

Furthermore, human-annotated MIRAGE analysis (Table 11) reveals that when verification is performed by human experts, INFOF1 achieves a high correlation with overall quality (EQJ \(\tau = 70.8\text{--}76.3\)), demonstrating that the subclaim-centric design itself is intrinsically sound and robust.

Key Findings

  • Subclaim granularity is indispensable for multimodal RAG: Sentence-level metrics (ALCE and ARGUE) completely fail on video grounding, producing negative correlations with human grounding judgments (\(\tau = -13.3\) and \(-8.5\)). They penalize informative sentences containing multiple clauses and artificially reward fragmented writing.
  • Surface metrics fail to capture true evidential grounding: While ROUGE-L and BERTScore show moderate correlations with text-based quality judgments in the presence of curated Wikipedia references, their correlation with actual video grounding collapses to zero or negative (\(\tau = -11.5\) and \(0.0\)), confirming they cannot detect video-contradicting hallucinations.
  • Divergence between perceived text quality and actual grounding: Holistic human quality scores (EQJs) often reward fluent text regardless of whether the video supports it. Only collection-based subclaim metrics (\(\text{INFOP-Col}\) at \(\tau = 73.3\) and \(\text{CITEP-Col}\) at \(\tau = 26.7\)) accurately capture true evidential support in raw video.

Highlights & Insights

  • Unified representation of multimodal evidence as atomic subclaims: Rather than struggling with the near-impossible task of exhaustively summarizing multi-minute video streams, MIRAGE maps all modalities onto discrete declarative subclaims, bridging the semantic gap between vision, audio, and language.
  • Orthogonal separation of factual content and citation validity: By decomposing evaluation into \(\text{INFOF1}\) (truth of content) and \(\text{CITEF1}\) (attribution accuracy), MIRAGE cleanly identifies pathological system outputs, such as correct statements equipped with fabricated or irrelevant citations.
  • Single-source inference circumvents GPU memory limits: By replacing multi-video joint attention with independent document-level verification and max-pooling, MIRAGE eliminates context window overflow, enabling scalable automated evaluation of long-form multimodal RAG pipelines.

Limitations & Future Work

  • Inability to evaluate faithful information refusal: The current framework does not reward or verify negative assertions or refusals (e.g., "The videos provide no information regarding the suspect's vehicle"), as evaluating unanswerability across large multimodal collections remains an open challenge.
  • Bottleneck in automated omnimodal verification: Automatic MIRAGE performance is bound by the accuracy of underlying VLMs and omnimodal models. On complex audiovisual tasks, top models like CLUE achieve only ~70% verification accuracy, leaving a noticeable gap compared to human-level video inspection.
  • Future directions: Developing specialized foundation models trained on cross-modal claim verification, optimizing spatio-temporal video sampling algorithms, and formalizing negative-claim verification protocols.
  • vs ALCE (Gao et al., 2023): ALCE requires concatenating all cited passages and scores citations at the sentence level, which triggers memory blowups on video and degrades rank correlation; MIRAGE uses subclaim-level verification and single-source max-pooling to achieve superior text alignment while extending seamlessly to video.
  • vs ARGUE (Mayfield et al., 2024): ARGUE evaluates sentence support against fixed question-answer nuggets, which limits flexibility for open-domain creative generation; MIRAGE extracts subclaims bottom-up directly from the model's output, achieving higher human correlation without requiring pre-authored QA pairs.
  • vs FActScore (Min et al., 2023): FActScore pioneered atomic subclaim verification for single-source plain text; MIRAGE scales this concept to the multi-source, multimodal, citation-attributed regime.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering claim-centric evaluation framework designed specifically for multimodal, citation-attributed RAG.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously validated across three modalities, three datasets, and three independent sets of human judgments.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulations and insightful diagnostic analyses of multimodal evaluation pitfalls.
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable and open-source evaluation standard for future research in multimodal reasoning and retrieval-augmented generation.