MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Daboolu/MG2-RAG
Area: Graph Learning
Keywords: Multimodal RAG / Multimodal Knowledge Graph / Visual Grounding / Multi-Granularity Retrieval / Cross-Modal Reasoning

TL;DR¶

Addressing the issues of flat vector retrieval overlooking structural dependencies and traditional graph methods discarding visual details through costly MLLM triplet extraction, MG2-RAG integrates lightweight dependency parsing with entity-driven open-vocabulary segmentation into unified multimodal nodes, enabling low-cost, high-fidelity multi-hop retrieval and reasoning via dense similarity aggregation and Personalized PageRank.

Background & Motivation¶

Multimodal Large Language Models (MLLMs) demonstrate impressive reasoning across vision and language, yet they frequently suffer from multimodal hallucinations and factual errors in knowledge-intensive domains due to static parametric pretraining. Multimodal Retrieval-Augmented Generation (MM-RAG) mitigates these hallucinations by dynamically grounding generation on relevant evidence retrieved from external multimodal corpora. However, conventional vector-based MM-RAG approaches project textual chunks and images into an unstructured shared embedding space, retrieving isolated elements via flat vector similarity. This paradigm severs logical dependencies among disparate pieces of evidence, creating a bottleneck for multi-hop cross-modal reasoning.

To overcome this lack of relational structure, recent research has explored Multimodal Knowledge Graph (MMKG) architectures. Despite their promise, existing graph-based MM-RAG solutions encounter two severe bottlenecks in practice: first, graph construction is excessively expensive, as they rely heavily on MLLMs to extract entity-relation triplets from large multimodal corpora; second, they predominantly embrace a text-centric topology by converting visual content into descriptive text before graph synthesis. This "translation-to-text" paradigm discards atomic visual structures and fine-grained spatial evidence, impairing cross-modal alignment whenever complex hybrid queries are processed.

The core opportunity lies in bypassing MLLM-based triplet extraction entirely, aligning textual entities directly with fine-grained visual regions at the conceptual level, and coupling this compact topology with an efficient graph diffusion algorithm. Core idea: combine lightweight grammatical dependency parsing with entity-driven open-vocabulary visual grounding to fuse textual entities and visual regions into unified multimodal nodes, performing multi-granularity retrieval over an integrated heterogeneous graph via dense similarity aggregation and Personalized PageRank.

Method¶

Overall Architecture¶

MG2-RAG comprises two primary modules: Hierarchical Multimodal Knowledge Graph Construction and Multi-Granularity Graph Retrieval. The system first converts an unstructured multimodal repository into an integrated heterogeneous graph containing document chunk nodes, image nodes, and unified multimodal nodes. During retrieval, dense similarity scores computed across four semantic granularities are aggregated onto multimodal nodes, followed by graph propagation to select top-ranked document chunks as grounded contexts for MLLM generation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Knowledge Base<br/>Document chunks and associated images"] --> B["Lightweight Hierarchical Graph Construction<br/>spaCy dependency parsing + SAM3 entity-driven grounding"]
    B --> C["Modality-Preserving Node Fusion<br/>Incidence matrix mapping + Heterogeneous graph topology"]
    D["Multimodal Query<br/>Textual query and visual query"] --> E["Multi-Granularity Graph Retrieval<br/>Multimodal seed activation + Personalized PageRank"]
    C --> E
    E --> F["Top-k Grounded Document Chunks<br/>MLLM factual response generation"]

Key Designs¶

1. Lightweight Hierarchical Graph Construction: Entity-driven grounding bypassing triplet extraction overhead

Prior graph MM-RAG approaches suffer prohibitive costs due to querying MLLMs for triplet extraction across extensive corpora. MG2-RAG replaces this with a deterministic, lightweight pipeline. For the text modality, a transformer-based spaCy parser (en_core_web_trf) performs Named Entity Recognition (NER) and token-level dependency parsing on document chunks, extracting grammatical relations via predicate-centered and nominal-modifier patterns without API overhead. For the visual modality, rather than running dense, indiscriminate object detection across the entire image, textual entity names are batched as semantic prompts into an open-vocabulary segmentation model (SAM3). An entity-relevant region is retained whenever the grounding confidence \(\sigma\) exceeds threshold \(\tau\):

\[ \mathcal{V}_o^e = \{o \mid (o, \sigma) \in \Phi(e, I), I \in \mathcal{I}_k, \sigma > \tau\} \]

This targeted grounding eliminates background visual noise, links visual regions directly to textual concepts, and slashes graph construction overhead by tens of times.

2. Modality-Preserving Multimodal Node Fusion: Incidence matrices bridging cross-modal atomic evidence

Converting images to plain text descriptions loses atomic spatial evidence and fine-grained visual properties. MG2-RAG introduces a modality-preserving strategy where an entity \(e\) and its localized visual objects \(\mathcal{V}_o^e\) are bound into a unified multimodal node \(v_m = (e, \mathcal{V}_o^e) \in \mathcal{V}_M\). To maintain a compact graph topology while retaining fine-grained internal membership, two incidence matrices are defined: an Object-Multimodal Incidence Matrix \(\mathbf{M}_{\text{OMI}} \in \{0, 1\}^{|\mathcal{V}_O| \times |\mathcal{V}_M|}\) and a Sentence-Multimodal Incidence Matrix \(\mathbf{M}_{\text{SMI}} \in \{0, 1\}^{|\mathcal{S}| \times |\mathcal{V}_M|}\).

The resulting heterogeneous graph \(\mathcal{G} = (\mathcal{V}, \mathcal{E}_G)\) operates over three node granularities: chunk nodes \(\mathcal{V}_C\), image nodes \(\mathcal{V}_I\), and unified multimodal nodes \(\mathcal{V}_M\). The edge set \(\mathcal{E}_G\) spans three complementary dimensions: (1) Contextual Edges, connecting chunks to their child images and multimodal nodes to preserve document provenance; (2) Semantic Edges, linking multimodal nodes sharing syntactic grammatical dependencies; and (3) Grounding Edges, connecting visual objects to their parent image nodes weighted by grounding confidence \(\sigma\). This topology jointly preserves structural hierarchy, language logic, and visual spatial grounding.

3. Multi-Granularity Graph Retrieval: Dense similarity aggregation and random-walk propagation

Flat vector retrieval often breaks down on multi-hop hybrid queries. MG2-RAG utilizes a shared EVA-CLIP-8B encoder to compute dense cosine similarities between queries (\(q_t, q_v\)) and embeddings across four granularities: sentences, chunks, images, and visual objects. Intermediate sentence and object similarities are projected onto primary multimodal nodes via the incidence matrices:

\[ \mathbf{s}_M^{(m)} = \mathbf{D}_S^{-1} \mathbf{M}_{\text{SMI}}^\top \mathbf{s}_S^{(m)} + \mathbf{D}_O^{-1} \mathbf{M}_{\text{OMI}}^\top \mathbf{s}_O^{(m)} \]

where \(\mathbf{D}_S\) and \(\mathbf{D}_O\) are diagonal degree normalization matrices. Modality activations are balanced via scaling factors \(\omega_C, \omega_I\) and fused into initial seed vector \(\mathbf{r}_0 = \lambda_t \mathbf{u}^{(t)} + \lambda_v \mathbf{u}^{(v)}\). Keeping the top-k seed distribution as a restart anchor, Personalized PageRank (PPR) propagates relevance across the heterogeneous graph:

\[ \mathbf{r}_{\ell+1} = \alpha \mathbf{W} \mathbf{r}_\ell + (1 - \alpha) \mathbf{r}_0 \]

where \(\mathbf{W}\) is the column-normalized transition matrix and \(\alpha \in (0, 1)\) controls diffusion reach. Upon convergence, chunk node activations determine the top-\(k\) most relevant document chunks to inject into the MLLM generation context.

Key Experimental Results¶

Main Results¶

The framework was evaluated across multimodal retrieval, knowledge-based VQA, multimodal reasoning, and classification. The table below presents the core retrieval and question answering results on E-VQA and InfoSeek:

Model	Retrieval Mechanism / Backbone	E-VQA R@1↑	E-VQA R@10↑	InfoSeek R@1↑	InfoSeek R@10↑	E-VQA VQA (All)↑	InfoSeek VQA (All)↑
CLIP ViT-L/14	Single Modality (V→V)	17.8	36.4	32.7	59.6	-	-
EVA-CLIP-8B	Cross-Modality (V→T)	42.0	69.5	56.5	82.2	-	-
EchoSight	Dense Vector Retrieval / LLaMA-3.1-8B	-	-	-	-	47.23	30.40
mKG-RAG	Multimodal Graph / LLaMA-3.1-8B	-	-	49.7	78.0	-	32.10
VaLiK (5k)	MLLM Triplet Graph / Qwen2.5-VL-7B	-	-	-	-	15.22	2.51
MMGraphRAG (5k)	Scene Graph / Qwen2.5-VL-7B	-	-	-	-	16.52	0.50
MG2-RAG (Ours)	Multi-Granularity Graph / LLaMA-3.1-8B	44.9	72.0	59.6	83.8	47.77	32.58
MG2-RAG (Ours)	Multi-Granularity Graph / Qwen2.5-VL-7B	44.9	72.0	59.6	83.8	48.59	35.48
MG2-RAG (Ours)	Multi-Granularity Graph / Qwen3.5-27B	44.9	72.0	59.6	83.8	52.19	37.87
MG2-RAG (Ours)	Multi-Granularity Graph / GPT-5.2	44.9	72.0	59.6	83.8	60.30	39.30

In terms of graph construction efficiency, MG2-RAG achieved up to a 77.1x speedup on ScienceQA, with an average 43.3x speedup and a 23.9x cost reduction across all evaluated benchmarks compared to MLLM-based baselines (VaLiK and MMGraphRAG).

Ablation Study¶

The table below summarizes component ablations across E-VQA (5k), ScienceQA, and CrisisMMD:

Config	E-VQA R@1 / R@5↑	E-VQA BEM Score (All)↑	ScienceQA Acc. (Avg.)↑	CrisisMMD Acc. (Avg.)↑	Note
MG2-RAG (Full Model)	57.8 / 83.1	60.24	97.85	59.12	Complete framework with HGC, MNF, and GP
w/o HGC (Hierarchical Graph Construction)	40.7 / 61.5	51.72	97.48	56.70	Collapses to flat retrieval; E-VQA R@1 drops 17.1%
w/o MNF (Multimodal Node Fusion)	43.8 / 78.0	55.38	97.67	58.62	Disconnects text entities from objects; BEM drops 4.86
w/o GP (Graph Propagation)	25.5 / 76.0	54.69	97.51	58.90	Relies only on initial seed similarity; R@1 plummets 32.3%

Key Findings¶

Graph Propagation (GP) drives precise top-1 evidence retrieval: Removing GP causes R@1 on E-VQA to plummet from 57.8% to 25.5%, demonstrating that dense similarity alone frequently gets trapped in superficial matches, whereas topological random-walk diffusion successfully identifies interconnected evidence.
Hierarchical and multimodal alignment (HGC & MNF) preserve factuality: Omitting HGC or MNF drops the VQA BEM score by 8.52 and 4.86 points, respectively. This highlights that grounding textual entities onto specific localized visual objects is essential for preventing MLLM hallucination during multi-hop reasoning.
Linear scalability across repository size: When the knowledge base scales from 1k to 100k documents, construction time grows almost linearly (scaling to 32.8 hours on dual RTX 6000 Ada GPUs for 100k items), proving the feasibility of large-scale deployment.

Highlights & Insights¶

Entity-driven grounding replaces exhaustive vision parsing: Prompting SAM3 directly with textual entities avoids generating hundreds of redundant bounding boxes per image, achieving high-precision visual localization at minimal computational cost.
Incidence matrices avoid graph over-expansion: Modeling sentences and object regions as incidence projection weights rather than explicit graph nodes keeps the heterogeneous graph compact, preventing topological sparsity during PPR diffusion.
Training-free, modular architecture: The entire framework requires no additional parameter fine-tuning, allowing seamless integration with any off-the-shelf vision-language embedding model and frontier MLLM generator.

Limitations & Future Work¶

Author-admitted limitations: For rare, long-tail technical entities, rule-based dependency parsing or SAM3 open-vocabulary prompts may miss subtle visual cues or suffer from false positives.
Observed limitations: Hyperparameters such as the PPR restart probability \(\alpha\) and modality scaling factors \(\omega_C, \omega_I\) currently require empirical tuning across different datasets.
Future work: Incorporating adaptive gating mechanisms to automatically estimate modality reliance based on query characteristics would enhance robustness across diverse data distributions.

vs VaLiK / MMGraphRAG: Traditional MMKGs rely on costly MLLM calls to extract text triplets or convert visual content into captions; MG2-RAG leverages lightweight parsing and SAM3 grounding, yielding a 43.3x speedup while preserving native visual regions.
vs EchoSight / Flat MM-RAG: Vector retrieval isolates textual and visual features in a flat embedding space, failing on multi-hop dependency chains; MG2-RAG uses graph diffusion over unified multimodal nodes to systematically retrieve linked evidence.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Decoupling graph construction with targeted visual grounding and incidence-based fusion offers an elegant alternative to MLLM triplet extraction.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across four diverse multimodal tasks, including 100k scalability tests and detailed ablation analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, crisp motivation, and clear mathematical formulations.
Value: ⭐⭐⭐⭐⭐ Drastically cuts the cost and latency bottlenecks of multimodal Graph RAG, providing high practical utility.