Skip to content

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/xyfyyds/ReGraP
Area: Multimodal Vision-Language Models
Keywords: Multimodal Large Language Models, Personalized Assistant, Knowledge Graph, Chain-of-Thought Reasoning, Graph Prompting

TL;DR

Addressing the limitation that existing personalized MLLMs focus only on isolated concepts while ignoring relational reasoning, this paper constructs the ReGraP dataset containing 120 personalized knowledge sets with knowledge graphs and CoT QA pairs, and introduces ReGraP-LLaVA that aligns structured relational graphs into language semantic space via soft and hard graph prompting.

Background & Motivation

Multimodal Large Language Models (MLLMs) have established impressive benchmarks across general vision-language understanding and visual question answering tasks. To transform these foundation models into practical personalized AI assistants, recent works introduce personalized concepts (e.g., private user belongings, unique pets, or specific characters) to generate tailored descriptions and recognize user-specific entities. However, current personalization methodologies suffer from structural deficiencies: prompt-based approaches depend exclusively on raw textual descriptions inserted into user prompts, which causes performance to degrade rapidly when prompts are noisy or when scenarios involve complex multi-object interactions; conversely, tuning-based approaches such as Yo'LLaVA and MyVLM adapt models at the concept-learning level by adding token embeddings or lightweight modules, primarily evaluating isolated concept recognition and simple captioning.

Human cognitive processing of personalized environments relies inherently on structured relational networks that interconnect entities, attributes, and contextual affordances. For instance, answering a natural user query such as "Who can help Bocchi in lyric writing?" demands far more than localizing individual band members in the visual field; it requires traversing an interconnected relational graph: recognizing that Bocchi needs songwriting guidance, that Ryo possesses lyric-writing skills within Kessoku Band, and that an established mentorship relation connects the two. Existing personalized datasets and models almost entirely lack relational knowledge graph structures and multi-step reasoning supervision, leaving models incapable of executing compositional deductions across linked concepts.

To bridge this gap, this paper integrates structured knowledge graphs (KGs) and multimodal Chain-of-Thought (CoT) reasoning into personalized visual assistance. Core idea: construct the ReGraP benchmark featuring multimodal triplets of images, knowledge graphs, and CoT reasoning QA pairs, and introduce ReGraP-LLaVA, which incorporates relational knowledge via dual-track soft GNN embeddings and discrete reasoning token hard prompts to empower personalized MLLMs with multi-step relational reasoning.

Method

Overall Architecture

The ReGraP-LLaVA framework is underpinned by two core technical components: an automated data generation pipeline that extracts structured knowledge graphs and multi-step reasoning routes from raw personal knowledge, and a graph-aligned multimodal training architecture that injects relational graph structures into LLaVA via soft and hard graph promptings. The data pipeline first disentangles entities, relational triples, and concept attributes from images and raw user text to synthesize comprehensive knowledge graphs; it subsequently applies reasoning tree topological reordering and depth-first search (DFS) to extract multi-hop reasoning subgraphs, prompting GPT-4o to formulate CoT question-answering pairs. During training, the framework supports both continuous soft graph embeddings produced by a GNN-MLP projector and discrete hard graph prompts composed of dedicated reasoning tokens added to the tokenizer. Parameter-efficient fine-tuning via LoRA on attention projection layers ensures scalable adaptation without catastrophic forgetting of foundational multimodal competencies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Personalized Multimodal Input<br/>Image Collection I + Textual Knowledge K_text"] --> B["Personalized KG & CoT Generation Pipeline<br/>Disentangle Entities/Attributes + Reasoning Tree DFS Traversal"]
    B --> C["Dual-Track Graph Structure Alignment<br/>GNN Continuous Soft Prompting / Reasoning Token Discrete Hard Prompting"]
    C --> D["Parameter-Efficient Tuning & Vocabulary Expansion<br/>Frozen Vision Backbone + Expanded Linear Head + LoRA Adaptation"]
    D --> E["Personalized Relational Reasoning & Contextual Output"]

Key Designs

1. Personalized KG and CoT Generation Pipeline: Disentangling Relational Knowledge and Traversing Reasoning Routes To overcome the bottleneck of isolated concept labels in prior benchmarks, the pipeline constructs complete personalized knowledge sets \(S = (I, G, \mathcal{P}_{(QA, R)})\). Raw user textual knowledge \(K_{\text{text}}\) is split into relational knowledge \(K_R\) and attribute knowledge \(K_A\). A Relation-Graph Builder prompts GPT-4o with \(K_R\), extracted entity set \(E\), and image collection \(I\) to construct an initial relation graph \(G_R = \{(h_i, r_i, t_i)\}_{i=1}^n\). A KG Enricher then incorporates attribute nodes \(N_{\text{new}}\) and relational edges to yield the final knowledge graph \(G\). To synthesize realistic deductive reasoning data, the pipeline performs hierarchical topological re-ordering of nodes into reasoning trees \(T\), applying depth-first search (DFS) to extract the longest reasoning routes \(S_R = \text{DFS}(T)\). Each route \(R \in S_R\) serves as the explicit thinking trajectory provided to GPT-4o to formulate relational questions \(Q\) and step-by-step CoT answers \(A_{\text{CoT}} = \text{GPT}(Q, R)\).

2. Dual-Track Graph Prompting Mechanisms: Seamlessly Mapping Graph Topology into Autoregressive Language Space To bridge the dimensional and structural mismatch between non-Euclidean graph topologies and sequential autoregressive language tokens, the authors formulate soft and hard graph prompting mechanisms. Under Soft Prompting, a target reasoning subgraph is transformed into a one-hot encoded graph \(g\), processed through a Graph Neural Network (GNN) to derive a structural embedding \(H_g = \text{GNN}(g)\), and projected by an MLP into the LLM's hidden vector space of dimension \(d\): $\(\hat{H} = \text{MLP}(H_g) \in \mathbb{R}^d\)$ This vector \(\hat{H}\) is directly concatenated with token embeddings derived from textual queries and visual projection tokens. Under Hard Prompting, the subgraph \(g = \{(E_i, r_i, E_{i+1})\}_{i=1}^N\) is linearized into a discrete sequence of natural language statements structured by dedicated reasoning tokens. The model introduces \(N+1\) entity tokens and \(N\) relation tokens (e.g., <Relation_i>), formatting the graph into templates like "<E1> and <E2> share <Relation1>...". This enables self-attention layers to explicitly calculate token-to-token cross-attention weights over relational dependencies.

3. Parameter-Efficient Tuning with Vocabulary Expansion: Safeguarding Foundation Capacities while Learning Personalization To inject dense relational knowledge without degrading general image-text capabilities, ReGraP-LLaVA enforces strict parameter isolation. The pre-trained vision encoder, vision projector, and base tokenizer are entirely frozen. To support the \(2N+1\) reasoning tokens under hard prompting, the tokenizer expands into a Reasoning Tokenizer, and the final language model classification head matrix \(W_{c \times n}\) is expanded by \(2N+1\) dimensions to \(W_{c \times (n+2N+1)}\). Only the newly added token embeddings, the expanded slice of the output projection matrix, and LoRA parameters applied to the query and value projection matrices (q_proj, v_proj) are set to trainable status: $\(\theta_{\text{new}} = \{\langle E_1 \rangle, \dots, \langle E_{N+1} \rangle, \langle \text{Relation}_1 \rangle, \dots, \langle \text{Relation}_N \rangle, W_{(:, n+2N+1)}\} \cup \theta_{\text{LoRA}}\)$ This lightweight configuration allows full adaptation on a single NVIDIA A6000 GPU while providing an architectural firewall against catastrophic forgetting.

A Worked Example

Consider the desktop environment set <electronic_workspace>, containing an Asus ROG laptop, Filco wired mechanical keyboard, Asus ROG wireless mouse, Philips monitor, and Bose wired headphones. Given the query: "How can the owner improve the gaming experience, and why?" 1. Subgraph Hard-Prompt Activation: Discrete reasoning tokens <ROG_mouse>, <ROG_laptop>, and <connected_via_Bluetooth> are instantiated alongside their property nodes. 2. CoT Reasoning Execution: The model activates the internalized relational chain: "Currently, the Asus ROG mouse is connected wirelessly via Bluetooth to the Asus ROG laptop, introducing minor latency; in gaming contexts, a wired connection delivers higher stability and minimal latency; the Filco keyboard is already situated at a 45-degree angle for ergonomic comfort." 3. Structured Contextual Output: The assistant correctly deduces and outputs that the owner should switch the ROG mouse from Bluetooth to a wired cable connection, demonstrating multi-hop deduction that seamlessly links hardware attributes to functional performance.

Loss & Training

The network optimizes standard autoregressive language modeling cross-entropy over instruction-response triplets \((X_i, I, X_a)\), calculating loss exclusively across the answer tokens \(X_a\): $\(\mathcal{L} = - \sum_{j=1}^{L} \log p_\theta (x_j \mid I, X_i, X_{a, <j})\)$ Training utilizes the AdamW optimizer with learning rate \(1 \times 10^{-5}\), weight decay 0.1, CosineAnnealingLR scheduling, and mixed-precision GradScaler. Converging a personalized knowledge set requires approximately 10 images and 20 CoT QA pairs.

Key Experimental Results

Main Results

The authors evaluate performance on the ReGraP Benchmark across 8,160 questions spanning Multiple-Choice, Fill-in-the-Blank, True/False, and closed Descriptive QA, categorized into basic attribute recognition ("Simple") and relational deduction ("Difficult"). As documented in Table 1, ReGraP-LLaVA outperforms all baselines across almost every task, achieving a 5.3% weighted accuracy gain over the strongest fine-tuned model LLaVA (CoT) and an 8.8% margin over the strongest prompt-based model GPT-4o (Prompt).

Model MC (Simple) MC (Diff.) Fill (Simple) Fill (Diff.) T/F (Simple) T/F (Diff.) Desc. (Simple) Desc. (Diff.)
LLaVA-7B (Prompt) 0.786 0.684 0.813 0.647 0.908 0.784 0.892 0.783
LLaVA-13B (Prompt) 0.829 0.705 0.883 0.673 0.920 0.888 1.000 0.913
Qwen2-VL-7B (Prompt) 0.794 0.688 0.858 0.633 0.898 0.878 0.925 0.842
Qwen2.5-VL-7B (Prompt) 0.798 0.683 0.865 0.642 0.922 0.874 0.958 0.858
Qwen2.5-VL-72B (Prompt) 0.875 0.714 0.882 0.677 0.920 0.878 0.992 0.950
GPT-4o (Prompt) 0.863 0.735 0.862 0.668 0.938 0.890 0.950 0.929
Yo'LLaVA (Raw) 0.814 0.695 0.862 0.668 0.887 0.765 0.900 0.767
Yo'LLaVA (CoT) 0.849 0.725 0.860 0.675 0.908 0.832 0.875 0.763
LLaVA (Raw) 0.865 0.762 0.863 0.753 0.893 0.840 0.850 0.796
LLaVA (CoT) 0.885 0.829 0.890 0.817 0.947 0.877 0.917 0.867
ReGraP-LLaVA (Ours) 0.942 0.892 0.940 0.858 0.967 0.916 0.975 0.950

Ablation Study

The ablation in Table 2 inspects variations across graph prompting modalities. All three configurations achieve close performance (within 0.4% variance), with standalone Hard Prompting answering 16 more questions correctly than the combination and 28 more than soft prompting out of 8,160 instances. Table 3 tests out-of-domain transfer on Yo'LLaVA and MyVLM benchmarks across concept-present (Positive) and concept-absent (Negative) splits.

Prompting Variant MC (Simple) MC (Diff.) Fill (Simple) Fill (Diff.) T/F (Simple) T/F (Diff.) Desc. (Simple) Desc. (Diff.)
ReGraP-LLaVA (Hard) 0.942 0.892 0.940 0.858 0.967 0.916 0.975 0.950
ReGraP-LLaVA (Soft) 0.938 0.893 0.938 0.852 0.967 0.910 0.942 0.929
ReGraP-LLaVA (Combination) 0.929 0.898 0.943 0.860 0.960 0.917 0.983 0.950
Benchmark Model Recognition Acc. (Pos.) Recognition Acc. (Neg.) Captioning Acc. (Pos.) Captioning Acc. (Neg.)
Yo'LLaVA 0.925 0.857 0.905 0.966
MyVLM 0.905 0.823 0.895 0.946
ReGraP-LLaVA 1.000 0.850 0.965 0.973

Key Findings

  • Structured graphs eliminate prompt degradation: In complex scenes with 5-6 objects, textual prompt-based models suffer significant relation confusion, whereas structured graph prompts ensure reliable entity-relation retrieval.
  • Chain-of-thought depth governs relational accuracy: Ablating CoT answer length shows that reducing reasoning steps has negligible impact on simple recognition, but causes dramatic performance drops on difficult multi-hop queries.
  • Hard prompting yields superior operational efficiency: By transforming subgraphs into discrete reasoning token sequences, hard prompting leverages Transformer self-attention natively without requiring extra GNN inference or projection matrices, achieving optimal accuracy at minimal computation.

Highlights & Insights

  • Topological DFS as reasoning supervision: Converting knowledge graph subgraphs into linear reasoning chains via DFS traversal on reasoning trees effectively solves the data bottleneck for multi-modal relational reasoning.
  • Systematic exploration of soft vs. hard graph injection: The empirical validation confirms that discrete reasoning tokens provide a cleaner, more interpretable alignment with LLM attention mechanisms than continuous GNN vector injection.
  • Grounded visual attention: Qualitative attention visual maps demonstrate that during reasoning, the model's cross-attention heads focus sharply on task-relevant physical regions (e.g., connection ports and mouse contours).

Limitations & Future Work

  • Acknowledged Limitations: The model exhibits occasional hallucinations inherited from foundation LLMs, inventing subtle unobserved details; for simple queries, it can produce overly detailed, verbose responses that diverge from human expectations.
  • Potential Enhancements: The knowledge graph remains static post-training; future work could investigate test-time dynamic graph retrieval or episodic memory updates when user environments undergo physical changes.
  • vs Yo'LLaVA (Nguyen et al., 2024): Yo'LLaVA introduces single token embeddings for isolated concepts without modeling relations; ReGraP-LLaVA constructs complete relation graphs and multi-step reasoning trees across multiple concepts.
  • vs CCoT (Mitra et al., 2024): CCoT uses zero-shot scene graphs to parse generic object layout compositions at inference; ReGraP-LLaVA personalizes models with user-specific knowledge networks via fine-tuning.

Rating

  • Novelty: โญโญโญโญโ˜† (Pioneering integration of knowledge graphs and CoT reasoning for personalized multimodal assistants)
  • Experimental Thoroughness: โญโญโญโญโญ (8,160 benchmark questions, comprehensive human/LLM evaluations, and extensive ablations)
  • Writing Quality: โญโญโญโญโญ (Clean mathematical formalization, transparent methodology, and detailed appendices)
  • Value: โญโญโญโญโ˜† (Highly practical blueprint for personalized assistants, embodied agents, and desktop AI copilots)