Skip to content

One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Schemas

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Jo-eyang/OMG-VLM
Area: LLM Reasoning
Keywords: Attributed Graph Learning, Vision-Language Models, Heterogeneous Modality Schemas, Graph Adapters, Cross-Graph Generalization

TL;DR

OMG-VLM presents a unified vision-language model (VLM) framework for attributed graph learning across heterogeneous modality schemas, utilizing a target-aware textual aggregation adapter and a graph-aware visual compression adapter to seamlessly process text-attributed graphs (TAGs), image-attributed graphs (IAGs), and multimodal-attributed graphs (MAGs) within a shared native token space for end-to-end node classification and link prediction.

Background & Motivation

Attributed graph learning is a foundational paradigm in machine learning and data mining, underpinning key applications across academic citation networks, e-commerce recommendation ecosystems, social media interaction graphs, and multimodal knowledge graphs. In real-world deployments, attributed graphs exhibit pronounced modality heterogeneity: certain graphs possess exclusively textual node features (such as titles and abstracts in citation networks, forming TAGs), others provide purely visual features (such as product photos or user avatars in social graphs, forming IAGs), while many real-world interaction graphs integrate both textual and visual modalities (forming MAGs). Such heterogeneous modality schemas arise naturally from domain-specific data collection constraints, creating an urgent practical need for general-purpose models that can represent and reason across disparate attribute configurations within a single architecture.

Despite this need, existing graph learning methodologies are predominantly anchored to rigid single-modality assumptions. Classical graph neural networks, including GCN, GraphSAGE, and RevGAT, operate over fixed numerical feature vectors and require isolated modality-specific encoders and pre-processing pipelines, precluding cross-graph model reusability. Emerging LLM-based graph learning frameworks such as LLaGA and GraphPrompter linearize neighborhood text into language prompts to enable generative reasoning, yet they remain strictly text-centric; incorporating visual attributes requires auxiliary vision encoders, rendering them incapable of handling text-free image-attributed graphs. Furthermore, existing multimodal graph learners like UniGraph2, Graph4MM, and MLaGA either limit unification to embedding-level contrastive alignment paired with detached task heads, or strictly enforce fully multimodal inputs, breaking down whenever one modality is absent. Meanwhile, directly serializing multi-hop visual-textual neighborhoods into VLM prompts, as in GraphVLM, suffers from severe input token explosion and excessive computational overhead.

The core tension stems from the mismatch between variable-sized, structure-dependent graph neighborhoods and the rigid sequential token interfaces of foundation models, alongside the challenge of injecting topological context without degrading pretrained multimodal alignment. To dissolve the boundaries separating specialized single-modality models, this work explores the potential of pretrained vision-language models whose native shared embedding space bridges text and image representations. Core idea: formulate multi-schema graph reasoning as conditional autoregressive generation over a pretrained VLM backbone, introducing target-aware textual retrieval probes and per-neighbor visual compression adapters to project heterogeneous topological context directly into native VLM tokens for end-to-end reasoning across TAGs, IAGs, and MAGs.

Method

Overall Architecture

OMG-VLM builds upon a pretrained vision-language model (primarily Qwen-VL-Chat 7B) to cast node classification and link prediction across heterogeneous attributed graphs into unified autoregressive sequence generation. Given a target node \(v\) associated with an available attribute subset \(\mathcal{A}_v \subseteq \{T_v, I_v\}\), modality-specific topological neighborhoods \(\mathcal{N}_T(v)\) and \(\mathcal{N}_I(v)\), and a natural language task prompt \(P\), the framework constructs a compact, unified input token sequence \(X_v\): $\(X_v = [\Phi_T(T_v, \mathcal{N}_T(v)) \parallel \Phi_I(I_v, \mathcal{N}_I(v))] \parallel \text{Embed}(P)\)$ Whenever a modality is missing in a given graph schema, its corresponding adapter representation is naturally omitted. The VLM decoder autoregressively generates the target prediction sequence \(Y\) by maximizing the likelihood \(P(Y \mid X_v; \Theta) = \prod_{t=1}^{|Y|} P_{\text{VLM}}(y_t \mid y_{<t}, X_v; \Theta)\). By eliminating external modality alignment modules and operating strictly within the VLM's native visual encoder and token embedding space, OMG-VLM achieves unified reasoning across heterogeneous schemas without cross-space distortion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Attributed Graph Query & Instruction<br/>TAG / IAG / MAG Modality Combinations"] --> B["Target-Aware Textual Aggregation<br/>Learnable Query RoPE Probes + Multi-Head Cross-Attention Retrieval"]
    A --> C["Graph-Aware Visual Representation Learning<br/>Per-Neighbor Token Compression + In-Place Center-Conditioned Aggregation"]
    B --> D["Native Token Concatenation & Instruction Alignment<br/>Compact Unified Sequence X_v Assembly"]
    C --> D
    D --> E["End-to-End Cross-Schema Co-Optimization<br/>LoRA Parameter Adaptation + Joint Multi-Task Training"]
    E --> F["Unified Generative Decoding Output<br/>Node Classification Label / Link Prediction Decision"]

Key Designs

1. Target-Aware Textual Aggregation: Mitigating Textual Neighborhood Semantic Redundancy and Alignment Distortion In text-attributed graphs, the relevance of neighboring nodes to the target entity varies drastically; naive text concatenation balloons input length and introduces distracting noise, while external language encoders demand cumbersome cross-modal projection. To address this, OMG-VLM develops the target-aware textual adapter \(\Phi_T\), operating directly over the VLM's native embedding matrix. Target text \(T_v\) is mapped to token embeddings \(E_{T_v} \in \mathbb{R}^{L_{T_v} \times d}\), while neighboring textual attributes are concatenated into a neighborhood token repository \(B_{T_v} \in \mathbb{R}^{L_T^N \times d}\). Because contemporary generative VLM tokenizers lack an explicit \([CLS]\) pooling token, the module introduces \(M_T\) learnable query tokens \(Q_T \in \mathbb{R}^{M_T \times d}\) as adaptive probes. These probes are concatenated with target embeddings and processed through a Rotary Position Embedding (RoPE) self-attention layer to extract target-conditioned query vectors: \(\hat{Q}_{T_v} = \text{Attn}_{\text{RoPE}}(E_{T_v} \parallel Q_T)_{[-M_T:, :]}\). Subsequently, \(\hat{Q}_{T_v}\) queries the neighbor token bank \(B_{T_v}\) through multi-head cross-attention (MHCA) to retrieve salient topological evidence: \(Z_{T_v} = \text{MHCA}(\hat{Q}_{T_v}, B_{T_v}, B_{T_v}) \in \mathbb{R}^{M_T \times d}\). Concatenated with a text bridge embedding \(E_{T_{\text{sep}}}\) and target tokens as \(X_{T_v} = E_{T_v} \parallel E_{T_{\text{sep}}} \parallel Z_{T_v}\), this design bounds textual graph context to a fixed token budget (default \(M_T=8\)), preserving fine-grained topological semantics without cross-space alignment error.

2. Graph-Aware Visual Representation Learning: Resolving Patch Token Explosion and Computational Bottlenecks Visual neighborhoods pose an acute efficiency challenge: when each neighboring image is unfolded into patch tokens, incorporating \(K\) neighbors expands context by \(K \times L_I\), which dilutes target visual focus and incurs quadratic self-attention cost. To resolve this, the graph-aware visual adapter \(\Phi_I\) introduces a two-stage paradigm of "per-neighbor visual compression" followed by "center-conditioned in-place aggregation". First, the frozen ViT encoder of the VLM extracts patch tokens \(E_{I_u} \in \mathbb{R}^{L_I \times d}\) for the target and each neighbor \(u \in \mathcal{N}_I(v)\). Rather than forwarding uncompressed patch sequences, \(M_I\) learnable query vectors \(Q_I \in \mathbb{R}^{M_I \times d}\) compress each neighbor independently via cross-attention: \(C_{I_u} = \text{MHCA}(Q_I, E_{I_u}, E_{I_u})\), with \(M_I \ll L_I\) (default \(M_I=32\)), reducing neighborhood visual overhead by approximately 75% while forming a compact visual bank \(B_{I_v} \in \mathbb{R}^{|\mathcal{N}_I(v)| M_I \times d}\). Next, target visual tokens \(E_{I_v}\) serve as queries across \(L\) layers (\(L=1\) provides optimal trade-offs) of cross-attention and feed-forward updates: $\(H_{I, (\ell+1)_v} = \text{FFN}(\text{MHCA}(H_{I, (\ell)_v}, B_{I_v}, B_{I_v})) + H_{I, (\ell)_v}\)$ Crucially, the refined visual representation \(X_{I_v} = H_{I, (L)_v}\) retains the exact token length \(L_I\) of a single image, enabling direct in-place substitution into the VLM sequence without adding downstream sequence length.

3. End-to-End Cross-Schema Co-Optimization: Overcoming Staged Optimization Suboptimality and Schema Disconnection To enable a single model to flexibly handle TAGs, IAGs, and MAGs, OMG-VLM bypasses isolated two-stage pretraining pipelines and dataset-specific parameter sets in favor of unified end-to-end co-optimization. The underlying ViT visual encoder and tokenizer remain completely frozen, while low-rank adaptation (LoRA, rank \(r=64\), scaling \(\alpha=16\)) is applied to the VLM transformer layers, co-trained alongside the trainable graph adapters \(\Phi_T\) and \(\Phi_I\). The training mixture comprises four diverse graphs spanning e-commerce, citation networks, and social media across multiple tasks (balanced via downsampling to prevent dominance by large graphs). The joint training objective updates all parameters simultaneously: $\(\mathcal{L}(\Theta) = \mathbb{E}_{z \sim \mathcal{D}} \left[ \sum_{t=1}^{|Y|} \log P_{\text{VLM}}(y_t \mid y_{<t}, X_v; \Theta) \right]\)$ This joint optimization ensures neighborhood retrieval, compression, and generative decoding co-adapt dynamically. Relative to two-stage training, co-optimization accelerates training time by 17.54% (dropping from 81.05 to 66.83 hours) while markedly boosting out-of-distribution transfer to unseen graph topologies, unencountered label spaces, and schema transitions.

Loss & Training

The framework optimizes the standard causal autoregressive cross-entropy loss across all generated target tokens. Optimization is performed using AdamW with an initial learning rate of \(1 \times 10^{-5}\), weight decay of 0.1, \(\beta_2 = 0.95\), a linear warmup ratio of 0.01, and a cosine learning rate scheduler. LoRA is integrated across all self-attention projection and feed-forward modules. On a single NVIDIA H100 80GB GPU, training converges within 3 epochs over balanced multi-graph batches.

Key Experimental Results

Main Results

The primary evaluation investigates node classification (NC) and link prediction (LP) across four in-domain datasets and five zero-shot transfer benchmarks. The benchmark spans MAG Amazon-Arts (NC), TAG ogbn-arXiv (NC), MAG Amazon-Movies (LP), and IAG RedditS (LP) for in-domain training, alongside unseen transfer benchmarks TAG PubMed (NC), MAG Amazon-CDs (NC), MAG RedditM (LP), MAG Amazon-VideoGames (LP), and TAG Cora (LP). All figures report Accuracy (%).

Category Model Arts(NC) arXiv(NC) Movies(LP) RedditS(LP) In-Domain Avg PubMed(NC) CDs(NC) RedditM(LP) VideoGames(LP) Cora(LP) Transfer Avg Overall Avg
Classical MLP 73.75 64.59 59.42 56.25 63.50 37.96 1.42 55.20 50.12 50.90 39.12 49.96
GNNs GCN 63.95 58.71 52.38 49.77 56.20 20.79 10.79 51.05 50.61 50.00 36.65 45.34
GraphSAGE 74.77 62.78 52.22 49.87 59.91 42.77 3.67 50.80 52.99 50.30 40.11 48.91
RevGAT 71.89 62.15 49.54 49.77 58.34 40.26 3.54 50.25 50.08 50.00 38.83 47.50
UniGraph2 78.87 41.26 91.51 78.85 72.62 38.40 10.83 62.35 54.41 51.10 43.42 56.40
LLM-based LLaGA 88.74 63.70 53.27 โ€“ 68.57 1.65 4.14 51.60 57.98 0.70 23.21 40.22
GraphPrompter 83.21 56.28 63.82 โ€“ 67.77 56.92 30.71 44.65 70.30 50.00 50.52 56.99
Multimodal LLM MLaGA 89.63 โ€“ 76.20 90.10 85.31 โ€“ 31.10 64.30 58.46 โ€“ 51.29 68.30
Graph4MM 84.05 52.75 88.45 โ€“ 75.08 25.95 10.25 72.30 59.40 14.90 36.56 51.01
VLM-based GraphVLM 82.55 63.99 50.74 55.70 63.25 72.13 29.47 57.35 48.73 50.20 51.58 56.76
Ours OMG-VLM 92.50 67.85 95.30 99.10 88.69 78.55 32.56 92.45 77.99 51.80 66.67 76.46

Note: Dash (โ€“) denotes that a model cannot process the dataset due to inherent architectural modality constraints.

Ablation Study

Extensive ablations evaluate the structural necessity of the proposed adapters, image compression efficiency, and optimization strategies.

Adapter Replacement Ablation (Table 8): Replacing adapters with Mean Pooling or Center-Query Attention Pooling degrades accuracy across schemas:

Dataset Full Model Text Adapter Pooled (Mean / Attn) Visual Adapter Pooled (Mean / Attn) Both Adapters Pooled (Mean / Attn)
Arts (MAG, NC) 92.50 92.43 (-0.07) / 92.20 (-0.30) 92.30 (-0.20) / 91.25 (-1.25) 92.20 (-0.30) / 91.67 (-0.83)
arXiv (TAG, NC) 67.85 67.38 (-0.47) / 67.09 (-0.76) 67.84 (-0.01) / 67.35 (-0.50) 67.12 (-0.73) / 67.09 (-0.76)
Movies (MAG, LP) 95.30 92.93 (-2.37) / 92.30 (-3.00) 93.56 (-1.74) / 85.33 (-9.97) 90.75 (-4.55) / 73.81 (-21.49)
RedditS (IAG, LP) 99.10 97.80 (-1.30) / 97.80 (-1.30) 76.25 (-22.85) / 74.95 (-24.15) 76.30 (-22.80) / 74.70 (-24.40)
PubMed (TAG, NC) 78.55 59.25 (-19.30) / 60.41 (-18.14) 78.27 (-0.28) / 78.32 (-0.23) 57.13 (-21.42) / 56.25 (-23.30)
RedditM (MAG, LP) 92.45 87.15 (-5.30) / 86.85 (-5.60) 79.85 (-12.60) / 82.30 (-10.15) 80.70 (-11.75) / 76.45 (-16.00)
CDs (MAG, NC) 32.56 26.34 (-6.22) / 31.53 (-1.03) 30.89 (-1.67) / 27.21 (-5.35) 26.34 (-6.22) / 30.76 (-1.80)

Optimization Strategy and Runtime Efficiency (Table 7 & Table 10): - Co-Optimization vs. Two-Stage: End-to-end co-optimization outperforms two-stage training across 7 of 9 benchmarks (e.g., RedditM improves from 85.70 to 92.45; Movies improves from 93.97 to 95.30), while reducing overall training time from 81.05 to 66.83 hours (-17.54%). - Computational Cost vs. GraphVLM: Compared to prompt-linearized GraphVLM on Amazon-Arts, OMG-VLM compresses average input prompt length from 1444.07 to 464.30 tokens, decreases training latency from 1428.30 to 925.42 ms per sample, and reduces inference latency from 397.85 to 358.17 ms per sample. - Visual Neighbor Compression: Compressing neighbor visual features to \(M_I=32\) tokens achieves roughly 75% FLOPs reduction (slashing computational overhead from 10.74Bโ€“107.37B to 2.68Bโ€“26.84B across 10 to 100 neighbors) while maintaining performance within 1โ€“2 percentage points of uncompressed patch modeling.

Key Findings

  • Superiority of Unified Multi-Schema Representation: Specialized baselines fail drastically when confronting schema shifts (e.g., text-only LLaGA cannot execute on IAG RedditS and collapses to 1.65% and 4.14% on unseen transfer benchmarks PubMed and CDs). OMG-VLM achieves a leading overall average of 76.46%, outperforming the strongest multimodal baseline MLaGA (68.30%) by +8.16 points.
  • Criticality of Target-Conditioned Adapters: Replacing structure-aware adapters with naive pooling degrades performance by up to 24.40 points on image-attributed graphs (RedditS) and up to 23.30 points on text-attributed graphs (PubMed), confirming that target-directed retrieval is vital for filtering neighborhood noise.
  • Stability Across Hyperparameter Budgets: Sensitivity analyses reveal that performance remains exceptionally stable across aggregated token lengths (\(M_T=8, M_I=32\)) and neighborhood sampling sizes (\(K=10\)), demonstrating that a compact neighborhood set fully captures decisive topological signals without requiring bloated receptive fields.

Highlights & Insights

  • Native Embedding Space Exploitation: By directly tapping into the VLM's inherent tokenizer and ViT token representations, the framework avoids external neural modality encoders and the corresponding cross-space alignment loss, preserving clean multimodal semantics.
  • In-Place Visual Refinement Trick: The visual adapter refines target image tokens in place rather than appending neighbor tokens to the sequence, keeping the visual context budget invariant to graph density and eliminating downstream attention explosion.
  • Conceptual Shift in Graph Learning: Rather than viewing TAGs, IAGs, and MAGs as segregated research silos requiring custom designs, OMG-VLM unifies graph reasoning into autoregressive sequence generation with missing-modality tolerance.

Limitations & Future Work

  • Supervised Static Task Formulation: The current architecture is optimized for static node classification and link prediction under supervised settings, without extending to dynamic evolving graphs or temporal interaction edges.
  • Absence of Interactive Agentic Reasoning: The model executes fixed-budget neighborhood aggregation in a single forward pass, lacking iterative exploration mechanisms such as agentic tool calling or multi-hop path inspection.
  • Future Directions: Extending OMG-VLM toward agentic graph reasoning with dynamic subgraph expansion, and scaling the backbone to support non-visual physical modalities such as 3D point clouds and audio streams.
  • vs UniGraph2: UniGraph2 targets representation-level graph pretraining by producing continuous embeddings for separate downstream linear classifiers. In contrast, OMG-VLM performs end-to-end generative reasoning directly outputting text tokens, yielding far superior zero-shot cross-graph generalization.
  • vs LLaGA / GraphPrompter: While LLaGA effectively maps graph structure into LLM prompt space, it is fundamentally text-bound and incapable of handling raw visual graphs. OMG-VLM elevates the foundation backbone to a VLM, natively assimilating image-attributed and multimodal graphs.
  • vs GraphVLM: GraphVLM incorporates multimodal graph context by directly flattening neighbor attributes into textual prompts, which triggers extreme prompt bloat and quadratic attention costs. OMG-VLM resolves this via target-conditioned and per-neighbor compressed adapters, dramatically accelerating training and inference.

Rating

  • Novelty: โญโญโญโญโญ Pioneering framework exploring pretrained VLMs as unified, general-purpose backbones across heterogeneous graph modality schemas.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 9 benchmarks, four task paradigms, transfer settings, computational profiling, and multiple VLM backbones (Qwen-VL, LLaVA-1.6, Qwen3-VL).
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, elegant problem formulation, rigorous mathematical specification, and transparent ablation reporting.
  • Value: โญโญโญโญโญ Provides a foundational blueprint for unifying graph representation learning and multimodal generative foundation models.