Skip to content

LLM-Powered Holistic Enhancement for Text-Attributed Graphs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Graph Learning
Keywords: Text-Attributed Graphs, Graph Neural Networks, Large Language Models, Node Feature Enhancement, Graph Structure Learning

TL;DR

Proposes EFS (Enhancing Features and Structures), a unified framework that enriches node features using LLM-extracted keywords, ranked category predictions, and explanatory contexts, while refining the graph topology with a lightweight edge refiner without requiring any per-edge LLM API calls, achieving state-of-the-art node classification across six benchmark datasets.

Background & Motivation

Text-attributed graphs (TAGs)—such as academic citation networks, e-commerce product co-purchasing graphs, and encyclopedia hyperlink structures—are ubiquitous across real-world domains. Nodes in TAGs carry rich unstructured textual attributes, while edges delineate topological relational dependencies. Graph Neural Networks (GNNs) have naturally emerged as the foundation for learning representations over TAGs by iteratively passing and aggregating node features along topological pathways. However, conventional GNN pipelines encounter two persistent bottlenecks: on the feature side, early paradigms depend on shallow Bag-of-Words or TF-IDF vectors that fail to capture contextual semantics and long-range syntactic patterns; on the structure side, real-world graphs inherently suffer from connection sparsity (missing true links) and spurious noisy interactions, which severely corrupt message passing.

The emergence of Large Language Models (LLMs) has opened new horizons for graph learning by bringing human-level language understanding and semantic reasoning into graph domains. Nonetheless, existing attempts predominantly suffer from fragmented optimization. Feature-focused methods like TAPE prompt LLMs to output node category predictions and textual explanations to bolster node embeddings, yet they leave the noisy or sparse graph topology completely untouched and fail to address the verbosity and domain redundancy inherent in raw text. Conversely, structure-centric methods like GraphEdit fine-tune LLMs across whole-graph traversals to predict edge dependencies, incurring prohibitive computational overhead and astronomical API expenses. Meanwhile, joint feature-structure approaches like GAugLLM rely on multi-prompt perturbations and require querying an LLM for every single candidate edge, making scaling to large-scale graphs computationally infeasible.

The central angle of this paper is to bridge this feature-structure dichotomy without suffering from runaway computation: the ranked category predictions and rich textual rationales already produced by an LLM for individual nodes contain high-order semantic affinities that can be repurposed to steer topology refinement at zero marginal API cost. Core idea: propose EFS (Enhancing Features and Structures), a holistic framework that extracts keyword summaries and ranked explanatory predictions via single-node LLM calls to construct multi-view node representations, and repurposes category prediction intersections alongside a lightweight bounded MLP refiner to adjust edge additions and deletions without any per-edge LLM queries.

Method

Overall Architecture

EFS operates as an efficient, modular, and plug-and-play framework designed to overcome both shallow semantic representations and corrupted graph topologies in TAGs. The overall execution flows through four decoupled stages: 1. Keywords and Predictions Generation: Given dataset-tailored prompt templates, an LLM is queried once per node to produce two distinct semantic enrichments—summarized keyword definitions \(s_s\) that strip away descriptive redundancy, and likelihood-ranked category predictions accompanied by textual justifications \(s_p\); 2. Multi-View Node Feature Enhancement: Three independent language model backbones (DeBERTa) and task-specific classification heads are fine-tuned on the labeled training set to map raw text \(s_o\), keyword text \(s_s\), and predictive text \(s_p\) into three separate semantic embeddings \(h_o, h_s, h_p\); 3. Lightweight Dual-Directional Edge Refiner: Using raw text representations \(h_o\), a four-layer MLP edge refiner is trained on labeled training pairs. At inference, candidate pairs are pruned via bounded neighborhood and blockwise cosine-similarity searches (slashing complexity from \(\mathcal{O}(|V|^2)\) to \(\mathcal{O}(|V|M + |E|)\)), and candidate edges are cross-verified against overlaps in the top-\(K_2\) ranked predictions to build addition matrix \(A_{add}\) and deletion matrix \(A_{del}\), producing the refined topology \(\hat{A}\); 4. Multi-Branch GNN Ensembling: Separate GNN backbones (RevGAT) ingest the three enhanced representations \(h_o, h_s, h_p\) across the refined adjacency matrix \(\hat{A}\), after which their predicted logit distributions are concatenated into an adaptive MLP fusion head to output final node classifications.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Text Attributes so<br/>and Graph Topology A"] --> B["Keywords and Predictions Generation<br/>Prompt-guided single-node LLM queries"]
    B --> C["Multi-View Feature Encoding<br/>Three DeBERTa encoders output ho, hs, hp"]
    C --> D["Lightweight Dual-Directional Edge Refiner<br/>Blockwise retrieval + TopK2 overlap reweighting"]
    C --> E["Multi-Branch GNN Ensembling<br/>Independent RevGATs with adaptive logit fusion"]
    D -->|Refined Adjacency A_hat| E
    E --> F["Node Classification Predictions Y"]

Key Designs

1. Keywords and Predictions Generation: Redundancy Stripping and Explicit Semantic Distillation Scientific articles and web documents frequently exhibit domain-specific terminology alongside verbose descriptive filler that dilutes downstream classification signals. To distill the core semantics, EFS constructs two distinct prompt templates that elicit complementary knowledge from the LLM via its standard API without fine-tuning. The first prompt directs the LLM to extract primary keywords and provide concise operational definitions, effectively filtering narrative noise. The second prompt asks the LLM to output a ranked sequence of probable target categories, demanding textual quotations and rationales for each candidate label. Both prompts are queried exactly once per node and cached offline, ensuring the total cost is bounded at \(2|V|\) one-time API calls while completely eliminating LLM latency during subsequent training and inference.

2. Multi-View Feature Encoding: Orthogonal Semantic Decoupling via Dedicated Encoders Fusing raw text, extracted keywords, and predictive explanations into a single concatenation creates gradient interference and dilutes heterogeneous linguistic granularities. Instead, EFS deploys three independent DeBERTa language models to encode the three semantic views into distinct vector spaces: $\(h_o = \text{LM}(s_o), \quad h_s = \text{LM}(s_s), \quad h_p = \text{LM}(s_p)\)$ Each encoder is topped with a single-layer MLP classifier and fine-tuned using standard cross-entropy loss against the ground-truth labels of training nodes \(V_{train}\). This decoupled fine-tuning allows \(h_o\) to retain faithful local attribute fidelity, \(h_s\) to model concise conceptual anchors, and \(h_p\) to capture high-order label-directed reasoning, establishing an orthogonal, robust multi-view feature foundation for both topological adjustment and downstream message passing.

3. Lightweight Dual-Directional Edge Refiner: Zero-API-Cost Graph Structure Refinement Traditional graph structure learning incurs an unsustainable \(\mathcal{O}(|V|^2)\) all-pair evaluation or requires calling LLMs on individual candidate edges. EFS circumvents this with a two-tier filtering and cross-verification strategy. First, a four-layer MLP edge refiner is trained on raw text embeddings \(h_o\) using only same-class (positive, \(y_e=1\)) and different-class (negative, \(y_e=0\)) node pairs from \(V_{train}\) via binary cross-entropy: $\(\mathcal{L}_{CE}(y_e, \hat{y}_e) = -[y_e \log(\hat{y}_e) + (1 - y_e)\log(1 - \hat{y}_e)]\)$ To bound inference complexity, candidate deletions \(S_{del}^i\) are strictly confined to existing neighbors, while candidate additions \(S_{add}^i\) are restricted to the top-\(M\) (\(M=50\)) non-neighbors retrieved via blockwise cosine similarity, curbing scoring cost to \(\mathcal{O}(|V|M + |E|)\). The refiner ranks candidates to isolate the top-\(K_1\) potential additions and bottom-\(K_1\) potential deletions. Crucially, EFS introduces the cached ranked predictions \(s_p\) as a semantic gate: an addition edge is materialized in \(A_{add}\) only if the top-\(K_2\) predicted category sets of the two nodes overlap (\(\text{TopK}_2(s_p(v_i)) \cap \text{TopK}_2(s_p(v_j)) \neq \emptyset\)), while an existing edge is marked for deletion in \(A_{del}\) if their top-\(K_2\) predictions share zero intersection (\(\text{TopK}_2(s_p(v_i)) \cap \text{TopK}_2(s_p(v_j)) = \emptyset\)). The refined topology is obtained symmetrically via: $\(\hat{A} = A + A_{add} - A_{del}\)$ This mechanism fuses metric-based pairwise affinity with high-level LLM categorical consensus without initiating a single additional LLM call.

4. Multi-Branch GNN Ensembling: Topology-Aware Propagation and Adaptive Logit Gating Rather than collapsing the multi-view representations into a monolithic vector that risks inter-view feature corruption during neighborhood aggregation, EFS instantiates three separate GNN branches (e.g., RevGAT), each dedicated to propagating one distinct feature view across the enhanced topology \(\hat{A}\): $\(Y_o = \text{GNN}_o(h_o, \hat{A}), \quad Y_p = \text{GNN}_p(h_p, \hat{A}), \quad Y_s = \text{GNN}_s(h_s, \hat{A})\)$ The resulting logit matrices \(Y_o, Y_p, Y_s\) are concatenated along the feature dimension and fed into a single-layer MLP to learn adaptive per-node gating weights: $\(Y = \text{MLP}(Y_o || Y_p || Y_s)\)$ This adaptive late fusion enables the model to dynamically prioritize the most reliable information modality for every node—relying on \(Y_o\) when textual attributes are clear, leaning on \(Y_s\) for technical keyword alignment, and leveraging \(Y_p\) when disambiguating difficult border cases.

Loss & Training

The EFS framework follows a modular three-phase training curriculum: 1. Language Model Fine-Tuning: Three DeBERTa encoders and their respective MLP heads are fine-tuned on \(V_{train}\) using label cross-entropy. Node embeddings are extracted and stored offline; 2. Edge Refiner Training: The 4-layer MLP is trained using binary cross-entropy on pairs constructed exclusively from \(V_{train}\). Validation labels are used solely to tune \(K_1\) and \(K_2\) (\(K_1=4, K_2=3\) across most datasets; \(K_2=1\) for 3-class Pubmed), guaranteeing zero test leakage; 3. Downstream GNN Optimization: With \(\hat{A}\) and embeddings fixed, the three RevGAT branches and the output fusion MLP are trained end-to-end using the AdamW optimizer (learning rate 0.002) under cross-entropy loss.

Key Experimental Results

Main Results

Evaluated across six standard TAG benchmarks covering citation networks (Cora, Pubmed, Citeseer, ogbn-arxiv), e-commerce co-purchasing (ogbn-products), and encyclopedia graphs (Wiki-CS) under a 6:2:2 train/val/test split. Results reflect node classification accuracy (% mean ± std over 5 random seeds).

Paradigm Model Cora Pubmed Citeseer ogbn-arxiv ogbn-products Wiki-CS
GNN GCN 86.46 ± 0.56 80.42 ± 0.42 72.56 ± 0.46 71.21 ± 0.45 70.12 ± 0.56 75.96 ± 0.24
GNN RevGAT 85.58 ± 0.86 88.26 ± 0.43 75.76 ± 0.62 70.82 ± 0.36 69.85 ± 0.56 78.64 ± 0.48
LLM GPT-3.5 67.69 ± 0.00 93.42 ± 0.00 59.29 ± 0.00 73.50 ± 0.00 73.56 ± 0.00 72.67 ± 0.00
GNN+LLM GIANT 85.75 ± 0.88 91.21 ± 0.29 73.45 ± 0.72 74.16 ± 0.12 75.14 ± 0.58 76.36 ± 0.35
GNN+LLM TAPE 89.42 ± 1.45 94.88 ± 0.16 78.58 ± 0.63 77.37 ± 0.12 80.86 ± 0.44 82.67 ± 0.29
GNN+LLM GraphEdit 88.21 ± 1.27 93.65 ± 0.33 80.84 ± 1.26 75.24 ± 0.25 78.18 ± 0.28 79.86 ± 0.78
GNN+LLM GAugLLM 78.84 ± 0.75 83.50 ± 0.84 74.12 ± 1.58 73.71 ± 0.08 71.87 ± 0.64 73.42 ± 0.41
GNN+LLM OFA 77.24 ± 0.98 79.62 ± 1.06 70.58 ± 0.81 75.24 ± 0.79 78.98 ± 0.61 78.27 ± 0.26
GNN+LLM InstructGLM 87.08 ± 0.32 93.84 ± 0.25 77.04 ± 0.12 76.42 ± 0.09 79.18 ± 0.41 80.79 ± 0.72
GNN+LLM ENGINE 89.96 ± 0.36 92.56 ± 0.17 78.34 ± 0.78 76.07 ± 0.34 77.75 ± 0.59 81.33 ± 0.87
GNN+LLM LLaGA 89.45 ± 0.25 94.95 ± 0.12 75.46 ± 0.36 75.78 ± 0.14 79.85 ± 0.23 81.85 ± 0.28
Ours EFS 91.64 ± 0.23 96.36 ± 0.08 87.49 ± 0.81 78.86 ± 0.19 82.27 ± 0.10 84.08 ± 0.09

Ablation Study

Systematic ablation on Cora and Citeseer isolating feature modalities, multi-view ensembling, edge addition, and edge deletion (mean accuracy ± std over 5 runs).

Configuration Cora Accuracy Citeseer Accuracy Architectural Scope
\(h_o + A\) 0.8708 ± 0.0105 0.7765 ± 0.0184 Raw text features only on original graph
\(h_s + A\) 0.8576 ± 0.0118 0.7761 ± 0.0133 Summarized keywords features only on original graph
\(h_p + A\) 0.8808 ± 0.0157 0.7664 ± 0.0109 Ranked predictions features only on original graph
\(h + A\) 0.9007 ± 0.0120 0.8031 ± 0.0101 Multi-view feature fusion (\(h_o, h_s, h_p\)) on original graph
\(h + A + A_{add}\) 0.9126 ± 0.0094 0.8561 ± 0.0069 Multi-view features + edge addition only
\(h + A - A_{del}\) 0.9118 ± 0.0086 0.8228 ± 0.0072 Multi-view features + edge deletion only
\(h + A + A_{add} - A_{del}\) (Full EFS) 0.9164 ± 0.0023 0.8749 ± 0.0081 Full model: multi-view features + dual topology refinement

Key Findings

  • Synergistic Feature and Structure Gains: On Citeseer, fusing multi-view features (\(h+A\)) boosts accuracy over raw features (\(h_o+A\)) by 2.66% (from 77.65% to 80.31%). Introducing edge addition further boosts it by 5.30% to 85.61%, and combining both addition and deletion achieves 87.49%, outperforming the strongest baseline GraphEdit (80.84%) by a dramatic +6.65%.
  • Massive Reduction in Training Latency: On Citeseer, EFS completes its post-LLM model-side training in just 20.20 minutes (14.52 min for DeBERTa fine-tuning, 1.28 min for edge refiner training, 3.2 s for candidate search and scoring, and 4.31 min for RevGAT branches). In contrast, GraphEdit requires 1205 minutes and InstructGLM requires 2706 minutes, rendering EFS 60× to 134× faster while achieving significantly higher accuracy.
  • Robustness Against LLM Hallucinations: On the specific subset of Citeseer nodes where the LLM's top-1 zero-shot prediction is erroneous, full EFS recovers a remarkable 75.74% accuracy (and 78.77% on Cora). This confirms that downstream GNN message passing over the refined topology actively corrects LLM mistakes rather than mechanically memorizing generative prompts.

Highlights & Insights

  • Dual Repurposing of Single-Node LLM Queries: Instead of querying an LLM repeatedly across graph edges, EFS shows that a single node-level prediction query can simultaneously act as a rich semantic attribute for feature encoders and as a sharp categorical consistency check for topological reweighting.
  • Extreme Candidate Pruning via Blockwise Search: By restricting edge addition candidates to the top-50 cosine-similar non-neighbors via blockwise search, candidate pairs are slashed by 1329× on ogbn-arxiv (from billions to 10.78M) and 30× on Citeseer, reducing edge scoring to mere seconds.
  • Late Logit Fusion Preserves Modality Purity: Running dedicated GNN instances over individual semantic views followed by adaptive logit fusion outperforms early feature concatenation by preventing high-entropy textual noise from drowning out structural embeddings.

Limitations & Future Work

  • Homophily Assumption in Edge Refinement: The edge refiner is trained on same-class vs. different-class node pairs, which relies heavily on the homophily assumption (connected nodes share labels). Adapting EFS to heterophilous graphs (where connections naturally span disparate classes) requires rethinking the pairing criteria.
  • Compounding of Upstream LLM Errors: While downstream GNN message passing corrects over 75% of LLM errors, systematically biased or hallucinated keywords from smaller LLMs could still permeate into the DeBERTa representations.
  • Multi-Stage Training Complexity: The current pipeline requires three discrete training phases (LM fine-tuning, edge refiner fitting, and GNN ensembling), leaving open the opportunity for an end-to-end differentiable framework.
  • vs TAPE (ICLR 2024): TAPE leverages LLM predictions and rationales strictly for node feature enhancement. EFS goes beyond TAPE by extracting concise keyword definitions to strip narrative redundancy, and repurposing the category rankings to drive zero-cost graph structure refinement, outperforming TAPE by +8.91% on Citeseer.
  • vs GraphEdit (arXiv 2024): GraphEdit relies on full-graph LLM fine-tuning to rewire edges, taking over 20 hours to train on small graphs. EFS achieves a 6.65% accuracy lead on Citeseer while completing model-side training in just 20.20 minutes.
  • vs GAugLLM (KDD 2024): GAugLLM executes self-supervised contrastive learning and invokes an LLM to evaluate candidate edges directly. EFS operates in a supervised setting with bounded search, avoiding all per-edge LLM calls and delivering superior performance.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant dual-repurposing of node-level ranked predictions for zero-call edge refinement, breaking the speed-accuracy trade-off.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across 6 diverse datasets, 11 strong baselines, full parameter sensitivity (\(K_1, K_2\)), error correction audits, and runtime complexity breakdowns.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, well-structured narrative, and thorough ablation analyses.
  • Value: ⭐⭐⭐⭐⭐ A highly practical, reproducible paradigm for deploying LLMs on large-scale text-attributed graphs in resource-constrained settings.