Skip to content

HiTeC: Hierarchical Contrastive Learning on Text-Attributed Hypergraph with Semantic-Aware Augmentation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Matilda0224/HiTeC
Area: LLM Reasoning
Keywords: Text-Attributed Hypergraph, Self-Supervised Learning, Contrastive Learning, Semantic-Aware Augmentation, s-walk Subgraph Sampling

TL;DR

Addressing the issues of semantic-topological disconnection, perturbation noise from random augmentations, and poor long-range structural modeling in text-attributed hypergraphs (TAHGs), HiTeC introduces a decoupled two-stage contrastive learning framework. By combining structure-aware text encoder pretraining, semantic-aware dual augmentations, and s-walk-induced multi-scale hierarchical contrastive objectives, HiTeC significantly surpasses strong baselines across six benchmark datasets.

Background & Motivation

Complex systems in the real world frequently exhibit multi-way group interactions that transcend pairwise graph edges, such as cortical co-activation in neural circuits, collective user-item shopping sessions in e-commerce, and multi-author collaborations in academic publication networks. Hypergraphs serve as a natural mathematical abstraction for modeling such higher-order relationships because each hyperedge can connect an arbitrary number of nodes. In these domains, node entities are almost invariably accompanied by rich textual information, such as product reviews and item descriptions in recommendation networks or titles and abstracts in citation networks. While Hypergraph Neural Networks (HNNs) have emerged as an effective paradigm for capturing high-order structural dependencies, they predominantly rely on numerical node features and task-specific supervised labels, leaving self-supervised learning on text-attributed hypergraphs (TAHGs) severely underexplored.

Directly porting conventional hypergraph contrastive learning (HCL) methods or text-attributed graph (TAG) models to TAHGs reveals three critical barriers. First, existing HCL approaches extract node features using graph-agnostic shallow encoders or off-the-shelf language models, creating a severe representational rift between high-order topological structures and textual semantics. Second, current hypergraph contrastive pipelines rely extensively on random data augmentations, such as random masking of text attributes or uniform dropping of hyperedges, which indiscriminately corrupt semantic integrity and inject detrimental noise into the contrastive objective. Third, existing hypergraph contrastive signals operate almost exclusively at the localized node and hyperedge levels or 1-hop incidence relations, failing to capture long-range topological dependencies and richer structural contexts essential for expressive global representations.

The core tension lies in injecting higher-order topological priors into textual embeddings without entangling joint training complexity, while constructing semantically consistent augmentations and multi-scale contrastive objectives capable of capturing long-range contextual scopes. To resolve these challenges, this paper proposes HiTeC, a two-stage decoupled hierarchical contrastive learning framework. HiTeC first aligns textual semantics with higher-order neighborhood topology via a structure-aware contrastive objective, and then freezes the text encoder to pretrain the hypergraph encoder using structure-contextualized text augmentation, semantic cohesiveness-guided hyperedge dropping, and an s-walk-based subgraph contrastive objective. The core idea is to decouple TAHG representation learning into structure-aware text prior injection and semantic-aware hierarchical hypergraph contrastive pretraining, leveraging semantic-preserving augmentations and s-walk-sampled subgraphs to simultaneously alleviate perturbation noise and capture long-range higher-order dependencies.

Method

Overall Architecture

HiTeC implements an efficient two-stage decoupled self-supervised learning pipeline comprising four sequential mechanisms: structure-aware text encoder pretraining, semantic-aware multimodal view augmentation, hypergraph message passing, and multi-scale hierarchical contrastive learning. In Stage 1, a text encoder with a projection head is pretrained using a triplet margin objective defined over 1-hop hyperedge structural neighbors, ensuring that textual embeddings naturally reflect local higher-order clustering. In Stage 2, with the text encoder frozen, HiTeC generates two augmented views by combining structure-contextualized text views (incorporating domain knowledge, global topology, and neighbor text) with semantic cohesiveness-guided hyperedge dropping. These views are encoded through a Hypergraph Neural Network (HGNN), and representations are jointly optimized across node, hyperedge, and s-walk-sampled subgraph scales using InfoNCE objectives.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input TAHG H = (V, E, T)"] --> B["Structure-Aware Text Encoder Pretraining<br/>1-hop neighbors as positives & complement as negatives"]
    B --> C["Semantic-Aware Data Augmentation<br/>Contextualized text views + cohesiveness-guided edge drop"]
    C --> D["Hypergraph Neural Network Encoding<br/>Node-to-hyperedge and hyperedge-to-node message passing"]
    D --> E["Hierarchical Multi-Scale Contrastive Objectives<br/>Node-level + Hyperedge-level + s-walk Subgraph InfoNCE"]
    E --> F["Transferable Representations (Z_V, Z_E)<br/>Downstream node classification & hyperedge prediction"]

Key Designs

1. Structure-Aware Text Encoder Pretraining: Injecting Higher-Order Topological Priors into Language Models

Conventional approaches use graph-agnostic text encoders that map textual attributes in isolation, yielding representations poorly correlated with hypergraph topology. HiTeC resolves this by explicitly guiding the text encoder \(f_\theta\) (parameterized by a pretrained language model backbone with a 2-layer MLP projection head) to align with hypergraph connectivity. For any node \(v_i\) with text \(t_i\), the representation is \(x_i = f_\theta(t_i) \in \mathbb{R}^d\). Rather than sampling arbitrary contrastive pairs, HiTeC treats all 1-hop structural neighbors \(\mathcal{N}^{(1)}_{v_i}\)—nodes sharing at least one hyperedge with \(v_i\)—as positives, and the remaining disconnected nodes \(V \setminus \mathcal{N}^{(1)}_{v_i}\) as negatives. Positive and negative representations are formed via mean-pooling over their corresponding text embeddings:

\[x_i^+ = \frac{1}{|\mathcal{N}^{(1)}_{v_i}|} \sum_{v_j \in \mathcal{N}^{(1)}_{v_i}} x_j, \quad x_i^- = \frac{1}{|V \setminus \mathcal{N}^{(1)}_{v_i}|} \sum_{v_j \in V \setminus \mathcal{N}^{(1)}_{v_i}} x_j\]

The text encoder is optimized using a triplet margin loss that pushes anchor \(x_i\) closer to its pooled 1-hop structural neighborhood than to the disconnected node set:

\[\ell_t(x_i, x_i^+, x_i^-) = \max\left(\cos(x_i, x_i^-) - \cos(x_i, x_i^+) + m, 0\right)\]

where \(\cos(\cdot)\) denotes cosine similarity and \(m\) is the margin hyperparameter. This objective aligns textual semantic geometry with hypergraph clustering tendencies before full hypergraph neural training begins.

2. Semantic-Aware Data Augmentation: Dual-Level Perturbation Suppression for Structure and Text

Random data augmentations, such as token masking or uniform edge dropping, distort semantic meaning and disrupt fragile higher-order graph-text alignments. HiTeC addresses this by introducing structure-contextualized text augmentation and semantic cohesiveness-guided hyperedge dropping. At the text level, one view is constructed through a deterministic template that enriches raw text \(t_i\) with three contextual anchors: \(\tilde{t}_{1,i} = [[\text{DK}] \parallel [\text{GT}] \parallel [\text{NC}] \parallel t_i]\), where \([\text{DK}]\) encodes shared domain knowledge, \([\text{GT}]\) injects node degree and local density, and \([\text{NC}]\) concatenates text snippets from sampled neighbors (capped at 10 to ensure sequence stability). The paired view retains original text \(\tilde{t}_{2,i} = t_i\). Encoding both with frozen \(f_\theta\) produces views \(\tilde{X}_1\) and \(\tilde{X}_2\) that exhibit structural variance while maintaining semantic equivalence.

At the structural level, HiTeC defines the semantic cohesiveness score \(s(e)\) of hyperedge \(e\) as the average pairwise cosine similarity among member node embeddings:

\[s(e) = \frac{1}{\binom{|e|}{2}} \sum_{i=1}^{|e|} \sum_{j=i+1}^{|e|} S_e(i, j), \quad S_e = X_e X_e^\top\]

Using this cohesiveness metric, the drop probability of node-hyperedge incidence \((v_i, e_j)\) is computed adaptively:

\[p_{\text{drop}}(v_i, e_j) = 1 - \sigma\left(\frac{s(e_j) - 0.5}{\tau_{\text{drop}}}\right)\]

where \(\sigma(\cdot)\) is the Sigmoid activation and \(\tau_{\text{drop}}\) is a temperature scale. Highly cohesive hyperedges strongly preserve their connections, while noisy or weakly aligned incidences are selectively pruned. This produces robust augmented hypergraphs \(\tilde{\mathcal{H}}_1\) and \(\tilde{\mathcal{H}}_2\) free from arbitrary topological disruption.

3. s-Walk-Induced Multi-Scale Hierarchical Contrastive Learning: Bridging Microscopic Entities to Macroscopic Subgraphs

Prior HCL methods focus almost exclusively on node- and hyperedge-level contrastive signals, which limits their receptive field to immediate topological neighborhoods. HiTeC establishes a comprehensive hierarchical objective covering node, hyperedge, and subgraph granularities. The central novelty is the s-walk-based subgraph sampling mechanism. An s-walk of length \(l\) defines an ordered sequence of hyperedges \([e_1, e_2, \dots, e_l]\) where every consecutive pair shares at least \(s\) common nodes, i.e., \(|e_j \cap e_{j+1}| \ge s\). Setting \(s \ge 2\) enforces strong high-order overlap and contextual continuity along the traversal path, preventing random walk wandering.

To maintain scalability and avoid dense subgraph construction across the entire hypergraph, HiTeC samples subgraphs exclusively for a representative subset of structurally central nodes \(V_r \subseteq V\), selected by ranking node degrees and retaining the top \(r\%\). For each sampled subgraph \(\mathcal{H}_c^s\), representations are obtained via mean pooling over node embeddings produced by the hypergraph encoder \(g_\phi\): \(z_{s_c} = P_{\text{mean}}(g_\phi(\mathcal{H}_c^s))\). Symmetric InfoNCE losses across node pairs \(\mathcal{L}_n\), hyperedge pairs \(\mathcal{L}_e\), and s-walk subgraph pairs \(\mathcal{L}_s\) are simultaneously optimized, capturing both local entity interactions and long-range higher-order topological dependencies.

Loss & Training

The overall self-supervised contrastive learning objective in Stage 2 is:

\[\mathcal{L} = \mathcal{L}_n + \lambda_e \mathcal{L}_e + \lambda_s \mathcal{L}_s\]

where \(\mathcal{L}_n\) and \(\mathcal{L}_e\) represent symmetric InfoNCE objectives over all nodes and hyperedges, respectively, and \(\mathcal{L}_s\) is the subgraph-level contrastive loss:

\[\ell_s(z_{1,s_m}, z_{2,s_m}) = -\log \frac{\exp(\cos(z_{1,s_m}, z_{2,s_m}) / \tau_s)}{\sum_{k=1}^{|V_r|} \exp(\cos(z_{1,s_m}, z_{2,s_k}) / \tau_s)}\]
\[\mathcal{L}_s = \frac{1}{2|V_r|} \sum_{m=1}^{|V_r|} \left[\ell_s(z_{1,s_m}, z_{2,s_m}) + \ell_s(z_{2,s_m}, z_{1,s_m})\right]\]

Hyperparameters \(\lambda_e, \lambda_s \in \{1, 2, 3, 4\}\). Stage 1 pretrains a BERT-Base model with LoRA (\(r=16, \alpha=32\)) on an NVIDIA A800 80GB GPU for 3–5 epochs at learning rate 2e-5. Stage 2 employs a single-layer HGNN encoder. Across the six benchmarks, default parameters are set to walk length \(l = 4\), overlap threshold \(s = 3\), and subgraph sampling ratio \(r = 30\%\), delivering up to \(2.2\times\) training acceleration with negligible loss in accuracy.

Key Experimental Results

Main Results

Evaluation follows the standard linear evaluation protocol: node embeddings are frozen after self-supervised pretraining, with downstream classifiers trained on top. For node classification, an \(\ell_2\)-regularized logistic regression classifier is trained on a 10%/10%/80% train/val/test split. For hyperedge prediction, Clique Negative Sampling (CNS) generates balanced negative hyperedges, evaluated via a two-layer MLP on a 60%/20%/20% split. Performance is reported as average test accuracy and standard deviation across 20 random splits with 5 random initializations each (% mean ± std).

Table 1: Node classification performance comparison (% mean ± std)

Category / Method Citeseer Cora History Photo Computers Fitness
Graph-based SSL
GraphCL + SE 59.63 ± 1.58 56.69 ± 1.99 62.86 ± 0.22 42.61 ± 0.21 34.45 ± 0.38 52.11 ± 0.43
GraphCL + GIANT 57.58 ± 1.90 55.64 ± 1.50 62.08 ± 0.24 42.74 ± 0.22 34.65 ± 0.23 52.04 ± 0.34
BGRL + SE 60.81 ± 1.45 59.72 ± 1.32 65.74 ± 0.30 44.60 ± 0.30 39.40 ± 0.46 55.66 ± 0.33
BGRL + GIANT 58.68 ± 1.92 57.44 ± 1.39 63.99 ± 0.35 43.65 ± 0.25 37.00 ± 0.37 55.05 ± 0.25
GraphMAE + SE 46.42 ± 2.75 50.33 ± 0.47 52.90 ± 9.01 41.52 ± 0.08 25.61 ± 0.11 40.65 ± 0.37
GraphMAE + GIANT 50.11 ± 1.74 53.59 ± 1.07 56.20 ± 0.12 41.54 ± 0.06 25.57 ± 0.12 42.04 ± 0.04
Hypergraph SSL
TriCL + SE 60.78 ± 1.00 58.79 ± 1.30 69.28 ± 0.32 47.11 ± 0.44 43.43 ± 0.66 45.46 ± 0.21
TriCL + BERT 64.50 ± 1.08 62.67 ± 1.13 75.01 ± 0.22 51.25 ± 0.62 46.60 ± 0.90 48.97 ± 0.63
TriCL + RoBERTa 64.48 ± 0.95 60.74 ± 1.01 70.62 ± 0.81 48.42 ± 0.67 43.82 ± 1.11 47.49 ± 0.60
SE-HSSL + SE 57.04 ± 3.31 59.87 ± 1.83 68.59 ± 0.74 43.75 ± 2.01 36.53 ± 2.32 54.67 ± 2.00
SE-HSSL + BERT 65.83 ± 1.21 64.30 ± 0.89 74.88 ± 0.42 51.80 ± 0.63 41.63 ± 2.21 56.92 ± 1.44
SE-HSSL + RoBERTa 65.04 ± 1.04 62.76 ± 1.15 72.13 ± 1.07 49.41 ± 2.62 40.86 ± 3.99 54.79 ± 0.64
HypeBoy + BERT 61.98 ± 1.70 32.73 ± 1.71 OOM OOM OOM OOM
VilLain (w/o feat.) 42.22 ± 1.72 53.72 ± 1.00 40.99 ± 3.06 33.94 ± 1.39 20.99 ± 0.91 42.04 ± 0.04
HyperBERT (TAHG) 58.25 ± 3.04 47.03 ± 0.77 62.12 ± 0.19 41.46 ± 0.14 26.79 ± 0.20 OOM
HiTeC (Ours) 68.69 ± 0.97 74.07 ± 0.88 79.81 ± 0.16 59.64 ± 0.16 55.81 ± 0.12 67.18 ± 0.19

Table 2: Hyperedge prediction performance comparison (% mean ± std)

Category / Method Citeseer Cora History Photo Computers Fitness
GraphCL + GIANT 72.70 ± 2.06 55.48 ± 1.96 63.02 ± 1.61 68.78 ± 1.16 70.87 ± 0.87 63.02 ± 0.01
BGRL + GIANT 72.61 ± 1.99 66.22 ± 2.38 65.13 ± 2.09 65.65 ± 1.10 67.67 ± 0.99 60.92 ± 0.34
TriCL + BERT 71.36 ± 1.78 64.02 ± 2.33 70.18 ± 0.31 71.71 ± 0.21 73.20 ± 0.17 65.49 ± 0.40
SE-HSSL + BERT 72.29 ± 1.74 65.66 ± 2.25 67.93 ± 0.44 71.27 ± 0.18 70.68 ± 0.91 66.00 ± 0.69
HypeBoy + BERT 74.56 ± 2.10 67.38 ± 1.89 OOM OOM OOM OOM
HyperBERT 71.84 ± 2.03 64.48 ± 2.10 69.52 ± 0.20 69.34 ± 0.18 70.40 ± 0.15 OOM
HiTeC (Ours) 75.22 ± 1.87 65.91 ± 2.32 71.83 ± 0.21 72.56 ± 0.17 73.91 ± 0.14 67.32 ± 0.09

Ablation Study

The ablation investigations dissect the complementary contributions of Stage 1 pretraining versus Stage 2 hypergraph contrastive learning, as well as the individual components of the text view and data augmentations.

Table 3: Controlled node classification performance using shared Stage 1 representations

Method Variant Cora Photo Computers Fitness
TriCL (w/ Stage-1 features) 66.17 ± 1.25 57.55 ± 1.06 48.20 ± 1.87 58.86 ± 0.48
SE-HSSL (w/ Stage-1 features) 66.30 ± 0.23 57.80 ± 1.13 47.63 ± 1.12 59.12 ± 0.34
HiTeC (Full model) 74.07 ± 0.88 59.64 ± 0.16 55.81 ± 0.12 67.18 ± 0.19

Table 4: Key component and text view ablation analysis

Variant Modification Empirical Impact
w/o pre Removes Stage 1 structure-aware text encoder pretraining Causes the steepest accuracy drop, decreasing by 13.41% on Fitness; highlights the necessity of topology-enhanced text features
w/o s Removes the s-walk subgraph contrastive objective \(\mathcal{L}_s\) Drops accuracy by 7.14% on Photo; proves the critical value of modeling long-range higher-order dependencies
w/o shd Replaces semantic-aware hyperedge dropping with random dropping Triggers substantial performance degradation; demonstrates that random perturbations corrupt semantic consistency
w/o sct Removes structure-contextualized text view (using only raw text) Lacks contextual richness and diversity; causes consistent performance declines across all datasets
w/o NC Omits Neighbor Context from the contextualized text prompt Produces the largest degradation among text view ablations, showing local text context is the most critical attribute
w/o GT / w/o DK Omits Global Topology or Domain Knowledge from the prompt Leads to steady drops, confirming that explicit graph metadata enhances textual semantic expressiveness

Key Findings

  • Two-stage synergy over feature upgrades: Feeding Stage 1's topology-enhanced text representations into the strongest baseline models (TriCL and SE-HSSL) yields modest gains on Cora (reaching 66.17% and 66.30%), but both remain far below HiTeC's 74.07% (a margin of nearly 8%), with an 8.06% gap persisting on Fitness. This proves that HiTeC's superiority stems from the synergistic interplay between semantic-aware hypergraph augmentations and s-walk hierarchical objectives in Stage 2.
  • The structural coherence vs. diversity tradeoff in s-walk: In sensitivity analyses of overlap constraint \(s\), setting \(s = 1\) reduces the traversal to an unconstrained random walk that lacks structural cohesion, whereas setting \(s = 10\) overly restricts candidate hyperedge pools, impairing contextual diversity. An intermediate overlap threshold of \(s = 3\) consistently strikes the optimal balance between topological cohesion and contextual variety.
  • Efficient representative subgraph sampling: Sampling subgraphs across all nodes is computationally prohibitive. Retaining only the top \(r = 30\%\) degree-ranked nodes achieves classification accuracy comparable to \(r = 50\%\) or full sampling while providing \(1.6\times\) to \(2.2\times\) training acceleration.

Highlights & Insights

  • Decoupled training circumvents memory bottlenecks: Prior TAHG approaches like HyperBERT rely on joint end-to-end training of Transformers interleaved with hypergraph layers, resulting in out-of-memory (OOM) failures on large-scale datasets. HiTeC cleanly decouples text prior injection from hypergraph contrastive learning, maintaining scalability comparable to pure hypergraph models.
  • Semantic cohesiveness guidance for structural augmentations: Unlike traditional graph contrastive learning that drops edges uniformly at random, HiTeC weights hyperedge incidences using the pairwise semantic similarity of connected nodes. This preserves tightly cohesive concepts while filtering weakly correlated noise.
  • Generalization potential of s-walk subgraph contrast: The s-walk sampling protocol provides a principled tool for extracting cohesive subgraphs from hypergraphs. Its parameter \(s\) provides tunable control over topological rigidity, offering immediate transferability to multi-hop reasoning over knowledge hypergraphs and molecular hypergraph learning.

Limitations & Future Work

  • Author-admitted limitations: The structure-contextualized text view relies on handcrafted prompt templates. While empirically robust across six diverse benchmarks, the optimal phrasing may differ across novel domains. Additionally, Stage 1 pretraining introduces modest upfront training overhead compared to feature-fixed HCL baselines.
  • Reviewer-identified limitations: The s-walk traversal operates strictly on observed hyperedge incidences; on extremely sparse or highly skewed graphs, high overlap requirements (\(s \ge 3\)) can encounter dead-ends or oversample dominant cliques. Furthermore, Stage 2 employs an elementary single-layer HGNN backbone without exploring advanced hypergraph attention mechanisms (such as AllSet or HNHN).
  • Future directions: Integrating LLM generative prompting to dynamically construct structure-contextualized descriptions, and generalizing s-walk to continuous or probabilistic overlaps to support evolving dynamic hypergraphs.
  • vs TriCL & SE-HSSL: TriCL designs tri-directional contrastive signals across nodes, hyperedges, and memberships, while SE-HSSL optimizes sampling-efficient membership contrast. However, both are graph-agnostic with respect to raw text, relying on shallow embeddings or frozen PLM features, and neither models multi-scale subgraph contexts. HiTeC bridges this gap through structure-aware text pretraining and s-walk subgraph learning.
  • vs HyperBERT: HyperBERT is the pioneering self-supervised approach tailored for TAHGs, but its end-to-end joint training architecture incurs prohibitive memory overhead (OOM on four out of six benchmarks) and restricts contrastive signals to 1-hop hyperedge scopes. HiTeC's two-stage decoupled paradigm achieves \(3\times\) faster training than TriCL on large datasets while scaling seamlessly to over 1.4 million hyperedges.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Decoupled structure-aware text injection paired with s-walk-induced multi-scale hypergraph contrastive learning provides a sound, principled framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across six real-world TAHG benchmarks spanning two distinct domains, supported by controlled feature experiments, extensive ablations, and sensitivity analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with crisp formal mathematical definitions and transparent methodological explanations.
  • Value: ⭐⭐⭐⭐☆ Establishes a standard benchmark suite and an effective architectural reference for self-supervised representation learning on text-attributed hypergraphs.