Dual-Prompt-Driven Textual Knowledge Graph Generation With Heterogeneous Information Embedding for CTR Prediction¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/fxl9/TKE4CTR
Area: Graph Learning
Keywords: Click-Through Rate Prediction, Knowledge Graph Generation, LLM Prompting, Cross-Attention Gated Fusion, Heterogeneous Information Embedding
TL;DR¶
Addressing the severe semantic loss of discrete feature encodings and the lack of fine-grained cross-modal alignment in CTR prediction, TKE4CTR designs a dual-prompt paradigm to transform tabular records into textual narratives and hallucination-mitigated local knowledge subgraphs, followed by bidirectional cross-attention gated fusion and a feature-gated MLP that achieve substantial AUC improvements across five diverse Amazon domains.
Background & Motivation¶
Click-through rate (CTR) prediction serves as the backbone of online advertising and modern e-commerce recommendation platforms, directly determining commercial monetization and user experience. Traditional CTR prediction workflows rely heavily on discrete categorical feature engineering, mapping multi-field tabular attributes into sparse one-hot encoded vectors before learning linear low-order combinations (e.g., via FM, FwFM) or modeling high-order non-linear feature interactions through deep networks such as PNN, DCN, DeepFM, xDeepFM, and AutoInt. However, this discrete paradigm suffers from intrinsic semantic deficiencies: categorical features are isolated into discrete indices, completely stripping away the semantic correlations between different categories (such as brand relationships, category hierarchies, and product descriptions), which inevitably introduces excessive dimensionality, brittle manual schema dependencies, and severe cold-start vulnerability.
With the rapid emergence of pre-trained language models (PLMs) and large language models (LLMs), text-enhanced models (e.g., CTR-BERT, ClickPrompt, Uni-CTR) attempt to verbalize tabular attributes into descriptive sentences to extract implicit contextual semantics, while KG-enhanced models (e.g., DKN, CKAN, MCCLK) exploit relational entities to mitigate data sparsity. Nevertheless, both branches encounter significant bottlenecks: textual representations excel at parsing free-form reviews and descriptions but struggle to capture explicit multi-hop relational dependencies across entities; conversely, graph-based methods depend heavily on static external knowledge graphs or labor-intensive schemas that fail to generalize across diverse domains. Furthermore, prompting LLMs to directly extract triples inevitably introduces rampant factual hallucinations and schema inconsistencies, while existing multimodal CTR architectures resort to naive concatenation or summation that lacks fine-grained alignment and adaptive modality weighting.
The fundamental technical tension lies in how to restructure multi-field discrete tabular records into faithful, complementary structured knowledge and contextual text without relying on external static graphs, while achieving fine-grained alignment between graph topology and unstructured semantics. Core idea: TKE4CTR formulates a dual-prompt-driven modality transformation framework that converts tabular data into standardized text chunks and hallucination-suppressed knowledge subgraphs, synergizing both modalities through an 8-head bidirectional cross-attention gated fusion network and a feature-gated MLP predictor.
Method¶
Overall Architecture¶
The end-to-end pipeline of TKE4CTR consists of four interconnected stages: First, in the dual-prompt modality transformation stage, raw multi-field tabular attributes and interaction sequences are converted into standardized text chunks, and structured relational triples are extracted via a constrained prompt with a two-stage hallucination mitigation pipeline to construct sample-level local knowledge subgraphs. Second, in the dual-modality feature encoding stage, a two-layer Graph Attention Network (GAT) aggregates neighbor information across the local subgraphs, while a frozen PLM (Qwen3-4B) generates dense textual semantic embeddings. Third, in the cross-attention gated modality fusion stage, the heterogeneous embeddings are mapped into a shared latent space, cross-pollinated via 8-head bidirectional cross-attention, and adaptively balanced via dynamic gating units. Finally, a feature-gated MLP prediction head estimates the user-item click probability.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Multi-field tabular data and historical interactions"] --> B["Dual-prompt modality transformation with hallucination mitigation<br/>Standardized text chunks + SCFP extraction with secondary validation"]
B --> C["Dual-modality feature encoding<br/>Two-layer GAT subgraph aggregation + Frozen PLM semantic extraction"]
C --> D["Bidirectional cross-attention gated modality fusion<br/>Shared space projection + 8-head cross-attention + Adaptive gating"]
D --> E["Feature-gated MLP prediction head<br/>Element-wise feature modulation + Dense MLP scoring"]
E --> F["Output: Predicted user-item click probability ลท"]
Key Designs¶
1. Dual-prompt modality transformation with hallucination mitigation: Semantic deficiency and generation hallucinations -> Standardized text templates and two-stage constrained extraction -> Faithful complementary graph and text generation
Discrete feature indices conceal natural semantics, whereas naive prompting of LLMs to generate graph triples leads to severe structural hallucinations, invalid relational labels, and ungrounded facts. To solve this, the authors introduce a dual-prompt paradigm. For text modality, a unified slot-filling template converts domain name, user identifiers, item static attributes, and user review texts into structured text chunks \(x_{text}\), providing consistent context for representation learning. For KG modality, the authors establish a two-stage hallucination mitigation workflow: during extraction, a Structured Constrained Few-shot Prompt (SCFP) enforces a predefined closed set of relations (e.g., has_username, has_price, has_brand, expresses_sentiment, compares_to) and strict JSON array formatting; during verification, a dedicated Secondary Validation prompt inspects textual grounding, schema compliance, and entity matching, allowing up to three regeneration attempts before applying heuristic fallback filtering. This creates a clean local subgraph \(g_{u,p}\) for each sample without the heavy computational overhead of a unified global graph.
2. Dual-modality feature encoding: Limitations of single-modal representation -> GAT local subgraph modeling and frozen PLM semantic encoding -> Collaborative modeling of explicit structure and deep semantics
Neither pure textual representations nor pure relational graphs can simultaneously handle multi-hop relational topology and complex sentiment semantics. TKE4CTR decouples the representation into specialized encoders. For the graph modality, each node \(v\) in subgraph \(g_{u,p}\) is initialized by concatenating heterogeneous features followed by L2 normalization:
$\(x_v^{(0)} = \text{L2Norm}\left(x_v^{type} \parallel x_v^{id} \parallel x_v^{bert} \cdot \mathbb{I}[v \in \mathcal{V}_{core}] \parallel x_v^{struct}\right)\)$
where \(x_v^{type}\) denotes node type embeddings, \(x_v^{id}\) denotes hash-based identity embeddings, \(x_v^{struct}\) encapsulates topological features (degree, clustering coefficient), and BERT embeddings \(x_v^{bert}\) are restricted strictly to core prediction entities (target items) to avoid noisy feature propagation. A two-layer GAT subsequently aggregates neighborhood features: the first layer computes normalized attention coefficients \(\alpha_{vu}^k\) using 8 parallel heads, while the second layer applies single-head attention and mean aggregation to produce the final graph embedding \(h_G \in \mathbb{R}^{d_G}\). For the text modality, the text chunk \(t_{u,p}\) is processed by a frozen PLM (Qwen3-4B); the hidden states of the sequence are mean-pooled and L2-normalized to yield \(h_T \in \mathbb{R}^{d_T}\), preserving rich world knowledge without risk of catastrophic forgetting.
3. Bidirectional cross-attention gated modality fusion: Cross-modal distribution discrepancy and naive fusion redundancy -> Shared space projection, dual cross-attention, and adaptive gating -> Precise cross-modal alignment and dynamic contribution weighting
Because structural graph embeddings \(h_G\) and textual embeddings \(h_T\) reside in divergent vector spaces, simple summation or concatenation cannot capture fine-grained dependencies. The fusion module first maps both embeddings into a shared latent space via Swish activation and LayerNorm:
$\(\tilde{h}_G = \text{LN}(\text{Swish}(W_G h_G + b_G)), \quad \tilde{h}_T = \text{LN}(\text{Swish}(W_T h_T + b_T))\)$
Next, 8-head bidirectional cross-attention computes interactive representations:
$\(A_{GT} = \text{Softmax}\left(\frac{\tilde{h}_G \tilde{h}_T^\top}{\sqrt{d_h}}\right), \quad H'_G = A_{GT}\tilde{h}_T\)$
$\(A_{TG} = \text{Softmax}\left(\frac{\tilde{h}_T \tilde{h}_G^\top}{\sqrt{d_h}}\right), \quad H'_T = A_{TG}\tilde{h}_G\)$
To dynamically trade off original representations and interactive signals, a lightweight gating unit computes an element-wise balance factor \(g = \sigma(W_g [\tilde{h}_G \parallel H'_G \parallel \tilde{h}_T \parallel H'_T] + b_g)\), yielding modulated vectors \(h_G^f = g \odot H'_G + (1-g) \odot \tilde{h}_G\) and \(h_T^f = g \odot H'_T + (1-g) \odot \tilde{h}_T\). Finally, a residual projection and feed-forward network synthesize the unified representation:
$\(z_f = \text{FFN}\left(\frac{h_G^f + h_T^f}{2} + \text{LN}\left(W_r [h_G^f \parallel h_T^f]\right)\right)\)$
4. Feature-gated MLP prediction head: Feature redundancy and channel importance variance -> Instance-level feature-gated modulation -> Suppressing noisy dimensions and boosting inference robustness
Even after cross-attention fusion, not all feature channels within \(z_f\) contribute equally to a specific user-item click decision. To prevent irrelevant dimensions from distorting the decision boundary, an adaptive gating vector modulates the fused embedding prior to the classification layers:
$\(g_f = \sigma(W_p z_f + b_p), \quad \tilde{z}_{u,p} = g_f \odot z_f\)$
This channel-wise gating mechanism dynamically amplifies salient decision signals while dampening uninformative noise. The modulated representation \(\tilde{z}_{u,p}\) passes through a multilayer perceptron (MLP) featuring a hidden layer of 256 units, ReLU activation, Layer Normalization, and a dropout rate of 0.3, with the final prediction generated by a Sigmoid output neuron:
$\(\hat{y}_{u,p} = \sigma\left(w_o^\top \tilde{z}_{u,p} + b_o\right)\)$
Loss & Training¶
The framework is optimized end-to-end using standard Binary Cross-Entropy (BCE) loss: $\(\mathcal{L} = -\frac{1}{N}\sum_{i=1}^N \left[ y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i) \right]\)$ Key training strategies include: - Strict Leakage Prevention: In the training partition, fine-grained review text and historical click interactions are incorporated into prompt generation to help the model learn rich preference associations. In validation and test partitions, review text and interaction features are completely stripped away, with subgraphs constructed solely from static attributes, preventing any post-click data leakage. - Optimization & Scheduling: Trained with AdamW optimizer at an initial learning rate of \(1 \times 10^{-5}\) and weight decay of \(1 \times 10^{-4}\), scheduled with cosine annealing and warm-up. - Training Lifecycle: Trained for up to 50 epochs with early stopping triggered after 25 consecutive epochs of unimproved validation AUC. All experiments are executed on an NVIDIA A100-40GB GPU using PyTorch with random seed fixed to 42.
Key Experimental Results¶
Main Results¶
Experiments are evaluated on five sub-domains of the 2018 Amazon Review Dataset spanning 12 years (1996โ2008), chronologically split into training (80%), validation (10%), and test (10%) sets. Performance is measured by Area Under the ROC Curve (AUC) and Relative Improvement (RelaImpr, \(RI = \frac{\text{AUC}_{target} - 0.5}{\text{AUC}_{baseline} - 0.5} - 1\)). Note that a 0.001 increase in AUC is widely recognized as a statistically and practically significant gain in CTR systems.
The table below summarizes the comparison between TKE4CTR and ten competitive baselines:
| Models | All Beauty (AUC / RI) | Amazon Fashion (AUC / RI) | Digital Music (AUC / RI) | Gift Cards (AUC / RI) | Mus. Instruments (AUC / RI) |
|---|---|---|---|---|---|
| PNN (ICDM 2016) | 0.6000 / 161.6% | 0.6979 / 56.7% | 0.5904 / 266.0% | 0.5959 / 240.0% | 0.6859 / 55.3% |
| DCN (ADKDD 2017) | 0.6200 / 118.0% | 0.6985 / 56.3% | 0.5919 / 260.0% | 0.6126 / 190.0% | 0.6893 / 52.5% |
| DeepFM (IJCAI 2017) | 0.6050 / 150.0% | 0.6982 / 56.5% | 0.5917 / 261.0% | 0.5937 / 248.0% | 0.6880 / 53.5% |
| xDeepFM (KDD 2018) | 0.6220 / 114.4% | 0.7031 / 52.7% | 0.5957 / 246.0% | 0.6121 / 191.0% | 0.6892 / 52.6% |
| DIEN (AAAI 2019) | 0.6180 / 121.7% | 0.6995 / 55.5% | 0.5900 / 267.7% | 0.6105 / 195.0% | 0.6881 / 53.4% |
| AutoInt (CIKM 2019) | 0.6080 / 142.2% | 0.7003 / 54.9% | 0.5913 / 262.4% | 0.5976 / 234.0% | 0.6888 / 52.9% |
| FiBiNET (RecSys 2019) | 0.6200 / 118.0% | 0.6770 / 75.3% | 0.5832 / 297.7% | 0.6120 / 191.0% | 0.6878 / 53.7% |
| IntTower (CIKM 2022) | 0.6180 / 121.7% | 0.6988 / 56.0% | 0.5920 / 259.7% | 0.6100 / 196.0% | 0.6888 / 52.9% |
| FinalMLP (AAAI 2023) | 0.6130 / 131.5% | 0.6907 / 62.7% | 0.5947 / 249.4% | 0.6082 / 201.0% | 0.6884 / 53.2% |
| Uni-CTR (TOIS 2025) | 0.7250 / 16.27% | 0.7523 / 23.0% | 0.6140 / 190.3% | 0.7246 / 45.2% | 0.7569 / 12.3% |
| TKE4CTR (Ours) | 0.7616 / โ | 0.8102 / โ | 0.8309 / โ | 0.8261 / โ | 0.7886 / โ |
Underlined indicates the strongest baseline; bold indicates the overall best performance.
Ablation Study¶
1. Input Modality Ablation Across All Domains (Table 3 & Appendix Table 12):
| Input Modality | All Beauty (AUC) | Amazon Fashion (AUC) | Digital Music (AUC) | Gift Cards (AUC) | Mus. Instruments (AUC) | Description |
|---|---|---|---|---|---|---|
| Text-KG (Full Model) | 0.7616 (0.7769) | 0.8102 | 0.8309 | 0.8261 | 0.7886 | Joint multimodal fusion |
| w/o KG (only Text) | 0.7411 | 0.7305 | 0.7878 | 0.7834 | 0.7240 | Lacks explicit relational structure |
| w/o Text (only KG) | 0.6624 | 0.7470 | 0.7955 | 0.7808 | 0.7545 | Lacks deep textual semantics |
| only ID | 0.5985 | 0.6518 | 0.6340 | 0.5837 | 0.6511 | Discrete categorical IDs only |
2. Component Ablation Across All Domains (Table 4 & Appendix Table 13):
| Model Variant | All Beauty (AUC) | Amazon Fashion (AUC) | Digital Music (AUC) | Gift Cards (AUC) | Mus. Instruments (AUC) | Impact Analysis |
|---|---|---|---|---|---|---|
| TKE4CTR (Full) | 0.7616 | 0.8102 | 0.8309 | 0.8316 | 0.7886 | Optimal architecture |
| w/o Cross-Attn Gate | 0.7593 | 0.8021 | 0.8228 | 0.8161 | 0.7801 | Degrades to unaligned fusion |
| w/o Feat-Gated Head | 0.7585 | 0.8082 | 0.8261 | 0.8212 | 0.7853 | Lacks dimension-level feature modulation |
3. Impact of Prompt Strategy and Secondary Validation on Hallucinations (Tables 6 & 7):
| Prompt Strategy / Validation Setup | Samples | Triples | NLI Fidelity (NLI-F) โ | Rule Fidelity (R-F) โ | Hallucination Rate (HR) โ |
|---|---|---|---|---|---|
| Baseline Prompt (BP) | 100 | 1128 | 15.41% | 14.76% | 84.92% |
| Structured Constrained Zero-shot (SCZP) | 100 | 896 | 58.37% | 61.42% | 59.16% |
| Structured Constrained Few-shot (SCFP, w/o Valid.) | 100 | 835 | 72.15% | 80.34% | 36.28% |
| SCFP + Secondary Validation (w/ Valid.) | 100 | 779 | 77.53% | 85.67% | 33.42% |
Key Findings¶
- Cross-modal complementarity exhibits clear domain divergence: Textual semantics dominate in All Beauty (0.7411 for only Text vs. 0.6624 for only KG), where cosmetic attributes and review descriptors guide user choice. Conversely, relational knowledge is far more influential in Amazon Fashion (0.7470 for only KG vs. 0.7305 for only Text), where explicit brand identity and product co-click relationships are critical. Fusing both yields significant gains in both domains (0.7616 and 0.8102).
- Constrained few-shot prompting and secondary validation curb hallucination dramatically: Unconstrained baseline prompting incurs a disastrous 84.92% hallucination rate. Enforcing structured few-shot constraints (SCFP) cuts this to 36.28%, while iterative secondary validation further pushes Rule-based Fidelity to 85.67% and NLI Fidelity to 77.53%, ensuring high-fidelity graph construction.
- Cross-attention head scaling reaches saturation at 8 heads: Increasing attention heads from 2 to 8 yields steady AUC gains across all datasets with modest latency increases (~0.15 ms/sample). Expanding to 16 heads causes performance saturation (marginal AUC drops or sub-0.001 gains) while doubling inference latency to ~0.32 ms/sample, validating the 8-head design choice.
Highlights & Insights¶
- Decoupled Text-Subgraph Dual Transformation: Bypasses the need for expensive external static knowledge bases by programmatically generating sample-specific, self-contained knowledge subgraphs and text chunks directly from tabular records via LLM prompting.
- Closed-Loop Hallucination Mitigation: Combines SCFP schema constraints with an iterative secondary verification filter, providing an empirical quality control standard (evaluated via NLI-F, R-F, and HR) for LLM-based graph generation.
- Principled Leakage-Free Protocol: Skillfully leverages rich post-interaction review content during training while strictly restricting validation and inference to basic static attributes, resolving the common data leakage pitfall in text-based recommender research.
Limitations & Future Work¶
- High Computational Overhead for Offline Graph Generation: Querying 8B-scale LLMs for triple extraction introduces significant offline pre-processing latency, constraining its immediate application to ultra-high-throughput real-time systems. Future work should explore lightweight distilled extractors.
- Static Subgraph Modeling: Current subgraphs are constructed from static snapshots and do not model long-term dynamic user interest migration or session-level evolving graph topologies over time.
- Reliance on Heuristic Prompt Templates: Templates and schema rules require manual domain-specific definitions; future explorations could investigate automated prompt optimization across unseen recommendation categories.
Related Work & Insights¶
- vs Uni-CTR (TOIS 2025): Uni-CTR relies strictly on LLMs to construct cross-domain textual representations; TKE4CTR demonstrates that textual narratives alone miss explicit relational topology, and integrating a GAT-encoded graph yields 12.3% to 190.3% relative AUC gains.
- vs CKAN / KGCAN (Graph-based CTR models): Prior KG-enhanced recommenders require pre-existing global KGs with tedious entity-linking; TKE4CTR constructs ad-hoc, task-adaptive subgraphs on-the-fly directly from tabular fields.
- vs ClickPrompt / FLIP: These methods align ID embeddings with PLM token representations; TKE4CTR elevates cross-modal alignment to the level of explicit graph structures versus contextual text via bidirectional cross-attention.
Rating¶
- Novelty: โญโญโญโญโ (Creative dual-prompt formulation that constructs hallucination-mitigated local subgraphs without external knowledge bases)
- Experimental Thoroughness: โญโญโญโญโญ (Comprehensive validation across five real-world e-commerce domains, covering baselines, ablations, hallucination metrics, and parameter sensitivity)
- Writing Quality: โญโญโญโญโญ (Clean mathematical formalization, comprehensive figures, and well-structured empirical analysis)
- Value: โญโญโญโญโ (Offers an actionable, high-performing paradigm for uniting LLM text capabilities with GNN structural modeling in tabular recommendation)