SAILRec: Steering LLM Attention to Dual-Side Semantically Aligned Collaborative Embeddings for Recommendation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/XuPhoenixyear/SAILRec
Area: Safety
Keywords: Recommender Systems, Large Language Models, Collaborative Filtering, Semantic Alignment, Hierarchical Attention Steering
TL;DR¶
To address the severe semantic gap between external collaborative embeddings and internal LLM representations as well as premature interference across Transformer depths, SAILRec introduces dual-side semantic alignment (via phrase-level user codebooks and item text targets) coupled with hierarchical attention steering (shallow suppression, neutral middle, and deep enhancement), achieving consistent performance gains in click-through rate prediction and personalized ranking.
Background & Motivation¶
Large language models (LLMs) have brought revolutionary semantic reasoning, contextual understanding, and instruction-following capabilities to recommender systems, yet they inherently lack the collaborative filtering signals and structural behavioral patterns distilled from historical user-item interactions. Recent research (e.g., CoLLM, LLaRA) attempts to bridge this gap by injecting collaborative embeddings learned from conventional models (such as Matrix Factorization) into the LLM as soft prompt tokens. However, making continuous collaborative embeddings available at the input layer does not guarantee their proper utilization during inference. These numerical vectors remain fundamentally disconnected from the linguistic semantic space of the LLM, often behaving as out-of-distribution noise that confuses standard self-attention mechanisms.
Diagnostic probing experiments reveal a deeper layer-wise contradiction: measuring the attention allocated from the final answer generation position toward collaborative and textual tokens demonstrates that collaborative signal utilization is intrinsically depth-dependent and alignment-sensitive. In standard Transformer architectures, shallow layers are predominantly dedicated to encoding surface-level lexical patterns, local syntax, and textual context. Unchecked exposure to raw collaborative tokens in early layers leads to premature interference with the natural comprehension of historical items. Conversely, top decision layers responsible for high-level preference matching and final ranking require consolidated collaborative evidence, yet they often fail to receive sufficient attention allocation. Existing approaches either inject unaligned vectors directly or restrict semantic alignment exclusively to the item side (e.g., SeLLa-Rec), neglecting the multi-layer dynamic regulation of collaborative signals inside the network.
Consequently, effective collaborative integration requires solving two coupled challenges: ensuring collaborative embeddings are linguistically understandable to the LLM semantic space, and precisely regulating when and where these external signals should influence Transformer inference. Core idea: SAILRec bridges the semantic representation gap through dual-side semantic alignmentโanchoring user collaborative signals via three-dimensional phrase-level codebooks and item signals via text semanticsโand introduces hierarchical attention steering inside the LLM (shallow-layer suppression, middle-layer neutral interaction, and deep-layer enhancement) to ensure collaborative evidence governs final decision making without corrupting early textual understanding.
Method¶
Overall Architecture¶
The end-to-end framework of SAILRec consists of three decoupled operational stages: collaborative embedding construction, dual-side semantic warm-up, and supervised fine-tuning under hierarchical attention steering. A frozen Matrix Factorization (MF) backbone first supplies foundational user and item collaborative embeddings. Two Collaborative Q-Formers (C-QFormers) project these latent vectors into the LLM's high-dimensional hidden space using contrastive alignment against semantic targets. Finally, the concatenated prompt tokens are processed by the LLM backbone (Qwen2-7B), where layer-wise attention biases steer the information flow toward collaborative tokens across Transformer depths before generating the binary click probability.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Interaction Data<br/>User History & Target Candidate Item"] --> B["Stage 1: Conventional Collaborative Modeling<br/>Frozen MF Distills Latent Embeddings"]
B --> C["Stage 2: Dual-Side Semantic Alignment<br/>C-QFormer Maps Codebook Profiles & Item Texts"]
C --> D["Prompt Construction<br/>Concatenate Text Prompts with Aligned Collab. Tokens"]
D --> E["Stage 3: Hierarchical Attention Steering<br/>Shallow Suppression ยท Middle Neutral ยท Deep Enhancement"]
E --> F["Output Prediction<br/>Binary Softmax over 'Yes'/'No' Token Logits"]
Key Designs¶
1. Codebook-Based User-Side Semantic Alignment: Grounding Anonymous IDs with Multi-Facet Preferences
Because anonymous user IDs lack intrinsic linguistic meaning in the pre-trained semantic space of language models, injecting raw user embeddings forces the model to interpret arbitrary numeric vectors. Furthermore, a user's latent preference is rarely monolithic; it spans stylistic presentation, affective resonance, and philosophical thematic orientation. To address this, SAILRec constructs three distinct phrase-level semantic codebooks: Style (\(C_A\)), Emotion (\(C_B\)), and Theme (\(C_C\)), each populated with carefully screened semantic tags. For a user's interaction history \(H_u = (i_1, \dots, i_T)\), an item-to-tag selector identifies the closest codebook tag for each historical item, and weighted voting incorporating positional and temporal decay aggregates these into a three-element semantic user profile \(P_u = \{p^A_u, p^B_u, p^C_u\}\). A lightweight Collaborative Q-Former (C-QFormer) with three learnable queries performs cross-attention over the frozen user MF embedding \(e^c_u\), producing high-dimensional collaborative tokens \(Z_u = [z^A_u, z^B_u, z^C_u] \in \mathbb{R}^{3 \times d_L}\) corresponding to <User_ID1>, <User_ID2>, and <User_ID3>. The module is trained using slot-wise InfoNCE contrastive losses against target embeddings \(T_u\) extracted from the frozen LLM:
$\(L^r_u = -\frac{1}{|B|} \sum_{u \in B} \log \frac{\exp(\text{sim}(z^r_u, t^r_u)/\tau)}{\sum_{v \in B} \exp(\text{sim}(z^r_u, t^r_v)/\tau)}\)$
This structured alignment anchors implicit collaborative user dynamics to explicit linguistic concepts, converting uninterpretable ID embeddings into semantically coherent prompt tokens.
2. Item-Side Single-Query Semantic Distillation: Synchronizing Item Representations with Natural Language Semantics
Unlike abstract user profiles, candidate items naturally possess descriptive textual metadata such as titles and genre descriptions that are already well-represented within the LLM's pre-trained vocabulary. To resolve the cross-modal discrepancy between item collaborative vectors and text descriptions, SAILRec introduces an item-side C-QFormer equipped with a single learnable query. This module takes the frozen item collaborative embedding \(e^c_i\) as key-value input and maps it into a single collaborative token embedding \(z_i \in \mathbb{R}^{d_L}\) associated with the <Item_ID> placeholder. Supervised by an InfoNCE loss against the frozen LLM semantic representation of the item's textual title \(t_i = E_{\text{LLM}}(\text{title}_i)\), the mapped item vector is directly pulled into the LLM's textual embedding space, preserving interaction topology while ensuring compatibility with prompt context.
3. Hierarchical Attention Steering: Mitigating Early Interference and Boosting Late Collaborative Evidence
To resolve the depth-dependent tension identified in attention diagnostics, SAILRec partitions the \(L\) Transformer layers into three functional stages: shallow layers (early 30%), middle layers (intermediate 50%), and deep layers (top 20%). During the forward pass, a structured bias matrix \(\Delta^\ell\) is added to the attention logits exclusively along paths targeting collaborative token key positions \(C\): $\(A^\ell = \text{softmax}\left(\frac{Q^\ell (K^\ell)^\top}{\sqrt{d_h}} + M + \Delta^\ell\right)\)$ In shallow layers, a constant negative bias \(\Delta^\ell_{q, k} = b_s < 0\) (\(b_s = -1.0\)) suppresses premature attention to collaborative tokens, allowing the network to focus unobstructed on parsing lexical, syntactic, and contextual history information. In middle layers, the bias is set to zero (\(\Delta^\ell_{q, k} = 0\)), providing an unconstrained zone for textual and collaborative features to interact naturally through residual streams. In deep layers approaching final decision logits, a depth-dependent positive bias \(\Delta^\ell_{q, k} = b_d(\ell) > 0\) (following a triangular schedule peaking at \(+2.0\)) actively reinforces the aggregation of collaborative evidence for final preference matching.
Loss & Training¶
SAILRec employs a three-stage decoupled training strategy: 1. Stage 1 (Collaborative Pre-training): Matrix Factorization (MF) is trained on historical click logs with latent dimension \(d_c = 256\), and its parameters are subsequently frozen. 2. Stage 2 (Dual-Side Contrastive Warm-Up): The user-side and item-side C-QFormers are pre-warmed independently using InfoNCE contrastive alignment (\(\tau = 0.07\), learning rate \(1 \times 10^{-3}\), batch size 128) against the semantic codebook profiles and item title representations. 3. Stage 3 (Supervised Fine-Tuning with Attention Steering): Keeping the MF backbone frozen, LoRA parameters applied to the LLM (Qwen2-7B, hidden size 3584) and both C-QFormers are jointly fine-tuned under hierarchical attention steering. Optimization is guided by binary cross-entropy: $\(L_{\text{rec}} = -y_{u,i} \log \hat{y}_{u,i} - (1 - y_{u,i}) \log (1 - \hat{y}_{u,i})\)$ where the prediction probability \(\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}}) + \exp(o_{\text{No}})}\) is computed via binary softmax over vocabulary logits.
Key Experimental Results¶
Main Results¶
On MovieLens-1M (dense movie rating scenario) and Amazon-Book (sparse, highly long-tailed e-commerce domain), SAILRec consistently outperforms traditional collaborative filtering models, LLM-only recommenders, and state-of-the-art collaborative-enhanced LLM architectures.
| Dataset | Method | AUC | UAUC | NDCG | MAP |
|---|---|---|---|---|---|
| MovieLens-1M | MF | 0.6429 | 0.6126 | 0.8422 | 0.7093 |
| MovieLens-1M | LightGCN | 0.5831 | 0.6493 | 0.8532 | 0.7320 |
| MovieLens-1M | SASRec | 0.7010 | 0.6742 | 0.8581 | 0.7411 |
| MovieLens-1M | TALLRec | 0.7039 | 0.6837 | 0.8701 | 0.7598 |
| MovieLens-1M | GDRT | 0.7249 | 0.6871 | 0.8689 | 0.7537 |
| MovieLens-1M | HatLLM | 0.7330 | 0.7058 | 0.8762 | 0.7645 |
| MovieLens-1M | CoLLM | 0.7137 | 0.6850 | 0.8757 | 0.7647 |
| MovieLens-1M | BinLLM | 0.7271 | 0.6980 | 0.8833 | 0.7779 |
| MovieLens-1M | CoRA | 0.7284 | 0.6956 | 0.8744 | 0.7614 |
| MovieLens-1M | CKF | 0.7320 | 0.7120 | 0.8884 | 0.7830 |
| MovieLens-1M | TokenRec | 0.7272 | 0.7036 | 0.8798 | 0.7702 |
| MovieLens-1M | SeLLa-Rec (Prev. Best) | 0.7428 | 0.7133 | 0.8885 | 0.7825 |
| MovieLens-1M | SAILRec (Ours) | 0.7571 | 0.7410 | 0.8985 | 0.7999 |
| Amazon-Book | MF | 0.7098 | 0.5630 | 0.8221 | 0.7420 |
| Amazon-Book | LightGCN | 0.7001 | 0.5608 | 0.8218 | 0.7415 |
| Amazon-Book | SASRec | 0.6646 | 0.5629 | 0.8208 | 0.7393 |
| Amazon-Book | TALLRec | 0.7198 | 0.6524 | 0.8624 | 0.7959 |
| Amazon-Book | GDRT | 0.7607 | 0.6280 | 0.8505 | 0.7800 |
| Amazon-Book | HatLLM | 0.7397 | 0.6147 | 0.8475 | 0.7754 |
| Amazon-Book | CoLLM | 0.7924 | 0.7197 | 0.8896 | 0.8351 |
| Amazon-Book | BinLLM | 0.8031 | 0.7234 | 0.8922 | 0.8384 |
| Amazon-Book | CoRA | 0.8156 | 0.7201 | 0.8930 | 0.8393 |
| Amazon-Book | CKF | 0.8048 | 0.7246 | 0.8929 | 0.8398 |
| Amazon-Book | TokenRec | 0.7975 | 0.6027 | 0.8360 | 0.7603 |
| Amazon-Book | SeLLa-Rec (Prev. Best) | 0.8191 | 0.7310 | 0.8940 | 0.8419 |
| Amazon-Book | SAILRec (Ours) | 0.8393 | 0.7506 | 0.9017 | 0.8524 |
Ablation Study¶
The systematic ablation analysis verifies the respective importance of hierarchical attention steering (S), user/item alignment (UI, U, I), semantic codebooks (C), and Stage 3 co-training strategies.
| Config | Variant Description | MovieLens-1M AUC | MovieLens-1M UAUC | Amazon-Book AUC | Amazon-Book UAUC |
|---|---|---|---|---|---|
| Full-SAILRec | Complete framework | 0.7571 | 0.7410 | 0.8393 | 0.7506 |
| SAIL-w/o-S | Remove attention steering | 0.7546 | 0.7268 | 0.8220 | 0.7194 |
| SAIL-w/o-UI | Remove both-side alignment | 0.7322 | 0.7236 | 0.8116 | 0.7391 |
| SAIL-w/o-I | Remove item-side alignment | 0.7346 | 0.7277 | 0.8267 | 0.7441 |
| SAIL-w/o-U | Remove user-side alignment | 0.7434 | 0.7205 | 0.8288 | 0.7369 |
| SAIL-w/o-C | Replace codebooks with sequence summary | 0.7475 | 0.7295 | 0.8310 | 0.7367 |
| SAIL-LoRA | Freeze C-QFormers in Stage 3 (tune LoRA only) | 0.7283 | 0.7116 | 0.7649 | 0.6928 |
| SAIL-LoRA-Map | Tune mappers separately without joint LoRA tuning | 0.7507 | 0.7225 | 0.8301 | 0.7375 |
Key Findings¶
- Dual-Side Alignment Establishes the Representational Floor, Attention Steering Unlocks the Performance Ceiling: Completely removing dual-side alignment (SAIL-w/o-UI) causes the steepest drops in global AUC (a decline of 0.0249 on MovieLens-1M and 0.0277 on Amazon-Book). In contrast, removing hierarchical attention steering (SAIL-w/o-S) damages user-level personalized discrimination most heavily, causing UAUC to drop by 3.12 percentage points on Amazon-Book, proving that aligned tokens still require layer-wise guidance to prevent cross-layer interference.
- Asymmetric Impact of User Alignment in Sparse Settings: Removing user-side alignment (SAIL-w/o-U) degrades UAUC more severely than removing item-side alignment on MovieLens-1M (dropping to 0.7205 vs 0.7277). This underscores that anonymous IDs absolutely necessitate structured anchor spaces, which cannot be compensated for by monolithic whole-history textual embeddings (w/o-C).
- Partition Ratio and Steering Bias Sensitivity: Varying the steering schedule demonstrates that the default S1E2 configuration (shallow \(-1.0\), deep \(+2.0\), with a 30%-50%-20% layer partition) achieves the best performance. Applying uniform suppression (SAIL-S) or uniform enhancement (SAIL-E) across all layers severely degrades ranking metrics. Furthermore, eliminating the neutral middle zone (e.g., S30-N0-E70) drops AUC and UAUC, verifying that maintaining an unconstrained intermediate layer buffer is crucial for steady representation fusion.
- Inference-Time Masking Proves Genuine Collaborative Reliance: Applying finite negative biases \(\lambda \in \{-1, -2, -5, -8, -10\}\) to collaborative attention paths on Amazon-Book demonstrates that user-side path attenuation causes severe AUC drops (from 0.8393 down to 0.6080 at \(\lambda = -5\)), whereas item-side path suppression primarily degrades personalized ranking quality (UAUC drops from 0.7506 to 0.7220), confirming asymmetric yet complementary functional roles.
Highlights & Insights¶
- Diagnostic-Driven Architectural Innovation: Rather than treating LLM attention as an untampered black box, the authors diagnose layer-wise attention mass from the answer position to uncover the empirical phenomenon of depth-dependent collaborative utilization, designing the steering mechanism directly from diagnostic insights.
- Structured Multi-Facet Preference Grounding: Deconstructing holistic user interests into orthogonal style, emotion, and theme codebooks offers an elegant blueprint for grounding arbitrary continuous IDs into high-level human semantic concepts, boasting a 93.9% cross-domain tag transferability rate from movies to books.
- Lightweight, Training-Free Inference Steering: The hierarchical attention steering mechanism requires zero additional parameters or complex head reparameterizations inside the Transformer layersโmerely injecting constant and scheduled logit biases at key positions during forward attention calculation.
Limitations & Future Work¶
- Hand-Crafted Layer Partitioning and Bias Magnitudes: The current 30%-50%-20% split and fixed \(-1.0 / +2.0\) bias values are heuristic hyper-parameters tuned on specific benchmarks; future work should explore sample-adaptive or learnable continuous bias functions across Transformer layers.
- Semi-Automated Codebook Construction: Adapting semantic codebooks across distant domains still demands modest human inspection to eliminate domain-specific idioms; automated discovery of preference dimensions via unsupervised LLM clustering would enhance transferability.
- Susceptibility Under Severe Cold-Start Scenarios: In extreme cold-start settings where interaction histories are virtually absent, collaborative vectors carry high variance and noise; blindly amplifying deep-layer collaborative attention can degrade performance relative to purely semantic zero-shot baselines.
Related Work & Insights¶
- vs CoLLM / BinLLM (Direct Feature Injection): CoLLM injects raw collaborative embeddings as soft prompts, while BinLLM encodes collaborative IDs as binary text tokens. Both leave representations unaligned with the LLM's internal manifold. SAILRec introduces C-QFormer dual-side contrastive alignment to bridge this semantic gap and explicitly guides layer-wise attention routing.
- vs SeLLa-Rec (Item-Side Semantic Distillation): SeLLa-Rec addresses representation gaps exclusively on the item side via semantic distillation. SAILRec demonstrates that user-side alignment is even more critical for personalized ranking and establishes hierarchical attention steering across network depth.
- vs HatLLM / CTR-Sink (Attention Mechanisms in Recommendation): HatLLM applies hierarchical masking exclusively over textual item histories, and CTR-Sink introduces sink tokens for long sequences. In contrast, SAILRec pioneers attention steering over externally injected continuous collaborative embeddings, clarifying the division of labor between textual syntax and collaborative evidence across Transformer layers.
Rating¶
- Novelty: โญโญโญโญโ Excellent diagnosis of depth-dependent attention patterns, successfully bridging codebook semantic alignment with layer-wise steering.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across two benchmarks, thorough ablations, sensitivity analyses, and finite-bias inference masking tests.
- Writing Quality: โญโญโญโญโญ Rigorous prose, clear formulation, self-consistent terminology, and clean diagnostic visualizations.
- Value: โญโญโญโญโ Provides a practical and lightweight blueprint for fusing multimodal collaborative embeddings into general-purpose LLM backbones.