Skip to content

curp codebook based continuous user representation for personalized generation w

content_hash: af2bbc1f6262a96b ---# CURP: Codebook-based Continuous User Representation for Personalized Generation with LLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/RaidonWong/CURP_code
Area: NLP Generation
Keywords: LLM Personalization, Prototype Codebook, Continuous User Representation, Dual-Aggregator, LaMP Benchmark

TL;DR

CURP proposes a plug-and-play personalization framework based on a discrete prototype codebook and a dual-aggregator, factorizing user interaction history into orthogonal subspace prototypes to capture query-adaptive dynamic preferences and stable intrinsic traits, achieving state-of-the-art personalized generation while training only 28M parameters (0.36% of the base LLM).

Background & Motivation

Personalized text generation with Large Language Models (LLMs) aims to mirror an individual's unique preferences, stylistic habits, and behavioral patterns, powering applications from personalized conversational agents and recommender systems to assistive writing and multi-agent social simulations. Contemporary LLM personalization paradigms predominantly fall into two categories: personalized prompt-based methods and personalized parameter-based methods. Prompt-based approaches directly prepend raw interaction histories or natural language persona profiles into the input context. While lightweight and versatile, raw historical records frequently contain surface lexical noise, uninformative filler, and redundant tokens; furthermore, excessively long prompts balloon inference latency and easily mislead the model into rigid stereotyping or hallucinatory responses. Conversely, parameter-based approaches adapt model weights by fine-tuning dedicated parameter modules (such as per-user LoRA adapters) on user-specific data. Although they achieve high fidelity, maintaining and serving separate parameter checkpoints for millions of distinct users incurs prohibitive storage overhead and engineering latency.

This division exposes a fundamental dilemma between personalization fidelity and computational scalability. A principled perspective emerges from Self-Categorization Theory (SCT) in social psychology: human identity is not an isolated, static self-concept, but rather a context-sensitive, continuous blend of shared social prototypes (e.g., #Teenager, #Faithful, #Sport Fan). If user behavior can be represented in a shared latent prototype space, systems can aggregate behavioral signals across users to mitigate data sparsity, filter out superficial lexical noise, and flexibly approximate complex individual personas through discrete prototype compositions.

CURP's angle of attack is to quantize user interaction histories into a discrete prototype codebook over orthogonal subspaces, followed by a dual-aggregator that decouples representations into query-adaptive dynamic preferences and long-term stable characteristics, injecting them as continuous soft prefixes into a frozen LLM. Core idea: Grounded in Self-Categorization Theory, construct a decoupled subspace prototype codebook and a dual-aggregator that explicitly decomposes historical behaviors into query-aware dynamic intent and long-term stable traits, achieving robust, interpretable, plug-and-play LLM personalization with only 28M trainable parameters.

Method

Overall Architecture

CURP is structured around four primary components: a frozen bidirectional User Encoder \(E\), a discrete Prototype Codebook \(C\), a Dual-Aggregator \(A\), and a Decoder LLM \(D\). In the end-to-end pipeline, historical user behaviors are first transformed into dense continuous embeddings by the user encoder, and then discretized via Product Quantization (PQ) against subspace codebooks to produce denoised prototype representations. Next, the dual-aggregator processes these prototype embeddings along two complementary pathways: a multi-head cross-attention mechanism that fuses the current task query to extract dynamic preferences, and a multi-layer Q-Former with learnable queries that summarizes long-term stable characteristics. Finally, a lightweight MLP projection module maps both representations into the decoder LLM's input embedding space, prepending them to the query embeddings for autoregressive generation. Training proceeds through two disjoint stages: Prototype Codebook Construction (PCC) and User Behavior Alignment (UBA).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Interaction History<br/>and Current Task Query"] --> B["User Encoding & Subspace Quantization<br/>Contriever Dense Encoding & Product Quantization"]
    B --> C["Prototype Codebook Construction<br/>Balanced K-Means with Diversity Regularization"]
    C --> D["Dual-Aggregator Representation Decoupling<br/>MHCA Dynamic Preferences & Q-Former Stable Traits"]
    D --> E["Latent Projection & Behavior Alignment<br/>Lightweight MLP Mapping & Autoregressive Decoding"]
    E --> F["Personalized Response Output"]

Key Designs

1. Prototype Codebook Construction: High-Dimensional Semantic Factorization and Balanced Subspace Clustering To eliminate the lexical noise and context-window expansion caused by concatenating raw text histories, CURP applies Product Quantization (PQ) to map continuous embeddings into combinations of discrete prototypes. A frozen Contriever encoder first transforms a large pool of historical behaviors into dense vectors. Each embedding \(e \in \mathbb{R}^d\) is partitioned into \(L\) orthogonal subspaces \(e = [e^{(1)}, \dots, e^{(L)}]\), where each subspace is quantized against a dedicated codebook \(C_l\) with \(K\) entries. To avoid centroid collapse and "dead codes", codebook initialization employs a Balanced K-Means algorithm enforcing equal-cardinality constraints across clusters. In the PCC stage, codebooks are trained with a composite objective combining reconstruction fidelity, codebook diversity, and usage entropy:

\[ \mathcal{L}_{\text{PCC}} = \alpha \mathcal{L}_{\text{reconstruction}} + \beta \mathcal{L}_{\text{diversity}} + \gamma \mathcal{L}_{\text{usage}} \]

where the reconstruction loss relies on the Straight-Through Estimator (STE) for gradient backpropagation, the diversity loss utilizes a smooth hyperbolic tangent penalty to maximize pairwise centroid separation, and the usage loss penalizes usage variance across batches while maximizing active code cardinality and normalized entropy. Through this formulation, any user behavior is abstracted into \(L\) discrete indices, filtering out surface lexical irregularities while preserving core semantic patterns.

2. Dual-Aggregator: Decoupling Query-Adaptive Contextual Intent and Long-Term Stable Traits User actions are driven by both transient context-specific intents and persistent, domain-wide behavioral tendencies. Conflating both aspects into a single vector leads to information entanglement and unstable generations. The dual-aggregator explicitly factorizes user modeling into two parallel branches: The dynamic branch employs an 8-head Multi-Head Cross-Attention (MHCA) module, taking the encoded current query \(E(Q_i)\) as Query and the quantized prototype sequence \(BP_i\) as Key and Value, yielding query-sensitive dynamic preferences \(R_{\text{dyn}} = \text{MHCA}(BP_i, E(Q_i))\). The stable branch employs a 2-layer Q-Former with \(M=4\) global learnable queries \(Q_{\text{learned}}\) interacting with the entire prototype sequence \(BP_i\) via cross-attention, compressing the user history into fixed-length intrinsic representations \(R_{\text{stb}} = \text{Q-Former}(BP_i, Q_{\text{learned}})\). Even when the current query diverges from historical topics, the stable branch anchors the generation on the user's habitual style and distribution, preventing catastrophic degradation.

3. User Behavior Alignment: Decoder-Agnostic Soft-Prompt Injection In Stage 2 (UBA), to seamlessly guide the generative LLM without expensive per-user model tuning, both dynamic and stable representations are projected into the decoder's embedding space via a shared Multi-Layer Perceptron (MLP, featuring a 768-3584-3584 linear structure with GELU activation). The projected tokens are prepended directly to the embedded task query \(Q_i\):

\[ x_{\text{in}} = [f(R_{\text{dyn}}, R_{\text{stb}}); \mathcal{D}_{\text{Emb}}(Q_i)] \]

The entire decoder backbone (Qwen-2.5-7B-Instruct) remains completely frozen throughout UBA training; only the aggregator and projection MLP (totaling ~28M parameters) are optimized via autoregressive negative log-likelihood:

\[ \mathcal{L}_{\text{UBA}} = -\sum_{t=1}^{|Y_i|} \log P(y_t \mid x_{\text{in}}, y_{<t}; \theta) \]

This modular decoupling enables zero-shot migration across different base LLMs (requiring only a lightweight UBA fine-tuning step) and supports privacy-preserving edge-cloud deployments where historical quantization is computed locally on-device.

Loss & Training

In Stage 1 (PCC), 100k behavior samples are used to initialize \(L=4\) codebooks with \(K=1,000\) entries each. The loss hyperparameters are set to \(\alpha=1.0, \beta=0.1, \gamma=0.3\) with an AdamW optimizer (batch size 1,024). In Stage 2 (UBA), full user histories are aligned on 8 NVIDIA RTX 4090 GPUs using DeepSpeed ZeRO Stage 2. The learning rate is \(1 \times 10^{-4}\) with a weight decay of 0.01, gradient clipping at 1.0, gradient accumulation of 4 steps, and training runs for 3 epochs.

Key Experimental Results

Main Results

CURP is evaluated across six diverse tasks on the LaMP benchmark (covering classification, regression, and open-ended text generation), compared against Zero-Shot (ZS), In-Context Learning (ICL), retrieval baselines (BM25, Dense), reasoning prompts (CoT, PAG), and parameter-based/soft-prompt methods (DiffMean, PROPER, Soft Prompt, PPlug). The main comparison results are summarized below:

Task Metric Zero-Shot ICL BM25 Dense PROPER PPlug CURP (Ours)
LaMP-1 (Citation Identification) Acc ↑ 0.514 0.557 0.579 0.555 - 0.502 0.518
LaMP-2 (Movie Tagging) Acc ↑ 0.345 0.445 0.431 0.448 0.587 0.565 0.629
F1 ↑ 0.191 0.291 0.281 0.300 0.504 0.485 0.547
LaMP-3 (Product Rating) MAE ↓ 0.502 0.368 0.314 0.284 0.284 0.258 0.276
RMSE ↓ 0.839 0.714 0.651 0.604 0.597 0.567 0.585
LaMP-4 (Headline Generation) R-1 ↑ 0.138 0.160 0.168 0.177 0.191 0.190 0.194
R-L ↑ 0.119 0.140 0.151 0.159 0.171 0.168 0.173
BERTScore (Sim) ↑ 0.845 0.852 0.854 0.855 0.858 0.857 0.859
LaMP-5 (Title Generation) R-1 ↑ 0.411 0.432 0.440 0.435 0.469 0.500 0.505
R-L ↑ 0.338 0.360 0.371 0.364 0.399 0.444 0.449
BERTScore (Sim) ↑ 0.876 0.884 0.885 0.884 0.890 0.897 0.898
LaMP-7 (Tweet Paraphrasing) R-1 ↑ 0.457 0.428 0.446 0.447 - 0.536 0.543
R-L ↑ 0.401 0.375 0.390 0.392 - 0.477 0.483
BERTScore (Sim) ↑ 0.903 0.894 0.898 0.898 - 0.909 0.911

Ablation Study

The contribution of each architectural module is ablated on LaMP-2 (classification), LaMP-3 (regression), LaMP-4 (generation), and LaMP-7 (generation):

Configuration LaMP-2 Acc ↑ LaMP-2 F1 ↑ LaMP-3 MAE ↓ LaMP-3 RMSE ↓ LaMP-4 R-1 ↑ LaMP-4 R-L ↑ LaMP-7 R-1 ↑ LaMP-7 R-L ↑
CURP (Full model) 0.629 0.547 0.276 0.585 0.194 0.173 0.543 0.483
w/o Quantize (continuous) 0.634 0.548 0.275 0.601 0.198 0.177 0.544 0.483
w/o Dynamic (no dynamic branch) 0.474 0.367 0.290 0.596 0.191 0.170 0.538 0.480
w/o Stable (no stable branch) 0.539 0.436 0.287 0.607 0.188 0.168 0.528 0.469
w/o Q-Former (mean pooling) 0.590 0.541 0.290 0.620 0.192 0.171 0.536 0.477
Dual MLP (separate projection) 0.604 0.558 0.282 0.605 0.194 0.175 0.545 0.485

Key Findings

  • Complementary Dual Representations: Removing either the dynamic or the stable branch induces severe performance degradation. Omitting the dynamic branch causes a 15.5% drop in accuracy on LaMP-2 (0.629 down to 0.474), while dropping the stable branch decreases accuracy by 9.0%, proving that both query-conditioned alignment and persistent long-term prior are indispensable.
  • Robustness from Discrete Quantization: While continuous unquantized embeddings (w/o Quantize) marginally edge out default PQ on clean data (e.g., +0.004 R-1 on LaMP-4), they are highly vulnerable to profile paraphrasing (dropping 2.7% under profile perturbation, whereas CURP exhibits ~0% drop). Moreover, substituting PQ with stronger quantizers like Residual Quantization (RQ) or Optimized PQ (OPQ) pushes LaMP-2 accuracy to 0.655 and 0.649, outperforming the continuous variant on clean inputs.
  • High Efficiency and Scalability: In contrast to PROPER which requires tuning 8M parameters per individual user (totaling ~57 min training and ~1056 min inference on LaMP-4), CURP requires only 28M shared parameters and completes inference in 4 minutes and 28 seconds—achieving a two-order-of-magnitude speedup.

Highlights & Insights

  • Sociological Grounding via Vector Quantization: Elegantly translates Self-Categorization Theory into machine learning practice by framing user modeling as discrete composition over learnable orthogonal prototype codebooks, balancing cross-user statistical sharing with fine-grained individual specificity.
  • Interpretable Semantic Factorization: PQ subspaces autonomously specialize into interpretable, hierarchical semantic roles without explicit supervision. For instance, in political headline generation, Subspaces 2 and 3 act as invariant topic anchors (capturing Trump immigration executive order content), Subspace 1 differentiates institutional sources (legal vs. media), and Subspace 4 captures framing stances (court proceedings vs. personal human impact).
  • Model-Agnostic Portability: The codebook and user representations are decoupled from the generator LLM. Switching between Contriever/RoBERTa encoders and Qwen/LLaMA decoders requires only retraining the lightweight UBA module, making it practical for real-world enterprise deployment.

Limitations & Future Work

  • Reliance on Behavioral History: LaMP captures user personas purely via behavioral text logs, lacking explicit demographic attributes, psychometric profiles, or rich cross-domain behavioral streams.
  • Information Loss in Rare Tail Patterns: Vector quantization intrinsically discards surface linguistic traits. Idiosyncratic token-level habits or niche preferences situated far from all codebook centroids cannot be easily reconstructed.
  • Metric-Preference Decoupling: Surface overlap metrics (ROUGE / BERTScore) penalize genuine personalization when a user's consistent personal habit (e.g., always authoring numbered listicle headlines) happens to differ from a single ground-truth reference headline. More expressive evaluation paradigms (such as LLM judges and human blind studies) are necessary.
  • vs. Prompt-based Methods (ICL / BM25 / PAG): Prompt concatenation exposes the generator to surface token noise and blows up prompt length. CURP compresses histories into discrete prototype embeddings, providing superior noise robustness and inference throughput.
  • vs. Parameter-based Methods (PROPER / DiffMean): PROPER fits dedicated LoRA parameters per user, creating linear storage bloat with user base growth. CURP keeps the LLM frozen and shares 28M parameters across all users.
  • vs. Single-Token Soft Prompts (PPlug): PPlug relies solely on query-weighted attention over raw histories into a single token; when no relevant history matches the query, it tends to hallucinate meaningless tokens (e.g., repeating social media @handles). CURP's dual-aggregator utilizes the Q-Former stable branch as a structural prior, preventing degenerate outputs during out-of-distribution queries.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers the use of Self-Categorization Theory to structure personalized generation via subspace prototype codebooks and dual-branch aggregation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluation across six LaMP tasks, comprehensive ablations, quantizer replacements, semantic interpretability case studies, and noise-perturbation robustness tests]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, coherent narrative, well-structured figures and insightful case analyses]
  • Value: ⭐⭐⭐⭐⭐ [Provides a scalable, parameter-efficient, and noise-resistant foundation for real-world personalized LLM deployment without per-user model sprawl]