Skip to content

MoF: Preference-Aware Mixture Modeling for Black-Box LLM Personalization

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/hoonpak/Mixture-of-Facets
Area: NLP Understanding
Keywords: Black-Box LLM Personalization, Mixture of Experts, Latent Preference Facets, Sparse Autoencoders, Cluster-Based Sampling

TL;DR

Addressing the parameter explosion and cold-start adaptation bottlenecks of user-specific scoring heads in black-box LLM personalization, MoF models user preferences as adaptive sparse compositions over shared latent facet heads guided by a history-conditioned sparse autoencoder (SAE) router and cluster-based sampling (CBS), achieving constant \(O(1)\) personalization parameters and strong generalization to unseen users without fine-tuning.

Background & Motivation

Large language models (LLMs) have achieved remarkable capabilities across a broad spectrum of general language understanding and complex reasoning tasks, becoming core backbones for personalized conversational assistants. Nevertheless, contemporary proprietary LLMs are predominantly pre-trained under a one-size-fits-all paradigm via next-token prediction and further aligned toward average human preferences during post-training. This training paradigm inherently biases models toward dominant, generic response styles, creating a fundamental obstacle when attempting to tailor responses to the nuanced, heterogeneous, and subjective preferences of individual users.

In black-box deployment scenarios where proprietary model parameters cannot be directly modified via gradient descent, early personalization approaches relied primarily on in-context prompting or retrieval-augmented generation (RAG). However, these prompt-based techniques suffer from context window limitations and are notoriously vulnerable to distracting or irrelevant context retrieved from user histories. To deliver more robust personalization, recent state-of-the-art frameworks like HYDRA established a two-stage "reranker and adapter" paradigm, wherein useful historical items are reranked to form a personalized prompt and candidate responses generated by the black-box LLM are rescored. Yet, HYDRA couples its architecture to dedicated, user-specific scoring heads. Consequently, the volume of personalized parameters scales linearly (\(O(N)\)) with the user population \(N\), requiring an expensive and non-scalable secondary fitting stage for every newly registered or unseen user before the system can serve personalized responses.

The core tension lies in representing highly disparate, dynamic user preferences without incurring unbounded parameter growth or requiring per-user gradient updates. This paper tackles the challenge by recognizing that human preferences across a population are not entirely disjoint but rather multifaceted combinations of common underlying dimensions. Core idea: decompose individual user preferences into weighted compositions of shared latent preference facets via a history-conditioned sparse autoencoder router and cluster-based sampling, achieving constant \(O(1)\) parameter complexity and zero-overhead generalization to unseen users in black-box LLM personalization.

Method

Overall Architecture

MoF is instantiated across both the reranker and adapter stages within a black-box personalization pipeline. Given a user \(u\), their interaction history \(H_u = \{(q_1^u, y_1^u), \dots, (q_T^u, y_T^u)\}\), and an incoming query \(q_u\), the reranker first selects the top-\(k\) most useful historical items to assemble a personalized context \(H_u^*\). Conditioned on this context, the black-box LLM generates a set of candidate responses \(O_u = \{o_1^u, \dots, o_B^u\}\). The adapter then scores all candidates according to user preference alignment, outputting the highest-scoring candidate as the final prediction \(\hat{y}_u\).

At the heart of both scoring stages, MoF replaces per-user dedicated heads with a unified mixture-of-facets architecture. A shared text encoder (BERT) extracts the \([CLS]\) representation of an input pair (query-history item for the reranker, or query-candidate for the adapter). This embedding is evaluated in parallel by \(F\) shared lightweight facet heads. Concurrently, a preference-aware router processes the user's interaction history: historical embeddings are aggregated through a Transformer encoder, passed through a Sparse Autoencoder (SAE) to isolate disentangled preference features, and projected via top-\(k\) sparse gating into routing weights over the facet heads. During training, Cluster-Based Sampling (CBS) partitions the user's history into semantic clusters to draw diverse training samples, driving the shared facet heads to specialize into distinct preference attributes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Query & Interaction History<br/>Query q + History H"] --> B["Cluster-Based Training Construction<br/>Cluster-Based Sampling"]
    B --> C["Shared Facet Scoring Matrix<br/>Shared Facet Heads"]
    A --> D["Sparse Preference Feature Extraction<br/>Sparse User Representation (SAE)"]
    D --> E["Preference-Aware Sparse Routing<br/>Preference-Aware Router (Top-k)"]
    C --> F["Convex Mixture Aggregation<br/>Mixture-of-Facets Prediction"]
    E --> F
    F --> G["Personalized Selection & Output Scoring<br/>Reranker Selection / Adapter Scoring"]

Key Designs

1. Shared Facet Heads: Decoupling Parameter Growth from User Population via Shared Latent Subspaces

Prior black-box personalization methods instantiate an isolated scoring head for each user, leading to linear storage growth and cold-start failures. MoF discards user-specific heads in favor of a fixed pool of \(F\) shared facet heads \(\{\psi_f\}_{f=1}^F\). Each head is implemented as a lightweight two-layer MLP to maintain computational efficiency while isolating the compositional effects. Given input pair \(x^R=(q_u, h_{u,m}')\) or \(x^A=(q_u, o_b^u)\), the shared base encoder produces the \([CLS]\) representation \(z_x = \delta(x)_{[CLS]} \in \mathbb{R}^d\). Each facet head independently produces a facet-specific logit: $$ \psi_f(z_x) = W_f^{(2)} \tanh\left(W_f^{(1)} z_x + b_f^{(1)}\right) + b_f^{(2)} $$ where weights \(W_f^{(1)} \in \mathbb{R}^{d \times d}\) and \(W_f^{(2)} \in \mathbb{R}^{o \times d}\) are shared globally across all users. Stacking all head outputs yields the facet logit matrix \(\Psi(x) = [\psi_1(z_x), \dots, \psi_F(z_x)] \in \mathbb{R}^{F \times o}\). Because \(F\) is constant, total trainable personalization parameters remain strictly \(O(1)\) regardless of user count \(N\).

2. Sparse User Representation: Disentangling Persistent Preferences from Contextual Noise via Sparse Autoencoders

The router requires a robust user preference representation extracted from the historical interaction set \(H_u\). However, raw historical texts contain dominant semantic and syntactic information that easily drowns out subtle user preference tendencies. MoF first embeds individual history items as \(e_t^u = \delta(h_t^u)_{[CLS]}\), aggregates them via a sequence Transformer encoder, and computes dense representation \(e_u' = \text{Mean}(\text{Encoder}(E_u))\). Crucially, query representations are excluded from this step to ensure routing decisions reflect persistent preferences rather than query semantics. To isolate true preference factors, an expansion Sparse Autoencoder (SAE) with latent expansion factor \(\tau=2\) projects \(e_u'\) into a higher-dimensional sparse space: $$ e_{\text{enc}}^u = \text{ReLU}\left(W_{\text{enc}} e_u' + b_{\text{enc}}\right), \quad e_{\text{dec}}^u = W_{\text{dec}} e_{\text{enc}}^u + b_{\text{dec}} $$ Trained via reconstruction loss and \(L_1\) penalty: \(\mathcal{L}_{\text{SAE}} = \|e_{\text{dec}}^u - e_u'\|_2^2 + \lambda_1 \|e_{\text{enc}}^u\|_1\), the SAE forces preference-relevant activations to be sparsely localized, providing a clean, noise-free input representation \(e_{\text{enc}}^u\) for the router.

3. Preference-Aware Sparse Router: History-Conditioned Top-k Routing for Plug-and-Play Generalization

To determine which latent facets govern a specific user's choices, the router maps the sparse user representation \(e_{\text{enc}}^u\) through a linear projection layer into routing logits \(g_u = W_r e_{\text{enc}}^u + b_r \in \mathbb{R}^F\). To encourage sharp specialization and avoid diluting signals with irrelevant facets, top-\(k\) routing activates only the \(K\) highest logits (denoted \(\tilde{g}_u \in \mathbb{R}^K\) with indices \(K_u\)), normalized through a Softmax function: $$ \alpha_{u,k} = \frac{\exp(\tilde{g}{u,k})}{\sum}^K \exp(\tilde{g{u,j})}, \quad \text{MoF}(x, H_u) = \sum(x) $$ Because routing weights are computed purely via forward evaluation on the user's past history, MoF completely eliminates the need for per-user parameter fitting. For unseen users entering the system, the model instantly produces personalized routing weights, enabling seamless zero-shot generalization.}^K \alpha_{u,k} \psi_{K_{u,k}

4. Cluster-Based Sampling: Inducing Facet Specialization by Exposing Diverse Preference Clusters

When training rerankers and adapters, standard instance generation typically retrieves history items based on lexical or embedding similarity to the query. This causes training instances to concentrate on narrow semantic regions, causing multiple facet heads to collapse into identical behaviors. MoF introduces Cluster-Based Sampling (CBS) to explicitly drive facet specialization. Specifically, the user's history embedding set \(E_u\) is clustered into distinct groups using \(k\)-means clustering. When forming training pairs for the reranker and adapter, candidate items are sampled independently from each separate cluster rather than from a single global pool. This exposes the facet heads to maximally diverse preference signals across differing interaction topics, ensuring each facet specializes in distinct latent behavioral modes.

Loss & Training

The framework optimizes the MoF scoring parameters and the Sparse Autoencoder jointly. The training objective couples binary cross-entropy with the SAE regularizer: $$ \mathcal{L} = \mathcal{L}{\text{MoF}} + \lambda \mathcal{L} $$ where prediction probability }\(p_i^u = \sigma(\text{MoF}(x_i^u, H_u))\) is evaluated against supervision label \(r_i^u \in \{0, 1\}\). In the reranker, positive labels denote history items that successfully trigger preference-aligned responses. In the adapter, positive labels identify candidate responses matching ground truth (classification) or exceeding average evaluation scores (generation). Training utilizes AdamW with a learning rate of \(5.0 \times 10^{-5}\), weight decay \(0.01\), batch size \(64\), \(\lambda = 0.5\), \(\lambda_1 = 8.6 \times 10^{-4}\), \(F = 7\) facet heads, and top-\(3\) sparse routing (\(K = 3\)).

Key Experimental Results

Main Results

MoF is evaluated on four benchmarks from LaMP: LaMP-2 (Movie Tagging Classification), LaMP-3 (Product Rating Classification), LaMP-4 (News Headline Generation), and LaMP-5 (Scholarly Title Generation). The underlying black-box LLM is gpt-3.5-turbo-1106, and the base encoder is bge-base-en-v1.5 (110M parameters).

Table 1 summarizes the main comparison against prompt-based baselines and reranker/adapter methods (averaged over 3 runs with different sampled user splits):

Method LaMP-2 Acc โ†‘ LaMP-2 F1 โ†‘ LaMP-3 MAE โ†“ LaMP-3 RMSE โ†“ LaMP-4 R-1 โ†‘ LaMP-4 R-L โ†‘ LaMP-4 BLEU โ†‘ LaMP-5 R-1 โ†‘ LaMP-5 R-L โ†‘ LaMP-5 BLEU โ†‘ Avg. Rank โ†“
Zero-Shot 0.420 0.257 0.500 0.777 0.147 0.129 0.891 0.446 0.375 5.822 13.5
ICL (\(k=4\)) 0.513 0.403 0.433 0.762 0.161 0.140 1.240 0.447 0.384 5.511 10.0
RAG (\(k=4\)) 0.547 0.409 0.440 0.765 0.176 0.148 1.456 0.457 0.395 5.888 6.9
PAG (\(k=1\)) 0.473 0.337 0.400 0.739 0.181 0.157 1.040 0.454 0.371 6.063 8.0
CFRAG (\(k=4\)) 0.593 0.447 0.440 0.780 0.176 0.151 0.966 0.456 0.392 5.748 7.4
HYDRA 0.593 0.417 0.393 0.761 0.183 0.163 1.849 0.459 0.396 6.652 3.5
MoF-Reranker 0.600 0.451 0.407 0.724 0.177 0.154 1.502 0.456 0.390 7.421 4.2
MoF-Adapter 0.467 0.308 0.327 0.682 0.161 0.142 1.275 0.477 0.400 6.515 6.5
MoF (Full) 0.627 0.438 0.300 0.605 0.199 0.176 1.662 0.481 0.413 6.967 1.4

When evaluated with gpt-4o-mini-2024-07-18 as the black-box LLM, MoF consistently maintains its lead with an overall average rank of 3.6 (outperforming HYDRA at 5.0 and RAG at 4.4), proving model-agnostic robustness.

Ablation Study

The contribution of each component within MoF is rigorously analyzed across LaMP-2, LaMP-3, and LaMP-4:

Variant LaMP-2 Acc โ†‘ LaMP-3 MAE โ†“ LaMP-4 R-1 โ†‘ LaMP-4 R-L โ†‘ Analysis
MoF (Full) 0.627 0.300 0.199 0.176 Full model with shared facets, SAE, top-k router, and CBS
w/o CBS 0.587 0.380 0.183 0.168 Most severe drop, proving cluster diversity is required for facet separation
w/o Router 0.600 0.360 0.184 0.161 Uniform routing allocation fails to capture distinct user tendencies
w/o Multi-Facet 0.607 0.353 0.183 0.160 Single head cannot capture multi-dimensional user preference nuances
w/o SAE 0.607 0.333 0.188 0.165 Dense history representations introduce semantic noise into routing logits

In unseen user generalization (50 test users): - HYDRA cannot function without its fit stage (Fit โœ— is inapplicable; Fit โœ“ yields Acc 0.593, MAE 0.393, R-1 0.183). - MoF without any fit stage (Fit โœ—) scores Acc 0.620, MAE 0.333, and R-1 0.200, cleanly outperforming HYDRA with full user-specific fitting. - In a large-scale test on 1,000 unseen users without adaptation, MoF achieves Acc 0.577, MAE 0.354, and R-1 0.175, decisively surpassing RAG (\(k=4\): Acc 0.508, MAE 0.449, R-1 0.164).

Key Findings

  • CBS is a synergistic necessity for mixture-of-facets: Ablation confirms that removing CBS triggers the steepest drop across tasks (Acc dropping from 0.627 to 0.587). In contrast, applying CBS to HYDRA degrades performance (Acc drops from 0.593 to 0.560), verifying that CBS is not a generic data augmentation heuristic, but specifically prevents representation collapse in shared-facet mixture models.
  • Constant storage scaling (\(O(1)\)) with zero per-user overhead: Whereas HYDRA incurs roughly 2.26 MB per user in dedicated parameter weights, MoF requires zero additional bytes per user, eliminating the operational complexity of hosting millions of user-specific adapter weights.
  • Optimal facet capacity: Sensitivity analysis demonstrates that performance peaks around \(F=6 \sim 8\) facet heads with top-\(2 \sim 3\) sparse activation. Beyond this range, activating excessive heads introduces redundant signals and degrades performance.

Highlights & Insights

  • Reconceptualizing Personalization as Latent Mixture Modeling: Overturns the dogma that black-box personalization requires user-specific parameters, proving that diverse human preferences can be faithfully approximated as convex combinations of shared, latent preference dimensions.
  • Sparse Autoencoders as Representation Filters: Demonstrates that projecting user sequence histories through an SAE effectively decouples persistent preference characteristics from dominant contextual semantics without needing query supervision.
  • Cluster-Based Sampling to Prevent Expert Collapse: Provides a concrete algorithmic solution for training MoE-style preference models, using feature clustering to ensure balanced gradient exposure across all expert heads.

Limitations & Future Work

  • Constrained Benchmark Scope: Evaluation is focused on the standard LaMP benchmark (LaMP-2 to LaMP-5); long-form multi-turn conversational personalization (e.g., LongLaMP) remains to be explored.
  • Fixed Facet Capacity: The current formulation utilizes a static number of shared heads (\(F=7\)). Future research could investigate dynamic facet allocation mechanisms that adapt model capacity to growing population heterogeneity.
  • Training User Population Scale: While inference scalability is validated on 1,000 unseen users, training was conducted on 100 users; behavior under massive, highly heterogeneous training populations deserves further study.
  • vs HYDRA (Zhuang et al., NeurIPS 2024): HYDRA pioneered the reranker-adapter pipeline for black-box LLMs but relies on user-specific scoring heads, causing linear parameter scaling and mandatory per-user fitting. MoF replaces individual heads with shared facets and dynamic routing, achieving constant \(O(1)\) parameter complexity and superior zero-shot performance on unseen users.
  • vs White-Box MoE/PEFT (PROPER / PER-PCS): PROPER and PER-PCS compose LoRA modules inside the LLM backbone, requiring white-box access to model weights and gradients. MoF operates strictly externally, making it readily deployable for commercial proprietary LLM APIs.
  • vs Prompt-Based Methods (LaMP / PAG / CFRAG): Prompt-based methods feed retrieved histories or user summaries directly into LLM prompts, leading to context bloat and vulnerability to irrelevant noise. MoF applies structured dual-stage selection, delivering significantly more consistent improvements across diverse tasks.

Rating

  • Novelty: โญโญโญโญโ˜† Introduces MoE principles and sparse autoencoders to black-box personalization, replacing dedicated heads with shared latent facets.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage across four LaMP tasks, two proprietary LLMs, 1,000-user cold-start tests, and extensive ablations.
  • Writing Quality: โญโญโญโญโญ Crisp mathematical formulations, strong motivation, clear structural comparisons, and cohesive narrative flow.
  • Value: โญโญโญโญโญ Solves the practical deployment bottleneck of parameter proliferation and cold-start adaptation for commercial LLM personalization.