Skip to content

UEmbed: Unified Sparse and Dense Multimodal Embeddings

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Learned Sparse Retrieval, Dense Retrieval, Multimodal Embeddings, Decoder-Only MLLM, Vocabulary Partitioning

TL;DR

UEmbed natively unifies dense vectors and inverted-index-compatible sparse lexical representations within a single causal forward pass of a decoder-only Multimodal Large Language Model (MLLM) by appending \(N\) learnable special tokens mapped to \(k\)-means-clustered disjoint vocabulary subsets.

Background & Motivation

Learned Sparse Retrieval (LSR), exemplified by the SPLADE family, underpins modern large-scale search infrastructure and retrieval-augmented generation (RAG) by learning context-aware term weights and expanding queries and documents into rich vocabulary spaces. Compared with traditional term-matching approaches such as BM25, LSR successfully bridges lexical gaps while retaining the operational efficiency, interpretability, and inverted-index compatibility of sparse representations. However, existing LSR methods remain structurally anchored to bidirectional encoder architectures (e.g., BERT), relying intrinsically on sequence-wide token-level linear projections followed by global max-pooling and ReLU saturating activations. As modern foundational AI overwhelmingly converges toward autoregressive decoder-only models, the unidirectional causal attention mask fundamentally prevents earlier tokens from accessing future context, rendering classical bidirectional pooling schemes inoperative.

Prior attempts to adapt decoder-style Large Language Models (LLMs) to learned sparse retrieval suffer from critical limitations. Researchers have predominantly either converted causal backbones into bidirectional architectures through expensive architectural modifications or adopted intricate multi-stage training curricula. These interventions not only impose heavy pre-training and distillation overheads but also break out-of-the-box compatibility with highly optimized causal inference serving engines, such as vLLM and its PagedAttention mechanisms. Furthermore, naively relying on a single pooling vector (such as an EOS or CLS token) to project global representations onto massive modern LLM vocabularies exceeding 150k to 250k tokens introduces a severe representational bottleneck, while extending sparse retrieval to non-text modalities has historically required clumsy auxiliary cross-modal adapters that do not scale across diverse multimodal inputs.

This paper's angle of attack exploits the intrinsic property of causal self-attention: because tokens placed at the end of a sequence attend autoregressively to all preceding content, appending a small set of specialized learnable tokens allows each to independently summarize the comprehensive multimodal context. By simultaneously partitioning the massive token vocabulary into semantically coherent, non-overlapping subspaces via clustering, each special token functions as a dedicated sub-vocabulary specialist. Core idea: append \(N\) learnable special tokens to a decoder-only MLLM and partition the vocabulary into \(N\) disjoint subsets via semantic \(k\)-means clustering, allowing each special token to causally summarize global context and predict sparse weights over its assigned semantic subspace in a single causal forward pass that simultaneously yields dense and sparse representations.

Method

Overall Architecture

UEmbed is instantiated upon a single decoder-only Multimodal Large Language Model (specifically the Qwen3.5 family) and supports arbitrary interleaved combinations of text, image, video, and visually-rich document inputs. The input sequence is first processed through modality-specific tokenizers and vision encoders. A standard EOS token followed by \(N=16\) learnable special tokens \(\langle s_1 \rangle, \dots, \langle s_N \rangle\) is appended at the very end of the sequence. Under standard causal attention, the EOS token aggregates comprehensive contextual information from all input tokens; its final-layer hidden state is directly extracted as the dense embedding vector \(d \in \mathbb{R}^D\). Concurrently, the subsequent \(N\) special tokens attend to the entire input sequence as well as preceding special tokens. Each token's hidden state is fed into a lightweight linear projection head responsible exclusively for its designated vocabulary partition \(V_k\), generating sparse lexical activation weights that are concatenated into the global sparse representation \(w \in \mathbb{R}^{|V|}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input Sequence<br/>Text / Image / Video / VisDoc"] --> B["Autoregressive MLLM Backbone<br/>Unidirectional Causal Attention"]
    B --> C["EOS Token Hidden State<br/>Last-pooling yields Dense Embedding d"]
    B --> D["Vocabulary Compression & Semantic Partitioning<br/>248k vocabulary normalized to 184k and K-Means clustered"]
    D --> E["Causal Multi-Token Summary & Partitioned Sparse Heads<br/>N=16 special tokens predict disjoint subspace weights"]
    E --> F["Full Sparse Vector Concatenation<br/>Directly indexes into Lucene Inverted Index w"]
    C & F --> G["Unified Joint Objective & Decoupled Temperature Optimization<br/>Dual InfoNCE with decoupled temperatures and FLOPs loss"]
    G --> H["Dual-Mode Serving & Hybrid Retrieval<br/>vLLM High-Throughput / Inverted Index / Agent Search"]

Key Designs

1. Vocabulary Compression & Semantic Partitioning: Vocabulary redundancy and dimensional explosion -> Normalization cleaning and K-means subspace clustering -> Semantically cohesive sub-vocabularies Modern open-source LLM tokenizers maintain expansive vocabularies (frequently exceeding 240,000 tokens) that contain substantial surface redundancy, including trivial case variants, accented forms, and trailing whitespace tokens. Projecting directly onto this uncurated space inflates memory footprints and causes sparse token activations to disperse haphazardly across redundant subwords. UEmbed first compresses the vocabulary through an NLTK-based pipeline encompassing accent stripping, lowercasing, and whitespace normalization, collapsing tokens sharing the identical canonical root into a single unified entry (e.g., Hello, HELLO, and héllò are merged into hello, with the maximum activation retained during inference scoring), shrinking the active vocabulary from 248,320 down to 184,016. To overcome the semantic scattering that would occur if subwords were split randomly across multiple output heads, UEmbed performs \(k\)-means clustering in the embedding space to partition \(V\) into \(N=16\) disjoint subsets \(V_1, \dots, V_N\) of approximately equal size. This geometric grouping ensures that each subset comprises semantically cohesive concepts, effectively enabling each special token to function as a focused semantic specialist.

2. Causal Multi-Token Summary & Partitioned Sparse Heads: Single-token bottleneck and causal pooling failure -> N appended special tokens with disjoint linear heads -> Seamless inverted-index sparse representation Because causal masking prohibits bidirectional max-pooling across the input sequence, compressing complex multimodal signals into sparse weights over hundreds of thousands of terms via a single terminal vector introduces an intolerable capacity bottleneck. UEmbed circumvents this by appending \(N=16\) learnable special tokens \(\langle s_1 \rangle, \dots, \langle s_N \rangle\) directly after the EOS token. By virtue of causal autoregressive attention, each special token naturally observes the complete multimodal context. Each special token \(\langle s_k \rangle\) is paired with a head-specific linear layer that predicts non-negative sparse weights exclusively over its assigned vocabulary partition \(V_k\): $\(w_t^{(k)} = \log\left(1 + \text{ReLU}\left(W_t^{(k)\top} h_{s_k} + b_t^{(k)}\right)\right), \quad \forall t \in V_k\)$ where \(h_{s_k}\) denotes the final hidden state of \(\langle s_k \rangle\), and \(W_t^{(k)}, b_t^{(k)}\) represent the head-specific weight vector and bias for term \(t \in V_k\). The comprehensive sparse lexical representation is formed by direct concatenation across all \(N\) partitioned subsets: $\(w = \bigoplus_{k=1}^N \left(w_t^{(k)}\right)_{t \in V_k} \in \mathbb{R}^{|V|}\)$ This partitioned formulation completely removes the need for bidirectional max-pooling over sequence lengths, enabling execution within standard autoregressive inference runtimes. Crucially, when an application requires only dense embeddings, the \(N\) special tokens can simply be omitted during generation, incurring zero computational penalty.

3. Unified Joint Objective & Decoupled Temperature Optimization: Objective conflict and scale divergence -> Joint InfoNCE with decoupled temperature scaling and FLOPs regularization -> Zero-interference dual-mode capability Training dense and sparse retrieval representations jointly within a shared neural backbone commonly triggers multi-task negative transfer and numerical instability. Dense embeddings compute cosine similarity constrained within \([-1, 1]\), whereas sparse embeddings rely on high-dimensional inner products whose unnormalized magnitudes can span several orders of magnitude. Forcing both modalities to share a single softmax temperature results in severe over-sharpening of the sparse distribution and gradient saturation. UEmbed unifies the training objective while completely decoupling the temperature hyperparameters: $\(\mathcal{L} = \mathcal{L}_{\text{dense}}^{\text{InfoNCE}} + \lambda \mathcal{L}_{\text{sparse}}^{\text{InfoNCE}} + \alpha_q \mathcal{L}_{q}^{\text{FLOPS}} + \alpha_d \mathcal{L}_{d}^{\text{FLOPS}}\)$ The dense contrastive loss employs \(\tau=0.03\), whereas the sparse contrastive loss sets a significantly broader temperature \(\tau_s=32\), with \(\lambda=1.0\) balancing the two components equally. To ensure search efficiency on inverted indices and avoid dense vocabulary activations, FLOPs regularization terms penalize the average squared \(L_1\) norm across queries (\(\alpha_q\)) and documents (\(\alpha_d\)) using a linear ramp-up over the initial 200 optimization steps reaching \(1 \times 10^{-4}\). Furthermore, to mitigate the acute shortage of hard negatives in existing multimodal retrieval datasets, Qwen3-VL-Embedding-8B is deployed as an offline teacher model to mine top-\(k\) non-relevant hard negatives for all training queries.

Loss & Training

UEmbed is initialized from pretrained Qwen3.5 backbones across three parameter scales: 2B, 4B, and 9B. The visual encoder is frozen throughout training, while LoRA is applied to the self-attention and MLP projection layers (q_proj, k_proj, v_proj, up_proj, down_proj, gate_proj). The model is trained on a curated mixture of 3.94 million publicly available query-document pairs spanning text (39.3%), image (27.1%), video (21.4%), and visually-rich documents (12.2%) sourced from Echo-Embeddings, MLDR, ViDoRe, VisRAG, and MMEB. Training is executed in bf16 mixed precision using DeepSpeed ZeRO and gradient checkpointing with a cosine learning rate schedule (peak rate \(3 \times 10^{-5}\), warmup ratio 0.1) for 1 epoch. The 2B and 4B models are trained on 16\(\times\)A100 GPUs (per-device batch size 16), while the 9B model uses 32\(\times\)A100 GPUs (per-device batch size 8), maintaining an identical global batch size of 256. Crucially, empirical analysis reveals that including text retrieval pairs acts as a vital regularizer: while multimodal-only sparse training requires ~500 steps before separating positive and negative margins, incorporating text data enables stable, rapid convergence within 100 optimization steps.

Key Experimental Results

Main Results

UEmbed is evaluated across 78 datasets on the massive multimodal embedding benchmark MMEB-v2 and 9 diverse text retrieval datasets from BEIR (evaluated with nDCG@10). Performance is benchmarked against leading open-source and proprietary multimodal dense retrievers as well as competitive sparse baselines.

Model Size / Mode Image (36) Video (18) VisDoc (24) MMEB-v2 All (78) BEIR (9) nDCG@10
Qwen3-VL-Embedding-8B 8B Dense 80.1 67.2 82.4 77.8 55.5
Qwen3-VL-Embedding-2B 2B Dense 75.0 61.9 79.2 73.2 53.8
RzenEmbed-V2-7B 7B Dense 75.9 55.7 75.5 71.1 -
Embed-RL-4B 4B Dense 71.2 53.0 74.7 68.1 -
Ops-MM-Embed-7B 7B Dense 72.7 53.8 68.7 67.1 -
UniME-7B 7B Dense 71.2 47.5 65.7 64.1 -
SPLADE-v3 110M Sparse - - - - 50.5
Echo-Mistral-SPLADE 7B Sparse - - - - 55.2
UEmbed-2B (dense) 2B Dense 67.8 50.0 77.0 66.5 53.6
UEmbed-2B (sparse) 2B Sparse 67.0 47.7 76.7 65.5 52.3
UEmbed-4B (dense) 4B Dense 71.4 57.0 78.8 70.4 56.0
UEmbed-4B (sparse) 4B Sparse 70.6 56.0 78.6 69.7 53.6
UEmbed-9B (dense) 9B Dense 73.2 59.0 79.2 71.8 56.3
UEmbed-9B (sparse) 9B Sparse 72.5 57.5 79.1 71.0 55.2

Ablation Study

On the image subsets of MMEB-v1 using the Qwen3.5-2B backbone trained on a 500k-instance subset, UEmbed is ablated against the standard bidirectional SPLADE formulation, alternative partitioning schemes, and varying head configurations.

Configuration / Variant CLS QA RET GRD Average Key Insight
SPLADE Bidirectional Baseline (Eq. 2) 55.8 57.1 62.9 80.9 61.3 Converted to bidirectional attention + Max-pooling
UEmbed-2B Dense Mode 55.5 65.3 65.1 83.3 64.5 +3.2 points over bidirectional baseline (+8.2 on QA)
UEmbed-2B Sparse Mode 54.7 63.1 64.8 81.8 63.4 +2.1 points over bidirectional baseline (+6.0 on QA)
Vocabulary Partition: Random 55.0 62.3 64.2 81.0 63.0 Random uniform assignment across 16 subsets
Vocabulary Partition: Max-distance 55.1 62.4 64.9 79.8 63.2 Maximally distant \(k\)-means clusters paired
Vocabulary Partition: Semantic (Ours) 54.7 63.1 64.8 81.8 63.4 Coherent \(k\)-means clusters assigned to special tokens
Special Tokens: \(N=2\) 56.0 62.9 65.0 81.2 63.7 Massive sub-vocabulary per token causes capacity strain
Special Tokens: \(N=16\) (Default) 54.7 63.1 64.8 81.8 63.4 Optimal balance between expressiveness and efficiency
Special Tokens: \(N=32\) 54.6 62.8 65.0 80.3 63.2 Fragments vocabulary subsets; increases sequence length
Single-Mode: Dense-Only (\(\lambda=0\)) 55.3 65.4 65.3 83.5 64.6 Matches jointly trained UEmbed dense score
Single-Mode: Sparse-Only 54.1 63.6 64.2 79.8 63.0 Matches jointly trained UEmbed sparse score

In the downstream agentic search benchmark BrowseComp-Plus (utilizing DeepResearch-30A3B as the iterative reasoning engine), UEmbed's sparse mode demonstrates exceptional operational efficiency. At the 2B scale, UEmbed sparse achieves 57.04% recall in an average of 32.67 search rounds, outperforming the dense mode (53.47% recall across 39.19 rounds) while slashing calibration error from 16.69% to 8.54%. At 9B scale, sparse mode delivers 64.33% recall in 31.05 rounds versus 64.72% in 33.68 rounds for dense mode, substantially curtailing expensive tool-calling latency in agentic workflows.

Key Findings

  • Near-Zero Parity Gap Between Modalities: Across all evaluated scales on MMEB-v2, the performance gap between UEmbed's sparse and dense modes is contained within 1.0 point (e.g., 71.8 dense vs. 71.0 sparse on 9B; 70.4 vs. 69.7 on 4B). On visually-rich document (VisDoc) retrieval, sparse representations virtually equal dense ones (79.1 vs. 79.2 on 9B), highlighting that document images with explicit lexical content are exceptionally receptive to inverted-index sparse retrieval.
  • Autoregressive Causal Formulation Outperforms Bidirectional Pooling: Contrary to the longstanding belief that bidirectional attention is indispensable for token-level sparse retrieval, UEmbed-2B outperforms the bidirectional SPLADE baseline by +3.2 points in dense and +2.1 points in sparse retrieval on MMEB-v1. The advantage is most pronounced on QA tasks (+8.2 and +6.0 points), proving that terminal causal aggregation effectively leverages the LLM's autoregressive reasoning capabilities.
  • Absence of Negative Multi-Task Transfer: Joint dense-sparse training introduces zero performance penalty relative to single-mode specialists (64.5 vs. 64.6 dense, 63.4 vs. 63.0 sparse), demonstrating that unified training successfully harmonizes dense and sparse objectives.

Highlights & Insights

  • Causal Decoupling via Vocabulary Partitioning: Instead of forcing a single token to represent all concepts or resorting to non-causal pooling, partitioning the vocabulary into \(N\) semantic clusters and assigning them to appended causal tokens circumvents the representational bottleneck without altering decoder architectures.
  • Zero-Cost Hybrid Scoring: Because dense and sparse embeddings are produced simultaneously in a single forward pass, hybrid scoring (\(s_{\text{hybrid}} = \alpha s_{\text{dense}} + \beta s_{\text{sparse}}\)) can be performed with zero additional model inference overhead, yielding reliable improvements on text (+0.3) and visual documents (+0.5).
  • Efficiency Gains in Agentic RAG: In iterative search environments where reasoning agents generate terse, keyword-dense queries, UEmbed's sparse mode delivers equivalent or superior recall with 15–20% fewer search rounds and lower calibration error, significantly cutting tool-calling costs.

Limitations & Future Work

  • Author-Acknowledged Limitations: The large vocabulary of modern LLMs occasionally leads to anomalous token activations (e.g., non-standard subwords or artifacts such as _alt); the model exhibits language and cultural skew toward English and Chinese due to the training data composition; and a more noticeable performance gap persists in the video domain due to the difficulty of compressing complex spatiotemporal dynamics into static vocabulary weights.
  • Potential Improvement Directions: Future research could explore targeted vocabulary pruning to suppress spurious subwords, integrate temporal timestamp tokens for frame-grounded video sparse indexing, and expand the pre-training mixture to cover broader multilingual corpora.
  • vs. SPLADE (Formal et al., 2021/2022) & SPLADE-v3: SPLADE mandates bidirectional BERT encoders and global token max-pooling, restricting it to text and rendering it incompatible with modern decoder LLMs. UEmbed establishes a causal multi-token projection framework that unlocks native sparse retrieval on autoregressive MLLMs.
  • vs. Qwen3-VL-Embedding & RzenEmbed: These leading multimodal embedding models produce exclusively dense vectors, incurring high memory and search costs when scaling to massive corpora and struggling on exact keyword queries. UEmbed matches open-data dense baselines while providing inverted-index-ready sparse representations from the same checkpoint.
  • vs. STAIR (Chen et al., 2023) & Multimodal LSR: Earlier multimodal sparse models required specialized cross-modal fusion modules or auxiliary token prediction heads. UEmbed unifies text, image, video, and document inputs within a single causal transformer without auxiliary networks.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ (Pioneering framework for native sparse and dense multimodal retrieval directly within a single causal decoder-only MLLM)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive evaluation across MMEB-v2 78 datasets, BEIR, BrowseComp-Plus, and detailed ablations across 2B/4B/9B scales)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Exceptionally well-structured paper with crisp problem formulation, intuitive diagrams, and clear ablation analysis)
  • Value: ⭐⭐⭐⭐⭐ (Seamlessly aligns with modern vLLM serving and Lucene inverted indices, unlocking highly scalable hybrid retrieval and cost-effective agentic search)