Jasper-Flash: Elastic Token Compression and Progressive Distillation for Inference-Scalable Text Embedding Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/infgrad/Jasper-Token-Compression-600M
Area: LLM/NLP
Keywords: text embedding models, elastic token compression, progressive distillation, multi-teacher distillation, inference acceleration
TL;DR¶
Jasper-Flash proposes an inference-scalable text embedding framework integrating Elastic Token Compression (ETC) and Compression-Adaptive Progressive Distillation (CAPD), enabling dynamic sequence reduction on the fly; its 600M model slashes 1024-token encoding latency from 24.24 ms to 4.48 ms (a 5.4x speedup) while achieving an MTEB score of 74.75, outperforming substantially larger 4B models.
Background & Motivation¶
With the widespread adoption of dense text retrieval, semantic search, and retrieval-augmented generation (RAG), text embedding models have become critical infrastructure across modern NLP systems. However, deploying top-tier embedding models in latency-critical and resource-constrained production settings faces dual operational bottlenecks: on one hand, expanding parameter scales impose heavy GPU memory burdens; on the other hand, the quadratic computational complexity \(\mathcal{O}(L^2)\) of Transformer self-attention with respect to sequence length leads to dramatic latency spikes as documents grow longer. While Matryoshka Representation Learning (MRL) has successfully tackled elasticity in vector representation dimensions to optimize downstream storage and search efficiency, the computational overhead during inference forward passes has remained rigid and unaddressed.
To mitigate sequence-length bottlenecks, existing token-level sequence reduction techniques—such as static word-vector elimination, intermediate layer pooling, or token merging (e.g., ToMe)—rely on static pruning budgets or complex dynamic clustering heuristics. Static hard truncation simply drops tail tokens, causing complete retrieval failure when critical evidence is positioned in the mid-to-late sections of long documents; conversely, similarity-based merging introduces non-trivial clustering and sorting overhead at runtime, frequently eroding real-world speedups on modern parallel hardware. Crucially, existing methods lock models into a single fixed reduction ratio after training, making it impossible to adapt sequence lengths dynamically across fluctuating inference latency budgets.
Directly coupling aggressive token compression with knowledge distillation triggers severe optimization difficulties: token compression is inherently a lossy spatial reduction, and forcing a student network to simultaneously learn semantic space alignment alongside destructive sequence reduction before establishing a stable feature foundation results in severe gradient conflict and catastrophic representational collapse. Core idea: decouple representation space alignment from sequence compression adaptation via a lightweight Elastic Token Compression (ETC) module and a four-stage Compression-Adaptive Progressive Distillation (CAPD) curriculum, first establishing full-sequence semantic grounding through heterogeneous multi-teacher fusion before progressively transitioning to fixed and dynamic compression sampling, enabling a single model to flexibly scale inference latency on the fly without performance degradation.
Method¶
Overall Architecture¶
Jasper-Flash is built around two tightly coordinated mechanisms: the lightweight Elastic Token Compression (ETC) module placed immediately after the embedding layer, and the four-stage Compression-Adaptive Progressive Distillation (CAPD) framework. Initialized from Qwen3-Embedding-0.6B, the model inserts an MLP and 1D adaptive average pooling layer before the first attention block to deterministically reduce spatial sequence dimensions early in the network. The learning procedure is structured into four progressive stages: Stage 1 full-context heterogeneous dual-teacher distillation, Stage 2 fixed-ratio compression adaptation, Stage 3 dynamic-ratio sampling generalization with semantic topology preservation, and Stage 4 contrastive retrieval task adaptation with semantic regularization.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Tokens (Lin, d)"] --> B["Length Threshold & Ratio Sampling<br/>Compute target length Ltgt"]
B --> C["Elastic Token Compression (ETC)<br/>Qwen3MLP Transformation + AdaptiveAvgPool1d"]
C --> D["Heterogeneous Multi-Teacher Distillation (Stage 1)<br/>Equal-weight fusion of Qwen3-8B & QZhou-7B"]
D --> E["Progressive Compression Curriculum (Stage 2 & 3)<br/>Fixed baseline adaptation + Dynamic sampling with topology loss"]
E --> F["Regularized Retrieval Adaptation (Stage 4)<br/>Contrastive InfoNCE + Hard Negatives + Semantic Anchoring"]
F --> G["Elastic Scalable Embedding (dim: 2048)"]
Key Designs¶
1. Elastic Token Compression (ETC): Threshold-protected lightweight pooling for zero-overhead dynamic scaling Addressing the tension between quadratic attention cost on long sequences and destructive information loss on short queries, the ETC module is positioned directly between the token embedding layer and the initial self-attention block. Input token embeddings first pass through a randomly initialized SwiGLU feed-forward network (Qwen3MLP) that refines token-level feature interactions while preserving the sequence length and hidden dimension. Next, a deterministic 1D adaptive average pooling operator (AdaptiveAvgPool1d) scales the sequence length to a target length \(L_{tgt}\). To prevent catastrophic feature dilution on short queries, ETC incorporates a length threshold \(L_{th}\) (defaulting to 80): $\(L_{tgt} = \begin{cases} L_{in}, & \text{if } L_{in} \le L_{th} \\ L_{th} + (L_{in} - L_{th}) \times \rho, & \text{if } L_{in} > L_{th} \end{cases}\)$ where \(\rho \in (0, 1]\) is the user-specified inference compression ratio. This design preserves short queries intact while scaling long passages proportionally, relying entirely on dense parallel matrix operations that execute efficiently on GPU hardware without graph-dispatch overhead.
2. Heterogeneous Multi-Teacher Distillation: Equal-weight fusion across complementary representational spaces To eliminate single-teacher inductive biases and avoid overwhelming the compact student during early training, CAPD preserves full context sequences in Stage 1 without compression. Two complementary top-performing models from the MTEB leaderboard are selected: Qwen3-Embedding-8B (strong in asymmetric retrieval, whose representation \(E_{qwen}\) is truncated to 1024 dimensions via MRL) and QZhou-Embedding-7B (strong in semantic textual similarity, whose 3072-dimensional representation is compressed to 1024 dimensions via parameter-free block-wise summation, denoted as \(E_{qzhou}\)). The normalized sub-vectors are concatenated and globally \(\ell_2\)-normalized: $\(E_t = \mathrm{Norm}\big(\mathrm{Norm}(E_{qwen}) \,\|\, \mathrm{Norm}(E_{qzhou})\big)\)$ Mathematically, the dot product of this concatenated vector equals the arithmetic mean of the cosine similarities from the individual teachers: \(U \cdot V = \frac{1}{n}\sum_{i=1}^n (u_i \cdot v_i)\). The student model replaces last-token pooling with mean pooling followed by a linear projection to 2048 dimensions, optimized using cosine distance \(\mathcal{L}_{cosine} = 1 - E_s \cdot E_t\), successfully integrating complementary retrieval and semantic similarity capabilities into a 0.6B backbone.
3. Progressive Compression Curriculum: Two-stage fixed-to-dynamic transition with semantic topology preservation To avoid training collapse caused by introducing volatile sequence scaling all at once, CAPD decouples compression adaptation into Stage 2 (fixed compression) and Stage 3 (dynamic sampling). Stage 2 trains the model on a fixed baseline ratio \(\rho_{base} = 0.33\) (\(L_{th} = 80\)) guided solely by \(\mathcal{L}_{cosine}\), establishing foundational stability in a compressed space. Stage 3 unlocks dynamic elasticity by stochastically sampling compression ratios \(r\) on a per-batch basis (with probability 0.2 from \([0.1, \rho)\), 0.4 at \(\rho\), 0.2 from \((\rho, 2\rho]\), and 0.2 from \((2\rho, 1.0]\)). To prevent relational semantic distortion under severe spatial reductions, a pairwise similarity loss is introduced: $\(\mathcal{L}_{similarity} = \mathrm{MSE}(E_s E_s^\top, E_t E_t^\top)\)$ The joint loss \(\mathcal{L}_{s3} = \lambda_1 \mathcal{L}_{cosine} + \lambda_2 \mathcal{L}_{similarity}\) forces the student model to maintain the teachers' global geometric manifold across wildly varying sequence lengths.
4. Regularized Retrieval Adaptation: Anti-forgetting semantic anchoring with hard-negative contrastive fine-tuning Because asymmetric dense passage retrieval is vulnerable to signal dilution caused by feature pooling, Stage 4 specializes the representation space for retrieval while mitigating catastrophic forgetting of general semantic structures. Dynamic compression ratio sampling continues in each batch. The model is fine-tuned using the InfoNCE contrastive loss \(\mathcal{L}_{cl}\) augmented with \(K=3\) mined hard negatives (documents ranked between 50 and 100 in the initial retrieval pool). To preserve foundational semantic stability, the cosine loss \(\mathcal{L}_{cosine}\) is retained as a semantic anchor alongside a soft KL divergence loss \(\mathcal{L}_{soft}\) against teacher distributions: $\(\mathcal{L}_{s4} = \lambda_1 \mathcal{L}_{cosine} + \lambda_3 \mathcal{L}_{soft} + \mathcal{L}_{cl}\)$ This multi-objective regularization sharpens retrieval decision boundaries across the full elastic spectrum while preventing degradation of broad classification and similarity capabilities.
Loss & Training¶
Training was executed on 4 NVIDIA RTX 4090 GPUs using FlashAttention-2 and the Adam optimizer, with a maximum sequence length of 1,030 tokens. Hyperparameters across the four stages were configured as follows: - Stage 1 (Multi-Teacher Distillation): 2 epochs on 12M balanced bilingual pairs; learning rate 1e-4, batch size 256, warmup ratio 0.005, cosine loss weight \(\lambda_1 = 10\), no compression. - Stage 2 (Fixed Compression Adaptation): 2 epochs; learning rate 7e-5, batch size 256, fixed \(\rho = 0.33, L_{th} = 80\), optimized with \(\mathcal{L}_{s2} = 10 \mathcal{L}_{cosine}\). - Stage 3 (Dynamic Sampling Generalization): 800 steps; learning rate 7e-5, batch size 512, dynamic ratio \(r \sim \mathcal{U}(0.1, 1.0)\), loss weights \(\lambda_1 = 10, \lambda_2 = 100\). - Stage 4 (Retrieval Adaptation): 5,000 steps on 2.5M curated pairs; learning rate 2e-5, batch size 64, InfoNCE temperature \(\tau=0.3\), KL temperature \(\alpha=0.1\) with weight \(\lambda_3 = 16\), and \(K=3\) hard negatives.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on both English MTEB and Chinese C-MTEB benchmarks across varying compression ratios \(\rho \in \{0.50, 0.33, 0.20, 0.10\}\). Despite having only 595M parameters, the resulting model outperformed its base architecture and surpassed substantially larger 4B baselines.
| Model | Params | Dim. | MTEB M.Task | MTEB M.Type | Clas. | Clus. | Retr. | STS | Sum. | C-MTEB M.Task |
|---|---|---|---|---|---|---|---|---|---|---|
| QZhou-Embedding (Teacher) | 7B | 3584 | 75.97 | 69.52 | 88.97 | 61.65 | 67.12 | 91.65 | 33.05 | 76.99 |
| Qwen3-Embedding-8B (Teacher) | 8B | 4096 | 75.23 | 68.71 | 90.43 | 58.57 | 69.44 | 88.58 | 34.83 | 73.84 |
| Qwen3-Embedding-4B | 4B | 2560 | 74.61 | 68.10 | 89.84 | 57.51 | 68.46 | 88.72 | 34.39 | 72.27 |
| Qwen3-Embedding-0.6B (Base) | 595M | 1024 | 70.47 | 64.72 | 84.58 | 54.05 | 61.83 | 86.57 | 33.43 | 66.33 |
| Jasper-TC-600M (\(\rho = 0.50\)) | 595M | 2048 | 74.75 | 68.46 | 90.35 | 59.44 | 66.19 | 88.79 | 33.66 | 73.51 |
| Jasper-TC-600M (\(\rho = 0.33\)) | 595M | 2048 | 74.71 | 68.43 | 90.37 | 59.33 | 66.17 | 88.71 | 33.66 | 73.51 |
| Jasper-TC-600M (\(\rho = 0.20\)) | 595M | 2048 | 74.58 | 68.33 | 90.32 | 59.22 | 65.77 | 88.75 | 33.53 | 73.38 |
| Jasper-TC-600M (\(\rho = 0.10\)) | 595M | 2048 | 74.22 | 67.93 | 90.27 | 58.80 | 64.76 | 88.77 | 32.16 | 73.17 |
Encoding latency (milliseconds) measured on an NVIDIA RTX 4090 demonstrates dramatic speedups as input lengths increase:
| Model Configuration | MTEB M.Task | 128 tokens | 256 tokens | 512 tokens | 1024 tokens | 2048 tokens |
|---|---|---|---|---|---|---|
| Qwen3-Embedding-0.6B | 70.47 | 3.22 ms | 6.47 ms | 12.20 ms | 24.24 ms | 49.99 ms |
| Jasper-TC-600M (\(\rho = 0.50\)) | 74.75 | 2.62 ms | 3.96 ms | 7.39 ms | 13.11 ms | 25.07 ms |
| Jasper-TC-600M (\(\rho = 0.33\)) | 74.71 | 2.52 ms | 3.52 ms | 5.41 ms | 9.38 ms | 17.52 ms |
| Jasper-TC-600M (\(\rho = 0.20\)) | 74.58 | 2.38 ms | 2.91 ms | 4.00 ms | 6.56 ms | 11.48 ms |
| Jasper-TC-600M (\(\rho = 0.10\)) | 74.22 | 2.09 ms | 2.56 ms | 3.18 ms | 4.48 ms | 6.95 ms |
Ablation Study¶
Ablation studies on the 23-task MTEB subset (1 epoch) and full MTEB isolate the contributions of teacher distillation, compression operators, and the progressive pipeline:
| Experiment / Variant | Dim. | Clas. | Clus. | Retr. | STS | Sum. | M.Task | Note |
|---|---|---|---|---|---|---|---|---|
| Teacher Distillation Ablation | (MTEB-Subset, 1 epoch) | |||||||
| Single Teacher (Qwen3-8B only) | 2048 | 90.12 | 53.03 | 57.87 | 84.70 | 32.74 | 68.91 | Lacks complementary STS geometry |
| Heterogeneous Dual-Teacher (Stage 1) | 2048 | 90.35 | 55.80 | 58.56 | 86.15 | 32.80 | 70.11 | +1.20 overall gain from dual teachers |
| Compression Operator Comparison | (MTEB-Subset, \(\rho=0.33\)) | |||||||
| Hard Truncation | 2048 | 76.53 | 41.66 | 22.54 | 78.57 | 11.45 | 51.62 | Drops tail context; retrieval plummets (-27.04) |
| Token Pruning (\(\ell_2\)-Norm) | 2048 | 80.19 | 43.98 | 29.78 | 78.53 | 29.39 | 55.38 | Preserves isolated tokens, destroying context |
| Token Merging (ToMe) | 2048 | 80.23 | 43.71 | 26.84 | 78.40 | 28.51 | 54.55 | Clustering discards fine details; slow on GPU |
| Learnable Conv1D | 2048 | 72.70 | 40.36 | 10.21 | 76.11 | 18.32 | 46.90 | Random initialization severely harms convergence |
| ETC Module (MLP + AvgPool) | 2048 | 89.94 | 51.39 | 49.58 | 85.03 | 29.27 | 66.28 | Dense global pooling retains core semantics |
| Progressive Training Pipeline | (Full MTEB Stage Analysis) | |||||||
| Skip Stage 1 (Direct Compression) | 2048 | 78.27 | 46.60 | 28.15 | 76.17 | 27.90 | 56.36 | Training collapse due to early spatial loss |
| Stop at Stage 2 (Fixed Ratio Only) | 2048 | 90.37 | 59.64 | 64.70 | 88.64 | 32.73 | 74.34 | Overfits fixed ratio; degrades at other budgets |
| w/o Retrieval Adaptation (Stage 4) | 2048 | 90.49 | 59.71 | 65.53 | 88.73 | 33.28 | 74.65 | Retrieval discrimination is not fully activated |
| Complete Model (Stages 1–4) | 2048 | 90.35 | 59.44 | 66.19 | 88.79 | 33.66 | 74.75 | Retrieval reaches 66.19 while preserving all tasks |
Key Findings¶
- Decoupled progressive staging is essential for successful compression training: Skipping Stage 1 to directly perform compression distillation on the base student (Skip Stage 1) caused MTEB Mean(Task) to collapse to 56.36 and Retrieval to drop to 28.15 (far below the uncompressed 0.6B baseline of 61.83). This confirms that simultaneously learning multi-teacher alignment and lossy sequence reduction induces severe optimization conflict; establishing an uncompressed foundation first is mandatory.
- Global average pooling vastly outperforms hard sequence truncation: Qualitative token-level heatmaps across five challenging retrieval scenarios (Extreme Tail, Lost in the Middle, Scattered Evidence, Dense Distraction, and Temporal Update) show that while hard truncation mechanically eliminates tokens beyond the cutoff boundary causing zero recall, ETC's global average pooling maintains strong attention activations on core evidence tokens regardless of their original position.
- Length threshold \(L_{th} = 80\) guards short queries: Setting \(L_{th} = 1\) compressed short queries destructively, lowering MTEB scores sharply. Setting \(L_{th} = 80\) fully protected short query semantics with negligible latency impact, while further increasing \(L_{th}\) to 256 yielded negligible accuracy gains at the expense of inflated inference latency.
Highlights & Insights¶
- Extending Matryoshka Representation Learning (MRL) to the sequence length dimension: While MRL provides storage elasticity across embedding dimensions, Jasper-Flash delivers computational elasticity along the sequence length axis. Users deploy a single model weight file and modulate \(\rho \in [0.1, 1.0]\) per inference request to navigate latency budgets from 4.48 ms to 24.24 ms.
- Mathematically verified heterogeneous teacher concatenation: By proving that the dot product of globally normalized concatenated vectors equals the arithmetic mean of the teachers' individual cosine similarities, the framework aggregates complementary representation spaces without requiring complex parameterized gating networks.
- Hardware-friendly early spatial reduction: Unlike token merging methods that require cross-token similarity sorting and irregular indexing, SwiGLU MLP projection paired with 1D adaptive average pooling executes as pure dense matrix multiplication, running at near-native hardware speed on modern tensor cores.
Limitations & Future Work¶
- Residual retrieval performance gap relative to specialized 8B teachers: The 0.6B student attains a retrieval score of 66.19, still trailing the 8B retrieval-specialized teacher (69.44). The uniform average pooling mechanism inevitably smooths over fine-grained entity-level token signals essential for exact-match asymmetric retrieval.
- Uniform spatial reduction lacks semantic routing: The current ETC module compresses tokens uniformly across the sequence rather than routing compression budgets based on token importance (e.g., preserving rare keywords while aggressively dropping stopwords).
- Evaluation context restricted to 1,030 tokens: Although latency scaling was benchmarked up to 4,096 tokens, downstream task evaluations were capped at 1,030 tokens. Validating elastic compression behavior across extreme context windows (32K to 128K) remains an area for future exploration.
Related Work & Insights¶
- vs Matryoshka Representation Learning (MRL): MRL targets representation dimension truncation for vector database storage and search efficiency; Jasper-Flash targets the quadratic self-attention bottleneck during forward encoding. The two approaches are completely orthogonal and can be stacked directly.
- vs Token Merging (ToMe) & Funnel-Transformer: Prior sequence reduction works insert clustering or downsampling operators within deep Transformer blocks and fix reduction ratios at compile time. Jasper-Flash places compression before the first self-attention layer and uses dynamic ratio training to provide continuous, zero-overhead inference elasticity within a single model.
- vs Classical Knowledge Distillation (TinyBERT, MiniLM): Traditional distillation focuses on reducing layer depth or hidden width against a single teacher. CAPD establishes a multi-teacher feature fusion and progressive curriculum specifically designed to decouple representation grounding from lossy sequence compression.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Combines sequence-length inference elasticity with mathematically proven multi-teacher feature fusion and progressive staging)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive bilingual benchmarks on MTEB/C-MTEB, complete 4-stage ablations, operator latency benchmarks, and qualitative heatmaps)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clean technical prose, rigorous mathematical formulation, and transparent decoupling of objectives)
- Value: ⭐⭐⭐⭐⭐ (Releasing a 600M model that outperforms 4B models while offering 5.4x latency reduction provides immense practical utility for dense retrieval deployments)