The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ThisIsManuel/HTEB
Area: Information Extraction
Keywords: Text Embedding Benchmark, Robustness Evaluation, Dynamic Evaluation, Multidimensional Robustness, LLM Transformations
TL;DR¶
Addressing the fundamental blind spots of static, single-scalar embedding leaderboards like MTEB, this paper introduces HTEB, a dynamic evaluation framework that stochastically injects 8 LLM-generated transformations along Lexical/Stylistic, Length, and Language axes, demonstrating that embedding robustness is multidimensional and that parameter scaling fails to close robustness gaps.
Background & Motivation¶
Dense text representations and vector embeddings form the fundamental cornerstone of modern information retrieval, retrieval-augmented generation (RAG), question answering, and semantic information extraction pipelines. For years, the community has evaluated embedding systems through standardized benchmarks such as MTEB and MMTEB, which compute an aggregated scalar average across fixed, static test collections. This paradigm implicitly formalizes robustness as an invariant, one-dimensional property. However, real-world deployment is inherently dynamic: queries and user documents experience natural, non-adversarial variations ranging from shifts between formal and colloquial styles, drastic length variations from terse queries to multi-paragraph expositions, to multilingual transfers and cross-lingual alignments. Static scalar scores inevitably obscure severe model vulnerabilities under specific operational conditions and invite overfitting to static benchmark corpora.
Recent literature has highlighted these structural deficiencies. SAGE explored rule-based character perturbations and synthetic noise injection on English benchmarks; PTEB pioneered evaluation-time stochastic paraphrasing across MTEB datasets; and FLUKE investigated task brittleness under minimal English linguistic shifts. Nonetheless, existing efforts either constrain themselves to a single perturbation modality (such as simple rephrasing), collapse diverse transformations into a single undifferentiated metric, or restrict analysis exclusively to monolingual English corpora. The underlying conflict is that robustness is fundamentally multidimensional: different architectures exhibit decoupled tolerances across distinct perturbation regimes. For instance, a model may remain exceptionally resilient to stylistic paraphrasing while deteriorating catastrophically when inputs expand in length or migrate across languages. Single-score aggregates conceal these axis-specific failure modes, preventing practitioners from selecting models aligned with their actual deployment constraints. Furthermore, dynamic LLM-based evaluation introduces critical methodological concerns regarding semantic drift and label preservation that prior studies failed to audit systematically.
To transcend these limitations, this paper proposes the Harder Text Embedding Benchmark (HTEB), a dynamic evaluation framework that systematically challenges embedding models along three practically interpretable, deployment-motivated axes: Lexical/Stylistic, Length, and Language. HTEB generates 8 carefully prompted transformations using open-weight generative models validated through rigorous error auditing and 4,800 human ratings. Core idea: abandon the assumption of static one-dimensional robustness by introducing HTEB, a dynamic evaluation benchmark that decomposes text embedding robustness along Lexical/Stylistic, Length, and Language axes at test time, exposing widespread ranking instability and decoupled model vulnerability profiles that parameter scale alone cannot resolve.
Method¶
Overall Architecture¶
HTEB establishes an automated, reproducible, dynamic evaluation pipeline that intervenes directly at evaluation time. Rather than evaluating pre-computed, static test sets, the framework applies a generative pipeline driven by an open-weight instruction-tuned LLM to transform inputs on the fly. The input data comprises 32 datasets across 15 tasks from (M)MTEB (covering classification, clustering, pair classification, reranking, retrieval, semantic textual relatedness STR, semantic textual similarity STS, and summarization). For asymmetrical query-document tasks (retrieval and reranking), HTEB strictly transforms the queries alone, mimicking realistic deployment scenarios where variable user requests target a stable corpus while circumventing prohibitive document-side inference overhead. The transformations span 8 discrete operations organized into three deployment-motivated axes. To account for generative variance, each benchmark is evaluated over 3 independent random seeds, and all findings are substantiated through non-parametric inferential statistics.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Original Benchmark Inputs<br/>(M)MTEB Datasets (Queries / Sentence Pairs)"] --> B["1. Transformation Engine Selection & Error Truncation<br/>4-stage filter: STS-B error audit โ LLM judge cross-eval โ human verification"]
B --> C["2. Three-Dimensional Eight-Operator Perturbation Taxonomy<br/>Lexical/Stylistic / Length / Language deployment-oriented axes"]
C --> D["3. Construct Validity & Human Label Preservation Audit<br/>4,800 ratings + Spanish source transfer + STS-B gold similarity MAD check"]
D --> E["4. Multi-Seed Non-Parametric Inferential Profiling<br/>3 seeded runs + Wilcoxon signed-rank + Hodges-Lehmann shifts"]
E --> F["Multidimensional Robustness Profiles<br/>(Per-axis score deltas, rank volatility via Kendall's ฯ)"]
Key Designs¶
1. Transformation Engine Selection and Error Truncation: eliminating generation hallucinations and format leaks
Dynamic evaluation critically hinges on the generative fidelity of the underlying transformation model. To avoid confounding embedding degradation with generation errors (such as outputting empty strings, leaking reasoning traces, or violating format constraints), HTEB adopts a rigorous four-step model selection protocol. The authors evaluated 16 open-weight candidate models across 5 prominent families (Apriel-1.5/1.6, Gemma-3, Ministral-3, Olmo-3, and Qwen3) on STS-B across 10 granular failure modes:
- These error rates include Identical Rate (verbatim copy of input), Empty Rate, Ellipsis Rate (refusal signified by ...), JSON Fragment Rate (unsolicited structured formatting), Reasoning Leak Rate (leaking chain-of-thought phrases like Let me think), Prefix Leak Rate (retaining prompt headers like Paraphrased text:), Wrong Language Rate, Runaway Generation Rate (>5x original length for non-expansion tasks), Truncated Rate (<20% original length for non-summarization tasks), and Summary Too Long Rate (summaries exceeding input length).
- The audit revealed that 4-bit Activation-aware Weight Quantization (AWQ) successfully constrained error rates while enabling execution on accessible dual-GPU hardware. A crossover round-robin LLM-as-a-Judge tournament among the top 5 lowest-error generators demonstrated that Gemma-3-27B-int4-AWQ achieved the lowest overall error rate (3.0%) and the highest quality score across judges (mean 4.45, 95% CI: [4.44, 4.46]), statistically outperforming all other candidates (\(p < 0.001\), Holm-corrected bootstrap). Consequently, it was selected as HTEB's standard transformation generator.
2. Three-Dimensional Eight-Operator Perturbation Taxonomy: comprehensive coverage of deployment shifts Rather than treating perturbations as unstructured noise, HTEB structures transformations along three interpretable, deployment-oriented dimensions via 8 tailored prompt templates: - Lexical/Stylistic Axis: Captures semantic-preserving superficial variations. (1) Paraphrasing reformulates syntactic structure and vocabulary while maintaining invariant meaning; (2) Backtranslation translates text into a randomly sampled intermediate language (from Arabic, French, German, Spanish, Turkish, or English) and subsequently translates it back into the source tongue; (3) Style Change bi-directionally transforms informal/colloquial phrasing into formal/academic text and vice versa. - Length Axis: Reflects input conciseness and verbosity shifts in real-world queries. (4) Expansion elaborates on the input by introducing context and descriptive details while strictly preserving the core premise; (5) Summarisation distills text down to salient propositional content; (6) Summarised Expansion cascades expansion followed by summarization, creating information-dense synthetic variants. - Language Axis: Tests model sensitivity to shifts in deployment language. (7) Translation translates all texts in a batch to the same randomly sampled target language; (8) Cross-Translation translates each text within a pair or batch into independent, disjoint random languages, subjecting the representation space to extreme cross-lingual alignment stress.
3. Construct Validity and Human Label Preservation Audit: establishing semantic invariance across 4,800 ratings To ensure that observed performance drops reflect representational brittleness rather than semantic destruction or label noise introduced by the LLM, the authors conducted extensive human validation studies. Annotators recruited through Amazon Mechanical Turk (filtered for \(\ge 95\%\) approval and \(\ge 1,000\) completed HITs) and native-speaker linguists evaluated text pairs along two 1-to-5 Likert scales: Transformation Quality and Fluency: - On English datasets, 4,800 individual ratings across all 8 transformations yielded a high overall quality mean of \(4.09 \pm 1.04\) (Gwet's \(AC_2 = 0.68\)) and fluency mean of \(4.42 \pm 0.86\) (\(AC_2 = 0.80\)), comfortably exceeding the predefined acceptance threshold of 3.0 ("acceptable"). - A non-English validation study on Spanish source texts collected 1,200 additional human ratings, returning mean quality of 4.03 and fluency of 4.34, confirming that the generator maintains high semantic fidelity beyond English. - An exploratory label preservation study on 50 STS-B sentence pairs evaluated human similarity scores before and after transformation across Paraphrasing, Summarised Expansion, and Translation. The mean absolute difference (MAD) between transformed and original ratings was just 0.65, 0.41, and 0.45, with 88%, 96%, and 90% of paired ratings differing by \(\le 1\) point on the 0-5 scale. This directly establishes construct validity: semantic similarity labels remain preserved, confirming that performance drops stem from representation failures.
4. Multi-Seed Non-Parametric Inferential Profiling: robust statistical guarantees Recognizing that LLM generation involves stochastic sampling, HTEB executes 3 independent seeded runs (seeds 1337, 1338, 1339) for every model-dataset-transformation combination. Rather than relying on parametric assumptions that fail in NLP benchmarks, HTEB executes non-parametric inferential statistics: - Two-sided Wilcoxon signed-rank tests with Holm family-wise error rate corrections evaluate paired dataset-level score deltas, accompanied by Hodges-Lehmann location shift estimators with 95% confidence intervals derived from Walsh averages. - Split-half reliability analysis using 1,000 stratified random dataset splits yielded median Spearman \(\rho\) values of 0.82โ0.96 for English and 0.80โ0.93 for multilingual data. This proves that axis-specific robustness profiles represent stable, reproducible architectural properties rather than stochastic sampling artifacts.
Experimental Results¶
Main Benchmark Results¶
The benchmark evaluates 16 diverse open-weight sentence transformer embedding models, spanning specialized bi-encoders, autoregressive LLM-based embeddings, and long-context architectures from 33M to 27B parameters.
Table 1: Embedding model performance on English datasets (19 datasets, 3 runs) across robustness axes and overall HTEB score (%)
| Model Architecture | Parameter Scale | Original MTEB Score | Lexical / Stylistic | Length Axis | Language Axis | HTEB Total Score | Absolute Delta (\(\Delta\)) | MTEB \(\rightarrow\) HTEB Rank |
|---|---|---|---|---|---|---|---|---|
| NV-Embed-v2 | 7B (Decoder) | 73.34 | 69.26 (-4.08) | 66.99 (-6.35) | 61.84 (-11.50) | 66.03 | -7.31 | 1 \(\rightarrow\) 4 |
| F2LLM-v2-4B | 4B (Multilingual) | 73.00 | 68.42 (-4.58) | 66.93 (-6.07) | 67.76 (-5.24) | 67.70 | -5.30 | 2 \(\rightarrow\) 1 |
| Qwen3-Emb-8B | 8B (Multilingual) | 72.77 | 68.12 (-4.65) | 66.62 (-6.14) | 67.09 (-5.68) | 67.28 | -5.49 | 3 \(\rightarrow\) 2 |
| Jina-Emb-v5-Small | 430M (Distilled) | 71.50 | 67.64 (-3.85) | 65.75 (-5.75) | 66.12 (-5.38) | 66.50 | -5.00 | 4 \(\rightarrow\) 3 |
| Stella-EN-400M-v5 | 400M (English) | 71.04 | 66.67 (-4.37) | 65.23 (-5.81) | 45.46 (-25.58) | 59.12 | -11.92 | 5 \(\rightarrow\) 7 |
| Jasper-600M | 600M (Bilingual) | 70.78 | 67.10 (-3.67) | 65.14 (-5.63) | 56.69 (-14.08) | 62.98 | -7.80 | 6 \(\rightarrow\) 5 |
| MXBAI-Emb-Large-v1 | 335M (English) | 70.75 | 66.34 (-4.41) | 65.32 (-5.43) | 40.72 (-30.03) | 57.46 | -13.29 | 7 \(\rightarrow\) 9 |
| E5-Mistral-7B-it | 7B (Instruction) | 67.28 | 63.32 (-3.96) | 62.71 (-4.57) | 56.47 (-10.82) | 60.83 | -6.45 | 8 \(\rightarrow\) 6 |
| All-MPNet-Base-v2 | 110M (Classical) | 67.09 | 62.84 (-4.25) | 62.57 (-4.52) | 36.34 (-30.75) | 53.92 | -13.17 | 9 \(\rightarrow\) 10 |
| All-MiniLM-L12-v2 | 33M (Classical) | 66.36 | 62.64 (-3.72) | 61.86 (-4.50) | 37.14 (-29.22) | 53.88 | -12.48 | 10 \(\rightarrow\) 11 |
| Llama-Nemotron-8B | 8B (Multilingual) | 63.17 | 59.91 (-3.26) | 62.49 (-0.68) | 54.16 (-9.01) | 58.85 | -4.32 | 11 \(\rightarrow\) 8 |
| Benchmark Average | - | 69.73 | 65.66 (-4.07) | 64.69 (-5.04) | 53.62 (-16.12) | 61.32 | -8.41 | - |
Table 2: Embedding model performance on multilingual datasets (13 datasets, 42 languages, 3 runs) (%)
| Model Architecture | Original Score | Lexical / Stylistic | Length Axis | Language Axis | HTEB Total Score | Score Delta (\(\Delta\)) | Key Dimensional Behavior |
|---|---|---|---|---|---|---|---|
| F2LLM-v2-4B | 67.90 | 65.09 (-2.81) | 63.74 (-4.16) | 67.91 (+0.01) | 65.58 | -2.32 | Retains rank #1; perfect zero-drop stability on Language |
| Qwen3-Emb-8B | 64.43 | 61.79 (-2.64) | 61.02 (-3.41) | 63.71 (-0.72) | 62.17 | -2.26 | Uniform degradation, retains rank #2 |
| Jina-Emb-v5-Small | 63.91 | 61.54 (-2.37) | 60.86 (-3.04) | 64.74 (+0.83) | 62.38 | -1.53 | Positive language gain (+0.83), overtakes Qwen3 on HTEB |
| BGE-M3 | 62.99 | 60.32 (-2.66) | 58.95 (-4.04) | 60.62 (-2.36) | 59.97 | -3.02 | Substantial degradation on Length transformations |
| GTE-ML-Base | 62.40 | 60.11 (-2.29) | 59.46 (-2.95) | 61.43 (-0.97) | 60.33 | -2.07 | Balanced moderate drops across all three axes |
| ML-E5-Large-it | 61.65 | 58.63 (-3.02) | 57.88 (-3.78) | 57.61 (-4.04) | 58.04 | -3.61 | Vulnerable to cross-lingual and length drift |
| Harrier-270M | 58.87 | 55.59 (-3.27) | 54.64 (-4.23) | 55.47 (-3.39) | 55.23 | -3.63 | Smaller capacity yields higher stylistic vulnerability |
| Llama-Nemotron-8B | 56.37 | 54.86 (-1.51) | 57.82 (+1.45) | 56.63 (+0.27) | 56.44 | +0.07 | Gains on Length (+1.45) offset stylistic drops; net positive |
| Paraphr.-ML-MPNet | 55.52 | 54.35 (-1.17) | 53.40 (-2.12) | 58.75 (+3.23) | 55.50 | -0.02 | Strong language axis robustness (+3.23) neutralizes drop |
| Benchmark Average | 61.56 | 59.14 (-2.42) | 58.64 (-2.92) | 60.76 (-0.79) | 59.52 | -2.04 | Significantly smaller degradation than on English corpora |
Parameter Scale Ablation¶
To determine whether scaling model capacity resolves perturbation-induced degradation, the authors ablated 14 checkpoints across three distinct embedding model families (F2LLM-v2, Qwen3-Embedding, and Harrier-OSS) across parameter scales on HTEB English datasets.
Table 3: Model size ablation across three embedding families on HTEB English datasets (averaged over 19 datasets, 3 runs)
| Model Family | Checkpoint Scale | Original Score | HTEB Total Score | Robustness Gap (\(\Delta\)) | Language Axis Score | Language Axis Gap |
|---|---|---|---|---|---|---|
| Qwen3-Emb | 0.6B | 68.22 | 61.94 | -6.28 | 58.12 | -10.10 |
| 4B | 71.95 | 66.30 | -5.65 | 65.48 | -6.47 | |
| 8B | 72.77 | 67.28 | -5.49 | 67.09 | -5.68 | |
| F2LLM-v2 | 80M | 59.20 | 53.64 | -5.56 | 49.30 | -9.90 |
| 160M | 62.45 | 56.91 | -5.54 | 54.12 | -8.33 | |
| 330M | 64.80 | 59.15 | -5.65 | 57.60 | -7.20 | |
| 0.6B | 67.10 | 61.55 | -5.55 | 61.20 | -5.90 | |
| 1.7B | 70.30 | 64.92 | -5.38 | 65.10 | -5.20 | |
| 4B | 73.00 | 67.70 | -5.30 | 67.76 | -5.24 | |
| 8B | 73.45 | 68.10 | -5.35 | 68.25 | -5.20 | |
| 14B | 74.12 | 68.75 | -5.37 | 69.10 | -5.02 | |
| Harrier-OSS | 270M | 63.85 | 57.62 | -6.23 | 54.70 | -9.15 |
| 0.6B | 67.40 | 61.35 | -6.05 | 60.10 | -7.30 | |
| 27B | 73.15 | 67.55 | -5.60 | 68.40 | -4.75 |
Key Findings¶
- Dramatic Leaderboard Reranking: The static MTEB leader on English data,
NV-Embed-v2(original 73.34), plunges by 7.31 pp under HTEB, dropping to 4th place. Conversely,F2LLM-v2-4BandQwen3-Emb-8Bsurpass it due to superior resilience across stylistic and multilingual shifts. Kendall's \(\tau\) rank correlation between static and dynamic rankings averages only 0.555 on English (with Expansion at \(\tau = 0.486\) and Translation at \(\tau = 0.386\)), demonstrating that static rankings poorly predict real-world performance under realistic input variations. - Decoupled Robustness Profiles: Model performance across perturbation axes is largely decoupled rather than driven by a single latent robustness factor.
Llama-Nemotron-8Bexhibits remarkable resilience on the Length axis (improving by +1.45 pp on multilingual data and gaining +4.7 pp under Expansion), yet suffers severe degradation under Cross-Translation (-12.6 pp on English). In contrast,Paraphrase-Multilingual-MPNetexhibits positive gains on the Language axis (+3.23 pp) while suffering standard drops on Length (-2.12 pp). - Scaling Law Limits on Robustness: Across 14 models spanning 80M to 27B parameters, increasing model capacity elevates absolute performance but fails to contract the gap between untransformed and transformed evaluations (the robustness delta remains persistently around ~5.3โ5.6 pp for F2LLM-v2). Parameter scale disproportionately benefits the Language axis (reducing F2LLM's gap from -9.90 pp to -5.02 pp) while offering virtually no additional protection against Lexical/Stylistic or Length transformations.
- English vs. Multilingual Sensitivity Asymmetry: Text embeddings exhibit substantially higher sensitivity to HTEB transformations on English datasets than on multilingual corpora (mean drops of -8.41 pp vs. -2.04 pp; or -4.56 pp vs. -2.67 pp when omitting the Language axis). This gap persists when controlling for identical models, parallel STS-B translations (English STS-B drop of -14.9 vs. German STS-B drop of -10.2), and confirmed zero-shot training metadata (8.84 pp vs. 2.10 pp), indicating that English benchmarks suffer from severe over-reliance on surface keyword overlap.
Highlights & Insights¶
- Multidimensional Decoupled Robustness Taxonomy: HTEB moves the field beyond simplistic aggregate benchmarks by formalizing Lexical/Stylistic, Length, and Language as three practical, deployment-oriented evaluation axes. This enables practitioners to select embeddings based on their specific operational profiles (e.g., prioritizing length resilience for conversational search vs. multilingual resilience for cross-border indexing).
- Construct Validity Grounded in Human Validation: Unlike prior dynamic evaluation studies that assume LLM perturbations preserve semantics without proof, HTEB establishes construct validity through 4,800 human quality and fluency annotations, non-English Spanish validations, and STS-B gold similarity label audits demonstrating that paired rating shifts remain strictly within 1 point.
- Critical Implications for Representation Learning: The empirical finding that parameter scaling fails to close the transformation gap proves that larger language model backbones do not automatically learn transformation-invariant vector spaces. Overcoming embedding fragility will require architectural inductive biases and explicit contrastive invariance constraints during fine-tuning.
Limitations & Future Work¶
- Risk of Circularity from Synthetic LLM Data: Because HTEB relies on generative LLMs for transformations, embedding models fine-tuned on synthetic data generated by similar LLMs may possess implicit distributional advantages. Conversely, models trained strictly on human corpora might be unfairly penalized.
- Computational Overhead of Evaluation-Time Generation: Running 8 LLM transformations across 3 random seeds on 32 datasets significantly escalates evaluation runtime compared to static embedding lookups. Future work could investigate adaptive subsetting strategies or lighter transformation models.
- Fixed Task Instructions for Instruction-Tuned Embeddings: The benchmark evaluates transformations on input passages while holding the task instruction fixed. Recent literature shows that instruction-tuned encoders are highly sensitive to prompt phrasing, which represents an orthogonal dimension of evaluation not yet integrated into HTEB.
Related Work & Insights¶
- vs. MTEB / MMTEB (Muennighoff et al., 2023; Enevoldsen et al., 2025): MTEB established the standard multi-task embedding benchmark but relies on static datasets and a single scalar average. HTEB builds directly on top of MTEB tasks, introducing dynamic, evaluation-time perturbations that reveal critical ranking instabilities hidden by static evaluation.
- vs. SAGE (Goel et al., 2025): SAGE applies rule-based typo and character-level noise injections to English datasets. HTEB targets higher-level semantic, stylistic, length, and cross-lingual shifts generated by state-of-the-art LLMs, better reflecting realistic user input variations.
- vs. PTEB (Frank and Afli, 2026) & CS-MTEB (Zeng et al., 2026): PTEB and CS-MTEB introduced dynamic evaluation for paraphrasing and code-switching, respectively. HTEB unifies and expands these efforts into a cohesive three-dimensional evaluation framework supported by non-parametric statistics and exhaustive human verification.
Rating¶
- Novelty: โญโญโญโญโญ (Pioneers a multidimensional dynamic evaluation paradigm that dismantles the single-scalar static leaderboard model)
- Experimental Rigor: โญโญโญโญโญ (Evaluates 16 models across 32 datasets and 42 languages, backed by 3 seeded runs, 4,800 human ratings, and non-parametric hypothesis testing)
- Writing & Clarity: โญโญโญโญโญ (Exceptionally well-structured paper with transparent methodological reflections and rigorous statistical formulations)
- Practical Impact: โญโญโญโญโญ (Highly actionable framework for selecting and diagnosing text embeddings in production RAG and retrieval architectures)