Skip to content

Length and Meaning for Language Model Maps

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/shimo-lab/modelmap
Area: LLM Reasoning
Keywords: Language model maps, Predictive importance sampling, Log-likelihood variance, Model representation and comparison, Domain remapping

TL;DR

Addressing the computational bottleneck where log-likelihood evaluation scales linearly with text volume when constructing macroscopic language model maps across domains, this work reveals that text length and semantic embeddings jointly determine inter-model likelihood variance. The authors propose Predictive Importance Sampling (PredIS), which achieves the KL divergence estimation accuracy of uniform sampling with only ~45% of the text volume while zero-shot generalizing across 24 unseen domains.

Background & Motivation

The open-source ecosystem has seen an explosive proliferation of Large Language Models (LLMs), with hundreds of thousands of checkpoints accessible on repositories like Hugging Face Hub. To systematically understand their evolutionary lineages, behavioral relationships, and emergent capabilities, recent literature has introduced Language Model Maps. By recording centered log-likelihood vectors across an evaluation text corpus, the squared Euclidean distance between two models rigorously approximates their symmetric Kullback–Leibler (KL) divergence. Applying dimensionality reduction techniques such as t-SNE to these distance matrices yields a two-dimensional geometric map that intuitively visualizes model clusters, alignment trajectories, and downstream capability profiles.

However, standard protocols construct these maps via uniform sampling over massive corpora like the Pile. This practice suffers from an acute trade-off between statistical fidelity and computational expenditure: achieving precise pairwise distance estimates necessitates evaluating tens of thousands of texts, where forward log-likelihood computation scales in direct proportion to the number of evaluated tokens and models. Prior diagnostic analyses revealed that uniform sampling retains roughly 50% redundant texts, because the discriminative power of individual texts—quantified by the inter-model variance of centered log-likelihoods—spans multiple orders of magnitude. Resampling proportionally to empirical variance can cut sample requirements in half; yet in real-world scenarios, practitioners must continually remap hundreds of models onto specialized domains (such as law, biomedicine, or code) where target model log-likelihoods are uncomputed, creating a fundamental chicken-and-egg bottleneck.

The core tension stems from the fact that inter-model likelihood variance is an unobserved posterior statistic before forward inference, whereas practitioners only have access to surface-level features of unannotated candidate texts prior to evaluation. While conventional wisdom presumed that variance is predominantly driven by token length, empirical evidence shows that variance spans extreme ranges even across texts of identical lengths, and the residual variance exhibits pronounced semantic clustering. Core idea: decompose text variance into orthogonal components of token length and semantic embeddings, train a lightweight linear regressor to predict inter-model log-likelihood variance a priori, and formulate smoothed defensive Predictive Importance Sampling (PredIS) to enable highly efficient zero-shot remapping across diverse domains without running target models in advance.

Method

Overall Architecture

The primary objective of PredIS is to select a highly informative, variance-prioritized subset of evaluation texts from an unannotated candidate corpus \(\mathcal{D}^\dagger\) prior to querying the target model set \(\mathcal{M}\), thereby minimizing the mean squared error of pairwise KL divergence estimations. The overall pipeline comprises three core phases: offline variance predictor training on a reference corpus, smoothed defensive importance sampling on the target domain, and self-normalized importance sampling (SNIS) for weighted distance matrix reconstruction. The inputs are an unannotated candidate pool \(\mathcal{D}^\dagger\) and a set of target models \(\mathcal{M}\); the output is an accurately estimated pairwise KL divergence matrix \(\hat{G}\) that preserves local and global geometric relationships in the resulting 2D language model map.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Candidate Corpus D† (10^6 texts)"] --> B["Text Feature Extraction<br/>GPT-2 length log L(x) + SimCSE vector v(x)"]
    B --> C["Variance Predictor<br/>Lightweight linear model predicts log V(x)"]
    C --> D["Smoothed Defensive Importance Sampling<br/>Exponential smoothing τ + Uniform mixture floor α"]
    D --> E["Sample Compact Subset D_PredIS<br/>without replacement (N = 4,500 texts)"]
    E --> F["Target Model Set M Forward Pass<br/>Self-Normalized Importance Weighted KL Estimation"]
    F --> G["High-Fidelity 2D Language Model Map<br/>(t-SNE projection & topology preservation)"]

Key Designs

1. Variance Predictor: Decoupled Linear Fusion of Text Length and Semantic Embeddings To overcome the impossibility of directly observing inter-model variance prior to running target models, this paper examines how variance \(V(x; \mathcal{M}) = \mathrm{Var}_{p_i \in \mathcal{M}} \xi_i(x)\) (where \(\xi_i(x)\) is centered log-likelihood with the length-dependent mean slope removed) correlates with intrinsic text attributes. While variance scales generally with token length \(L(x)\), the residual \(\Delta z(x) = \log V(x; \mathcal{M}) - \log L(x)\) shows distinct clustering patterns in embedding space across different text genres. Consequently, a 769-dimensional feature vector is formed by concatenating the scalar log-length \(\log L(x)\) (measured via the GPT-2 tokenizer) with a 768-dimensional sentence embedding \(v(x)\) obtained from an off-the-shelf SimCSE-RoBERTa-base encoder. An ordinary least squares linear regression model \(f(v(x), \log L(x))\) is trained to predict \(\log V(x; \mathcal{M})\) on a log scale due to the multi-order dynamic range of variance. DialoGPT-small is diagnosed as a significant distributional outlier based on log-likelihood vector norms and excluded from the training target. The resulting predictor trained on 99 open-source models attains a high Spearman rank correlation of \(\rho = 0.862\) and \(R^2 = 0.846\) on held-out validation texts.

2. Smoothed Defensive Importance Sampling: Bounding Extremes and Stabilizing SNIS Estimators Directly normalizing predicted variance \(\hat{V}(x) = \exp(f(v(x), \log L(x)))\) into a proposal distribution introduces severe numerical instability: extreme variance spikes cause the proposal distribution to collapse onto a handful of high-variance instances, while importance weights \(w(x) = 1/\pi(x)\) for low-variance samples explode, sharply inflating the variance of Monte Carlo estimators. To ensure bounded estimation error, PredIS introduces two complementary safeguards: an exponential smoothing parameter \(\tau \in (0, 1]\) to flatten peaked proposal probabilities, and a defensive uniform mixture coefficient \(\alpha \in [0, 1]\) that guarantees a non-zero probability floor \(\alpha / |\mathcal{D}^\dagger|\) for every text in the candidate pool: $\(\pi_{\mathrm{PredIS}}(x) = (1 - \alpha) \frac{\hat{V}(x)^\tau}{\sum_{x' \in \mathcal{D}^\dagger} \hat{V}(x')^\tau} + \frac{\alpha}{|\mathcal{D}^\dagger|}\)$ From this distribution, a compact subset \(\mathcal{D}_{\mathrm{PredIS}}^N\) of \(N\) texts is drawn without replacement. Pairwise population-level squared distances \(g_{ij} = \sum_{x \in \mathcal{D}^\dagger} (\xi_i(x) - \xi_j(x))^2\) are then reconstructed via the Self-Normalized Importance Sampling (SNIS) framework: $\(\hat{g}_{ij} = \sum_{x \in \mathcal{D}_{\mathrm{PredIS}}^N} \tilde{w}(x) (\hat{\xi}_i(x) - \hat{\xi}_j(x))^2, \quad \tilde{w}(x) = \frac{w(x)}{\sum_{x'} w(x')}\)$ where \(\hat{m}_i = \sum w(x)\log p_i(x) / \sum w(x)L(x)\) provides a weighted centering correction, guaranteeing asymptotically unbiased and scale-invariant distance estimation under finite sample budgets.

3. Zero-Shot Cross-Domain Remapping & Topological Ordering Preservation In practice, evaluating models on specialized domains (e.g., medicine, law, or programming) would become prohibitively expensive if the variance predictor required domain-specific retraining. PredIS establishes that a single linear predictor trained purely on general Pile samples generalizes zero-shot to 24 distinct domains in Common Pile v0.1. From an information-theoretic standpoint, minimizing forward KL divergence \(D_{\mathrm{KL}}(\pi^*_{ij} \parallel \pi)\) toward the pair-specific optimal sampling distribution \(\pi^*_{ij}(x) \propto (\xi_i(x) - \xi_j(x))^2\) is the key mechanism for variance reduction. PredIS systematically achieves divergence reduction (\(\Delta D_{ij} > 0\)) across 96.7% of all 4,947 model pairs. Qualitatively, PredIS selectively upweights high-residual texts (such as Pile-CC web articles, GitHub source code, and English Wikipedia) where model opinions diverge sharply, while systematically downweighting non-discriminative samples: both over-simplistic boilerplate (e.g., SQL queries and shell scripts that all LLMs predict effortlessly) and ultra-specialized or archaic content (e.g., dense mathematical equations and antique literature that all LLMs fail to model).

Benchmark & Evaluation Protocol

The experimental evaluation benchmarks \(K = 100\) open-source autoregressive LLMs from Hugging Face Hub (ranging from 0.1B to 22B parameters, spanning Qwen, Llama, Mistral, DeepSeek, Phi, Gemma, and others, all evaluated in full precision). Evaluations cover 25 distinct corpora: the uncopyrighted Pile (\(10^6\) 1,024-byte chunks) and 24 diverse textual domains from Common Pile v0.1 each exceeding 10GB (e.g., ArXiv Papers, GitHub Archive, Caselaw Access Project, PubMed, StackExchange). Metrics are averaged across \(T = 1,000\) independent sampling trials. Primary quantitative criteria include Relative RMSE against ground-truth population KL divergence (\(e_{ij} = \sqrt{\mathrm{MSE}_{ij}} / g_{ij}\)), median Efficiency Gain over uniform sampling (\(R_{ij} = \mathrm{MSE}_{ij}^{\mathrm{Unif}} / \mathrm{MSE}_{ij}^{\mathrm{Method}}\)), and Spearman rank correlation \(\bar{\rho}\) assessing the global rank preservation of pairwise distance matrices.

Key Experimental Results

Main Results

Under a sample budget of \(N = 10,000\), the performance of PredIS against baselines across representative domains and macro averages is summarized in Table 1:

Domain Mean Rel. RMSE (↓)
Unif / Length / PredIS / Oracle
Median Efficiency Gain (↑)
Length / PredIS / Oracle
Spearman Corr. (N=10, ↑)
Unif / Length / PredIS / Oracle
The Pile (General) 0.048 / 0.036 / 0.032 / 0.030 1.72 / 2.04 / 2.16 0.886 / 0.905 / 0.916 / 0.916
ArXiv Papers (Academic) 0.046 / 0.040 / 0.029 / 0.027 1.15 / 1.81 / 1.93 0.903 / 0.907 / 0.920 / 0.921
GitHub Archive (Code) 0.156 / 0.217 / 0.108 / 0.042 1.40 / 1.72 / 3.36 0.872 / 0.892 / 0.899 / 0.898
Caselaw Project (Legal) 0.027 / 0.024 / 0.022 / 0.020 1.26 / 1.37 / 1.58 0.900 / 0.908 / 0.909 / 0.913
StackExchange (Q&A) 0.114 / 0.140 / 0.072 / 0.037 1.42 / 2.41 / 3.60 0.918 / 0.931 / 0.938 / 0.937
UK Hansard (Parliament) 0.054 / 0.040 / 0.030 / 0.029 1.86 / 2.60 / 3.15 0.800 / 0.809 / 0.844 / 0.870
YouTube (Transcripts) 0.232 / 0.170 / 0.099 / 0.045 1.89 / 5.54 / 16.72 0.858 / 0.858 / 0.857 / 0.860
PubMed (Biomedical) 0.023 / 0.020 / 0.023 / 0.021 1.51 / 1.10 / 1.37 0.927 / 0.957 / 0.955 / 0.958

In accuracy-matched settings (matching Uniform at \(N = 10,000\)), log-likelihood computation times on a single NVIDIA A100 (40GB) GPU and break-even model counts \(K^* = \lceil a/b \rceil\) (offsetting the \(a = 30\) min offline embedding cost) are reported in Table 2:

Model Architecture & Size Uniform (N=10k) Wall Time Length (N=6k) Wall Time PredIS (N=4.5k) Wall Time vs. Uniform Time Saved (b) / Break-Even Models \(K^*\) vs. Length Time Saved (b) / Break-Even Models \(K^*\)
Qwen3-0.6B 4.6 min 2.8 min 2.2 min +2.4 min / 13 +0.6 min / 50
Qwen3-1.7B 4.7 min 2.8 min 2.2 min +2.5 min / 12 +0.6 min / 50
Qwen3-4B 6.0 min 3.7 min 2.9 min +3.1 min / 10 +0.8 min / 38
Qwen3-8B 6.6 min 4.1 min 3.3 min +3.3 min / 10 +0.8 min / 38
Qwen3-14B 9.9 min 6.2 min 4.9 min +5.0 min / 6 +1.3 min / 24

Ablation Study

The ablation comparing variance predictors trained with and without the outlier model DialoGPT-small (PredIS-100 vs. PredIS-99) across 25 domains is presented in Table 5 (\(N = 10,000, T = 100\) trials):

Domain Mean Rel. RMSE (↓)
Uniform / Length / PredIS-100 / PredIS-99
Median Efficiency Gain (↑)
PredIS-100 / PredIS-99
Mean RMSE (↓)
Uniform / Length / PredIS-100 / PredIS-99
The Pile 0.047 / 0.035 / 0.043 / 0.033 1.18 / 1.80 1.28e+04 / 7.55e+03 / 9.35e+03 / 6.94e+03
ArXiv Papers 0.041 / 0.036 / 0.037 / 0.028 0.85 / 2.00 3.75e+03 / 2.66e+03 / 2.74e+03 / 2.41e+03
GitHub Archive 0.139 / 0.258 / 0.150 / 0.089 1.06 / 1.71 6.73e+03 / 9.44e+03 / 4.55e+03 / 3.16e+03
StackExchange 0.114 / 0.059 / 0.124 / 0.078 1.54 / 2.09 3.69e+03 / 2.67e+03 / 2.77e+03 / 2.24e+03
YouTube 0.211 / 0.179 / 0.060 / 0.103 9.67 / 4.46 2.88e+03 / 1.91e+03 / 1.08e+03 / 1.27e+03
USPTO (Patents) 0.058 / 0.041 / 0.059 / 0.038 0.78 / 2.42 5.23e+03 / 3.15e+03 / 6.72e+03 / 2.96e+03

Key Findings

  • Substantial Sample Reduction: On the Pile benchmark, PredIS requires only \(N = 4,500\) texts to achieve the identical relative RMSE (0.048) of uniform sampling with \(N = 10,000\) texts, cutting the required evaluation data volume by 55% and markedly outperforming pure length-based sampling (\(N = 6,000\)).
  • Remarkable Zero-Shot Generalization: Across 24 unseen Common Pile domains, PredIS outperforms Uniform and Length in relative RMSE on 23 domains without any retraining, yielding massive median efficiency gains on YouTube (5.54x), UK Hansard (2.60x), and StackExchange (2.41x). In the singular outlier domain (PubMed), even the theoretical Oracle upper bound fails to surpass Length, confirming an atypical variance distribution in medical abstracts rather than predictor failure.
  • Superior Neighborhood Topological Preservation: Connecting the 10 nearest neighbors on 2D maps whose relative RMSE falls below the uniform threshold (\(\gamma = 0.0477\)), PredIS preserves 499 high-precision edges, exceeding Length (440 edges) and Uniform (335 edges), proving that variance-based sampling specifically refines local model cluster fidelity.
  • Low Break-Even Threshold: Although computing embeddings for \(10^6\) candidate texts takes ~30 minutes on an A100 GPU, the per-model inference savings (e.g., 5.0 minutes on Qwen3-14B) allow PredIS to achieve net computational savings once 6 to 13 models are evaluated, delivering dramatic net benefits when evaluating fleets of dozens or hundreds of LLMs.

Highlights & Insights

  • Decoupling the Variance Mechanism: The paper shatters the naive assumption that text length alone dictates model discriminability. It rigorously proves that length merely provides a baseline magnitude, while the residual variance is intrinsically governed by semantic clustering in representation space.
  • Minimalist Linear Elegance: Rather than employing complex non-linear neural regressors, simply training an ordinary least squares model on concatenated 768-D SimCSE vectors and log-length achieves an exceptional Spearman \(\rho = 0.862\), offering near-instant training and immune resistance to cross-domain overfitting.
  • Principled Defensive Formulation: Combining power smoothing (\(\tau = 0.5\)) with defensive mixture (\(\alpha = 0.1\)) prevents importance weight explosions on long-tail instances, presenting a clean, robust template for general LLM active evaluation and subset selection.

Limitations & Future Work

  • Reliance on White-Box Log-Likelihoods: The methodology requires token-level conditional probability outputs from target models, restricting its utility to open-weight LLMs and excluding commercial black-box API endpoints.
  • Embedding Overhead for Small Model Fleets: For minimal benchmarking suites (\(K < 6\)), the 30-minute offline embedding computation outweighs the cumulative inference savings, rendering uniform or length sampling more economical.
  • Monolingual & Fixed Encoder Scope: Empirical validations remain restricted to English corpora and the SimCSE-RoBERTa encoder; evaluating multi-lingual texts and modern retrieval-augmented dense embeddings (e.g., Contriever, BGE) remains an open research avenue.
  • vs Active Testing (Kossen et al., 2021; Berrada et al., 2025): Active testing focuses on optimizing the evaluation efficiency of a single model on labeled classification/QA benchmarks via surrogate models; PredIS targets the full pairwise KL divergence geometry across an entire fleet of 100+ LLMs on unlabeled corpora in an a priori zero-shot setup.
  • vs Anchor Points (Vivek et al., 2024): Anchor points subsample existing benchmark datasets based on sample difficulty; PredIS samples from unstructured, raw web corpora specifically for unsupervised language model mapping.
  • vs Empirical Variance Resampling (Oyama et al., 2025a): While prior work identified the theoretical benefit of variance proportional sampling, it required running all target models on candidate texts first to obtain posterior variances; PredIS establishes the first predictive framework that enables practical cross-domain remapping before target models are executed.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Pioneers the integration of text length and semantic embeddings for a priori variance prediction and importance sampling in language model maps.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously validated on 100 open-weight models across 25 domains with thousands of Monte Carlo trials and comprehensive break-even analyses.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant mathematical formulations, and insightful graphical representations.
  • Value: ⭐⭐⭐⭐⭐ Provides a foundational, computationally efficient tool for large-scale model auditing, phylogenetic charting, and specialized domain remapping.