TBLS: Transition-Based Language Similarity through Activation Perturbation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/WildRaincoat/TBLS
Area: Interpretability
Keywords: language similarity, activation perturbation, multilingual models, cross-lingual transfer, representation geometry
TL;DR¶
To address the cross-model instability and static nature of conventional geometric language similarity measures, this paper proposes Transition-Based Language Similarity (TBLS), which injects controlled Gaussian noise into intermediate activations to trigger cross-lingual output transitions, benchmarking representation proximity via calibrated escape noise.
Background & Motivation¶
Language similarity represents a foundational concept bridging theoretical linguisticsโsuch as phylogenetic genealogical classification and typological profilingโand multilingual natural language processing. In computational setups, reliable similarity metrics are essential for data selection in pretraining, identifying transfer languages for zero-shot cross-lingual adaptation, and evaluating low-resource model behaviors. With the proliferation of multilingual language models (mLMs) like mGPT, XLM-R, mT5, and Qwen3, extensive research has attempted to extract language similarities directly from the internal geometry of learned representation spaces. Dominant paradigms typically rely on static geometric snapshots, calculating cosine distances between parallel sentence embeddings (REP/mPLM-Sim), distances between language centroids (CentroidSim), or neuron-level activation correlations (ANC).
However, such static geometric measurements suffer from fundamental theoretical and practical vulnerabilities. First, static distance metrics are known to vary dramatically across transformer layer depths, model architectures, tokenizer vocabulary designs, and evaluation corpora domains. Consequently, different models often yield contradictory language taxonomies, lacking cross-architecture universality. Second, static distances offer merely a static spatial slice of embedding distributions, completely omitting the dynamical behavior of the model during sequence generation and token projection. If two languages genuinely inhabit adjacent, locally connected sub-manifolds in a model's latent representation space, local perturbations to hidden states should naturally prompt the output distribution to shift probability mass from the source language toward its close neighbor, while distant languages should remain unreachable.
To overcome the blind spots of static geometric probes, this paper adopts a behavior-driven perspective centered on perturbation-induced output transitions. By injecting controlled Gaussian noise into the intermediate feed-forward activations of mLMs during forward passes, the framework investigates at what perturbation strength a model escapes its source language region and how its probability mass reallocates across the multilingual spectrum. Core idea: by perturbing feed-forward layer activations with calibrated language-level escape noise, TBLS quantifies cross-lingual output transition probabilities and symmetrizes them via bidirectional minima, providing a behavior-driven metric that directly captures the local connectivity and proximity of multilingual representation geometry.
Method¶
Overall Architecture¶
The TBLS pipeline operates through three cohesive stages: first, for an input text in a given source language, Gaussian noise is incrementally added to intermediate feed-forward sublayer activations during inference until the decoded output sequence is no longer identified as the source language, yielding a sample escape noise that is averaged across inputs to establish a standardized language-level escape noise (\(\sigma_{L_i}\)); second, running a single forward pass under this calibrated escape noise, the model extracts the full token probability distribution at the first perturbed position, resolving subword language ambiguity by combining MC4 corpus frequency weighting for shared tokens with hard language identification labels from XLM-V; third, the directional transition probabilities are estimated empirically by taking expectations across the dataset, and the symmetric TBLS matrix is formed by taking the minimum of bidirectional transitions, ensuring mutual geometric proximity.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual Input Sequence<br/>x โ D_Li"] --> B["Language-level Escape Noise Calibration<br/>Incremental noise sweep for transition boundary"]
B --> C["Vocabulary Soft Aggregation & Token Disambiguation<br/>Frequency weighting over shared subwords"]
C --> D["Directional Transition Probability & Symmetrization<br/>Bidirectional minimum TBLS matrix"]
D --> E["Output Language Similarity Matrix<br/>TBLS(Li, Lj) & Transfer Prediction"]
Key Designs¶
1. Language-level Escape Noise Calibration: Disparate cross-lingual robustness -> Incremental noise sweep -> Standardized operational scale
Because languages differ significantly in pretraining volume, subword fragmentation, and representation manifold compactness, applying a single uniform noise level \(\sigma\) across all languages would unduly overwhelm low-resource representations while barely perturbing high-resource ones. To address this disparity, the authors introduce the concept of sample-level and language-level escape noise. For any input \(x \in \mathcal{D}_{L_i}\), Gaussian noise added to intermediate activations is stepped upward until an external language identification (LID) system detects that the generated sequence leaves language \(L_i\). The minimal threshold is recorded as \(\sigma_{\text{esc}}(x)\), and averaging across the corpus yields the language-level escape noise \(\sigma_{L_i} = \frac{1}{|\mathcal{D}_{L_i}|} \sum_{x \in \mathcal{D}_{L_i}} \sigma_{\text{esc}}(x)\). Empirically, scaling the perturbation strength to this threshold consistently brings the labeled dependency overlap (LAS Overlap) down to an aligned baseline across all languages (as illustrated in Figure 9), proving that escape noise provides an equitable and comparable operating point.
2. Vocabulary Soft Aggregation and Token Disambiguation: Cross-lingual subword overlap -> Frequency-weighted projection and hard filtering -> Robust language probability mass estimation
Modern multilingual vocabularies (e.g., 250k tokens in XLM-R) contain numerous short subwordsโespecially in Latin and Cyrillic scriptsโthat appear across multiple genealogical families. Forcing a discrete one-to-one mapping for these shared tokens would inject substantial surface-level artifacts into transition measurements. To resolve this, TBLS adopts a hybrid aggregation strategy: for shared subwords \(\tilde{v}\) extracted from MC4, their contribution to language \(L_j\) is weighted softly by empirical occurrence frequencies \(w_{\tilde{v}, L_j} = f_{\tilde{v}, L_j} / \sum_k f_{\tilde{v}, L_k}\); for non-shared vocabulary items, hard language tags are assigned using XLM-V predictions, excluding non-linguistic tokens \(S\) (like padding and special symbols) and low-confidence tokens (\(<0.2\)). This soft aggregation allows the full output logit distribution at the first perturbed token to map smoothly and accurately into a language probability mass \(P(L_k \mid x, \sigma_{L_i})\).
3. Directional Transition Probability and Symmetrization: Asymmetric generation shifts -> Expected corpus transition with bidirectional minima -> Robust mutual language proximity
In generative and masked language models, shifting from a dominant high-resource language to a related low-resource language does not necessarily exhibit the same transition probability as the reverse process due to asymmetric vocabulary priors and frequency skews. To capture fine-grained transferability while enforcing a rigorous mathematical similarity metric, the method separates directional and symmetric formulations: the directional transition probability \(p_{i \to j} = \mathbb{E}_{x \sim \mathcal{D}_{L_i}} [P(L_j \mid x, \sigma_{L_i})]\) is retained to predict directional cross-lingual zero-shot transfer performance; simultaneously, the symmetric similarity is defined as the lower bound of mutual transition, \(\text{TBLS}(L_i, L_j) = \min(p_{i \to j}, p_{j \to i})\). Enforcing this minimum ensures that two languages are considered close only if they can mutually transition into each other under localized perturbations, effectively suppressing spurious one-sided artifacts.
Perturbation Mechanism and Similarity Metric¶
Perturbation is introduced directly at the output of feed-forward sublayers (FFN) at layer \(\ell\) during inference:
$$ \tilde{a}\ell = a\ell + \sigma \xi_\ell, \quad \xi_\ell \sim \mathcal{N}(0, I) $$
where the underlying model parameters \(\theta\) remain frozen, and Gaussian noise is sampled independently across layers and forward passes. For non-autoregressive encoder-only models such as XLM-R where sequence generation is absent, a deterministic left-to-right mask-and-fill procedure constructs pseudo-generated sentences \(\tilde{y}(x, \sigma)\) to evaluate language escape; for encoder-decoder models (mT5), single-token span infilling (<extra_id_0>) is applied; for decoder-only LLMs (mGPT, Qwen3-8B, Apertus-8B), greedy decoding is capped at 25 tokens. The language distribution over the effective vocabulary \(V' = V \setminus S\) is computed as:
$$ P(L_k \mid x, \sigma) = \sum_{v \in V_{L_k}} p_\theta(v \mid x, \sigma) $$
from which \(p_{i \to j}\) and \(\text{TBLS}(L_i, L_j)\) are derived.
Key Experimental Results¶
Main Results¶
The authors evaluated TBLS across five diverse multilingual language models (mGPT, mT5-Large, XLM-R Large, Qwen3-8B, Apertus-8B) on two benchmark corpora (FLORES and TATOEBA, spanning 45 languages with a high genealogical entropy score of 0.95). The table below reports the normalized leaf-distance between similarity-reconstructed hierarchical trees and the Glottolog gold genealogical tree (lower distance indicates superior genealogical alignment):
| Model | TBLS | REP (mPLM-Sim) | Centroid (Cosine) | ANC (Neuron Corr.) | Random Baseline (Mean ยฑ Std) |
|---|---|---|---|---|---|
| mGPT | 0.151 | 0.188 | 0.204 | 0.337 | 0.450 ยฑ 0.082 |
| mT5-Large | 0.074 | 0.128 | 0.125 | 0.203 | 0.450 ยฑ 0.082 |
| XLM-R Large | 0.138 | 0.072 | 0.075 | 0.123 | 0.450 ยฑ 0.082 |
| Qwen3-8B | 0.338 | 0.128 | 0.122 | 0.179 | 0.450 ยฑ 0.082 |
| Apertus-8B | 0.195 | 0.171 | 0.138 | 0.197 | 0.450 ยฑ 0.082 |
Cross-Model Consistency & Transfer Analysis¶
To assess metric stability across distinct model architectures and corpora, Representational Similarity Analysis (RSA) was conducted on the resulting language similarity matrices. The table below details average RSA across resource tiers and presents downstream zero-shot transfer correlations:
| Resource Tier / Evaluation Setting | (A) Same-Model Cross-Corpus RSA | (B) Cross-Model Same-Corpus RSA | (C) Cross-Model Cross-Corpus RSA | Cross-Script Transfer RSA (CS) |
|---|---|---|---|---|
| High-Resource Languages | 0.981 | 0.915 | 0.912 | - |
| Medium-Resource Languages | 0.887 | 0.678 | 0.647 | - |
| Low-Resource Languages | 0.965 | 0.870 | 0.867 | - |
| Global TBLS (MASSIVE Intent) | - | - | - | 0.440 (Token Overlap is NA) |
| Global TBLS (XNLI NLI) | - | - | - | 0.341 (Token Overlap is NA) |
| Global TBLS (WikiAnn NER) | - | - | - | 0.002 (Token Overlap is NA) |
| Global TBLS (UD POS Tagging) | - | - | - | 0.028 (Token Overlap is NA) |
Key Findings¶
- High Cross-Model Agreement: TBLS achieves remarkably consistent pairwise similarity structures across models and corpora, with RSA values ranging between 0.883 and 0.982 (overall average 0.882, Figure 12), substantially outperforming static representations such as REP (peak layer 0.487) and ANC (peak layer 0.241).
- Stability Across Resource Tiers: As shown in the resource breakdown, cross-model cross-corpus consistency is exceptionally high for both high-resource (0.912) and low-resource (0.867) languages, confirming that TBLS does not rely solely on high-frequency representation artifacts.
- Superior Token-Level Transfer Prediction: On token-level tasks like WikiAnn NER and UD POS, after excluding only the top 1โ2 lowest-escape target languages, TBLS matches or exceeds static similarity baselines in predicting zero-shot transfer performance. Furthermore, under cross-script conditions where tokenizer overlap degenerates into an uninformative constant vector (NA), TBLS preserves strong transfer signals (MASSIVE RSA 0.440, XNLI RSA 0.341).
- Syntactic Scaffolding Preservation: Positional and lexical edit analyses (Figures 7 & 17) demonstrate that over 60%โ90% of token mutations induced by activation perturbation occur on content words rather than function words, confirming that perturbation explores adjacent semantic neighborhoods while maintaining grammatical structure.
Highlights & Insights¶
- From Static Distance to Dynamical Probing: Shifting the paradigm from static cosine distances to perturbation-induced transition probabilities allows researchers to directly observe the dynamical flow of language boundaries in generative models.
- Escape Noise Normalization: Introducing escape noise establishes an objective operating point that normalizes cross-lingual comparisons across varying manifold densities and vocabulary tokenizations.
- Local Connectivity of Multilingual Semantics: The observation that local perturbations yield coherent translations while preserving sentence frames provides empirical behavioral support for the hypothesis that multilingual concepts occupy locally connected representation neighborhoods.
Limitations & Future Work¶
- Instability on Extremely Low-Escape Languages: Languages with marginal pretraining representation (e.g., Basque) can fail to maintain source generation even without perturbation, degrading the reliability of calibrated escape noise.
- Residual Tokenizer Bias: While MC4 frequency weighting mitigates subword ambiguity, shared Latin or Cyrillic characters still impart minor baseline probability mass between unrelated languages sharing a script.
- Inference Search Overhead: Calibrating escape noise necessitates multi-step iterative forward sweeps (averaging 445 passes for mGPT, 370 for mT5, and 155 for Qwen3 per example), which incurs higher computational cost than single-pass embedding extraction.
Related Work & Insights¶
- vs REP / mPLM-Sim: REP computes cosine similarity across aligned sentence representations, which is heavily sensitive to layer depth choice and displays poor cross-architecture agreement (RSA < 0.5); TBLS relies on first-token transition dynamics, attaining higher cross-model stability (RSA > 0.88).
- vs CentroidSim: Centroid similarity compares language-wide mean vectors; while relatively robust across corpora (RSA ~ 0.62), it collapses fine-grained manifold topology, whereas TBLS evaluates local transition barriers directly.
- vs ANC: Average Neuron-wise Correlation evaluates individual neuron activation correlations across languages, which is computationally sensitive to neuron-level noise; TBLS measures aggregate output probabilities, yielding better interpretability.
Rating¶
- Novelty: โญโญโญโญโญ Introduces a perturbation-driven dynamical transition perspective that breaks away from conventional static distance metrics.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 5 model families, 45 typologically diverse languages, 4 downstream transfer tasks, and thorough robustness checks.
- Writing Quality: โญโญโญโญโญ Clear exposition with seamless transitions from intuitive motivation to rigorous mathematical definitions and linguistic analyses.
- Value: โญโญโญโญโญ Offers valuable insights into multilingual representation geometry and practical guidance for cross-lingual transfer selection.