Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Output Homogeneity, Semantic Convergence, Instruction Tuning, Base Models, Latent Preferences
TL;DR¶
By systematically tracking full alignment stages, conducting sample-level controlled SFT intervention experiments, and directly probing unaligned base models via prompting, this work reveals that language model output homogeneity (semantic convergence) is not introduced by post-training preference alignment, but is an inherent latent property already encoded during pretraining, where SFT merely serves as an amplifying catalyst.
Background & Motivation¶
Language models (LMs) routinely suffer from an acute lack of output diversity in open-ended generation tasks, collapsing onto narrow semantic spaces and generating repetitive, highly predictable responses across independent samplings. The community has long attributed this semantic convergence to the post-training alignment pipeline—specifically supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning with human or verifiable feedback (RLHF/RLVR)—under the assumption that human preference optimization and strict safety guardrails inevitably squash distributional entropy and homogenize generation.
However, prevailing arguments that "alignment destroys diversity" overlook the granular phase transitions within the alignment pipeline itself and fail to rigorously disentangle whether semantic collapse is genuinely introduced by alignment or merely unmasked from pretraining distributions. If convergence is indeed introduced by post-training preference learning, then diversifying preference datasets or relaxing reward constraints should naturally mitigate the collapse. Conversely, if semantic convergence is an inevitable byproduct of foundational pretraining objectives, post-alignment interventions alone will prove fundamentally insufficient to resolve the issue.
To decouple the causal effects of alignment procedures from pretraining priors, this paper investigates the lifecycle of output homogeneity using open-ended queries and metaphor generation as delicate diagnostic probes. Through three rigorous experimental axes—cross-stage alignment tracking, sample-level SFT interventions, and unsupervised base model prompting—the authors systematically test the origin and persistence of semantic collapse. Core idea: Semantic convergence originates during pretraining rather than post-training; SFT and preference alignment merely act as catalysts that reveal and amplify latent base model preferences, while SFT data remains incapable of introducing out-of-distribution convergent patterns.
Method¶
Overall Architecture¶
The investigation is structured around three progressive and causal evaluation modules. First, the pipeline tracks output distributions across full alignment trajectories (SFT \(\to\) DPO \(\to\) RLVR) on open-ended queries to pinpoint the exact onset of semantic collapse. Second, it conducts controlled sample-level interventions during SFT by injecting structural formats and conceptual metaphors (spanning high-frequency, low-frequency, and out-of-distribution semantic vehicles) to determine whether SFT injects new convergent modes or merely magnifies pre-existing priors. Finally, it probes raw, unaligned base models under varying prompting conditions (Basic completion, Few-shot examples, and Assistant persona simulation) to verify whether instruct-like semantic collapse can emerge purely from pretraining representations without any gradient updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Open-Ended Queries & Metaphor Probes<br/>INFINITY-CHAT100 and 5 metaphor topics"] --> B["Full Alignment Stage Tracking<br/>Evaluate SFT / DPO / RLVR cosine similarity & rank correlation"]
B --> C["Sample-Level Controlled SFT Interventions<br/>Structural format enforcement & plausible / OOD vehicle injection"]
C --> D["Base Model Latent Mode Elicitation<br/>Basic / Few-shot / Assistant persona prompting"]
D --> E["Causal Empirical Conclusion<br/>Convergence is latent in pretraining; alignment acts as catalyst"]
Key Designs¶
1. Full Alignment Stage Tracking: Pinpointing the Onset of Semantic Collapse
Addressing the open debate regarding where in the alignment pipeline diversity is lost, this design tracks semantic similarity across the three canonical alignment checkpoints: SFT, DPO, and RLVR. Evaluations are performed on 100 open-ended prompts from the INFINITY-CHAT100 benchmark across four model families: OLMo 3 (7B, 32B) and Tülu 3 / 3.1 (8B, 70B). Sampling 50 responses per prompt (\(T=1.0, p=0.9\)) and encoding texts via OpenAI's text-embedding-3-small, the protocol measures mean pairwise cosine similarity and lexical diversity metrics (1-distinct-1/2, Self-BLEU). Furthermore, it computes the Spearman rank correlation \(\rho\) of per-prompt mean cosine similarity between consecutive alignment stages. The empirical distributions show that semantic similarity shifts heavily rightward immediately after SFT (jumping to mean cosine values of 0.70–0.81), with subsequent DPO and RLVR stages causing only marginal increases. Strikingly, per-prompt convergence rank correlations range from 0.870 to 0.994, proving that post-SFT preference optimization does not alter which prompts converge, but merely preserves and reinforces the homogeneity established in the very first alignment stage.
2. Sample-Level Controlled SFT Interventions: Disentangling Format Exposure from Conceptual Memorization
To determine whether SFT actively introduces convergence or simply unlocks latent pretraining preferences through format learning, this design conducts point-edited SFT interventions on a curated, single-turn subset of the LIMA dataset (950 examples) using Llama-3.1-8B with LoRA (\(r=64, \alpha=64\)). The study focuses on metaphor generation across five abstract topics: {time, nostalgia, doubt, language, hope}. The experiment executes two phases: first, a "Structure Injection" baseline appends 30 metaphor instruction-response pairs formatted as [topic] is a [vehicle]... without containing any of the five evaluated topics or their canonical vehicles. Second, an "Idea Injection" phase introduces single instances of specific vehicles into the training set under four independent conditions: the 1st, 2nd, and 6th most frequent vehicles identified in the structure baseline, alongside completely implausible out-of-distribution (OOD) food-related vehicles (e.g., mapping time to a sandwich or doubt to sushi). If SFT induces convergence via rote memorization, models should reproduce injected OOD vehicles at test time; if convergence is latent, injection should only amplify vehicles aligned with the base distribution while failing to absorb out-of-distribution concepts.
3. Base Model Latent Mode Elicitation: Surfacing Collapse via Prompting Alone
To eliminate any residual confounding factors such as training data leakage during fine-tuning, this design directly interrogates raw, unaligned base models (Llama-3.1-8B, OLMo-3-7B, Qwen3-8B) using three distinct prompting paradigms: Basic completion (Here is a metaphor about [topic]: [topic] is ...), Few-shot in-context learning (prepending three canonical, non-overlapping metaphors), and Assistant persona prompting (Sure! Happy to help. Here is a metaphor about [topic]: [topic] is ...). Outputs are constrained to 30 new tokens to maintain focus and analyzed via PCA projections and pairwise cosine similarities. This design establishes that without any parameter updates or alignment gradients, activating an assistant persona or providing common in-context formatting causes base models to collapse onto the exact same preferred vehicles (e.g., river and thief for time) observed in fully aligned checkpoints.
Loss & Training¶
All SFT intervention runs are trained with standard autoregressive cross-entropy loss on a single NVIDIA A6000 GPU in bfloat16 precision. Training hyper-parameters are uniformly set across runs: effective batch size of 64 (per-device batch size 2, gradient accumulation 32), learning rate \(1\times 10^{-4}\) with a linear schedule and 15 warmup steps, weight decay 0.1, trained over 15 epochs. LoRA is applied to the projection layers (q, k, v, o) with rank \(r=64\) and alpha \(\alpha=64\). Checkpoints are selected at epoch 14 based on response quality scores evaluated by the Tülu 3 reward model on the LIMA validation set. Supplementary experiments confirm that full parameter fine-tuning yields identical empirical dynamics.
Key Experimental Results¶
Main Results¶
The evolution of pairwise cosine similarity and lexical diversity across consecutive alignment stages on the INFINITY-CHAT100 benchmark is detailed in the table below:
| Model | Metric | SFT | DPO | RLVR / Instruct | Consecutive Stage Rank Correlation Median \(\rho\) |
|---|---|---|---|---|---|
| OLMo 3 7B | Cosine Similarity | 0.790 | 0.822 | 0.831 | 1.000 |
| OLMo 3 7B | 1 − distinct-1 | 0.788 | 0.799 | 0.818 | 0.730 |
| OLMo 3 7B | 1 − distinct-2 | 0.479 | 0.490 | 0.531 | 0.440 |
| OLMo 3 7B | Self-BLEU | 0.469 | 0.480 | 0.525 | 0.310 |
| OLMo 3 32B | Cosine Similarity | 0.812 | 0.837 | 0.837 | 1.000 |
| OLMo 3 32B | 1 − distinct-1 | 0.822 | 0.831 | 0.844 | 0.710 |
| OLMo 3 32B | 1 − distinct-2 | 0.553 | 0.569 | 0.596 | 0.330 |
| OLMo 3 32B | Self-BLEU | 0.562 | 0.580 | 0.619 | 0.190 |
| Tülu 3.1 8B | Cosine Similarity | 0.706 | 0.802 | 0.807 | 1.000 |
| Tülu 3.1 8B | 1 − distinct-1 | 0.765 | 0.806 | 0.813 | 0.770 |
| Tülu 3.1 8B | 1 − distinct-2 | 0.443 | 0.498 | 0.518 | 0.590 |
| Tülu 3.1 8B | Self-BLEU | 0.415 | 0.484 | 0.506 | 0.400 |
| Tülu 3 70B | Cosine Similarity | 0.732 | 0.819 | 0.815 | 1.000 |
| Tülu 3 70B | 1 − distinct-1 | 0.789 | 0.820 | 0.817 | 0.780 |
| Tülu 3 70B | 1 − distinct-2 | 0.485 | 0.524 | 0.524 | 0.560 |
| Tülu 3 70B | Self-BLEU | 0.466 | 0.513 | 0.513 | 0.380 |
Ablation Study¶
Ablation of idea injection under Full Supervised Fine-Tuning (Full SFT) on Llama-3.1-8B, tracking dominant vehicle generation frequency (%) and the adoption rate of injected implausible vehicles:
| Metaphor Topic | Dominant Vehicle | Baseline Structure SFT (%) | Plausible Vehicle Injection (%) | Implausible (Food) Vehicle Injection (%) | Injected OOD Vehicle Adoption Rate (%) |
|---|---|---|---|---|---|
| time | river | 57 | 90 (+33) | 39 (−18) | 0% |
| nostalgia | photograph | 79 | 75 (−4) | 32 (−47) | 0% |
| doubt | shadow | 20 | 39 (+19) | 16 (−4) | 0% |
| language | map | 31 | 59 (+28) | 18 (−13) | 0% |
| hope | seed | 30 | 60 (+30) | 26 (−4) | 0% |
Decoding parameter sensitivity ablations (\(T \in [1.0, 1.2], p \in [0.7, 1.0]\)) confirm that dominant vehicle identity remains strictly invariant across 19 out of 20 tested configurations. Mean dominant vehicle frequency across topics reaches 48% under conservative decoding (\(T=1.0, p=0.7\)) and remains robust at 28% even under highly permissive decoding (\(T=1.2, p=0.9\)), rising to 78% and 56% respectively following plausible vehicle injection.
Key Findings¶
- SFT marks the true inflection point for convergence: Pairwise cosine similarity surges dramatically during SFT (reaching 0.70–0.81), whereas DPO and RLVR introduce only minimal additional increases (0.01–0.09). This definitively refutes the assumption that preference optimization and RL are the primary drivers of output homogenization.
- SFT amplifies pre-existing modes but cannot introduce new ones: Injecting the top pre-existing vehicle during SFT boosts its generation rate up to 48%–92%. In contrast, injecting implausible OOD food metaphors yields a strict 0% adoption rate across all topics; models steadfastly retain their intrinsic latent vehicles. Only through extreme repetition (5–7 repeated training instances) can an OOD vehicle be forced into generation.
- Prompting raw base models replicates aligned collapse: In unaligned base models, activating an assistant persona through prompting alone creates tight PCA semantic clusters (e.g.,
riverandthieffor time) and high intra-vehicle cosine similarities (0.69–0.91), matching the behavior of instruction-tuned and RL-aligned models.
Highlights & Insights¶
- Instruction tuning operates as a catalyst, not a cause: SFT does not instill preferences for specific conceptual associations; instead, by enforcing canonical response templates and assistant personas, it unlocks the densest, most probable paths already carved into the pretraining distribution.
- Data diversification during SFT cannot cure homogeneity: Because instruction-tuned models filter out or reject patterns that contradict pretraining distributions, merely introducing diverse or novel examples into SFT data will fail to shift the model's fundamental convergent tendencies.
- Corpus raw token frequency is decoupled from conceptual convergence: Preliminary corpus investigations indicate that the vehicles language models converge upon do not linearly track raw word frequencies in pretraining corpora, pointing to more intricate multi-token semantic aggregation during foundational pretraining.
Limitations & Future Work¶
- Domain scope constrained to metaphor and short-form queries: The causal analysis relies primarily on five metaphor generation topics and open-ended chatbot prompts. While conceptually clean and highly informative, extending these causal probes to complex multi-step reasoning, mathematical derivation, and long-form coding remains open for exploration.
- Synthetic proxy setup for SFT: Controlled SFT interventions were conducted on the 1,000-sample LIMA dataset with Llama-3.1-8B. While isolating individual sample effects cleanly, production-grade alignment pipelines leverage millions of heterogeneous samples where subtle multi-task interactions might occur.
- Pretraining mechanisms remain a black box: Although the paper establishes that convergence originates in pretraining, the exact interplay between pretraining loss formulations, data filtering curricula, and transformer inductive biases that crystallize these latent modes requires further mechanistic study.
Related Work & Insights¶
- vs Jiang et al. (2025) (Artificial Hivemind): Jiang et al. demonstrated pervasive open-ended homogeneity across aligned models. This work substantially deepens the diagnosis by proving causally that hivemind convergence is not an artifact of alignment, but an explicit manifestation of latent pretraining structures.
- vs Chu et al. (2025) (SFT Memorizes, RL Generalizes): Chu et al. posited that SFT fundamentally memorizes training examples. This paper nuances that thesis by showing that memorization in SFT is heavily gated by prior knowledge: models effortlessly absorb in-domain patterns while strictly resisting out-of-distribution examples under standard tuning budgets.
- vs West & Potts (2025) & Murthy et al. (2025): While prior literature argued that alignment actively degrades creativity and randomness, this study shows that unaligned base models possess identical latent convergence under matching prompt constraints, revealing that alignment merely crystalizes a pre-existing latent state.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Overturns the dominant consensus regarding alignment-induced collapse by establishing the SFT catalyst hypothesis through meticulous causal interventions.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Combines cross-stage lifecycle tracking, sample-level point editing, full parameter fine-tuning, decoding sweeps, and base model persona elicitation into a watertight empirical argument.
- Writing Quality: ⭐⭐⭐⭐⭐ Elegantly structured, crystal-clear scientific narrative, and exceptionally precise articulation of hypotheses and results.
- Value: ⭐⭐⭐⭐⭐ Provides critical foundational insights for generative model diversity, synthetic data recursive collapse, and future pretraining data curation strategies.