Skip to content

OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/nict-astrec-att/optimer
Area: Model Compression
Keywords: Continual Pre-Training, Distribution Vectors, Model Merging, Bayesian Optimization, Resource-Constrained Adaptation

TL;DR

Addressing the rigid pre-training bottleneck and prohibitive tuning overhead of data mixture ratios in continual pre-training, OptiMer decouples ratio selection from model training by extracting independent distribution vectors per dataset and searching continuous merge weights post-hoc via TPE Bayesian optimization, outperforming joint data mixing and uniform model averaging across 16 benchmarks with 15–35× lower search cost.

Background & Motivation

Adapting large language models (LLMs) to specialized languages and technical domains is indispensable for deploying capable models under strict computational and memory budgets on resource-constrained devices. Continual pre-training (CPT) represents the established paradigm for infusing target-domain knowledge and multilingual capabilities into base models. In standard CPT pipelines, the training corpus comprises a heterogeneous mixture of diverse datasets spanning general text, target languages, mathematics, and programming code. However, the data mixing ratio across these sources constitutes an exceptionally sensitive hyperparameter: an over-represented domain can trigger catastrophic forgetting of foundational reasoning, whereas an under-represented domain fails to instill meaningful task competence.

Existing data mixture optimization approaches—such as proxy-model estimation via DoReMi or offline regression prediction via RegMix—mitigate blind trial-and-error, but still require freezing the mixture ratio before launching full-scale distributed training. Once an expensive pre-training run on billions of tokens exhibits suboptimal downstream balance days or weeks later, the resulting checkpoint cannot be dynamically rectified, squandering extensive GPU cluster budgets and forcing practitioners to restart from scratch. Furthermore, conventional post-hoc model merging approaches that rely on naive uniform weight averaging suffer from destructive cross-distribution interference, precipitating severe syntactic degradation in sensitive domains like code synthesis.

To break the impasse between pre-training inflexibility and prohibitive retraining costs, this work approaches the problem from the geometry of model parameter trajectories. Because parameter shifts induced by distinct datasets remain approximately orthogonal in parameter space, the effective behavior of multi-data joint pre-training can be closely approximated by a linear combination of individual distribution vectors in the local tangent space. The core idea is to decouple data mixture ratio selection from continual pre-training by training independent models per dataset to extract distribution vectors, followed by post-hoc continuous merge weight optimization via TPE-based Bayesian optimization, eliminating retraining overhead while preserving base instruction alignment.

Method

Overall Architecture

OptiMer decomposes conventional joint continual pre-training into three decoupled phases: distribution vector extraction, multi-vector parameter space composition, and black-box Bayesian weight optimization. First, taking the pre-trained base model \(\theta_{pt}\) as an anchor, individual CPT models are trained independently on each dedicated dataset \(D_i\) to yield parameter shifts \(\tau_i\); an instruction vector \(\tau_{it}\) is simultaneously derived from an instruction-tuned checkpoint \(\theta_{it}\) to recover conversational alignment without additional supervised fine-tuning. Second, the extracted vectors are synthesized via a parameterized merge operator \(\theta_{merge}(\alpha)\) (e.g., DARE-Linear). Finally, a Tree-structured Parzen Estimator (TPE) iteratively optimizes continuous merge weights over lightweight proxy evaluation tasks, converging to the optimal configuration within minutes per candidate without requiring backward gradients.

The overall architecture and execution pipeline are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    PT["Pretrained Base Model θ_pt"] -->|Independent Single CPT| SingleModels["Domain CPT Checkpoints {θ_CPT,i} (Ja, Zh, Math, Code)"]
    SingleModels --> VectorExtract["Distribution Vector Extraction<br/>τ_i = θ_CPT,i − θ_pt and τ_it = θ_it − θ_pt"]
    VectorExtract --> LinearCompose["Multi-Vector Parameter Composition<br/>θ_merge(α) = θ_pt + α_it·τ_it + ∑ α_i·τ_i"]
    LinearCompose --> TPE["TPE Bayesian Optimization<br/>Partition via quantile γ into ℓ(α) and g(α), maximize ℓ/g"]
    TPE -->|Fast proxy evaluation feedback| LinearCompose
    TPE --> Out["Optimal Merged Target Model θ*_merge"]

Key Designs

1. Distribution Vector Extraction: Projecting continual pre-training into orthogonal parameter shifts

To eliminate conflicting gradient updates across heterogeneous datasets during joint pre-training, OptiMer generalizes the task vector formulation to continual pre-training, defining dataset-specific distribution vectors: $\(\tau_i = \theta_{CPTi} - \theta_{pt}, \quad \tau_{it} = \theta_{it} - \theta_{pt}\)$ Empirical geometric analysis reveals that pairwise cosine similarities between different CPT distribution vectors range strictly between 0.03 and 0.31, occupying mutually orthogonal subspaces within the overall parameter space. This near-orthogonality ensures that linear superposition incurs minimal cross-task disruption. Furthermore, extracting and injecting \(\tau_{it}\) enables the CPT model to immediately inherit strong instruction-following and safety capabilities without incurring secondary alignment fine-tuning that might overwrite newly acquired domain knowledge.

2. Multi-Vector Parameter Composition: Continuous relaxation and architecture preservation

Equipped with decoupled parameter displacements, OptiMer constructs a continuous composition manifold parameterized by weight vector \(\alpha = (\alpha_{it}, \alpha_1, \dots, \alpha_n)\): $\(\theta_{merge}(\alpha) = \theta_{pt} + \alpha_{it}\tau_{it} + \sum_{i=1}^n \alpha_i \tau_i\)$ To maintain vocabulary integrity and structural stability, composition is executed via DARE-Linear (Drop and REscale) using mergekit, explicitly freezing token embeddings (embed_tokens), the output projection (lm_head), and rotary position encodings (rotary). Along the local trajectory manifold, adjusting continuous weight \(\alpha_i\) acts analogously to calibrating the effective pre-training step budget on dataset \(D_i\), providing a smooth, fine-grained control knob post-training.

3. TPE Bayesian Optimization: Bypassing exponential search complexity via density-ratio modeling

Identifying the optimal weight combination \(\alpha^* = \arg\max_\alpha S(\theta_{merge}(\alpha), D_{dev})\) constitutes a derivative-free black-box optimization challenge. Discrete grid search requires \(O(G^{n+1})\) evaluations for \(G\) steps per dimension—prohibitive for 27B parameter models. OptiMer deploys the Tree-structured Parzen Estimator (TPE), partitioning evaluated candidate trials into upper and lower sets via a performance quantile threshold \(\gamma\), and modeling their parameter densities through two separate kernel estimators: $\(p(\alpha | S) = \begin{cases} \ell(\alpha), & S \ge s^* \\ g(\alpha), & S < s^* \end{cases}\)$ The next evaluation candidate is sampled by maximizing the density ratio \(\ell(\alpha)/g(\alpha)\). Because top-performing configurations concentrate within an extremely narrow hypervolume (with \(\alpha_{it}\) sharply bounded in \([0.5, 0.7]\)), TPE converges in \(O(10n)\) trials (typically within 100 trials). Each trial evaluates candidate weights on only 100 proxy validation samples (taking ~5 minutes), with the top-3 configurations subsequently confirmed on 300 samples, slashing search overhead by 15–35× relative to full mixture pre-training.

Benchmark & Evaluation Protocol

Experiments are anchored on Gemma 3 27B PT and Gemma 3 27B IT. Continual pre-training is executed on 1B token subsets from the LLM-jp Corpus v4 across Japanese, Chinese, Math, and Code using DeepSpeed ZeRO Stage 3 on an 8 × NVIDIA H200 (141GB) cluster for 1 epoch (2,000 steps). Evaluation is conducted with lm-evaluation-harness on a vLLM backend using a standardized 1-shot protocol across 16 benchmarks organized into five task suites: English (MMLU, ARC-Challenge, HellaSwag, TruthfulQA), Japanese Leaderboard (JA LB, 8 tasks averaged), Chinese (C-Eval), Math (GSM8K), and Code (HumanEval pass@1, MBPP). Generalization is further validated on Zh+Math, negative weight regularizations, and the Southeast Asian multilingual model Gemma-SEA-LION-v4-27B.

Key Experimental Results

Main Results

The primary evaluation comparing OptiMer against individual CPT models, uniform DataMix pre-training, and classical uniform model merging baselines (Task Arithmetic, TIES, DARE) is detailed below (transcribed directly from main Table 1):

Combination Method English (MMLU / TQA) Math (GSM8K) Code (HE / MBPP) Japanese (JA LB) Chinese (C-Eval) Overall Avg.
Baselines Gemma 3 27B PT 31.59 / 54.71 0.83 0.00 / 0.60 39.04 27.41 25.54
Gemma 3 27B IT 51.03 / 64.87 15.39 2.44 / 37.80 54.64 37.96 39.44
Ja + Math DataMix (\(\alpha_{it}=0.6\)) 73.17 / 49.57 83.62 50.00 / 68.80 73.34 61.74 67.86
Task Arithmetic 75.44 / 41.49 74.45 60.37 / 1.60 75.11 63.67 60.59
TIES 71.14 / 30.11 69.52 26.22 / 0.00 72.40 58.84 52.66
DARE-Linear 75.02 / 42.96 78.17 58.54 / 65.00 72.37 63.30 67.65
OptiMer (Ours) 73.17 / 55.20 84.46 59.76 / 70.40 72.53 63.45 69.98
Ja + Code DataMix (\(\alpha_{it}=0.6\)) 71.23 / 43.08 78.62 64.02 / 65.00 72.38 60.40 67.23
Task Arithmetic 72.33 / 42.72 78.17 45.12 / 60.20 74.04 61.96 65.26
DARE-Linear 72.33 / 42.84 79.08 46.34 / 60.60 74.07 62.18 65.58
OptiMer (Ours) 71.41 / 54.22 83.47 64.63 / 70.20 73.06 61.81 69.68
Ja + Zh + Math DataMix (\(\alpha_{it}=0.6\)) 73.83 / 37.70 79.61 32.32 / 65.40 72.77 62.18 63.71
Task Arithmetic 75.27 / 38.92 73.54 45.73 / 0.40 73.17 64.41 58.29
DARE-Linear 74.48 / 38.68 76.42 50.00 / 62.00 70.92 63.89 65.48
OptiMer (Ours) 72.94 / 51.77 83.93 67.07 / 70.20 72.65 63.22 70.37

Ablation Study

The sensitivity of the instruction vector weight \(\alpha_{it}\) (Table 6) and the regularizing impact of permitting negative merge weights (Table 4) are summarized below:

Configuration / Weight GSM8K (Math) HE (Code) MBPP (Code) JA LB (Ja) C-Eval (Zh) Overall Avg. Analytical Mechanism
\(\alpha_{it}\) Sensitivity Sweep Merging single English CPT (\(\alpha_{En}=1.0\)) with varying \(\alpha_{it}\)
\(\alpha_{it} = 0.0\) 29.42 0.00 16.40 39.02 54.23 42.11 Missing IT vector collapses instruction compliance
\(\alpha_{it} = 0.3\) 65.20 20.73 54.00 42.13 59.96 56.12 Basic alignment restored; complex tasks still underperform
\(\alpha_{it} = 0.6\) (Optimal) 80.82 66.46 67.60 42.11 62.11 66.36 Optimal balance preserving CPT knowledge and IT capability
\(\alpha_{it} = 0.8\) 80.82 28.05 70.00 49.43 58.62 61.31 Excessive IT weight dilutes domain-specific CPT gains
\(\alpha_{it} = 1.0\) 14.86 1.83 37.20 54.36 38.11 39.17 Overwritten by base IT parameters; sharp regression
Negative Weight Search Expanding search bounds from \([0, 1]\) to \([-1, 1]\)
Ja Objective \([0, 1]\) - - - 73.37 - - \(\alpha_{en}=0.129, \alpha_{math}=0.489, \alpha_{code}=0.033\)
Ja Objective \([-1, 1]\) - - - 73.85 - - \(\alpha_{math}=-0.060\), subtracting conflicting mathematical drift
Zh Objective \([0, 1]\) - - - - 63.60 - \(\alpha_{it}=0.450, \alpha_{zh}=0.034, \alpha_{en}=0.001\)
Zh Objective \([-1, 1]\) - - - - 65.68 - \(\alpha_{en}=-0.121, \alpha_{code}=-0.057\), actively canceling English interference

Key Findings

  • Eliminating Catastrophic Code Synthesis Failure: Standard model averaging methods (Task Arithmetic, TIES) severely degrade on code generation benchmarks (MBPP collapsing from 68+ down to 0.00–1.60). Inspection indicates that models do not hallucinate, but rather generate syntactically corrupted code lacking indentation; OptiMer maintains precise syntactic integrity, securing 70.20 on MBPP.
  • Robust Preservation of Base Factuality: On adversarial TruthfulQA evaluations, DataMix baselines degrade severely to 37–49 points and uniform merge baselines fall to 26–42 points. OptiMer consistently preserves factual calibration in the 51–55 range, outperforming all pre-training baselines by over 10 points.
  • Informing Future Data Mixtures (DataMixOptiMer ratio): Converting OptiMer's post-hoc weights back into proportional training mixtures and retraining a joint model yields substantial gains (e.g., Ja+Zh+Math improving from 63.71 to 68.66), proving that DataMix underperformance is predominantly driven by poor ratio selection rather than intrinsic limits of multi-task learning.
  • Interference Removal via Negative Weights: Broadening weight bounds to \([-1, 1]\) enables OptiMer to assign negative values to unrelated domains (e.g., \(\alpha_{en} = -0.121\) for Chinese evaluation), effectively neutralizing negative transfer and acting as a structural parameter regularizer.

Highlights & Insights

  • Post-Hoc Reformulation of Pre-Training Hyperparameters: By establishing that pre-training trajectories in parameter space are approximately linear, OptiMer reframes data ratio selection from an uncorrectable pre-training commitment into an agile post-hoc optimization, shifting how practitioners reason about continual pre-training.
  • Agile Domain Specialization from a Single Vector Pool: A single pool of independently pre-trained distribution vectors can be rapidly re-optimized for divergent objective functions (e.g., targeting Chinese, Japanese, or Mathematics) in under 10 minutes per search, delivering bespoke specialized models on demand without retraining.
  • Density-Ratio Search over Knife-Edge Optima: The optimization landscape reveals a sharp performance plateau: configurations with high accuracy are tightly concentrated within a narrow band of instruction weights. TPE's non-parametric density modeling navigates this sharp geometry with exceptional sample efficiency.

Limitations & Future Work

  • Divergence Under Large Token Budgets: The empirical validation focuses on 1B tokens per domain. For large-scale CPT spanning tens or hundreds of billions of tokens, parameters may diverge beyond the linear regime; iterative branch-train-merge extensions remain necessary.
  • Architecture and Quantization Generalization: Experiments were conducted exclusively on dense Gemma 3 27B and Gemma-SEA-LION models; validating scalability across Mixture-of-Experts (MoE) architectures and low-bit post-training quantization schemes remains open for future research.
  • Coupling with Upfront Ratio Predictors: Comparing or combining OptiMer with proxy predictive frameworks like DoReMi and RegMix could yield even tighter initialization priors for Bayesian search.
  • vs DataMix (Gururangan et al., 2020; Xie et al., 2023): Conventional data mixing statically fixes dataset ratios before multi-GPU training, risking weeks of wasted compute upon miscalibration; OptiMer trains isolated distribution vectors and finds the optimal balance post-hoc in minutes, achieving 15–35× lower search overhead.
  • vs Task Arithmetic & DARE (Ilharco et al., 2023; Yu et al., 2024): Existing merging frameworks primarily target supervised fine-tuning (SFT) adapters and rely on uniform averaging, which catastrophically breaks down on complex code generation at the 27B scale; OptiMer optimizes continuous non-uniform coefficients specifically for continual pre-training.
  • vs DEM (Ram et al., 2024): DEM relies on discrete grid search over task vectors, suffering from \(O(G^{n+1})\) exponential complexity; OptiMer utilizes TPE Bayesian optimization to achieve continuous convergence within \(O(10n)\) function evaluations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering post-hoc continuous distribution vector merging as an alternative to rigid data mixture pre-training]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive 27B scale evaluation covering 16 benchmarks, trajectory dynamics, geometric orthogonality, and case studies]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear problem formulation, rigorous mathematical grounding, and rich empirical insights]
  • Value: ⭐⭐⭐⭐⭐ [High practical impact for cost-efficient enterprise and resource-constrained adaptation of modern LLMs]