Skip to content

MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates

Conference: ICLR 2026 arXiv: 2510.05361 Code: None Area: Distributed Optimization / LLM Pre-training Keywords: Distributed Training, Adaptive Optimizer, Multi-timescale Momentum, Communication Efficiency, Local SGD

TL;DR

This paper proposes MT-DAO, a multi-timescale distributed adaptive optimizer that introduces a slow momentum (high \(\beta\)) to address the timescale mismatch caused by excessively rapid decay of standard momentum under infrequent communication. MT-DAO provides the first convergence guarantee in this setting, eliminates the performance gap with fully synchronized DDP in language model pre-training, and reduces end-to-end training time by 6–27%.

Background & Motivation

Communication bottleneck in Distributed Data Parallel (DDP): DDP requires gradient synchronization at every step, making communication the primary training efficiency bottleneck when network bandwidth is limited (e.g., cross-datacenter or Ethernet-interconnected clusters).

Performance gap in infrequent communication strategies (e.g., Local SGD): Synchronizing parameters every \(K\) steps greatly reduces communication, but applying this approach to adaptive optimizers such as Adam incurs a notable performance degradation relative to DDP. Charles et al. (2024) found that even with Nesterov momentum as the outer optimizer, performance lags behind DDP for models below 2.4B parameters with more than 2 workers.

Root cause of the performance gap: timescale mismatch: Standard Adam uses a fast momentum with \(\beta_1 \approx 0.9\), whose half-life is \(\tau_{0.5}(\beta) = \frac{\ln 0.5}{\ln \beta} \approx 6.6\) steps. When the communication interval \(K \gg \tau_{0.5}\) (e.g., \(K = 32\)), the influence of the global momentum decays to \(\beta^K \approx 0.03\) after \(K\) steps, forcing workers to rely on high-variance local gradients.

Directly increasing \(\beta\) is not viable: High-momentum optimizers are insufficiently sensitive to changes in the loss landscape and tend to oscillate, often performing worse in practice.

Multi-momentum methods offer a solution: Methods such as QHM, AggMo, and AdEMAMix have demonstrated that mixing fast and slow momenta can capture long-term memory without sacrificing responsiveness, yet they have not been extended to the distributed infrequent-communication setting.

Method

Overall Architecture

MT-DAO introduces multi-momentum optimizers into the distributed infrequent-communication setting. Each worker maintains \(N\) first-order momenta with distinct decay rates \(\beta_{1,j}\) and one second-order momentum. The parameter update direction is a convex combination of the current gradient and the \(N\) momenta (quasi-hyperbolic form). Parameters, momenta, and second-order states can be synchronized independently at different frequencies.

Key Designs

1. Multi-timescale momentum mixing

  • Function: Blends the current gradient (fast signal) with slow momentum (long-term memory) in the update direction.
  • Mechanism: Parameter update \(\Delta_t^m = \frac{1}{\sqrt{v_t^m} + \epsilon}\left[(1-\sum_{j=1}^N \omega_j)\hat{g}_t^m + \sum_{j=1}^N \omega_j u_t^{j,m}\right]\)
  • Design Motivation: Slow momentum (\(\beta \approx 0.999\)) retains global optimization direction information across synchronization intervals, while the fast gradient preserves responsiveness to local loss landscape changes.

The simplest form (QH, \(N=1\)) requires no additional memory overhead and introduces only one extra hyperparameter pair \((\omega_1, \beta_1)\).

2. Decoupled communication frequencies

  • Function: Allows parameters, each momentum, and the second-order state to be synchronized at different frequencies.
  • Mechanism: Parameters are synchronized every \(K_x\) steps, the \(j\)-th momentum every \(K_j\) steps, and the second-order state every \(K_v\) steps.
  • Design Motivation: Theoretical analysis shows that momenta with larger \(\beta\) are less sensitive to synchronization frequency, so slow momenta can be synchronized less often. Communication cost is reduced by a factor of \((1/K_x + \sum_{j=1}^N 1/K_j + 1/K_v)^{-1}\).

3. Mutual information retention analysis

  • Function: Quantifies, from an information-theoretic perspective, the global optimization signal retained by momentum across communication intervals.
  • Mechanism: \(I(U_{t+K}; U_t) = \frac{1}{2}\log\det(I + \beta^{2K}\Sigma_{U_t}\Sigma_L^{-1})\)
  • Design Motivation: When \(\beta^K \to 0\), mutual information vanishes (as with standard fast momentum at \(K=32\)); when \(\beta^K \to 1\), the global signal is preserved (as with slow momentum).

Loss & Training

Convergence guarantee (Theorem 1): Under standard non-convex smoothness assumptions, MT-DAO-SGDM achieves the optimal \(\mathcal{O}(1/\sqrt{T})\) asymptotic convergence rate. Key constants:

\[\beta_\omega = \sum_{j=1}^N \frac{\omega_j \beta_j}{1 - \beta_j}, \quad \psi = \frac{4(1-p_x)}{p_x^2}\sum_{j=1}^N \omega_j \frac{(1-\beta_j)(1-p_j)}{1-(1-p_j)\beta_j}\]
  • \(\beta_\omega\) constrains step size: larger \(\beta\) limits the step size.
  • \(\psi\) reflects the communication penalty: larger \(\beta\) reduces \(\psi\) (greater robustness to infrequent communication).
  • Distributed factors (client drift, data heterogeneity) are confined to higher-order \(\mathcal{O}(1/T)\) terms and do not affect the asymptotic rate.

Practical configuration: - Uses the ADOPT optimizer (an Adam variant) with \(\beta_2 = 0.9999\). - Default \(K = K_x = K_1 = K_v = 32\). - Uses CompleteP parameterization to transfer learning rates from a 16M model to larger models without re-tuning.

Key Experimental Results

Main Results

720M-parameter language model (SmolLM2 dataset), 4 H100 GPUs, Ethernet interconnect:

Method Performance (vs. DDP) Communication (vs. DDP)
ADOPT-DDP Baseline 1×
QHADOPT-DDP Slightly better than DDP 1×
Local ADOPT Below DDP 10× reduction
Nesterov outer optimizer Below DDP 10× reduction
MT-DAO Matches/exceeds DDP 10× reduction

Key figures for MT-DAO at 720M: - Reaches target perplexity with 24% fewer steps and 35% less time than single-momentum DDP. - Approximately 8% faster and 5% fewer tokens than QHADOPT-DDP. - End-to-end training time reduced by 6–27% depending on interconnect bandwidth.

Ablation Study

Effect of communication frequency on performance (16M model, \(K_1=K_v=16\)):

\(K_x\) Degradation (\(\beta_1=0.99\)) Degradation (\(\beta_1=0.995\))
32 +1.7% +1.0%
128 +3.9% +3.2%
512 +5.6% +3.4%
1024 +6.2% +3.7%

Higher \(\beta_1\) leads to less performance degradation as the communication interval increases.

Worker alignment (cosine similarity):

Metric MT-DAO Local ADOPT Nesterov
Local pseudo-grad ↔ global momentum >0.95 ~0.7 ~0.8
Local ↔ global pseudo-grad >0.95 ~0.7 ~0.85

Key Findings

  1. MT-DAO is the first to eliminate the performance gap between infrequent communication and DDP: At the 720M scale, MT-DAO not only matches but surpasses DDP.
  2. Slow momentum acts as a regularizer: It aligns worker update directions to a high degree (cosine similarity >0.95), reducing worker drift.
  3. Mutual information retention is critical: MT-DAO's slow momentum maintains statistical dependence on the global optimization direction across communication intervals, whereas the mutual information of standard fast momentum decays rapidly.
  4. Higher \(\beta\) = greater robustness to infrequent communication: Theoretical predictions are consistent with experiments; \(\beta=0.995\) degrades approximately 40% less than \(\beta=0.99\) under extreme infrequent communication (\(K=1024\)).
  5. The QH form (\(N=1\)) is the optimal practical choice: It adds no memory overhead, introduces only one extra hyperparameter, and already delivers sufficient performance.

Highlights & Insights

  1. The timescale mismatch diagnosis is precise and compelling: The problem is quantified from two perspectives—half-life and mutual information—providing a principled basis for the solution design.
  2. First convergence guarantee for distributed multi-momentum optimizers: The theoretical analysis reveals the unique advantage of slow momentum in distributed settings (insensitivity to synchronization frequency).
  3. Complementary relationship with AdEMAMix: AdEMAMix demonstrates the memory advantage of slow momentum in single-machine settings; MT-DAO brings this advantage into the distributed setting.
  4. Practical significance for cross-datacenter training: MT-DAO enables training over high-latency networks without quality loss, making it feasible to leverage geographically distributed GPU resources.

Limitations & Future Work

  1. Maximum scale is only 720M: Whether MT-DAO's advantages hold for models of 7B+ parameters and hundreds of GPUs remains to be verified.
  2. Evaluated only under IID data distributions: Performance may differ in federated learning scenarios with strong data heterogeneity (non-IID).
  3. Hyperparameter sensitivity: Although CompleteP can transfer learning rates, the optimal combination of \(\omega\) and \(\beta\) still requires search on small models.
  4. Joint use with gradient compression: Compatibility of MT-DAO with compression methods such as quantization and sparsification is unexplored (the paper notes this as a complementary direction).
  5. Evaluated only on language models: Applicability to vision models, multimodal architectures, and other model families has not been validated.
  • Local SGD/Adam (McMahan et al., 2017; Reddi et al., 2021): Classical distributed infrequent-communication frameworks; MT-DAO addresses their performance gap through multi-timescale momentum.
  • QHM (Ma & Yarats, 2018): The originator of quasi-hyperbolic momentum; MT-DAO extends it to the distributed setting.
  • AdEMAMix (Pagliardini et al., 2025): A single-machine multi-momentum optimizer demonstrating that slow momentum reduces forgetting; MT-DAO exploits this property to reduce worker drift in distributed training.
  • Charles et al. (2024): Diagnosed the performance gap of infrequent-communication Adam and improved but did not eliminate it with a Nesterov outer optimizer; MT-DAO fully closes this gap.
  • Insight: Timescale design may be an underappreciated dimension in distributed optimization; future work could explore adaptive timescales and hierarchical multi-timescale approaches.

Rating

  • Novelty: ⭐⭐⭐⭐ — Brings the multi-momentum idea into the distributed infrequent-communication setting with a clear diagnosis of timescale mismatch; however, the core idea builds upon QHM/AdEMAMix.
  • Experimental Thoroughness: ⭐⭐⭐⭐ — Systematic evaluation at three scales (16M/125M/720M); cosine similarity and mutual information visualizations strongly support the theoretical analysis; however, larger-scale validation is absent.
  • Writing Quality: ⭐⭐⭐⭐⭐ — The narrative flow from problem diagnosis → theoretical analysis → algorithm design → experimental validation is exceptionally smooth, with well-crafted figures and tables.
  • Value: ⭐⭐⭐⭐ — Directly applicable to communication-constrained distributed training, particularly well-suited for cross-datacenter and edge computing scenarios.