Skip to content

Evo: Autoregressive–Diffusion Large Language Models with Evolving Balance

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Architecture / Autoregressive–Diffusion Hybrid Modeling
Keywords: Autoregressive Models, Diffusion Models, Latent Flow, Semantic Progression, Adaptive Decoding

TL;DR

Evo establishes a mathematical duality between autoregressive and diffusion language generation under a continuous probability flow, using a learned token-wise semantic progression variable and a unified time-conditioned Transformer to adaptively balance coarse global planning with fine-grained local refinement at near-autoregressive inference speed.

Background & Motivation

Autoregressive (AR) Transformers have served as the foundational pillar of modern large language models, powering scalable pre-training and alignment via causal, token-by-token next-token prediction. However, this strictly unidirectional left-to-right decoding paradigm suffers from an inherent structural vulnerability: when faced with complex multi-step reasoning, mathematical proof construction, or structured code generation, the model must commit irrevocably to early prefix tokens without having formed a global semantic plan. Under greedy or approximate sampling heuristics, early local errors compound monotonically across the generation trajectory, leading to cascade failures and hallucination.

To overcome the rigid unidirectional bias of AR generation, recent efforts have explored diffusion-based language models that iteratively synthesize text through bidirectional denoising over corrupted latent or discrete token representations. By operating over the entire sequence simultaneously, diffusion models unlock non-sequential revision and flexible global coordination. Nevertheless, diffusion language models encounter severe operational hurdles: they demand dozens to hundreds of denoising iterations to converge, incurring prohibitive decoding latency; moreover, their lack of explicit causal factorizations frequently degrades local linguistic perplexity and token-level surface fidelity compared to mature AR architectures.

Recent hybrid attempts attempt to bridge this divide by stitching AR and diffusion modules together, such as applying block-wise diffusion (e.g., BD3-LM) or imposing fixed positional noise schedules (e.g., AR-Diffusion). Yet, these heuristics impose rigid, handcrafted structural boundaries that partition generation by arbitrary block indices or physical token positions rather than underlying semantic certainty. Core idea: autoregressive and score-based diffusion models are two continuous parameterizations of the same underlying probability flow in latent space, allowing generation to be formulated as an evolving latent trajectory where learned token-level progression variables adaptively allocate diffusion-style planning to uncertain semantic scaffolds and autoregressive refinement to mature tokens.

Method

Overall Architecture

Evo models the discrete target sequence \(X = (x_1, \dots, x_N)\) as a continuous latent flow over representations \(Z = \{z_i\}_{i=1}^N\) with \(z_i \in \mathbb{R}^d\). Each token position is assigned an evolving progression variable \(t_i \in [0, 1]\) that quantitatively indexes its semantic maturity: values near \(t_i \approx 1\) evoke stochastic diffusion-style global planning, while values near \(t_i \approx 0\) govern deterministic autoregressive refinement and token projection. Rather than utilizing distinct networks, Evo implements a single unified decoder-only Transformer with full-sequence bidirectional self-attention governed by a shared time-conditioned vector field \(F_\theta\).

During inference, initial latent vectors and progression variables are sampled from a prior, and the representations evolve deterministically along an ordinary differential equation (ODE) trajectory. Tokens that achieve early semantic maturity are truncated from further refinement, while uncertain tokens undergo extended multi-step updates before final projection into vocabulary space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Prompt Context and Latent Prior Distribution"] --> B["Latent Flow Duality Mapping<br/>Time reparameterization and probability flow equivalence"]
    B --> C["Continuous Latent Trajectory Evolution<br/>Full-sequence attention over shared velocity field"]
    C --> D["Semantic Progression Gating<br/>Two-layer MLP predicting token-wise maturity steps"]
    D --> E["Unified Variational ELBO Objective<br/>Joint likelihood bound linking NLL and score matching"]
    E --> F["Adaptive Latent Decoding Output<br/>Linear projection to target discrete token sequence"]

Key Designs

1. Latent Flow Duality Mapping: Unifying causal autoregression and diffusion under a shared probability flow

Prior paradigms typically treat discrete autoregressive factorization and continuous Gaussian diffusion as fundamentally incompatible generative philosophies, forcing hybrid models into unnatural block-level heuristics. Evo resolves this conceptual fracture by mathematically demonstrating that AR and diffusion represent different time discretizations of a singular probability flow in continuous space. Specifically, an autoregressive process over continuous completion scale \(s \in [0, 1]\) satisfies the neural ODE \(\frac{dx_s}{ds} = F_\theta(x_s, s)\). Meanwhile, the reverse-time dynamics of a variance-preserving diffusion SDE over diffusion time \(t \in [0, 1]\) yield a deterministic probability flow ODE: \(\frac{dz_t}{dt} = -\sigma(t)^2 \nabla_z \log p_t(z_t)\).

By defining an invertible, strictly monotonic time reparameterization \(s(t) \triangleq \int_0^t \sigma(\tau)^2 d\tau / \int_0^1 \sigma(\tau')^2 d\tau'\), the paper establishes that both processes trace the exact same geometric curves in latent space when the velocity field matches the scaled score function \(F_\theta(x_s, s) \equiv -C \nabla_x \log p_s(x_s)\). This theoretical proof confirms that autoregressive commitment corresponds to deterministic score-following near the origin, whereas diffusion decoding corresponds to stochastic score-following across noise regimes, providing a rigorous mathematical foundation for joint modeling.

2. Continuous Latent Trajectory Evolution: Time-conditioned feature dynamics via bidirectional Transformer states

To circumvent the information bottlenecks and gradient discontinuities associated with discrete masking or token replacement, Evo formulates generation as continuous trajectory updates governed by a learned velocity field. Each token's continuous latent state \(z_i(t)\) evolves over time according to:

\[\frac{dz_i(t)}{dt} = F_\theta(z_i(t), t_i)\]

Rather than enforcing a causal triangular attention mask, Evo deploys full-sequence bidirectional self-attention across all active tokens, enabling holistic contextual coordination throughout the evolution process. Continuous progression times \(t_i\) are mapped into high-dimensional embeddings via sinusoidal projections and added directly to token hidden representations. This conditioning allows the model's self-attention mechanism to modulate its behavior dynamically along the coarse-to-fine spectrum, synthesizing global semantic scaffolds while simultaneously sharpening local token representations within a single forward pass.

3. Semantic Progression Gating: Adaptive computation allocation via learned maturity prediction

Fixed-budget diffusion models uniformly update all sequence tokens across every denoising step, incurring immense computational redundancy on trivial tokens (such as punctuation, common functional words, and syntactic templates) while starving complex reasoning tokens of sufficient capacity. Evo introduces a lightweight, end-to-end learned progression head consisting of a two-layer MLP with GELU activations and a sigmoid mapping, operating directly on the initial latent representation of each token to predict \(t_i \in [0, 1]\).

During inference, given a maximum global refinement budget \(K_{\max} = 20\), the per-token update depth is dynamically allocated via \(K_i = \lfloor K_{\max} \cdot t_i \rfloor\). Tokens with low progression values converge and freeze early in the decoding process, while tokens with high values receive the full budget of iterative updates. This uncertainty-adaptive gating eliminates arbitrary block boundaries and permits near-instantaneous decoding for confident tokens, directly bridging the throughput gap between multi-step diffusion and single-pass autoregressive models.

4. Unified Variational ELBO Objective: Joint training of next-token prediction and score-based denoising

Training hybrid architectures with disjoint objectives often induces catastrophic representation drift between autoregressive generation and score matching. Evo derives a unified variational evidence lower bound (ELBO) on sequence marginal likelihood \(\log p_\Theta(X)\):

\[\log p_\Theta(X) \ge \mathbb{E}_{Z, t \sim q(Z, t \mid X)} \left[ \log p_\theta(Z, t) + \log p(X \mid Z) - \log q(Z, t \mid X) \right]\]

Discretizing the trajectory into \(K\) latent transition steps yields the operational flow prediction objective:

\[\mathcal{L}_{\text{Flow}}(\theta) = \sum_{t=0}^{K-1} \mathbb{E}_{X, Z} \left[ - \log p_\theta(z^{(t+1)} \mid z^{(t)}, t) \right]\]

During training, Gaussian noise is perturbed onto latent embeddings via a cosine variance schedule \(\alpha_t = \cos^2(\frac{\pi}{2} t)\). As \(t_i \to 0\), the variational objective smoothly reduces to standard autoregressive negative log-likelihood; as \(t_i \to 1\), it transitions into score-matching denoising. Annealing the relative weighting during warmup enables the single shared Transformer parameter set to smoothly integrate semantic planning and discrete lexical realization end-to-end.

Loss & Training

Evo is pre-trained on a comprehensive 3.4-trillion token multi-domain corpus encompassing web text, curated multilingual documents across 30+ languages, source code repositories, and mathematical publications. Optimization utilizes AdamW (\(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay 0.1) with a cosine learning rate schedule peaking at \(1.5 \times 10^{-4}\) after 3,000 warmup steps, a batch size of 4 million tokens, and a maximum sequence context of 4K tokens. Training is executed using DeepSpeed ZeRO-3 with pipeline and sequence parallelism on NVIDIA A100 GPU clusters. Supervised fine-tuning follows the post-training protocol of LLaDA, focusing on complex reasoning, symbolic mathematics, and instruction alignment.

Key Experimental Results

Main Results

The authors evaluate Evo 8B across 15 standard benchmarks in both pre-trained and post-trained configurations against leading open-source AR baselines (LLaMA3 8B, Qwen2.5 7B), diffusion-based LLMs (LLaDA 8B, MDLM 7B), and prior AR-diffusion hybrids (BD3-LM 7B, ARD 7B).

Pre-trained Benchmark Comparison (Extracted from Table 1):

Model E2E Latency (s) Speed (tokens/s) MMLU (5-shot) GSM8K (4-shot) MATH (4-shot) HumanEval (0-shot) MBPP (4-shot) CMMLU (5-shot)
Evo 8B 8.6 52 76.8 86.4 54.9 60.6 77.4 88.1
BD3-LM 7B 14.2 28 70.0 79.8 42.9 50.6 63.7 84.2
ARD 7B 32.5 12 46.2 15.1 3.6 13.4 18.9 33.1
LLaDA 8B 21.8 16 65.6 70.9 27.0 33.2 38.5 70.2
MDLM 7B 18.9 22 64.0 36.5 10.0 29.0 51.3 47.0
LLaMA3 8B 7.4 58 65.1 52.7 15.4 34.6 47.1 50.4
Qwen2.5 7B 8.1 46 74.5 85.8 50.2 58.3 75.2 -

Post-trained Benchmark Comparison (Extracted from Table 2):

Model Alignment E2E Latency (s) Speed (tokens/s) MMLU (5-shot) ARC-C (0-shot) GSM8K MATH (0-shot) HumanEval (0-shot) MBPP
Evo 8B SFT 8.6 52 78.6 92.5 89.3 (4) 78.8 86.7 81.1 (4)
BD3-LM 7B SFT+RL 14.2 28 75.1 86.7 85.7 (0) 74.3 82.9 67.2 (0)
ARD 7B SFT+RL 32.5 12 48.9 56.9 31.0 (4) 15.6 16.5 20.6 (4)
LLaDA 8B SFT 21.8 16 65.5 88.5 78.6 (4) 26.6 47.6 34.2 (4)
MDLM 9B SFT+RL 18.9 22 70.2 - 76.7 (0) 44.3 68.9 74.9 (0)
LLaMA3 8B SFT+RL 7.4 58 68.4 82.4 78.3 (4) 29.6 59.8 57.6 (4)
Qwen2.5 7B SFT+RL 8.1 46 73.8 - 91.6 (0) 75.5 84.8 79.2 (0)

Ablation Study

The authors perform extensive ablations isolating the contributions of time conditioning, progression step scheduling, and parameter sharing strategies on validation perplexity (PPL) and average downstream accuracy across MMLU and GSM8K.

Effect of Time Conditioning and Progression Steps (Extracted from Figure 3):

Ablation Variant Validation PPL (\(\downarrow\)) Task Acc Avg (\(\uparrow\)) Key Observations
Evo (Full Model) 8.1 81.6 Unified latent flow with adaptive learned progression times \(t_i\)
w/o Time Conditioning 9.6 69.1 Eliminating time embeddings prevents model from sensing trajectory maturity
Fixed \(t_i = 0\) (Pure AR mode) 10.2 69.6 Strictly unidirectional causal decoding; worst PPL and severe reasoning drops
Fixed \(t_i = 1\) (Pure Diffusion mode) 8.6 77.0 Full iterative bidirectional denoising; high computational overhead and suboptimal PPL
Fixed \(t_i = 0.5\) (Uniform schedule) 8.8 77.9 Rigid midpoint compromise; fails to allocate computation adaptively
Heuristic \(t_i\) (AR token entropy) 8.4 78.2 Heuristic entropy reflects local uncertainty but misses long-range planning needs

Architecture and Pre-training Design Choices (Extracted from Figure 4):

Architecture Variant Validation PPL (\(\downarrow\)) Task Acc Avg (\(\uparrow\)) Key Observations
Evo (Full Model) 8.1 81.6 Single shared Transformer parameter set across all evolution steps
Two-Module (Separate AR + Diff) 8.6 77.9 Disjoint networks create representational divergence across generation stages
No Shared Embeddings 8.4 78.3 Independent embeddings degrade multi-step semantic alignment along trajectories
No Pre-trained Initialization 9.1 76.2 Cold-start training suffers from slower convergence and reduced final accuracy
Frozen AR Backbone 8.7 80.4 Validates that Evo can serve as an efficient adaptation layer for frozen AR models

Key Findings

  • Substantial Breakthrough in Mathematical and Code Reasoning: Pre-trained Evo 8B achieves 54.9 on MATH and 60.6 on HumanEval, outpacing LLaMA3 8B (+39.5 on MATH, +26.0 on HumanEval) and the strongest hybrid baseline BD3-LM 7B (+12.0 on MATH, +10.0 on HumanEval). The bidirectional latent flow allows the model to lay down symbolic outlines before finalizing fine-grained algebraic and syntactic tokens.
  • Decoding Speed on Par with Pure Autoregressive Models: Despite incorporating multi-step diffusion refinement, Evo 8B delivers an end-to-end latency of 8.6 seconds (52 tokens/s), closely tracking LLaMA3 (7.4s, 58 tokens/s) and Qwen2.5 (8.1s, 46 tokens/s), while running nearly \(2\times\) faster than BD3-LM (14.2s) and \(2.5\times\) faster than full-sequence diffusion LLaDA (21.8s).
  • Superiority of Learned Progression over Heuristic Rules: The ablation demonstrates that end-to-end learned progression steps \(t_i\) via gradient descent outperform entropy-based heuristic gating (81.6% vs. 78.2%), confirming that true semantic uncertainty cannot be captured solely by next-token softmax sharpness.

Highlights & Insights

  • Theoretical Unification via Time Reparameterization: The paper formally proves Proposition 1, establishing that autoregressive causal generation and score-based diffusion probability flows are mathematically equivalent under a smooth monotonic transformation of time, unifying two dominant paradigm families within a shared ODE trajectory framework.
  • Continuous Latent Flow with Semantic Gating: By parameterizing progression via a continuous variable \(t_i\) rather than discrete token masking, Evo allows tokens with differing semantic entropy to coexist in the same bidirectional attention layer while executing variable refinement depths.
  • Plug-and-Play Adaptability to Existing Backbones: The ablation showing that freezing the base AR backbone retains 80.4% task accuracy indicates that continuous latent flow evolution can be retrofitted onto existing pre-trained LLM checkpoints as an efficient post-training upgrade.

Limitations & Future Work

  • Higher Pre-training Computational Footprint: Simulating continuous latent trajectory flows and calculating variational ELBO gradients introduces additional backward pass overhead, rendering Evo moderately more compute-intensive to pre-train from scratch than standard causal Transformers.
  • Controllability in Unconstrained Open-Ended Generation: Because Evo evolves an abstract semantic scaffold bidirectionally rather than strictly proceeding left-to-right, enforcing strict output length constraints or exact lexical matching in open-ended creative tasks poses greater optimization challenges.
  • Future Directions: Extending continuous progression flow modeling to multimodal vision-language generation, and formulating trajectory-level reinforcement learning objectives (such as flow-based PPO or DPO) to steer intermediate planning dynamics.
  • vs. BD3-LM: BD3-LM relies on discrete block-wise diffusion, enforcing strict autoregressive generation between blocks and fixed denoising iterations within blocks; Evo eliminates rigid block boundaries, operating continuously over the full sequence with token-adaptive compute allocation.
  • vs. AR-Diffusion (ARD): ARD applies a monotonic position-dependent noise schedule that forces later tokens to undergo more denoising regardless of semantic difficulty; Evo predicts maturity dynamically based on semantic ambiguity rather than linear sentence position.
  • vs. LLaDA / MDLM: Pure diffusion LLMs apply uniform iterative denoising to the entire sequence, incurring severe latency penalties and degraded perplexity; Evo transitions mature tokens rapidly into deterministic autoregressive refinement, achieving near-AR decoding speeds.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates a rigorous continuous probability flow duality between AR and diffusion models, replacing ad-hoc hybrid heuristics with a principled mathematical framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 15 standard benchmarks, 5 parameter scales, full-stack latency benchmarking, and deep structural ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical exposition, structured narrative flow, and well-designed experimental methodology.
  • Value: ⭐⭐⭐⭐⭐ Provides a viable, high-throughput architectural path forward to mitigate autoregressive compounding errors in complex reasoning and code synthesis.