Skip to content

Evo: Autoregressive–Diffusion Large Language Models with Evolving Balance

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Text Generation
Keywords: Autoregressive Models, Diffusion Language Models, Continuous Probability Flow, Evolving Balance, Semantic Maturity

TL;DR

This paper introduces Evo, a duality latent flow model that unifies autoregressive next-token prediction and diffusion denoising into a continuous probability trajectory, adaptively balancing global planning and local refinement via learned per-token progression variables while preserving near-AR decoding throughput (52 tokens/s).

Background & Motivation

Autoregressive (AR) Transformers dominate contemporary natural language processing, delivering exceptional scalability across foundation pretraining and post-training alignment. However, their strictly sequential, left-to-right factorization imposes an unyielding causal bottleneck: early decoding deviations or heuristic mistakes inevitably trigger compounding error cascades, as the model lacks native mechanisms to revisit or revise previously committed tokens. Conversely, diffusion language models synthesize sequences by iteratively denoising corrupted inputs across the entire sequence via bidirectional attention, demonstrating natural strengths in non-sequential global planning, holistic coordination, and iterative refinement. Nevertheless, standard diffusion models typically suffer from prohibitive decoding latency, requiring tens or hundreds of sampling steps, and consistently trail comparable AR architectures in perplexity and fine-grained local syntax realization.

Prior attempts to construct hybrid AR–diffusion models predominantly rely on rigid structural stratifications. For instance, block diffusion architectures (e.g., BD3-LM) enforce an autoregressive sequence of blocks while executing fixed-step diffusion denoising within each block, squandering extensive computation on segments that are already semantically finalized. Similarly, position-dependent schedules such as AR-Diffusion artificially force later token indices to undergo more denoising steps regardless of actual semantic certainty. The fundamental dilemma of these hybrid designs lies in their artificial bifurcation into static stages or fixed chunk boundaries, which prevents the generative system from dynamically allocating computational depth according to the genuine semantic maturity of individual concepts.

To break this impasse, the authors trace generative modeling back to its continuous dynamical roots, proving that monotonic AR generation paths and reverse-time probability flow ODEs in score-based diffusion models are exact mathematical reparameterizations of the same underlying continuous latent flow. Core idea: formulate language generation as an evolving latent flow governed by token-specific progression variables \(t_i \in [0, 1]\) representing semantic maturity, adaptively balancing coarse diffusion planning and fine-grained autoregressive refinement within a single, end-to-end trained Transformer.

Method

Overall Architecture

Evo conceptualizes sequence generation as a continuous flow in latent representation space. For a target sequence of \(N\) tokens \(X = (x_1, \dots, x_N)\), the model maintains a collection of latent representations \(Z = \{z_i\}_{i=1}^N \subset \mathbb{R}^d\) coupled with learned continuous progression time variables \(\{t_i\}_{i=1}^N \subset [0, 1]\). The progression variable \(t_i\) measures the semantic maturity of token \(x_i\): high values (\(t_i \approx 1\)) invoke stochastic diffusion-style semantic exploration and global scaffolding, whereas low values (\(t_i \approx 0\)) transition into confident, deterministic autoregressive token realization. Generation proceeds via a single shared Transformer equipped with full-sequence bidirectional self-attention that integrates progression time embeddings and evaluates a unified vector field \(F_\theta\). Upon trajectory completion, final latent representations are projected onto the discrete token vocabulary.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Prompt Input & Latent Prior Initialization"] --> B["AR-Diffusion Duality Probability Flow<br/>Establish unified continuous latent ODE"]
    B --> C["Semantic Maturity Progression Sensing<br/>Predict token progression variables ti ∈ 0,1"]
    C --> D["Evolving Balance Latent Trajectory Modeling<br/>Shared vector field Fθ multi-step updates"]
    D --> E["Adaptive Truncation Decoding & Projection<br/>Ki = ⌊Kmax · ti⌋ early stop and token realization"]

Key Designs

1. AR-Diffusion Duality Probability Flow: Formulating Continuous Dynamical Equivalence

To bridge the conceptual divide between autoregressive and diffusion paradigms, the authors formulate both as discrete approximations of a shared continuous probability flow. Let \(x_s\) denote the continuous sequence realization at generation scale \(s \in [0, 1]\), evolving under the deterministic ODE \(\frac{dx_s}{ds} = F_\theta(x_s, s)\). Simultaneously, the deterministic probability flow ODE of a variance-preserving (VP) score-based diffusion model with marginal variance schedule \(\sigma(t)\) satisfies \(\frac{dz_t}{dt} = -\sigma(t)^2 \nabla_z \log p_t(z)\). By introducing a smooth, strictly monotonic temporal reparameterization:

\[s(t) = \frac{\int_0^t \sigma(\tau)^2 d\tau}{\int_0^1 \sigma(\tau')^2 d\tau'}\]

Applying the chain rule directly confirms that the reverse diffusion probability flow matches the functional form of the continuous AR flow, where the generative vector field aligns identically with the scaled score function. Hence, autoregressive next-token prediction corresponds to the zero-noise, deterministic limit near the trajectory origin, while diffusion represents score-following dynamics across noisy semantic manifolds. This formal duality establishes that AR and diffusion represent opposite poles of a continuous generation continuum, allowing both to be naturally unified within a shared parameter space.

2. Evolving Balance Latent Trajectory Modeling: Adaptive Token-Wise Computation Allocation

Building upon the unified flow formulation, Evo eliminates rigid block boundaries and replaces them with an adaptive token-level progression mechanism. Built upon a decoder-only Transformer backbone, Evo removes the causal attention mask in favor of full-sequence bidirectional attention during latent updates, ensuring rich contextual coordination across all stages of generation. Each token's initial latent vector \(z_i^{(0)}\) passes through a two-layer MLP with GELU activations and a Sigmoid output to predict its progression time \(t_i\). This progression scalar is subsequently mapped into the network through sinusoidal embeddings added to intermediate hidden representations. Latent updates follow a discretized ODE over step size \(\Delta t\):

\[z_i^{(t+1)} = z_i^{(t)} + \Delta t \cdot F_\theta(z_i^{(t)}, t_i)\]

For deterministic or syntactically trivial tokens (such as punctuation or deterministic keywords), the model predicts small \(t_i\), converging in very few update steps. For semantically ambiguous tokens or critical reasoning pivot tokens, higher \(t_i\) values maintain diffusion-style bidirectional plasticity across extensive steps. During inference, token update depths are dynamically truncated at \(K_i = \lfloor K_{max} \cdot t_i \rfloor\) with \(K_{max} = 20\), concentrating computational effort strictly where semantic uncertainty persists.

3. Unified Variational ELBO Objective: Joint End-to-End Optimization of Trajectories and Times

Because continuous marginalization over both latent paths \(Z\) and progression variables \(t\) is analytically intractable, Evo derives an evidence lower bound (ELBO) on sequence log-likelihood:

\[\log p_\Theta(X) \ge \mathbb{E}_{Z, t \sim q(Z, t | X)} \left[ \log p_\theta(Z, t) + \log p(X | Z) - \log q(Z, t | X) \right]\]

This variational lower bound translates into a multi-step flow matching loss \(\mathcal{L}_{Flow}(\theta)\). Continuous Gaussian perturbations are injected into the latent trajectory: \(z_i^{(t)} = \sqrt{\alpha_t} z_i^{(0)} + \sqrt{1 - \alpha_t} \epsilon\) under a cosine schedule \(\alpha_t = \cos^2(\frac{\pi}{2} t)\). As \(t_i \to 0\), perturbation vanishes and the objective naturally collapses to standard autoregressive negative log-likelihood; as \(t_i \to 1\), score-matching denoising gradients govern the trajectory; across intermediate values, both learning signals interact smoothly. Backpropagating gradients directly through the variational posterior \(q(Z, t | X)\) enables joint, end-to-end learning of latent vector fields and progression allocations, regularized by an entropy term to prevent degenerate temporal assignments.

Loss & Training

The unified objective incorporates flow matching, autoregressive likelihood, and score matching:

\[\mathcal{L}_{Total} = \mathcal{L}_{Flow}(\theta) + \lambda_{AR}(t) \mathcal{L}_{AR} + \lambda_{Diff}(t) \mathcal{L}_{Score} + \mathcal{L}_{Entropy}(t_i)\]

During initial training phases, \(\lambda_{AR}\) and \(\lambda_{Diff}\) are set to unity and gradually annealed during warmup, allowing latent flow loss to guide long-term training. Pretraining is conducted on a 3.4-trillion-token corpus (60% English, 25% multilingual across 30+ languages, 10% code, and 5% scientific/mathematical literature) using a 64K SentencePiece BPE tokenizer and a maximum sequence context of 4K tokens. Optimization utilizes AdamW (\(\beta_1 = 0.9, \beta_2 = 0.95\), weight decay 0.1) with a cosine learning rate decay schedule starting from peak values of \(1.5 \times 10^{-4}\) to \(2.0 \times 10^{-4}\). Distributed training is executed on clusters of NVIDIA A100 GPUs using DeepSpeed ZeRO-3, sequence parallelism, and bfloat16 mixed precision. Post-training supervised fine-tuning mirrors the LLaDA instruction mixtures and configurations to ensure unbiased benchmarking.

Key Experimental Results

Main Results

Evaluations span 15 established benchmarks across general reasoning (MMLU, BBH, ARC-C), mathematical problem solving (GSM8K, MATH, GPQA), coding (HumanEval, MBPP), and end-to-end decoding latency. Baselines include leading autoregressive models (LLaMA3 8B, Qwen2.5 7B), diffusion-based language models (LLaDA 8B, MDLM 7B), and hybrid architectures (BD3-LM 7B, ARD 7B).

Model E2E Latency (s) ↓ Speed (tokens/s) ↑ MMLU (5-shot) GSM8K (4-shot) MATH (4-shot) HumanEval (0-shot) MBPP (4-shot) ARC-C (25-shot)
Evo 8B (Ours) 8.6 52 76.8 86.4 54.9 60.6 77.4 65.6
BD3-LM 7B (Hybrid) 14.2 28 70.0 79.8 42.9 50.6 63.7 60.2
ARD 7B (Hybrid) 32.5 12 46.2 15.1 3.6 13.4 18.9 46.7
LLaDA 8B (Diffusion) 21.8 16 65.6 70.9 27.0 33.2 38.5 48.2
MDLM 7B (Diffusion) 18.9 22 64.0 36.5 10.0 29.0 51.3 59.6
LLaMA3 8B (AR) 7.4 58 65.1 52.7 15.4 34.6 47.1 52.8
Qwen2.5 7B (AR) 8.1 46 74.5 85.8 50.2 58.3 75.2 64.0

Under post-trained (SFT) evaluations, Evo 8B retains dominant performance margins, scoring 92.5 on ARC-C, 78.8 on MATH, and 86.7 on HumanEval pass@1, outperforming heavily aligned RLHF/DPO baselines of similar capacity.

Ablation Study

Systematic ablations inspect the influence of temporal conditioning, progression step allocation, multi-module separation, and pretraining weight transfer:

Ablation Variant Validation PPL ↓ Task Acc (MMLU+GSM8K) ↑ Analysis & Conclusion
Evo (Full Model) 8.1 81.6 Full model with continuous flow conditioning and learned \(t_i\)
w/o Time Conditioning 9.6 69.1 Eliminates progression trajectory, dropping task accuracy by 12.5%
Fixed \(t_i = 0\) (AR-only) 10.2 69.6 Forces deterministic one-pass decoding, losing global self-correction
Fixed \(t_i = 1\) (Diff-only) 8.6 77.0 Full diffusion denoising retains refinement but harms language fluency
Fixed \(t_i = 0.5\) (Uniform) 8.8 77.9 Static progression schedule lacks semantic awareness
Heuristic \(t_i\) (AR Entropy) 8.4 78.2 Confidence heuristics underperform variational end-to-end learning
Two-Module (AR + Diff) 8.6 77.9 Decoupled parameters impede smooth latent flow alignment
No Shared Embeddings 8.4 78.3 Weakens cross-temporal manifold consistency across steps
No Pretrained Init 9.1 76.2 Cold starts lead to prolonged convergence and 5.4% accuracy degradation
Frozen AR Backbone 8.7 80.4 Preserves strong performance, highlighting efficient modular adaptation

Key Findings

  • Imperative Role of Dynamic Time Conditioning: Discarding sinusoidal time embeddings drops benchmark performance by 12.5% and inflates perplexity to 9.6. Restricting to pure AR (\(t_i = 0\)) causes devastating regression on symbolic reasoning (69.6% vs 81.6%), proving that bidirectional trajectory evolution is paramount for overcoming early reasoning failure modes.
  • Superiority of Learned over Heuristic Maturity: Deriving progression steps from autoregressive entropy signals achieves 78.2% accuracy, trailing the learned variational \(t_i\) parameterization by 3.4%. Semantic uncertainty requires contextual trajectory inference rather than superficial token-level probabilities.
  • Transforming Diffusion Decoding Throughput: By adaptively halting updates on mature tokens via \(K_i = \lfloor K_{max} \cdot t_i \rfloor\), Evo sustains 52 tokens/s on a single A100 GPU. This marks a \(3.25\times\) speedup over full diffusion (LLaDA at 16 tokens/s) and almost doubles block diffusion (BD3-LM at 28 tokens/s), approaching LLaMA3's 58 tokens/s while delivering drastically superior reasoning capability.

Highlights & Insights

  • Rigorous Mathematical Unification: Demonstrates that autoregressive generation and reverse diffusion probability flows represent identical ODE trajectories under an invertible continuous time reparameterization, unifying two fundamentally divergent modeling schools.
  • Maturity-Driven Computation Allocation: Replaces arbitrary positional or chunk-based scheduling with intrinsic token semantic maturity, directing iterative refinement strictly toward regions of persistent semantic ambiguity.
  • Single-Architecture Simplicity: Avoids cumbersome dual-backbone or cascaded pipeline architectures, executing end-to-end training and inference inside a single Transformer backbone that seamlessly repurposes standard pretrained model weights.

Limitations & Future Work

  • Higher Training Overhead: Optimizing joint variational posteriors across latent trajectories incurs 30%–50% greater wall-clock training compute compared to pure autoregressive baselines, posing scalability considerations for training runs beyond 10T tokens.
  • Latency Sensitivity on Highly Ambiguous Prompts: Because inference speed relies on early convergence of low-entropy tokens, unstructured generation tasks with uniformly high semantic entropy may force widespread iteration near \(K_{max}\), reducing the decoding speed advantage over AR models.
  • Controllability under Bidirectional Attention: By eliminating causal masking during refinement steps, fine-grained positional constraints (such as exact token count budgets or prefix anchoring) are occasionally susceptible to contextual drift.
  • vs BD3-LM (Arriola et al., 2025): BD3-LM constrains diffusion refinement to fixed discrete blocks in an AR chain; Evo operates continuously across the full sequence and modulates computation at the token level, surpassing BD3-LM by 6.6 points on GSM8K and doubling decoding speed.
  • vs AR-Diffusion (Wu et al., 2023): ARD imposes a left-to-right positional noise schedule, artificially assuming later tokens always require deeper denoising; Evo learns semantic uncertainty adaptively, eliminating positional bias and accelerating throughput from 12 tokens/s to 52 tokens/s.
  • vs LLaDA (Zhu et al., 2026): LLaDA applies discrete masked diffusion globally across all positions; Evo leverages continuous probability flows and early termination on confident tokens, slashing latency from 21.8 s to 8.6 s.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First principled mathematical proof and unified latent trajectory formulation bridging autoregressive and diffusion language models via continuous probability flows.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 15 benchmarks, multi-scale variants (1.3B–13B), pretraining and post-training regimes, and exhaustive architectural ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical exposition, compelling intuitive motivation, and seamless narrative progression from theoretical ODE derivations to empirical implementation.
  • Value: ⭐⭐⭐⭐⭐ Provides a practical, highly efficient blueprint for eliminating autoregressive compounding errors without bearing the crushing sampling cost of traditional diffusion LLMs.