Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/aSleepyTree/Drift-AR
Area: Image Generation
Keywords: Autoregressive Generation, Drifting Models, Single-Step Generation, Speculative Decoding, Entropy-Parameterized Prior

TL;DR

Drift-AR reveals that per-position prediction entropy in continuous-space autoregressive models simultaneously captures speculative draft quality and local generation uncertainty, utilizing causal-normalized entropy alignment and an entropy-parameterized anti-symmetric drifting field to achieve genuine 1-NFE visual decoding with 3.8x to 5.5x end-to-end acceleration.

Background & Motivation

Autoregressive-Diffusion (AR-Diffusion) hybrid architectures, exemplified by MAR, TransDiff, and NextStep-1, have emerged as a premier paradigm for transcending the discrete tokenization ceiling. By partitioning visual synthesis into two consecutive stages—continuous-space autoregressive transformers for structured global semantic modeling, followed by diffusion or flow matching decoders for high-fidelity visual detail realization—these hybrids strike an impressive balance between long-range semantic coherence and photorealistic texture synthesis.

However, this two-stage formulation inevitably introduces a dual speed bottleneck: the autoregressive transformer generates continuous tokens strictly sequentially, accumulating high latency over sequence length, while the diffusion decoder demands iterative multi-step numerical denoising. Prior acceleration efforts tackle these stages in strict isolation—applying speculative decoding (e.g., EAGLE) to the autoregressive stage and diffusion distillation (e.g., CD/DMD) to the vision decoding stage. This segregated paradigm lacks a unifying principle; distillation methods remain bound to iterative sampling with severe training instability, while directly porting discrete speculative decoding to continuous latent spaces triggers severe entropy collapse where draft models degenerate into overconfident low-entropy modes and face pervasive rejection.

Real-world visual images exhibit highly non-uniform information density across spatial coordinates, where flat backgrounds (e.g., sky) and intricate structures (e.g., edges and textures) present drastically disparate generation difficulties. The authors identify that the per-position prediction entropy of continuous-space autoregressive models naturally quantifies this spatial uncertainty and carries a dual physical significance: it reflects draft prediction fidelity in speculative decoding, while strongly correlating with autoregressive prediction error (Pearson correlation \(r=0.64\)), directly indicating the magnitude of corrective effort required by the vision decoder. Core idea: per-position autoregressive prediction entropy serves as a unifying bridge across both bottlenecks—calibrating speculative draft distributions via a causal-normalized entropy loss while parameterizing the physical initial variance of an anti-symmetric drifting field for genuine 1-NFE single-step decoding without distillation.

Method

Overall Architecture

Drift-AR establishes an end-to-end acceleration architecture centered around per-position prediction entropy. In the first phase, Entropy-Informed Speculative Decoding utilizes a lightweight draft transformer to predict continuous feature sequences in parallel, regularized by a causal-normalized entropy loss to prevent overconfident collapse; accepted features and their associated entropy map are verified and output by the target model. In the second phase, the vision decoder abandons iterative denoising altogether; instead, it adopts an Anti-Symmetric Drifting Field whose initial Gaussian state is centered on the autoregressive feature and parameterized by the entropy-derived variance via a bounded exponential mapping. Driven by kernelized attraction and repulsion forces in the feature space, the pushforward distribution evolves toward the real data distribution in a single forward pass (1-NFE), naturally vanishing at equilibrium.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Context & Condition Inputs"] --> B["Entropy-Informed Speculative Decoding<br/>Continuous regression with causal-normalized entropy alignment"]
    B --> C["Context-Aware Early Stopping<br/>Shallow-layer entropy monitoring & dynamic threshold truncation"]
    C --> D["Autoregressive Latents & Entropy Map"]
    D --> E["Entropy-Parameterized Prior<br/>Bounded temperature variance mapping for initial distribution"]
    E --> F["Single-Step Anti-Symmetric Drifting Decoder<br/>Multi-temperature kernelized attraction-repulsion for 1-NFE"]
    F --> G["Synthesized High-Fidelity Image"]

Key Designs

1. Entropy-Informed Speculative Decoding: Mitigating Draft Overconfidence in Continuous Latent Space

Directly applying EAGLE-style speculative decoding to continuous visual autoregressive models induces severe draft rejection. Due to constrained capacity and low visual information density, smaller draft models readily collapse toward dominant modes with unnaturally low entropy, deviating from the broad entropy distribution of the target model. Drift-AR removes the discrete classification cross-entropy and constructs a continuous feature regression objective conditioned on continuous representations:

\[\mathcal{L}_{\text{reg}} = \text{SmoothL1}\left(z_{j+1},\; \text{DraftModel}(Z_{1:j},\, F_{1:j})\right)\]

where \(Z_{1:j}\) represents the sequence of continuous autoregressive latents and \(F_{1:j}\) denotes the target hidden states. Furthermore, because a causal attention mask restricts row \(r\) to attend only to columns \(1, \dots, r\), its theoretical upper bound of entropy is \(\log(r)\). Directly averaging raw entropy across rows would unfairly penalize early sequence positions and destabilize optimization gradients. Drift-AR therefore formulates a causal-normalized entropy indicator and loss:

\[\mathcal{E}_{\text{entropy}}^{(r)} = -\frac{1}{\log(r)} \sum_{c=1}^{r} \Omega_{r,c} \log \Omega_{r,c}, \quad \mathcal{L}_{\text{entropy}} = -\frac{1}{R} \sum_{r=2}^{R} \mathcal{E}_{\text{entropy}}^{(r)}\]

where \(\Omega_{r,c}\) is the softmax weight from the penultimate attention layer. Maximizing causal-normalized entropy aligns the draft model's uncertainty with the target model, substantially enhancing draft acceptance rates.

2. Context-Aware Early Stopping: Dynamic Entropy Thresholding to Prevent Low-Quality Speculation

Allowing the draft model to propose speculative tokens blindly for fixed-length windows wastes computational verification budget whenever a draft prefix degrades early. Drift-AR monitors the draft model's shallow-layer attention entropy as a computationally efficient confidence proxy. Because uniform flat regions (e.g., clear sky) and dense texture regions inherently exhibit different entropy levels, a fixed static threshold causes premature or delayed termination. Instead, the framework maintains an exponential moving average (EMA) of the target model's normalized entropy \(\bar{E}_{c}^{\text{target}} = 0.9 \bar{E}_{c-1}^{\text{target}} + 0.1 E_{c}^{\text{target}}\), defining a dynamic threshold \(\tau_c = \gamma \cdot \bar{E}_{c}^{\text{target}}\) (with \(\gamma=0.8\)). Once the draft entropy falls below \(\tau_c\), speculative proposing is halted immediately and handed over to the target model for parallel prefix verification, avoiding wasted target forward passes.

3. Entropy-Parameterized Prior: Grounding Physical Drifting Variance in Autoregressive Uncertainty

Standard AR-Diffusion hybrids treat autoregressive representations merely as conditioning embeddings while drawing initial latents from an isotropic standard Gaussian prior \(\mathcal{N}(0, \mathbf{I})\), ignoring spatial variations in autoregressive generation confidence. Empirical diagnostics demonstrate a strong positive correlation (\(r=0.64\)) between autoregressive prediction error \(\|z_{\text{AR}}^{(r)} - x_{\text{gt}}^{(r)}\|\) and per-position entropy. Drift-AR treats autoregressive features as the physical center of the initial distribution and maps per-position entropy \(\mathcal{E}\) to a local variance scale via a bounded exponential temperature mapping:

\[\sigma(\mathcal{E}) = \sigma_{\max} \cdot \frac{e^{\mathcal{E} / \tau_\sigma}}{e^{\mathcal{E}_{\max} / \tau_\sigma}}\]

yielding the localized Gaussian prior \(q(x^{(r)} \mid z_{\text{AR}}^{(r)}, \mathcal{E}^{(r)}) = \mathcal{N}\left(x^{(r)};\, z_{\text{AR}}^{(r)},\, \sigma^2(\mathcal{E}^{(r)})\mathbf{I}\right)\). For confident, low-entropy coordinates, \(\sigma \to 0\), creating a tight distribution centered on the autoregressive feature; for uncertain, high-entropy regions, the variance smoothly scales up to \(\sigma_{\max}\), reserving sufficient sampling dispersion for corrective drifting.

4. Anti-Symmetric Drifting Field in Latent Space: Genuine 1-NFE Visual Decoding

Rather than executing multi-step diffusion denoising or distilling a multi-step teacher, the visual decoder \(f_\theta\) takes the reparameterized sample \(x_0 \sim q\) and condition \(c\) to output latent representation \(\hat{x} = f_\theta(x_0, c, \mathcal{E})\). To align decoder outputs with the drifting field, \(\hat{x}\) is projected into feature space via a frozen encoder \(\phi\), implemented as a fixed copy of the target autoregressive model's penultimate layer. The drifting loss is optimized directly in \(\phi\)-space:

\[\mathcal{L}_{\text{drift}} = \mathbb{E}_{x \sim q}\left[ \left\| \phi(\hat{x}) - \text{stopgrad}\Big(\phi(\hat{x}) - V_{p,q}(\phi(\hat{x}))\Big) \right\|^2 \right]\]

The drifting field satisfies anti-symmetry \(V_{p,q} = -V_{q,p}\), driven by positive attraction from real data \(V_p^+\) and negative repulsion among generated samples \(V_q^-\) across multiple kernel temperatures. As pushforward distribution \(q\) reaches equilibrium with data distribution \(p\), net velocity vanishes (\(V_{p,q} \to 0\)). Under the entropy-parameterized prior, high-entropy positions experience strong corrective drift toward the data manifold, while low-entropy positions naturally yield near-zero drift, enabling principled single-step (1-NFE) generation without iterative sampling.

Loss & Training

Naive end-to-end joint training suffers from the "moving prior problem": updating the autoregressive model continuously alters the initial distribution \(q\), violating the stationary source assumption required by anti-symmetric drifting dynamics. Drift-AR introduces a two-phase annealed training schedule:

\[\mathcal{L}_{\text{total}} = \alpha(t) \cdot (\mathcal{L}_{\text{reg}} + \mathcal{L}_{\text{entropy}}) + (1 - \alpha(t)) \cdot \mathcal{L}_{\text{drift}}\]

The scalar weighting \(\alpha(t)\) follows a piecewise linear decay. During Phase I (Joint Optimization, \(t \le T_{\text{freeze}} = 0.8 \cdot T_{\text{total}}\)), \(\alpha(t)\) decays linearly from \(\alpha_0 = 0.95\) to 0. Initializing \(\alpha_0 < 1\) provides gentle supervisory gradients to prevent decoder cold-start collapse while the autoregressive prior stabilizes. During Phase II (Pure Drifting, \(t > T_{\text{freeze}}\)), \(\alpha(t) = 0\), and autoregressive latents and entropy maps are detached, completely freezing the prior distribution \(q\) and eliminating gradient leakage. The drifting field is optimized stably against a stationary prior. Key training hyper-parameters include \(\sigma_{\max}=0.5\), \(\tau_\sigma=2.0\), multi-kernel temperatures \(\tau \in \{0.02, 0.05, 0.2\}\), AdamW optimizer with learning rate \(1 \times 10^{-4}\), and batch size 32.

Key Experimental Results

Main Results

Evaluated on ImageNet \(256 \times 256\) class-conditional generation across MAR and TransDiff architectures, alongside text-to-image synthesis on NextStep-1 (GenEval and MJHQ-30K), Drift-AR achieves superior acceleration and generation quality.

Model / Framework	Acceleration Scheme	Latency (s)	Speedup	FID \(\downarrow\)	IS \(\uparrow\)
MAR-L [13]	Baseline (iterative denoising)	5.31	1.00×	1.78	296.0
MAR-L + DMD [39]	Diffusion Distillation	1.99	2.67×	1.81	295.5
MAR-L + LazyMAR [37]	Feature Caching	2.29	2.32×	1.93	297.4
MAR-L + Drift-AR (Ours)	Speculative + 1-NFE Drifting	0.96	5.53×	1.76	297.4
MAR-H [13]	Baseline (iterative denoising)	9.97	1.00×	1.55	303.7
MAR-H + DMD [39]	Diffusion Distillation	4.17	2.39×	1.73	301.0
MAR-H + LazyMAR [37]	Feature Caching	4.24	2.35×	1.69	299.2
MAR-H + Drift-AR (Ours)	Speculative + 1-NFE Drifting	1.93	5.16×	1.53	304.6
TransDiff-L [42]	Baseline (iterative denoising)	3.17	1.00×	1.61	295.1
TransDiff-L + SD [15]	Vanilla Speculative Decoding	1.75	1.81×	1.88	283.6
TransDiff-L + DMD [39]	Diffusion Distillation	1.61	1.97×	1.79	288.3
TransDiff-L + Drift-AR (Ours)	Speculative + 1-NFE Drifting	0.64	4.96×	1.61	295.8
TransDiff-H [42]	Baseline (iterative denoising)	6.72	1.00×	1.55	297.9
TransDiff-H + SD [15]	Vanilla Speculative Decoding	3.52	1.91×	1.68	291.1
TransDiff-H + DMD [39]	Diffusion Distillation	3.48	1.93×	1.71	289.9
TransDiff-H + Drift-AR (Ours)	Speculative + 1-NFE Drifting	1.33	5.06×	1.57	298.1

On NextStep-1 text-to-image synthesis, Drift-AR achieves a 3.81× speedup while outperforming the base teacher: GenEval score rises from 0.63 to 0.66, FID improves from 6.71 to 6.66, and CLIP score reaches 29.02 (vs. 28.67 for baseline, 27.96 for SD, and 25.19 for DMD).

Ablation Study

Ablations on TransDiff-H (ImageNet \(256 \times 256\)) demonstrate the criticality of each constituent design:

Configuration	FID \(\downarrow\)	IS \(\uparrow\)	Note
TransDiff-H Baseline	1.55	297.9	Original 20-step diffusion reference
Ours (Full Model)	1.57	298.1	Single-step 1-NFE + entropy alignment (5.06× speedup)
w/o A: Entropy Parameterization	1.72	289.3	Constant variance prior; largest degradation
w/o B: Stage-Wise Reweighting	1.69	292.6	Fixed \(\alpha=0.5\); moving prior destabilizes training
w/o C: Anti-Symmetric Kernel	1.69	291.8	Attraction only; violates \(V_{p,q}=0\) equilibrium
w/o D: Context-Aware Early Stopping	1.62	295.5	Constant threshold; redundant verification overhead
w/o E: Phase II Prior Freezing	1.67	291.9	Unfrozen AR gradients; violates stationary source assumption

In the decoder step analysis on TransDiff-H, forcing the standard diffusion decoder to 1 step leads to catastrophic collapse (FID 14.72). Distilled DMD achieves FID 2.93 at 1 step. In contrast, Drift-AR attains FID 1.57 in a single forward pass, matching the 20-step baseline (1.55).

Key Findings

• Entropy parameterization is paramount: Ablation A incurs the sharpest performance drop (FID degrading to 1.72, IS to 289.3), proving that allocating prior dispersion based on local generation uncertainty is essential for single-step drift convergence.
• Two-phase scheduling preserves equilibrium: Ablations B and E confirm the "moving prior" hypothesis. Training the drifting field on an evolving prior breaks the stationary source assumption, leading to residual drift and degraded synthesis.
• Variance boundary robustness: Exploring \(\sigma_{\max} \in [0.1, 1.0]\) confirms performance remains stable across \([0.3, 0.7]\) (FID 1.57–1.58). Oversmall values (\(\sigma_{\max}=0.1\), FID 1.68) collapse the prior to a Dirac delta and sacrifice sample diversity, while excessive values (\(\sigma_{\max}=1.0\), FID 1.64) place an unsustainable transport burden on the single-step field.

Highlights & Insights

• Dual-purpose signal reuse: Computing per-position prediction entropy once during speculative decoding and directly reusing it as physical variance for single-step drifting eliminates secondary uncertainty models and incurs zero auxiliary overhead.
• Causal normalization solves early-token scale distortion: Accounting for the theoretical row-wise entropy bound \(\log(r)\) avoids penalizing early autoregressive tokens, resolving gradient skew in continuous latent space.
• Distillation-free 1-NFE generation: Grounded in the anti-symmetric equilibrium property (\(q=p \implies V=0\)), the framework bypasses multi-step teachers and trajectory distillation instabilities, deriving 1-step generation from physical first principles.

Limitations & Future Work

• Causal architecture restriction: The normalized entropy computation relies on lower-triangular causal attention maps, which cannot be directly ported to bidirectional or non-causal visual encoders without structural adaptation.
• Extended training duration: The two-phase schedule requires stabilizing the autoregressive prior before fine-tuning the drifting field, resulting in longer total training iterations compared to certain one-step post-hoc distillations.
• Complex spatiotemporal scaling: Extending entropy parameterization to video generation requires modeling 3D spatiotemporal uncertainty, where motion blur and temporal dynamics introduce more intricate entropy distributions.

Related Work & Insights

• vs EAGLE / EAGLE-2: While EAGLE accelerates discrete LLMs via feature regression and token classification, Drift-AR tailors speculative decoding to continuous visual spaces using continuous-feature regression and causal-normalized entropy alignment to prevent draft collapse.
• vs DMD / DMD2: Distribution matching distillation relies on multi-step diffusion teachers and adversarial/score distillation losses that frequently suffer from instability. Drift-AR trains an anti-symmetric drifting field from scratch, achieving 1-NFE visual decoding with superior fidelity.
• vs MAR / TransDiff: MAR and TransDiff suffer from 20- to 50-step iterative decoding latencies. Drift-AR serves as a drop-in acceleration framework, delivering roughly 5x speedups across scales while maintaining or exceeding baseline generative quality.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant physical insight establishing prediction entropy as the bridge between speculative decoding and single-step drifting.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning three benchmark architectures (MAR, TransDiff, NextStep-1), detailed ablations, step count analyses, and hyper-parameter sensitivity sweeps.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear intuitive motivation, and compelling empirical correlation analyses.
• Value: ⭐⭐⭐⭐⭐ Resolves the primary inference bottleneck of continuous autoregressive generative models, enabling practical real-time synthesis.