LISA: Locality-Informed Speculative Decoding for Accelerating Autoregressive Image Generation¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://neuraliying.github.io/LISA
Area: Image Generation
Keywords: Autoregressive Image Generation, Speculative Decoding, Locality Geometry, Knowledge Distillation, Soft Verification
TL;DR¶
Addressing the collapse of acceptance rates in speculative decoding caused by flat, non-discriminative visual token distributions, LISA exploits the local geometry of visual codebooks via locality-informed distillation and geometry-aware soft verification, achieving up to 4.12× lossless acceleration.
Background & Motivation¶
Autoregressive (AR) models have demonstrated impressive visual fidelity and versatile unified multimodal capabilities by framing image synthesis as sequential next-token prediction. However, this strict sequential decoding paradigm inevitably suffers from substantial inference latency, presenting a critical bottleneck for real-world interactive deployment. In large language models (LLMs), speculative decoding has become the gold-standard acceleration paradigm, where a lightweight drafter predicts candidate token sequences in parallel and a large target model verifies them in a single forward pass. Yet, when state-of-the-art speculative decoding techniques such as EAGLE-3 are applied directly to visual autoregressive models, the acceleration gains diminish significantly or even collapse into slowdowns.
This failure stems from the intrinsic distributional discrepancy between discrete visual tokens and natural language tokens. Natural language tokens exhibit spiky probability distributions with mass concentrated on a small set of confident candidates, whereas visual tokens—quantized from continuous signals via vector quantization (VQ)—exhibit flat and non-discriminative profiles that impede drafter–target alignment. Furthermore, high-quality image synthesis fundamentally relies on stochastic sampling (e.g., temperature \(T>0\), Top-\(K\) sampling), which further disperses probability mass across the vocabulary. Under stochastic sampling, strict pointwise verification criteria frequently reject valid candidates proposed by the drafter, causing acceptance rates to plummet.
A diagnostic investigation of visual codebooks reveals that despite spatial entropy heterogeneity and global distribution dispersion, visual codebooks retain rich geometric continuity where perceptually coherent patches map to neighboring codebook entries. Under high predictive uncertainty, target probability mass concentrates locally within the geometric neighborhood around the drafter's prediction. Core Idea: Leverage the intrinsic local geometry of discrete visual codebooks to reweight alignment supervision over structured uncertainty regions via locality-informed distillation, and adaptively loosen acceptance boundaries via geometry-aware soft verification, systematically bridging the drafter–target misalignment gap during both training and inference.
Method¶
Overall Architecture¶
LISA builds upon the standard speculative decoding pipeline consisting of draft tree expansion followed by target model parallel verification. To conquer the flat probability landscape and stochastic sampling fragility of visual tokens, LISA establishes a dual-stage closed loop: during training, Locality-Informed Distillation evaluates the target centroid concentration in codebook space against drafter prediction errors, dynamically upweighting hard, high-entropy positions via progressive soft gating; during inference, Geometry-Aware Soft Verification (GASV) modulates acceptance thresholds according to target predictive entropy and jointly checks geometric drift in a whitened feature space alongside cluster posterior mass, enabling permissive yet faithful token acceptance.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Context and Prompt"] --> B["Drafter Tree Expansion<br/>Draft Tree Generation"]
B --> C["Target Parallel Forward Pass<br/>Logits & Hidden State Extraction"]
C --> D["Uncertainty-Modulated Thresholds<br/>Dynamic Scaling via Entropy & Margin"]
D --> E["Geometry-Aware Soft Verification<br/>Joint Geometric Drift & Cluster Posterior Gate"]
E -->|Candidate Acceptance / Resampling| F["Output Accepted Visual Token Sequence"]
Key Designs¶
1. Locality-Informed Selection: Pinpointing Structurally Concentrated Yet Misaligned Regions
Standard distillation objectives evenly distribute supervision across all token positions, leaving drafters prone to severe misalignment in ambiguous, high-entropy regions such as intricate textures and object boundaries. LISA exploits the geometric structure of the VQ codebook by identifying local geometric anchors. For each position, it computes the probability-weighted centroid \(\mathbf{c}_{b,t}\) of the target model's top-\(K_c\) tokens in the \(L_2\)-normalized embedding space: $\(\mathbf{c}_{b,t} = \frac{\sum_{k=1}^{K_c} \tilde{p}_t^{(k)} \mathbf{e}_{i_k}}{\left\| \sum_{k=1}^{K_c} \tilde{p}_t^{(k)} \mathbf{e}_{i_k} \right\|_2}\)$ With cosine neighborhood radius \(r\), the neighborhood \(\mathcal{N}_{b,t} = \{v \in \mathcal{V} : 1 - \langle \mathbf{c}_{b,t}, \mathbf{e}_v \rangle \le r\}\) captures the target's local mass \(\alpha^t_{b,t} = \sum_{v \in \mathcal{N}_{b,t}} p_t(v \mid b,t)\). When this target mass is dense but the drafter's Top-1 prediction falls outside the neighborhood (\(a_{b,t} = 0\)), the position is assigned a high locality-mismatch score: $\(s_{b,t} = \alpha^t_{b,t} (1 - a_{b,t})\)$ Positions exceeding an annealed quantile threshold are filtered into a selection mask, directing the model's capacity toward structurally critical regions where target confidence is localized but the drafter errs.
2. Progressive Soft Gating and Truncated Distillation: Smoothing Optimization and Stabilizing Gradient Flow
Directly applying hard binary selection early in training can destabilize optimization due to excessive noise in high-uncertainty regions. LISA mitigates this with a progressive soft gating mechanism \(g_{\mathrm{align}}(b,t)\), which scales supervision according to local alignment progress computed over the joint top-\(K\) candidate set \(\mathcal{K}(b,t)\): $\(g_{\mathrm{align}}(b,t) = \exp\left( -\mathrm{KL}\left(\tilde{p}_t^{(\mathcal{K})}(\cdot \mid b,t) \;\|\; \tilde{p}_s^{(\mathcal{K})}(\cdot \mid b,t)\right) \right)\)$ Combined with a scheduled veto factor \(w_{\mathrm{veto}}(b,t)\), the composite position weight \(w_{b,t} = w_{\mathrm{veto}}(b,t) \tilde{g}_{b,t}\) assigns modulated weights to selected positions while preserving a baseline weight for unselected ones to ensure continuous gradient flow. Supervision is then formulated via truncated cross-entropy over the target nucleus set \(\mathcal{S}_t(b,t)\) and KL divergence over the joint support set \(\mathcal{K}\).
3. Uncertainty-Modulated Thresholds: Entropy-Aware Dynamic Acceptance Boundaries
Fixed rejection sampling thresholds rigidly penalize visual tokens in high-entropy regions, where multimodal distributions naturally dilute individual token probabilities despite semantic validity. In GASV, LISA measures the target distribution's normalized entropy \(U = \frac{-\sum_v p_t(v)\log p_t(v)}{\log V}\) and normalized logit margin \(m_{\mathrm{norm}} = \sigma(\ell_{(1)} - \ell_{(2)})\). Guided by the empirical finding that higher target uncertainty correlates with smaller perceptual preference gaps between Top-1 and near-neighbor tokens, the base thresholds are dynamically adapted: $\(\kappa_{\mathrm{eff}} = f_\kappa(\kappa; U, m_{\mathrm{norm}}), \quad \lambda_{\mathrm{eff}} = f_\lambda(\lambda; U, m_{\mathrm{norm}})\)$ As predictive uncertainty increases, the effective acceptance boundary automatically relaxes, permitting semantically compatible draft candidates to pass verification while strictly enforcing identity matching in deterministic low-entropy regions.
4. Dual-Branch Soft Verification: Reconciling Geometric Drift and Cluster Posterior Support
For a draft candidate token \(v\) against the target Top-1 token \(i\), GASV extracts two complementary signals in the whitened output-embedding space. First, it measures the normalized geometric drift distance \(s_{\mathrm{geo}}(v) = \frac{\ell_i - \ell_v}{\|(\mathbf{w}_i - \mathbf{w}_v)\odot \mathrm{invstd}\|_2 + \varepsilon}\) and compares it with a calibrated cluster budget \(\beta_{c_i}\), yielding geometric confidence score \(g_{\mathrm{geo}}(v) = \sigma\left(\theta_{\mathrm{geo}} (\kappa_{\mathrm{eff}} \beta_{c_i} - s_{\mathrm{geo}}(v))\right)\). Second, by partitioning the codebook into offline clusters, it calculates the regional posterior log-mass \(\pi_c = \log \sum_{u \in \mathcal{R}(c)} \exp(\ell_u)\) to obtain cluster confidence \(g_{\mathrm{clu}}(v) = \sigma\left(\theta_{\mathrm{clu}} (\pi_{c_v} - \pi_{c_i} - \log \lambda_{\mathrm{eff}})\right)\). These two scores are aggregated via an OR-style gate: $\(g(v) = \max\left(g_{\mathrm{geo}}(v), \; g_{\mathrm{clu}}(v)\right)\)$ Multiplying this gate into the raw sampling distribution \(p_{\mathrm{soft}}(v) \propto p_{\mathrm{raw}}(v) g(v)\) yields a renormalized target distribution that safely accommodates geometrically and semantically coherent draft tokens.
Loss & Training¶
The overall training loss for the drafter combines weighted truncated cross-entropy and weighted KL divergence over the joint support: $\(\mathcal{L}_{\mathrm{tot}} = \lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}} + \lambda_{\mathrm{KD}}\mathcal{L}_{\mathrm{KD}}\)$ where \(\mathcal{L}_{\mathrm{CE}} = -\sum_{b,t} w_{b,t}\sum_{v\in\mathcal{S}_t(b,t)}\tilde{p}_t(v\mid b,t)\log p_s(v\mid b,t)\) and \(\mathcal{L}_{\mathrm{KD}} = \tau_s^2 \sum_{b,t} w_{b,t} \mathrm{KL}\left(\tilde{p}_t^{(\mathcal{K})}(\cdot\mid b,t) \;\|\; \tilde{p}_s^{(\mathcal{K})}(\cdot\mid b,t)\right)\). Drafters are built on the EAGLE-3 architecture and trained on A100 GPUs for 6 epochs using BF16 precision and a learning rate of \(3\times 10^{-4}\). Training data uses a 100k LAION-COCO subset for LlamaGen-XL-T2I and 30k target-generated samples for Lumina-mGPT-7B.
Key Experimental Results¶
Main Results¶
End-to-end inference latency and generation quality were evaluated on a single NVIDIA RTX 4090 GPU for Text-to-Image (MS-COCO 2017 validation set, 5,000 samples) and Class-to-Image (ImageNet-1K, 20,000 samples) benchmarks.
| Dataset / Model | Method | Speedup↑ | Acc. Len.↑ | FID↓ | Quality Metric (CLIP↑ / IS↑) |
|---|---|---|---|---|---|
| LlamaGen-XL-T2I (MS-COCO) | Vanilla | 1.00× | – | 45.5 | 28.6 (CLIP) |
| LlamaGen-XL-T2I | EAGLE-2 | 0.97× | 1.20 | 43.1 | 28.9 (CLIP) |
| LlamaGen-XL-T2I | LANTERN | 1.70× | 2.40 | 47.2 | 28.6 (CLIP) |
| LlamaGen-XL-T2I | SJD | 1.61× | 1.69 | 48.1 | 28.7 (CLIP) |
| LlamaGen-XL-T2I | LISA (Ours) | 2.11× | 3.05 | 46.7 | 28.5 (CLIP) |
| Lumina-mGPT-7B (MS-COCO) | Vanilla | 1.00× | – | 30.1 | 33.0 (CLIP) |
| Lumina-mGPT-7B | EAGLE-3 | 1.86× | 2.77 | 30.2 | 32.1 (CLIP) |
| Lumina-mGPT-7B | LANTERN | 2.56× | 3.63 | 33.9 | 32.7 (CLIP) |
| Lumina-mGPT-7B | SJD | 2.05× | 2.23 | 31.1 | 31.3 (CLIP) |
| Lumina-mGPT-7B | GSD (G=25) | 3.77× | 3.53 | 33.1 | 31.2 (CLIP) |
| Lumina-mGPT-7B | ZipAR-16 | 2.31× | 3.12 | 32.6 | 31.2 (CLIP) |
| Lumina-mGPT-7B | AR-Sample | 2.35× | 2.61 | 30.9 | 26.1 (CLIP) |
| Lumina-mGPT-7B | LISA (Ours) | 4.12× | 4.79 | 31.0 | 31.5 (CLIP) |
| LlamaGen-XL-C2I (ImageNet) | Vanilla | 1.00× | 1.00 | 9.78 | 152.3 (IS) |
| LlamaGen-XL-C2I | EAGLE-3 | 0.70× | 1.00 | 9.88 | 144.2 (IS) |
| LlamaGen-XL-C2I | LANTERN | 1.04× | 1.50 | 10.72 | 149.5 (IS) |
| LlamaGen-XL-C2I | LISA (κ=1.0, λ=0.6) | 1.39× | 2.02 | 10.50 | 148.6 (IS) |
| LlamaGen-XL-C2I | LISA (κ=1.1, λ=0.4) | 1.53× | 2.69 | 12.60 | 142.1 (IS) |
Ablation Study¶
Ablations decouple training-side distillation benefits, inference-side GASV threshold sensitivity, and architectural portability across drafters.
| Study Focus | Model & Drafter Backbone | Verification Mechanism | Acc. Len.↑ | Speedup↑ | Note |
|---|---|---|---|---|---|
| Distillation & GASV | LlamaGen-XL-T2I (EAGLE-3) | w/o GASV (Exact Match) | 1.00 | ~1.00× | Vanilla EAGLE-3 baseline under sampling |
| Distillation & GASV | LlamaGen-XL-T2I (Ours) | w/o GASV (Exact Match) | 1.24 | - | Locality distillation alone improves acceptance |
| Distillation & GASV | LlamaGen-XL-T2I (Ours) | GASV (κ=1.0, λ=0.4) | 2.92 | ~2.05× | Joint distillation and soft verification |
| Drafter Architecture | LlamaGen-XL-T2I (Medusa-style) | Vanilla Medusa Rule | 1.19 | 1.04× | Multi-head non-autoregressive drafter baseline |
| Drafter Architecture | LlamaGen-XL-T2I (LISA-Medusa) | w/o GASV (Exact Match) | 1.53 | 1.37× | Distillation transferred to Medusa drafter |
| Drafter Architecture | LlamaGen-XL-T2I (LISA-Medusa) | w/ GASV (Full Framework) | 2.25 | 1.82× | Significant speedup and FID 46.2 on Medusa |
Key Findings¶
- Training-side alignment is vital for visual speculative decoding: Even under rigid exact-match verification (w/o GASV), the drafter trained with locality-informed distillation increases the accepted length from 1.00 to 1.24 on LlamaGen-XL-T2I. Addressing the flat distribution during training resolves fundamental misalignment that inference-only methods cannot fix.
- Controllable speed–quality trade-off via \((\kappa, \lambda)\): Relaxing the geometric drift boundary \(\kappa\) from 0.6 to 1.0 steadily boosts the accepted length from 2.16 to 2.92, whereas tighter cluster posterior thresholds \(\lambda\) enforce stricter preservation. Users can calibrate these two hyperparameters to prioritize maximum throughput or extreme visual fidelity.
- Strong architectural generalizability: Beyond EAGLE-3's autoregressive tree drafter, LISA seamlessly transfers to Medusa-style parallel prediction heads, lifting its speedup from 1.04× to 1.82× while improving FID from 46.9 to 46.2.
Highlights & Insights¶
- Root-cause analysis of visual token dynamics: Identifies that the breakdown of speculative decoding in vision models stems from flat, non-discriminative probability distributions and sampling dispersion, rather than drafter capacity limitations.
- Geometric codebook prior as an inductive bridge: Overcomes discrete token isolation by exploiting the continuous latent geometry preserved in VQ codebooks, constructing whitened drift metrics and cluster representations for safe verification.
- Full-stack co-design across training and inference: Synthesizes structured uncertainty reweighting during distillation with entropy-guided soft verification at runtime, maintaining high structural fidelity up to 4.12× speedup.
Limitations & Future Work¶
- Dependency on discrete codebook geometry: The formulation relies on explicit continuous embedding spaces provided by VQ-VAE or VQ-GAN codebooks. Extending this geometry prior to continuous-valued autoregressive visual generators remains an open challenge.
- Offline calibration overhead: Whitening statistics, cluster assignments, and geometry budgets require an offline rollout profiling step on the target model before deployment.
- Future directions: The authors outline promising opportunities in scaling the geometric speculative framework to autoregressive video generation (leveraging spatio-temporal redundancy) and interleaved vision-language generation in MLLMs.
Related Work & Insights¶
- vs EAGLE-2 / EAGLE-3: While EAGLE excels in language generation, flat visual token distributions cause its acceptance rate to stall under sampling (only 0.70× on LlamaGen-C2I, causing a net slowdown); LISA restores viability with 1.53× to 4.12× practical speedups.
- vs LANTERN: LANTERN relaxes verification criteria during inference but overlooks drafter-target misalignment during training; LISA pairs locality-informed training with soft verification, yielding higher acceptance lengths (3.05 vs 2.40 on LlamaGen-T2I).
- vs SJD / ZipAR / GSD: Parallel Jacobi or grouped verification methods alter the target model's execution pipeline or require significant auxiliary compute; LISA adheres to the plug-and-play drafter–target speculative decoding paradigm without target model modification.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Insightfully uncovers visual token distributional bottlenecks and leverages codebook geometric continuity to guide both distillation and verification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across T2I and C2I benchmarks with leading open-source models, thorough ablations on hyperparameters, and successful cross-architecture validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational narrative, precise mathematical definitions, and comprehensive figures and diagnostics.
- Value: ⭐⭐⭐⭐⭐ Provides an effective, practical acceleration framework for deploying autoregressive vision models in production pipelines.