title: >- [Paper Note] STAT: Soft Tail-dropping for Adaptive Visual Tokenization description: >- [ECCV 2026][LLM Efficiency][Adaptive Visual Tokenization] STAT introduces a discrete 1D visual tokenizer with soft tail-dropping and structured priors to adaptively allocate tokens based on visual complexity, achieving SOTA reconstruction and superior causal autoregressive generation. tags: - ECCV 2026 - LLM Efficiency - Adaptive Visual Tokenization - Autoregressive Generation - Soft Tail-dropping date: 2026-09-19 content_hash: fd1c76cac01e78d1
STAT: Soft Tail-dropping for Adaptive Visual Tokenization¶
Conference: ECCV 2026
Paper: ECCV 2026
Project Page: https://zeyuan-chen.com/STAT/
Area: LLM Efficiency
Keywords: adaptive visual tokenization, autoregressive generation, soft tail-dropping, discrete representation, probabilistic tokenization
TL;DR¶
STAT presents a discrete 1D visual tokenizer with a soft tail-dropping mechanism that predicts token-wise keep probabilities guided by content-adaptive and decreasing importance priors, aligning allocated token counts with image perceptual complexity to achieve SOTA reconstruction and scalable causal autoregressive generation.
Background & Motivation¶
Natural images exhibit immense variation in visual complexity, ranging from simple, homogeneous geometric compositions to intricate scenes with rich high-frequency textures and dense objects. In traditional compression codecs such as JPEG for still images and H.264/HEVC for video, as well as modern text tokenizers in large language models, allocating bits or token counts adaptively according to content complexity is an established core principle. However, visual tokenizers in contemporary visual generative models (such as VQ-GAN, DiT, and LlamaGen) almost universally anchor representations to rigid 2D grids or fixed-length 1D sequences, enforcing a constant compression ratio regardless of the underlying content.
This uniform representation paradigm creates a severe structural imbalance: visually simple images are over-represented with redundant tokens that waste computation and complicate generative modeling, whereas complex scenes suffer from under-representation due to token capacity bottlenecks. Although recent attempts like FlexTok and One-D-Piece leverage nested dropout to train ordered 1D sequences for variable-length decoding, they still rely on manually predetermined token counts during inference rather than enabling the model to dynamically infer the appropriate representation length. Furthermore, existing adaptive approaches either require expensive multi-pass recurrent updates or rely on external heuristics, remaining largely incompatible with simple causal autoregressive (AR) models.
This paper tackles the challenge by internalizing length adaptivity within the visual tokenizer itself. Core Idea: STAT introduces a soft tail-dropping adaptive tokenizer that outputs discrete tokens alongside token-wise keep probabilities learned via a content-adaptive prior (correlating expected token count with perceptual reconstruction error) and a decreasing importance prior (enforcing monotonic decay), producing content-aligned representations in a single forward pass and enabling seamless, adaptive-length causal autoregressive generation with favorable scaling behavior.
Method¶
Overall Architecture¶
STAT builds upon a 1D Transformer-based tokenizer architecture that decouples visual tokens from rigid 2D spatial patches. Given an input image \(x \in \mathbb{R}^{B \times C \times H \times W}\), a patch embedding layer with downsampling factor \(f\) maps it to flattened patch tokens \(P \in \mathbb{R}^{B \times N \times D}\). These patch embeddings are concatenated with \(L\) learnable 1D latent tokens \(L \in \mathbb{R}^{B \times L \times D}\) and fed into a ViT encoder for cross-token interaction and representation distillation. The encoder outputs only the latent tokens \(z_l\), which are then quantized by a vector quantizer (VQ) into discrete latent tokens \(z_q\).
To achieve content adaptivity, STAT augments the encoder with a position-aware probability prediction head that predicts a keep probability \(p_{j,i}\) for each token. During training, Bernoulli sampling determines which tokens are retained, using a straight-through estimator (STE) for end-to-end differentiable gradient propagation. A ViT decoder then reconstructs the full image from the masked discrete tokens combined with learnable output tokens \(O\). During generative modeling, an End-of-Sequence (EoS) token is adaptively assigned based on sampled probability thresholds, allowing standard causal next-token prediction models to execute variable-length generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image & Learnable 1D Latent Tokens"] --> B["ViT Encoder & Feature Distillation"]
B --> C["Vector Quantizer & Position-Aware Keep Probability Head"]
C --> D["Dual Structural Priors<br/>Content-Adaptive Prior + Decreasing Importance Prior"]
D --> E["Soft Tail-Dropping & Straight-Through Estimator Masking"]
E --> F["ViT Decoder Variable-Length Reconstruction"]
E -.->|Generative Stage| G["Adaptive EoS Truncation & Causal AR Generation"]
Key Designs¶
1. Progressive Two-Stage Training and Soft Tail-Dropping: Overcoming Rigid Sequence Truncation Directly training adaptive token selection on unaligned latent spaces often causes training instability or trivial collapse. STAT resolves this through an elegant two-stage learning pipeline. In stage one, the model is trained with hard tail-dropping, where a keep length \(K \sim \mathcal{U}(L_{\min}, L_{\max})\) is sampled uniformly to reconstruct images from random prefixes, instilling strong prefix-decoding capacity. In stage two, STAT introduces a position-aware MLP \(g_\theta\) that outputs token-wise keep probabilities \(p_{j,i} = \sigma(g_\theta(z_l[j,i]))\). Binary token selection masks \(m_{j,i} \sim \text{Bernoulli}(p_{j,i})\) are sampled stochastically, and gradients are propagated back through the non-differentiable sampling via a straight-through estimator (STE). Rather than using hard tail cuts, this stochastic soft tail-dropping introduces token-level perturbations during training, acting as an effective sequence regularizer that drastically improves robustness against imperfect token predictions during autoregressive generation.
2. Dual Structural Priors and Budget Regularization: Aligning Perceptual Complexity with Causal Order To shape the predicted probabilities into an effective and structured profile, STAT incorporates two complementary inductive priors. The Content-Adaptive Prior aligns token allocation with perceptual complexity. For each image \(x_j\), perceptual reconstruction error (such as LPIPS) serves as a complexity proxy \(L_{\text{perc}, j}\), which is encouraged to positively correlate with the expected token budget \(T_j = \sum_{i=0}^{L-1} p_{j,i}\) across the training batch via the Pearson correlation coefficient: $\(\mathcal{L}_{\text{content}} = 1 - \mathrm{corr}(L_{\text{perc}}, T)\)$ By contrasting samples across a batch, this objective directs more tokens to visually complex scenes and fewer to simple ones without requiring brittle absolute threshold normalization. Simultaneously, the Decreasing Importance Prior penalizes upward steps along the sequence to enforce monotonic decay: $\(\mathcal{L}_{\text{decrease}} = \sum_{i=1}^{L-1} \max\big(0, p_{j,i} - p_{j,i-1}\big)\)$ This forces earlier tokens to capture primary semantic structures and later tokens to refine local high frequencies, matching the left-to-right causal conditioning of AR language models. Finally, global sequence length is controlled by a Token Budget Regularization term that penalizes the divergence between the image-level mean keep probability \(\bar{p}_j = \frac{1}{L} \sum_{i=0}^{L-1} p_{j,i}\) and a target sparsity prior \(p^* = 0.5\) using Bernoulli KL divergence: $\(\mathcal{L}_{\text{sparse}} = \mathrm{KL}(\mathrm{Bern}(p^*) \parallel \mathrm{Bern}(\bar{p}_j))\)$
3. Adaptive-Threshold EoS Mechanism: Bridging to Vanilla Causal Autoregressive Models Prior adaptive tokenizers frequently struggle to integrate with standard causal AR models, often necessitating custom non-causal attention masks or multi-scale tree structures. STAT leverages the monotonic decay of its probability profiles to introduce a natural End-of-Sequence (EoS) termination. During AR training, a probability threshold \(\tau\) is sampled dynamically in each iteration, designating the first index with \(p_i < \tau\) as the sequence boundary: \(t_{\text{eos}} = \min \{i \mid p_i < \tau\}\). Tokens beyond \(t_{\text{eos}}\) are replaced by EoS and discarded. The AR generative model is trained strictly under the canonical next-token prediction objective: $\(p_\phi(q) = \prod_{t=0}^{t_{\text{eos}}} p_\phi(q_t \mid q_{<t})\)$ Sampling \(\tau\) across iterations functions as continuous data augmentation across varied truncation depths. During inference, the generative model dynamically halts generation upon emitting the EoS token, achieving adaptive-length sampling natively without modifying the standard decoder-only architecture.
Loss & Training¶
The overall objective function during the second stage integrates reconstruction, generative, codebook, and structural regularization terms: $\(\mathcal{L} = \mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{GAN}} + \mathcal{L}_{\text{VQ}} + \lambda_{\text{content}} \mathcal{L}_{\text{content}} + \lambda_{\text{decrease}} \mathcal{L}_{\text{decrease}} + \lambda_{\text{sparse}} \mathcal{L}_{\text{sparse}}\)$ Here, \(\mathcal{L}_{\text{recon}}\) combines pixel-level \(\ell_1\) loss and perceptual LPIPS loss, \(\mathcal{L}_{\text{GAN}}\) is a patch discriminator adversarial loss, and \(\mathcal{L}_{\text{VQ}}\) is the codebook commitment loss. Evaluated on ImageNet at \(256 \times 256\) resolution, the downsampling factor is \(f=16\), latent sequence length is \(L=256\), codebook size is 4,096 with 12-dimensional code entries. During inference for reconstruction, tokens satisfying \(p > 0.5\) are retained for decoding.
Key Experimental Results¶
Main Results¶
STAT was evaluated extensively on ImageNet \(256 \times 256\) image reconstruction and class-conditional image generation against prominent 2D grid tokenizers, 1D sequential tokenizers, diffusion models, and autoregressive architectures.
Table 1: ImageNet \(256 \times 256\) Image Reconstruction Across Tokenizers
| Tokenizer Type | Tokenizer Model | #Tokens | Codebook Size (CB) | rFID ↓ | PSNR ↑ |
|---|---|---|---|---|---|
| 2D Grid | Taming VQ-GAN | 256 | 16,384 | 4.98 | 19.40 |
| 2D Grid | MaskGIT VQ-GAN | 256 | 1,024 | 2.28 | - |
| 2D Grid | Open-MAGVIT2 | 256 | 262,144 | 1.17 | 22.64 |
| 2D Grid | LlamaGen | 256 | 16,384 | 2.19 | 20.79 |
| 1D Sequence | TiTok-S | 128 | 4,096 | 1.71 | 17.80 |
| 1D Sequence | FlexTok | 256 | 64,000 | 1.08 | 17.70 |
| 1D Sequence | One-D-Piece-L | 256 | 4,096 | 1.08 | 19.04 |
| 1D Sequence | STAT (Default Thr 0.5) | 220 (Avg. Adaptive) | 4,096 | 1.15 | 20.22 |
| 1D Sequence | STAT (Thr 0.01) | 230 (Avg. Adaptive) | 4,096 | 0.99 | 20.25 |
| 1D Sequence | STAT (Thr 0.01 +10) | 240 (Avg. Adaptive) | 4,096 | 0.88 | 20.02 |
Table 2: ImageNet \(256 \times 256\) Class-Conditional Generation Across Tokenizers (LlamaGen-XL 775M)
| Tokenizer | AR Architecture | Avg. #Tokens | Codebook Size | gFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|---|---|
| LlamaGen | Vanilla Causal AR (LlamaGen-XL) | 256 | 16,384 | 3.39 | 227.1 | 0.81 | 0.54 |
| LlamaGen | Vanilla Causal AR (LlamaGen-XL) | 576 | 16,384 | 2.62 | 244.1 | 0.80 | 0.57 |
| TiTok-S | Vanilla Causal AR (LlamaGen-XL) | 128 | 4,096 | 4.90 | 191.7 | 0.77 | 0.56 |
| One-D-Piece-L | Vanilla Causal AR (LlamaGen-XL) | 256 | 4,096 | 2.99 | 235.1 | 0.81 | 0.59 |
| STAT | Vanilla Causal AR (LlamaGen-STAT-XL) | 223 | 4,096 | 2.36 | 244.0 | 0.78 | 0.62 |
Ablation Study¶
The core mechanisms of STAT—including adaptive token allocation, soft vs. hard tail-dropping, and variable EoS thresholds—were ablated using the standardized 775M parameter setting.
Table 3: Ablation Study on Dropping Strategies and Training Thresholds
| Dropping Configuration | Recon. #Tokens | rFID ↓ | Recon. PSNR ↑ | Gen. #Tokens | gFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|---|---|---|
| Fixed Token Allocation | 220 | 1.15 | 20.35 | 220 | 2.73 | 234.2 | 0.77 | 0.62 |
| Hard Tail-Dropping | 222 | 1.15 | 20.30 | 223 | 2.67 | 231.2 | 0.76 | 0.63 |
| Fixed EoS Threshold (\(\tau = 0.5\)) | 220 | 1.15 | 20.22 | 221 | 2.49 | 240.7 | 0.79 | 0.61 |
| Soft Tail-Dropping (STAT Full) | 220 | 1.15 | 20.22 | 223 | 2.36 | 244.0 | 0.78 | 0.62 |
Key Findings¶
- Content-Adaptivity Unlocks Generative Superiority: While fixed-token tokenizers achieve similar reconstruction quality (rFID 1.15 vs. 1.15) due to a simplified optimization surface, adaptive tokenization yields substantially superior generative modeling, slashing gFID from 2.73 to 2.36. Content-dependent sequence lengths allocate model capacity proportionally to information density.
- Soft Tail-Dropping Imparts Generation Robustness: Replacing soft tail-dropping with deterministic hard tail-dropping during training deteriorates generative gFID from 2.36 to 2.67. The Bernoulli stochasticity in soft tail-dropping exposes the decoder and AR model to realistic sequential noise, fortifying generative robustness against autoregressive error accumulation.
- Superior Capacity Scaling Over Prior Baselines: As LlamaGen-STAT scales from 775M (XL) to 1.4B (XXL) and 3.1B, gFID steadily improves from 2.36 to 1.91 and 1.75 (with Recall expanding to 0.64). In contrast, standard LlamaGen with 256 tokens plateaus near 1.4B (gFID 3.09) and barely improves at 3.1B (gFID 3.06). STAT successfully re-enables the favorable scaling behavior expected of causal language models.
- Strong Physical Correlation with Compression Size: Empirically, predicted token counts correlate tightly with standard JPEG compressed file sizes (from ~175 tokens / 63 KB for smooth images to 256 tokens / 164 KB for high-frequency textures), proving that STAT genuinely captures visual information entropy.
Highlights & Insights¶
- Endogenous Complexity-Adaptive Allocation: Unlike external LLM-guided or recurrent heuristics, STAT embeds visual complexity assessment directly into its internal token-wise probability head via a correlation loss with perceptual error (\(\mathcal{L}_{\text{content}}\)), achieving single-pass content adaptivity.
- Natural Alignment with Autoregressive Causality: By regularizing probability curves into monotonic decay with \(\mathcal{L}_{\text{decrease}}\), STAT aligns the representation with left-to-right next-token generation, ensuring earlier tokens carry high-level semantic foundations while trailing tokens provide fine detail refinements.
- Minimalist Generative Compatibility via EoS: STAT dispenses with non-causal multi-scale schedules (VAR) or diffusion heads (MAR). A simple sampled EoS token allows an off-the-shelf decoder-only GPT architecture to perform variable-length visual generation with state-of-the-art fidelity.
Limitations & Future Work¶
- Upper Bound Saturation in Extreme Densities: Because the latent token sequence has a predefined hard ceiling (\(L=256\)), highly congested scenes (such as dense crowds or fine-print document text) push almost all token probabilities toward 1.0, saturating the budget without allowing dynamic expansion beyond \(L\).
- Multi-Loss Optimization Balance: Balancing reconstruction, GAN, VQ, content correlation, monotonic decrease, and Bernoulli KL divergence requires careful hyperparameter tuning of \(\lambda_{\text{content}}\) and \(\lambda_{\text{decrease}}\) to prevent profile collapse or gradient competition.
- Future Direction: Multi-Frame Video and Multimodal Unification: Extending soft tail-dropping to spatiotemporal video tokens and unified multimodal autoregressive foundation models could allow models to dynamically expand or contract token budgets based on temporal and linguistic density.
Related Work & Insights¶
- vs LlamaGen (Sun et al., 2024): LlamaGen pioneered scalable causal AR image generation on discrete 2D VQ tokens, but suffered from early performance saturation at 3.1B parameters due to rigid 256/576 token grids. STAT uses fewer tokens (223 on average) while enabling 3.1B models to reach a gFID of 1.75, unlocking true scaling capacity.
- vs TiTok & FlexTok (Yu et al., 2024; Bachmann et al., 2025): TiTok compresses images into fixed 1D tokens, while FlexTok provides variable-length decoding via nested dropout but requires hardcoded, manually chosen token budgets at test time. STAT autonomously infers image complexity to determine representation length on a per-sample basis.
- vs ALIT & KARL (Duggal et al., 2024/2025): ALIT relies on multi-pass recurrent loops to find minimal token counts, significantly multiplying latency; KARL employs external loss heuristics. STAT operates in a single forward pass and demonstrates superior direct applicability to causal AR generative models.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Soft tail-dropping combined with dual structural priors provides an elegant, principled adaptive mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning ImageNet reconstruction, class-conditional generation, GenEval text-to-image, 512x512 scaling, and UCF-101 video.
- Writing Quality: ⭐⭐⭐⭐⭐ Method derivation is rigorous, narrative motivation is clear, and figures seamlessly match design components.
- Value: ⭐⭐⭐⭐⭐ Revitalizes vanilla causal autoregressive modeling for vision with proven scaling laws and dynamic token efficiency.