HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/bovod-sjtu/HoliTok
Area: Audio & Speech
Keywords: Speech Tokenizer, Continuous Latent Space, Unified Generation and Understanding, Progressive Training, AR+DiT Architecture
TL;DR¶
HoliTok introduces a holistic continuous speech tokenizer that compresses 48 kHz audio into a compact 25 Hz sequence of 128-dimensional continuous latents, achieving high-fidelity waveform reconstruction while resolving the representation conflict between speech generation and understanding within a shared AR+DiT backbone.
Background & Motivation¶
Recent progress in multimodal foundation models has shifted the research frontier toward building unified architectures capable of understanding, reasoning, and generating across modalities within a single shared parameter space. In the speech domain, this paradigm imposes stringent demands on the front-end tokenization space: the representation must simultaneously guarantee high-fidelity waveform decodability, facilitate predictable and continuous generative learnability for downstream language models, and encode sufficient high-level semantic abstractions for complex audio understanding. In the absence of such a unified continuous interface, state-of-the-art systems must resort to ad-hoc architectural compromises—such as deploying decoupled multi-stream tokenizers, cascading specialized front-end encoders, or separating understanding and generation into disjoint pipelines, shifting the burden of modality unification onto fragile and cumbersome system designs.
Existing speech representations only satisfy subsets of these criteria. Classical acoustic front-ends such as Mel-spectrograms and filter banks faithfully preserve raw spectral details, yet their dense frame rates and high redundancy make them notoriously inefficient and intractable for autoregressive sequence modeling. Pretrained self-supervised representations such as WavLM and HuBERT offer rich contextual semantics, but they discard phase and fine acoustic textures, cannot be inverted into high-fidelity waveforms, and present noisy, unbounded targets that destabilize generative diffusion heads. While discrete neural codecs provide compact symbols compatible with language models, vector quantization intrinsically introduces irreversible information loss, and multi-codebook residual structures dramatically complicate downstream sequence generation. Recent continuous tokenizers attempt to avoid quantization errors, but they are predominantly optimized in isolation for reconstruction or pure synthesis, failing to maintain semantic structure under unified generation-understanding conditions.
This representational fragmentation is acutely revealed in unified AR+DiT models that execute speech recognition (ASR) and text-to-speech (TTS) simultaneously: representations that perform adequately in standalone TTS collapse under joint multi-task training due to severe negative transfer between acoustic reconstruction and semantic extraction. The central insight of this work is that modality unification should not be solved by engineering patchwork in downstream networks, but rather by progressively shaping a continuous latent space that embeds semantic richness into an acoustic manifold without sacrificing generative smoothness. Core idea: train a continuous speech tokenizer through a three-stage progressive recipe—moving from deterministic autoencoding pretraining to temporal variational bottleneck transfer with implicit fidelity guarantees, and finally to downstream-aware multi-task supervision with multi-granularity distillation—to establish a 25 Hz continuous latent space that robustly serves as a shared interface for unified speech generation and understanding.
Method¶
HoliTok is designed as a foundational representation interface that encodes 48 kHz raw waveforms into a compact 25 Hz stream of 128-dimensional continuous latent vectors (a 7.5× nominal compression ratio). To evaluate its representational integrity, the authors deploy it within a unified downstream spoken language model combining an autoregressive backbone with a diffusion transformer (AR+DiT).
Overall Architecture¶
The core tokenizer consists of a low-latency causal convolutional encoder, a temporal variational bottleneck parameterized by LSTM layers with normalizing flows, and a BigVGAN-style upsampling waveform decoder. In the final training stage, an auxiliary encoder-decoder supervision network (a 0.6B Transformer encoder paired with a pretrained Qwen2.5-0.5B decoder) is temporarily attached to inject cross-modal gradients and is discarded before downstream deployment. In the shared downstream AR+DiT architecture, an autoregressive language model (Qwen2.5-0.5B) consumes interleaved text and audio patch embeddings. For speech generation, the LLM generates causal conditioning hidden states that drive an 18-layer DiT flow-matching module to predict continuous latent patches; for understanding, the LLM predicts text tokens autoregressively.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
IN["Raw Waveform 48 kHz"] --> ENC["Causal Convolutional Encoder<br/>6 downsampling stages·hop size 1920·dilated stacks"]
ENC --> S1
subgraph S1["Stage I: Reconstruction-Oriented Autoencoder Pretraining"]
direction TB
AE["Deterministic Latents z_AE (128-dim)"] --> DEC["BigVGAN-Style Generator<br/>AMPBlocks + SnakeBeta activations"]
DEC --> LOSS1["Multi-scale Mel-spectrogram + Multi-period/CQT Discriminator Loss"]
end
S1 --> S2
subgraph S2["Stage II: Autoencoding-to-Variational Latent Transfer"]
direction TB
LSTM["Temporal Variational Bottleneck<br/>4-layer LSTM + Normalizing Flow"] --> SAMP["Reparameterization Sampling z_VAE"]
SAMP --> FROZEN["Freeze Pretrained Encoder & Decoder<br/>Weak KL regularization (β_low = 0.1)"]
end
S2 --> S3
subgraph S3["Stage III: Downstream-Aware Semantic Enrichment"]
direction TB
DIST["Multi-Granularity Distillation<br/>WavLM layer 23 alignment + x-vector speaker embedding"]
SUP["Multi-Task Language Supervision Network<br/>ASR / Emotion / Sound Events / Captioning"]
REG["Strong Variational Prior Regularization (β_high = 7.0)"]
end
S3 --> LAT["25 Hz Compact Continuous Latent Space (HoliTok)"]
LAT --> ARDIT["Shared Downstream AR+DiT Backbone<br/>Joint TTS Generation & ASR Understanding"]
Key Designs¶
1. Reconstruction-Oriented Autoencoder Pretraining: Preventing Early Prior Collapse and Grounding Acoustic Manifolds
When training a variational autoencoder directly from scratch with standard KL regularization, strong prior constraints force the encoder to discard subtle, high-frequency acoustic details before the decoder can learn an accurate inverse mapping. To eliminate this pathology, Stage I bypasses stochastic sampling and optimizes a purely deterministic autoencoder \(z_{\text{AE}} = E_\phi(x)\). The encoder employs 6 causal convolutional downsampling blocks (kernel sizes 4, 4, 4, 8, 12, 20 with strides 2, 2, 2, 4, 6, 10, totaling a hop size of 1920 corresponding to 25 Hz), yielding a 128-dimensional acoustic stream. The generator mirrors this hierarchy using BigVGAN-style AMPBlocks with SnakeBeta activations, trained alongside multi-period and multi-scale sub-band CQT discriminators via spectral and adversarial losses across 500K steps to establish a high-fidelity acoustic manifold.
2. Autoencoding-to-Variational Latent Transfer: Bounded Perturbation via Implicit Fidelity Transfer
In Stage II, the pretrained encoder \(E_\phi\) and decoder \(G_\psi\) are kept completely frozen, while a temporal variational bottleneck consisting of a 4-layer LSTM network and a normalizing flow is inserted to model the posterior \(q_\eta(z_{\text{VAE}}|z_{\text{AE}})\). Rather than training from scratch, this stage applies weak KL regularization (\(\beta_{\text{low}} = 0.1\)) to gently regularize the latent distribution. The design is grounded in an implicit fidelity transfer theorem: assuming decoder \(G_\psi\) is locally \(L_\psi\)-Lipschitz, with Stage-I reconstruction error \(\epsilon_{\text{AE}} = \mathbb{E}[\|x - G_\psi(z_{\text{AE}})\|_2^2]\) and latent shift \(\delta_{\text{shift}} = \mathbb{E}[\|z_{\text{VAE}} - z_{\text{AE}}\|_2^2]\), the expected waveform distortion satisfies: $\(\mathbb{E}_{x, z_{\text{VAE}}}\left[\|x - G_\psi(z_{\text{VAE}})\|_2^2\right] \le 2\epsilon_{\text{AE}} + 2L_\psi^2 \delta_{\text{shift}}\)$ Because the decoder is frozen and \(\beta_{\text{low}}\) is small, \(\delta_{\text{shift}}\) is tightly constrained, transferring the deterministic autoencoding manifold into a continuous, regularized latent space without degrading audio fidelity.
3. Downstream-Aware Semantic Enrichment: Multi-Granularity Distillation and Multi-Task Language Supervision
In Stage III, the encoder and variational bottleneck are jointly unfrozen under a significantly elevated KL penalty (\(\beta_{\text{high}} = 7.0\)). To prevent representational collapse under strong regularization, the latent trajectory is anchored using two complementary mechanisms: multi-granularity representation distillation and multi-task language supervision. Frame-level latents are projected and aligned with the 23rd layer of a frozen WavLM teacher via cosine similarity, while utterance-pooled latents are aligned with ECAPA-TDNN x-vector speaker embeddings. Simultaneously, an auxiliary Transformer encoder paired with a frozen Qwen2.5-0.5B language decoder enforces causal cross-entropy loss over diverse spoken language tasks (speech recognition, emotion classification, audio captioning, and sound event detection). This corresponds to optimizing an evidence lower bound (ELBO) that jointly explains raw audio, SSL representations, and task targets, ensuring the continuous space retains essential linguistic and paralinguistic semantics for downstream understanding.
Loss & Training¶
The tokenizer optimization proceeds across three sequential phases using the AdamW optimizer (learning rate initialized at \(1 \times 10^{-4}\) with exponential decay to \(1 \times 10^{-6}\), \(\beta=(0.8, 0.99)\)): - Stage I (500K steps): \(\mathcal{L}_{\text{I}} = 45.0 \mathcal{L}_{\text{spec}} + 1.0 \mathcal{L}_{\text{adv}}^{\text{G}} + 2.0 \mathcal{L}_{\text{fm}}\) on 9.6s audio segments. - Stage II (50K steps): \(\mathcal{L}_{\text{II}} = \mathcal{L}_{\text{gen}} + 0.1 \mathcal{L}_{\text{KL}}\), optimizing only the bottleneck parameters. - Stage III (200K steps): Full encoder-bottleneck fine-tuning: $\(\mathcal{L}_{\text{III}} = \mathcal{L}_{\text{gen}} + 7.0 \mathcal{L}_{\text{KL}} + 1.0 \mathcal{L}_{\text{distill}}^{\text{WavLM}} + 1.0 \mathcal{L}_{\text{distill}}^{\text{x-vector}} + 1.0 \mathcal{L}_{\text{sup}}\)$ For downstream unified spoken language modeling, the AR+DiT architecture is trained on a 5:1 mixture of Emilia TTS and multi-source ASR datasets (LibriSpeech, AISHELL-1/2, GigaSpeech, Common Voice, FLEURS), optimizing continuous flow matching \(\mathcal{L}_{\text{FM}}\) for speech generation alongside standard cross-entropy for text prediction.
Key Experimental Results¶
Main Results: Reconstruction Fidelity and Unified Generation-Understanding¶
Evaluated on LibriSpeech test-other, HoliTok achieves competitive acoustic reconstruction while operating at an aggressive 25 Hz frame rate (25 TPS, 7.5× compression ratio):
| Model | Compression Ratio (CR) | TPS | NB/WB PESQ ↑ | STOI ↑ | WER(%) ↓ | SPKSIM ↑ | EMOSIM ↑ | UTMOS ↑ |
|---|---|---|---|---|---|---|---|---|
| Ground Truth | 1.00× | – | – | – | 3.91 | 1.000 | 1.000 | 3.75 |
| Mel Spectrogram | 2.00× | 86 | 4.15 / 4.05 | 0.988 | 3.96 | 0.957 | 0.988 | 3.75 |
| SemanticVAE | 2.73× | 40 | 3.99 / 3.80 | 0.969 | 4.15 | 0.963 | 0.993 | 3.76 |
| MingTok-Audio | 2.19× | 50 | 4.23 / 4.12 | 0.981 | 4.27 | 0.950 | 0.992 | 3.75 |
| Vanilla VAE | 7.50× | 25 | 3.18 / 2.65 | 0.925 | 5.41 | 0.859 | 0.988 | 3.75 |
| HoliTok (Ours) | 7.50× | 25 | 4.10 / 4.01 | 0.974 | 4.22 | 0.968 | 0.995 | 3.75 |
When subjected to unified spoken language modeling under the shared AR+DiT architecture (joint ASR and TTS tasks), baseline representations exhibit catastrophic collapse, while HoliTok maintains robust dual capabilities:
| Model Setting | TTS: Seed-en WER(%) ↓ | TTS: Seed-en SIM ↑ | TTS: Seed-zh WER(%) ↓ | TTS: Seed-hard WER(%) ↓ | ASR: test-clean WER(%) ↓ | ASR: test-other WER(%) ↓ | ASR: AISHELL-1 WER(%) ↓ |
|---|---|---|---|---|---|---|---|
| Semantic-VAE | 102.32 | 0.47 | 99.30 | 97.31 | 9.69 | 21.32 | 15.81 |
| MingTok-Audio | 51.06 | 0.42 | 18.17 | 50.35 | 4.62 | 9.06 | 5.01 |
| HoliTok-Base | 27.85 | 0.52 | 4.40 | 30.44 | 6.45 | 16.51 | 14.92 |
| HoliTok-Unite | 7.20 | 0.55 | 1.78 | 16.79 | 5.48 | 12.65 | 5.93 |
| (DiT Init + HoliTok-Unite) | 3.64 | 0.59 | 1.13 | 12.53 | 5.32 | 13.12 | 7.17 |
Ablation Study: Stage-III Objectives and Downstream Integration¶
Ablation analysis on the unified spoken language modeling setting isolates the contribution of each training signal in HoliTok-Base and evaluates semantic encoder integration in HoliTok-Unite:
| Configuration & Ablation | Seed-en WER(%) ↓ | Seed-en SIM ↑ | Seed-zh WER(%) ↓ | Seed-hard WER(%) ↓ | ASR clean WER(%) ↓ | ASR other WER(%) ↓ | AISHELL-1 WER(%) ↓ |
|---|---|---|---|---|---|---|---|
| HoliTok-Base (Default) | 27.85 | 0.52 | 4.40 | 30.44 | 6.45 | 16.51 | 14.92 |
| — w/o distillation | 22.80 | 0.51 | 3.65 | 19.89 | 6.27 | 16.04 | 15.23 |
| — w/o supervision | 110.05 | 0.39 | 97.56 | 96.81 | 6.44 | 16.59 | 16.63 |
| — w/o both | 103.14 | 0.37 | 98.11 | 97.24 | 6.12 | 16.00 | 14.99 |
| HoliTok-Unite (Default) | 7.20 | 0.55 | 1.78 | 16.79 | 5.48 | 12.65 | 5.93 |
| — freeze semantic encoder | 8.46 | 0.56 | 1.86 | 15.60 | 6.44 | 15.92 | 7.42 |
| — DiT-specialized init | 3.64 | 0.59 | 1.13 | 12.53 | 5.32 | 13.12 | 7.17 |
Key Findings¶
- Multi-task language supervision prevents generative collapse: Removing Stage-III downstream supervision (
w/o supervise) results in near-total failure on TTS generation (Seed-en WER skyrockets from 27.85% to 110.05%), indicating that cross-modal autoregressive task supervision imparts vital structural regularity to the continuous latent space. - Direct SSL distillation impairs diffusion learnability: As demonstrated by Semantic-VAE's high TTS failure rate (102.32% WER), forcing continuous latents into SSL semantic representations without progressive fidelity transfer makes them poorly conditioned targets for DiT-based flow matching.
- Joint modeling exposes representational fragility: While baseline tokenizers perform acceptably on isolated synthesis tasks (Table 2), their capabilities disintegrate when tasked with sharing a backbone with ASR (Table 3), highlighting that HoliTok is the only continuous representation capable of navigating multi-task cross-modal interference robustly.
Highlights & Insights¶
- Theoretical Lipschitz grounding for progressive VAE training: By mathematically bounding the variational reconstruction distortion relative to the Stage-I autoencoder error via local Lipschitz continuity, the authors establish a rigorous theoretical justification for multi-stage continuous tokenization.
- Dismantling the acoustic-semantic dual-stream crutch: Traditional speech foundation models require disjoint encoders for understanding (HuBERT/WavLM) and generation (discrete codecs). HoliTok proves that a 25 Hz continuous latent space can natively serve both modalities without architectural bifurcation.
- Downstream-aware tokenizer evaluation protocol: Instead of confining tokenizer assessment to signal reconstruction metrics (PESQ/STOI), evaluating representations inside a shared AR+DiT multi-task network provides a definitive, realistic benchmark for token usability in foundation models.
Limitations & Future Work¶
- Domain limitation to human speech: While the training mixture incorporates environmental sound and music subsets (AudioSet, MusicCaps), experimental evaluations are exclusively centered on clean/expressive speech reconstruction, TTS, and ASR. Its generalization to complex polyphonic music generation remains unverified.
- Restricted downstream architecture search: Downstream validation is conducted solely within a hybrid AR+DiT framework; the performance of HoliTok within pure non-autoregressive masked transformers or fully autoregressive continuous diffusion architectures warrants future exploration.
- Voice cloning and synthetic speech risks: The high fidelity and controllable paralinguistic expressiveness of HoliTok could be exploited for unauthorized voice impersonation and audio deepfakes, necessitating the integration of robust watermarking and provenance safeguards.
Related Work & Insights¶
- vs Semantic-VAE (Niu et al., 2025): Semantic-VAE directly distills self-supervised speech features into continuous latents, which severely impairs generative learnability in joint ASR–TTS environments; HoliTok balances semantic accessibility with acoustic decodability through progressive fidelity transfer.
- vs MingTok-Audio / Ming-UniAudio (Yan et al., 2025): MingTok connects a frozen low-level acoustic VAE with an external semantic module, creating an asymmetric representation space; HoliTok directly shapes the primary continuous latent space, resulting in superior generative intelligibility (8.59% vs 39.86% average TTS WER).
- vs Discrete Speech Codecs (WavTokenizer, CosyVoice, EnCodec): Discrete codecs introduce quantization artifacts and require multi-codebook modeling strategies; HoliTok provides a single-stream, 25 Hz continuous latent representation natively aligned with continuous flow-matching heads.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (A well-reasoned, progressive training paradigm bridging continuous speech synthesis and spoken language understanding.)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exhaustive evaluations spanning reconstruction, zero-shot TTS, emotional/style controllability, and rigorous shared-backbone ASR+TTS benchmarks.)
- Writing Quality: ⭐⭐⭐⭐⭐ (Clear mathematical formulation of fidelity transfer, cohesive narrative, and well-structured comparative analysis.)
- Value: ⭐⭐⭐⭐⭐ (Provides an essential, open-source continuous tokenization foundation for unified speech-language models.)