Skip to content

BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ZhangXinWhut/BiMTokenizer
Area: Audio & Speech
Keywords: speech codec, state-space models, spherical Leech quantization, low-bitrate speech tokenization, semantic-acoustic balance

TL;DR

BiMTokenizer challenges the recent architectural bloat of dual-tower speech codecs by revitalizing the single-tower paradigm with a bidirectional Mamba backbone and non-parametric Residual Spherical Leech Quantization (RSLQ), delivering state-of-the-art acoustic reconstruction and downstream semantic performance at ~1.1 kbps with only 253M parameters.

Background & Motivation

In extending large language models to audio and speech modalities, neural speech codecs serve as the critical bridge transforming continuous waveforms into discrete tokens. An ideal speech tokenizer must satisfy two conflicting demands: maintaining high acoustic fidelity (timbre, prosody, and fine spectral details) for faithful audio reconstruction, and preserving linguistic and phonetic content to align with the discrete reasoning space of language models. At very low bitrates (around 1.1 kbps), this competition becomes acute. Early single-tower codecs such as SpeechTokenizer and Mimi attempted to balance these goals using Residual Vector Quantization (RVQ) or split-RVQ structures with self-supervised distillation, but under aggressive compression they suffer from codebook under-utilization, code collapse, harmonic smearing, and compromised semantic alignment.

To circumvent these single-tower bottlenecks, recent research has gravitated towards dual-tower architectures—exemplified by X-Codec, XCodec2, DualCodec, and XY-Tokenizer. These frameworks dedicate separate encoders to semantic and acoustic feature extraction before fusing them for quantization. While this decoupling relieves the modeling tension within a single encoder, it incurs severe architectural penalties: parameter counts surge to 500M–800M+, training dynamics become unwieldy, and inference latency escalates sharply when tokenizing large-scale speech corpora. This migration raises a fundamental question: has the single-tower approach truly reached its structural limit, or have its core components—the temporal sequence backbone and the vector quantizer—merely remained under-explored?

The authors argue for the latter and systematically upgrade both foundational pillars. Speech is an intrinsically continuous time series where selective state-space models (Mamba) provide natural inductive bias with linear complexity; equipping Mamba with bidirectional scanning grants every latent frame global contextual grounding without the quadratic overhead of self-attention. On the quantization side, Residual Spherical Leech Quantization (RSLQ) maps representations onto the 24-dimensional Leech lattice—the densest known sphere packing in 24 dimensions. Its fixed geometric configuration eliminates codebook collapse by construction and offers an enormous, well-separated discrete space capable of hosting both semantic and acoustic features within a single unified stream. Core idea: Revisit the single-tower codec paradigm by coupling an external bidirectional Mamba backbone with fixed-lattice Residual Spherical Leech Quantization (RSLQ) under training-only dual semantic supervision, achieving acoustic and semantic performance superior to dual-tower baselines at 1.1 kbps using only 253M parameters.

Method

Overall Architecture

BiMTokenizer is an end-to-end, single-stage GAN-based neural speech codec operating in an offline (non-streaming) setting. The codec takes a 16 kHz speech waveform, extracts an 80-channel mel-spectrogram, and processes it through a convolutional front-end that projects the feature dimension to 768 while downsampling the temporal rate from 100 Hz to 50 Hz. An encoder stack of 8 bidirectional Mamba blocks captures long-range temporal dependencies. Next, a lightweight downsampling module stacks adjacent frames by a factor of 4 and applies dilated residual convolutions to yield a 12.5 Hz latent representation. At this frame rate, the 5-layer Residual Spherical Leech Quantization (RSLQ) module partitions the representation into 1 semantic token and 4 acoustic residual tokens. Symmetrically, the decoder upsamples the discrete tokens back to 50 Hz, processes them through 8 mirrored bidirectional Mamba blocks, reconstructs the mel-spectrogram, and converts it into waveform using a jointly trained Vocos vocoder. During training, a frozen semantic teacher (Whisper-small or SenseVoice-small) provides frame-level semantic distillation and synthesized-audio reconstruction alignment; this teacher branch is entirely discarded at inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Waveform & Mel-Spectrogram (16kHz / 80-mel)"] --> B["Bidirectional Mamba Backbone<br/>ExtBiMamba temporal modeling"]
    B --> C["Temporal Downsampler<br/>4x frame reduction to 12.5Hz"]
    C --> D["Residual Spherical Leech Quantization (RSLQ)<br/>1 semantic + 4 acoustic residual levels"]
    D --> E["Temporal Upsampler & BiMamba Decoder<br/>12.5Hz to 50Hz reconstructed mel"]
    E --> F["Vocos Vocoder & Discriminators<br/>Multi-scale mel & adversarial waveform synthesis"]
    G["Frozen Semantic Teacher (Whisper/SenseVoice)<br/>Auxiliary semantic distillation & reconstruction alignment"] -.->|Training-only supervision| D
    G -.->|Training-only supervision| E

Key Designs

1. Bidirectional Mamba Backbone: Causal Horizon Limits and Quadratic Attention Overhead -> External Bidirectional State-Space Recurrence -> Global Context Modeling with Linear Complexity

Speech exhibits dense local correlations alongside long-range phonetic structures. While Transformer self-attention offers a global receptive field, its quadratic compute and memory complexity render it expensive for long speech sequences; conversely, standard causal convolutions or unidirectional Mamba lack future contextual awareness, resulting in harmonic discontinuities during reconstruction. BiMTokenizer incorporates an external bidirectional Mamba (ExtBiMamba) architecture across both its encoder and decoder. For a hidden feature sequence \(h \in \mathbb{R}^{T \times d}\), the sequence and its time-reversed counterpart are processed by parallel Mamba operators with independent linear input and output projections, and their outputs are fused via element-wise addition. Each block follows a Pre-RMSNorm residual topology combined with a 1536-dimensional SwiGLU feed-forward network:

\[h' = h + \mathrm{ExtBiMamba}(\mathrm{RMSNorm}(h))$$ $$h_{\mathrm{out}} = h' + \mathrm{SwiGLU}(\mathrm{RMSNorm}(h'))\]

The per-frame non-linearity provided by SwiGLU complements the linear-in-time recurrent state-space operators, granting the single-tower backbone high expressivity to support semantic distillation and fine spectral reconstruction simultaneously, while scaling linearly in runtime and memory across long audio inputs.

2. Residual Spherical Leech Quantization (RSLQ): Learnable RVQ Codebook Collapse and Feature Entanglement -> 24-Dimensional Leech Lattice Projection -> Non-Parametric High-Capacity Residual Decomposition

Standard Vector Quantization relies on straight-through estimators to update learned codebook centroids, frequently suffering from codebook collapse at low bitrates where majority clusters fall dormant. BiMTokenizer introduces Spherical Leech Quantization to speech codecs, adapting the non-parametric lattice codebook derived from the 24-dimensional Leech lattice. The codebook \(\mathcal{C} \subset \mathbb{S}^{23}\) is fixed to the \(|C| = 196,560\) minimal vectors of the lattice's first shell, conferring an information capacity of \(\log_2 196,560 \approx 17.58\) bits per token. Because the Leech lattice provides the densest known sphere packing in 24 dimensions, its code vectors are uniformly distributed and maximally separated on the hypersphere, eliminating codebook collapse by construction and removing the need for auxiliary commitment or entropy losses.

For an input frame \(f \in \mathbb{R}^C\), a learnable matrix \(W_{\mathrm{down}} \in \mathbb{R}^{24 \times C}\) projects the feature to 24 dimensions, which is normalized onto the unit hypersphere: \(\tilde{f} = \frac{W_{\mathrm{down}}f}{\|W_{\mathrm{down}}f\|_2}\). The nearest codeword is identified via maximum inner product \(k = \arg\max_j \langle \tilde{f}, c_j \rangle\) with \(c_j \in \mathcal{C}\), and mapped back to codec space via \(e = \gamma W_{\mathrm{up}} c_k\) where \(\gamma\) is a learnable scaling factor.

RSLQ employs \(K=5\) quantization levels configured in a split residual structure. Given encoder latent \(u\), the first level quantizes \(u\) directly into semantic token \(e_0 = \mathrm{RSLQ}_0(u)\), guided by semantic distillation. The remaining four levels quantize the successive residual errors \(r_0 = u, r_i = r_{i-1} - e_i\) for \(i=1,\dots,4\), producing the combined quantized latent \(\hat{u} = e_0 + \sum_{i=1}^4 e_i\). At a 12.5 Hz frame rate, this yields an exact bitrate of \(5 \times 12.5 \times 17.58 \approx 1,099\) bps (~1.1 kbps). Empirical analysis reveals that the semantic layer converges to a compact normalized entropy of 0.81 (grouping linguistic units), while the residual acoustic levels maintain high entropy above 0.94, confirming effective division of labor.

3. Auxiliary Semantic Supervision and Reconstruction Alignment: Single-Tower Acoustic Bias -> Dual Frozen Teacher Distillation -> Zero-Overhead Inference

Without explicit guidance, single-tower reconstruction objectives inevitably prioritize low-level spectral energy over phonetic intelligibility. BiMTokenizer addresses this with a dual semantic supervision framework active solely during training, leveraging a frozen ASR teacher \(\mathcal{S}_\tau(\cdot)\) (Whisper-small or SenseVoice-small).

First, frame-level cosine semantic distillation maps the semantic token embedding \(q_{\mathrm{sem}}\) through a lightweight projector \(P_\psi\) to match teacher space \(\tilde{s} = P_\psi(q_{\mathrm{sem}})\), minimizing the cosine distance against reference representations \(s = \mathcal{S}_\tau(x)\):

\[\mathcal{L}_{\mathrm{sem}} = 1 - \frac{1}{T}\sum_{t=1}^T \frac{\tilde{s}_t^\top s_t}{\|\tilde{s}_t\|_2 \|s_t\|_2}\]

Second, a reconstruction alignment loss enforces semantic consistency at the synthesized waveform level. Reconstructed audio \(\hat{x}\) is passed back through the frozen teacher to extract synthesized representations \(\hat{s} = \mathcal{S}_\tau(\hat{x})\), penalizing semantic drift via an L1 objective:

\[\mathcal{L}_{\mathrm{align}} = \frac{1}{T}\sum_{t=1}^T \|\hat{s}_t - s_t\|_1\]

This ensures that the acoustic decoder preserves intelligible linguistic information rather than hallucinating phonetically corrupt acoustic textures. At inference time, the teacher model, projector, and alignment path are entirely removed, maintaining a pure single-tower execution graph.

Loss & Training

The generator is optimized end to end via a multi-task objective:

\[\mathcal{L}_G = \lambda_{\mathrm{rec}} \mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{adv}} \mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{feat}} \mathcal{L}_{\mathrm{feat}} + \lambda_{\mathrm{sem}} \mathcal{L}_{\mathrm{sem}} + \lambda_{\mathrm{align}} \mathcal{L}_{\mathrm{align}}\]

Adversarial training employs a multi-period discriminator (MPD) and a multi-scale STFT discriminator (MS-STFTD) trained with least-squares GAN (LSGAN) loss. The reconstruction loss \(\mathcal{L}_{\mathrm{rec}}\) computes L1 errors across multi-scale mel-spectrograms spanning FFT window sizes \(2^5\) through \(2^{11}\). The loss weights are fixed at \(\lambda_{\mathrm{rec}}=15, \lambda_{\mathrm{adv}}=1, \lambda_{\mathrm{feat}}=1, \lambda_{\mathrm{sem}}=15, \lambda_{\mathrm{align}}=1\). The model is trained on the 960-hour LibriSpeech dataset for 1,000,000 steps on two NVIDIA H100 GPUs using AdamW (\(\beta_1=0.8, \beta_2=0.9\), weight decay 0.01) with an effective batch size of 128 and cosine learning rate decay from \(1 \times 10^{-4}\) to 0 after 30k warmup steps.

Key Experimental Results

Main Results

On LibriSpeech test-clean (16 kHz), BiMTokenizer is compared against competitive low-bitrate neural codecs across codebook specifications, bitrate, frame rate, architecture type, speaker similarity (SIM), intelligibility (STOI, WER), and subjective/objective perceptual quality (PESQ, UTMOS, ViSQOL).

Model Single Tower Params Bitrate (bps) Frame Rate (Hz) SIM (↑) STOI (↑) PESQ-WB (↑) UTMOS (↑) ViSQOL (↑) WER (%) (↓)
Ground Truth – – – – 1.00 1.00 4.64 4.09 5.00 2.16
EnCodec (24 kHz) ✓ 15M 1500 75 0.60 0.85 1.56 1.58 3.59 5.63
DAC (16 kHz) ✓ 74M 1500 50 0.47 0.80 1.25 1.48 3.38 7.80
SpeechTokenizer (16 kHz) ✓ 103.7M 1000 50 0.36 0.77 1.25 2.28 3.15 4.49
BigCodec (16 kHz) ✓ 159M 1040 80 0.84 0.94 2.68 4.11 4.21 2.92
Mimi (24 kHz) ✓ 79M 1100 12.5 0.74 0.91 2.25 3.63 3.85 3.28
SimWhisper-Codec (16 kHz) ✓ 291M 1100 12.5 0.83 0.93 2.72 4.00 4.31 2.75
XCodec (16 kHz) ✗ (Dual) 177M 1000 50 0.68 0.86 2.11 4.06 3.72 2.73
XCodec2.0 (16 kHz) ✗ (Dual) 820M 800 50 0.82 0.92 2.43 4.13 3.83 2.61
DualCodec (24 kHz) ✗ (Dual) 664M 1075 12.5 0.84 0.93 2.71 4.12 4.19 2.46
XY-Tokenizer (16 kHz) ✗ (Dual) 520M 1000 12.5 0.85 0.92 2.50 4.03 4.22 2.46
BiMTokenizer-Whisper (Ours) ✓ 253M 1100 12.5 0.87 0.95 3.03 4.21 4.33 2.44
BiMTokenizer-SenseVoice (Ours) ✓ 253M 1100 12.5 0.85 0.94 2.85 4.18 4.34 2.53

Ablation Study

The table below details component-level ablations evaluating backbone architectures, quantizer types, and semantic supervision loss components at ~1.1 kbps on LibriSpeech test-clean (acoustic reconstruction) and the ARCH benchmark (SLURP intent classification and AudioMNIST digit recognition).

Group Teacher \(\lambda_{\mathrm{sem}}\) \(\lambda_{\mathrm{align}}\) Backbone Quantizer SIM (↑) UTMOS (↑) PESQ (NB/WB) (↑) WER (%) (↓) SLURP (↑) AudioMNIST (↑)
Core Design None 0 0 BiMamba RSLQ 0.88 4.12 3.56 / 2.99 2.55 7.94 78.26
Backbone Ablation None 0 0 Bi-Self-Attn RSLQ 0.85 3.90 3.28 / 2.77 4.15 7.62 75.18
Quantizer Ablation None 0 0 BiMamba GroupFSQ 0.84 4.03 3.42 / 2.84 2.69 7.75 75.31
Quantizer Ablation None 0 0 BiMamba RVQ 0.84 4.00 3.42 / 2.75 2.75 7.81 76.26
Semantic Ablation SenseVoice 0 1 BiMamba RSLQ 0.86 4.15 3.50 / 2.95 2.56 8.58 81.25
Semantic Ablation SenseVoice 15 0 BiMamba RSLQ 0.84 4.18 3.46 / 2.89 2.58 18.22 97.79
Final Variant SenseVoice 15 1 BiMamba RSLQ 0.85 4.18 3.45 / 2.85 2.53 18.48 98.02
Final Variant Whisper 15 1 BiMamba RSLQ 0.87 4.21 3.56 / 3.03 2.44 12.51 86.20

Key Findings

  • Backbone Efficiency & Acoustic Purity: Replacing BiMamba with bidirectional Transformer self-attention (with RoPE) in the reconstruction-only setting degraded WER sharply from 2.55% to 4.15% and PESQ-WB from 2.99 to 2.77. Length-scaling tests show that as sequence duration expands from 5s to 320s, BiMamba exhibits virtually flat MACs/s and RTF, whereas Transformer attention explodes quadratically.
  • Superiority of Leech Lattice Geometry: At identical bitrates without semantic supervision, RSLQ outperformed GroupFSQ (WER 2.69%, PESQ-WB 2.84) and standard RVQ (WER 2.75%, PESQ-WB 2.75). The 196,560-entry Leech lattice provides dense, uniform spatial coverage that avoids codebook under-utilization, maintaining normalized entropy above 0.94 across residual levels.
  • Tuning the Semantic-Acoustic Trade-off: Adding reconstruction alignment (\(\lambda_{\mathrm{align}}=1\)) consistently improved intelligibility and downstream semantic accuracy without hurting perceptual scores. Distillation weight \(\lambda_{\mathrm{sem}}=15\) achieved optimal balance; elevating \(\lambda_{\mathrm{sem}}\) to 20 improved SLURP at the cost of speaker similarity and PESQ.
  • Drastic Reduction in Computational Footprint: BiMTokenizer incurs only 4.79G encoding MACs and 10.03G decoding MACs (total 14.82G MACs/s) with an RTF of 0.007. Compared to dual-tower competitors DualCodec (40.98G MACs, 0.019 RTF) and XY-Tokenizer (82.02G MACs, 0.007 RTF), BiMTokenizer slashes arithmetic complexity by 63.8% and 81.9% respectively.

Highlights & Insights

  • Geometric Regularity Beats Learned Quantization: By replacing learned vector codebooks with the rigid, mathematically optimal 24-dimensional Leech lattice, the model completely bypasses codebook collapse and auxiliary entropy tuning, providing a dependable baseline for multi-scale residual coding.
  • Single-Tower Paradigm Resurgence: Demonstrates that the industry-wide move toward cumbersome dual-tower architectures is not fundamentally necessary for low-bitrate codecs; enhancing single-tower sequence modeling and quantization bottlenecks yields higher performance at half the parameter budget.
  • Extensible Tokenization for Speech LLMs: Downstream autoregressive TTS experiments on Seed-TTS-Eval (after subsampling RSLQ to 2048 entries) show that the resulting discrete representations achieve 1.87% WER on English and 1.72% on Chinese, proving strong cross-lingual utility in generative language modeling.

Limitations & Future Work

  • Codebook Scale vs. Autoregressive Vocabulary Head: The 196,560-entry Leech codebook creates an excessively large prediction space for standard autoregressive language models, necessitating codebook subsampling (e.g. to 2048) in downstream generative TTS. Developing hierarchical prediction techniques that retain full Leech lattice precision without parameter explosion remains open.
  • Architectural Heterogeneity: The current pipeline combines 1D convolutional stems and downsamplers with Mamba blocks. Designing a fully homogeneous, pure-Mamba codec from waveform to bottleneck is a promising next step.
  • Offline Bidirectional Constraint: The ExtBiMamba layers require the entire sequence context, confining BiMTokenizer to offline processing. Developing causal or chunk-wise streaming variants is necessary for interactive low-latency duplex agents.
  • vs SpeechTokenizer / Mimi: Early single-tower codecs relied on RVQ and were constrained by learned codebook capacity, leading to severe harmonic loss at 1.1 kbps; BiMTokenizer leverages the high-capacity Leech lattice to avoid collapse and preserve crisp harmonic transitions.
  • vs XCodec / XCodec2 / XY-Tokenizer: Dual-tower architectures use separate acoustic and semantic encoders, bloating models to 520M–820M parameters and requiring 40G–82G MACs; BiMTokenizer achieves superior PESQ (3.03 vs 2.50) and WER (2.44% vs 2.46%) with a single 253M parameter network requiring only 14.82G MACs.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering introduction of bidirectional state-space modeling and 24-dimensional spherical Leech lattice quantization into end-to-end speech tokenization.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations covering LibriSpeech clean/other, ARCH understanding, Seed-TTS-Eval generation, scaling complexity, and granular ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual narrative, clear problem formulation, and meticulous comparative analyses.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient, single-tower architectural benchmark for low-bitrate neural audio compression and Speech LLM tokenization.