Skip to content

FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/AmphionTeam/FlexiSLM
Area: audio_speech
Keywords: Spoken Language Model, Dynamic Frame Rate, Frame Merging, Direct Frame-Rate Conditioning, Thinker-Talker

TL;DR

FlexiSLM is the first spoken language model featuring dynamic and controllable frame rates across both speech input and output, leveraging adaptive cosine frame merging and explicit sinusoidal conditioning to operate anywhere \(\le 12.5\text{ Hz}\) (including \(6.25\text{ Hz}\), \(5.0\text{ Hz}\), and \(4.0\text{ Hz}\)) without retraining, roughly halving inference time at \(6.25\text{ Hz}\) while outperforming fixed-rate 7B baselines.

Background & Motivation

Spoken language models (SLMs) unify speech perception and generation within large language models, powering speech-to-speech dialogue, automatic speech recognition (ASR), text-to-speech (TTS), and general audio understanding under an end-to-end framework. Existing mainstream SLMs, such as Qwen2.5-Omni (25 Hz input / 50 Hz output) and Kimi-Audio (12.5 Hz parallel representations), rely strictly on fixed frame rates. However, natural human speech exhibits highly uneven temporal information density: steady-state vowel prolongations and pauses convey far fewer linguistic and acoustic changes than rapid consonant transitions. Modeling speech at a rigid, uniform frame rate squanders precious autoregressive compute cycles on low-information segments and entirely robs the serving stack of the flexibility to trade fidelity against latency across diverse devices, network conditions, and computational budgets.

Recent advances in neural audio codecsโ€”most notably FlexiCodec, CodecSlime, and VARSTokโ€”demonstrate that temporal redundancy can be compressed via similarity-based frame merging to achieve remarkably low average frame rates while preserving audio reconstruction quality. Nonetheless, prior investigations remained confined to standalone neural codecs or compact 0.3B TTS pipelines. Extending dynamic and variable-rate tokenization to full-fledged end-to-end multi-task SLMs presents substantial system-level hurdles: dynamic frame merging alters sequence lengths unpredictably across utterances and domains, introducing modeling ambiguity into the talker prediction heads; furthermore, steering frame rates by tuning local merge thresholds produces high variance and an intractable one-to-many mapping, preventing operators from guaranteeing bounded real-time factors (RTF).

To resolve this conflict between computational efficiency and controllable deployment budgets, this work integrates dynamic-frame-rate representations directly into an end-to-end thinker-talker SLM architecture. By pairing newly trained input-side frame merging with pretrained FlexiCodec discrete output tokens, and introducing direct frame-rate conditioning via sinusoidal embeddings, the model unifies diverse temporal resolutions within a single parameter set. Core idea: develop an end-to-end spoken language model with dynamic frame rates on both input and output via adaptive cosine frame merging, steered at inference time by direct sinusoidal target frame-rate conditioning to deliver continuous, highly accurate quality-latency trade-offs without retraining.

Method

Overall Architecture

FlexiSLM follows a speech-in, speech-out parallel Thinker-Talker paradigm comprising an audio encoder, an input-side frame merging module, a Thinker LLM backbone, a dual-stream Talker Transformer, and a non-autoregressive (NAR) flow-matching audio decoder with a neural vocoder. Raw speech is initially converted by a 640M parameter pretrained audio encoder into 25 Hz continuous representations. The input-side frame merging module then dynamically compresses these dense vectors into a \(\le 12.5\text{ Hz}\) sequence based on consecutive cosine similarity and local attention alignment. The Thinker LLM backbone (initialized from Qwen2.5-7B-Instruct) processes the compressed continuous tokens and autoregressively generates text hidden states. The 630M parameter Talker Transformer consumes the Thinker hidden states, a sinusoidal encoding of the target frame rate, and previous speech embeddings to predict both FlexiCodec Finite Scalar Quantization (FSQ) discrete tokens and per-token frame lengths via two parallel LM heads. Finally, repeated 12.5 Hz tokens are upsampled to 50 Hz and synthesized into 24 kHz audio waveforms by a frozen flow-matching decoder and a Vocos vocoder. An optional Talker-to-Thinker feedback link allows the Thinker to retain acoustic awareness across generation steps.

The end-to-end information flow across all modules is structured as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Audio Input<br/>Raw Waveform (16/24 kHz)"] --> B["Audio Encoder (Qwen2.5-Omni)<br/>Extracts 25 Hz Continuous Features"]
    B --> C["Input Frame Merging Module<br/>Dynamic Compression to <=12.5 Hz"]
    C --> D["Thinker LLM Backbone (Qwen2.5-7B)<br/>Cross-modal Reasoning & Hidden States"]
    D --> E["Direct Frame-Rate Conditioning<br/>Sinusoidal Target Rate Embedding Injection"]
    E --> F["Talker Transformer Dual-Stream Prediction<br/>Parallel FSQ Semantic Tokens & Frame Lengths"]
    F --> G["Flow-Matching Audio Decoder & Vocos<br/>Reconstructs 24 kHz Waveform via Repeat"]
    F -.->|"Stage 3 Full Fine-Tuning Active"| D

Key Designs

1. Input Frame Merging Module: Eliminating Temporal Redundancy via Local Attention Re-alignment

To compress the 25 Hz continuous feature sequence \(x_1, x_2, \dots, x_T\) from the audio encoder into an efficient dynamic-rate representation, the input frame merging module computes the cosine similarity between adjacent frames: $\(s_t = \frac{x_t \cdot x_{t+1}}{\|x_t\| \|x_{t+1}\|}, \quad t = 1, \dots, T-1\)$ When \(s_t\) exceeds an utterance-specific threshold \(\tau\), contiguous redundant frames are grouped and averaged into a single representation \(\bar{x}_k\) with duration attribute \(l_k\). Because naive average pooling disrupts the temporal manifold and semantic continuity required by LLMs, the module interleaves original and averaged representations into an augmented sequence and passes it through a lightweight 20M parameter Transformer equipped with local attention. The final merged sequence is retrieved from positions corresponding to the averaged features. Ablation studies demonstrate that bypassing this Merging Transformer and directly feeding pooled features to the LLM degrades LibriSpeech ASR test-other WER from 7.20% to 12.33% and drops spoken QA from 68.7/63.0 to 65.7/60.1, confirming its pivotal role in realigning compressed representations.

2. Talker Transformer Dual-Stream Prediction: Synchronous Generation of Variable-Duration Speech Units

Because dynamic-rate tokens each span a variable physical duration, the speech generation module must synthesize both the semantic identity and temporal extent of every speech unit. FlexiSLM deploys a 630M Talker Transformer (hidden size 1280, 20 layers, 8 attention heads) that projects a joint embedding formed by concatenating the Thinker's top hidden state, the target frame rate encoding, and embeddings of previously emitted speech codes and lengths. The Talker uses two parallel prediction heads to output FlexiCodec FSQ semantic tokens and discrete frame duration lengths simultaneously. To prevent speech generation from running ahead of text planning, the Talker FSQ stream applies a 5-token lookahead delay relative to the text sequence. Concurrently, the frame-length stream is delayed by an additional position relative to the FSQ stream, establishing an autoregressive dependency where each token's duration is predicted conditionally on its known phonetic identity.

3. Direct Frame-Rate Conditioning: Overcoming Threshold Ambiguity with Sinusoidal Guidance

Prior dynamic-rate systems controlled average rate indirectly through a scalar merging threshold \(\tau\). However, threshold-to-rate mappings are highly utterance-dependent: an identical threshold of \(\tau = 0.90\) results in wide per-utterance variations spanning 3.91 to 10.74 Hz (\(\sigma \approx 0.70\)) across different test sets, creating an ambiguous training target for the Talker and preventing deterministic latency guarantees in production. FlexiSLM introduces direct frame-rate conditioning. During training, merging thresholds are sampled randomly, the empirical average rate \(r = \frac{\text{total merged frames}}{\text{duration in seconds}}\) is computed per utterance, and \(r\) is embedded using sinusoidal positional encoding: $\(PE(r) = [\sin(r\omega_1), \cos(r\omega_1), \dots, \sin(r\omega_d), \cos(r\omega_d)]\)$ where \(\omega_i = 10000^{-2i/d}\). This continuous conditioning vector is broadcast to every position of the Talker Transformer. At inference time, the user or runtime scheduler directly specifies a target rate (e.g., 6.25 Hz or 4.0 Hz); the model achieves the requested rate with errors under 0.1 Hz and standard deviations down to 0.03โ€“0.06, turning dynamic compression into a predictable, fine-grained control dial.

4. Three-Stage Training Pipeline with Talker-to-Thinker Feedback Integration

To ensure stable convergence of the 7B multi-task architecture, FlexiSLM employs a staged training regimen: - Stage 1 (Talker Pre-training): Freezes the Thinker backbone and trains the randomly initialized Talker Transformer on ~100K hours of TTS data (Emilia and MLS) to master acoustic token prediction. - Stage 2 (Multi-Task LoRA Fine-Tuning): Unfreezes the input frame merging module, trains Thinker parameters using LoRA (\(r=32, \alpha=64\)), and fine-tunes the Talker on 1.4M dialogue samples from FlexiSLM-Data distilled from Qwen3-Omni alongside ASR, TTS, and audio understanding tasks. - Stage 3 (Full Fine-Tuning & Feedback Activation): Merges LoRA weights into the backbone for full-parameter training and activates the Talker-to-Thinker connection. This feedback channel projects the Talker's previous speech and length embeddings back into the Thinker's hidden states, providing explicit acoustic context for subsequent tokens. Ablations reveal that activating this link under LoRA in Stage 2 destabilizes training (dropping spoken QA to 60.5/55.9 and blowing up ASR WER to 12.67%), whereas full-parameter fine-tuning in Stage 3 absorbs the bidirectional feedback smoothly, lifting 12.5 Hz overall score from 70.6/66.1 to 72.4/67.2.

Loss & Training

The network is optimized end to end using a composite cross-entropy objective: $\(\mathcal{L} = \lambda_{\text{text}} \mathcal{L}_{\text{text}} + \lambda_{\text{speech}} \mathcal{L}_{\text{speech}} + \lambda_{\text{speech\_len}} \mathcal{L}_{\text{speech\_len}}\)$ where \(\mathcal{L}_{\text{text}}\) models autoregressive text generation, \(\mathcal{L}_{\text{speech}}\) guides FlexiCodec FSQ token prediction, and \(\mathcal{L}_{\text{speech\_len}}\) supervises duration classification. Weights are configured as \(\lambda_{\text{text}} = 2\) and \(\lambda_{\text{speech}} = \lambda_{\text{speech\_len}} = 1\) (set to 0 for non-speech tokens). During training across 24 A100 (80GB) GPUs, the input frame rate target is sampled uniformly from \(\mathcal{U}(4, 12.5)\text{ Hz}\) while the output threshold \(\tau \sim \mathcal{U}(0.85, 1.0)\). Training utilizes AdamW with a 5% linear warmup and cosine decay to 0.1 of peak learning rates (\(2 \times 10^{-4}\) for Talker/Merging modules; \(2 \times 10^{-5}\) in Stage 2 and \(1 \times 10^{-6}\) in Stage 3 for the backbone).

Key Experimental Results

Main Results

Evaluations were conducted on Kimi-Audio-Evalkit using GPT-5.5 as an LLM judge across OpenAudioBench (Llama Q, Web Q, TriviaQA), VoiceBench (Alpaca, Common, IFEval, SD-QA, AdvBench), and LibriSpeech ASR test splits.

The table below contrasts FlexiSLM-7B against leading large-scale models and 7B fixed-frame-rate competitors in speech-to-text (s2t) and speech-to-speech (s2s) settings:

Model Input FR (Hz) Output FR (Hz) OpenAudioBench ACC (s2t/s2s) VoiceBench IFEval (s2t/s2s) Overall AVG (s2t/s2s) LibriSpeech clean (WERโ†“) LibriSpeech other (WERโ†“)
>7B Models
Qwen3-Omni-30B-A3B 12.5 12.5 82 / 72 3.86 / 2.60 76.5 / 69.7 1.44 2.82
GLM-4-Voice-9B 12.5 12.5 76 / 62 2.32 / 1.97 61.1 / 55.2 2.82 7.66
Fun-Audio-Chat-8B 5.0 5.0 80 / 74 3.81 / 2.55 74.5 / 67.3 - -
7B Fixed-Rate Baselines
Qwen2-Audio-7B 25 - 68 / - 2.25 / - 54.9 / - 1.74 4.04
Mimo-Audio-7B 6.25 6.25 79 / 71 3.24 / 2.12 70.6 / 59.0 - -
Kimi-Audio-7B 12.5 12.5 79 / 69 3.26 / 2.11 69.7 / 57.2 1.80 2.45
Qwen2.5-Omni-7B 25 50 76 / 72 2.68 / 2.12 66.7 / 63.3 2.38 4.21
FlexiSLM-7B-Stage3 (Ours)
FlexiSLM-7B (12.5/12.5) 12.5 12.5 80 / 75 3.22 / 2.33 72.4 / 67.2 1.98 5.79
FlexiSLM-7B (12.5/6.25) 12.5 6.25 80 / 74 3.22 / 2.26 72.3 / 66.2 1.98 5.79
FlexiSLM-7B (6.25/6.25) 6.25 6.25 78 / 71 3.11 / 2.21 70.2 / 64.3 2.55 6.37
FlexiSLM-7B (5.0/5.0) 5.0 5.0 77 / 67 3.14 / 2.09 69.0 / 60.4 3.34 7.85
FlexiSLM-7B (4.0/4.0) 4.0 4.0 75 / 64 2.94 / 1.85 67.2 / 56.5 4.47 9.53

On inference efficiency (single A100 GPU, batch size 1, average prompt duration 7.7 s): - Real-Time Factor (RTF): Qwen2.5-Omni-7B operates at 1.57 RTF. FlexiSLM at 12.5/12.5 Hz reaches 1.17 RTF (\(1.3\times\) speedup, 4.57 TFLOPs). Halving output frame rate to 6.25 Hz slashes RTF to 0.59 (\(2.7\times\) speedup, 3.41 TFLOPs at 12.5 Hz input) and 0.57 (2.73 TFLOPs at 6.25 Hz input). - First 1-Second Chunk Latency: Decreases from 1.62 s at 12.5 Hz output to 1.21 s at 6.25 Hz output (\(0.75\times\)), significantly boosting conversational responsiveness.

Ablation Study

The table below isolates the impact of dynamic tokenization and direct frame-rate conditioning on an 8-GPU, 160K-step Stage 2 baseline evaluated at 6.25 Hz input and 6.25 Hz output:

Model Configuration OAB+VB AVG (s2t / s2s) โ†‘ LibriSpeech clean (WERโ†“) LibriSpeech other (WERโ†“) LibriSpeech-PC TTS (WERโ†“)
FlexiSLM-Stage 2 Baseline 68.7 / 63.0 2.92 7.20 3.11
w/o dynamic output FR (uniform 2-frame merge) 67.7 / 61.0 3.14 7.67 4.95
w/o dynamic input FR (uniform 2-frame merge) 67.5 / 62.9 2.97 7.97 3.12
w/ threshold-controlled output FR 68.2 / 61.7 2.96 7.24 3.53
w/o input Merging Transformer 65.7 / 60.1 6.45 12.33 3.66
Activate Talker-to-Thinker link in Stage 2 (LoRA) 60.5 / 55.9 7.75 12.67 3.15

Key Findings

  • Dynamic frame merging prevents severe quality loss at low rates: Replacing adaptive frame merging with uniform 2-frame pooling at 6.25 Hz causes TTS WER to surge by 59% (from 3.11% to 4.95%) and drops s2s accuracy by 2.0 points. Adaptive compression preserves the temporal resolution of dense phonemes while compressing steady states.
  • Direct frame-rate conditioning provides superior stability: In addition to raising s2s score from 61.7 to 63.0 over threshold control, direct conditioning eliminates per-utterance frame-rate volatility. At target 6.25 Hz, direct control maintains an empirical rate of \(6.24 \sim 6.25\text{ Hz}\) (\(\sigma \le 0.06\)) across all datasets, whereas threshold control at \(\tau=0.86\) drifts between \(3.43\text{ Hz}\) and \(8.86\text{ Hz}\) (\(\sigma \approx 0.60\)).
  • Bottleneck at extreme low rates lies in duration prediction: At 4.0 Hz, discrete audio token loss remains flat at 3.59 (vs. 3.55 at 12.5 Hz), but duration-length prediction Top-1 accuracy collapses from 100% down to 19% (loss rises from 0 to 0.93). Degradation under extreme compression stems from coarse temporal granularity and duration modeling errors rather than phonetic token misclassification.
  • Global audio perception is highly compression-tolerant: On LLaSO-Eval classification benchmarks (emotion, accent, instruments), FlexiSLM maintains 64.1% accuracy even under aggressive 4.0 Hz input compression (compared to 65.8% at 12.5 Hz), demonstrating that sequence-level classification requires only global acoustic cues.

Highlights & Insights

  • Unified zero-retraining multi-rate serving: Conventional systems demand deploying separate checkpoints for different compute tiers. FlexiSLM parameterizes frame rate as a continuous control input, enabling a single model to adapt dynamically from 12.5 Hz down to 4.0 Hz depending on instantaneous server loads.
  • Dual-end dynamic compression loop: Unlike previous works focusing solely on generation or speech recognition, FlexiSLM dynamically manages temporal resolutions on both the input side (continuous features) and output side (discrete FSQ codes) while keeping the LLM backbone stable.
  • Foundational architecture for full-duplex SLMs: The Talker-to-Thinker feedback loop provides the reasoning backbone with real-time awareness of emitted acoustic tokens, offering an architectural stepping stone toward concurrent listening, reasoning, and speaking.

Limitations & Future Work

  • Non-streaming constraint: Both the frame merging module and the flow-matching audio decoder currently process full-sequence audio, preventing true causal streaming dialogue; future work must adapt them into chunk-based or causal pipelines.
  • Acoustic degradation below 6.25 Hz: At 5.0 Hz and 4.0 Hz, human MOS evaluations show pronounced drops in naturalness (3.69) and fidelity (3.66) despite preserving intelligibility (4.21).
  • Absence of RL post-training: FlexiSLM relies entirely on supervised multi-task distillation; incorporating reinforcement learning from human feedback (RLHF/DPO) tailored to prosody, naturalness, and turn-taking is necessary to match closed-source voice models.
  • vs Qwen2.5-Omni / Kimi-Audio: Dominant open-source SLMs enforce fixed frame rates (25/50 Hz or 12.5 Hz). FlexiSLM outperforms Qwen2.5-Omni-7B (72.4/67.2 vs. 66.7/63.3 overall) at 12.5 Hz and remains superior at 6.25 Hz while reducing latency and computational overhead.
  • vs FlexiCodec-TTS: FlexiCodec proposed similarity merging for 0.3B TTS, relying on indirect threshold tuning. FlexiSLM scales the paradigm to 7B multi-task SLMs and replaces ambiguous thresholds with direct sinusoidal frame-rate conditioning.
  • vs Grouped Patching (Mimo-Audio / Fun-Audio-Chat): Fixed patching groups \(N\) consecutive tokens uniformly. While patching offers a strong fixed-rate baseline, FlexiSLM's content-adaptive compression better respects natural speech information density and grants continuous runtime controllability.

Rating

  • Novelty: โญโญโญโญโญ First spoken language model with dynamic, controllable frame rates across input and output, introducing direct sinusoidal frame-rate conditioning.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous benchmarks spanning OpenAudioBench, VoiceBench, ASR, TTS, audio understanding, MOS listening tests, and training-eval prompt deduplication.
  • Writing Quality: โญโญโญโญโญ Clear architecture taxonomy, crisp motivation, and thorough exploration of efficiency-quality trade-offs.
  • Value: โญโญโญโญโญ Provides a foundational blueprint for elastic, resource-aware conversational AI deployment on edge devices and cloud infrastructure.