MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech¶
Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/dvlab-research/MGM-Omni
Area: Audio & Speech
Keywords: Omni-modal understanding, long-form speech generation, chunk-based parallel decoding, zero-shot voice cloning, timbre consistency
TL;DR¶
MGM-Omni separates multimodal understanding and speech synthesis into brain-mouth tracks, combining dual audio encoders with chunk-based parallel decoding to achieve 4.98% English WER and 0.19 RTF with a 2B SpeechLM on its Long-TTS-Eval benchmark while conditioning generation on a reference voice.
Background & Motivation¶
An omni-modal model must not only understand inputs but also speak its answers to image, video, or audio questions fluently. Connecting a text model to a short-utterance text-to-speech (TTS) system provides basic interaction, but long narration poses a different problem: text advances quickly while speech unfolds at a much finer temporal resolution. As generation continues, text and its corresponding speech become increasingly separated, allowing omissions, repetitions, or speaker drift to accumulate. Extending the context window alone does not resolve this cross-modal rate mismatch.
The CosyVoice2 speech tokenizer used here produces 25 discrete tokens per second, whereas people typically speak only two or three words per second. Words are not equivalent to text tokens, but this comparison illustrates the substantial rate difference. Forcing the understanding model to follow every acoustic token would slow text responses and complicate long-sequence alignment. Meanwhile, understanding more than an hour of audio requires both linguistic content and general acoustic cues; treating all sound as text awaiting transcription is insufficient.
MGM-Omni therefore modifies both ends of the system. Its input side combines representations specialized for general audio and speech recognition. Its output side preserves an independent text track and lets SpeechLM synthesize speech within shorter local alignment spans while retaining historical context. Core Idea: prevent text and speech from drifting apart across an entire long sequence by constraining local alignment through history-preserving chunks, then predict multiple acoustic tokens in parallel to narrow the generation-rate gap.
Method¶
Overall Architecture¶
Inputs can include text, images, video, and audio; outputs are text and its spoken realization, not arbitrary generated modalities. The understanding model builds on Qwen2.5-VL with dual audio encoders and an information-mining module. Its generated text tokens pass to a separate SpeechLM, which also receives reference speech, produces discrete speech tokens through chunk-based parallel decoding, and converts them into waveforms using flow matching and a HiFi-GAN vocoder.
The brain-mouth design is a token interface between two models, not a requirement to finish the entire response before reading it aloud. The understanding model determines answer content, while SpeechLM handles acoustic expansion. They are trained separately, each in two stages. Solid arrows below indicate inference data flow; dashed arrows indicate training supervision only, not training data required at deployment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Audio["Input audio"] --> Mining["Dual-Audio Information Mining"]
Mining --> Tracks["Brain-Mouth Token Interface"]
Other["Text, images, video"] --> Tracks
Tracks --> Chunk["History-Preserving Chunked Decoding"]
Chunk --> Parallel["Parallel Speech-Token Prediction"]
Reference["Reference speech"] --> Parallel
Parallel --> Wave["Flow matching and vocoder<br/>Output speech"]
Understanding["Transcription and multimodal QA supervision"] -.-> Tracks
Synthesis["Paired text-speech supervision"] -.-> Parallel
Key Designs¶
1. Dual-Audio Information Mining: general acoustic features retrieve linguistic cues
The primary encoder comes from Qwen2-Audio, continually trained from Whisper-large-v3 for general sound perception. The auxiliary Belle-Whisper-large-v3 encoder emphasizes Chinese and English speech recognition. These are not simply duplicate features: the primary branch preserves acoustic information beyond transcription, while the auxiliary branch contributes finer linguistic content. Primary features supply queries, and auxiliary features supply keys and values. The retrieved semantic cues are combined with primary features and passed through an MLP to produce audio tokens for the language model.
This asymmetric fusion keeps general audio representations in the leading role while allowing them to retrieve recognition-oriented information as needed. The auxiliary encoder alone is not necessarily better for speech tasks; it actually performs worse in the CommonVoice ablation. The benefit comes from complementary fusion. Long-audio training additionally groups samples by length and adjusts batch size dynamically: smaller batches prevent memory exhaustion for long inputs, while larger batches improve utilization for short inputs. This is a scheduling strategy for joint training across lengths, not an additional memory-retrieval module or a new context-compression algorithm.
2. Brain-Mouth Token Interface: separate answer content from acoustic expansion
The brain inherits Qwen2.5-VL's image and video encoding capabilities and incorporates the new audio tokens into multimodal understanding. The mouth is initialized from Qwen3 and adds a TTS-Adapter consisting of 6 randomly initialized Qwen3 blocks at its output. The brain's 7B/32B variants and the mouth's 0.6B/2B/4B variants describe different components. MGM-Omni-TTS-2B in the experimental tables does not mean that the entire omni-modal system has only 2B parameters.
Text tokens are central to this interface. SpeechLM conditions on the brain's generated words and uses reference speech to establish the target timbre, without fine-tuning for each new speaker. It generates CosyVoice2 finite scalar quantization (FSQ) tokens rather than waveforms. CosyVoice2's chunk-aware flow-matching model subsequently converts these tokens into Mel-spectrograms, and HiFi-GAN synthesizes the waveform. This reuses a mature streaming acoustic backend while avoiding repeated visual reasoning at the rate of 25 acoustic tokens per second.
3. History-Preserving Chunked Decoding: bound local alignment distance without resetting context
Text from the understanding model first enters a buffer. A chunk is formed when a sentence boundary is detected and the buffer contains more than 50 text tokens. If the buffer exceeds 150 tokens, it is forcibly chunked even without a suitable sentence boundary. Boundaries are detected from periods, question marks, exclamation marks, and line breaks; each chunk typically corresponds to about 20–40 seconds of speech. The thresholds 50 and 150 count tokens, not characters or seconds, so chunk duration varies with language and speaking rate.
Within a chunk, a delay strategy places padding at the speech positions corresponding to the first 4 text tokens before speech prediction begins, giving synthesis a small textual lead. Successive chunks are not independent TTS requests: previously generated text and speech remain conditioning context, and the reference voice continues to constrain generation. This prevents unbounded local text-speech separation without sacrificing cross-chunk prosodic and timbral continuity through complete resets. Retaining history also means that computational cost cannot simply be claimed to be independent of total duration.
4. Parallel Speech-Token Prediction: expand several acoustic positions in one model step
Each decoding step combines the embeddings of one text token and several speech tokens by averaging them into the model input representation. The output hidden state passes through the TTS-Adapter and prediction head to produce the next group of speech tokens at once. The default parallel size is 4, so the number of model steps no longer needs to equal the number of acoustic tokens. Its role differs from chunking: chunking primarily constrains long-distance alignment, while parallel prediction reduces serial acoustic-generation steps and further narrows the rate gap between the tracks.
The authors emphasize parallel prediction with FSQ representations rather than merely adopting the common multi-codebook parallelism of residual vector quantization (RVQ). Predicting more positions simultaneously is not lossless: increasing the parallel size from 1 to 4 slightly raises short-speech error rates but lowers RTF from 0.57 to 0.19. The cached vocabulary-size, embedding, and attention equations have extraction damage, so this note describes operations explicitly supported by readable prose rather than presenting guessed repairs as the authors' equations.
A Worked Example¶
Suppose a user requests a long explanation of a video and supplies a short, authorized reference voice clip. After processing the video and instruction, the understanding model streams text into SpeechLM's buffer. The reference clip specifies speaker timbre, not the content of the explanation.
If a sentence boundary is reached when the buffer contains 60 text tokens, the first chunk is formed. If no sentence boundary arrives, chunking is forced after the buffer exceeds 150 tokens. The first 4 text positions lead the speech stream, after which acoustic tokens are generated 4 at a time and converted into waveform chunks by the backend. These numbers illustrate the chunking rules; they are not additional experimental findings.
When the second chunk arrives, the first chunk's text and speech history are not cleared. Narration continues under the same context and reference timbre. The objective is coherent long-form narration rather than concatenating unrelated short utterances; the paper does not yet support fully duplex interaction in which users can interrupt while the system listens and speaks simultaneously.
Loss & Training¶
The understanding model first aligns audio to text using transcription data, then performs unified omni-modal training with transcription, audio question answering, speech-instructed visual question answering, and text instruction data. Both stages use LoRA. Length grouping and dynamic batch sizes support this training path, exposing a single model to both short and long audio.
SpeechLM uses a next-speech-token prediction objective. Pretraining freezes the existing Qwen3 parameters and updates only the randomly initialized TTS-Adapter, aligning the new speech representations with text. Post-training updates both components, with the Adapter learning rate set to 5 times that of the base LLM. This avoids letting an initially random acoustic module substantially disrupt the language model, while later allowing joint adaptation for fluent synthesis.
The speech corpus contains approximately 400k hours: about 300k hours of public raw speech and 100k hours of Chinese and English synthesized speech. For synthesis, the authors sample 900k Chinese and 700k English conversations by length from Belle-10M and Lamini-Instruct, regenerate responses with Qwen2.5-72B, and synthesize them with megatts3 and randomly assigned reference voices. Post-training primarily uses high-fidelity synthesized speech with a smaller raw-speech portion. The paper variously says "over," "around," and "under 400k," so this should be treated as an approximate scale rather than a basis for an exact data-efficiency ratio.
Key Experimental Results¶
Main Results¶
Long-TTS-Eval contains 360 English and 356 Chinese samples in its long subset, covering literature, news, knowledge, talks, comments, and papers. Its hard subset contains 262 English and 265 Chinese samples testing URLs, emails, phone numbers, large numbers, and formulas. English uses word error rate (WER), and Chinese uses character error rate (CER), both lower-is-better. Evaluation computes errors against both original references and GPT-5-normalized references and takes the smaller value; these results are not directly equivalent to single-reference evaluation.
The following results are selected from Table 6b. Real-time factor (RTF) is generation time divided by generated audio duration, with lower values indicating faster synthesis. Speed is evaluated on a Long-TTS-Eval subset. Section 4.3 specifies a single H800 and 16 Chinese plus 16 English samples for the parallel-decoding ablation; throughput under this hardware condition should not be interpreted as time to first audio.
| Model | Speech Model Size | RTF ↓ | long EN WER (%) ↓ | long ZH CER (%) ↓ | hard EN WER (%) ↓ | hard ZH CER (%) ↓ |
|---|---|---|---|---|---|---|
| CosyVoice2 (chunk mode) | 0.5B | 0.34 | 14.80 | 5.27 | 42.48 | 32.76 |
| MOSS-TTSD-v0.5 | 2B | 0.23 | 8.69 | 6.82 | 62.61 | 62.97 |
| Higgs-Audio-v2 | 6B | 0.33 | 27.09 | 31.39 | 98.61 | 98.85 |
| MGM-Omni-TTS-2B | 2B | 0.19 | 4.98 | 5.58 | 26.26 | 23.58 |
MGM-Omni leads in most conditions shown here, but its ordinary Chinese long-subset CER is worse than CosyVoice2's 5.27%. Model sizes and training corpora differ, making these system-level comparisons rather than controlled comparisons at equal training budgets.
Table 6c divides each long generation into 30-second segments, measures speaker similarity between each segment and the reference voice, and reports the mean and standard deviation. A higher mean indicates a closer reference timbre; a lower standard deviation indicates greater consistency across segments.
| Model | EN SIM (%) ↑ | EN Std. ↓ | ZH SIM (%) ↑ | ZH Std. ↓ |
|---|---|---|---|---|
| CosyVoice2 | 83.88 | 1.74 | 92.23 | 1.82 |
| MOSS-TTSD-v0.5 | 80.68 | 3.60 | 88.93 | 2.23 |
| MGM-Omni-TTS-2B | 84.66 | 1.63 | 94.32 | 0.97 |
Ablation Study¶
The three parts of Table 7 use different tasks and must be read separately. Audio encoders are evaluated on CommonVoice, without the long-audio QA training data used for the final model. Parallel-decoding quality is evaluated on Seed-TTS-Eval, with speed measured on the H800 subset described above. Chunking quality is evaluated on Long-TTS-Eval.
| Ablation and Data Setting | Config | EN WER (%) ↓ | ZH CER (%) ↓ | RTF ↓ |
|---|---|---|---|---|
| CommonVoice, Table 7a | Qwen2-Audio encoder | 13.0 | 3.9 | Not reported |
| CommonVoice, Table 7a | Belle-Whisper encoder | 21.7 | 5.0 | Not reported |
| CommonVoice, Table 7a | Dual-encoder information mining | 9.1 | 3.5 | Not reported |
| Seed-TTS-Eval + speed subset, Table 7b | Parallel size 1 | 1.86 | 1.15 | 0.57 |
| Seed-TTS-Eval + speed subset, Table 7b | Parallel size 2 | 2.02 | 1.23 | 0.32 |
| Seed-TTS-Eval + speed subset, Table 7b | Parallel size 4 | 2.28 | 1.28 | 0.19 |
| Long-TTS-Eval, Table 7c | Without chunking | 31.84 | 8.97 | Not reported |
| Long-TTS-Eval, Table 7c | With chunking | 4.98 | 5.64 | Not reported |
The chunked Chinese CER is 5.64% in Table 7c but 5.58% in Table 6b. The main text does not explain the discrepancy, and this note preserves both values. Removing chunking has a much larger effect on English error rates than changing parallel size, but the evaluations use different datasets, so their absolute differences cannot directly rank all components by importance.
Key Findings¶
- Figure 5 reports an average long-audio retrieval success rate of 94% over 5 materials from different topics, versus 58% for Qwen2.5-Omni. Although each source recording exceeds 3 hours, evaluated needle-in-the-haystack durations reach at most 4,500 seconds; this is not an evaluation of complete 3-hour recordings.
- In Table 6a, the 2B SpeechLM achieves 2.28% English WER and 1.28% Chinese CER on Seed-TTS-Eval. Short-utterance accuracy and long-form stability require separate evaluation rather than substituting one metric for the other.
- In Table 4, the 7B model improves ChartQA from Qwen2.5-VL's 87.3 to 87.8, but AI2D falls from 85.8 to 83.2. Visual ability is largely retained after adding audio, but a claim of no single-modality degradation does not hold for every benchmark.
Highlights & Insights¶
- Chunking does more than avoid memory limits: it constrains the distance between text and acoustic sequences. Retaining history allows this local constraint without reverting to independent short-utterance synthesis.
- Content accuracy, text-normalization accuracy, and stable timbre are distinct objectives. Separating long/hard error rates from segment-level speaker-similarity means and standard deviations diagnoses long-speech failures more effectively than one aggregate similarity score.
- Decoupling the brain and mouth allows understanding and speech model sizes to be chosen separately. This is a reusable interface design, but system parameters and costs must include all components rather than only the speech model.
Limitations & Future Work¶
- The authors acknowledge limitations in extremely long interactions, fine-grained speaking-style control, and rare acoustic conditions. Visual understanding trails specialized VLMs, and the system lacks tool use and fully duplex speech interaction.
- Assessment in this note: the long-audio needle test covers only 5 source materials and mainly measures retrieval. It does not independently establish long-meeting reasoning or persistent multi-turn memory. The main text refers to an appendix, but the assigned cache ends with references; no conclusions from an unread appendix are added here.
- Assessment in this note: comparing approximately 400k hours with other systems' larger corpora does not isolate architecture-driven data efficiency. Learning curves matched for data, compute, and model size, together with time-to-first-audio and listening evaluations, remain necessary.
- The authors require speaker consent for reference voices and recommend provenance metadata, watermarking, and synthetic-speech detection in deployment. These are responsible-use recommendations, not demonstrated anti-misuse features already implemented in the system.
Related Work & Insights¶
- Compared with Qwen2.5-Omni: both separate understanding from speech production. MGM-Omni emphasizes reference-voice conditioning and long-horizon chunk-based parallel generation, so the brain-mouth division itself should not be portrayed as an entirely unprecedented idea.
- Compared with CosyVoice2: MGM-Omni reuses its FSQ tokenizer and flow-matching backend, placing them after an interface between omni-modal understanding and a separate SpeechLM. Ordinary Chinese long-form results still show that a specialized TTS baseline remains competitive.
- Compared with Lyra and Mini-Gemini: the understanding branch draws respectively on audio-training procedures and information mining, adding unified length-aware training scheduling. The transferable lesson is to establish representation complementarity before designing asymmetric retrieval, rather than simply adding encoders.
Rating¶
- Novelty: 4/5. Chunking, parallel prediction, and dual tracks have precedents; the contribution is their combination and validation for FSQ-based long-form speech.
- Experimental Thoroughness: 4/5. Understanding, short/long synthesis, timbre, and ablations are covered, but the custom long-speech evaluation and budget fairness need further support.
- Writing Quality: 4/5. The mechanisms are clear overall, with inconsistencies in corpus-size wording and some table values.
- Value: 4/5. The paper offers a relatively complete open-system design and evaluation framework for personalized long-form speech.