SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://hasaki321.github.io/SimulS2ST-Omni.demo/
Area: Audio & Speech
Keywords: Streaming Speech-to-Speech Translation, Speech Language Models, Trajectory Supervision, Two-Stream Architecture, Low-Resource Data Efficiency
TL;DR¶
SimulS2ST-Omni establishes an explicit joint text-code trajectory supervision framework paired with a decoupled Thinker–Talker architecture, delivering state-of-the-art sentence-level and long-form streaming S2ST using only ~2,000 hours of paired cross-lingual audio.
Background & Motivation¶
Real-time, continuous speech-to-speech translation (Streaming S2ST) demands incremental comprehension, cross-lingual semantic planning, and high-fidelity target audio synthesis over unbounded source speech streams under rigorous latency constraints. Traditional cascaded architectures (ASR + MT + TTS) incur prohibitive pipeline wait times and severe cascading error propagation, rendering them suboptimal for natural simultaneous interpretation. While end-to-end Speech Language Models (SLMs) have recently shown impressive mastery in offline, sentence-bounded processing, existing streaming systems heavily rely on massive synthetic supervision—routinely exceeding 40,000 hours of paired source-target audio—or necessitate intricate reinforcement learning and human-annotated alignments. Achieving robust, continuous streaming S2ST under practical, data-constrained regimes has thus stood as a fundamental, long-standing bottleneck.
This data bottleneck is profoundly compounded by two structural obstacles in streaming audio generation. First, long-form simultaneous translation is inherently a latent path optimization problem, wherein the system must dynamically choose when to read, wait, or write based solely on incomplete source context. While trajectory-based methods successfully schedule streaming in text-only spaces, they fail when directly applied to speech generation, which requires the irrevocable, time-sensitive emission of dense acoustic semantic codes along speech boundaries. Second, training a single unified decoder to simultaneously handle source speech understanding, target text planning, and high-rate acoustic code prediction creates severe modality representation interference: maximizing linguistic comprehension capacity actively degrades dense acoustic modeling, and vice versa.
To resolve this conflict under tight data budgets, this paper proposes that linguistic reasoning and dense acoustic generation should be structurally factored rather than forced to compete within a single decoder vocabulary. Core idea: Introduce explicit joint text-code trajectory supervision to schedule target text and acoustic semantic tokens along a unified monotonic commitment path, coupled with a decoupled Thinker–Talker two-stream architecture that isolates linguistic planning from acoustic prediction, enabling data-efficient long-form streaming S2ST with only ~2,000 hours of paired audio.
Method¶
Overall Architecture¶
SimulS2ST-Omni consists of an offline trajectory mining pipeline and an online two-stream streaming generation engine. Given an unbounded source speech stream \(X\), the goal is to incrementally emit target speech waveforms \(Y_{\text{wav}}\). The target speech is represented as discrete acoustic semantic codes \(Y_{\text{code}} = Q(Y_{\text{wav}})\) via a frozen DualCodec tokenizer. Word-level forced alignments and cross-lingual SimAlign projections map each target word to its earliest source-speech prefix, and a monotonicity filter schedules target words and their acoustic codes into fixed-duration read/wait/write chunks \(\tau\). Online, the two-stream backbone decouples translation into a 3B Thinker (modeling target text tokens and high-level hidden representations \(H_\theta\)) and a lightweight 0.4B Talker (conditioned on \(H_\theta\) to emit dense acoustic codes). The emitted codes are then reconstructed into continuous audio by a frozen chunk-wise Flow Matching decoder and vocoder conditioned on sliding audio context.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
InAudio["Continuous Source Speech X"] --> Enc["Speech Encoder<br/>Extracts acoustic representations"]
subgraph DataPipeline["Trajectory Construction"]
direction TB
FA["Forced Alignment & SimAlign<br/>Extract word & cross-lingual boundaries"] --> TrajBuild["Monotonization & Chunk Grouping<br/>Build unified text-code trajectory τ"]
end
Enc --> Thinker["Thinker Linguistic Stream<br/>Autoregressively plans target text & Hθ"]
TrajBuild -.->|Teacher trajectory supervision| Thinker
Thinker --> Talker["Talker Acoustic Stream<br/>Predicts dense acoustic codes Ycode"]
TrajBuild -.->|Chunk boundary alignment| Talker
Talker --> CodecFM["Frozen Flow Matching & Vocoder<br/>Sliding audio context smoothing"]
CodecFM --> OutWave["Low-Latency Target Speech Ywav"]
Key Designs¶
1. Monotonized Joint Text-Code Trajectory Supervision: Problem of Latent Path Decisions -> Mechanism of Aligned Discrete Trajectories -> Effect of Eliminating Unstable Emission Controllers
Streaming S2ST requires early commitments before source utterances finish, but cross-lingual syntactic inversion frequently misaligns target words to delayed source frames, risking premature and irreversible speech emission. To establish stable, step-by-step supervision without relying on complex policy networks or reinforcement learning, the authors extract cross-lingual word alignments using SimAlign and map each target word \(y_i\) to its corresponding source-word end timestamp \(t_{a(i)}\). A monotonicity recursion resolves local word reordering:
$\(\tilde{t}_i = \max(\tilde{t}_{i-1}, t_{a(i)}), \quad \text{with } \tilde{t}_0 = 0\)$
Both target text words and their corresponding segmented semantic speech codes inherit the identical monotonized source boundary \(\tilde{t}_i\). These paired tokens are then grouped into fixed-interval source chunks (e.g., 1-second intervals). Chunks without new target content act as explicit wait steps, while chunks with content act as write steps emitting \((Y_{\text{text}}^c, Y_{\text{code}}^c)\). This converts the continuous latency policy into a discrete, supervised multi-task classification objective.
2. Thinker–Talker Two-Stream Decoupling: Problem of Modality Interference & Decoding Overhead -> Mechanism of Factorized Conditioning -> Effect of Preserving Intelligence and Slashing Latency
Forcing a unified 3B decoder to predict 16,384 discrete codec tokens alongside natural text vocabulary creates conflicting gradient demands between high-level cross-lingual understanding and low-level acoustic token prediction, while also requiring massive compute for dense code generation. The method resolves this by factorizing the chunk joint probability: $\(p(Y_{\text{text}}^c, Y_{\text{code}}^c \mid C_c) = p_\theta(Y_{\text{text}}^c \mid C_c) \cdot p_\phi(Y_{\text{code}}^c \mid C_c, Y_{\text{text}}^c, H_\theta)\)$ where \(C_c\) denotes the chunk context (consumed source speech and previously generated target tokens), and \(H_\theta\) is the intermediate hidden state sequence emitted by the Thinker \(\theta\). The 3B Thinker focuses exclusively on linguistic translation at word-level rates, while the lightweight 0.4B Talker \(\phi\) consumes \(H_\theta\) to rapidly predict dense acoustic codes. This decoupling eliminates vocabulary pollution, protects translation intelligence, and significantly reduces the active Real-Time Factor (RTF).
3. Cross-Chunk Sliding Audio Context: Problem of Boundary Acoustic Discontinuities -> Mechanism of Multi-Second Prompt Caching -> Effect of Smooth Waveform Synthesis
Synthesizing continuous speech chunk-by-chunk using flow-matching vocoders inevitably introduces transient clicks, energy discontinuities, and abrupt timbre shifts across chunk joins. To ensure acoustic stability across long-form streams, the system buffers and passes the most recent 4 seconds of source audio and the most recent 2 seconds of previously generated target audio as conditioning prompts into the Flow Matching acoustic model. This sliding history guides the vector field integration, smoothing spectral transitions across chunk seams and maintaining speaker timbre coherence.
Loss & Training¶
The system is optimized through a three-stage progressive curriculum: 1. Stage 1 (Warmup): The Thinker (initialized from Qwen2.5-Omni) is frozen. The Talker is initialized with DualCodec embeddings and output heads, then trained for 2 epochs on multi-speaker TTS data (~89.6k hours) to establish robust text-to-code synthesis. 2. Stage 2 (Joint Pretraining): LoRA adapters (rank 32) are added to the Thinker and co-trained with the Talker across a balanced multi-task mixture (ASR:S2TT:MT:TTS:S2ST = \(0.2:1:0.5:1:1.5\)) totaling 11.7B tokens, where paired-S2ST audio constitutes only 2,104.8 hours. 3. Stage 3 (Streaming Finetuning): Adapters are merged, fresh LoRAs (rank 8) are attached with frozen embedding/head layers, and the model is finetuned for 1 epoch on quality-filtered streaming trajectories. To enable a single model checkpoint to support multiple latency operating points, a latency multiplier \(m \in \{1, \dots, 12\}\) is sampled during training using a linear decaying distribution \(\tilde{w}_m = \max(a - b(m-1), 0)\) to emphasize low-latency operating regimes.
Key Experimental Results¶
Main Results¶
Evaluations are conducted on the RealSI benchmark (both sentence-level and long-form continuous settings) and the CVSS-T offline benchmark, tracking translation fidelity (BLEU, ASR-BLEU), latency (LAAL, StreamLAAL), and acoustic quality (SIM-O, AutoPCP).
Table 1: Offline S2ST on CVSS-T Benchmark (Direct architectural comparison against leading MLLMs and specialized S2ST baselines under standardized evaluation)
| Model Category | Model | Backbone # | ASR-BLEU (En→Zh) | ASR-BLEU (Zh→En) | Text-BLEU (En→Zh) | Text-BLEU (Zh→En) | A.PCP (En→Zh) | SIM-O (En→Zh) |
|---|---|---|---|---|---|---|---|---|
| MLLM | GPT-4o | — | 25.42 | 31.64 | — | — | 2.66 | — |
| MLLM | Qwen-Omni2.5 | 7B | 8.04 | 22.66 | 34.85 | 24.39 | 1.90 | — |
| S2ST | Seamless-expressive | 1.7B | 23.70 | 15.89 | 25.45 | 16.67 | 2.73 | 0.42 |
| S2ST | UniSS (Q) | 1.5B | 32.04 | 24.72 | 32.95 | 25.51 | 2.71 | 0.40 |
| Ours | Dec-only Baseline | 3B | 27.12 | 23.41 | 31.59 | 24.90 | 2.96 | 0.45 |
| Ours | Thinker–Talker | 3B | 31.12 | 25.18 | 33.73 | 26.41 | 2.70 | 0.35 |
Table 2: RealSI Streaming S2ST ASR-BLEU across Operating Latency Tiers (Operating points \(m2 \sim m6\) denote source chunk aggregation scales compared against leading commercial system LiveInterpret 2.0)
| Evaluation Setting | Direction | Ours m2 | Ours m3 | Ours m4 | Ours m5 | Ours m6 | LiveInterpret 2.0 |
|---|---|---|---|---|---|---|---|
| Sentence-level (Sent) | En→Zh | 24.21 | 25.01 | 25.54 | 25.92 | 26.32 | 29.09 |
| Sentence-level (Sent) | Zh→En | 19.55 | 21.07 | 22.47 | 22.94 | 23.47 | 22.19 |
| Long-form Stream (Doc) | En→Zh | 26.58 | 28.42 | 29.69 | 30.04 | 29.73 | 29.33 |
| Long-form Stream (Doc) | Zh→En | 22.40 | 22.62 | 24.37 | 26.53 | 25.60 | 26.80 |
Ablation Study¶
Table 3: Paired-S2ST Data Budget Ablation on RealSI Sentence-Level S2ST (ASR-BLEU) (Greedy decoding across latency multipliers to evaluate auxiliary multitask dependence)
| Direction | Model Architecture | Paired S2ST Budget | m1 (Lowest Latency) | m2 | m3 | m4 |
|---|---|---|---|---|---|---|
| En→Zh | Thinker–Talker | Full (2,105h) | 17.30 | 21.49 | 23.02 | 24.04 |
| En→Zh | Thinker–Talker | 10% (~200h) | 15.50 | 20.00 | 21.49 | 23.47 |
| En→Zh | Thinker–Talker | 10% w/o aux tasks | 9.47 | 18.46 | 21.12 | 21.90 |
| En→Zh | Dec-only | Full (2,105h) | 12.70 | 14.98 | 15.61 | 17.44 |
| En→Zh | Dec-only | 10% (~200h) | 13.02 | 14.07 | 15.38 | 16.52 |
| En→Zh | Dec-only | 10% w/o aux tasks | 12.96 | 13.97 | 14.22 | 15.93 |
| Zh→En | Thinker–Talker | Full (2,105h) | 12.49 | 16.88 | 17.55 | 18.78 |
| Zh→En | Thinker–Talker | 10% (~200h) | 9.76 | 15.57 | 17.53 | 18.59 |
| Zh→En | Thinker–Talker | 10% w/o aux tasks | 8.45 | 14.62 | 16.91 | 18.24 |
| Zh→En | Dec-only | Full (2,105h) | 10.88 | 12.94 | 14.01 | 15.45 |
Table 4: Trajectory Filtering (NIR) and Multiplier Sampling on RealSI Long-Form S2TT (BLEU)
| Direction | Trajectory Strategy | m1 (BLEU) | m2 (BLEU) | m3 (BLEU) | m4 (BLEU) |
|---|---|---|---|---|---|
| En→Zh | Random Selection (RANDOM) | 4.59 | 24.86 | 26.22 | 26.54 |
| En→Zh | Monotonicity Filter (NIR) | 21.14 | 26.44 | 28.00 | 28.53 |
| En→Zh | NIR + Linear Schedule (NIR+LINEAR) | 22.26 | 26.94 | 28.19 | 28.33 |
| Zh→En | Random Selection (RANDOM) | 3.56 | 14.55 | 17.32 | 17.08 |
| Zh→En | Monotonicity Filter (NIR) | 11.98 | 18.42 | 19.89 | 21.73 |
| Zh→En | NIR + Linear Schedule (NIR+LINEAR) | 12.83 | 18.83 | 20.40 | 21.63 |
Key Findings¶
- High Data Efficiency of Two-Stream Factorization: When paired-S2ST supervision is reduced by 90% (to ~200 hours), the Thinker–Talker model retains strong performance, suffering only 0.5–1.8 BLEU loss when supported by auxiliary ASR/S2TT/MT tasks, still outperforming the fully supervised unified decoder by a wide margin.
- Unified Decoder Modality Conflict Ceiling: Dec-only models exhibit a hard structural ceiling: despite consuming twice the GPU training time (via dual-branch warmup), they trail the Talker by 6–8 ASR-BLEU points. Removing auxiliary tasks does not impact Dec-only performance, indicating that internal text-code interference prevents it from leveraging multitask representations.
- Monotonicity (NIR) Filtering is Crucial for Low Latency: Without NIR-based difficulty bucketing, low-latency (\(m=1\)) streaming collapses entirely (En→Zh BLEU drops from 22.26 to 4.59) due to erratic, non-monotonic word alignments triggering premature speech commits before source semantic clues arrive.
Highlights & Insights¶
- Explicit Trajectory Unifies Text and Code Commitments: Rather than engineering separate, unstable speech-side emission triggers, binding discrete speech codes to monotonized word boundaries lets the model inherit the native causal streaming capabilities of the language model.
- Asymmetric Parameter Allocation Optimizes Quality and Latency: Allocating 3B parameters to high-level linguistic reasoning (Thinker) and 0.4B to acoustic code generation (Talker) achieves an optimal Pareto frontier: linguistic capability is insulated from acoustic noise, while active compute RTF is slashed compared to unified 3B decoders.
- Generalizable Design Pattern: For complex multimodal generation tasks requiring both deep semantic reasoning and high-rate physical synthesis (e.g., real-time duplex conversational speech or vision-driven digital avatars), two-stream factorization consistently outperforms monolithic token-interleaved architectures.
Limitations & Future Work¶
- Bilingual Focus: The current empirical validation is restricted to Chinese-English; evaluating trajectory stability across morphologically complex or drastically non-monotonic language pairs (e.g., German, Japanese) remains future work.
- Synthetic Training Data vs. Real Interpretation: Paired streaming audio is constructed via voice cloning and alignment rather than collected from live human simultaneous interpreters, omitting genuine disfluencies, filler strategies, and adaptive speech rate adjustments.
- Non-Native Streaming Acoustic Backend: While the SLM predicts codes in a streaming fashion, the downstream Flow Matching vocoder uses chunk-wise stitching with history prompts rather than an end-to-end causal flow architecture.
Related Work & Insights¶
- vs LiveInterpret 2.0 / Hibiki: While commercial and top-tier academic streaming S2ST models demand 40k+ hours of paired audio or complex multi-agent reinforcement learning, SimulS2ST-Omni matches their translation quality using only 2k hours of paired data via explicit trajectory supervision.
- vs InfiniSST: InfiniSST pioneered long-form streaming text trajectories for speech translation; this work successfully extends the trajectory paradigm to speech synthesis by solving the challenge of irrevocable acoustic code commitment.
- vs SeamlessStreaming: Seamless relies on dedicated UnitY multi-stage decoders and external policy controllers; SimulS2ST-Omni unifies decision policy and generation within the autoregressive SLM, achieving superior long-form streaming stability.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Pioneering extension of trajectory supervision to joint text-code streaming, paired with a principled two-stream architectural solution to modality interference]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluations across offline, sentence, and long-form streaming settings, covering LAAL_CA, RTF, multiple ASR backends, human listening MOS, and LLM-as-judge]
- Writing Quality: ⭐⭐⭐⭐⭐ [Crystal-clear structural motivation, rigorous ablation protocols, and detailed diagnostic case studies]
- Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly data-efficient blueprint for academic researchers to train state-of-the-art streaming S2ST systems without commercial-scale compute or data]