SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://hasaki321.github.io/SimulS2ST-Omni.demo/
Area: Audio and Speech Processing
Keywords: Streaming Speech-to-Speech Translation, Simultaneous Interpretation, Explicit Trajectory Supervision, Two-Stream Architecture, Data Efficiency
TL;DR¶
SimulS2ST-Omni introduces a joint text-code trajectory supervision mechanism paired with a decoupled Thinker–Talker architecture, achieving state-of-the-art long-form streaming speech-to-speech translation using only ~2k hours of synthesized paired audio.
Background & Motivation¶
Real-time, cross-lingual communication demands simultaneous speech-to-speech translation (S2ST) systems capable of continuously translating incoming speech streams under tight latency budgets. Conventional cascaded pipelines—consisting of pipelined ASR, MT, and TTS modules—inevitably suffer from compounded latencies and error propagation across module boundaries. While end-to-end Speech Language Models (SLMs) such as Hibiki and LiveInterpret 2.0 have recently demonstrated impressive streaming capabilities, they depend overwhelmingly on massive corpora of synthetic parallel speech exceeding 40,000 hours, or require delicate reinforcement learning frameworks with expert-annotated alignments. Building an effective, unbounded streaming S2ST system under limited paired cross-lingual audio remains an open and pressing challenge.
This data bottleneck is rooted in two fundamental architectural dilemmas. First, continuous streaming translation represents an unobserved latent path problem where the model must autonomously make read, wait, and write commitments without explicit step-by-step guidance. Although trajectory-based supervision has shown great success in streaming text generation by formalizing read-write schedules, direct extension to speech generation breaks down because acoustic semantic codes require irrevocable, highly synchronized time-domain commitment. Second, forcing a single unified decoder to perform both high-level semantic planning and dense acoustic code prediction introduces severe gradient conflicts across modalities. In a unified decoder, predicting high-rate acoustic code tokens rapidly interferes with the language model's cognitive capacity, degrading translation accuracy.
This paper tackles these challenges by reformulating the streaming translation policy into an explicit joint text-code commitment path supported by structural decoupling. By combining word-level cross-lingual forced alignment with monotonicity-aware trajectory filtering, the system eliminates the need for separate external emission controllers. Core idea: schedule target text tokens and discrete acoustic codes onto a synchronized chunk-level trajectory, and decouple linguistic planning from dense acoustic generation via a Thinker–Talker factorization, enabling competitive long-form streaming S2ST with only ~2k hours of paired audio.
Method¶
Overall Architecture¶
SimulS2ST-Omni models streaming speech translation as a chunk-wise commitment process driven by a teacher trajectory. The system reads source speech incrementally and jointly emits target text along with discrete acoustic semantic codes, which are subsequently synthesized into continuous waveforms via a frozen Flow Matching model and neural vocoder. Training follows a three-stage progressive curriculum: Talker warmup on single-modality TTS, joint multitask pretraining across auxiliary tasks, and streaming trajectory finetuning with dynamic latency multipliers. During inference, dynamic key-value caching and cross-chunk acoustic prompt caching ensure seamless long-form generation across arbitrary latency tiers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: Continuous Source Audio Stream X"] --> Step1["Text and Audio Alignment<br/>Forced alignment + SimAlign cross-lingual mapping"]
Step1 --> Step2["Discrete Trajectory Construction<br/>Boundary monotonization + code slicing & 1s chunk grouping"]
Step2 --> Filter["Monotonicity-Aware Trajectory Selection<br/>NIR difficulty stratification & ASR content filtering"]
Filter --> TT["Thinker–Talker Two-Stream Factorization<br/>Thinker plans text + Talker predicts dense acoustic codes"]
TT --> Stage3["Progressive Multitask Training<br/>TTS warmup → Multitask pretraining → Trajectory finetuning"]
Stage3 --> Out["Output: Target text stream + Flow Matching synthesized speech"]
Key Designs¶
1. Discrete Trajectory Construction: Turning Latent Policy into Explicit Joint Commitments
Simultaneous S2ST traditionally struggles with unstable emission timing because the model receives no direct supervisory signal on when to hold and when to emit irrevocable speech units. To eliminate complex external emission controllers or reinforcement learning loops, this work formalizes streaming translation as a causal, chunk-wise joint trajectory \(\tau = \{(Y_{text}^c, Y_{code}^c, g_c)\}_{c=1}^C\). Given source-target pairs \((X, Y_{text})\), phoneme-level forced alignment extracts word boundaries, and SimAlign computes cross-lingual word mappings \(\hat{A}\). If target word \(y_i\) aligns to source word \(x_{a(i)}\) ending at frame \(t_{a(i)}\), this timestamp dictates the earliest source boundary required to commit \(y_i\). To prevent non-monotonic word reordering from breaking streaming causality, boundaries are monotonized via:
$\(\tilde{t}_i = \max(\tilde{t}_{i-1}, t_{a(i)}), \quad \tilde{t}_0 = 0\)$
Target audio \(Y_{wav}\) is quantized into discrete semantic tokens \(Y_{code}\) via DualCodec and sliced along the target word boundaries. Adjacent target words and codes sharing the same 1-second source chunk window are grouped into discrete decision steps. Chunks without target content serve as implicit <wait> steps, while chunks containing content serve as joint <write> steps, training the model to emit synchronized text and speech units in an autoregressive fashion.
2. Monotonicity-Aware Trajectory Selection: Eliminating Inversion Noise and Hallucinations Because paired S2ST data is synthesized via machine translation and voice cloning, imperfect word alignments and severe cross-lingual word reorderings can corrupt low-latency decision boundaries. The authors introduce a difficulty-stratified sampling scheme using the Normalized Inversion Rate (NIR). For an aligned source index sequence \(\hat{A} = [\hat{a}_1, \hat{a}_2, \dots]\) with inversion count \(I_{\hat{A}}\), NIR is defined as: $\(\text{NIR} = \frac{2 I_{\hat{A}}}{|\hat{A}|(|\hat{A}| - 1)} \times 100\%\)$ While random sampling clusters around NIR \(\approx 0\) (producing trivial monotonic examples that harm real-world generalization), the proposed policy stratifies candidates across difficulty weights \(\{0.1, 0.3, 0.4, 0.2\}\) and length weights \(\{0.1, 0.5, 0.4\}\), deliberately reserving a 10% quota for high-NIR complex reorderings. Furthermore, dual ASR engines (Qwen3-ASR-1.7B and Whisper-large-v3) cross-validate the synthesized audio against target transcripts under strict error thresholds (5% for Chinese, 7% for English), pruning over 20% of corrupted pairs to protect the acoustic code sequence from hallucinated tokens.
3. Thinker–Talker Two-Stream Factorization: Resolving Representational Modality Conflicts In unified decoder baselines (Dec-only), 16,384 discrete acoustic codes are appended directly to the language model vocabulary, forcing a single autoregressive head to emit text followed immediately by dense codes. Dense acoustic predictions produce high-frequency gradient updates that disrupt long-range semantic reasoning. To resolve this representational clash, the authors decouple generation into two specialized streams: $\(p(Y_{text}^c, Y_{code}^c \mid C_c) = p_\theta(Y_{text}^c \mid C_c) \, p_\phi(Y_{code}^c \mid C_c, Y_{text}^c, H_\theta)\)$ The Thinker \(\theta\) (a 3B-parameter LLM) focuses exclusively on source speech comprehension and target text planning. The Talker \(\phi\) (a lightweight 0.4B-parameter decoder) conditions on the Thinker’s intermediate hidden states \(H_\theta\) and target text token embeddings to predict dense acoustic semantic codes. This structural separation prevents acoustic interference with linguistic cognition while reducing inference computational overhead (RTF) by more than half.
4. Progressive Multitask Training: Anchoring Low-Resource S2ST Generalization To maximize data efficiency under a restricted budget of only 2,104.8 hours of paired S2ST audio, the model is trained in three stages. In Stage 1 (Talker Warmup), the Thinker is frozen while the Talker is trained on 89.6k hours of single-speaker and multi-speaker TTS data for 2 epochs. In Stage 2 (Multitask Joint Pretraining), the Thinker is equipped with LoRA adapters (rank 32) and jointly trained with the Talker on an ASR, S2TT, MT, TTS, and S2ST mixture with a sampling ratio of \(0.2 : 1 : 0.5 : 1 : 1.5\). In Stage 3 (Streaming Trajectory Finetuning), previous adapters are merged, fresh LoRAs (rank 8) are attached to both modules, and the model is finetuned on augmented multi-task streaming trajectories. A latency multiplier \(m \in \{1, \dots, 12\}\) is dynamically sampled during training, enabling a single checkpoint to seamlessly serve all latency operating points at test time.
Loss & Training¶
During the final streaming finetuning stage, the Thinker and Talker are jointly optimized on the chunk-factorized log-likelihood objective given the causal prefix \(C_c = (X_{1:g_c}, Y_{text}^{<c}, Y_{code}^{<c})\): $\(\mathcal{L} = - \sum_{c=1}^C \left[ \log p_\theta(Y_{text}^c \mid C_c) + \log p_\phi(Y_{code}^c \mid C_c, Y_{text}^c, H_\theta) \right]\)$ All models are trained across 8 NVIDIA A800 GPUs using the AdamW optimizer with cosine learning rate schedules (\(4 \times 10^{-4} \to 4 \times 10^{-5}\) in Stage 1; \(1 \times 10^{-4} \to 2 \times 10^{-5}\) in Stage 2; \(3 \times 10^{-5} \to 2 \times 10^{-6}\) in Stage 3). To prevent spectral discontinuities and acoustic clicks at chunk boundaries during streaming speech synthesis, the downstream Flow Matching model is supplied with 4 seconds of source audio and 2 seconds of recently generated speech as acoustic prompt context.
Key Experimental Results¶
Main Results¶
The framework is comprehensively evaluated on the offline CVSS-T benchmark and real-world simultaneous interpretation benchmarks RealSI and ACL60/60-dev. In offline S2ST, the 3B Thinker–Talker significantly outperforms the matched 3B Dec-only baseline and matches or exceeds prior SOTA systems such as UniSS(Q). In streaming S2ST on RealSI, it matches and even surpasses the commercial closed-source benchmark LiveInterpret 2.0 at higher latency operating points.
Table 1: Offline S2ST Evaluation on CVSS-T
| Category | Model | Params | ASR-BLEU ↑ (En→Zh) | ASR-BLEU ↑ (Zh→En) | Text-BLEU ↑ (En→Zh) | Text-BLEU ↑ (Zh→En) | A.PCP ↑ (En→Zh / Zh→En) | SIM-O ↑ (En→Zh / Zh→En) |
|---|---|---|---|---|---|---|---|---|
| MLLM | GPT-4o | — | 25.42 | 31.64 | — | — | 2.66 / 2.58 | — / — |
| MLLM | Qwen-Omni2.5 | 7B | 8.04 | 22.66 | 34.85 | 24.39 | 1.90 / 1.92 | — / — |
| MLLM | Step-Audio 2 mini | 7B | 32.81 | 25.35 | — | — | — / — | — / — |
| S2ST | Seamless-m4t-v2-large | 2.3B | 20.86 | 22.25 | 20.75 | 22.60 | 2.43 / 2.67 | 0.26 / 0.11 |
| S2ST | Seamless-expressive | 1.7B | 23.70 | 15.89 | 25.45 | 16.67 | 2.73 / 2.76 | 0.42 / 0.34 |
| S2ST | UniSS (Q) | 1.5B | 32.04 | 24.72 | 32.95 | 25.51 | 2.71 / 2.75 | 0.40 / 0.42 |
| Baseline | Ours (Dec-only) | 3B | 27.12 | 23.41 | 31.59 | 24.90 | 2.96 / 2.75 | 0.45 / 0.59 |
| Proposed | Ours (Thinker–Talker) | 3B | 31.12 | 25.18 | 33.73 | 26.41 | 2.70 / 2.64 | 0.35 / 0.45 |
Table 2: RealSI Streaming S2ST Translation Quality (ASR-BLEU) vs. LiveInterpret 2.0
| Setting | Direction | Ours m2 | Ours m3 | Ours m4 | Ours m5 | Ours m6 | LiveInterpret 2.0 (Closed-source) |
|---|---|---|---|---|---|---|---|
| Sentence-level (Sent) | En→Zh | 24.21 | 25.01 | 25.54 | 25.92 | 26.32 | 29.09 |
| Sentence-level (Sent) | Zh→En | 19.55 | 21.07 | 22.47 | 22.94 | 23.47 | 22.19 |
| Long-form (Doc) | En→Zh | 26.58 | 28.42 | 29.69 | 30.04 | 29.73 | 29.33 |
| Long-form (Doc) | Zh→En | 22.40 | 22.62 | 24.37 | 26.53 | 25.60 | 26.80 |
(Note: Ours uses num_beams=4 trajectory decoding. On long-form En→Zh, Ours overtakes LiveInterpret 2.0 from tier m4 onwards.)
Ablation Study¶
Ablations systematically probe trajectory filtering methods, auxiliary multitask supervision, and the lower limits of paired S2ST audio.
Table 3: Trajectory Quality and Multiplier Sampling Ablation on RealSI Long-Form S2TT (BLEU)
| Direction | Variant | m1 | m2 | m3 | m4 |
|---|---|---|---|---|---|
| En→Zh | RANDOM (unfiltered random sampling) | 4.59 | 24.86 | 26.22 | 26.54 |
| En→Zh | NIR (monotonicity-filtered, uniform sampling) | 21.14 | 26.44 | 28.00 | 28.53 |
| En→Zh | NIR + LINEAR (linear multiplier schedule) | 22.26 | 26.94 | 28.19 | 28.33 |
| En→Zh | NIR + GAUSSIAN (Gaussian multiplier schedule) | 22.08 | 26.91 | 27.60 | 27.90 |
| Zh→En | RANDOM (unfiltered random sampling) | 3.56 | 14.55 | 17.32 | 17.08 |
| Zh→En | NIR (monotonicity-filtered, uniform sampling) | 11.98 | 18.42 | 19.89 | 21.73 |
| Zh→En | NIR + LINEAR (linear multiplier schedule) | 12.83 | 18.83 | 20.40 | 21.63 |
| Zh→En | NIR + GAUSSIAN (Gaussian multiplier schedule) | 11.95 | 17.78 | 19.87 | 21.46 |
Table 4: Paired-S2ST Budget Ablation on RealSI Sentence-Level S2ST (ASR-BLEU, Greedy Decoding)
| Direction | Model | Paired-S2ST Budget | m1 | m2 | m3 | m4 |
|---|---|---|---|---|---|---|
| En→Zh | Talker | Full (100% ~2k hrs) | 17.30 | 21.49 | 23.02 | 24.04 |
| En→Zh | Talker | 10% (~200 hrs) | 15.50 | 20.00 | 21.49 | 23.47 |
| En→Zh | Talker | 10% w/o aux multitask | 9.47 | 18.46 | 21.12 | 21.90 |
| En→Zh | Dec-only | Full (100%) | 12.70 | 14.98 | 15.61 | 17.44 |
| En→Zh | Dec-only | 10% | 13.02 | 14.07 | 15.38 | 16.52 |
| En→Zh | Dec-only | 10% w/o aux multitask | 12.96 | 13.97 | 14.22 | 15.93 |
| Zh→En | Talker | Full (100%) | 12.49 | 16.88 | 17.55 | 18.78 |
| Zh→En | Talker | 10% | 9.76 | 15.57 | 17.53 | 18.59 |
| Zh→En | Talker | 10% w/o aux multitask | 8.45 | 14.62 | 16.91 | 18.24 |
| Zh→En | Dec-only | Full (100%) | 10.88 | 12.94 | 14.01 | 15.45 |
| Zh→En | Dec-only | 10% | 10.29 | 12.82 | 14.16 | 15.12 |
| Zh→En | Dec-only | 10% w/o aux multitask | 10.24 | 11.68 | 13.83 | 15.12 |
Key Findings¶
- Monotonicity filtering is essential for low-latency stability: Omitting NIR filtering (RANDOM in Table 3) causes catastrophic collapse at the lowest latency tier \(m=1\) (En→Zh drops from 21.14 to 4.59 BLEU; Zh→En plummets from 11.98 to 3.56). This collapse occurs because severe reordering noise forces the model to emit predictions prematurely without adequate source context.
- Auxiliary multitask data anchors low-resource regimes: When paired S2ST audio is slashed by 90% to just ~200 hours, retaining auxiliary ASR/S2TT/MT supervision keeps Talker performance nearly on par with the full-data model (Table 4). Removing auxiliary supervision triggers an immediate drop of nearly 8.0 ASR-BLEU at \(m=1\) on En→Zh.
- Architectural superiority over unified decoders: The unified Dec-only baseline remains bounded below 17.5 ASR-BLEU regardless of data budget or multitask support. Its internal representational interference between linguistic reasoning and acoustic modeling prevents it from benefiting from auxiliary supervision.
Highlights & Insights¶
- Synchronous commitment of text and discrete acoustic units: By leveraging forced alignment and recursive monotonization, continuous acoustic speech emission is mapped directly onto a causal discrete trajectory, eliminating the need for complex emission controllers or reinforcement learning rewards.
- Asymmetric two-stream compute factorization: Delegating heavy linguistic planning to a 3B Thinker and dense acoustic unit synthesis to a lightweight 0.4B Talker resolves modality interference and halves runtime computational RTF.
- Single-checkpoint latency scalability: Dynamic sampling of the latency multiplier \(m \in \{1, \dots, 12\}\) during finetuning empowers a single trained model to operate across arbitrary latency tiers at inference time.
Limitations & Future Work¶
- Language coverage: Experiments are limited to Chinese-English bidirectional translation due to the scarcity of high-quality aligned speech corpora in other language pairs.
- Synthesized vs. human interpreting dynamics: Training audio is synthesized via cross-lingual voice cloning rather than collected from live professional interpreters, meaning specific interpreting strategies (e.g., proactive summarization, pause filler management) are not explicitly learned.
- Non-native streaming acoustic backend: The downstream Flow Matching model operates over concatenated chunks rather than in a natively causal streaming mode, which can cause subtle boundary clicks during ultra-short chunk synthesis.
Related Work & Insights¶
- vs LiveInterpret 2.0: LiveInterpret 2.0 achieves top-tier performance via 40k+ hours of data and reinforcement learning. SimulS2ST-Omni matches its long-form ASR-BLEU on RealSI using only ~2k hours of synthesized data via explicit trajectory supervision and auxiliary multitask pretraining.
- vs SeamlessStreaming: SeamlessStreaming relies on specialized monotonic attention mechanisms and unified representations that are sensitive to hyperparameter tuning. SimulS2ST-Omni’s decoupled Thinker–Talker architecture provides superior discourse stability and lower computation-aware latency.
- vs InfiniSST: While InfiniSST demonstrated explicit trajectory supervision for streaming text translation, SimulS2ST-Omni successfully extends this framework to speech output by jointly scheduling discrete acoustic codes alongside target text.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first explicit joint text-code commitment trajectory for streaming S2ST, bypassing the need for complex RL or external emission controllers.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across CVSS-T, RealSI, and ACL60/60-dev, complete with computation-aware latency, RTF, multi-seed variance, dual-ASR validation, and human/LLM evaluation.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured and clear, featuring transparent failure mode analyses and rigorous architectural controls.
- Value: ⭐⭐⭐⭐⭐ Drastically reduces the data barrier for building production-grade simultaneous speech-to-speech translation systems.