Skip to content

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://hasaki321.github.io/SimulS2ST-Omni.demo/
Area: Audio & Speech
Keywords: Streaming Speech-to-Speech Translation, Speech Language Models, Trajectory Supervision, Two-Stream Architecture, Low-Resource Data Efficiency

TL;DR

SimulS2ST-Omni establishes an explicit joint text-code trajectory supervision framework paired with a decoupled Thinker–Talker architecture, delivering state-of-the-art sentence-level and long-form streaming S2ST using only ~2,000 hours of paired cross-lingual audio.

Background & Motivation

Real-time, continuous speech-to-speech translation (Streaming S2ST) demands incremental comprehension, cross-lingual semantic planning, and high-fidelity target audio synthesis over unbounded source speech streams under rigorous latency constraints. Traditional cascaded architectures (ASR + MT + TTS) incur prohibitive pipeline wait times and severe cascading error propagation, rendering them suboptimal for natural simultaneous interpretation. While end-to-end Speech Language Models (SLMs) have recently shown impressive mastery in offline, sentence-bounded processing, existing streaming systems heavily rely on massive synthetic supervision—routinely exceeding 40,000 hours of paired source-target audio—or necessitate intricate reinforcement learning and human-annotated alignments. Achieving robust, continuous streaming S2ST under practical, data-constrained regimes has thus stood as a fundamental, long-standing bottleneck.

This data bottleneck is profoundly compounded by two structural obstacles in streaming audio generation. First, long-form simultaneous translation is inherently a latent path optimization problem, wherein the system must dynamically choose when to read, wait, or write based solely on incomplete source context. While trajectory-based methods successfully schedule streaming in text-only spaces, they fail when directly applied to speech generation, which requires the irrevocable, time-sensitive emission of dense acoustic semantic codes along speech boundaries. Second, training a single unified decoder to simultaneously handle source speech understanding, target text planning, and high-rate acoustic code prediction creates severe modality representation interference: maximizing linguistic comprehension capacity actively degrades dense acoustic modeling, and vice versa.

To resolve this conflict under tight data budgets, this paper proposes that linguistic reasoning and dense acoustic generation should be structurally factored rather than forced to compete within a single decoder vocabulary. Core idea: Introduce explicit joint text-code trajectory supervision to schedule target text and acoustic semantic tokens along a unified monotonic commitment path, coupled with a decoupled Thinker–Talker two-stream architecture that isolates linguistic planning from acoustic prediction, enabling data-efficient long-form streaming S2ST with only ~2,000 hours of paired audio.

Method

Overall Architecture

SimulS2ST-Omni consists of an offline trajectory mining pipeline and an online two-stream streaming generation engine. Given an unbounded source speech stream \(X\), the goal is to incrementally emit target speech waveforms \(Y_{\text{wav}}\). The target speech is represented as discrete acoustic semantic codes \(Y_{\text{code}} = Q(Y_{\text{wav}})\) via a frozen DualCodec tokenizer. Word-level forced alignments and cross-lingual SimAlign projections map each target word to its earliest source-speech prefix, and a monotonicity filter schedules target words and their acoustic codes into fixed-duration read/wait/write chunks \(\tau\). Online, the two-stream backbone decouples translation into a 3B Thinker (modeling target text tokens and high-level hidden representations \(H_\theta\)) and a lightweight 0.4B Talker (conditioned on \(H_\theta\) to emit dense acoustic codes). The emitted codes are then reconstructed into continuous audio by a frozen chunk-wise Flow Matching decoder and vocoder conditioned on sliding audio context.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    InAudio["Continuous Source Speech X"] --> Enc["Speech Encoder<br/>Extracts acoustic representations"]
    subgraph DataPipeline["Trajectory Construction"]
        direction TB
        FA["Forced Alignment & SimAlign<br/>Extract word & cross-lingual boundaries"] --> TrajBuild["Monotonization & Chunk Grouping<br/>Build unified text-code trajectory τ"]
    end
    Enc --> Thinker["Thinker Linguistic Stream<br/>Autoregressively plans target text & Hθ"]
    TrajBuild -.->|Teacher trajectory supervision| Thinker
    Thinker --> Talker["Talker Acoustic Stream<br/>Predicts dense acoustic codes Ycode"]
    TrajBuild -.->|Chunk boundary alignment| Talker
    Talker --> CodecFM["Frozen Flow Matching & Vocoder<br/>Sliding audio context smoothing"]
    CodecFM --> OutWave["Low-Latency Target Speech Ywav"]

Key Designs

1. Monotonized Joint Text-Code Trajectory Supervision: Problem of Latent Path Decisions -> Mechanism of Aligned Discrete Trajectories -> Effect of Eliminating Unstable Emission Controllers

Streaming S2ST requires early commitments before source utterances finish, but cross-lingual syntactic inversion frequently misaligns target words to delayed source frames, risking premature and irreversible speech emission. To establish stable, step-by-step supervision without relying on complex policy networks or reinforcement learning, the authors extract cross-lingual word alignments using SimAlign and map each target word \(y_i\) to its corresponding source-word end timestamp \(t_{a(i)}\). A monotonicity recursion resolves local word reordering: $\(\tilde{t}_i = \max(\tilde{t}_{i-1}, t_{a(i)}), \quad \text{with } \tilde{t}_0 = 0\)$ Both target text words and their corresponding segmented semantic speech codes inherit the identical monotonized source boundary \(\tilde{t}_i\). These paired tokens are then grouped into fixed-interval source chunks (e.g., 1-second intervals). Chunks without new target content act as explicit wait steps, while chunks with content act as write steps emitting \((Y_{\text{text}}^c, Y_{\text{code}}^c)\). This converts the continuous latency policy into a discrete, supervised multi-task classification objective.

2. Thinker–Talker Two-Stream Decoupling: Problem of Modality Interference & Decoding Overhead -> Mechanism of Factorized Conditioning -> Effect of Preserving Intelligence and Slashing Latency

Forcing a unified 3B decoder to predict 16,384 discrete codec tokens alongside natural text vocabulary creates conflicting gradient demands between high-level cross-lingual understanding and low-level acoustic token prediction, while also requiring massive compute for dense code generation. The method resolves this by factorizing the chunk joint probability: $\(p(Y_{\text{text}}^c, Y_{\text{code}}^c \mid C_c) = p_\theta(Y_{\text{text}}^c \mid C_c) \cdot p_\phi(Y_{\text{code}}^c \mid C_c, Y_{\text{text}}^c, H_\theta)\)$ where \(C_c\) denotes the chunk context (consumed source speech and previously generated target tokens), and \(H_\theta\) is the intermediate hidden state sequence emitted by the Thinker \(\theta\). The 3B Thinker focuses exclusively on linguistic translation at word-level rates, while the lightweight 0.4B Talker \(\phi\) consumes \(H_\theta\) to rapidly predict dense acoustic codes. This decoupling eliminates vocabulary pollution, protects translation intelligence, and significantly reduces the active Real-Time Factor (RTF).

3. Cross-Chunk Sliding Audio Context: Problem of Boundary Acoustic Discontinuities -> Mechanism of Multi-Second Prompt Caching -> Effect of Smooth Waveform Synthesis

Synthesizing continuous speech chunk-by-chunk using flow-matching vocoders inevitably introduces transient clicks, energy discontinuities, and abrupt timbre shifts across chunk joins. To ensure acoustic stability across long-form streams, the system buffers and passes the most recent 4 seconds of source audio and the most recent 2 seconds of previously generated target audio as conditioning prompts into the Flow Matching acoustic model. This sliding history guides the vector field integration, smoothing spectral transitions across chunk seams and maintaining speaker timbre coherence.

Loss & Training

The system is optimized through a three-stage progressive curriculum: 1. Stage 1 (Warmup): The Thinker (initialized from Qwen2.5-Omni) is frozen. The Talker is initialized with DualCodec embeddings and output heads, then trained for 2 epochs on multi-speaker TTS data (~89.6k hours) to establish robust text-to-code synthesis. 2. Stage 2 (Joint Pretraining): LoRA adapters (rank 32) are added to the Thinker and co-trained with the Talker across a balanced multi-task mixture (ASR:S2TT:MT:TTS:S2ST = \(0.2:1:0.5:1:1.5\)) totaling 11.7B tokens, where paired-S2ST audio constitutes only 2,104.8 hours. 3. Stage 3 (Streaming Finetuning): Adapters are merged, fresh LoRAs (rank 8) are attached with frozen embedding/head layers, and the model is finetuned for 1 epoch on quality-filtered streaming trajectories. To enable a single model checkpoint to support multiple latency operating points, a latency multiplier \(m \in \{1, \dots, 12\}\) is sampled during training using a linear decaying distribution \(\tilde{w}_m = \max(a - b(m-1), 0)\) to emphasize low-latency operating regimes.

Key Experimental Results

Main Results

Evaluations are conducted on the RealSI benchmark (both sentence-level and long-form continuous settings) and the CVSS-T offline benchmark, tracking translation fidelity (BLEU, ASR-BLEU), latency (LAAL, StreamLAAL), and acoustic quality (SIM-O, AutoPCP).

Table 1: Offline S2ST on CVSS-T Benchmark (Direct architectural comparison against leading MLLMs and specialized S2ST baselines under standardized evaluation)

Model Category Model Backbone # ASR-BLEU (En→Zh) ASR-BLEU (Zh→En) Text-BLEU (En→Zh) Text-BLEU (Zh→En) A.PCP (En→Zh) SIM-O (En→Zh)
MLLM GPT-4o — 25.42 31.64 — — 2.66 —
MLLM Qwen-Omni2.5 7B 8.04 22.66 34.85 24.39 1.90 —
S2ST Seamless-expressive 1.7B 23.70 15.89 25.45 16.67 2.73 0.42
S2ST UniSS (Q) 1.5B 32.04 24.72 32.95 25.51 2.71 0.40
Ours Dec-only Baseline 3B 27.12 23.41 31.59 24.90 2.96 0.45
Ours Thinker–Talker 3B 31.12 25.18 33.73 26.41 2.70 0.35

Table 2: RealSI Streaming S2ST ASR-BLEU across Operating Latency Tiers (Operating points \(m2 \sim m6\) denote source chunk aggregation scales compared against leading commercial system LiveInterpret 2.0)

Evaluation Setting Direction Ours m2 Ours m3 Ours m4 Ours m5 Ours m6 LiveInterpret 2.0
Sentence-level (Sent) En→Zh 24.21 25.01 25.54 25.92 26.32 29.09
Sentence-level (Sent) Zh→En 19.55 21.07 22.47 22.94 23.47 22.19
Long-form Stream (Doc) En→Zh 26.58 28.42 29.69 30.04 29.73 29.33
Long-form Stream (Doc) Zh→En 22.40 22.62 24.37 26.53 25.60 26.80

Ablation Study

Table 3: Paired-S2ST Data Budget Ablation on RealSI Sentence-Level S2ST (ASR-BLEU) (Greedy decoding across latency multipliers to evaluate auxiliary multitask dependence)

Direction Model Architecture Paired S2ST Budget m1 (Lowest Latency) m2 m3 m4
En→Zh Thinker–Talker Full (2,105h) 17.30 21.49 23.02 24.04
En→Zh Thinker–Talker 10% (~200h) 15.50 20.00 21.49 23.47
En→Zh Thinker–Talker 10% w/o aux tasks 9.47 18.46 21.12 21.90
En→Zh Dec-only Full (2,105h) 12.70 14.98 15.61 17.44
En→Zh Dec-only 10% (~200h) 13.02 14.07 15.38 16.52
En→Zh Dec-only 10% w/o aux tasks 12.96 13.97 14.22 15.93
Zh→En Thinker–Talker Full (2,105h) 12.49 16.88 17.55 18.78
Zh→En Thinker–Talker 10% (~200h) 9.76 15.57 17.53 18.59
Zh→En Thinker–Talker 10% w/o aux tasks 8.45 14.62 16.91 18.24
Zh→En Dec-only Full (2,105h) 10.88 12.94 14.01 15.45

Table 4: Trajectory Filtering (NIR) and Multiplier Sampling on RealSI Long-Form S2TT (BLEU)

Direction Trajectory Strategy m1 (BLEU) m2 (BLEU) m3 (BLEU) m4 (BLEU)
En→Zh Random Selection (RANDOM) 4.59 24.86 26.22 26.54
En→Zh Monotonicity Filter (NIR) 21.14 26.44 28.00 28.53
En→Zh NIR + Linear Schedule (NIR+LINEAR) 22.26 26.94 28.19 28.33
Zh→En Random Selection (RANDOM) 3.56 14.55 17.32 17.08
Zh→En Monotonicity Filter (NIR) 11.98 18.42 19.89 21.73
Zh→En NIR + Linear Schedule (NIR+LINEAR) 12.83 18.83 20.40 21.63

Key Findings

  • High Data Efficiency of Two-Stream Factorization: When paired-S2ST supervision is reduced by 90% (to ~200 hours), the Thinker–Talker model retains strong performance, suffering only 0.5–1.8 BLEU loss when supported by auxiliary ASR/S2TT/MT tasks, still outperforming the fully supervised unified decoder by a wide margin.
  • Unified Decoder Modality Conflict Ceiling: Dec-only models exhibit a hard structural ceiling: despite consuming twice the GPU training time (via dual-branch warmup), they trail the Talker by 6–8 ASR-BLEU points. Removing auxiliary tasks does not impact Dec-only performance, indicating that internal text-code interference prevents it from leveraging multitask representations.
  • Monotonicity (NIR) Filtering is Crucial for Low Latency: Without NIR-based difficulty bucketing, low-latency (\(m=1\)) streaming collapses entirely (En→Zh BLEU drops from 22.26 to 4.59) due to erratic, non-monotonic word alignments triggering premature speech commits before source semantic clues arrive.

Highlights & Insights

  • Explicit Trajectory Unifies Text and Code Commitments: Rather than engineering separate, unstable speech-side emission triggers, binding discrete speech codes to monotonized word boundaries lets the model inherit the native causal streaming capabilities of the language model.
  • Asymmetric Parameter Allocation Optimizes Quality and Latency: Allocating 3B parameters to high-level linguistic reasoning (Thinker) and 0.4B to acoustic code generation (Talker) achieves an optimal Pareto frontier: linguistic capability is insulated from acoustic noise, while active compute RTF is slashed compared to unified 3B decoders.
  • Generalizable Design Pattern: For complex multimodal generation tasks requiring both deep semantic reasoning and high-rate physical synthesis (e.g., real-time duplex conversational speech or vision-driven digital avatars), two-stream factorization consistently outperforms monolithic token-interleaved architectures.

Limitations & Future Work

  • Bilingual Focus: The current empirical validation is restricted to Chinese-English; evaluating trajectory stability across morphologically complex or drastically non-monotonic language pairs (e.g., German, Japanese) remains future work.
  • Synthetic Training Data vs. Real Interpretation: Paired streaming audio is constructed via voice cloning and alignment rather than collected from live human simultaneous interpreters, omitting genuine disfluencies, filler strategies, and adaptive speech rate adjustments.
  • Non-Native Streaming Acoustic Backend: While the SLM predicts codes in a streaming fashion, the downstream Flow Matching vocoder uses chunk-wise stitching with history prompts rather than an end-to-end causal flow architecture.
  • vs LiveInterpret 2.0 / Hibiki: While commercial and top-tier academic streaming S2ST models demand 40k+ hours of paired audio or complex multi-agent reinforcement learning, SimulS2ST-Omni matches their translation quality using only 2k hours of paired data via explicit trajectory supervision.
  • vs InfiniSST: InfiniSST pioneered long-form streaming text trajectories for speech translation; this work successfully extends the trajectory paradigm to speech synthesis by solving the challenge of irrevocable acoustic code commitment.
  • vs SeamlessStreaming: Seamless relies on dedicated UnitY multi-stage decoders and external policy controllers; SimulS2ST-Omni unifies decision policy and generation within the autoregressive SLM, achieving superior long-form streaming stability.

Rating

  • Novelty: ⭐⭐⭐⭐ [Pioneering extension of trajectory supervision to joint text-code streaming, paired with a principled two-stream architectural solution to modality interference]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluations across offline, sentence, and long-form streaming settings, covering LAAL_CA, RTF, multiple ASR backends, human listening MOS, and LLM-as-judge]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Crystal-clear structural motivation, rigorous ablation protocols, and detailed diagnostic case studies]
  • Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly data-efficient blueprint for academic researchers to train state-of-the-art streaming S2ST systems without commercial-scale compute or data]