Skip to content

RASST: Retrieval-Augmented Simultaneous Speech Translation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Simultaneous Speech Translation, Retrieval-Augmented Generation, Terminology Translation, Multi-Scale Speech Retrieval, Speech LLM

TL;DR

This paper proposes RASST, the first retrieval-augmented framework tailored for simultaneous speech translation, combining a lightweight multi-scale cross-modal speech-text retriever with noise-robust Speech LLM training to accurately retrieve domain-specific terminology under partial streaming speech and decide when to apply retrieved translations, boosting terminology accuracy by 12.3%โ€“31.22% and BLEU by up to 3 points over InfiniSST with negligible computational overhead.

Background & Motivation

Simultaneous Speech Translation (SST) aims to incrementally generate target-language text as continuous, unsegmented streaming audio is received. It serves as an indispensable bridge for real-time communication across international conferences, academic symposiums, and cross-border diplomacy. Recently, capitalizing on the rapid emergence of Speech Large Language Models (Speech LLMs), streaming architectures such as InfiniSST have achieved state-of-the-art general translation quality by interleaving audio feature chunks with causal autoregressive text generation over unbounded streams. Nevertheless, even multi-billion-parameter speech foundation models consistently stumble when translating rare proper nouns, specialized technical jargon, named entities, and domain-specific acronyms, frequently degenerating into inaccurate phonetic transliterations or outright hallucinations (e.g., rendering "Shinzo Abe" as the phonetically transliterated non-standard name "ๆ–ฐไธ‰ยท้˜ฟ่ด" instead of the official "ๅฎ‰ๅ€ๆ™‹ไธ‰").

In professional human simultaneous interpretation, interpreters heavily rely on external glossaries and reference term sheets to handle specialized vocabulary in real time. Although Retrieval-Augmented Generation (RAG) and contextual biasing techniques have witnessed remarkable success in offline Neural Machine Translation (NMT) and Automatic Speech Recognition (ASR), directly extending retrieval augmentation to streaming SST encounters fundamental structural hurdles. First, SST operates under incremental streaming conditions where the input arrives in short, partial audio chunks (often shorter than 1 second), and specialized terms exhibit arbitrary temporal spans and boundaries; naive global pooling over partial speech dilutes salient phonetic cues and causes severe retrieval failure. Second, unlike ASR transcripts that remain monotonically time-aligned with audio frames, SST translations naturally lag behind speech by several seconds due to word order reordering and streaming anticipation delays. The system must therefore decide not merely whether a candidate term is phonetically present in the current audio chunk, but when to execute and emit its target-language translation during subsequent autoregressive decoding steps.

To bridge the dual chasms of inaccurate partial cross-modal retrieval and asynchronous translation emission timing, the authors decouple acoustic retrieval from macro-linguistic generation through a coordinated, noise-robust design. Core idea: train a lightweight dual-encoder speech-text retriever that performs multi-scale windowed pooling over partial speech chunks to supply accurate terminology hints, while synthesizing Speech LLM training data with realistic retrieval distractors and explicit term-tag supervision so the model learns to filter retrieval noise and autonomously determine the precise emission timing of domain terms.

Method

Overall Architecture

RASST adopts an interleaved streaming SST formulation consisting of audio chunking, multi-scale cross-modal retrieval, prompt composition, and incremental Speech LLM decoding. At each streaming step \(i\), the system ingests a fixed-duration speech chunk \(c_i\) of length \(l\) seconds. A causal speech encoder extracts frame-level acoustic representations. Concurrently, a cross-modal retriever operates over the current chunk combined with an acoustic look-back buffer \(t_{\text{lookback}}\) to query an external bilingual glossary \(\mathcal{G}\), returning a top-\(K\) candidate term subset \(G_i\). The retrieved source-to-target pairs are appended to the speech representations as prompt context. Conditioned on prior speech, preceding emitted translations, and current acoustic-lexical hints, the Speech LLM incrementally generates the next translation segment \(\hat{y}_i\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Streaming Speech Chunk Input c_i<br/>(concatenated with t_lookback buffer)"] --> B["Causal Speech Encoder & BGE-M3<br/>Qwen3-Omni extracts frame sequence f_i"]
    B --> C["Multi-Scale Windowed Retrieval<br/>Multi-scale average pooling + FAISS inner product"]
    C --> D["Two-Stage Filtering & Deduplication<br/>Top-K candidate pool + threshold ฯ„ cutoff"]
    D --> E["Noise-Robust Data Synthesis<br/>Retrieved distractor injection + term tag supervision"]
    E --> F["Speech LLM Incremental Autoregressive Decoding<br/>Verify relevance and emit terms at correct timing"]
    F --> G["Streaming Target Translation Segment ลท_i"]

Key Designs

1. Multi-Scale Cross-Modal Retrieval: Mitigating temporal span divergence and acoustic boundary blurring under partial speech

In streaming inference, spoken terminology exhibits immense duration variance (mean duration 0.588s, with P99 reaching 1.497s). Simply collapsing the incoming chunk into a single average-pooled embedding destroys local phonetic resolution and obscures brief terms. RASST implements a dual-encoder architecture: textual glossary terms are mapped to vectors \(g_j \in \mathbb{R}^d\) via a frozen pre-trained BGE-M3 encoder, while acoustic frames are extracted by Qwen3-Omni's Audio Transformer. To capture terms that straddle chunk boundaries, the acoustic input \(c'_i\) prepends a look-back window of \(t_{\text{lookback}}=1.92\) seconds to the current chunk \(c_i\). For the resulting \(m\) acoustic frames \(f_i = (f_i^1, \dots, f_i^m)\), the retriever enumerates all possible sub-windows \(1 \le u < v \le m\) and applies localized average pooling:

\[f_i^{u,v} = \text{AvgPool}(f_i^u, \dots, f_i^v) \in \mathbb{R}^d\]

The similarity between the acoustic context \(c'_i\) and candidate term \(e_j\) is defined as the maximum cosine similarity over all valid window candidates: \(\text{Sim}(c'_i, e_j) = \max_{1 \le u < v \le m} \text{CosSim}(f_i^{u,v}, g_j)\). In practice, the system queries a FAISS inner-product index with each window vector whose right boundary terminates inside the active chunk, aggregates the candidate pool, retains the highest score per term upon deduplication, and applies a calibrated threshold \(\tau=0.78\) to prune false-positive noise before passing the hints \(G_i\) forward.

2. Noise-Robust Data Synthesis with Term-Tag Supervision: Resolving train-inference mismatch and asynchronous copying latency

If a Speech LLM is fine-tuned solely with oracle ground-truth terminology, it becomes excessively brittle at test time, frequently succumbing to false positives and copying irrelevant distractor terms into the output stream. To eliminate this train-inference distribution shift, the authors build a massive 140K glossary from GigaSpeech transcripts and Wikidata entities. During Speech LLM training data preparation, the cross-modal retriever is actively run across training audio chunks to mine realistic distractor terms. The ground-truth terms and retrieved negative distractors are mixed to populate \(G_i\) (capped at 20 entries). Furthermore, within target-language reference segments, translations corresponding to valid glossary entries are wrapped with explicit <term>...</term> markup tags. Because translations naturally lag behind acoustic realization in streaming audio, fine-tuning the model with standard next-token cross-entropy loss provides implicit temporal supervision: the LLM learns to identify which retrieved hints actually match the observed acoustic stream and learns exactly when to deploy the translated term within the generated sentence structure. The markup tags are stripped prior to final evaluation.

3. Hard-Negative Mined Contrastive Learning: Enhancing fine-grained acoustic discrimination for rare named entities

To ensure the lightweight linear projection and dual encoders effectively discriminate phonetically similar words and domain terminology, retriever training couples in-batch negative sampling with aggressive hard-negative mining. Every 50 training steps, the retriever checkpoint indexes the training lexicon to retrieve the top-\(N\) (\(N=1024\)) hardest negative candidate terms for each ground-truth term \(e_{\text{pos}}\), assembling a challenging negative pool \(E_{\text{neg}}\). During training, since precise word-level boundaries of \(e_{\text{pos}}\) within audio segment \(s'\) are pre-extracted using the Montreal Forced Aligner (MFA), the model bypasses the costly inference-time multi-scale window maximization and directly optimizes the minimal bounding window enclosing \(e_{\text{pos}}\) under a multi-positive InfoNCE loss:

\[\mathcal{L}_{\text{retriever}} = -\log \frac{\exp(\text{Sim}(c'_i, e_{\text{pos}})/\gamma)}{\sum_{e \in \{e_{\text{pos}}\} \cup E_{\text{neg}}} \exp(\text{Sim}(c'_i, e)/\gamma)}\]

This decoupled formulation maintains minimal training overhead while forcing the acoustic and text representations into a shared, highly discriminative metric space.

Loss & Training

Both the cross-modal retriever and the Speech LLM are trained using parameter-efficient LoRA. For the retriever, LoRA is applied to the Qwen3-Omni Audio Transformer's query/key/value/output projections and MLP layers (\(r=32\), \(\alpha=64\)), and to BGE-M3's Q/K/V projections (\(r=16\), \(\alpha=32\)). Retriever training runs on 8 NVIDIA RTX A6000 GPUs with temperature \(\gamma=0.03\), global batch size 8192, AdamW optimizer (learning rate \(1 \times 10^{-4}\)), and cosine learning rate decay. For the Speech LLM, Qwen3-Omni-30B-A3B-Instruct is fine-tuned with LoRA (\(r=32\), \(\alpha=32\)) across 4 RTX A6000 GPUs using Distributed Data Parallel and expert parallelism (EP=2), sequence packing up to 3072 tokens, and global batch size 4, optimized via standard autoregressive cross-entropy computed strictly over translation tokens.

Key Experimental Results

Main Results

Evaluation is conducted using SimulEval across the ACL 60/60 development set (En-Zh, En-De, En-Ja, comprising 5 technical ACL conference talks) and the ESO oncology test set (En-De, comprising 5 specialized medical training videos). Chunk durations vary across 0.96s, 1.92s, 2.88s, and 3.84s. Evaluated metrics include exact-match Terminology Accuracy (%), SacreBLEU, semantic metric XCOMET-XXL, and StreamLAAL (latency in milliseconds).

Dataset & Direction Latency Multiplier (Chunk) System Configuration StreamLAAL (ms) โ†“ Term Acc (%) โ†‘ BLEU โ†‘ XCOMET-XXL โ†‘
ACL 60/60 En-Zh 1.92s (Multiplier 2) Offline ST 2650.0 83.10 48.00 84.10
ACL 60/60 En-Zh 1.92s (Multiplier 2) Offline + GT Terms 2650.0 92.50 50.80 85.20
ACL 60/60 En-Zh 1.92s (Multiplier 2) InfiniSST (Baseline) 1860.2 75.17 45.82 83.50
ACL 60/60 En-Zh 1.92s (Multiplier 2) RASST (Ours) 1814.3 90.00 47.88 83.63
ACL 60/60 En-De 1.92s (Multiplier 2) InfiniSST (Baseline) 1920.0 68.21 30.27 83.10
ACL 60/60 En-De 1.92s (Multiplier 2) RASST (Ours) 1880.0 82.99 30.87 83.97
ACL 60/60 En-Ja 1.92s (Multiplier 2) InfiniSST (Baseline) 2240.0 65.96 27.72 68.20
ACL 60/60 En-Ja 1.92s (Multiplier 2) RASST (Ours) 2210.0 84.57 29.89 70.38
ESO Oncology En-De 1.92s (Multiplier 2) Offline ST 2900.0 58.50 28.20 75.20
ESO Oncology En-De 1.92s (Multiplier 2) InfiniSST (Baseline) 2050.0 52.00 26.50 73.80
ESO Oncology En-De 1.92s (Multiplier 2) RASST (Ours) 2010.0 78.21 29.80 74.00

Ablation Study

Component-wise ablations analyze the contribution of multi-scale windowing versus fixed windowing during inference and training on ACL En-Zh (latency multiplier 2), as well as the impact of training speech context lengths on retrieval Recall@10 across varying glossary scales.

End-to-end ablation of retrieval strategies on ACL En-Zh (latency multiplier 2):

Strategy Variant BLEU โ†‘ StreamLAAL (ms) โ†“ Term Acc (%) โ†‘ Mechanism & Analysis
RASST Multi-scale (Full) 47.88 1814.34 90.00 MFA-localized training + multi-scale dynamic pooling at inference
Largest-infer 47.83 1868.37 84.72 MFA-localized training, but queries only the largest window at inference
Largest-train 45.80 1863.20 73.93 Trains and queries retriever directly with largest window without localization
InfiniSST + RAG (No train-time terms) 39.50 1850.00 89.20 Zero term exposure during training leads to severe hallucinations and verbatim copy
RASST w/ LLM Distractors 46.50 1830.00 87.50 Synthesizing negative distractors via pure LLM prompts underperforms mined negatives

Recall@10 (%) across glossary sizes for retrievers trained with different speech context lengths:

Context Duration Setting 1K Glossary 10K Glossary 100K Glossary Observations
Fixed 1.92s 98.11 97.47 95.18 Insufficient context; struggles with boundary-straddling and compound phrases
Fixed 3.84s 98.65 98.31 97.58 Consistent improvement but lacks adaptability to variable streaming chunks
Fixed 5.76s 99.64 99.58 99.33 High recall but rigid temporal window during short-latency streaming
Variable 2.88sโ€“5.76s (Ours) 99.20 98.97 98.58 Matches the recall of 5.76s while offering superior inference flexibility

Key Findings

  • Dramatic Terminology Improvements: RASST outperforms InfiniSST by 12.3 to 31.22 percentage points in exact-match terminology accuracy across all 12 language-latency settings. Crucially, RASST even beats the full-context Offline ST baseline, demonstrating that complete speech history alone is inadequate for handling rare domain terms without explicit glossary retrieval. The performance gain is most pronounced on ESO oncology (+26.0%โ€“31.2%), where medical jargon causes unaugmented models to fail catastrophically.
  • Train-Inference Noise Matching Preserves Syntax: Providing retrieved terms to a baseline model not trained to handle them (InfiniSST + RAG) yields high terminology copy rates but degrades BLEU by over 8 points. Unsupervised models blindly copy hints into the translation stream without respecting grammatical structure. Training with mined distractors and <term> tags enables the model to accurately gauge relevance and maintain fluid syntactic cohesion.
  • Negligible Latency Overhead and Scalability: The retriever requires only 0.036sโ€“0.043s of wall-clock time per chunk, which accounts for less than 3.75% of the chunk duration. When scaling the glossary from 200 entries to 10,000 entries, RASST experiences less than 2% drop in terminology accuracy and less than 1 BLEU point loss, confirming exceptional practical scalability.

Highlights & Insights

  • Decoupled Cross-Modal Streaming RAG Architecture: RASST elegantly decouples fine-grained acoustic phonetic retrieval (handled efficiently by BGE-M3 and Qwen3-Omni Audio Transformer) from high-level autoregressive sequence generation. This plug-and-play design allows glossary expansion and runtime modification without expensive retraining of the underlying LLM.
  • Noise-Conditioned Counterfactual Training: By proactively exposing the Speech LLM to realistic hard-negative retrieval distractors during SFT, the framework effectively inoculates the language model against hallucination and false-positive copy errors, solving a notorious hurdle in streaming RAG.

Limitations & Future Work

  • Restricted Source Language Coverage: The entire framework is presently evaluated exclusively with English audio. Validating whether the multi-scale pooling and dual-encoder retrieval generalize seamlessly to tonal languages (e.g., Mandarin) or agglutinative tongues remains an open challenge.
  • Absence of Explicit Phonetic / Homophone Modeling: Unlike traditional ASR contextual biasing frameworks that incorporate grapheme-to-phoneme (G2P) pronunciations or trie-constrained decoding, RASST relies purely on dense vector similarities, occasionally confusing acoustic homophones in noisy acoustic environments.
  • Dependence on Curated Glossary Coverage: When using automatically extracted glossaries from paper PDFs rather than human-curated ones, glossary coverage drops to 23.95%, substantially dampening the resulting terminology gains. Future research should investigate unsupervised real-time terminology discovery.
  • vs InfiniSST (Ouyang et al., 2025b): While InfiniSST pioneered unbounded causal streaming translation with Speech LLMs, it relies entirely on parametric memory, failing catastrophically on rare or domain-specific terms; RASST augments its interleaved stream with real-time multi-scale retrieval, achieving up to 31.2% terminology accuracy gains.
  • vs Contextual Biasing in ASR (Gong et al., 2025; Kong et al., 2025): Contextual ASR biasing aligns words monotonically with identical-language audio; RASST solves the far harder problem of asynchronous, cross-lingual translation where target terms must be generated with variable temporal delays relative to speech.
  • vs Offline Speech Translation RAG (Li et al., 2024; Wu et al., 2025): Prior retrieval methods in ST targeted complete, unsegmented utterances offline; RASST is the first to introduce causal history look-back buffers and multi-scale sliding windows to make RAG viable in low-latency streaming translation.

Rating

  • Novelty: โญโญโญโญโญ [Pioneers retrieval-augmented generation for streaming simultaneous speech translation with elegant multi-scale windowing]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive cross-lingual benchmarks, multiple streaming latency regimes, stress tests up to 10K terms, and comprehensive ablations]
  • Writing Quality: โญโญโญโญโญ [Crystal clear problem formulation, rigorous mathematical definitions, and compelling illustrative case analyses]
  • Value: โญโญโญโญโญ [Provides an immediately deployable, low-overhead blueprint for deploying Speech LLMs in professional international conference translation]