OmniForcing: Unleashing Real-time Joint Audio-Visual Generation¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: joint audio-visual generation, block-causal attention, attention sinks, joint Self-Forcing distillation, streaming decoding
TL;DR¶
OmniForcing distills the 19B bidirectional audio-visual model LTX-2 into a causal model that jointly generates one-second blocks, stabilizes training with audio sinks, and mitigates error accumulation through joint Self-Forcing, achieving approximately 0.7-second first-chunk latency and 25 FPS at 480p on one H200 without preserving every aspect of teacher quality losslessly.
Background & Motivation¶
Joint audio-visual diffusion models coordinate visual actions, speech, and ambient sounds within one generation process, but bidirectional attention requires processing the entire timeline together. Even a user who only needs the first second must wait for full-sequence sampling. LTX-2 takes 197 seconds in the paper's five-second clip setting. Cascades that generate video before adding audio can use specialized sound models, but introduce a dependency in which the second modality waits for the first, making synchronized streaming difficult to obtain directly.
Video methods such as CausVid and Self-Forcing already convert bidirectional teachers into causal students, but their masks cannot simply be copied into a dual-stream model. LTX-2 produces 3 video latent frames and 25 audio latent frames per second; each video latent frame contains 384 spatial tokens, whereas each audio latent frame contains only one token. Audio therefore has fewer attention tokens per second despite its higher latent frame rate. Removing future context causes unequal conditional distribution shifts in the two streams. The paper observes NaN failures under direct causal conversion, while subsequent generation can propagate errors from one stream into the other.
The paper consequently treats temporal alignment, training stability, and adaptation to self-generated history as connected problems, rather than merely reducing sampling steps. Core Idea: organize asymmetric audio-visual blocks around complete physical seconds, retain a global temporal origin and position-agnostic audio sinks, and jointly distill the student on its own dual-stream history so that few-step generation, causal dependencies, and cross-modal synchronization work together.
Method¶
Overall Architecture¶
The input is text describing visual content and acoustic events; the output is audio and video that can be played block by block. The backbone retains LTX-2's 14B video branch and 5B audio branch, their respective VAEs, and interaction in both cross-modal directions, but restricts temporal visibility to the current and preceding blocks. Causality here is defined over one-second blocks, with bidirectional interaction inside each block. It is not strictly sample-by-sample or video-frame-by-video-frame causality.
Training first obtains few-step denoising through bidirectional Distribution Matching Distillation (DMD), then regresses teacher trajectories under asymmetric block-causal alignment and audio-sink constraints, and finally applies joint Self-Forcing distillation. Inference runs only the resulting student, using modality-independent rolling caches and streaming VAE decoding. The teacher, critic, and distillation objectives are absent from the online generation path.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
T["Bidirectional teacher and<br/>few-step DMD initialization"] -.->|Stage II trajectory supervision| B["Asymmetric Block-Causal Alignment"]
B --> C["Position-Agnostic Audio Sinks"]
C --> D["Joint Self-Forcing Distillation"]
T -.->|Stage III distribution supervision| D
D -.->|Trained student| E["Modality-Independent Rolling Cache"]
A["Text and dual-stream noise"] --> E
E --> O["Causal decoding and<br/>synchronized audio-visual output"]
Dashed edges indicate training supervision and transfer of trained parameters. The online path supplies text and noise to the student's cached generation process before decoding. The first two designs below configure the student architecture, the third defines its training procedure, and the fourth describes deployment.
Key Designs¶
1. Asymmetric Block-Causal Alignment: define shared history by physical time rather than token count
Forcing individual latent frames in the two streams into one-to-one correspondence would cause truncation or boundary mismatch under their 25:3 frequency ratio. Instead, each one-second macro-block contains 3 video latent frames and 25 audio latent frames. In the evaluated configuration, these expand into 1152 video tokens and 25 audio tokens. All four attention pathways, comprising video self-attention, audio self-attention, and cross-attention in both directions, use physical block indices to determine visibility. The current block can attend to the other modality within the same block and to retained preceding blocks, but not to future blocks. Thus, unequal token counts do not produce unequal physical-time receptive fields.
Causal VAEs use different temporal strides for the absolute first frame and subsequent frames, so the initial video and audio latents cannot simply be placed inside an ordinary one-second block. The paper groups them into a separate global prefix B0, allows bidirectional attention within that prefix, and keeps it visible to subsequent blocks. This addresses both boundary alignment and persistent semantic reference: the prefix is not an extra content prompt, but a way to preserve the VAE's special initial-frame structure without dropping boundary information. The latent-count equations are damaged in the cached extraction, so this note follows the surrounding explanation rather than reconstructing them as exact author formulas.
2. Position-Agnostic Audio Sinks: provide sparse audio attention with a learnable global buffer
The causal mask replaces full-sequence audio context with a small amount of history. The authors attribute training collapse to concentrated attention under sparse context, conditional distribution shift, and numerical instability. They prepend 16 learnable sink tokens to the audio sequence, assign them to B0, and keep them visible to subsequent queries. These tokens do not represent actual sounds to be emitted and receive no separate sink-label supervision; they learn a global contextual buffer through training. Particularly near the beginning of a sequence, they provide additional attention targets and reduce dependence on a very small number of real audio tokens.
Adding tokens alone is insufficient. Assigning ordinary consecutive position indices to abstract memory slots would impose a physical temporal ordering that they do not possess. Identity RoPE makes their rotary transformation an identity mapping, avoiding artificial temporal phases. With the same 16 sinks, the ablation reports a one-step loss of 0.402 for incremental RoPE versus 0.081 for Identity RoPE. This supports the importance of memory-slot position semantics, but the authors' qualitative account of Softmax denominators and exploding gradients is not a general mathematical proof: Softmax saturation does not by itself imply an unbounded local derivative.
3. Joint Self-Forcing Distillation: expose both streams to their own historical errors during training
A student adapted through causal trajectory regression can denoise under restricted context, but online generation conditions on its own preceding images and sounds rather than consistently reliable history. An action occurring too early or a sound arriving too late can become the next block's conditioning context and amplify cross-modal mismatch. Joint Self-Forcing therefore unrolls the student block by block during training, writes noise-free KV representations of generated blocks into the cache, and uses them to generate subsequent blocks. The training distribution now includes the imperfect histories encountered at deployment.
Supervision comes from a frozen bidirectional teacher and a critic trained to estimate the student's output distribution. Noise is added to student-generated audio-visual trajectories, and the difference between their scores supplies the DMD update, with weighted video and audio terms. The teacher supplies a target joint distribution, while the student approaches it under simultaneous dual-stream autoregression and learns to accommodate existing deviations in the other modality. This is not an added explicit lip-sync or event-synchronization loss, nor a separate inference-time correction module. Self-correction instead follows from closer agreement between training and deployment histories. The extracted main equation is incomplete, so its exact gradient and norm expressions are not guessed here.
4. Modality-Independent Rolling Cache: exploit dual-stream computation for online execution
The video branch is heavier than the audio branch, but their within-layer self-attention and feed-forward sublayers can run independently, exchanging information at cross-modal attention boundaries. The method exploits this structure to maintain separate rolling KV caches and schedule concurrent execution, avoiding full-sequence recomputation for each generation step. The paper describes per-step context complexity within the cache window as O(L), where L counts latent frames in that window. This does not imply constant end-to-end cost for arbitrarily long outputs, particularly because audio decoding grows differently. Asymmetric tensor parallelism across devices is a proposed scaling direction; the headline measurements use one GPU.
Decoding must also remain causal: streaming latent generation alone does not ensure that users can immediately play the output. Video uses the current block and W preceding blocks in a sliding-window decoder, with W=2 selected for deployment. Audio is cumulatively re-decoded from the beginning and passed through a vocoder to preserve agreement with full-sequence decoding. The two decoders run concurrently, and decoding completed blocks can overlap generation of later blocks. Their different strategies reflect modality-specific costs: video needs a bounded window, whereas cumulative audio decoding remains within 14 ms over the measured 30-second range. This does not establish the same bound for unlimited durations.
A Worked Example¶
Consider the scene in the paper's Figure 1, where an elderly woman at a sewing machine talks about making a quilt for her granddaughter. The text jointly specifies the person, sewing action, narration, and machine sounds. The generator establishes B0 and jointly generates the first one-second block B1, containing 3 video latent frames and 25 audio latent frames. The streams exchange context within B1 without waiting for the entire five-second sequence.
Time-To-First-Chunk (TTFC) covers generation and decoding of both B0 and B1. Once emitted, B1's noise-free KV representations condition B2, while subsequent blocks retain the origin reference. Video decoding looks back at most two blocks, and audio decoding processes the accumulated generated prefix. The distinction between "3 video latent frames" and "approximately 25 FPS output throughput" matters: the former describes compressed temporal resolution, while the latter describes generation speed after reconstruction. They are not interchangeable.
Loss & Training¶
Stage I applies weighted audio-visual DMD while retaining bidirectional receptive fields to learn few-step denoising. Stage II adds causal masks, sinks, and Identity RoPE, then regresses the Stage I teacher's ODE velocity predictions to adapt the architecture. Stage III performs causal DMD with jointly self-generated history. These stages run for 2000, 3000, and 2000 steps, respectively, instead of learning every constraint simultaneously from scratch. Stages I and III use backward simulation. The cached text does not clearly specify the final sampling-step count or modality loss weights, so "few-step" should not be expanded into invented numerical settings.
Training uses 32 GPUs, bf16, a global batch size of 32, and a learning rate of 0.00002. Video and audio CFG scales are 3 and 5, respectively. Text comes from Mixkit clips with Open-Sora-Plan captions and from AudioCaps descriptions, rewritten by Gemma 3 12B into coherent audio-visual descriptions. These should be understood as prompt sources for distillation. The presence of acoustic descriptions does not mean every example provides newly collected, strictly paired audio-visual supervision; the pretrained teacher remains the main source of generative quality.
Key Experimental Results¶
Main Results¶
The main evaluation uses JavisBench. FVD and FAD measure feature-distribution distances between generated and reference video and audio, respectively; lower is better. CLIP measures text-video semantic consistency, with higher being better, while DeSync measures temporal audio-visual mismatch, with lower being better. The following representative models are taken from the paper's Table 1. These metrics are not interchangeable measures of a single quality score. The main text does not state evaluation sample counts, the exact test split, or random seeds, which limits reproducibility of cross-model comparisons.
| Model | Parameters | FVD ↓ | FAD ↓ | CLIP ↑ | DeSync ↓ | Runtime ↓ |
|---|---|---|---|---|---|---|
| JavisDiT++ | 2.1B | 141.5 | 5.5 | 0.316 | 0.832 | 10 s |
| LTX-2 bidirectional teacher | 19B | 125.4 | 4.6 | 0.318 | 0.384 | 197 s |
| OmniForcing | 19B | 137.2 | 5.7 | 0.322 | 0.392 | 5.7 s |
Efficiency measurements concern five-second, 480p audio-visual clips on one NVIDIA H200 with bf16, CUDA 12.8, PyTorch 2.9.1, and SDPA MemoryEfficient. Timing includes VAE decoding and the vocoder, uses 3 warmups and 100 iterations, and inserts CUDA synchronization barriers. The text reports median TTFC=0.71 s and FPS=24.8, rounded to 0.7 s and 25 FPS in Table 2. Approximately 35 times faster refers to total runtime, 197 s versus 5.7 s, not the first-chunk latency ratio. The text does not sufficiently establish whether every other baseline was remeasured under an identical implementation protocol.
On the same-prompt VBench comparison, student aesthetic quality is 0.595 versus the teacher's 0.569, and imaging quality is 0.594 versus 0.574 (paper Table 2). Some visual metrics therefore improve after distillation, but FVD, FAD, and DeSync remain worse. The results should not be summarized as lossless quality preservation or improvement over the teacher on every metric.
Ablation Study¶
The paper's Table 3 compares stabilization strategies after 3000 Stage II training steps. The loss is one-step denoising loss at noise level 0.5 averaged over the evaluation set, not a final JavisBench score. The following settings isolate important mechanisms; standard sink variants use Identity RoPE.
| Config | Convergence | Maximum Gradient Norm | One-Step Loss ↓ | Output Observation |
|---|---|---|---|---|
| No stabilizer | NaN | Infinite | Not reported | Noise |
| 2 sinks | NaN | Infinite | Not reported | Noise |
| 4 sinks | Stable | 49.71 | 0.141 | Normal |
| 16 sinks | Stable | 9.23 | 0.081 | Normal |
| 24 sinks | Stable | 9.15 | 0.110 | Normal |
| 16 sinks + incremental RoPE | Stable | 11.21 | 0.402 | Noisier |
| QK-Norm | Stable | 4.45 | 0.232 | Normal |
| Tanh-Gated Attention | Plateau without NaN | 10.61 | 1.258 | Block artifacts |
The paper's Table 4 separately evaluates streaming-decoder fidelity against full-sequence decoding. Higher PSNR means closer agreement with the reference decoding of the same latent sequence, not greater realism of generated content. W counts preceding latent blocks and excludes the current block. Hardware matches the single-H200 setup above.
| Modality and Decoder Config | MAE ↓ | PSNR / dB ↑ | Decoding Latency per Block |
|---|---|---|---|
| Video: LTX-2 built-in causal mode | 0.083 | 20.5 | 426 ms |
| Video: sliding window W=1 | 0.065 | 19.8 | 201 ms |
| Video: sliding window W=2 | 0.002 | 47.4 | 288 ms |
| Video: sliding window W=4 | 0.002 | 47.4 | 444 ms |
| Audio: cumulative decoding and vocoder | 0 | Infinite | ≤14 ms, worst case measured at 30 seconds |
Key Findings¶
- More audio latent frames do not imply richer context: spatially expanded video has many more tokens, so stabilization should target the genuinely sparse stream.
- The smallest gradient norm does not guarantee the best generation behavior. QK-Norm's 4.45 is below the 9.23 of 16 sinks but accompanies a higher loss; 24 sinks also fail to improve over 16.
- W=2 and W=4 both achieve 47.4 dB, at 288 and 444 ms respectively. The paper describes the latency as approximately halved, but its table does not show an exact halving; the numerical entries take precedence.
Highlights & Insights¶
- Physical time is a more suitable cross-modal interface than token index for heterogeneous VAEs. The global prefix also turns special initial-frame encoding into a persistent semantic anchor instead of discarding boundary information.
- Sinks contribute more than extra parameters: abstract memory should not be forced to carry temporal positions. The incremental-RoPE comparison provides more specific evidence than simply showing that additional tokens help.
- End-to-end latency includes decoding and the vocoder, connecting few-step diffusion gains to playable output. Training stability, context caching, and decoder design must be considered together.
Limitations & Future Work¶
- The authors explicitly acknowledge that causal receptive fields sacrifice some consistency and synchronization quality. The teacher remains better on FVD, FAD, and DeSync, so real-time execution has a cost.
- This note's observation: headline runtime concerns five-second, 480p clips, while cumulative audio decoding is measured up to 30 seconds. Minute-scale joint-quality curves, prompt-switching tests, and long-horizon cache-eviction studies are missing; a streaming architecture does not by itself demonstrate indefinitely stable generation.
- This note's observation: Table 3 primarily validates sinks and positional encoding. It does not independently quantify the contributions of the global prefix or joint Self-Forcing to final synchronization metrics, and the stability explanation remains primarily empirical and qualitative.
- This note's observation: a 19B model running on one H200 does not establish real-time performance on consumer hardware. Training on 32 GPUs must also be distinguished from single-GPU deployment. Higher resolutions and multi-GPU asymmetric parallelism still require measurements.
Related Work & Insights¶
- vs LTX-2: the method retains the dual-stream teacher's architecture and joint-distribution capabilities, changing temporal visibility, training history, and execution rather than training a smaller model from scratch.
- vs CausVid / Self-Forcing: it inherits bidirectional-to-causal distillation and training on self-generated history, concentrating its extensions on dual-modality temporal-density mismatch and cross-modal error propagation.
- vs cascades such as MMAudio: those approaches generate sound conditioned on existing video, whereas this method jointly generates both streams within a block. This changes generation dependencies and cannot be judged solely through isolated audio quality.
Rating¶
- Novelty: 4/5. The combination of temporal alignment and stabilization directly addresses heterogeneous dual-stream generation, while the basic distillation ideas build on prior work.
- Experimental Thoroughness: 3/5. Quality, stabilizer, and decoder comparisons are provided, but long-sequence evaluation and complete component ablations are missing.
- Writing Quality: 3/5. The three-stage pipeline is clear, while some mechanistic claims and latency descriptions require careful interpretation.
- Value: 4/5. The paper provides actionable architectural and training choices for low-latency deployment of large joint audio-visual models.