Skip to content

MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation

Conference: ECCV 2026
arXiv: 2604.19679
Code: None
Area: Video Generation
Keywords: Joint Audio-Video Generation, Multi-Modal Control, Diffusion Transformer (DiT), Controllable Generation, Audio-Video Synchronization

TL;DR

MMControl proposes the first unified multi-modal control framework for joint audio-video generation. Through MMCU, heterogeneous control signals (including reference image, reference audio, depth maps, and pose sequences) are uniformly encoded. They are then non-intrusively injected into a frozen Joint DiT backbone via a dual-stream bypass architecture. During inference, modality-specific guidance scale factors are utilized to independently adjust the visual and acoustic control strengths. This approach comprehensively outperforms existing uni-modal control methods in identity consistency, timbre fidelity, and structural alignment.

Background & Motivation

Background: Joint audio-video generation models built on DiT backbones (e.g., Sora 2, LTX-2, MoVa) can simultaneously synthesize videos and synchronized audio within a single model, far surpassing traditional cascaded approaches in cross-modal temporal consistency and generation quality.

Limitations of Prior Work: Current controllable generation frameworks (e.g., ControlNet, VACE, OmniControl) focus almost entirely on the visual modalityโ€”either controlling video structure while neglecting audio, or performing audio-to-video lip-syncing while ignoring visual identity and scene structure. This leads to three practical issues: (1) Users cannot specify both the character's appearance and voice in a single workflow; (2) Visual and acoustic control signals originate from different systems, causing cross-modal alignment to rely on ad-hoc post-processing, which leads to poor consistency and synchronization; (3) The lack of a unified inference-time regulation mechanism prevents users from independently adjusting the control strengths of vision and audio.

Key Challenge: Joint generation models naturally possess cross-modal synchronization priors, but existing control injection methods are designed for single modalities. Directly inserting two independent ControlNet structures into the video and audio streams respectively not only disrupts the synergy of the joint latent space but also introduces modality conflicts.

Goal: To achieve unified controllable generation of visual identity/structure (reference images, depth, poses) and acoustic timbre (reference audio) while preserving the cross-modal synchronization capability of the joint DiT.

Core Idea: By tokenizing all heterogeneous control signals into a unified representation, they are injected into the even layers of a frozen Joint DiT using zero-initialized gated residuals via two independent, trainable bypass branches (visual and acoustic). During inference, two scalar factors \(\gamma_v\) and \(\gamma_a\) are used to independently scale the control strengths of the two branches. This achieves decoupled multi-modal control without disrupting the pre-trained generation priors.

Method

Overall Architecture

The problem that MMControl aims to address is generating synchronized video and audio from a frozen, pre-trained Joint DiT (LTX-2 19B), given text prompts, an optional reference image (specifying identity), an optional reference audio (specifying timbre), and optional depth/pose sequences (specifying structural constraints). The overall workflow consists of four stages: First, the Multi-Modal Control Unit (MMCU) encodes heterogeneous inputs into a unified synchronized representation with masks; then, the dual-stream bypass architecture injects these representations into even layers of the backbone via independent visual and acoustic branches; next, modality-specific guidance scaling factors \(\gamma_v\) and \(\gamma_a\) independently adjust the control strengths during inference; finally, a two-stage progressive inference first generates a semantic base at low resolution, then upsamples it to full resolution for detail refinement.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt + Reference Image/Audio<br/>+ Depth/Pose Sequences"] --> B["MMCU: Unified Multimodal Control Encoding<br/>Visual Tokenization + Acoustic Tokenization + Mask Gen"]
    B --> C["Dual-Stream Bypass: Modality-Specific Injection<br/>Visual Bypass Branch | Acoustic Bypass Branch<br/>Interleaved Injection into Frozen Joint DiT Even Layers"]
    C --> D["Modality-Specific Guidance Scaling<br/>ฮณ<sub>v</sub> regulates vision ยท ฮณ<sub>a</sub> regulates audio"]
    D --> E["Two-Stage Progressive Inference<br/>Stage 1: Low-Resolution Semantic Base Generation"]
    E --> F["Stage 2: High-Resolution Detail Refinement<br/>LoRA Upsampling + VAE Decoding"]
    F --> G["Output: Synchronized Video + Audio"]

Key Designs

1. MMCU: Unified Encoding and Synchronization of Multi-Modal Control Signals

The challenge lies in the fact that the reference image (a static image), reference audio (a waveform), and depth/pose sequences (frame-by-frame structural maps) are three completely different modalities with distinct durations. They must be aligned into a unified temporal representation to form a valid input for the joint diffusion model. MMCU resolves this as follows: On the visual side, the VAE latent of the reference image \(z_{\text{img}}\) is placed at the beginning of the sequence, followed by the latents of the \(t\) structural guidance frames \(\{z_{s,1}, \dots, z_{s,t}\}\). Concurrently, a binary mask \(M_v\) is generated, where the reference frame contains \(0\)s and target generation frames contain \(1\)s. This forces the model to explicitly distinguish between "preserving identity" and "following structure". On the acoustic side, the \(k\) latent tokens of the reference audio \(\{z_{\text{aud},1}, \dots, z_{\text{aud},k}\}\) are prepended, followed by \(t\) silent latent tokens as generation placeholders; the acoustic mask \(M_a\) is set to all \(0\)s for the reference segment and all \(1\)s for the generative segment. The elegance of this design is that the mask channels enable the model to determine whether each frame is "conditioning" or "to-be-generated" without extra position coding or segment embeddings. Furthermore, the design of silent placeholders leaves interfaces for future extensions like precise acoustic control (e.g., pitch or energy curves).

2. Dual-Stream Bypass Architecture: Non-Intrusive Modality-Specific Control Injection

Directly injecting control signals into the frozen Joint DiT via extra cross-attention or concatenation disrupts the pre-trained cross-modal attention maps, which degrades audio-video synchronization. To address this, MMControl establishes two parallel, trainable bypass branchesโ€”a visual bypass and an acoustic bypassโ€”to process their respective MMCU representations. Inside each bypass, the Self-Attention, Text Cross-Attention, and FFN layers inherit weights from the pre-trained backbone, but an extra Context Projector is added to project the concatenated MMCU sequence and mask along the channel dimension into the DiT hidden dimension. Crucially, the projector weights corresponding to the latents are inherited from the backbone's embedder, whereas the mask channel weights are initialized from scratch. The bypass branches only intervene at even layers, and their outputs are converted into hint vectors via a zero-initialized projection layer, which are then injected into the main backbone layers as gated residuals:

\[ \mathbf{x}_{\text{main}}^{(l)} = \text{MainBlock}^{(l)}(\mathbf{x}_{\text{main}}^{(l-1)}) + \gamma_m \cdot \text{Hint}_m^{(l)}, \quad m \in \{v, a\} \]

Zero-initialization ensures that the hint starts as a zero vector at the beginning of training, avoiding disruption to pre-trained priors, while progressively learning to inject useful control signals. Additionally, cross-attention between audio and video is intentionally omitted inside the bypasses, encouraging each branch to focus on its own modality-specific control while reducing computational overhead.

3. Modality-Specific Guidance Scaling: Decoupled Inference-Time Control Strength Adjustment

A practical issue during inference is that users might want strict identity preservation while allowing flexible speech variations, or vice versa. Directly training a fixed set of control strengths cannot satisfy diverse user preferences. MMControl introduces two scalar factors, \(\gamma_v\) and \(\gamma_a\), to independently scale the hints from the visual and acoustic bypasses. When \(\gamma = 0\), the corresponding modality is purely driven by text (reverting to the pre-trained prior), and when \(\gamma = 1\), it strictly follows the conditioning signals. A typical application is demonstrated in the experiments: setting \(\gamma_v = 1\) to anchor the identity visual + \(\gamma_a = 0\) to let the model freely synthesize speech, resulting in visual consistency with diverse voices. Conversely, setting \(\gamma_v = 0\) + \(\gamma_a = 1\) preserves the reference timbre while generating novel character appearances. This essentially extends standard single-scalar ControlNet scale adjustments to multi-modal, multi-scale decoupled regulation.

4. Two-Stage Progressive Inference: Efficient High-Resolution Generation

Directly generating full-resolution (\(h \times w\)) audio-video content from noise is computationally expensive. Following the strategy of LTX-2, MMControl divides the process into two stages: Stage 1 runs the full denoising pipeline at half resolution (\(h/2 \times w/2\)) using the complete MMCU conditions, standard flow-matching schedules, and CFG, thereby generating a low-resolution but semantically complete and structurally correct draft. Stage 2 utilizes a spatial latent upsampler to scale the video latent to full resolution, and then runs only a 3-step truncated noise schedule for detail refinement. During this stage, CFG is disabled to preserve the established semantic layout, and MMCU signals are re-encoded at full resolution to guide high-frequency details. Frozen LoRA weights distilled from LTX-2 are activated in Stage 2, enhancing fine-grained texture synthesis without additional training costs.

A Complete Example: Joint Generation with Reference Image, Reference Audio, and Depth Maps

Suppose the input consists of: (1) a reference portrait (half-body), (2) a 3-second reference audio (the speaker saying "The weather is so nice today"), (3) a 5-second depth sequence (extracted from a video of hand waving), and (4) the text prompt "[VISUAL]: A person waves hand while speaking outdoors [SPEECH]: The weather is so nice today". MMCU first encodes the reference image into \(z_{\text{img}}\) using a VAE, and the 5 depth frames into \(\{z_{\text{depth},1}, \dots, z_{\text{depth},5}\}\). These are concatenated into a visual sequence \([z_{\text{img}}, z_{\text{depth},1}, \dots, z_{\text{depth},5}]\) with the mask \([0, 1, 1, 1, 1, 1]\). Simultaneously, the reference audio is encoded into \(\{z_{\text{aud},1}, \dots, z_{\text{aud},k}\}\), concatenated with \(t=5\) frames of silent placeholders, and assigned the mask \([0, \dots, 0, 1, 1, 1, 1, 1]\). These two representations enter the visual and acoustic bypasses, respectively, and are injected into the even layers of the frozen Joint DiT with strengths \(\gamma_v=1.0\) and \(\gamma_a=0.8\). Stage 1 produces a low-resolution draft (already with waving gestures and synchronized lip movements), and Stage 2 refines it to full resolution in 3 steps. The VAE decodes the output as a 5-second synchronized video and audioโ€”where the character's appearance matches the reference image, gestures follow the depth guidance, and the speech matches the reference timbre with synchronized lip movements.

Loss & Training

Training adopts the flow-matching objective of LTX-2, keeping the backbone completely frozen while updating only the bypass branches and projector parameters. The optimizer is AdamW with a peak learning rate of 1e-5 and a cosine annealing schedule over 7,200 steps. Training is conducted on 4 H200 GPUs with a batch size of 2 per GPU and gradient accumulation of 2, totaling about 12 hours. The training dataset consists of 30,000 selected high-quality samples from the Hallo3 dataset, covering three task combinations: reference image + audio, reference image + depth, and reference image + pose. To support CFG, the visual and acoustic control signals are independently dropped with a probability of 0.1.

Key Experimental Results

Main Results

Audio-Driven Joint Generation Comparison (Table 1): In a comprehensive evaluation spanning text alignment (Text CLIP Similarity), identity preservation (Subject DINO Similarity), dynamic degree, aesthetic quality, motion smoothness, and audio-video synchronization metrics (Sync-C/Sync-D), MMControl achieves the highest Text CLIP (0.2546), Subject DINO (0.8948), and Sync-C (2.716). Moreover, its Sync-D (10.506) outperforms most baselines using ground-truth audio. Notably, baselines like Hallo3 use ground-truth audio and the first frame, which are much easier conditioning setups than MMControl's. Still, MMControl wins in synchronization and identity preservation, proving that the cross-modal correlation captured by the joint DiT is superior to post-concatenation pipelines.

Method Text CLIP Similarity โ†‘ Subject DINO Similarity โ†‘ Dynamic Degree โ†‘ Aesthetic Quality โ†‘ Motion Smoothness โ†‘ Sync-D โ†“ Sync-C โ†‘
SadTalker 0.2410 0.8752 0.0057 0.5434 0.9977 10.696 2.474
AniPortrait 0.2302 0.8857 0.0368 0.5350 0.9965 12.383 1.322
HunyuanCustom 0.2203 0.7816 0.6150 0.5003 0.9955 10.912 1.628
Hallo3 0.2337 0.8873 0.9184 0.5169 0.9667 10.243 2.550
MMControl 0.2546 0.8948 0.5750 0.5461 0.9954 10.506 2.716

Structural Control: For depth control, MMControl achieves a Mean MAE (x100) of 4.52, significantly outperforming VideoComposer's 15.41 and VACE's 5.35. For pose control, it achieves an MAE of 3.07, surpassing VACE's 3.29. Meanwhile, it maintains the best Subject DINO similarity and motion smoothness across both tasks, demonstrating that MMCU closely follows structure signals without sacrificing visual semantic consistency. In the human evaluation, MMControl scores an overall 3.58 (on a 4-point scale) across six metrics: lip synchronization, facial naturalness, motion naturalness, text alignment, subject alignment, and visual quality, outperforming Hallo3's 3.22.

Ablation Study

Configuration Sync-C โ†‘ SIM-o โ†‘ Depth MAE (x0.01) โ†“ Pose MAE (x0.01) โ†“ Key Findings
Full model 2.73 0.22 4.65 3.78 Full model
M1: W/o MMCU mask 2.53 0.17 5.01 3.92 Masks are crucial for audio-video synchronization and identity preservation
M2: W/o projector weight inheritance 2.36 0.16 18.32 14.42 Weight inheritance is the cornerstone of structural control; without it, depth/pose MAE spikes by 3-4x
B1: Unified attention instead of dual-stream 2.15 0.17 5.13 4.00 Dual-stream separation is critical for cross-modal synergy
B2: W/o acoustic bypass 2.20 0.10 4.97 4.24 Acoustic bypass is indispensable for timbre preservation; SIM-o is halved
B3: W/o visual bypass 2.15 0.11 19.35 16.83 Visual bypass is the primary channel for structural control; without it, MAE spikes
B4: Add cross-attention inside bypasses 2.73 0.14 4.87 4.01 Adding cross-attention unexpectedly degrades SIM-o, validating the rationale of intra-modality focus

Key Findings

  • Projector weight inheritance is the core enabler of structural control: Without it, depth MAE spikes from 4.65 to 18.32 (~4x) and pose MAE rises from 3.78 to 14.42 (~3.8x). This is far more severe than the impact of other ablation variants, indicating that the representation space inherited from the pre-trained embedder is irreplaceable for understanding spatial structural signals.
  • Dual-stream separation is clearly superior to unified attention: In B1, merging the two bypass streams into unified attention drops the Sync-C score sharply from 2.73 to 2.15. This proves that visual and acoustic control signals must be processed independently at the feature level; hard-fusion introduces inter-modality interference.
  • The decoupling effect of the guidance scaling factor is significant: As \(\gamma_v\) increases from 0.5 to 1.3, Subject DINO improves from 0.860 to 0.911, while SIM-o remains nearly constant. As \(\gamma_a\) scales from 0.5 to 1.3, SIM-o increases from 0.16 to 0.22 before dropping to 0.14 (excessive strength becomes counterproductive). The two factors influence their respective modalities independently without noticeable cross-modal leakage.
  • MMControl trails Hallo3 in dynamic degree: Hallo3's Dynamic Degree of 0.9184 is significantly higher than MMControl's 0.5750. This is because Hallo3 is explicitly optimized for large-amplitude movements. However, MMControl is superior in aesthetic quality and identity preservation, showing a quality-versus-dynamics trade-off.
  • Comparison with AVControl: MMControl leads AVControl by a large margin in both depth control MAE (4.52 vs 11.49) and pose control MAE (3.07 vs 7.57). This shows that the unified MMCU + dual-stream bypass approach is more effective than training task-specific LoRA adapters.

Highlights & Insights

  • Mask channels provide a low-cost means of multi-modal signal decoupling: MMCU not only concatenates tokens of different modalities at the sequence level, but also concatenates binary masks along the channel dimension. This operation adds almost zero parameters and computational cost, yet allows the model to differentiate "reference frames" vs. "generated frames" and "reference audio" vs. "generated audio" without any extra learning. This design is almost trivial, yet ablation experiments show that removing it drops Sync-C by 0.2 and SIM-o by 0.05, indicating that explicit signal separation is highly important for joint diffusion models.
  • The combination of zero-initialization and weight inheritance is key to training stability: The latent weights of the projector are inherited from the backbone (preserving the pre-trained representation space), the mask weights are trained from scratch (to adapt to the new channel), and the bypass output projectors are zero-initialized (preventing early-stage disruption during training). This three-pronged strategy allows a lightweight bypass trained on a frozen 19B model to converge in just 12 hours, representing a highly replicable engineering methodology.
  • "Omitting cross-attention" is an intentional design choice: The B4 ablation demonstrates that introducing audio-video cross-attention within the bypasses actually degrades SIM-o, challenging the intuition that multi-modal networks always benefit from interaction. The reason is that the bypass's goal is to extract purified control features, and cross-modal interactions should be handled by the original attention layers of the frozen backboneโ€”adding interaction inside the bypass is over-engineering.
  • Transferable Paradigm: The three-step framework of MMCU + dual-stream bypass ("unified encoding -> modality-separated injection -> independent inference-time adjustment") can be transferred to any pre-trained generative model requiring multi-modal conditional control. For example, simultaneously controlling content and audio style in video editing models, or controlling movement sequences and footstep sounds in 3D human motion generation.

Limitations & Future Work

  • The authors acknowledge: The current framework only supports single-person scenes, leaving cross-speaker synchronization and long-term consistency in multi-character dialogue scenarios unaddressed. Currently, control modalities are limited to four types (reference image, audio, depth, and pose); extending to arbitrary new modalities still requires task-specific adaptation.
  • Potential Limitations: (1) Although two-stage inference is efficient, the low-resolution bottleneck of Stage 1 might lose details in highly dynamic scenes. Additionally, using CFG only in Stage 1 and disabling it in Stage 2 warrants examination to see if it causes semantic drift for certain prompts. (2) The training data comprises only 30k samples from Hallo3, casting doubt on its generalization to complex outdoor scenes and rare characters. (3) There is a lack of quantitative analysis concerning improvements in audio quality (e.g., pronunciation accuracy, naturalness of prosody); although the WER metric (7.96% weighted) is acceptable, there is still room for improvement.
  • Future Directions: (1) Replace silent placeholders with learnable acoustic control tokens to support fine-grained prosodic control such as pitch, energy, and speech rate. (2) Introduce speaker diarization mechanisms to support multi-person dialogue scenarios. (3) Explore extending \(\gamma_v / \gamma_a\) from static scalars to adaptive schedules across different layers or timesteps, precisely regulating the conditioning dependency strength across different denoising stages.
  • vs ControlNet/T2I-Adapter: These established the classic paradigm for spatial condition injection in image generation (bypass module + zero-initialization). MMControl extends this from uni-modal image generation to multi-modal joint audio-video generation, with the core difference being the transition from a single bypass to a dual-stream visual/acoustic bypass, alongside the introduction of modality-specific guidance scaling for independent control adjustment.
  • vs VACE/OmniControl: VACE is a unified framework for video visual control, and OmniControl handles arbitrary joint control for human motionโ€”but both only process the visual modality. MMControl's contribution is not simply "unifying more visual conditions" (which VACE already does), but expanding the concept of "unified control" from pure vision to a cross-visual-acoustic multi-modal space.
  • vs MoCha/Hallo3: These methods focus on audio-to-video generation, where audio is the input and video is the output. MMControl operates as a bidirectional joint generation framework (audio-and-video) where both modalities are generated simultaneously, constrained by text and control signals, naturally yielding superior cross-modal synchronization.
  • vs AVControl: AVControl is a concurrent work that trains independent IC-LoRA adapters for different control modalities. MMControl's advantage lies in sharing a single MMCU + dual-stream bypass architecture across all control modalities without task-specific adapters, and outperforming AVControl in MAE by a large margin (depth: 4.52 vs 11.49, pose: 3.07 vs 7.57). However, AVControl supports loudness control on the acoustic side, whereas MMControl currently supports only timbre control, making their acoustic control scopes complementary.

Rating

  • Novelty: โญโญโญโญโ˜† Introducing unified multi-modal control to joint audio-video generation is a clear gap filled. The MMCU + dual-stream bypass architecture is well-designed, although the individual techniques (masks, bypasses, guidance scaling) are not entirely novel at the component level. The novelty shines through system integration and problem formulation.
  • Experimental Thoroughness: โญโญโญโญโญ Covers 4 control signal combinations, compares with 7 baselines, and includes automatic metrics, human evaluations, and audio quality assessments. The ablation studies dissect 7 variants and analyze the sensitivity curves of the \(\gamma\) factors. Furthermore, the appendix provides a head-to-head comparison with the concurrent work AVControl.
  • Writing Quality: โญโญโญโญโ˜† Clear problem formulation, structured method descriptions, and high diagram quality (7 figures total, including teaser, architecture, two-stage inference, decoupled control, and qualitative comparison). However, some experimental tables lack specificity (e.g., the sample size of the test set is not specified).
  • Value: โญโญโญโญโ˜† Establishes a reproducible baseline framework for controllability in joint audio-video generation. Training lightweight bypasses on a frozen 19B model in 12 hours yields SOTA control performance, offering high engineering utility. Still, the limited control modalities and restriction to single-person setups highlight the remaining gap toward truly "arbitrary modality, arbitrary scene" general controllable generation.