BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration

Conference: ICLR 2026 arXiv: 2510.00438 Code: https://lzy-dot.github.io/BindWeave/ (project page) Area: Video Generation / Subject Consistency Keywords: Subject-to-Video, MLLM Condition Injection, DiT, Multi-Reference Images, Cross-Modal Reasoning

TL;DR

BindWeave replaces conventional shallow fusion mechanisms with a Multimodal Large Language Model (MLLM) to parse complex multi-subject textual instructions, generating subject-aware hidden states as conditioning signals for a DiT. Combined with CLIP semantic features and VAE fine-grained appearance features, it achieves high-fidelity, subject-consistent video generation.

Background & Motivation

Background: DiT-based video generation models (Wan, HunyuanVideo, etc.) can produce high-quality long videos, yet precise control over subject identity and appearance remains insufficient.

Limitations of Prior Work: - Existing S2V methods (Phantom, VACE, etc.) follow a "separate-then-fuse" shallow processing paradigm — independent encoders extract image and text features respectively, which are then combined via concatenation or cross-attention for late fusion. - This paradigm handles simple appearance-preservation instructions adequately, but fails to establish deep cross-modal semantic associations when prompts involve complex spatial relationships, temporal logic, or multi-subject interactions. - This results in identity confusion, action misalignment, and attribute blending.

Key Challenge: Complex semantics in text prompts (e.g., "Person A hands a gift to Person B") require deep cross-modal reasoning to be correctly interpreted, which shallow fusion cannot achieve.

Goal: Establish deep semantic associations between textual commands and visual entities to accurately resolve the roles, attributes, and interactions of multiple subjects.

Key Insight: Leverage a pretrained MLLM as an "intelligent instruction parser" to perform deep cross-modal reasoning prior to generation.

Core Idea: Replace shallow encoder fusion with the deep reasoning capability of an MLLM to generate conditioning signals that jointly encode subject identity and interaction relationships for guiding the DiT.

Method

Overall Architecture

The input consists of a text prompt \(\mathcal{T}\) and \(K\) reference images \(\{I_k\}\). The MLLM parses the multimodal input to produce hidden states, which are projected and concatenated with T5 text features to form \(c_{\text{joint}}\), then injected into the DiT via cross-attention. Simultaneously, CLIP features \(c_{\text{clip}}\) provide semantic anchoring, and VAE features \(c_{\text{vae}}\) supply pixel-level detail via channel concatenation.

Key Designs

1. Intelligent Instruction Planning:

   • Function: Qwen2.5-VL-7B processes an interleaved text-image sequence and generates hidden states encoding subject roles, attributes, and interactions.
   • Mechanism: A unified multimodal sequence \(\mathcal{X} = [\mathcal{T}, \langle\text{img}\rangle_1, ..., \langle\text{img}\rangle_K]\) is constructed; the MLLM binds textual commands to corresponding visual entities through deep reasoning, producing \(H_{\text{mllm}} = \text{MLLM}(\mathcal{X}, \mathcal{I})\), which is then projected into the DiT feature space via a lightweight two-layer MLP+GELU connector.
   • Design Motivation: The multimodal reasoning capability of MLLMs far exceeds the shallow feature extraction of independent encoders such as CLIP or T5, enabling comprehension of complex logic such as "who does what, to whom, and where."

2. Collectively Conditioned Video Diffusion:

   • Function: Integrates three levels of conditioning signals within the DiT.
   • High-level relational reasoning: \(c_{\text{joint}} = \text{Concat}(c_{\text{mllm}}, c_{\text{text}})\) injected via cross-attention.
   • Semantic identity guidance: \(c_{\text{clip}} = \mathcal{E}_{\text{CLIP}}(\{I_{\text{ref}}^i\})\) injected via a separate cross-attention stream.
   • Low-level appearance details: \(c_{\text{vae}} = \mathcal{E}_{\text{VAE}}(\{I_{\text{ref}}^i\})\) injected at the input layer via channel concatenation.
   • Attention layer output: \(H_{\text{out}} = H_{\text{vid}} + \text{Attn}(Q, K_{\text{joint}}, V_{\text{joint}}) + \text{Attn}(Q, K_{\text{clip}}, V_{\text{clip}})\)

3. Adaptive Multi-Reference Conditioning Strategy:

   • Function: Extends the temporal axis with \(K\) dedicated slots to accommodate VAE features from reference images.
   • Mechanism: \(K\) zero-padded positions are appended to the temporal dimension of the video latent; VAE features of reference images and binary masks are placed at these positions, then channel-concatenated before PatchEmbed.
   • Design Motivation: Reference images are not video frames and should not be directly mixed with them; dedicated temporal slots combined with binary masks emphasize subject regions explicitly.

Loss & Training

• Rectified Flow + MSE velocity field prediction loss: \(\mathcal{L} = \|u_\Theta(z_t, t, c_{\text{joint}}, c_{\text{clip}}, c_{\text{vae}}) - v_t\|^2\)
• 1 million high-quality video-text pairs curated from OpenS2V-5M
• Two-stage training: 1,000 steps on core data for stability + 5,000 steps on the full dataset for expansion
• 512 xPUs, batch size 512, lr=5e-6, AdamW
• Random rotation/scaling augmentation on reference images to prevent copy-paste artifacts
• Inference: 50 steps, CFG scale \(\omega=5\)

Key Experimental Results

Main Results — OpenS2V-Eval Benchmark (180 prompts, 7 scenario categories)

Method	NexusScore↑	NaturalScore↑	GmeScore↑	Total↑
Phantom	Low	Medium	Medium	Medium
VACE	Medium	Low (unnatural motion)	Medium	Medium
SkyReels-A2	High	Low (distortion)	Medium	Medium-Low
Kling-1.6	Medium	High	High	High
BindWeave	Highest	Competitive	Competitive	Highest

• BindWeave achieves a significant lead over all open-source and commercial models on NexusScore, the primary subject consistency metric.
• It remains competitive on FaceSim, Aesthetics, MotionSmoothness, and other metrics.

Ablation Study

Configuration	Effect
Full BindWeave	Best overall
w/o MLLM (replaced with simple encoder)	Identity confusion and incorrect interaction logic in multi-subject scenes
w/o CLIP features	Degraded subject identity preservation
w/o VAE detail injection	Loss of appearance detail

Key Findings

• MLLM deep reasoning is the core advantage: Benefits are most pronounced in complex multi-subject interaction scenarios, where shallow fusion methods degrade severely.
• Three-level conditioning signals are complementary: MLLM provides semantic reasoning, CLIP preserves identity, and VAE preserves fine-grained details — removing any level causes performance degradation.
• Commercial models excel in aesthetics but lag in subject consistency: Kling and Vidu produce visually appealing outputs but frequently exhibit commonsense violations (e.g., distorted limbs).

Highlights & Insights

• Paradigm shift: MLLM as instruction parser — replacing "encode separately then fuse" with "deep understanding via MLLM before generation" is more principled in theory and more effective in practice.
• Design philosophy of three-level conditioning: High-level reasoning (MLLM) → mid-level semantics (CLIP) → low-level pixels (VAE), with a clear hierarchical structure where each level serves a distinct purpose.
• Lightweight connector strategy is effective: Aligning MLLM and DiT feature spaces with only a two-layer MLP suggests that MLLM hidden states already carry sufficiently structured information.

Limitations & Future Work

• The MLLM (Qwen2.5-VL-7B) introduces additional inference computational overhead.
• Training data is limited to 1 million samples; scaling up the dataset may further improve generalization.
• Occlusion and recovery of subjects within generated videos have not been addressed.
• The number of reference images is restricted (1–4); scenarios with a very large number of subjects remain unvalidated.

Related Work & Insights

• vs. Phantom: Processes text and image in dual independent branches before injection into the DiT, constituting shallow fusion; BindWeave performs end-to-end deep reasoning via MLLM.
• vs. VACE: Adopts a unified input format injected through residual blocks, but still lacks cross-modal reasoning capability.
• vs. per-subject optimization (CustomVideo, etc.): Requires separate fine-tuning for each subject; BindWeave is end-to-end and requires no fine-tuning.

Rating

• Novelty: ⭐⭐⭐⭐ Using MLLM as an instruction parser to replace shallow fusion represents a meaningful architectural contribution.
• Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive evaluation on the OpenS2V standard benchmark with comparisons against both open-source and commercial methods.
• Writing Quality: ⭐⭐⭐⭐ Architecture is described clearly with well-motivated design choices.
• Value: ⭐⭐⭐⭐⭐ Addresses a critical bottleneck in multi-subject video generation with high practical applicability.

Related Papers

• [CVPR 2026] UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
• [AAAI 2026] MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation
• [ICLR 2026] JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
• [ICLR 2026] MoSA: Motion-Coherent Human Video Generation via Structure-Appearance Decoupling
• [CVPR 2026] Gloria: Consistent Character Video Generation via Content Anchors