Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Generation¶

Conference: ECCV 2026
arXiv: 2605.31603
Project: https://jiazheng-xing.github.io/nexus-lumos-home/
Code: None
Area: Video Generation / Diffusion Models
Keywords: Unified Video Generation, Frequency Bridging, Homogeneous Latent Space, Reasoning-Driven Generation, Training-Efficient

TL;DR¶

Lumos-Nexus proposes a training-efficient, two-stage unified video generation framework. During training, only a lightweight generator is fine-tuned to absorb VLM semantics. During inference, the generation responsibility is progressively transferred from the lightweight generator to a large-capacity pretrained generator via Unified Progressive Frequency Bridging (UPFB). This achieves coarse-to-fine video synthesis within a shared homogeneous latent space, balancing reasoning-driven semantic accuracy and high-fidelity visual quality.

Background & Motivation¶

Background: Unified models integrate multimodal understanding and generation into a single framework, where the understanding module provides structured semantic priors for the generator to interpret complex instructions and produce logical, intent-aligned outputs. In the video domain, this unification is particularly crucial: videos are sequences of events unfolding over time, requiring temporal consistency, causal progression, and coherent motion, which naturally demands stronger reasoning capabilities than image generation. Existing video unified models fall into two categories: joint-attention-based schemes (which share self-attention for long-context interactions, offering high scalability but extremely high training costs) and connector-based schemes (which inject understanding representations into the generator's condition injection space through explicit connectors, decoupling the two modules).

Limitations of Prior Work: Although connector-based video unified models are structurally more decoupled, in practice they still require expensive fine-tuning of large-scale diffusion generators (e.g., Wan2.1-14B) to align the understanding outputs with the generator inputs. This makes simultaneously achieving tight semantic alignment and high visual fidelity exceedingly difficult in practice.

Key Challenge: The high-fidelity detail synthesis capability of large generators is highly desirable, but incorporating them into an end-to-end understanding-generation unified training loop is computationally prohibitive. Conversely, while light generators are computationally affordable to train, their generation quality suffers from a "short-board effect"—the precise semantic priors from the understanding side are bottlenecked by the lightweight generator's limited capacity, failing to be effectively executed in the generated video frames.

Goal: To introduce the high-fidelity synthesis capabilities of large generators into the unified video generation process within the connector paradigm, without increasing training costs.

Key Insight: The authors observe that fine-tuning in unified models does not alter the latent representation space of the diffusion backbone. Consequently, small and large generators belonging to the same model family naturally share a homogeneous latent space (sharing the same VAE). This establishes the foundation for separating training and inference: during training, a small generator learns to "absorb and encode" high-level semantic priors from the understanding module; during inference, the small generator acts as a "semantic kickstarter", converting understanding-derived semantic representations into structural priors that can be seamlessly inherited by the large generator, which then contributes its pretrained high-fidelity synthesis capabilities to further reinforce the execution of reasoning-driven semantics.

Core Idea: To replace "end-to-end training of large generators" with "progressive frequency-domain bridging in a homogeneous latent space," decoupling semantic learning and high-fidelity synthesis into separate training and inference stages, thereby achieving large-model-level generation quality at an extremely low training cost.

Method¶

Overall Architecture¶

The core mechanism of Lumos-Nexus is "train small, infer large": the light generator learns to receive and encode semantic knowledge from the understanding module during the training stage, and during the inference stage, the generation task is progressively transferred to the large-capacity pretrained generator via UPFB. The entire framework operates in two stages, taking textual instructions as input and outputting high-fidelity reasoning-driven videos.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Textual Instruction"] --> B["Training Stage<br/>Understanding Module + Connector<br/>+ Light Generator Fine-tuning"]
    B --> C["Inference Stage: UPFB Progressive Bridging"]
    C --> D["Temporal Semantic Gating<br/>Cosine Schedule Weight wt"]
    D --> E["Frequency Decomposition<br/>Gaussian Low-pass Filtering Separates LF/HF"]
    E --> F["Dual-frequency Asymmetric Fusion<br/>LF leans small / HF leans large"]
    F --> G["RMS Energy Alignment<br/>Pre-alignment + Post-rebalancing"]
    G --> H["High-fidelity Reasoning-driven Video"]

Training Stage: Using Omni-Video (a connector-based video unified model based on Wan2.1-T2V-1.3B) as the baseline, only the connector and the lightweight generator \(G^S\) (Wan2.1-1.3B) are fine-tuned to learn to transform the VLM-aligned embedding \(c^S\) output by the understanding module into structured generation signals. The large generator \(G^L\) (Wan2.1-14B) is completely omitted from the training loop and kept frozen.

Inference Stage: UPFB dynamically bridges the velocity field predictions of the two generators at each denoising step. \(G^S\) and \(G^L\) predict CFG velocities \(v_t^S\) and \(v_t^L\) for the same intermediate latent \(z_t\), respectively. These predictions then undergo four steps—temporal gating, frequency decomposition, asymmetric fusion, and RMS alignment—yielding a fused velocity field \(v_t\) for flow-matching latent updates. This process is training-free and can be seamlessly plugged into existing connector-based unified model inference pipelines.

Key Designs¶

1. Two-stage Training-Inference Separation: Semantic Learning via Small Generator, Detail Refinement via Large Generator

To address the high cost of end-to-end training for large generators, the core architectural decision of Lumos-Nexus is to completely separate semantic alignment and high-fidelity synthesis into the training and inference stages. The training phase only optimizes Wan2.1-1.3B (requiring only 2.25 s/it and 26.3 GB of VRAM), while Wan2.1-14B is only introduced during inference. The key premise is that both share a homogeneous latent space (from the same Wan family VAE, with an MMD of only 0.523), allowing the semantic representations learned by the small model to be directly understood and inherited by the large model. The elegance of this design lies in completely avoiding large model training (Wan2.1-14B fine-tuning requires 7.21 s/it and 72.8 GB of VRAM), yielding a substantial quality gain (+0.43 VBench and +1.05 VR-Bench) for a modest inference latency increase of roughly 1.2\(\times\) (from 35.1 to 40.5 seconds/step). Ablation experiments show that directly upgrading Omni-Video to the 14B backbone with LoRA fine-tuning (Omni-Video*) actually degrades VBench to 81.73 and VR-Bench to 70.63, demonstrating that simply scaling up the backbone cannot resolve structural cross-modal alignment incompatibilities.

2. Unified Progressive Frequency Bridging (UPFB): Joint Time-Frequency Domain Fusion for Coarse-to-Fine Semantic-Texture Transition

UPFB is the technical core of the proposed method, designed to resolve the issues of semantic instability, structural duplication, and texture conflicts that occur when directly blending outputs from two generators. It incorporates four sub-mechanisms:

Temporal Semantic Gating: A cosine schedule function \(w_t = \frac{1}{2}(1 + \cos(\pi(1 - \tau_t)^{\gamma_w}))\) controls the transition of dominance between the two generators, where \(\tau_t = (T-1-t)/(T-1)\) and \(\gamma_w\) controls the transition sharpness (set to 0.3). In early steps, \(w_t\) is larger to favor semantic construction by the small generator; in later steps, \(w_t\) is smaller to progressively hand over detail refinement to the large generator.

Time-Varying Frequency Decomposition: Gaussian low-pass filtering \(G_{\sigma_t}(\cdot)\) is applied to the predicted velocity fields of the two generators to isolate the low-frequency structural components \(LF(v) = G_{\sigma_t}(v)\) and high-frequency texture components \(HF(v) = v - LF(v)\). The bandwidth parameter \(\sigma_t = \sigma_{\min} + (\sigma_{\max} - \sigma_{\min}) w_t\) decays over time (\(\sigma_{\min}=0.35\), \(\sigma_{\max}=0.70\)). A larger bandwidth in early stages suppresses high-frequency noise to stabilize layout, while a smaller bandwidth in later stages progressively restores fine details.

Dual-Frequency Asymmetric Fusion: Low-frequency components are fused progressively using the temporal weight \(LF_t = w_t LF_t^S + (1-w_t) LF_t^L\), allowing the large model to contribute to the semantic structure. High-frequency components utilize an additional attenuation factor \(\gamma_{hf}=0.7\) to suppress high-frequency noise from the small model: \(HF_t = w_t \gamma_{hf} HF_t^S + (1-w_t) HF_t^L\). The final fused velocity field is \(v_t = LF_t + HF_t\).

RMS Energy Alignment: Before fusion, the amplitude of the large model's velocity field is aligned to that of the small model: \(\mathbf{v}_t^L \leftarrow \mathbf{v}_t^L \cdot \frac{\text{RMS}(\mathbf{v}_t^S)}{\text{RMS}(\mathbf{v}_t^L)}\). After fusion, a global rebalancing based on the average RMS of both models is applied: \(\mathbf{v}_t \leftarrow \mathbf{v}_t \cdot \frac{\frac{1}{2}(\text{RMS}(\mathbf{v}_t^S) + \text{RMS}(\mathbf{v}_t^L))}{\text{RMS}(\mathbf{v}_t)}\), preventing overexposure and unstable activations caused by amplitude mismatches.

3. VR-Bench: The First Benchmark for Reasoning-Driven Video Generation

Prior video generation evaluations primarily focused on visual fidelity and temporal coherence, largely ignoring the reasoning dimension—namely, whether a model can infer, plan, and act according to semantic intent. VR-Bench fills this gap by designing 216 evaluation test cases covering 8 reasoning dimensions across 3 high-level categories: (1) High-level Physical World Reasoning—Dynamic Reference Frame (DRF), Energy Transfer Visualization (ETV), Material Memory Consistency (MMC); (2) High-level Commonsense Reasoning—Conceptual Action Reasoning (CAR), Cultural Commonsense Reasoning (CCR), Preventative Causal Reasoning (PCR); (3) Embodied Physical Reasoning—Biological Behavior Reasoning (BBR), Concurrent Action Coordination (CAC). The evaluation employs Qwen3-VL-30B-A3B-Instruct to perform an 8-question, three-level diagnostic (L1 Basic Perception \(\rightarrow\) L2 Intermediate Relational Reasoning \(\rightarrow\) L3 Advanced Causal/Semantic Reasoning) for each dimension, aggregated into a 0-1 score and validated by human evaluation for ranking consistency (Kendall \(\tau = 0.73\)).

Loss & Training¶

The training phase adheres to the standard unified model training objectives of Omni-Video (fine-tuning only the connector and the lightweight generator, while keeping the large generator frozen and entirely uninvolved in training). The UPFB mechanism during inference is completely training-free; all hyperparameters are set during inference: \(\gamma_w = 0.3\), \(\sigma_{\min}=0.35\), \(\sigma_{\max}=0.70\), \(\gamma_{hf}=0.7\), CFG scale = 5, 50 sampling steps, 480p resolution, 81 frames (5 seconds @ 16fps).

Key Experimental Results¶

Main Results¶

VBench T2V Evaluation (Tab. 1):

Model	Total	Quality	Semantic	Type
CogVideoX1.5-5B	82.17	82.78	79.76	Generation Only
HunyuanVideo	83.43	85.07	76.88	Generation Only
CausVid	83.88	85.21	78.57	Generation Only
Wan2.1-14B	83.69	85.59	76.11	Generation Only
Omni-Video	83.82	85.00	79.10	Unified Model
Show-o2	81.34	82.10	78.31	Unified Model
Lumos-Nexus	84.12	85.03	80.52	Unified Model (Ours)

Lumos-Nexus achieves the highest total score of 84.12, which represents a 0.30 improvement over the baseline Omni-Video and a 0.43 improvement over the generation-only Wan2.1-14B model. The semantic score increases from 79.10 (Omni-Video) to 80.52, validating that UPFB effectively mitigates the bottleneck where understanding-side semantics are precise but generation-side expressiveness is restricted.

VR-Bench Reasoning Evaluation (Tab. 2, Wan2.1 class):

Model	Total	HL-Phys.	HL-Comm.	Emb.-Phys.
Veo 3.1 (Closed-source)	93.95	94.25	95.37	91.37
Kling 2.6 (Closed-source)	91.13	92.96	88.71	92.01
CogVideoX1.5-5B	66.27	71.92	67.03	56.67
HunyuanVideo	75.38	76.79	69.73	81.39
Wan2.1-14B	78.23	80.34	75.96	78.46
Omni-Video	72.78	72.39	72.79	73.33
Lumos-Nexus	79.28	79.49	77.57	81.54
Lumos-Nexus* (Wan2.2)	81.90	83.02	79.43	83.93

Notably, Omni-Video underperforms compared to the generation-only Wan2.1-1.3B on VR-Bench (72.78 vs. 77.00). This occurs because, despite accurate semantic priors from the understanding module, the small generator fails to execute them correctly in the video frames. Lumos-Nexus reverses this deficit by bridging to the large generator. Furthermore, replacing the large generator with Wan2.2-T2V-A14B (Lumos-Nexus*) boosts VR-Bench to 81.90, demonstrating the scalability of the framework.

Ablation Study¶

Temporal Gating Sharpness \(\gamma_w\) (Tab. 3):

\(\gamma_w\)	VBench Total	VBench Semantic	VR-Bench Total
0.2	84.08	80.04	79.09
0.3	84.12	80.52	79.28
0.4	84.02	79.63	76.49
0.5	84.05	79.29	75.10

\(\gamma_w = 0.3\) yields the best performance. A higher value (0.5) causes the small model to dominate for too long, inheriting too many base structural behaviors from the small model at the expense of fine details; a lower value (0.2) transitions too quickly, causing the large model's fine details to run ahead of a stable semantic foundation.

Bandwidth Range \((\sigma_{\min}, \sigma_{\max})\) (Tab. 4):

\((\sigma_{\min}, \sigma_{\max})\)	VBench Total	Quality	Semantic
(0.05, 0.10)	83.99	84.90	80.34
(0.35, 0.70)	84.12	85.03	80.52
(1.00, 2.00)	83.56	84.86	78.36

A moderate bandwidth range performs best. If too narrow, filtering is barely active, and the conflict between the two models persists; if too wide, low-frequency structural components are overly suppressed, degrading the semantic foundation and significantly dropping semantic scores.

RMS Alignment (Tab. 5):

Configuration	VBench Total	Quality	Semantic
w/o RMS	84.07	84.98	80.43
w/ RMS	84.12	85.03	80.52

RMS alignment brings a minor but consistent improvement (+0.05 Total), primarily stabilizing the magnitude of the fused velocity field and preventing overexposure or unstable activations.

Direct Add vs. UPFB (Tab. 7, Appendix):

Bridging Strategy	VBench Total	VR-Bench Total
Direct Add	83.28	77.33
UPFB	84.12	79.28

Direct addition leads to severe artifacts, such as duplicated subjects and structural inconsistencies (e.g., dual-headed birds). UPFB's time-frequency domain control is substantially superior (+0.84 VBench, +1.95 VR-Bench).

Key Findings¶

UPFB's frequency decomposition is a core contribution: Disabling frequency-domain controls (degrading to Direct Add) drops VBench by 0.84 points, indicating that temporal weighting alone is insufficient to reconcile conflicts between two heterogeneous generators.
Reasoning capabilities are highly sensitive to \(\gamma_w\): Increasing \(\gamma_w\) from 0.3 to 0.5 drops VR-Bench by 4.18 points, a far more severe impact than on VBench (-0.07). This suggests that reasoning-driven semantic execution heavily depends on the handoff timing; transitioning too quickly or too slowly disrupts the reasoning chain.
A homogeneous latent space is a strict prerequisite: Replacing the large generator with a model from a different family with a different VAE (e.g., HunyuanVideo) increases the MMD from 0.523 to 1.031, resulting in severe degradation and blurring. The framework is verified on both Wan2.1 and Wan2.2 families, demonstrating its applicability within the same VAE family.
Model capacity vs. bridging mechanism: Directly upgrading Omni-Video's generator to 14B and fine-tuning with LoRA (Omni-Video*) yields a VR-Bench score of only 70.63 (vs. Lumos-Nexus's 79.28). This indicates that Wan2.1-14B cannot natively accept VLM tokens and that lightweight LoRA adaptation is insufficient to bridge this structural mismatch. The bridging mechanism itself remains the key.

Highlights & Insights¶

Combine "Train Small, Infer Large" with Frequency Bridging: The integration of large-scale generators is shifted from "how to train" to "how to bridge." Using a homogeneous latent space and frequency-domain decomposition alleviates direct blending conflicts. This approach exploits the "coarse-to-fine" progression of diffusion denoising—where early steps dictate global layout and later steps refine textures—mapping naturally to the small model's semantic dominance and the large model's detail dominance.
Training-Free Aspect of UPFB: The entire bridging process occurs exclusively during inference without requiring any fine-tuning of the large model. Consequently, any model pair sharing a homogeneous latent space can benefit at virtually zero training cost. Preliminary tests on CogVideoX-2B/5B demonstrate valid video generation without retraining.
Tiered Diagnosis in VR-Bench: Replacing raw total scores with an "8-question, 3-tier" structure allows the benchmark not only to rank models but also to diagnose precisely where their reasoning breaks down (from basic perception to causal reasoning). This design is highly transferable to other tasks requiring fine-grained evaluation, such as embodied AI or physics simulation.
The RMS Energy Alignment Trick: Aligning amplitudes in the velocity field, rather than in the pixel or latent space, prevents exposure anomalies and unstable denoising trajectories. This technique is applicable to any scenario involving the fusion of multiple diffusion model predictions (e.g., multi-conditional generation, model merging).

Limitations & Future Work¶

Constraint of Homogeneous Latent Space: The current method requires that the small and large generators share the exact same VAE latent space, restricting the scope of usable generator pairs. Although the authors suggest that a latent space adaptation step on the small generator could accommodate heterogeneous models, this remains experimentally unverified.
Limited Scope of VR-Bench: The authors acknowledge that 8 dimensions cannot exhaustively cover open-world reasoning, particularly long-horizon causal chains. The sample size of 216 cases also requires balancing between breadth of coverage and statistical significance.
Inference Overhead: While avoiding large-model training, inference requires running two generators simultaneously (resulting in approximately 1.2\(\times\) the latency of the large model alone), which is challenging for real-time applications. Under the Wan2.1 configuration, steps take around 40 seconds, making deployment demanding on consumer-grade hardware.
Tested Only on DiT Architectures: All experiments were conducted on the Wan family DiT architecture. The performance of UPFB on U-Net architectures or other flow-matching variants remains unexplored.
Lack of Direct Evaluation on "Reasoning Chains": VR-Bench measures the consistency between the generated video and the reasoning intent, but does not directly evaluate the quality or interpretability of the internal reasoning chain within the understanding module. Future work could introduce explicit evaluations of intermediate reasoning steps.

vs. Omni-Video: Omni-Video is the direct baseline of Lumos-Nexus, sharing the connector-based video unified model architecture but with the generator fixed to an end-to-end fine-tuned Wan2.1-1.3B. Lumos-Nexus introduces the high-fidelity capabilities of Wan2.1-14B during inference without additional training cost, leading to across-the-board improvements on VBench and VR-Bench.
vs. Show-o / Show-o2 / UniVideo: These joint-attention-based video unified models share self-attention to achieve long-context understanding-generation interactions, which is more scalable but computationally prohibitive to train. Lumos-Nexus follows the connector pathway, accepting the training cost constraint and resolving generation quality deficiencies via inference-side innovation. The debate between these two paradigms remains unresolved.
vs. Multi-Model Ensemble Methods (e.g., Show-1's Pixel-Latent Hybrid): Show-1 also blends two generators but does so at the pixel or latent space level. Lumos-Nexus innovates by shifting fusion to the frequency domain and velocity field, exploiting the time-varying nature of the diffusion process for more granular control. Frequency-domain decomposition serves as a novel bridging mechanism.
Inspiration: The paradigm of "training a small model to capture semantics and bridging a large model during inference to enhance quality" can be extended to other unified model domains—such as text-to-image (T2I) unified models or multimodal conversational generation—provided the models share a latent space. With the growing availability of diverse model families (e.g., Wan's 1.3B/14B, CogVideoX's 2B/5B), this paradigm's applicability is bound to expand.

Rating¶

Novelty: ⭐⭐⭐⭐ Combining "train small, infer large" with frequency bridging is highly intuitive, and the training-free nature of UPFB distinguishes it from standard ensembles or knowledge distillation. However, the foundational blocks (low-pass filtering and cosine gating) are standard; the novelty lies in their integration rather than individual components.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ The experiments are exceptionally comprehensive: evaluating across VBench, VR-Bench, and GenEval, ablating all hyperparameters (gating sharpness \(\gamma_w\), bandwidth, RMS, and comparing against direct adding), scaling to 14B backbones for capacity controls, verifying adaptability on Wan2.2, testing robustness with alternative judge models, validating human alignment, quantifying latent alignment via MMD, and analyzing training/inference efficiency.
Writing Quality: ⭐⭐⭐⭐ The methodology is clear, with rigorous and complete mathematical formulations. The ablation analyses are deep, particularly in distinguishing capacity contributions from bridging mechanisms. However, the descriptions of VR-Bench's 8 dimensions are somewhat abstract and would benefit from concrete prompt examples to illustrate the evaluation granularity.
Value: ⭐⭐⭐⭐ Provides a highly practical, low-cost solution in the highly competitive field of unified video generation. VR-Bench fills a crucial gap in evaluating reasoning-driven video generation and has the potential to become a standard community benchmark. The high transferability of the method (to any homogeneous latent space model pair) amplifies its practical impact.