DiTex4D: Direct Text-Driven 4D Generation with Structured Latent Diffusion

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/3dv-casia/DiTex4D
Area: 3D Vision
Keywords: 4D Generation, 3D Animation, Structured Latent Diffusion, Mixed-4D RoPE, Correlated Noise Injection

TL;DR

DiTex4D builds a native text-driven 4D generation framework upon the 3D foundation model TRELLIS, eliminating intermediate video generation via mixed-4D RoPE and spatial-keyed correlated noise injection to natively support both 4D generation from scratch and mesh animation.

Background & Motivation

Directly generating dynamic 4D digital assets—represented as 3D Gaussian Splatting (3DGS) sequences—from pure textual descriptions is a longstanding goal in augmented reality, game development, and robotics simulation. However, text prompts lack explicit geometric shapes, camera viewpoints, and motion constraints, leaving an immense representational gap between compact linguistic instructions and high-degree-of-freedom spatiotemporal 3D dynamics. Prior text-to-4D methodologies have predominantly relied on two main paradigms: score distillation sampling (SDS) based per-scene optimization using video models, which suffers from multi-hour compute costs, oversaturated colors, and multi-face Janus artifacts; or two-stage text-to-video-to-4D pipelines, which suffer from catastrophic error accumulation between stages and impose severe constraints on video dynamics such as requiring zero occlusion and slow motion.

Another emerging branch attempts to bypass video generators by directly generating text-conditioned vertex trajectories to animate input meshes (e.g., AnimateAnyMesh). Nonetheless, because such methods fail to leverage pre-trained large-scale 3D foundation models, they remain severely bottlenecked by the extreme scarcity of dynamic 4D datasets, frequently failing to synthesize plausible large-amplitude motions on out-of-distribution meshes. Furthermore, unordered point cloud latent representations (such as VecSet) introduce non-deterministic sampling across frames, provoking severe temporal jitter in the latent space.

This paper's core angle of attack is that voxel-based Structured Latents (SLat) from 3D foundation models (e.g., TRELLIS) provide deterministic spatial coordinates without stochastic sampling and inherently decouple geometry and texture. Core idea: directly inject text conditions into a pre-trained structured 4D latent diffusion backbone by inflating 3D attention with mixed-4D RoPE, coupling it with spatial-keyed correlated noise injection and a masked multi-view branch to seamlessly unify generation from scratch and static mesh animation.

Method

Overall Architecture

DiTex4D operates through a coarse-to-fine two-stage diffusion process built on TRELLIS structured representations: first, the Sparse Structure Flow (SS Flow) predicts activated 3D voxel coordinate sequences; second, the Structured Latent Flow (SLat Flow) predicts continuous aggregated multi-view DINOv2 feature vectors bound to these active voxels. The framework natively supports two tasks: in 4D generation from scratch, text prompt embeddings directly guide the conditional flow matching fields; in 3D animation from static meshes, the input mesh is replicated across time, encoded into conditioning latents, and concatenated with binary masks and multi-view CLIP embeddings. Finally, decoded 3DGS sequences are temporally upsampled using a masked 4D interpolation model to ensure smooth, high-frame-rate dynamics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Text Prompt / Static 3D Mesh"] --> B["Multi-Condition Branch & Mask Construction<br/>Mesh Voxelization + Multi-View CLIP Encoding"]
    B --> C["4D Attention Inflation & Mixed-4D RoPE<br/>Spatial 3D APE + Decomposed 4D Relative RoPE"]
    C --> D["Spatial-Keyed Correlated Noise Injection<br/>Continuous SS Noise + Discrete Voxel-Keyed SLat Noise"]
    D --> E["Two-Stage Flow Matching Generation<br/>4D Sparse Voxel Sequences → 4D SLat Sequences"]
    E --> F["3DGS Decoding & 4D Spatiotemporal Interpolation<br/>Gaussian Sequence Reconstruction + Masked Interpolation"]

Key Designs

1. 3D Attention Inflation with Mixed-4D RoPE: Preserving Structural Geometry under Unconstrained Motion Pre-trained 3D foundational models contain rich per-frame spatial priors but lack inter-frame temporal interactions, leading to static degenerations when conditioned solely on text. DiTex4D inflates spatial self-attention into spatiotemporal full attention, enabling unrestrained interactions across all spatial voxels and time steps. Because text prompts lack visual motion tracking cues, existing 1D temporal RoPE techniques from video-to-4D methods lead to severe geometric drift. To counteract this, DiTex4D retains the pre-trained spatial 3D Absolute Positional Embeddings (APE) while applying independent 1D RoPE across coordinates \((x, y, z, t)\) concatenated along the feature dimension to form a 4D RoPE. This mixed-4D RoPE strategy preserves canonical 3D geometric structures via absolute coordinates while capturing flexible, extrapolated temporal dynamics through relative rotations.

2. Spatial-Keyed Correlated Noise Injection: Suppressing Inter-Frame Flickering in Sparse Voxel Diffusion When sampling diffusion trajectories independently per frame, abstract text prompts fail to enforce smooth latent trajectories, causing prominent temporal jitter and visual flickering. DiTex4D adopts a correlated noise formulation balancing a global shared noise vector \(\epsilon_{\text{shared}}\) and frame-independent noise \(\epsilon_{\text{ind}}^i\) (with sharing ratio \(\alpha=1\)). While continuous SS latents admit direct linear superposition, SLat latents reside on dynamically shifting active voxel subsets where coordinates vary over time. To resolve this structural mismatch, DiTex4D introduces a spatial-keyed alignment mechanism:

\[ \epsilon_{\text{SLat}}^i = \epsilon_{\text{shared}}^{k_i} + \epsilon_{\text{ind}}^i, \quad \text{where } k_i = (x_i, y_i, z_i) \]

Activated voxels possessing the same spatial coordinates across different temporal frames share an identical base noise vector \(\epsilon_{\text{shared}}^{k_i}\), while individual noise components preserve dynamic flexibility. This eliminates abrupt noise discrepancies caused by coordinate shifts.

3. Masked Diffusion Model with Multi-View CLIP Global Conditioning: Anchoring Static Mesh Identity For text-guided 3D mesh animation, the model must faithfully preserve the geometry and texture of the initial mesh while generating faithful text-guided motion. DiTex4D duplicates the single-frame input mesh \(T\) times along the temporal axis, processes it through the structural VAE into conditioning latents \(z_c\), and introduces a binary spatiotemporal mask \(M \in \{0, 1\}^{B \times T \times L \times 1}\) (1 for the preserved initial mesh frame, 0 for frames to be synthesized). The noisy latent \(z_t\), condition \(z_c\), and mask \(M\) are concatenated channel-wise and passed into the diffusion backbone via a zero-initialized linear projection. Furthermore, to eliminate identity drift across unseen dynamic views, the mesh is rendered into four orthogonal views (\(0^\circ, 90^\circ, 180^\circ, 270^\circ\)), processed by a CLIP image encoder, and projected into a global context vector injected via Decoupled Cross-Attention. This enforces a clear division of labor: multi-view images ground visual identity while text prompts guide dynamic trajectories.

Loss & Training

The framework minimizes the Conditional Flow Matching (CFM) objective:

\[ \mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \| v_\theta(x_t, t) - (\epsilon - x_0) \|^2 \]

The models adopt a progressive curriculum, training first on 8-frame sequences before scaling to 16 frames. Both the base generation model and the animation model are trained using AdamW at a learning rate of \(1 \times 10^{-4}\) under FP16 mixed precision on 8 GPUs with an aggregate batch size of 32. The 4D interpolation model masks frames \(I\) and \(I+2\) to predict the intermediate frame \(I+1\), achieving smooth \(T \to 2T - 1\) frame upsampling with minimal fine-tuning iterations.

Key Experimental Results

Main Results

The framework is trained on 20,000 animated 3D sequences curated from Objaverse and ObjaverseXL, filtered for salient motion amplitude and captioned with Qwen2.5-VL. Evaluations are conducted across 32 out-of-domain objects covering diverse categories (humans, animals, rigid objects) rendered from four orthogonal camera angles (\(0^\circ, 90^\circ, 180^\circ, 270^\circ\)). Evaluated metrics include VBench indicators: Motion Smoothness (M.Sm), Aesthetic Quality (Aest.Q), Dynamic Degree (Dy.Deg), Image-to-Video fidelity (I2V), and CLIP text alignment.

Main comparison results from Table 1 of the original paper:

Paradigm	Method	Scratch 4D: CLIP↑	Scratch 4D: M.Sm↑	Scratch 4D: Aest.Q↑	3D Animation: I2V↑	3D Animation: M.Sm↑	3D Animation: Aest.Q↑
SDS Optimization	Dream-in-4D [75]	24.17	0.973	0.416	–	–	–
SDS Optimization	Animate3D [24]	23.96	0.986	0.509	0.877	0.987	0.514
Two-Stage Feedforward	L4GM [45]	24.97	0.991	0.482	0.848	0.992	0.498
Video-to-4D Diffusion	GVFDiffusion [69]	25.03	0.989	0.539	0.902	0.989	0.535
Trajectory Diffusion	AnimateAnyMesh [58]	25.42	0.994	0.527	0.948	0.997	0.546
Direct 4D Diffusion	DiTex4D (Ours)	25.51	0.995	0.559	0.946	0.997	0.562

Ablation Study

Ablation experiments conducted on 8-frame sequences isolate the individual impact of mixed-4D RoPE, correlated noise injection, 4D interpolation, and multi-view CLIP conditioning (Table 2 of the original paper):

Configuration	Task Scope	Primary Metric (CLIP / I2V)↑	Motion Smoothness (M.Sm)↑	Aesthetic Quality (Aest.Q)↑	Dynamic Degree (Dy.Deg)↑	Design Role & Impact
Ours (Full Base Model)	4D Scratch	25.54	0.996	0.559	0.281	Combines mixed-4D RoPE with spatial-keyed correlated noise
w/o mixed-4D RoPE (1D RoPE + 3D APE)	4D Scratch	25.48	0.994	0.553	0.312	Lacks 4D spatial relative rotation, suffering structural drift under large motion
w/o mixed-4D RoPE (only 4D RoPE)	4D Scratch	23.64	0.964	0.469	0.531	Stripping 3D APE discards pre-trained spatial anchors; geometry collapses
w/o Correlated Noise Injection	4D Scratch	25.17	0.977	0.544	0.406	Independent per-frame sampling causes severe inter-frame flickering
w/o 4D interpolation	4D Scratch	25.52	0.986	0.556	0.281	Without frame doubling, motion remains coarse across large intervals
Ours (Full Animation Model)	3D Animation	0.948	0.997	0.563	0.250	Masked conditioning combined with multi-view CLIP embeddings
w/o Multi-View Condition	3D Animation	0.918	0.991	0.557	0.377	Relying solely on text fails to anchor unobserved viewpoints of the mesh

Key Findings

• 3D APE is indispensable for RoPE extension: Replacing mixed-4D RoPE with pure 4D RoPE causes CLIP text alignment to plunge from 25.54 to 23.64 and Aesthetic Quality to decline by ~0.09. Pre-trained 3D coordinate priors cannot be easily re-learned from limited 4D data using only relative rotations.
• Correlated noise eliminates temporal jitter: Spatial-keyed noise injection boosts Motion Smoothness on the base model from 0.977 to 0.996, effectively suppressing spurious voxel flickering across consecutive timesteps.
• Decoupled multi-view conditioning anchors identity: Adding multi-view CLIP conditioning raises initial-frame fidelity (I2V) from 0.918 to 0.948, showing that multi-view image prompts provide crucial appearance guidance that text prompts cannot convey alone.

Highlights & Insights

• Bypassing the 2D video intermediary: By generating directly within the 3D structured latent space, DiTex4D completely avoids occlusion fragility and multi-view geometric discrepancies inherent to 2D video generators.
• Spatial-keyed noise for unstructured dynamic voxels: Rather than forcing matrix broadcasting across variable-length active voxel sets, using continuous 3D spatial coordinates \(k_i=(x_i, y_i, z_i)\) as hash keys to anchor shared noise offers an elegant solution for sparse 4D grids.
• A unified diffusion-and-masking pipeline: Unifying unconstrained 4D asset creation and static mesh animation inside a single model architecture via channel concatenation and masked conditioning showcases the versatility of structured latent diffusion.

Limitations & Future Work

• Author-admitted limitations: The model can struggle to preserve fine topological structures (such as thin hair strands or porous geometries) during extreme, non-rigid deformations.
• Independent observations: Because the framework is anchored on TRELLIS structured latents, any geometric artifacts present in the pre-trained static representation propagate directly into the 4D dynamics. Furthermore, the framework currently targets isolated objects rather than full 4D scenes with background interactions.
• Future directions: Integrating physics-based constraints or reinforcement learning dynamics, alongside adaptive octree voxel scaling, could enable higher-resolution dynamic simulation.

Related Work & Insights

• vs Dream-in-4D / Animate3D: Dream-in-4D relies on costly per-scene SDS optimization taking hours per asset; DiTex4D achieves feed-forward inference in seconds while naturally preserving multi-view consistency.
• vs L4GM / GVFDiffusion: These video-to-4D pipelines are heavily constrained by video generation artifacts and camera tracking failures; DiTex4D operates directly on 4D latents without requiring intermediate video generation.
• vs AnimateAnyMesh: AnimateAnyMesh trains trajectory diffusion from scratch without large-scale 3D foundation model priors, resulting in negligible motion on out-of-distribution meshes; DiTex4D inherits powerful open-world geometry priors from TRELLIS.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Pioneering direct text-driven 4D generation on structured 3D latents with mixed-4D RoPE and spatial-keyed noise)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive qualitative and quantitative evaluations spanning two generation pathways, rigorous ablations, and user studies)
• Writing Quality: ⭐⭐⭐⭐⭐ (Clean narrative structure, clear technical problem formulation, and cohesive design presentation)
• Value: ⭐⭐⭐⭐☆ (Establishes a solid feed-forward baseline and release codebase for text-to-4D and 3D mesh animation)

Related Papers

• [ECCV 2026] SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion
• [ECCV 2026] One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control
• [ICML 2026] RelaxFlow: Text-Driven Amodal 3D Generation
• [ECCV 2026] PixGS: Pixel-Space Diffusion for Direct 3D Gaussian Splat Generation
• [ECCV 2026] Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation