Decoupling Complexity from Scale in Latent Diffusion Model

Conference: ECCV2026
Paper: ECCV Paper
Authors: Tianxiong Zhong, Xingye Tian, Xuebo Wang, Boyuan Jiang, Xin Tao, Pengfei Wan
Area: Image Generation / Video Generation
Keywords: latent diffusion model, scale decoupling, hierarchical latents, asynchronous denoising, progressive generation

TL;DR

DCS-LDM uses latent levels to control content detail and a decoder to control output scale, enabling progressive generation through causal hierarchies and asynchronous denoising; its single-level 512px configuration reduces the end-to-end latency in Table 4 from DiT's 6743ms to 1943ms, at the cost of higher decoder memory.

Background & Motivation

Conventional latent diffusion models first downsample images or videos by a fixed ratio and then denoise the resulting latents. This directly ties resolution and frame rate to the diffusion token count: more pixels or frames require more denoising tokens even when the content remains simple. Yet a high-resolution flat background need not require more representation capacity than dense low-resolution textures, and frame rate does not directly measure motion complexity. The paper therefore treats scale as an upper bound on representable information rather than a reliable proxy for actual content information.

Existing 1D tokenizers can already compress variable numbers of pixels into token sequences through learnable queries, but retain different restrictions. VFRTok supports variable frame rates with primarily fixed-length latents, whereas FlexTok supports variable latent lengths primarily for fixed-size images. DetailFlow studies a statistically sublinear relationship between scale and capacity, while this paper emphasizes that individual samples have different rate-distortion curves. Here, a rate-distortion curve relates representation budget to reconstruction error; the term does not imply that the authors introduce an exact theoretical measure of each sample's complexity.

Making the token count adjustable alone cannot ensure that additional tokens refine details without changing composition. The authors therefore constrain both information ordering in the latents and temporal ordering during denoising: structure comes first, and detail must depend on structure that is already relatively clear. Users can then establish content with a small budget, add detail level by level, and finally select a resolution or frame rate for decoding. Core Idea: delegate scale selection to the decoder and detail capacity to hierarchical latents, using one-way information flow and staggered denoising windows to make these controls independently adjustable.

Method

Overall Architecture

DCS-LDM combines the deterministic tokenizer DCS-Tok with the diffusion Transformer DCS-DiT. During training, DCS-Tok maps differently scaled versions of the same sample into a structurally consistent multilevel latent space and learns to reconstruct with different numbers of levels. DCS-DiT subsequently learns conditional generation of these latents; generation requires only noise, a class or text condition, and a selected level count, not an input image. The output scale is specified at decoding, when DCS-Tok converts the generated latents into an image or video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training sample<br/>Different scale variants"] --> B["Scale Alignment"]
    B --> C["Causal Hierarchical Representation"]
    C -.->|Latent training targets| D["Level-Asynchronous Denoising"]
    N["Noise + condition<br/>Selected level count"] --> D
    D --> E["Cache-Based Progressive Generation"]
    E --> F["DCS-Tok decoding<br/>Selected output scale"]
    F --> G["Image / video"]

The dashed edge denotes a training dependency, not a requirement to encode real samples during generation. Level count and scale are not entirely unconstrained controls: the latents retain spatial and temporal layouts, so changing aspect ratio or video duration changes the corresponding grid. The main decoupling concerns resolution or frame rate at a fixed layout and duration, versus the number of levels used to describe detail.

Key Designs

1. Scale Alignment: fix regional partitions rather than pixel patch sizes

Ordinary patchification fixes the number of pixels in each patch, increasing the patch count when an image becomes larger. DCS-Tok fixes the number of patches along the shorter side and the number of temporal patches per second, allowing their pixel dimensions to vary with input scale. For video height and width \(h,w\), frame rate \(f\), shorter-side patch count \(k\), and temporal patches per second \(k_t\), Equation (1) gives:

\[ p_{h,w}=\frac{\min(h,w)}{k},\qquad p_t=\frac{f}{k_t}. \]

Corresponding patches then cover the same spatial regions and time intervals across resolution and frame-rate variants of the same video. Images are treated as single-frame videos with temporal patch size \(p_t=1\); aspect ratio is preserved through the proportions of the spatial grid. Pixels within each patch are individually projected into tokens and aggregated by learnable latent queries, allowing the same local encoder to consume variable pixel counts. The decoder reverses this process, using learnable pixel queries to read local latents and produce the pixels required by the target scale.

Accepting variable inputs alone is insufficient, because the network could still bind its latents to the input scale. The authors use asymmetric scale training: the input and reconstruction target are randomly chosen scale variants of the same sample. This encourages latents to represent shared content across scales while leaving pixel expansion at different sampling densities to the decoder. It neither guarantees recovery of real details absent from the input nor makes decoding cost independent of the pixel count.

2. Causal Hierarchical Representation: let lower levels carry structure independently and higher levels add information

Each patch has several ordered latent tokens, with one token corresponding to one level for that patch. The local patchifier aggregates within patches, while the subsequent encoder and decoder handle cross-patch interaction, so the representation is not a single sequence that discards spatial layout entirely. The authors add a level dimension to the spatial and temporal grid; retaining more levels allocates more representation capacity to each patch. Causality here primarily concerns level dependencies and should not be conflated with conventional temporal causality in video.

Pixel-to-Latent Attention lets pixel information enter latents, while Latent-to-Pixel Attention lets latent information enter pixels. Pixel tokens can attend fully to one another, but exchange between pixels and latents is unidirectional to prevent higher levels from leaking information back to lower levels through a pixel branch. Level Causal Attention permits cross-patch interaction while allowing a level to read only its own and lower levels, never higher ones. This does not mean access is restricted to the immediately preceding level: the description of Figure 4 explicitly includes the same level and all earlier levels.

During training, each patch independently retains a random number of levels from \(1\) to \(n\), with remaining tail levels disabled by attention masks. Reconstructing frequently without higher levels pressures lower levels to preserve essential structure and higher levels to learn supplementary detail. At inference with batch size \(1\), unused tokens can be discarded rather than computing the full masked sequence. Deterministic decoding and level causality jointly support fixing a draft before adding detail, but visual stability also depends on the following denoising schedule.

3. Level-Asynchronous Denoising: establish higher-SNR structure before generating detail

If all levels begin together at the same noise level, detail levels initially read uncertain lower-level structure and may form textures misaligned with the final composition. DCS-DiT assigns staggered time windows to the levels so that lower levels accumulate structural information before higher levels denoise. Global time is mapped to a local time for each level and supplied as conditioning; velocity is predicted only within that level's active window. DCS-DiT uses a level-causal architecture resembling the tokenizer, with AdaLN and optional cross-attention for time and condition injection.

The factor \(\epsilon\) controls the proportion of global time occupied by a single level's window, with \(1/n\leq\epsilon\leq1\). At \(\epsilon=1\), levels are synchronous; smaller values impose stronger temporal separation, and the default \(\epsilon=1/n\) gives the strongest separation. This changes relative denoising progress across levels, rather than training another super-resolution model or assigning a different output resolution to each level. The authors also normalize local time and predicted velocity accordingly, but Equation (2) has displaced characters in the cache, so this note does not present a guessed reconstruction as the original equation.

Table 1 directly tests this design: adding levels worsens gFID under the synchronous setting, while asynchronous settings improve as levels are added. The point is therefore not simply that more tokens are better, but that additional capacity needs compatible information dependencies and noise ordering. Level asynchrony trades parallelism against generation quality; level count alone cannot determine real speed on every device.

4. Cache-Based Progressive Generation: reuse established structure instead of regenerating all levels

Level causality prevents higher levels from changing lower-level computations, allowing completed lower-level representations to remain fixed. Later queries still attend to lower-level keys and values, which the model caches to avoid repeated projection and lower-branch computation when adding detail. The same principle applies to level attention in DCS-DiT and the DCS-Tok decoder, supporting an interaction that displays a draft before further refinement. Once a level exits its active window, its velocity is set to zero and its representation stays unchanged, providing cacheable conditioning for subsequent levels.

The authors' claim of unchanged total computation means that their causal decomposition and caching avoid repeating lower-level computations already required by joint execution. It does not mean that additional levels are free or that arbitrary preview workflows have identical end-to-end latency. If a user changes the text condition and resamples the structure, the old structural cache should not be assumed reusable without qualification. Actual efficiency also depends on decoding, batch size, memory access, and selected levels; Table 4 exposes these system-level differences.

A Worked Example

Consider a square image with unchanged aspect ratio and the implementation's \(k=16\). At \(256\times256\), patches are \(16\times16\); at \(512\times512\), patches are \(32\times32\), with a \(16\times16\) spatial grid in both cases. A single level therefore contains \(256\) latent tokens, and \(4\) levels contain \(1024\) tokens; these counts follow from multiplying the grid size by the level count. A user can generate one level to inspect composition, then add levels for texture; changing decoding resolution does not itself require enlarging the diffusion latent grid. Figure 8 demonstrates decoding fixed \(4\)-level image latents from \(256\) to \(512\) pixels, and single-level video latents at \(12/24/48\) fps. However, changing the aspect ratio to \(1:2\) yields a \(16\times32\) grid, so scale independence must not be generalized to an identical token count for every output specification.

Loss & Training

DCS-Tok uses deterministic reconstruction rather than an additional stochastic flow decoder for pixels. Equation (7) combines L1 reconstruction, perceptual loss, and adaptively weighted adversarial loss, transcribed below with explicit weight notation:

\[ \mathcal{L}_{\mathrm{Tok}}=\mathcal{L}_{\mathrm{recon}}+\lambda_1\mathcal{L}_{\mathrm{percept}}+\lambda_2\lambda'\mathcal{L}_{\mathrm{adv}},\qquad \lambda_1=1,\quad \lambda_2=0.2. \]

Here, \(\lambda'\) is an adaptive weight; this note does not supply calculation details absent from the available cache. Training proceeds through low-resolution image initialization, spatial asymmetric training, and spatial-temporal asymmetric training; DCS-DiT follows LightningDiT's objective and sampling method. DCS-Tok's local patchifier and depatchifier each use a \(3\)-block tiny ViT, its encoder and decoder use \(12\)-block ViT-B backbones, and the main DCS-DiT configuration uses a \(28\)-block ViT-XL. Tokenizer training uses ImageNet, Koala-36M, and LAVIB, covering video scales from \(128\)p to \(720\)p and \(12\) to \(60\) fps. The qualitative text-to-image model uses ImageNet and SA-1B with captions generated by Qwen-VL 2.5; quantitative class-conditioned image and video tasks use ImageNet and UCF101, respectively. Class-conditioned configurations remove cross-attention blocks; results with and without classifier-free guidance (CFG) are reported separately and must not be conflated.

Key Experimental Results

Main Results

The following extracts Table 2 on page 13: ImageNet \(256\times256\) class-conditioned image generation. Lower gFID and higher IS are better; DCS-LDM uses only its first level in this system comparison, not an optimized multilevel configuration.

Method	gFID without CFG	IS without CFG	gFID with CFG	IS with CFG
DiT	9.62	121.5	2.77	278.2
SiT	8.61	131.7	2.06	270.3
REPA	5.90	157.8	1.42	305.7
LightningDiT	2.17	205.6	1.35	295.3
DCS-LDM, 1 level	7.26	119.0	2.44	277.3
DCS-LDM, 1 level + REPA	4.26	147.1	1.39	311.6

Adding REPA reduces guided gFID from 2.44 to 1.39, showing that representation alignment complements the new latent design, but the results do not surpass LightningDiT on every metric. Table 3 on page 13 reports guided gFVD of 77.10 for single-level DCS-LDM, 71.34 for VFRTok, and 85.22 for Cosmos on UCF101 \(256\times256\), 24 fps class-conditioned video generation. These video results support competitiveness rather than best performance; lower gFVD is also better.

Ablation Study

The following extracts Table 1 on page 13: the asynchronous-factor ablation with DCS-DiT/B, measured by gFID without CFG, where lower is better. This table uses a B-scale model, so its absolute scores cannot be treated as results from the same training setup as the preceding system comparison.

Asynchronous factor	1 level	2 levels	3 levels	4 levels
\(\epsilon=1\)	40.42	48.26	62.23	71.25
\(\epsilon=1/2\)	42.31	40.30	38.48	38.03
\(\epsilon=1/4\)	42.50	40.23	38.39	37.80

The synchronous configuration is slightly better at one level but worsens with additional levels; \(\epsilon=1/4\) improves from 42.50 at one level to 37.80 at four. The ablation therefore supports asynchronous structural guidance for multilevel generation, not superiority of asynchrony at every level count.

The next table extracts Table 4 on page 14: XL diffusion models with 50 NFE; latency and peak memory use batch=4, and memory retains the source unit G. Only latency and decoder peak memory are included, without mixing in throughput measured at batch=16; latency columns can differ slightly because of source-table rounding.

Output / method	Decode latency ms	Diffusion latency ms	Total latency ms	Decode peak memory G
256px / DiT	30.42	1572	1602	0.67
256px / DCS-LDM, 1 level	87.04	1573	1660	3.54
512px / DiT	131.5	6612	6743	2.18
512px / DCS-LDM, 1 level	370.4	1573	1943	14.9
512px / DCS-LDM, 2 levels	391.1	3280	3671	15.1
512px / DCS-LDM, 4 levels	414.2	6972	7386	15.4

Key Findings

• At one level, diffusion latency is 1573ms for both 256px and 512px, but decoding rises from 87.04ms to 370.4ms; decoupling does not make total system cost constant.
• At 512px with 4 levels, total latency reaches 7386ms, exceeding DiT's 6743ms, so computational savings depend on the required detail budget.
• Section 4.7 on page 15 reports validation PSNR of 31.59 for uniform allocation and 32.42 for patch-entropy allocation at an average reconstruction budget of 4 tokens per patch; this is reconstruction evidence, not a solved generation-time allocation policy.

Highlights & Insights

• Separating output specification from modeling capacity avoids automatically assigning full high-resolution diffusion cost to simple high-resolution content. The reusable interface is local querying over variable pixel counts, not merely shortening one global sequence.
• Level causality jointly supports truncatable representations, structural stability, and cache reuse. Asynchronous windows further ensure that detail levels can read relatively reliable structure, illustrating the need to co-design architecture constraints and training dynamics.
• Establishing composition before spending on detail suits interactive creation better than committing to the largest budget immediately. The value lies in stopping and resuming computation, not in every configuration outperforming existing models.

Limitations & Future Work

• The authors acknowledge the extra memory from per-pixel tokenization and suggest patch-parallel decoding across GPUs or grouped decoding with smaller patch batches; Table 4's 14.9G versus 2.18G shows a substantial cost.
• Complexity control still largely depends on choosing a level count; entropy allocation is tested only for reconstruction, and choosing optimal levels for unknown generated content remains open.
• This note's assessment: quantitative results focus on ImageNet and UCF101, while text-to-image evaluation is largely qualitative, so the evidence does not establish thoroughly validated open-domain high-quality text-to-video generation.
• Arbitrary scale describes an architectural interface, while experiments cover reported ranges; extrapolation to extreme resolutions, frame rates, durations, and aspect ratios needs separate testing.
• The available cache contains the main paper and references but not the repeatedly cited Appendix A; Equation (2), some symbols, and metric arrows suffer extraction corruption. This note retains readable values, states directions from metric definitions, and does not infer appendix details for attention masks, positional embeddings, or the complete time equation.

Related Work & Insights

• vs VFRTok: Both use cross-scale alignment to loosen the link between frame rate and latent length; DCS-LDM adds hierarchical capacity control, but its guided UCF101 gFVD remains worse than VFRTok's, so flexibility is not a free quality advantage.
• vs FlexTok / DetailFlow: Variable lengths and coarse-to-fine generation are not unique to this work; the distinction is jointly addressing scale-independent decoding, sample complexity differences, and stable hierarchical refinement.
• vs REPA / LightningDiT: Semantic alignment and training optimization improve generation quality, whereas this paper changes representation and computation organization; the REPA addition in Table 2 indicates complementary directions.
• Research implication: A learned budget predictor could use the prompt and a lower-level draft to choose additional levels or patches, but should be evaluated for content preservation, decoder memory, and total latency; this is a proposed direction, not an implemented component of the paper.

Rating

• Novelty: 4/5. The combination of scale independence, level causality, and asynchronous diffusion is clear, with identifiable predecessors for individual techniques.
• Experimental Thoroughness: 4/5. Reconstruction, generation, asynchronous ablations, and end-to-end cost are covered, but open-domain video and automatic budgeting need further validation.
• Writing Quality: 4/5. The problem and architecture align clearly, while corrupted cached equations and the missing appendix limit independent verification of implementation details.
• Value: 4/5. The method provides a useful interface for interactive, budget-adjustable generation, with decoder memory requiring careful deployment trade-offs.