EcoVideo: Entropy-Orchestrated Video Generation Paradigm in Cloud-Edge Dynamics¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/IF-LAB-PKU/EcoVideo
Area: Video Generation
Keywords: cloud-edge collaboration, attention entropy, keyframe selection, video frame interpolation, dynamic resource scheduling

TL;DR¶

EcoVideo replaces cloud-edge video generation through sequential large/small-model denoising with cloud generation of informative keyframes and edge reconstruction of intermediate frames, organizing computation through attention entropy and resource-aware scheduling to achieve 1.84× end-to-end speedup in the main Wan2.1 experiment and 2.91× under cloud compute contention.

Background & Motivation¶

The inference cost of video diffusion Transformers comes not only from model size but also from processing the entire video at every step. Even when adjacent frames change only slightly, the full sequence repeatedly undergoes spatiotemporal modeling and denoising, turning temporal redundancy into repeated computation. Cloud-edge methods such as HybridSD and EC-Diff divide generation along denoising steps: a large cloud model establishes the content, and a smaller edge model continues generation. This transfers cloud workload but does not reduce the number of frames being processed, and it leaves final texture and temporal-consistency refinement to the smaller model. When the two models produce frame-dependent differences in noise estimates for the same latent state, switching models can alter the sampling trajectory and introduce flicker, blur, or detail collapse.

Reducing cloud computation also does not automatically shorten the user's wait. Intermediate latents must cross the network, and the edge device must finish denoising; reduced bandwidth or edge contention can offset the cloud-side savings. The paper treats end-to-end latency as the sum of cloud, network, and edge costs, rather than optimizing a model's isolated forward-pass speed. Its central question is which video content must be generated directly by the large model and which content can be reconstructed from already generated images. This changes the unit of partitioning from denoising steps to frames, and changes the edge model's responsibility from continued denoising to conditional interpolation.

Frame-level partitioning still faces two difficulties: important frames are unknown before generation, and sparse generation must not break motion or texture between keyframes. EcoVideo estimates frame-level information density from early denoising self-attention distributions to locate temporally complex positions. The cloud preserves direct generation at these positions while retaining frozen context from unselected positions; the edge prioritizes intervals that are difficult to interpolate. Finally, the system selects the number of keyframes and interpolation refinement depth from current resource conditions rather than fixing a large-to-small-model switching point. Core Idea: let the large model generate informative temporal anchors, let the small model reuse those anchors to reconstruct continuous motion, and treat their workloads as configurations that adapt to available resources.

Method¶

Overall Architecture¶

Inputs are conditioning information such as text, a target video length, and random noise in the full video latent space; the output is a video completed to the target length. The system first performs a short full-sequence warm-up on the cloud to obtain stable frame-wise entropy from self-attention. Attention-Entropy Selection then supplies candidate keyframe sets for different budgets, and Resource-Aware Scheduling chooses the request's keyframe budget and edge refinement depth. Frozen-Context Denoising continues updating only keyframe latents while retaining other frames' warm-up latents as attention context. After the keyframes are decoded into pixels and transmitted to the edge, Multi-Cue Greedy Interpolation progressively fills the missing frames. Online profiling updates the unit costs used by scheduling, allowing the next request or scheduling window to use another configuration. The scheduler is part of the inference system, not another model that generates video content. The architecture is documented in Figures 2–3 and §3.1–3.3 of the paper, pages 6–10.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Condition and full noise sequence"] --> B["Attention-Entropy Selection"]
    B --> C["Resource-Aware Scheduling"]
    R["Bandwidth and cloud-edge compute"] --> C
    C -->|Keyframe budget| D["Frozen-Context Denoising"]
    C -->|Refinement depth| E["Multi-Cue Greedy Interpolation"]
    D -->|Decode and transmit keyframes| E
    E --> F["Complete video"]

Key Designs¶

1. Attention-Entropy Selection: estimate which temporal positions deserve direct generation from early states

Selection is not post-processing over an already generated full video, which would fail to save the main generation cost. EcoVideo reads self-attention weights during denoising, computes the entropy of each query token's attention distribution, and averages token entropies belonging to the same frame. A query whose attention spreads over more positions has higher entropy; the paper interprets this as a proxy for more complex dynamics and higher information density. This is the method's heuristic interpretation, not a proof that attention entropy exactly measures motion complexity. The frame-level mean in Equations (4)–(5) can be combined as:

\[ e_f^{(t)}=-\frac{1}{|\mathcal I_f|}\sum_{i\in\mathcal I_f}\sum_{j=1}^{N}A_{ij}^{(t)}\log A_{ij}^{(t)}. \]

Here, \(A_{ij}^{(t)}\) is an attention weight at denoising step \(t\), \(\mathcal I_f\) is the set of query tokens mapped to frame \(f\), and \(N\) is the total token count. Early attention changes with noise, so relying on a single step can destabilize selection; Equation (6) therefore combines warm-up frame entropies using an exponential moving average. The entropy estimate is fixed after warm-up, avoiding repeated full-sequence selection-signal computation during subsequent steps and limiting overhead. For a keyframe budget \(K\), the method retains the first and last frames, fills the remaining budget by entropy ranking, and restores temporal order. The endpoints anchor the full time span, while the other positions preferentially cover complex changes rather than following mechanically uniform sampling. The method section specifies the first 10% of steps as warm-up, whereas the implementation section specifies the first 5 steps; these agree for 50-step generation but not exactly for the 40-step Wan2.2 setting. Both source descriptions are retained here rather than asserting that an identical percentage is established for every backbone.

2. Resource-Aware Scheduling: jointly choose cloud keyframe budget and edge refinement depth

Entropy ranking determines which frames to select first; scheduling determines how many to select and how deeply the edge should refine. The scheduler monitors bandwidth \(B\), available cloud compute \(P_c\), and available edge compute \(P_e\), searching small candidate sets for keyframe count \(K\) and refinement depth \(D\). More keyframes generally provide denser temporal anchors but increase cloud computation and transmission; greater refinement depth assigns additional work to the edge. These controls interact because fewer keyframes mean more frames to reconstruct, potentially across more difficult intervals. Equations (14)–(15) divide total cost into fixed warm-up, keyframe generation, keyframe transmission, and intermediate-frame reconstruction. Cloud cost is approximated as proportional to keyframe count, network cost depends on per-keyframe payload and bandwidth, and edge cost depends on missing-frame count and the unit latency at the chosen depth. An exponential moving average updates these unit statistics online instead of assuming permanently fixed GPU or network performance.

The search maximizes quality utility minus weighted predicted latency, with the weight reflecting quality-latency preference or a service target. Small candidate sets permit enumeration without training a complex policy network. This does not mean that the scheduler computes actual VBench scores for every request; it uses a lightweight quality utility \(Q(K,D)\). The paper refers to an appendix for the utility's diminishing-return formulation, but the available cache contains only the main paper and references, not that appendix. Exact candidate sets, utility calibration, and some scheduling hyperparameters therefore cannot be fully reproduced from this material, and no guessed version is supplied here. Moreover, a utility-latency trade-off objective should not be interpreted as a strict mathematical guarantee of quality for every video.

3. Frozen-Context Denoising: update fewer frames while retaining global temporal cues

Warm-up produces a full video latent containing both selected keyframe positions and unselected positions. The most direct acceleration would crop out the keyframes and continue large-model denoising over that shorter sequence. However, this can discard the temporal layout established during full-sequence warm-up and change motion relationships or structure between keyframes. EcoVideo instead separates the latent along time into two groups: keyframe latents continue to update, while non-keyframe latents remain fixed at the end-of-warm-up state. The latter become frozen context tokens supplying additional keys and values in subsequent attention; noise predictions are retained only at keyframe positions. Conditioning information remains part of generation, and the large cloud model completes the remaining keyframe denoising steps before VAE decoding produces keyframe images.

This does not treat unselected frames as completed video, nor does the edge model continue repairing these noisy latents. Unselected frames serve as temporal context on the cloud, whereas edge reconstruction uses the cloud's pixel-space keyframe outputs. Equation (8) uses stop-gradient notation for frozen context, but the mechanism is primarily inference-time state reuse; this notation does not establish a new teacher-student training procedure. Keeping extra keys and values also retains global-context processing overhead, so the frame-cropping ratio is not itself a theoretical speedup factor. The savings principally come from no longer maintaining a full iterative update path for every frame while attempting to preserve relationships between keyframes. This distinguishes the method from generating a complete video and dropping frames afterward: omission occurs before the main denoising stage.

4. Multi-Cue Greedy Interpolation: prioritize refinement of intervals with the strongest changes

The edge model, EcoVFI-160M, modifies interval scoring based on EDEN's interpolation approach rather than running the second half of full video diffusion denoising. For each interval between adjacent keyframes, it examines motion, information density, structure, and texture differences. Motion is described by RAFT bidirectional optical-flow magnitude, density by the keyframe entropy gap, structure by DINO-small endpoint feature distance, and texture by pixel-domain appearance difference. These cues respectively address rapid displacement, information change, semantic structural change, and local appearance change, avoiding reliance on one signal alone. The four cues are normalized within each video to \([0,1]\), averaged with equal weights, and passed through a sigmoid as in Equation (13):

\[ g(I_a,I_b)=\sigma\!\left(\frac{\hat d_1+\hat d_2+\hat d_3+\hat d_4}{4}\right). \]

Here, \(I_a,I_b\) are the interval endpoints, and \(\hat d_1\) through \(\hat d_4\) denote normalized motion, density, structure, and texture cues, respectively. A higher score indicates an interval considered more susceptible to interpolation artifacts and therefore deserving earlier refinement; it is a ranking signal, not a calibrated failure probability. The system repeatedly selects the highest-scoring interval, synthesizes its midpoint with EcoVFI, splits the original interval into two, and updates their scores. This continues until the target video length is reached, progressively reusing keyframe information rather than filling every interval uniformly in one pass. The scheduler's \(D\) controls the edge refinement budget, but the main paper does not fully explain its mapping to concrete interpolation-model execution depth. It also does not detail how entropy cues are updated for newly inserted frames or how nonuniform midpoint insertion maps exactly onto the final fixed-frame-rate time grid; these interfaces need clarification for reproduction.

A Worked Example¶

Consider the paper's Wan2.1 setting with 81 frames and 50 steps: the system first warms up the full latent sequence instead of immediately skipping low-information frames. Attention entropies from the first 5 steps are averaged over time to form a temporal ranking for selection. The first and last frames are retained, the scheduler chooses a budget from current bandwidth and compute, and entropy ranking selects the remaining positions within it. The cloud continues denoising those positions while referencing the other positions frozen after warm-up through attention. Decoded keyframes are sent to the edge, where interval scoring prioritizes gaps with greater motion or structural change. Candidate intervals are compared again after each midpoint insertion until the sequence contains 81 frames. This example illustrates the source's data flow only; because the main paper does not give the request's actual \(K\) and \(D\), it is not an execution trace with a known fixed keyframe count.

Loss & Training¶

The main contribution is inference organization and scheduling rather than a new end-to-end video training loss. Attention-entropy analysis is explicitly training-free, and warm-up means early inference steps of an existing diffusion model, not additional pretraining. EMA stabilizes both frame entropy and runtime cost estimates; both uses concern runtime statistics. The cloud backbones are Wan2.1-14B, Wan2.2-A14B, and CogVideoX-5B, paired with EcoVFI-160M on the edge. The available full text does not provide separate EcoVFI training data, optimizer, or loss details, so it does not justify extending the training-free claim to every system component. Likewise, missing loss terms, EMA coefficients, and scheduling candidate values should not be filled with conventional defaults.

Key Experimental Results¶

Main Results¶

Evaluation uses the VBench prompt set, a single NVIDIA H200 with 141GB memory, and an NVIDIA Jetson Thor; the experimental setup states an average bandwidth of 20 Mbps. Wan2.1/Wan2.2 generate 5-second, 81-frame, 720P videos using 50/40 denoising steps, respectively; CogVideoX generates 6-second, 49-frame, 480P videos with 50 steps. Collaborative methods share the same cloud backbone; step-wise Wan baselines use Wan2.1-1.3B on the edge, and CogVideoX baselines use CogVideoX-2B. The following selection from Table 1, page 11, retains overall quality and end-to-end speedup over the corresponding cloud-only baseline; it does not compare absolute latency across backbones.

Cloud backbone	Method	VBench Overall ↑	Speedup ↑
Wan2.1-14B	Cloud-only	0.837	1.00×
Wan2.1-14B	HybridSD	0.677	0.89×
Wan2.1-14B	EC-Diff	0.683	1.27×
Wan2.1-14B	EcoVideo	0.846	1.84×
Wan2.2-A14B	Cloud-only	0.842	1.00×
Wan2.2-A14B	EcoVideo	0.830	1.59×
CogVideoX-5B	Cloud-only	0.819	1.00×
CogVideoX-5B	EcoVideo	0.775	2.03×

EcoVideo outperforms the step-wise collaborative baselines in the source's Table 1 across all three backbones, but it does not preserve cloud-only quality on every backbone. In particular, CogVideoX decreases from 0.819 for cloud-only to 0.775, demonstrating a quality cost accompanying acceleration. The next breakdown comes from Table 2, page 11, restricted to the same Wan2.1-14B, 81-frame, 720P setting; times are in seconds, and communication retains the source's MB unit.

Method	Cloud latency ↓	Edge latency ↓	Communication MB ↓	Total latency ↓
Cloud-only	1996.25	Not applicable	1.92	1996.35
HybridSD	1597.24	648.92	17.23	2247.02
EC-Diff	798.96	778.68	17.23	1578.50
EcoVideo	988.00	96.23	1.10	1084.29

EcoVideo does not have lower cloud latency than every collaborative baseline: EC-Diff uses only 798.96 seconds on the cloud, but its greater edge latency leads to a longer total runtime. This supports optimizing end-to-end cost rather than cloud cost alone. The communication term requires caution: HybridSD's total minus cloud and edge latency is only about 0.86 seconds, which does not directly reconcile with simple serial transfer of 17.23 MB at 20 Mbps. The table is transcribed as reported; no unit is silently corrected, and the source is not claimed to have clearly explained transfer overlap or this discrepancy.

Ablation Study¶

The following values come from Table 3, page 13, using Wan2.1-14B and the same cloud-only baseline to examine frame selection, interpolation, and warm-up duration. Full EcoVideo has a reported VBench of 0.845 here, slightly different from 0.846 in Table 1; both original values are retained rather than treated as interchangeable exact measurements.

Configuration	VBench ↑	Total latency (seconds) ↓	Speedup ↑
Full EcoVideo	0.845	1084.29	1.84×
Uniform keyframes without entropy selection	0.832	1049.73	1.90×
Without EcoVFI; keyframe-only video	0.835	988.82	2.02×
Naive interpolation with original EDEN	0.835	1019.33	1.96×
Warm-up 2%	0.834	938.17	2.13×
Warm-up 20%	0.842	1180.13	1.69×
Warm-up 30%	0.847	1336.45	1.49×

Uniform selection is slightly faster but lowers the overall score, suggesting that gains depend not only on reducing frame count but also on placing large-model computation at suitable positions. Removing EcoVFI changes temporal completion, so that row is not an equivalent substitute under completely identical output conditions. The difference between original EDEN interpolation and EcoVideo supports the contribution of interval scoring and refinement organization, but Table 3 does not ablate all four cues separately. Longer warm-up does not improve quality monotonically: 20% yields 0.842, below the default full configuration's 0.845; 30% reaches 0.847 at substantially greater latency.

Key Findings¶

Figure 5 and §4.3, pages 13–14, report 2.91× speedup with cloud compute at 50%, edge compute at 100%, and bandwidth set to 20; this is neither the default result nor a lower bound across resource conditions.
With edge compute at 50%, EcoVideo achieves 1.48×, versus 0.69× for HybridSD and 0.84× for EC-Diff, showing that edge workload can determine whether collaboration is faster at all.
At the reduced bandwidth labeled 10 in the figure, EcoVideo reaches 2.32×; this is another resource experiment and should not be combined with the default 1.84× as a bandwidth curve for one unchanged configuration.
Figures 4 and 6 provide qualitative comparisons of texture, structure, and motion continuity; no additional artifact rate or user-preference percentage is inferred from them here.

Highlights & Insights¶

The most important change is the edge model's responsibility: instead of imitating later large-model denoising, it reconstructs frames with explicit endpoint constraints. This reduces the risk of a large/small-model capability gap directly perturbing the same sampling trajectory.
Frozen non-keyframe context shows that updating less need not mean losing visibility entirely. It preserves global temporal cues, but its attention overhead must still be measured rather than estimating gains from frame ratios alone.
Entropy and interpolation difficulty operate at different stages: one allocates direct-generation budget, while the other allocates reconstruction budget. Their combination better matches nonuniform video changes than optimizing only a global sparsity ratio.
Online unit-cost estimation can transfer to other collaborative inference systems. The reusable principle is recomputing the workload partition as bottlenecks change, not transferring this paper's speedup factors to other hardware.

Limitations & Future Work¶

The authors explicitly identify extremely fast motion, heavy occlusion, and abrupt scene changes as interpolation failure cases; more keyframes or a stronger interpolator would consume some of the saved budget.
Attention entropy is only an information-density proxy: the paper does not prove that high-entropy frames are necessarily hardest to reconstruct or extensively report sensitivity to attention-layer or head selection.
The resource experiments cover representative settings rather than guaranteeing stability on long-running real network traces; heterogeneous edge devices and sustained online adaptation are identified as future directions.
The missing appendix, differing warm-up descriptions, main/ablation score discrepancy, and communication-time/unit concerns limit exact reproduction from the available full text alone.
Frozen context lacks a standalone removal ablation, and the four interpolation cues are not individually separated, so existing tables cannot establish every subdesign's independent contribution.
Reasonable next steps include feeding interpolation uncertainty back into keyframe insertion and combining the system with cloud caching or sparse attention; these are proposed directions, not already validated joint acceleration results.

Versus HybridSD, source reference [30]: it splits work by denoising steps and passes latents to the edge for further denoising; EcoVideo splits by temporal frames and reconstructs video from keyframes at the edge. The change concerns task boundaries, not just switching points.
Versus EC-Diff, source reference [29]: noise-gradient approximation and switching-point search reduce cloud inference while retaining step-wise collaboration. EcoVideo's latency breakdown shows why fewer cloud calls must be evaluated alongside edge execution cost.
Versus EDEN, source reference [36]: EDEN provides the large-motion interpolation foundation, while EcoVideo modifies interval scoring and refinement organization within edge reconstruction after sparse generation, rather than claiming to invent interpolation from scratch.
Relation to caching, sparse attention, and parallel inference: these primarily reduce the cost of cloud generation, whereas frame-level collaboration reduces the scope of directly generated content. The paper describes them as complementary but does not evaluate every combination comprehensively.
Sources are §3, pages 6–10; Tables 1–2, page 11; Table 3 and Figure 5, pages 13–14; and §6, page 15 of the paper PDF. The code link comes from the first page; its operational status was not checked during this offline reading.

Rating¶

Novelty: 4/5. The combination of frame-level generation partitioning, attention entropy, and resource scheduling defines a clear systems contribution, while interpolation and entropy signals build on prior work.
Experimental Thoroughness: 3/5. Three generation backbones, real cloud-edge hardware, and resource variations are covered, but fuller component isolation, scheduling detail, and long-running evidence are missing.
Writing Quality: 3/5. The main argument is clear, but numerical, unit, and implementation inconsistencies require further verification against code and the appendix.
Value: 4/5. Useful for studying end-to-end generation-service bottlenecks and task partitioning, but not a basis for a universal lossless video-acceleration claim.