FreeSwim: Revisiting Sliding-Window Attention Mechanisms for Training-Free Ultra-High-Resolution Video Generation¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Video Generation
Keywords: training-free resolution extrapolation, inward sliding-window attention, cross-attention override, coarse-to-fine generation, feature caching

TL;DR¶

FreeSwim divides high-resolution refinement in a pretrained video DiT between an inward-window branch for detail and a full-attention branch for global semantics, using cross-attention override and cross-step caching to achieve a 73.7% 1080P VBench overall score with Wan2.1-1.3B without training new parameters.

Background & Motivation¶

Video Diffusion Transformers (DiTs) jointly process spatial and temporal tokens, so enlarging a frame increases not only pixel count but also the number of relationships modeled by attention. Wan2.1-1.3B is trained only at 832 × 480; asking it to generate 1920 × 1088 video introduces unfamiliar receptive fields and positional relationships as well as greater computational demand. Figure 3 shows structural disorder under direct high-resolution inference, and resolution-aware attention scaling alone does not adequately restore the layout. Positional adjustments, sampling changes, and low-resolution guidance that work for images must also handle three-dimensional dependencies in video, so their extrapolation behavior cannot simply be assumed to transfer.

Generating a low-resolution video before upsampling, adding noise, and denoising preserves the broad composition but still does not guarantee natural local textures. The authors observe that high-resolution full attention preserves layout with weak detail, whereas local windows improve detail but tend to generate similar content repeatedly across regions. Thus, more context is not necessarily better for unadapted resolution extrapolation: global interaction expands the out-of-distribution input, while excessive localization interrupts coordination between objects. Boundary tokens introduce another issue: truncating their windows creates a different attention distribution even when central regions retain the training scale.

FreeSwim therefore reorganizes the interaction range and semantic transfer points of an existing model at inference time instead of training a new high-resolution generator. Its evidence comes from attention-map comparisons and generation ablations, not a theoretical proof that training-scale receptive fields are optimal. Global information remains necessary, but the branch generating texture need not perform global self-attention itself at every layer. Core Idea: preserve the pretrained spatial receptive field in local self-attention, then inject the global branch's text cross-attention outputs into the local branch, replacing unrestricted interaction expansion with complementary roles.

Method¶

Overall Architecture¶

The inputs are a text prompt, an existing video DiT, and a target resolution; the output is a video refined through high-resolution denoising. The first stage uses the model's native text-to-video pipeline without modification to establish the main objects, composition, and motion. The second stage upsamples in pixel space, encodes the result with the VAE, and adds Gaussian noise before refinement, rather than enlarging noise and generating from scratch. Noise permits detail regeneration while the coarse video constrains structure; its strength controls the trade-off between retaining the initial result and regenerating content.

The high-resolution stage contains three designs: inward sliding-window attention, cross-attention override, and full-branch feature caching. The window branch generates the final video's details, while the full branch provides semantic guidance; both receive the same current-step latent input. The full branch is neither an independently generated video later blended into the output nor a low-resolution teacher: it sees the global context of the current high-resolution latents. Caching determines when that branch is recomputed without changing the local branch's responsibility for step-by-step denoising.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text and pretrained DiT"] --> B["Native video generation<br/>Upsampling, VAE encoding, noise"]
    B --> C["Inward Sliding-Window Attention"]
    B --> D["Full-attention branch"]
    C --> E["Cross-Attention Override"]
    D --> E
    E --> F["Full-Branch Feature Caching"]
    F -->|Reuse semantic features at subsequent steps| E
    E --> G["Iterative denoising and decoding<br/>High-resolution video"]

Key Designs¶

1. Inward Sliding-Window Attention: retain the complete training-scale window at boundaries

The authors first distinguish matching total token count from matching the scale of each dimension. Reducing frame count to offset a larger spatial resolution controls sequence length but does not restore the original spatiotemporal relationships; Figure 3(e) shows limited improvement. The window therefore does not discard arbitrary key/value positions but assigns each query a local neighborhood corresponding to the native spatial resolution. The model still processes the temporal dimension: the method neither treats frames as independent images nor achieves higher resolution by shortening the output video. All self-attention layers in the high-resolution window branch replace their original full 3D attention with this local mechanism. Each position consequently generates detail from spatial context closer to its training experience instead of suddenly attending to an entire oversized frame.

If a centered window crosses the left boundary, the whole window shifts right; crossing the right boundary shifts it left, with the same treatment vertically. The essential operation is shifting rather than clipping, so edge and corner queries retain a full neighborhood of key/value tokens. Near a boundary, the query is no longer centered in its window, which is the geometric consequence of preserving window size. Unlike padding with zeros and treating them as real content, inward shifting uses valid context from inside the video. The implementation compiles sparse masks with PyTorch FlexAttention; Table 3 reports 32 min 29 s for ordinary windows and 32 min 30 s for inward windows. The extracted mask conditions and formatting in Equation (1) are corrupted, so this account follows Section 3.2's description without reconstructing the authors' exact indexing formula.

2. Cross-Attention Override: inject global semantics into the detail-preserving branch

A fixed local receptive field makes textures more natural, but distant windows do not know what has already been generated elsewhere and may independently instantiate the same textual object. The full-attention branch has complementary behavior: it coordinates global layout but loses local detail under resolution extrapolation. At every denoising step requiring the full branch, FreeSwim duplicates the current latents along the batch dimension and applies full attention and inward-window self-attention in the respective branches. Interaction occurs at latent-to-text cross-attention outputs rather than by averaging final RGB frames or directly merging the two self-attention matrices. Although both branches read the same prompt, their cross-attention queries have been updated by their respective self-attention operations, so the full-branch outputs interpret text through global context. Global information can therefore guide the local branch at the semantic entry point without removing its detail-producing local self-attention.

Equation (2) interpolates the two cross-attention outputs as follows; superscripts identify the branches and \(\lambda\) controls full-branch guidance strength.

\[ O_{\mathrm{Cross}}^{\mathrm{Window}} \leftarrow \lambda O_{\mathrm{Cross}}^{\mathrm{Full}} + (1-\lambda)O_{\mathrm{Cross}}^{\mathrm{Window}}. \]

The default is \(\lambda=1\), meaning a complete override rather than a small global correction. Only the cross-attention output is replaced, not the window branch's entire hidden state, so the window branch does not collapse into the full branch. In Figure 7, intermediate strengths resolve repetition in one scene but not all illustrated scenes, motivating full override as the robust default. The supported conclusion is reduced repetition with preserved detail, not a guarantee that arbitrary prompts will never produce duplicates; the figure itself shows scene-dependent behavior. The \(\lambda=0\) row in Table 4 disagrees numerically with the window-only configurations in Table 3, so its rows should not all be treated as an unambiguous single-variable causal comparison.

3. Full-Branch Feature Caching: recompute semantic guidance less frequently

The dual path adds expensive high-resolution full attention, potentially offsetting the savings from local sparsity if executed at every step. FreeSwim caches output features from the full branch's cross-attention layers, rather than only text embeddings or the final video latents. These guidance features are refreshed every \(P\) steps; intervening steps still execute the window branch and override its cross-attention outputs using the cache. Reuse relies on the empirical assumption that semantic guidance changes slowly between nearby denoising steps; it is not exactly equivalent to recomputing every step. Increasing \(P\) reduces full-branch calls but may make guidance stale, so quality preservation must be established experimentally. The paper defaults to \(P=2\) and also evaluates \(P=5\) and \(P=8\), without presenting the largest interval as uniformly optimal.

For classifier-free guidance (CFG), the authors additionally remove full-attention guidance from the unconditional branch, retaining only inward-window computation there. The text-conditional branch still receives global semantic override; removing CFG-Full does not remove CFG or skip unconditional prediction. Caching therefore reduces work in the auxiliary global branch without modifying pretrained parameters or learning a cache predictor. Storing features adds memory demand, and the authors use CPU offloading to control VRAM, so caching does not inherently save both time and memory. The time expressions and symbol descriptions in Equation (3) are corrupted in the extraction; this note describes the invocation logic and takes timings directly from Table 3 instead. Reusing structural guidance while retaining stepwise detail updates is what allows quality and speed to be adjusted separately.

A Worked Example¶

Consider Wan2.1-1.3B generating 1920 × 1088 video from its native 832 × 480 resolution; this is an explanatory walkthrough of the published procedure, not an additional experiment. The model first generates a native-resolution video from the prompt, establishing the main objects and their motion. Each frame is upsampled in pixel space, encoded by the VAE, and noised to produce the common refinement-stage latent input. At a refresh step, the full branch sees the complete current video context, while the window branch reads a native-scale spatial region around each query. For a query at the upper-left corner, the window shifts down and right so that two boundaries do not shrink its valid neighborhood. Each cross-attention layer uses \(\lambda=1\) to pass full-branch semantic outputs to the window branch, after which subsequent network operations produce the step's prediction.

With \(P=2\), the next non-refresh step can reuse the existing layerwise guidance without rerunning the full branch. The window branch nevertheless reads the new current latents, so neither texture nor motion is frozen. At the following refresh step, semantic guidance is recomputed from the updated latents and replaces the old cached features. The unconditional CFG path uses only windows, without constructing another full-attention guidance path. Further denoising and VAE decoding yield the high-resolution video; the procedure does not stitch independently generated local video clips together. Distinguishing periodically reused guidance from generation states updated every step is essential to understanding that this acceleration does not simply skip sampling steps.

Loss & Training¶

The method adds no training objective, dataset adaptation, or parameter updates; all changes operate within the pretrained model's inference pipeline. High-resolution refinement borrows SDEdit's upsample, noise, and denoise strategy, which is a generation procedure rather than a training loss. The available main-text extraction does not specify every noise strength, sampling-step setting, and hardware combination needed for complete reproduction, so generic defaults should not be inserted. Explicit core settings are full override with \(\lambda=1\), a default refresh interval of \(P=2\), and inward windows implemented with FlexAttention. Window scale should follow the backbone's native training resolution rather than mechanically applying the Wan2.1-1.3B setting to every model.

Key Experimental Results¶

Main Results¶

Section 4.1 randomly selects 60 VBench prompts with 5 random seeds each, yielding 300 videos per method at each evaluated resolution. The actual 1080P dimensions are 1920 × 1088; the paper also evaluates 3380 × 1920 and calls this 3K, a convention retained here. The following excerpt from Table 1, page 9, covers Wan2.1-1.3B; all metrics are percentages and higher is better. Imaging quality captures degradations such as noise and blur, aesthetic quality concerns composition and color, and overall consistency measures semantic and style alignment rather than being synonymous with the overall score.

Method / Config	Aesthetic Quality	Imaging Quality	Overall Consistency	Overall Score
Real-ESRGAN	55.9	63.9	24.8	72.3
Upscale-A-Video	55.6	62.9	22.9	71.1
I-Max	59.6	61.1	23.0	71.5
HiFlow	54.8	54.4	24.7	69.4
CineScale, without LoRA	45.9	40.9	16.8	65.5
CineScale, with LoRA	56.7	60.7	23.5	71.5
FreeSwim, without cache	57.9	63.4	25.1	72.7
FreeSwim, cache \(P=2\)	61.2	65.3	24.9	73.7
FreeSwim, cache \(P=5\)	56.2	62.6	25.2	72.3
FreeSwim, cache \(P=8\)	54.3	61.0	23.6	71.7

FreeSwim with \(P=2\) exceeds CineScale with high-resolution LoRA by 2.2 percentage points overall, but this does not imply leading every component metric. For example, \(P=5\) has overall consistency of 25.2 versus 24.9 for \(P=2\), despite a lower overall score, so selecting a configuration on one dimension can be misleading. In the same table, LTX-Video scores 66.2% both without caching and with \(P=5\), versus 63.3% for CineScale without LoRA; uncached Wan2.1-14B scores 73.6% versus 73.4% for its comparator. At 3K in Table 2, page 11, \(P=5\) scores 70.1, while both uncached FreeSwim and CineScale with LoRA score 67.4; that preferred cache interval should not automatically be transferred to 1080P.

Ablation Study¶

The following excerpt from Table 3, page 12, uses Wan2.1-1.3B at 1080P; quality metrics are percentages and lower inference time is better. The full-attention baseline is a coarse-to-fine high-resolution configuration, not the from-scratch direct high-resolution generation shown in Figure 3(a).

Config	Aesthetic Quality	Imaging Quality	Overall Score	Inference Time
Wan-Only Full	45.9	40.9	65.5	66 min 46 s
Wan-Only Window, ordinary window	57.0	57.1	71.4	32 min 29 s
Wan-Only Window, inward window	58.2	61.9	72.5	32 min 30 s
FreeSwim, without cache	57.9	63.4	72.7	79 min 15 s
FreeSwim, without CFG-Full	58.6	61.8	73.2	64 min 5 s
FreeSwim, cache \(P=2\)	61.2	65.3	73.7	54 min 40 s

Key Findings¶

Inward shifting raises imaging quality from 57.1% to 61.9%, a gain of 4.8 percentage points, while time changes only from 32 min 29 s to 32 min 30 s, isolating a measurable benefit from boundary treatment.
Cross-attention override raises the inward-window score from 72.5% to 72.7% while substantially increasing runtime; Figures 3 and 7 provide visual evidence of reduced repetition, but not every semantic metric increases simultaneously.
Runtime falls from 79 min 15 s without caching to 54 min 40 s at \(P=2\), approximately a 1.45-fold speedup calculated from the table; relative to the 66 min 46 s full-attention baseline, it is approximately 1.22-fold.
The discussion near Figure 8 on page 14 separately reports a 2.8-fold speedup at 4K with \(P=8\); this different resolution and configuration must not replace the 1080P comparison above.
The user study includes 29 participants; Figure 6, page 11, reports FreeSwim preference shares of 55.0%, 52.3%, and 59.7% for text alignment, aesthetic appeal, and detail richness, respectively.

Highlights & Insights¶

Receptive-field matching is more than matching total token count. Effective boundary neighborhoods also form part of the pretrained distribution, and Table 3 connects this detail to measurable quality improvements.
Self-attention and cross-attention serve different intervention roles. Retaining local detail computation while inserting global information at the text-semantic entry point is more targeted than directly averaging branch outputs.
The cache stores auxiliary semantic guidance rather than the entire generation process. As a reader inference, refresh frequency could adapt to guidance-feature changes, but this extension is not evaluated in the paper.

Limitations & Future Work¶

Caching is not lossless across all configurations: the 1080P score falls from 73.7% at \(P=2\) to 71.7% at \(P=8\). Claims of nearly lossless reuse at larger intervals need to be interpreted alongside the specific visuals and metrics.
The native video still determines coarse layout and motion, so high-resolution refinement cannot guarantee correction of errors from the first stage; systematically quantifying these failures remains useful future work.
Evaluation covers 60 prompts and 5 seeds, supporting comparisons within that sample but not establishing benefits for every backbone, video duration, or complex motion pattern.
Reported VRAM at 1080P, 2K, 3K, and 4K is 10.9, 18.0, 30.3, and 37.3 GB, respectively; CPU offloading keeps the cached version at those values, but practical cost also includes host memory and transfers.
The cache contains no readable appendix, Equations (1) and (3) are corrupted, and the code link survives only as “here” without a URL; unknown code and arXiv links are omitted, and implementation parameters are not invented.
Table 4 reports 65.5% at \(\lambda=0\), inconsistent with the inward-window-only score of 72.5% in Table 3 and identical to the full-attention baseline. A label or configuration difference may be involved, but the available text cannot resolve it and should not be silently corrected.

Relation to SDEdit: upsampling, adding noise, and denoising a low-resolution video provide the coarse-to-fine starting point; FreeSwim focuses on attention organization inside refinement rather than introducing noise-based editing anew.
Compared with I-Max and HiFlow: these provide image-side approaches to beyond-training-resolution generation, whereas FreeSwim targets scale mismatch in video interactions; their adapted Table 1 results remain specific to the evaluated backbone.
Compared with CineScale: FreeSwim requires no high-resolution LoRA and explicitly separates local detail from global semantics; the with-LoRA and without-LoRA variants must be compared separately.
Relation to DeepCache and other caching methods: the shared principle is cross-step feature reuse, but FreeSwim caches cross-attention outputs from its auxiliary full-attention branch while continuing local-branch updates at each step.
Classification: the central task is training-free ultra-high-resolution video synthesis, with attention optimization serving video quality, so video_generation is more precise than image restoration or general model compression.

Rating¶

Novelty: 4/5. The combination of inward windows, semantic override, and specialized caching targets video extrapolation, although coarse-to-fine generation and caching have established foundations.
Experimental Thoroughness: 4/5. Multiple backbones, resolutions, component ablations, and a user study are covered, with limitations from prompt scope and incomplete configuration details.
Writing Quality: 3/5. The mechanism and figures provide a clear narrative, but Table 4's configuration ambiguity and some acceleration claims require tighter alignment with evidence.
Value: 4/5. Existing video DiTs improve without high-resolution retraining, although deployment still needs to account for the global branch and cache resource costs.