SyncCache: Exploiting Asymmetric Dynamics for Fast Audio-Driven Portrait Animation¶

Conference: ECCV 2026
arXiv: 2606.30849
Code: None
Area: Video Understanding / Portrait Animation
Keywords: Diffusion Transformer Acceleration, Feature Caching, Audio-Driven Portrait Animation, Modal Asymmetry, Training-Free Acceleration

TL;DR¶

SyncCache proposes a training-free feature caching acceleration scheme for DiT-based audio-driven portrait animation. By employing Spatial Asymmetric Detection (SAP) to prioritize recalculations in the facial region, Modal Decoupled Caching (MDC) to skip heavy visual modules while refreshing lightweight audio modules step-by-step, and memory-adaptive offline DP-based cache planning, it achieves 4.12x and 3.75x acceleration on HunyuanVideo-Avatar and Wan-S2V respectively, with near-lossless visual quality and lip-sync performance.

Background & Motivation¶

DiT architectures have significantly improved the visual fidelity and expression controllability of portrait animation, but inference latency remains a critical bottleneck for practical deployment. Generating a 15-second video on 8x A800 GPUs takes approximately 10 minutes, severely hindering creative iteration efficiency. Training-free feature caching (such as TeaCache, MagCache, TaylorSeer) has recently proven effective in accelerating diffusion inference. However, these methods mainly target text-to-video generation, where text conditions serve as low-frequency global signals, and caching strategies naturally favor smoothness. In audio-driven portrait animation, however, modal dynamics present high asymmetry: the reference image provides low-frequency static visual priors (background, identity), while the driving audio consists of high-frequency local conditions driving frame-by-frame lip and muscle movements. Scaling temporal-level caching (skipping entire timesteps) interrupts the continuous injection of audio signals, leading to severe loss of high-frequency details. On the other hand, module-level caching (such as the TaylorSeer family) stores feature maps for every module, causing memory overhead to explode from \(O(1)\) to \(O(L)\), which quickly triggers out-of-memory (OOM) errors for long videos or high resolutions. Similar asymmetry exists in the spatial dimension: human lips and facial muscles are high-frequency dynamic regions while the background remains almost static, yet existing caching strategies treat detection errors uniformly across the entire frame, missing the opportunity to trigger recalculation at key moments to preserve fine facial movements. The root cause of these contradictions is that existing methods assume uniform modal dynamics, whereas portrait animation dynamics are inherently asymmetric. Core Idea: Explicitly model the dual asymmetry (spatial asymmetry + modal asymmetry) inherent in audio-driven portrait animation into the caching policy—spatially using a human mask to weight detection errors to prioritize high-frequency facial details, and modally decoupling heavy visual modules from lightweight audio modules, safely reusing stable cache residuals while refreshing audio signals at every timestep.

Method¶

Overall Architecture¶

SyncCache is a training-free cache acceleration framework designed specifically for DiT-based audio-driven portrait animation, operating within standard denoising sampling loops. At each timestep, it comprises three decoupled yet collaborative modules: (1) Spatial Asymmetric Detection—computes only the output of the first DiT block, using a human mask-weighted L1 relative error to determine whether to trigger a full forward pass to refresh the cache; (2) Modal Decoupled Caching—caches the cross-block residuals between audio blocks at the reference timesteps where full computation is triggered, and subsequently skips heavy self-attention/MLP computations in repeated steps, only recovering the residuals and refreshing the lightweight audio blocks; (3) Memory-Adaptive Optimal Selection—calculates residual stability scores using an offline calibration pass and selects the optimal subset of caching blocks via DP, providing flexible control over GPU memory usage through the cache ratio sigma. The temporal relationship among the three modules is: the detection module runs at every step (consuming only the computation of the first block), and when the accumulated error exceeds the threshold, a full-computation timestep is triggered. During this step, MDC caches residuals and runs complete audio blocks. Subsequent repeated steps reuse the cached residuals to skip massive visual computations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Current latent x_t<br/>Reference Image + Audio"] --> B["Compute first block output p^t<br/>& mask-weighted error ê_t"]
    B --> C{"Accumulated error > Threshold δ?"}
    C -->|No: Repeated step| D["Reuse cached cross-block residual<br/>Skip visual self-attention / MLP"]
    D --> E["Step-by-step refresh of lightweight audio block<br/>Maintain lip-sync"]
    C -->|Yes: Full-computation step| F["Full DiT Forward Pass"]
    F --> G["Cache cross-block residual r_i,j between audio blocks"]
    G --> H["Output updated latent x_{t-1}"]
    E --> H
    H --> B

Key Designs¶

1. Spatial Asymmetric Detection: Weighting errors with a human mask to prioritize recalculations in the facial region

To decide when to cache and when to recalculate, it is necessary to estimate the temporal redundancy of features between adjacent timesteps. Existing methods directly use a global L1 relative error as a detection metric. However, in portrait animation, this suffers from severe limitations—high-frequency critical changes are concentrated on human regions such as the face and lips, while the background remains nearly static. SyncCache exploits a piece of prior information: the human mask. This mask is already a standard auxiliary input in many DiT models (e.g., Hallo3, HunyuanVideo-Avatar). For models without masks (e.g., Wan-S2V), a lightweight human detection model can extract it in less than a second. By weighting the detection error with the mask, the region inside the mask is multiplied by a weight coefficient \(\omega\) (experimentally, \(\omega=2\) is sufficient, showing robustness in the 2-4 range), amplifying errors in high-frequency human regions and smoothing out background errors. Consequently, when dynamic changes in the face are drowned out by global averaging, the weighted error will be the first to exceed the threshold \(\delta\), forcing the model to trigger a complete forward pass at that timestep, thereby precisely preserving lip movements and expression details.

2. Modal Decoupled Caching: Skipping heavy visual modules while refreshing audio modules at each timestep to address modal computational asymmetry

The key observation of SyncCache is that self-attention and MLP layers account for over 99% of computation in DiT portrait animation, whereas audio blocks (responsible for injecting lip-sync control signals) occupy less than 1% but are essential. More importantly, cross-block residuals between audio blocks exhibit high temporal stability across different denoising steps, meaning they can be cached and reused without significant quality loss. Based on this dual asymmetry of computation and dynamics, SyncCache structurally decouples the computational pipelines of the two modalities. In full-computation timesteps, the cross-block residuals from audio block \(i\) to \(j\) are cached. In subsequent repeated steps, the cached residuals are directly reused to skip the entire heavy visual DiT computation, while the lightweight audio blocks are executed as usual. This ensures that high-frequency control signals are refreshed and synchronized at every timestep without incurring extra computational costs.

3. Memory-Adaptive Optimal Selection: Offline DP planning of optimal cache block subsets scaled by memory budget

Although MDC significantly reduces computational cost, it defaults to caching residuals at every audio block boundary, still keeping a memory overhead of \(O(L_a)\) (where \(L_a\) is the number of audio blocks). For long videos or deployments on memory-constrained devices, a more flexible memory footprint control is required. SyncCache introduces a continuous cache ratio \(\sigma \in (0, 1]\) to specify caching \(n = \lfloor L_a \sigma \rfloor\) residual boundaries. However, arbitrarily choosing which boundaries to cache is sub-optimal since the temporal stability of residuals varies drastically across audio blocks of different depths. Therefore, SyncCache quantifies the temporal instability of cross-block residuals as:

\[\gamma_{i,j} = \sum_{t=0}^{T-1} \frac{\| r_{i,j}^{t} - r_{i,j}^{t+1} \|_1}{\| r_{i,j}^{t+1} \|_1}\]

Then, dynamic programming (DP) is used to select the \(n\) boundaries that minimize the global temporal instability across the full denoising trajectory. A crucial finding is that for any given model, this optimal cache path is highly consistent across different input samples (different audio, different reference images)—even paths calibrated using silent samples show almost no difference from conversational ones. Consequently, with just a single offline calibration forward pass (taking about a few minutes for one sample), the optimal caching plan can be predetermined for any \(\sigma\), incurring zero overhead during online inference.

Loss & Training¶

SyncCache is entirely training-free and introduces no additional training loss or fine-tuning steps. All parameters (detection threshold \(\delta\), human weighting coefficient \(\omega\), cache ratio \(\sigma\)) are hyper-parameters that can be tuned during inference without retraining the model.

Key Experimental Results¶

Main Results¶

Model Config	Metric	Baseline (50 steps)	TeaCache 2.25x	MagCache 2.30x	CGCache 3.18x	SyncCache-slow 3.34x	SyncCache-fast 4.12x
HunyuanVideo-Avatar	LPIPS↓	-	0.1730	0.1696	0.1848	0.1016	0.1172
	SSIM↑	-	0.8428	0.8455	0.8249	0.8618	0.8493
	FID↓	25.27	26.83	26.43	27.75	25.65	26.89
	Sync-C↑	6.963	6.842	6.830	6.814	0.944	6.902

Model Config	Metric	Baseline (40 steps)	TeaCache 2.93x	MagCache 2.96x	DiCache 2.99x	SyncCache 3.75x
Wan-S2V	LPIPS↓	-	0.1863	0.1839	0.1835	0.1775
	PSNR↑	-	19.04	19.46	19.43	19.80
	Sync-C↑	6.712	6.678	6.709	6.716	6.791
	Sync-D↓	8.632	8.642	8.641	8.614	8.541

SyncCache consistently outperforms existing methods across both models in terms of visual quality and audio synchronization metrics. Notably, the TaylorSeer series (module-level caching) directly ran OOM on an 8-GPU A800 cluster, confirming the memory bottleneck of \(O(L)\) module-level caching strategies on large-scale models. SyncCache-fast maintains near-lossless Sync-C (6.902 vs. original 6.963) even under 4.12x acceleration, whereas TeaCache and MagCache show significant performance drops at 2.3x.

Ablation Study¶

Config	LPIPS↓	SSIM↑	Sync-C↑	Latency (s)	Note
Full SyncCache	0.1016	0.8618	6.944	157	Full SyncCache
w/o SAP	0.1259	0.8534	6.867	161	w/o Spatial Asymmetric Detection (degenerates to uniform detection)
w/o MDC	0.1571	0.8437	6.822	156	w/o Modal Decoupled Caching (both modalities cached/computed simultaneously)

Memory-Adaptive Selection	LPIPS↓	Sync-C↑	Note
DP Optimization (\(\sigma=0.4\))	0.1016	6.944	Offline DP selects optimal boundaries
w/o DP (Random Choice)	0.1369	6.847	Randomly select cache boundaries
Silent Sample Calibration	0.1021	6.932	Calibration with nearly silent samples

Key Findings¶

Modal decoupling is the key to lip-sync: Removing MDC (i.e., caching visual and audio blocks simultaneously) drops Sync-C from 6.944 to 6.822, indicating that refreshing the audio signal step-by-step is indispensable for maintaining high-precision lip-sync.
Spatial asymmetric detection primarily preserves visual quality: Removing SAP increases LPIPS from 0.1016 to 0.1259, meaning that detection without mask weighting easily misses critical changes in high-frequency human regions.
DP optimization yields significant effects: Random boundary selection degrades significantly compared to DP optimization (LPIPS increases from 0.1016 to 0.1369), and the choice of calibration samples has almost no impact—even silent samples can calibrate a high-quality cache plan.
Temporal-level caching has an acceleration limit: TeaCache/MagCache saturate around 2.3x, whereas SyncCache achieves up to 4.12x with superior visual and audio-visual synchronization.

Highlights & Insights¶

Addressing dual asymmetry is the core insight: Rather than treating caching as a generic acceleration technique, this work dives deep into the domain structure of the target task—spatial asymmetry between humans and backgrounds, and modal asymmetry between vision and audio. Encoding these domain priors into cache decisions yields far superior results in portrait animation compared to general caching schemes.
Elaborate design of offline calibration + zero online overhead: By discovering that cross-block residual stability is highly consistent across different inputs, the complex NP-hard cache planning problem is transformed into a one-time offline DP optimization. This incurs zero additional overhead during online inference, making it more efficient than methods requiring dynamic evaluation during inference (such as TeaCache's online cumulative error estimation).
Cache ratio \(\sigma\) serves as a continuous control knob: Providing a continuous trade-off adjustment between memory and speed, a single parameter can adapt to different scenarios, ranging from short demos to long videos and from high-end GPUs to edge devices, which is highly practical.
Soft mask instead of hard cropping: SAP utilizes a soft multiplication mask rather than hard cropping. If mask detection fails, it naturally degenerates to uniform detection, demonstrating good robustness.
The method is directly applicable to existing DiT portrait animation models, requiring no fine-tuning or architectural modifications, offering plug-and-play capability.

Limitations & Future Work¶

The method currently relies on the availability of masks (or an additional detection model). Under challenging detection scenarios (such as heavy occlusion or extreme angles), decreased mask accuracy can lead to acceleration degradation.
The cache path is based on a one-time offline calibration assuming a fixed model architecture, which is unsuitable for scenarios using dynamic computational graphs or switching architectures during inference.
While the effectiveness of the method is validated on two popular models (HunyuanVideo-Avatar and Wan-S2V), its generalizability to a wider range of DiT-based portrait animation models remains to be verified.
Currently, the method operates strictly under "training-free" acceleration. If lightweight fine-tuning is permitted, cache decisions might be jointly optimized with model parameters to preserve quality under even more aggressive acceleration ratios.

vs. TeaCache / MagCache: Temporal-level caching methods decide to skip entire timesteps through polynomial estimation or magnitude-aware strategies. They fail to consider modal asymmetry, leading to significant loss of high-frequency signals in audio-driven scenarios. SyncCache maintains audio continuity through modal decoupling.
vs. TaylorSeer / SpeCa / ClusCa: Module-level caching methods independently predict or cache features for each module, resulting in an \(O(L)\) memory overhead, which triggers OOM errors on large models. SyncCache only caches residuals across audio blocks, where memory overhead can be controlled within \(O(1)\) to \(O(L_a)\) via \(\sigma\).
vs. Delta-DiT / HiCache: These methods improve feature prediction accuracy using incremental updates or superior mathematical bases (e.g., Hermite polynomials), but similarly miss modeling asymmetric modalities. SyncCache's domain-aware design is complementary, and integrating them could be a promising future direction.
Discrepancy with general acceleration methods: SyncCache's domain specificity is a double-edged sword—it is significantly superior to general schemes on portrait animation but cannot be directly transferred to text-to-video or image generation. The core takeaway is that designing domain-specific caching strategies tailored to concrete task structures can yield a much larger performance gain than general optimization.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Extending diffusion cache acceleration from the general domain to audio-driven portrait animation, and modeling dual asymmetric dynamics for the first time. The design methodology of domain-aware caching is highly inspiring.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comparing against numerous baselines on two mainstream models with multiple metrics. The ablation studies fully cover the three core components, and the robustness analysis of memory-adaptive selection is well-conducted.]
Writing Quality: ⭐⭐⭐⭐⭐ [The motivation is clearly stated, with the concept of asymmetric dynamics consistently implemented from problem formulation to method design. The charts are complete and illustrative.]
Value: ⭐⭐⭐⭐⭐ [The practical acceleration is significant (4.12x) with plug-and-play capability without training, offering direct practical value for DiT inference deployment. The methodology of domain-aware caching can inspire other multi-modal generation tasks.]