Skip to content

Structured-Noise Masked Modeling for Video, Audio and Beyond

Conference: ECCV 2026
Paper: ECCV Official
Code: https://carloshinojosa.me/project/structured-noise-masking
Area: Audio & Speech
Keywords: Masked Autoencoders, Structured-Noise Masking, Self-Supervised Learning, Video Representation Learning, Audio Spectrogram

TL;DR

This paper introduces a data-independent structured-noise masking framework that generates spatiotemporally smooth 3D Green noise masks for video and regularized blue noise (R-BN) masks with controlled time-frequency patch separation for audio spectrograms, consistently outperforming random masking and expensive data-adaptive strategies with zero additional computational overhead.

Background & Motivation

Self-supervised masked modeling has established itself as a leading paradigm for representation learning across images, videos, and audio. By dropping a substantial portion of input tokens—such as image patches, spatiotemporal tubes, or spectrogram bins—and training an asymmetric encoder-decoder network to reconstruct the missing information, models learn deep contextual representations without human labels. Nevertheless, prevailing architectures such as VideoMAE and AudioMAE uniformly apply random masking, dropping tokens uniformly at random across the input tensor. This standard practice fundamentally overlooks the intrinsic physical and spectral characteristics of natural signals: visual data exhibits strong spatial coherence and temporal continuity following an approximate \(1/f\) spectral distribution, while audio spectrograms are characterized by structured time-frequency harmonic energy distributions. Uniform random masking disrupts these natural structures, leading to suboptimal pretraining signals.

To overcome this limitation, several recent works have explored data-adaptive masking strategies guided by semantic priors, attention maps, or optical flow (such as MGMAE, MGM, and AdaMAE). Although empirically promising, these data-driven approaches require auxiliary pretrained networks, reinforcement learning policies, or online motion vector estimation. For instance, computing optical flow increases training time by approximately 1.5 times, incurring heavy computational burdens and limiting seamless generalization across multiple modalities. Conversely, ColorMAE demonstrated that filtering white noise into distinct color noise distributions yields structured 2D masks for static images in a data-independent manner. However, directly applying static 2D color noise to complex temporal and spectral domains fails: videos demand coherent spatiotemporal evolution across consecutive frames rather than rigid static tubes or jittery frame-by-frame cuts, whereas audio spectrograms require a well-dispersed distribution across time and frequency axes to prevent blocking entire harmonic bands.

This paper addresses this challenge by designing modality-aware structured-noise masks that strictly mirror the inductive biases of video, audio, and multimodal signals without introducing any online computation or auxiliary models. Core idea: align the inductive biases of diverse modalities with color noise spectral filtering by generating spatiotemporally smooth 3D Green noise masks for video via dynamic scale sampling and constructing cluster-free Regularized Blue Noise (R-BN) masks for audio spectrograms via four-directional dispersion optimization, delivering a plug-and-play, data-independent framework that enhances self-supervised learning with zero training overhead.

Method

Overall Architecture

The framework provides modality-aligned structured masks for self-supervised masked autoencoding. For video signals, traditional static tube masking fails to capture dynamic motion, whereas independent 2D masking destroys inter-frame continuity; the method generates 3D Green noise masks via 3D Gaussian bandpass filtering and bounded stochastic scale sampling. For audio spectrograms, standard clustering masks obscure critical harmonic overtones, while standard 2D blue noise still suffers from local patch aggregation; the method introduces Regularized Blue Noise (R-BN) to enforce spatial dispersion along four directional orientations. All structured masks are precomputed offline as high-dimensional tensor banks and matched to arbitrary patch configurations during training with random geometric augmentations, seamlessly integrating into VideoMAE, SIGMA, AudioMAE, and CAV-MAE backbones.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Multimodal Data<br/>Video Spatiotemporal Tubes / Audio Spectrogram Patches"] --> B["3D Green Video Masking<br/>3D Difference-of-Gaussians + Dynamic Scale Sampling"]
    A --> C["Regularized Blue Noise Audio Masking<br/>Four-Directional Orientation Constraints + De-clustering"]
    B --> D["Precomputed Offline Tensor Bank<br/>Offline Storage + Random Geometric Augmentations"]
    C --> D
    D --> E["Masked Autoencoder Backbone<br/>Visible Token Encoding + Contextual Reconstruction"]
    E --> F["Downstream Benchmarks<br/>Action Recognition / Video Object Segmentation / Audio Event Classification"]

Key Designs

1. 3D Green Video Masking: Bandpass Filtering with Smooth Spatiotemporal Evolution Standard video masked modeling architectures (e.g., VideoMAE, SIGMA) rely on random tube masking, which stamps an identical 2D random mask across all video frames. Although this enforces temporal consistency, it remains completely static and fails to adapt to dynamic object motion. Conversely, independently sampling 2D green noise per frame produces severe temporal jitter and disrupts spatiotemporal continuity. To resolve this dilemma, the authors extend green noise filtering to the 3D spatiotemporal domain. Convolving a 3D white noise tensor \(n_w\) with two 3D isotropic Gaussian kernels \(G_\sigma(x) = (2\pi)^{-3/2}\sigma^{-3}\exp(-\|x\|^2/(2\sigma^2))\) of bandwidths \(\sigma_1 < \sigma_2\) yields a 3D bandpass filtered signal: $\(n_g^{\text{3D}} = G_{\sigma_1} * n_w - G_{\sigma_2} * n_w\)$ Crucially, fixing \((\sigma_1, \sigma_2)\) to static values restricts mask diversity and risks producing rigid or overly noisy temporal transitions. Instead, the authors stochastically sample \((\sigma_1, \sigma_2)\) within a bounded interval \([0.5, 2.0]\) (enforcing \(\sigma_1 < \sigma_2\)) for each training sequence. This bounded stochasticity generates diverse mid-frequency spatiotemporal patterns, ensuring that the masks evolve smoothly across consecutive frames without abrupt transitions and preventing the encoder from overfitting to any single spatial or temporal frequency.

2. Regularized Blue Noise Audio Masking: Four-Directional Constraints and Patch De-clustering Audio spectrograms display structured time-frequency harmonic tracks, where semantic energy is concentrated along low-to-mid frequency bands and specific temporal events. In this domain, red noise (low-pass) or green noise (bandpass) produces large contiguous clusters of visible and masked regions, frequently blanking out entire fundamental frequencies or time intervals and depriving the network of reconstruction anchors. Blue noise (high-pass filtering) naturally suppresses low frequencies and distributes visible tokens across the plane. However, standard Gaussian-filtered blue noise lacks explicit spatial separation constraints, still exhibiting localized patch clustering. To enforce maximal patch dispersion, the authors introduce the Regularized Blue Noise (R-BN) optimization algorithm. Starting with an initial candidate set of \(K\) thresholded masks \(\{M^i\}_{i=1}^K\), the algorithm scans each spatial coordinate \(P=(x, y)\) within a local window \(U_P^i \in \mathbb{R}^{\Delta \times \Delta}\). It quantifies clustering by counting aligned visible patches along four directional axes—horizontal (\(d_1^i\)), vertical (\(d_2^i\)), and both diagonals (\(d_3^i, d_4^i\)): $\(S_P^i = \sum_{k=1}^4 w_k d_k^i\)$ At each coordinate, the configuration minimizing the clustering score \(\hat{i} = \arg\min_i S_P^i\) is updated until the target visible patch budget \((1-\gamma)N\) is reached. R-BN rigorously enforces spatial separation, eliminating local micro-clusters and providing a uniform time-frequency scaffolding that empowers masked autoencoders to reconstruct complex acoustic harmonics accurately.

3. Offline Precomputation and Augmentation Pipeline: Zero-Overhead Plug-and-Play Integration Existing adaptive masking methods (such as MGMAE and MGM) require online motion vector extraction or optical flow estimation, introducing severe training bottlenecks (MGMAE runs approximately \(1.5\times\) slower than VideoMAE). In sharp contrast, structured-noise masking is entirely data-independent. Both the 3D Green masks and 2D R-BN masks are generated offline prior to training and stored as standard tensor archives (e.g., \(N \times 64 \times 64 \times 64\) tensors for video). During pretraining, cached masks are sampled and mapped to the target input patch grid (such as \(14 \times 14 \times 8\)) using standard bilinear or trilinear interpolation, combined with stochastic geometric augmentations including random spatial flipping and normalization. This preserves the desired spectral noise properties while injecting fresh geometric diversity into each mini-batch, completely eliminating runtime overhead and enabling seamless drop-in integration across arbitrary vision and audio backbones.

Loss & Training

During self-supervised pretraining, only visible patches \(X_p^{\text{visible}} = X_p \odot M\) are fed into the ViT encoder. A lightweight decoder subsequently consumes the processed visible tokens along with learnable mask tokens to reconstruct the complete input \(\hat{X}\) under standard mean squared error (MSE) supervision: $\(\mathcal{L}_{\text{recon}} = \|X - \hat{X}\|_2^2\)$ For video experiments, a ViT-B backbone is trained with a 90% masking ratio for 800 epochs on Kinetics-400 or Something-Something V2. For audio experiments, ViT-B is trained with an 80% masking ratio on AudioSet-2M and fine-tuned with a 30% masking ratio. For audio-visual experiments, CAV-MAE is pretrained for 25 epochs on VGGSound using a 75% masking ratio for both modalities. Upon completing pretraining, the decoder is discarded and the encoder is fine-tuned on downstream tasks.

Key Experimental Results

Main Results

The method was evaluated across video action recognition (Something-Something V2, Kinetics-400), unsupervised video object segmentation (DAVIS, YouTube-VOS), SEVERE generalization benchmarks, audio classification (AudioSet-20k/2M, ESC-50, Speech Commands V2), and multimodal classification (VGGSound).

Task / Domain Baseline Framework Masking Strategy Downstream Benchmark / Metric Performance Gain over Baseline
Video Action Recognition (SSv2 Pretrain) VideoMAE (ViT-B) Green3D (Ours) SSv2 Top-1 Accuracy (%) 70.8 +1.2% (vs 69.6 Random)
Video Action Recognition (K400 Pretrain) VideoMAE (ViT-B) Green3D (Ours) SSv2 Top-1 / K400 Top-1 (%) 69.7 / 80.5 +1.2% / +0.5% (vs Random)
Video Action Recognition (SSv2 Pretrain) SIGMA (ViT-B) Green3D (Ours) SSv2 Top-1 Accuracy (%) 72.0 +0.8% (vs 71.2 Random)
Video Action Recognition (K400 Pretrain) SIGMA (ViT-B) Green3D (Ours) SSv2 Top-1 / K400 Top-1 (%) 71.8 / 82.1 +0.7% / +0.6% (vs Random)
Audio Classification (AudioSet-2M Pretrain) Audio-MAE R-BN (Ours) AS-20k / AS-2M / ESC-50 (%) 36.8 / 47.2 / 94.6 +0.7% / +0.9% / +0.5%
Audio Classification (AudioSet-2M Pretrain) MaskSpec R-BN (Ours) AS-20k / AS-2M / ESC-50 (%) 33.4 / 47.6 / 90.4 +1.1% / +0.5% / +0.8%
Audio-Visual Classification (VGGSound) CAV-MAE Green3D + R-BN Audio / Video / Audio-Visual (%) 59.1 / 46.4 / 64.9 +0.6% / +0.8% / +0.6%

In unsupervised video object segmentation, VideoMAE equipped with Green3D masking demonstrates dramatic gains on DAVIS clustering mIoU, jumping from 29.5% to 38.2% (+8.7% absolute gain) and clearly outperforming motion-guided alternatives like MGMAE (31.0%) and MGM (36.5%). On the SEVERE benchmark evaluating domain shift, sample efficiency, action granularity, and task shift across eight experiments, Green3D boosts the mean score by +1.3% for VideoMAE and +3.0% for SIGMA.

Ablation Study

Ablation experiments systematically isolate the impact of noise color across modalities, comparing 3D continuous masking against 2D baselines and contrasting R-BN with vanilla blue noise.

Modality / Dataset Masking Strategy Reconstruction Loss \(\mathcal{L}_{\text{recon}}\) Downstream Metric (Top-1 / Acc) Note
Video (mini-Kinetics / mini-SSv2) Random (Tube) 0.67 51.6% / 52.8% Default baseline: static tube lacks temporal dynamics
Video (mini-Kinetics / mini-SSv2) Blue Noise (High-pass) 0.41 50.9% / 52.1% Task overly trivial; low loss leads to poor representations
Video (mini-Kinetics / mini-SSv2) Red Noise (Low-pass) 0.85 51.0% / 52.3% Excessive difficulty and large masked holes degrade learning
Video (mini-Kinetics / mini-SSv2) Green3D (Bandpass, Ours) 0.60 52.7% / 54.5% Optimal balance between solvability and challenge
Video (mini-Kinetics / mini-SSv2) Green2D (Frame-independent) 0.73 51.9% / 52.9% Lacks temporal coherence; marginal gain over random
Audio (AS-20k / ESC-50) Random Masking 0.52 36.1% / 94.1% Standard audio random baseline
Audio (AS-20k / ESC-50) Green Noise 0.57 36.4% / 94.1% Clustered patches disrupt time-frequency context
Audio (AS-20k / ESC-50) Red Noise 0.61 35.5% / 92.6% Severe harmonic occlusion drops accuracy significantly
Audio (AS-20k / ESC-50) Vanilla Blue Noise 0.45 36.5% / 94.2% Uniform high-pass distribution outperforms Green/Red
Audio (AS-20k / ESC-50) Regularized Blue (R-BN, Ours) 0.49 36.8% / 94.6% Directional regularization eliminates clustering, best audio accuracy

Key Findings

  • Modality Inductive Biases Invert Optimal Noise Spectra: For video modeling, Green noise (mid-frequency bandpass) delivers superior representation quality by matching the natural spatio-temporal scale of objects and motion, whereas Blue noise makes reconstruction trivial (\(\mathcal{L}_{\text{recon}}=0.41\)) and Red noise creates unbridgeable blind spots (\(\mathcal{L}_{\text{recon}}=0.85\)). In contrast, audio spectrograms exhibit the exact opposite pattern: Blue noise substantially outperforms Green and Red noise, confirming that spectrograms require finely dispersed visible tokens across time and frequency axes to resolve harmonic structures.
  • Spatiotemporal Continuity is Essential in Video: Naively extending 2D Green noise to video by applying it independently across frames (Green2D) achieves only 51.9% on mini-Kinetics, barely outperforming random tube masking (51.6%). Generating true 3D Green noise with continuous spatiotemporal coherence boosts accuracy to 52.7%, validating the necessity of temporal smoothness.
  • Impact of R-BN De-clustering: Optimizing vanilla blue noise with directional separation constraints raises AudioSet-20k accuracy from 36.5% to 36.8% and ESC-50 from 94.2% to 94.6%, proving that eliminating local micro-clusters provides superior time-frequency conditioning for masked audio autoencoders.

Highlights & Insights

  • Zero-Overhead Data-Independent Innovation: The method replaces expensive auxiliary networks, optical flow estimators, and complex online heuristics with precomputed frequency-filtered noise tensors, matching or outperforming data-adaptive methods like MGMAE at zero additional GPU training cost.
  • Deep Spectral-Domain Insights: By grounding masking designs in the spectral characteristics of modalities (spatiotemporal \(1/f\) distributions in video vs. dispersed time-frequency harmonics in audio), the work establishes clear, principled design guidelines for self-supervised masking.
  • Broad Multi-Domain Versatility: Seamlessly functions as a plug-and-play drop-in replacement across unimodal video, unimodal audio, and joint audio-visual architectures without requiring hyperparameter retuning.

Limitations & Future Work

  • Author-Acknowledged Limitations: The mask generation process relies on static frequency filtering and localized greedy optimization; while empirically effective, it does not adaptively reconfigure itself in response to sudden scene cuts or extreme camera velocity within individual video clips.
  • Scope and Scaling Limitations: Evaluations were primarily benchmarked on ViT-B backbones; empirical behavior on multi-billion parameter foundation models or generative video diffusion frameworks remains unverified. Furthermore, the R-BN optimization algorithm requires iterative offline multi-directional counting, which can be computationally slow when generating high-resolution candidate pools.
  • Future Directions: Developing learnable, differentiable frequency bandwidth parameters \((\sigma_1, \sigma_2)\) conditioned on clip dynamics, and exploring structured-noise schedules for forward and reverse diffusion processes in video synthesis.
  • vs ColorMAE: While ColorMAE introduced 2D color noise masks for static images, this paper extends the paradigm to temporal video via 3D Green noise and to audio spectrograms via Regularized Blue Noise, establishing a unified multimodal masking framework.
  • vs VideoMAE / AudioMAE: Traditional architectures rely on rigid random tube masking or uniform random patch dropping. Structured-noise masking serves as an exact drop-in replacement that boosts performance across all downstream metrics without altering model weights or loss functions.
  • vs MGMAE / MGM: Motion-guided masking requires optical flow extraction that adds \(\sim 1.5\times\) computational overhead. The proposed method is completely data-independent, runs at zero additional training cost, and significantly outperforms MGMAE in unsupervised video object segmentation (DAVIS mIoU +8.7%).

Rating

  • Novelty: ⭐⭐⭐⭐☆ Elegantly adapts color noise principles to 3D video spatiotemporal dynamics and 2D audio spectral structures through novel 3D Green filtering and R-BN optimization.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations spanning video action recognition, unsupervised video segmentation, SEVERE generalization, audio event classification, and audio-visual benchmarks.
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-written, clearly articulated motivation, transparent mathematical derivations, and rigorous comparative analyses.
  • Value: ⭐⭐⭐⭐⭐ Zero computational overhead, data-independent, plug-and-play usability, and broad applicability across diverse self-supervised modeling domains.