Escaping the Low-Frequency Bias: Adversarial Frequency Perturbation for Generalisable Gaze Estimation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Human Understanding
Keywords: Gaze Estimation / Domain Generalization / Frequency Analysis / Adversarial Perturbation / Adaptive Instance Normalization
TL;DR¶
This paper demonstrates that cross-domain performance degradation in gaze estimation is primarily driven by overfitting to source-domain low-frequency components; it proposes an Adversarial Low-Frequency Perturbation (ALFP) framework that dynamically mixes low-frequency amplitude statistics via AdaIN using an instance-specific generator (PGNet), significantly improving cross-domain generalization without requiring any target-domain data.
Background & Motivation¶
Deep learning-based appearance gaze estimation has advanced substantially in recent years, playing an increasingly crucial role in virtual and augmented reality, human-computer interaction, and medical diagnosis. Despite remarkable accuracy in within-domain benchmarks, existing models suffer severe performance degradation when deployed in unseen environments. Prior studies typically attribute this cross-domain gap to spatial variations, such as changes in illumination, background clutter, subject skin tone and appearance, and camera sensor characteristics. While existing solutions employ adversarial feature alignment, rotation consistency, contrastive learning, and feature purification to bridge this gap, almost all of them operate exclusively in the spatial domain without examining the underlying physical and statistical causes of domain shifts from a frequency-domain perspective.
From a frequency perspective, domain-specific environmental factors such as background scenes, illumination falloff, and skin tone vary smoothly across facial regions, naturally carrying substantial low-frequency components. Through a gradient-based frequency sensitivity analysis on standard CNN architectures (such as ResNet-18), the authors uncover an abnormal sensitivity peak concentrated in the low-frequency spectrum. Diagnostic experiments demonstrate that discarding source-domain low-frequency components altogether triggers a catastrophic collapse in gaze estimation accuracy (average angular error surges from 8.54° to 14.53°); conversely, randomly perturbing low-frequency amplitude scaling during training significantly reduces cross-domain error to 7.20°. These observations establish that low-frequency bands contain indispensable geometric dependencies for eye tracking that cannot be naively excised, and that the fundamental bottleneck in cross-domain generalization is the model's pathological overfitting to the specific low-frequency distributions of the source domain.
Therefore, the central challenge is to break the network's reliance on source-domain low-frequency statistics while preserving the spatial and geometric structures essential for accurate gaze regression. Core idea: propose the Adversarial Low-Frequency Perturbation (ALFP) framework, which employs a Perturbation Generation Network (PGNet) to synthesize instance-specific perturbation images and mixes their low-frequency amplitude statistics with source images via AdaIN, driving the gaze estimator to discard spurious low-frequency styles while learning robust, domain-invariant gaze representations.
Method¶
Overall Architecture¶
ALFP aims to mitigate low-frequency overfitting and promote cross-domain generalization without relying on any target-domain samples. The overall pipeline operates via two alternating steps and an adversarial game between two sub-networks: the Perturbation Generation Network (PGNet) and the Gaze Estimation Network (GazeNet). First, PGNet takes intermediate features extracted by GazeNet concatenated with random Gaussian noise to synthesize an instance-specific perturbation image in the bounded spatial domain. Second, both the source image and the generated perturbation image are transformed into the frequency domain via 2D Fast Fourier Transform (2D FFT); within a circular low-frequency region, their amplitude statistics (mean and standard deviation) are smoothly interpolated via Adaptive Instance Normalization (AdaIN), before being recombined with the original phase spectrum to reconstruct the augmented input via inverse FFT. Finally, GazeNet and PGNet are jointly optimized in an alternating adversarial paradigm to maximize robustness and perturbation effectiveness.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Face Image Is"] --> B["Spatial Adversarial Perturbation Generation<br/>PGNet combines feature f and noise z to synthesize Ip"]
B --> C["AdaIN-based Low-Frequency Amplitude Mixing<br/>2D FFT separates amplitude and phase; AdaIN blends statistics"]
C --> D["Inverse Fourier Transform Reconstruction<br/>Reconstruct Iaug using mixed amplitude Aaug and original phase Φs"]
D --> E["Alternating Adversarial Joint Optimization<br/>GazeNet enforces prediction consistency; PGNet maximizes error"]
E --> F["Output: Cross-domain robust gaze estimator GazeNet"]
Key Designs¶
1. Spatial-Domain Conditional Adversarial Perturbation Generation: Ensuring Optimization Stability and Instance Specificity
Directly searching for adversarial perturbations in the frequency domain is prone to severe numerical instability and gradient explosion because high-dimensional amplitude spectra exhibit unbounded, heavy-tailed distributions. To circumvent this issue, ALFP confines PGNet's output to the normalized spatial domain \([0, 1]^{3 \times H \times W}\). For each source input image \(I_s\), GazeNet extracts an intermediate feature vector \(f\), which is concatenated with a random Gaussian noise vector \(z \sim \mathcal{N}(0, I)\) (\(d_z = 100\)) and fed into PGNet. Through four convolutional upsampling blocks equipped with a terminal Sigmoid activation, PGNet generates a spatial perturbation image \(I_p\): $\(I_p = \sigma(P(z, f)) \in [0, 1]^{3 \times H \times W}\)$ This design provides two decisive structural advantages: first, because the perturbation image serves purely as an intermediate carrier for extracting frequency amplitude statistics, it does not require explicit facial semantic structures; second, conditioning the generation directly on the instance feature \(f\) produces tailored, instance-specific perturbations that target the model's precise low-frequency vulnerabilities for that individual input.
2. AdaIN-Based Low-Frequency Amplitude Mixing: Decoupling Frequency Style from Gaze Geometry
Directly substituting or independently modifying discrete frequency bins within the Fourier spectrum disrupts the internal spectral correlation between adjacent frequencies, creating severe ringing artifacts and corrupting fine gaze-directional features. ALFP resolves this by performing distribution-level interpolation using Adaptive Instance Normalization (AdaIN). Applying 2D FFT to both \(I_s\) and \(I_p\) yields amplitude spectra \(A_s, A_p\) and the source phase spectrum \(\Phi_s\). A circular low-frequency mask \(R\) centered at the DC component is defined with radius \(r = \beta \cdot \min(H, W)/2\), where \(\beta \sim \mathcal{U}(0, 0.5]\) is randomly sampled per batch. After extracting channel-wise mean and standard deviation scalars \((\mu_s, \sigma_s)\) and \((\mu_p, \sigma_p)\) within \(R\), an instance-wise mixing coefficient \(\lambda \sim \mathcal{U}(0, 1)\) interpolates the statistics: $\(A_{\text{lowmix}} = [\lambda \sigma_s + (1-\lambda)\sigma_p] \cdot \frac{A_s[R] - \mu_s}{\sigma_s} + [\lambda \mu_s + (1-\lambda)\mu_p]\)$ The augmented amplitude spectrum \(A_{\text{aug}}\) is formed by substituting \(A_s[R]\) with \(A_{\text{lowmix}}\) while keeping all high-frequency components intact. The augmented face image is subsequently reconstructed via inverse Fourier transform: \(I_{\text{aug}} = \text{IFFT}(A_{\text{aug}}, \Phi_s)\). Because Fourier phase spectra preserve structural content and geometric boundaries (such as eye orientation and facial landmarks) while amplitude statistics govern global style (illumination, skin tone, and contrast), this mechanism perturbs environmental styles while preserving the validity of the ground-truth gaze label.
3. Batch-Level Spectral Statistical Constraint and Multi-Branch Adversarial Optimization: Preventing Shortcut Collapse and Preserving Invariance
During alternating optimization, if PGNet is guided solely to maximize GazeNet's regression error, it trivially converges to degenerate shortcuts—such as producing fully saturated white images or binary noise patterns that completely obliterate facial identity. To regularize the search space without restricting sample diversity, ALFP introduces a batch-level statistical constraint loss \(L_{\text{stat}}\) formulated in log space: $\(L_{\text{stat}} = \|\log(\bar{\mu}_p) - \log(\bar{\mu}_s)\|_2^2 + \|\log(\bar{\sigma}_p) - \log(\bar{\sigma}_s)\|_2^2\)$ where \(\bar{\mu}_s, \bar{\sigma}_s\) and \(\bar{\mu}_p, \bar{\sigma}_p\) denote the batch-averaged mean and standard deviation of low-frequency amplitudes. By constraining statistics at the batch level rather than per-sample, individual perturbations can explore diverse styles while keeping the overall distribution within a realistic physical range. The total loss for updating PGNet is \(L_P = -L_{\text{gaze}}(G(I_{\text{aug}}), g_{\text{gt}}) + L_{\text{stat}}\). Simultaneously, GazeNet is updated via a composite multi-branch loss function to guarantee both clean-sample precision and perturbation invariance: $\(L_G = L_{\text{ori}} + L_{\text{aug}} + L_{\text{cons}}\)$ where \(L_{\text{ori}} = \|G(I_s) - g_{\text{gt}}\|_1\) preserves source benchmark accuracy, \(L_{\text{aug}} = \|G(I_{\text{aug}}) - g_{\text{gt}}\|_1\) supervises perturbed images with ground-truth gaze vectors, and \(L_{\text{cons}} = \|G(I_{\text{aug}}) - G(I_s)\|_1\) penalizes any gaze prediction discrepancies triggered by low-frequency perturbations.
Loss & Training¶
Both networks are optimized using the Adam optimizer with a learning rate of \(10^{-4}\) on NVIDIA RTX 3090 GPUs. Training runs for 10 epochs with a mini-batch size of 256. Face crops are resized to \(224 \times 224\) and normalized to \([0, 1]\) without spatial data augmentations (such as random rotation or color jitter). Crucially, during inference, PGNet is discarded entirely; GazeNet executes standard spatial feed-forward inference without any Fourier transforms or computational overhead, preserving native execution speed.
Key Experimental Results¶
Main Results¶
Experiments are conducted across four standard cross-domain gaze benchmarks: ETH-XGaze (DE), Gaze360 (DG), MPIIFaceGaze (DM), and EyeDiap (DD). The evaluation metric is the mean angular gaze error in degrees (lower is better).
| Method | Backbone | DE→DM | DE→DD | DG→DM | DG→DD | Avg Error (°) |
|---|---|---|---|---|---|---|
| Baseline (ERM) | ResNet-18 | 7.29 | 9.77 | 8.05 | 9.03 | 8.54 |
| Baseline (ERM) | ResNet-50 | 6.84 | 9.06 | 7.31 | 9.09 | 8.08 |
| PureGaze (AAAI 2022) | ResNet-50 | 7.08 | 7.48 | 9.28 | 9.32 | 8.29 |
| CDG (CVPR 2023) | ResNet-18 | 6.73 | 7.95 | 7.03 | 7.27 | 7.25 |
| Xu et al. (AAAI 2023) | ResNet-18 | 6.50 | 7.44 | 7.55 | 9.03 | 7.63 |
| FSCI (CVPR 2024) | ResNet-18 | 5.79 | 6.96 | 7.06 | 7.99 | 6.95 |
| GFAL (CVPR 2024) | ResNet-18 | 5.72 | 6.97 | 7.18 | 7.38 | 6.81 |
| CLIP-Gaze (ECCV 2024) | ViT-B/16 | 6.41 | 7.51 | 6.89 | 7.06 | 6.97 |
| LG-Gaze (ECCV 2024) | ViT-B/16 | 6.45 | 7.22 | 6.83 | 6.86 | 6.84 |
| ALFP (Ours) | ResNet-18 | 5.92 | 6.36 | 6.10 | 7.26 | 6.41 |
| ALFP (Ours) | ResNet-50 | 5.68 | 5.90 | 5.84 | 7.19 | 6.15 |
Ablation Study¶
Ablation studies on ResNet-18 validate key architectural choices, including the perturbation generation domain, mixing schemes, statistical constraints, and perturbation data sources.
| Dimension | Configuration Variant | DE→DM | DE→DD | DG→DM | DG→DD | Avg Error (°) | Note |
|---|---|---|---|---|---|---|---|
| Generation Domain | Frequency-domain direct amplitude (Softplus) | 6.25 | 6.56 | 6.60 | 7.87 | 6.82 | Heavy-tailed amplitudes hinder adversarial stability |
| Spatial-domain image generation (Ours) | 5.92 | 6.36 | 6.10 | 7.26 | 6.41 | Bounded [0,1] space ensures stable optimization | |
| Mixing Mechanism | Element-wise independent frequency mixing | 7.04 | 7.87 | 7.68 | 13.08 | 8.92 | Breaks spectral coherence, triggering large degradation |
| AdaIN-based statistics mixing (Ours) | 5.92 | 6.36 | 6.10 | 7.26 | 6.41 | Preserves internal spectral structure and geometry | |
| Statistical Loss \(L_{\text{stat}}\) | Completely removed (w/o \(L_{\text{stat}}\)) | 6.35 | 6.86 | 7.12 | 9.64 | 7.49 | Collapses to saturated white images as shortcuts |
| Sample-level constraint | 6.40 | 7.09 | 6.92 | 7.37 | 6.95 | Restricts perturbation space; yields source-like styles | |
| Pixel-level mean/variance constraint | 6.40 | 6.82 | 6.63 | 9.19 | 7.26 | Aggregates all frequencies; lacks targeted control | |
| Diversity maximization (variance loss) | 6.12 | 8.42 | 6.56 | 8.81 | 7.48 | Exploits binary black/white extreme patterns | |
| Batch-level frequency log constraint (Ours) | 5.92 | 6.36 | 6.10 | 7.26 | 6.41 | Balanced diverse perturbation and physical validity | |
| Perturbation Sources | Within-domain random pairing | 6.33 | 6.68 | 6.40 | 7.75 | 6.79 | Static samples cannot maximize adversarial hardness |
| External human faces (VGGFace2) | 6.54 | 6.76 | 6.83 | 8.44 | 7.14 | Fixed distribution fails to target model weaknesses | |
| External animal faces (AP-10K) | 6.51 | 6.53 | 6.69 | 7.94 | 6.92 | Cross-species distribution still inferior to PGNet | |
| Adversarial generation via PGNet (Ours) | 5.92 | 6.36 | 6.10 | 7.26 | 6.41 | Synthesizes instance-adaptive challenging styles |
Key Findings¶
- Substantial Gains in Challenging Domain Shifts: On the demanding DE→DD transfer task (migrating from a high-resolution, multi-camera studio rig to low-resolution VGA video), ALFP slashes the angular error of ResNet-18 from 9.77° to 6.36° (a 34.9% relative error reduction). On DG→DM, ALFP achieves 5.84° with ResNet-50, surpassing the prior best method by 15.0%.
- Optimal Frequency Cutoff Boundary \(\beta\): Sweeping the upper bound of the low-frequency radius ratio \(\beta\) from 0.1 to 1.0 reveals an optimal threshold at \(\beta = 0.5\) (yielding 6.41° error). Smaller values (\(\beta < 0.5\)) insufficiently cover domain-specific style bands, whereas excessively large radii (\(\beta > 0.5\)) corrupt fine high-frequency structural details (such as pupil edges and iris borders).
- Flattening the Frequency Sensitivity Peak: Radial frequency sensitivity profiles \(S(r)\) confirm that while the ERM baseline exhibits extreme gradient spikes in the low-frequency region, ALFP suppresses this sensitivity peak while maintaining moderate response levels, balancing low-frequency robustness with essential geometric feature perception.
Highlights & Insights¶
- Rethinking Domain Gaps Through the Frequency Lens: Instead of struggling to decouple tangled spatial attributes (illumination, skin tone, background), the paper elegantly demonstrates that these diverse factors unify into low-frequency amplitude statistics, offering a simpler and more principled formulation for domain generalization.
- Harmonious Synergy Between Spatial Synthesis and Spectral AdaIN: By generating bounded images in pixel space and executing style mixing via AdaIN in frequency space, the framework simultaneously avoids heavy-tailed spectral optimization instability and guarantees label-preserving phase consistency.
- High Utility for Practical Deployment: The entire frequency perturbation and generation pipeline operates strictly as a training-time regularizer. At test time, PGNet is discarded, enabling high-accuracy, zero-overhead gaze tracking on resource-constrained embedded platforms.
Limitations & Future Work¶
- Unaddressed Spatial Domain Mismatches: ALFP specifically tackles frequency amplitude shifts, but does not explicitly resolve spatial domain gaps such as extreme head pose distribution biases, gaze angle range discrepancies, and severe facial occlusions.
- Symmetric Circular Frequency Masking: The low-frequency band is partitioned via isotropic circular regions, which overlooks potential directional preferences in real-world lighting and facial texture distributions. Future work could investigate learnable elliptical masks or orientation-selective filter banks.
Related Work & Insights¶
- vs PureGaze / Xu et al.: While PureGaze attempts feature-level disentanglement to suppress identity and illumination and Xu et al. generate spatial texture noise, ALFP operates directly on the Fourier amplitude spectrum via AdaIN, naturally isolating style from geometric phase without requiring complex orthogonal feature projections.
- vs CLIP-Gaze / LG-Gaze: Recent methods integrate massive Vision-Language Models (such as CLIP) to regularize the gaze feature space, introducing heavy parameter footprints and complex adaptation pipelines. ALFP with a standard ResNet-18 outperforms ViT-B/16-based VLM approaches across the board while introducing zero additional inference parameters.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering frequency sensitivity diagnostic for gaze estimation with an elegant spatial-synthesis / spectral-AdaIN hybrid pipeline]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-domain evaluations across four benchmarks, thorough frequency sensitivity profilings, and exhaustive ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, compelling diagnostic experiments, cohesive mathematical exposition, and rigorous empirical validation]
- Value: ⭐⭐⭐⭐⭐ [Improves cross-domain gaze estimation SOTA by up to 15% with zero inference compute overhead, offering strong practical and academic value]