\(S^{2}\)-FracMix: Label-Preserving Self-Saliency Mixup Augmentation

Conference: ECCV 2026
arXiv: 2606.25784
Code: None (Project page: fracmix-data-augmentation.github.io)
Area: Object Detection / Image Classification
Keywords: Data Augmentation, Mixup, Saliency-Guided, Fractal Mixing, Self-Supervised Augmentation

TL;DR

S²-FracMix proposes a label-preserving self-saliency mixup augmentation framework: extracting multi-scale saliency patches within the same image, then mixing them back into the original image after rotation, blurring, and fractal texture injection, thereby avoiding cross-sample semantic interference; when paired with a multi-mode high-level mixing strategy, it comprehensively outperforms SOTA methods such as AdAutoMix on 7 benchmarks including classification, detection, and robustness, while introducing negligible training overhead.

Background & Motivation

Mixup-based data augmentation has become a standard approach to enhance the generalization ability of deep vision models. Classical methods (Mixup, CutMix, ResizeMix) generate new samples by linearly interpolating or regionally mixing two random images. However, such cross-sample mixing inherently introduces semantic interference—forcing the superposition of semantic regions from the source and target images can create synthetic samples with contradictory visual cues, which degrades the learned features of the model. To address this, saliency-guided methods such as SaliencyMix, PuzzleMix, and Co-Mixup have been proposed. They prioritize mixing the most discriminative regions to preserve semantic integrity, but at the cost of significant computational overhead (requiring online optimization of complex mask problems), which substantially increases training time and relies on high-end hardware. Another line of work is fractal augmentation (PixMix, DiffuseMix), which mixes fractal textures into the entire image to increase diversity. However, indiscriminately injecting fractal textures into the whole image disrupts the visual structures of key target objects, introducing unwanted distribution shifts that degrade model robustness.

Key Challenge: Existing methods fail to simultaneously achieve "sample diversity", "label consistency", and "computational efficiency". Cross-sample mixing sacrifices label consistency, saliency optimization sacrifices efficiency, and global fractal injection sacrifices structural fidelity.

Goal: To design a unified framework that preserves labels (intra-image operations), generates highly diverse augmented samples, and maintains extremely low computational overhead. The Key Insight is to constrain both "saliency guidance" and "fractal injection" inside a single image: saliency patches are not retrieved from other images, and fractal textures are only injected into saliency regions. By narrowing down the constraint scopes, the conflict is naturally resolved.

Core Idea: Augmenting an image using its own saliency regions: extracting multi-scale saliency patches, applying geometric/photometric transformations, injecting fractal textures, and re-inserting them back into non-salient locations. The entire process remains intra-image, preserving labels while increasing diversity, complemented by a lightweight multi-mode random mixing to cover auxiliary augmentation signals of different granularities.

Method

Overall Architecture

The input of \(S^{2}\)-FracMix is a standard training image, and the output is an augmented image through multi-scale self-saliency mixing and fractal injection, while the label remains unchanged. The overall pipeline consists of five stages: (1) computing the saliency map of the input image using the Spectral Residual method; (2) randomly sampling patches at two scales (\(h/2\) and \(h/4\)) and filtering them using a saliency threshold to select patches containing sufficient discriminative information; (3) mixing each accepted patch with a fractal texture (randomly sampled from a pre-computed database of 400 fractal images, mixed at \(\lambda=0.20\)); (4) applying transformations to the patches—random rotation (\(\pm 30^\circ\)) for salient regions and Gaussian blur for non-salient regions, then resizing them back to the original image dimensions; (5) overlaying all transformed patches with equal weights, and mixing the combination with the original image at a random ratio \(\alpha\) to obtain the final augmented image. During training, cross-sample modes such as Mixup, CutMix, and ResizeMix are randomly stacked to form a multi-mode high-level mixing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image I"] --> B["Spectral Residual Saliency Detection"]
    B --> C["Multi-scale Patch Extraction<br/>h/2 and h/4 scales"]
    C --> D["Saliency Filtering<br/>Threshold t ~ Uniform(0.5, 1.0)"]
    D -->|Accept| E["Fractal Mixing<br/>P_k^f = λF + (1-λ)P_k"]
    D -->|Reject| C
    E --> F["Transformation: Rotation + Blur<br/>T_k = R·S̃_k + B·(1-S̃_k)"]
    F --> G["Resize + Aggregation<br/>P_m = Σ R_k"]
    G --> H["Weighted Mixing Output<br/>Ĩ = αI + (1-α)/n_k · P_m"]
    H --> I["Multi-mode High-level Mixing<br/>Randomly select Mixup / CutMix / ResizeMix"]
    I --> J["Augmented Image Ĩ"]

Key Designs

1. S² Self-Saliency Mixing: Self-Augmentation Within the Same Image

This lies at the core of the proposed method, directly addressing the semantic interference introduced by cross-sample mixing. Traditional saliency methods (SaliencyMix, PuzzleMix) paste the salient region of image A onto image B. When the contents of the two images are incompatible (e.g., placing a cat's head on a car), the semantic consistency of the synthesized sample is disrupted, leading to ambiguous labels. In contrast, S² enforces that both the source and destination are the exact same image. Specifically, a spectral residual detector \(f(\cdot)\) is first applied to the original image \(I_i\) to compute a saliency map \(S_i = f(I_i)\), which is a single-channel heatmap indicating the pixel-wise discriminative importance. Then, patches \(P_k\) are randomly sampled at two scales \(\mathcal{P} = \{(h/2, w/2), (h/4, w/4)\}\). For each patch, its saliency mask \(S_k\) is normalized and binarized using a threshold \(t \sim \text{Uniform}(0.5, 1.0)\), preserving only those patches with sufficiently high saliency proportions: \(\sum \tilde{S}_k / (h_k w_k) \geq (1-t)\). This threshold sampling is elegantly designed: a larger \(t\) loosens the saliency constraint (accepting more patches), while a smaller \(t\) strictly filters for only the most salient patches. Randomizing \(t\) dynamically scales the augmentation intensity across iterations without manual tuning.

The accepted patches then undergo a transformation \(T_k\): a random rotation \(R(P_k^f, \theta)\) (\(\theta \sim \text{Uniform}(-30^\circ, 30^\circ)\)) is applied to the salient regions (\(\tilde{S}_k=1\)), while Gaussian blur \(B(P_k^f)\) is applied to the non-salient regions:

\[T_k = R(P_k^f, \theta) \cdot \tilde{S}_k + B(P_k^f) \cdot (1 - \tilde{S}_k)\]

The transformed patches are resized back to the original image dimensions \(R_k = \text{Resize}(T_k, (h, w))\), and all accepted patches are stacked with equal weights to mix with the original image:

\[\tilde{I}_i = \alpha I_i + (1-\alpha) \sum_{k=1}^{n_p} \lambda_k R_k, \quad \lambda_k = 1/n_p, \quad \alpha \sim \text{Uniform}(0, 1)\]

This design is highly effective because it simultaneously accomplishes three goals: (a) naturally preserving labels (intra-image operation); (b) forcing the model to learn scale-invariant features via multi-scale resizing; (c) providing rich geometric and photometric perturbations restricted precisely within their respective regions by the saliency masks, avoiding unnatural artifacts along the boundaries.

2. FracMix Fractal Injection: Injecting Textures strictly inside Salient Regions, Not Background Noise

A common issue with fractal-based methods (PixMix, DiffuseMix) is the indiscriminate mixing of fractal textures into the entire image—treating the background, noisy areas, and key objects equally. Consequently, texture alterations in non-discriminative regions drown out key visual cues of the objects, triggering distribution shifts. The key insight of FracMix is that fractal textures serve as effective regularization signals only when injected into salient regions, acting as meaningless noise when applied to the background. Therefore, FracMix strictly constrains fractal mixing within the selected saliency patches. Specifically, a fractal image \(F\) is randomly sampled from a pre-computed database of 400 fractal images \(\mathcal{F}\) and mixed with the patch \(P_k\):

\[P_k^f = \lambda F + (1-\lambda) P_k, \quad \lambda = 0.20\]

The mixed patch \(P_k^f\) then enters the S² transformation pipeline. Here, the weight \(\lambda=0.20\) is determined via validation sweep (as shown in Figure 5b of the paper). An overly small \(\lambda\) leads to under-regularization, while an excessively large \(\lambda\) (e.g., 0.50) dominates the original semantics and degrades accuracy—which perfectly aligns with the theoretical analysis in Section 4 stating that "the stability penalty term grows with \(\lambda^2\)".

Comparative experiments strongly support this design: on Stanford-Cars, global fractal injection (whole-image mixing) achieves 92.27% accuracy, whereas local fractal injection (FracMix) achieves 92.78% (Table 6). In security benchmarks, S²-FracMix consistently outperforms global PixMix in adversarial robustness (92.9% vs 89.2%) and calibration error (7.12% vs 8.1%) (Table 10). This indicates that limiting fractal textures to salient regions is a fundamental architecturally sound decision rather than a trivial hyperparameter trick.

3. Multi-mode High-level Mixing: Covering Multi-granularity Augmentation Signals via Random Collections

The authors observed an interesting phenomenon during experiments: relying solely on S²-FracMix yields solid improvements but leaves room for further gains. This is because S²-FracMix excels at providing "intra-image geometric and textural diversity" but lacks "cross-sample global semantic diversity"—the former teaches the model invariance to poses/textures of the same object, while the latter establishes decision boundaries between different classes. Thus, the authors propose randomly stacking three classical lightweight cross-sample modes on top of S²-FracMix: Mixup (global interpolation for cross-image smoothing), CutMix (patch replacement for local occlusion diversity), and ResizeMix (scaled overlay for resolution robustness).

Specifically, for each training batch, the framework randomly decides whether to apply S²-FracMix alone or to overlay additional combinations of \(M_m\) (Mixup), \(M_c\) (CutMix), and \(M_r\) (ResizeMix). The ablation studies (Table 5) show the complementarity of these modes: S²-FracMix alone achieves 81.73%/82.22% on ResNet-18/ResNeXt-50, which increases to 82.74%/84.91% when all three modes are overlaid (gains of +1.01% and +2.69%, respectively). Crucially, this high-level mixing utilizes only computationally cheap classical methods (Mixup/CutMix/ResizeMix), deliberately excluding expensive saliency methods like PuzzleMix and Co-Mixup. Replacing S²-FracMix with FMix drops performance significantly (to 80.24%/82.27%), demonstrating that the core gains originate from S²-FracMix itself rather than the multi-mode wrapper.

A Complete Example: Step-by-Step Augmentation of a CIFAR-100 'Cat' Image

Taking a 32x32 "Cat" image from CIFAR-100 as an example, the full process of S²-FracMix is outlined. First, the spectral residual detector computes a saliency map \(S_i\)—the cat's face and body yield high saliency values (~0.8-1.0) while the background stays close to 0. Second, multi-scale sampling is performed: at the scale \(h/2=16\), a patch covering the cat's face is cropped. With a random threshold \(t=0.65\), the ratio of salient pixels in the patch is 72%, which is higher than \(1-t=0.35\), so the patch is accepted. At scale \(h/4=8\), a patch covering mostly the background is cropped; its salient pixel ratio is only 12%, which is rejected, triggering re-sampling until an acceptable patch is found. Third, a fractal image \(F\) is randomly fetched from the fractal database, mixing with the accepted patch at \(\lambda=0.20\) (\(P_k^f = 0.20F + 0.80P_k\))—the cat's face thus gains mild fractal textures without losing its recognizable shape. Fourth, transformations are applied: the patch is resized to 32x32, the salient region (cat face) is randomly rotated (e.g., +18°), and the non-salient region is blurred (Gaussian blur, \(\sigma=2.0\)). Fifth, the transformed patch \(R_k\) and the original image are mixed with \(\alpha=0.35\)—yielding a final output where approximately 65% of the pixels come from the transformed multi-scale patch and 35% from the original image. Through this process, the model learns from a cat with a slightly rotated face, textured fur, and a blurred background, while the label remains "cat". Sixth (during training), if CutMix is randomly selected in the high-level mixing, a rectangular patch from another image is pasted on top, further enhancing occlusion robustness.

Loss & Training

Standard cross-entropy loss is used for training. Crucially, the augmentation occurs during the data loading stage rather than inside the loss function, meaning no extra loss terms are introduced. In Section 4, the paper presents a theoretical analysis from the perspective of Vicinal Risk Minimization (VRM). By decomposing the S²-FracMix augmentation into a structured transformation \(T_\theta(x)\) and an additive fractal perturbation \(\Delta_\theta(x) = (1-\alpha)\lambda(M_\theta \odot F)\), the second-order Taylor expansion approximates the VRM as:

\[\text{VRM}(h_\omega) \approx \underbrace{\mathbb{E}_{x,y}\mathbb{E}_\theta[\ell(h_\omega(T_\theta(x)), y)]}_{\text{Invariance Term}} + \underbrace{\frac{\lambda^2}{6} \mathbb{E}_x[\text{tr}(H_g(T_\theta(x)) \Sigma_{\text{loc}}(x))]}_{\text{Saliency Local Stability Penalty}}\]

The first term enforces prediction invariance of the model toward rotation, blur, and scale transformations. The second term is a data-driven Hessian regularization term—where \(\Sigma_{\text{loc}}(x)\) restricts the penalty to salient regions (where the mask \(M_\theta\) is non-zero). Consequently, the model is forced to be smooth only within key regions rather than undergoing uniform global regularization. This theoretical derivation explains why an excessively large \(\lambda\) degrades performance (as the penalty grows with \(\lambda^2\), causing over-smoothing) and why the improvement in robustness is more pronounced than clean accuracy (the stability penalty directly suppresses sensitivity to local perturbations).

Key Experimental Results

Main Results

S²-FracMix consistently outperforms the previous SOTA method, AdAutoMix, across three benchmarks (CIFAR-100, Tiny-ImageNet, and ImageNet-1K), covering both CNN (ResNet/ResNeXt) and Transformer (Swin/ConvNeXt/ViT) architectures.

Dataset	Backbone	Vanilla	AdAutoMix	Ours	Gain
CIFAR-100	ResNet-18	78.04	82.32	82.74	+0.42
CIFAR-100	ResNeXt-50	81.09	84.22	84.91	+0.69
CIFAR-100	Swin-T	78.41	84.33	85.35	+1.02
CIFAR-100	ConvNeXt-T	78.70	83.54	84.41	+0.87
Tiny-ImageNet	ResNet-18	61.68	69.19	70.38	+1.19
Tiny-ImageNet	ResNeXt-50	65.04	72.89	74.27	+1.38
ImageNet-1K	ResNet-18	70.04	70.86	71.37	+0.51
ImageNet-1K	ResNet-50	76.83	78.04	78.54	+0.50
ImageNet-1K	ViT-B	76.7	—	81.2	—

Stable leading performance is also observed on fine-grained classification (Caltech Birds-200, FGVC-Aircraft, Stanford-Cars), with a +1.27% improvement over AdAutoMix using ResNet-50 on Stanford-Cars. For transfer learning (ImageNet-1K pre-training followed by fine-tuning), ViT-B achieves 89.84% on CUB-200 (vs 88.76% for AdAutoMix) and 92.86% on Stanford-Cars (vs 91.38% for AdAutoMix).

Ablation Study

Configuration	ResNet-18	ResNeXt-50	Description
Vanilla (No Augmentation)	78.04	81.09	Baseline
Mixup Only (\(M_m\))	79.12	82.10	Single cross-sample mode
CutMix Only (\(M_c\))	78.17	81.67	Single cross-sample mode
ResizeMix Only (\(M_r\))	80.01	81.82	Single cross-sample mode
S²-FracMix Only	81.73	82.22	Core method without multi-mode
S²-FracMix + \(M_m\) + \(M_c\) + \(M_r\)	82.74	84.91	Full-mode combination (Ours)
Replacing S²-FracMix with FMix	80.24	82.27	Ablation: irreplaceable core

Key Findings: (1) S²-FracMix alone (81.73%) substantially outperforms any single classical cross-sample mode (the best being ResizeMix at 80.01%), verifying that self-saliency mixing is a strong regularization signal on its own; (2) Multi-mode scaling yields an additional +1.01% gain, which is much smaller than the +3.69% jump from Vanilla to S²-FracMix, confirming that the main contribution lies in S²-FracMix itself rather than the multi-mode wrapper; (3) Replacing S²-FracMix with FMix (global fractal mixing) drops performance significantly to 80.24%/82.27% (on par with AdAutoMix+MM), demonstrating the necessity of the proposed local fractal + self-saliency design.

In robustness evaluations (Corruption testing on CIFAR-100-C + FGSM adversarial attack), S²-FracMix outperforms AdAutoMix across clean accuracy (82.74% vs 81.55%), corruption accuracy (53.84% vs 51.44%), and FGSM error rate (72.52% vs 75.66%). The boost in corruption accuracy (+2.40%) is the most significant, aligning with the theoretical derivation that the stability penalty suppresses sensitivity to local perturbations.

Key Findings

• Saliency threshold \(t\) and fractal mixing coefficient \(\lambda\) are two key hyperparameters: \(t\) controls patch acceptance rate (smaller is stricter), while \(\lambda\) regulates the intensity of fractal texture injection. Experiments reveal \(t=0.5\) and \(\lambda=0.20\) as optimal, and both demonstrate stable performance within reasonable bounds—scaling \(\lambda\) from 0.20 to 0.50 only mildly drops performance from 82.74% to 82.22%, and \(t\) also shows minimal variation from 0.5 to 0.9.
• Sweeping database sizes from 100 to 500 images shows that performance saturates at 400 images (90.56% on Stanford-Cars), implying that the role of the database is to provide "sufficiently diverse structured noise" rather than dense coverage of a fractal manifold.
• Grad-CAM visualizations demonstrate that models trained with S²-FracMix generate more focused attention heatmaps with clearer boundaries. t-SNE plots exhibit better category boundaries, indicating that the model learns more discriminative features.
• Expected Calibration Error (ECE) decreases from AdAutoMix's 3.1% to 2.8%, indicating that the model is more accurate and has improved confidence calibration, which is critical for safety-critical applications.

Highlights & Insights

• Paradigm Shift toward "Self-Augmentation": Classical Mixup operates under the premise that blending two images produces a new sample. S²-FracMix breaks this design regime by demonstrating that extracting, transforming, and re-pasting patches within the same image is sufficient to generate rich augmented signals while naturally avoiding label drift. This design can be readily extended to tasks requiring pixel-level label alignment (e.g., semantic segmentation, keypoint detection) by adapting patch operations onto pixel-level annotations.
• "Pinpoint-Accuracy" Fractal Injection: Instead of generically showing "fractals are useful", this work mathematically and experimentally demonstrates that where you inject matters more than how much. The contrast between local and global fractal injection (92.78% vs 92.27%) might seem incremental (+0.51%), but in out-of-distribution scenarios (Corruption: 27.8 vs 30.5, Adversaries: 89.2 vs 92.9), the advantages are magnified. This translates to a reusable design principle: regularization signals should be concentrated inside regions with high information density.
• "Complementary Thinking" in Multi-Mode Combinations: The authors do not attempt to cover all augmentation needs using a single mode. Recognizing the blind spot of S²-FracMix (lack of cross-sample diversity), they utilize lightweight classical methods to fill the gaps. This strategy of "deploying a core method for main performance and using cheap methods to patch blind spots" is highly valuable for real-world production systems.

Limitations & Future Work

• Reliance on Saliency Detectors: Both patch extraction and fractal injection in S²-FracMix heavily rely on the quality of the spectral residual saliency detector. Although the paper claims that when the detector fails, rotation/blur/fractal transformations degrade gracefully and still provide regularization, this was only verified on small-scale datasets and not under extreme visual domains (e.g., medical imaging, low-light). Replacing the detector with more robust alternatives (e.g., attention maps from pre-trained ViTs) represents a straightforward improvement direction.
• Randomized instead of Learned Mode Selection: Currently, high-level mixing simply chooses mode combinations randomly. Introducing a lightweight gating network to adaptively select augmentation modes based on sample difficulty (e.g., using S²-FracMix for easy samples, and CutMix for hard samples to introduce occlusion diversity) could further boost performance, as acknowledged in the limitations.
• Limited to 2D Natural Images: The fractal database, saliency detection, and patch transformations are fully tailored for 2D images. Extending this framework to video requires spatiotemporal saliency detection and time-consistent fractal injection, whereas 3D (point cloud/voxel) requires redefining "saliency patches" and "fractal textures". These tasks represent non-trivial transitions.
• The Gains in Fine-grained Classification Warrant Closer Inspection: The improvement of S²-FracMix over AdAutoMix on Stanford-Cars (+1.37%/+1.27%) is more pronounced than on CIFAR-100 (+0.42%/+0.69%). This implies that self-saliency mixing is highly effective for fine-grained tasks where discriminative cues are extremely subtle, and local fractal injection may be aligning well with these nuances. However, the paper lacks detailed analysis on this specific mechanism.

Related Work & Insights

• vs SaliencyMix / PuzzleMix / Co-Mixup: All three use saliency-guided mixing, but across two images (pasting salient regions from image A to image B). S²-FracMix fundamentally differs by being an intra-image operation, preventing cross-sample semantic interference. Furthermore, PuzzleMix and Co-Mixup require online optimization (transportation problem/submodular optimization), which entails much higher computational overhead than S²-FracMix.
• vs PixMix / DiffuseMix / IPMix: All three utilize fractal textures but via global injection (whole-image fractal mixing). S²-FracMix's FracMix component limits the injection strictly within salient patches, preventing background noise from drowning out key objects. Direct comparisons in Table 10 show that the local strategy outperforms global fractal mixing on all benchmarks.
• vs SalfMix: The most closely related intra-image self-mixing method. While SalfMix also transfers salient regions to non-salient targets, it lacks the multi-scale patch creation and fractal injection. On CIFAR-100 (ResNet-18), SalfMix achieves only 78.35% compared to S²-FracMix's 82.74% (+4.39%), where the gap is mainly driven by the diversity in scale and texture dimensions.
• vs RandomMix: RandomMix also combines multiple modes (Mixup+CutMix+ResizeMix+FMix) but is limited by the individual quality of each component, reaching only 81.02% on CIFAR-100 (ResNet-18). S²-FracMix demonstrates that "a high-quality core mode supplemented by cheap, complementary modes" is far superior to "a simple stack of mediocre modes".

Rating

• Novelty: ⭐⭐⭐⭐ While combining self-saliency mixing and local fractal injection merges existing concepts, the design perspective of restricting both from the global level to the local level is highly insightful, and this is the first work proving local fractals superior to global ones.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extremely comprehensive, covering 7 datasets, 9 baselines, 5 downstream tasks (classification/fine-grained/detection/transfer/contrastive learning) alongside theoretical proofs, ablations, visualizations, and safety benchmarks.
• Writing Quality: ⭐⭐⭐⭐ Well-structured. The theoretical section decomposing the augmentation into transformations and perturbations under the VRM framework to derive Hessian regularizers is highly persuasive. The only minor drawback is the slightly condensed notation for ablation components (\(M_m\), \(M_c\), etc. require referencing the body text).
• Value: ⭐⭐⭐⭐ As a plug-and-play augmentation module, S²-FracMix introduces negligible training overhead (no online optimization, no extra networks, and the fractal database is pre-computed oncely). It performs consistently across architectures (CNN/Transformer) and tasks, offering exceptional utility. Its biggest value is a transferable design logic: regularization signals should be concentrated where information density is highest.

Related Papers

• [ECCV 2026] The Label Imitation Game: Turing Test Network for Zero-Shot Pseudo-Label Pruning
• [ICLR 2026] Point2RBox-v3: Self-Bootstrapping from Point Annotations via Integrated Pseudo-Label Refinement and Utilization
• [CVPR 2026] Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
• [NeurIPS 2025] ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detection
• [CVPR 2026] Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection