Skip to content

title: >- [Paper Note] SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation description: >- [ECCV 2026][segmentation][adversarial attack] A class-centric decision-based sparse black-box attack using discrepancy reward to severely degrade semantic segmentation with minimal pixel perturbations. tags: - ECCV 2026 - segmentation - adversarial attack - black-box attack - sparse attack date: 2026-09-19 content_hash: ee4257ad915ca4d0

SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/KAU-QuantumAILab/SegPAR
Area: Segmentation
Keywords: semantic segmentation / black-box attack / decision-based attack / sparse adversarial attack / reinforcement learning

TL;DR

Addressing the immense search space and misleading step-transition rewards in decision-based black-box sparse attacks on semantic segmentation, SegPAR introduces a class-centric exploration framework coupled with a discrepancy reward that penalizes misclassification reversion, achieving unprecedented MIoU degradation under extreme pixel sparsity.

Background & Motivation

Deep neural networks have achieved remarkable success across diverse visual perception tasks, yet adversarial attacks have repeatedly exposed their extreme vulnerability to subtle input perturbations. In safety-critical real-world deployments—such as perception systems for autonomous driving and robot navigation—adversaries rarely possess white-box access to network gradients, architectural topologies, or continuous confidence score distributions. Instead, they can only probe the victim model through discrete hard-label outputs. In this context, decision-based sparse black-box attacks, which aim to fool the network by perturbing only a tiny fraction of pixels using exclusively discrete label feedback, reflect realistic physical threat models such as sensor hardware flaws (e.g., dead or hot pixels) or localized physical artifacts (e.g., small patches on road signs).

However, migrating decision-based sparse attacks from image classification to semantic segmentation encounters severe structural challenges. In semantic segmentation, high-resolution inputs yield dense, multi-class spatial prediction maps rather than a single categorical label. Existing state-of-the-art sparse pixel attack frameworks, such as RFPAR, rely on an image-centric pixel accumulation paradigm where a reinforcement learning agent explores perturbation locations across the entire image space. This global approach inevitably causes query budgets to be squandered on large, easily perturbed background areas, leaving heterogeneous, class-specific decision boundaries across disparate objects severely under-explored.

Furthermore, conventional step-transition decision rewards suffer from severe feedback distortion in cumulative attack settings. Standard formulations reward any change in pixel prediction between successive steps. Consequently, when newly accumulated noise disrupts previously successful attacks and causes a misclassified pixel to revert to its ground-truth label (reversion failure), or causes a pixel to oscillate redundantly among different incorrect classes (unstable fluctuation), the conventional reward still provides a positive reinforcement signal. This counterproductive feedback misguides the agent toward suboptimal search trajectories and wastes valuable black-box queries. The core idea is: reframe dense sparse exploration into parallel class-centric local boundary searches, and introduce a discrepancy mask reward anchored to the initial clean prediction to strictly reward net misclassification gains while penalizing reversion failures.

Method

Overall Architecture

The SegPAR framework operates through five coordinated stages: class-centric state construction, mask-constrained location and color action sampling, victim model black-box discrete querying, discrepancy reward computation, and memory-driven policy optimization. Given an original input image, the victim model generates an initial discrete prediction map, from which class-wise binary masks and tight axis-aligned bounding boxes are extracted to crop the image into independent per-class local states. Next, conditioned on each class-wise state, the policy network leverages a masked spatial Softmax distribution to sample sparse pixel coordinates strictly within the target class contour, alongside Bernoulli decisions for extreme RGB channel values. After perturbing the image and querying the victim model, a discrepancy mask tracks deviations relative to the initial clean prediction, producing an isolated discrepancy reward per class that penalizes reversion. Finally, the policy is updated via REINFORCE and archives optimal candidates in an external memory buffer, triggering periodic resets upon convergence to incrementally accumulate perturbations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Original image & initial label prediction"] --> B["Class-centric state cropping<br/>extract predicted class masks & tight bounding boxes"]
    B --> C["Mask-constrained action sampling<br/>spatial masked Softmax coordinates & RGB decisions"]
    C --> D["Synthesize adversarial candidate & query victim model"]
    D --> E["Discrepancy reward computation<br/>track net misclassification gains against initial prediction"]
    E --> F["Memory update & policy optimization<br/>REINFORCE update & periodic restart upon convergence"]

Key Designs

1. Class-centric state representation: bypassing global search bottlenecks

To prevent query budgets from concentrating in large, easily perturbed background regions, SegPAR decomposes the global image space into class-specific sub-problems. Given an input image \(x \in \{0, \dots, 255\}^{C \times H \times W}\), querying the victim model yields an initial discrete prediction map \(y^{(0)} = f(x) \in \{0, \dots, K-1\}^{H \times W}\). For every class \(k\) present in the prediction set \(\mathcal{K}\), SegPAR constructs a fixed binary spatial mask: $$ M^k_{i,j} = \mathbb{I}\left[f(x)_{i,j} = k\right], \quad \forall (i, j) \in {0, \dots, H-1} \times {0, \dots, W-1} $$ To eliminate background distractions and reduce state dimensionality, an axis-aligned bounding box operator \(b_k = \mathcal{B}(M^k) = [i_{\min}^k, j_{\min}^k, i_{\max}^k, j_{\max}^k]\) crops the image into a class-specific local state \(x_k = \text{Crop}(x, b_k)\). The reinforcement learning policy network \(\pi_\phi\) is conditioned on \(x_k\), enabling parallel and focused exploration across distinct semantic decision boundaries throughout the image.

2. Mask-guided action sampling: eliminating cross-class signal contamination

Operating within cropped bounding boxes introduces a geometric complication: tight rectangular bounding boxes frequently contain pixels belonging to neighboring classes. Sampling coordinates via unconstrained distributions (such as the Gaussian sampling used in RFPAR) inevitably perturbs non-target pixels, contaminating the reward feedback and destabilizing policy optimization. To solve this, SegPAR predicts a 4-channel spatial feature map \(S_k \in \mathbb{R}^{4 \times h_k \times w_k}\) for each cropped state \(x_k\), using the first channel \(S_k^0\) as spatial logits. By intersecting with the cropped binary class mask \(m^k = \text{Crop}(M^k, b_k)\), the framework defines a masked Softmax distribution over all candidate spatial locations: $$ P_k = \text{Softmax}\left(S_k^0 \odot m^k - \infty \cdot (1 - m^k)\right) $$ Sampling \(n\) locations without replacement via \(I = \{I_1, \dots, I_n\} \sim \text{MultinomialNR}(p_k; n)\) guarantees that every perturbed pixel falls strictly within the true geometric boundary of class \(k\). For perturbation color assignment, the remaining three logit channels \(S_k^c\) (\(c \in \{1, 2, 3\}\)) are converted via Sigmoid activations into independent Bernoulli distributions \(z_{l,c} \sim \text{Bernoulli}(\text{Sigmoid}(s^c_{I_l}))\), mapping perturbations directly to \(\{0, 255\}\) extreme color values to maximize attack impact.

3. Discrepancy mask reward: resolving optimization distortions from reversion and oscillation

In cumulative perturbation pipelines, standard step-transition rewards \(R_s \propto \sum_{i,j} \mathbb{I}[f(\hat{x}^{(t)})_{i,j} \neq f(x^{(t)})_{i,j}]\) naively grant positive reinforcement whenever a pixel changes its label. Analyzing the state transitions between step \(t\) and step \(t+1\) reveals four distinct subsets: - Maintained misclassification (\(\mathcal{S}_{\text{maintain}}\)): pixels stably remaining misclassified from previous steps; - Reversion failure (\(\mathcal{S}_{\text{revert}}\)): previously misclassified pixels reverting back to their original ground truth due to interference from newly accumulated noise; - New misclassification (\(\mathcal{S}_{\text{new}}\)): pixels successfully flipped away from the original class in the current step; - Unstable fluctuation (\(\mathcal{S}_{\text{fluctuate}}\)): pixels oscillating between different incorrect classes without contributing to true attack progression.

Standard rewards assign \(+1\) to both \(\mathcal{S}_{\text{revert}}\) and \(\mathcal{S}_{\text{fluctuate}}\), generating misleading gradients. SegPAR overcomes this pathology by introducing a discrepancy mask anchored to the initial clean prediction, \(d^{(t)}_{i,j} = \mathbb{I}[f(x^{(t)})_{i,j} \neq y^{(0)}_{i,j}]\) and \(\hat{d}^{(t)}_{i,j} = \mathbb{I}[f(\hat{x}^{(t)})_{i,j} \neq y^{(0)}_{i,j}]\). The class-wise discrepancy reward is formulated as: $$ (R_d(x^{(t)}, \hat{x}^{(t)}))k = \frac{1}{o^2 \cdot n} \sum}^{H-1} \sum_{j=0}^{W-1} M_{i,j}^k \cdot \left(\hat{d{i,j}^{(t)} - d\right) $$ where }^{(t)\(o=5\) is a scaling factor. Under this formulation, the effective pixel-level reward \(r = \hat{d}_{i,j}^{(t)} - d_{i,j}^{(t)}\) yields \(r = 0\) for \(\mathcal{S}_{\text{maintain}}\), \(r = +1\) for \(\mathcal{S}_{\text{new}}\), \(r = 0\) for \(\mathcal{S}_{\text{fluctuate}}\), and an explicit penalty of \(r = -1\) for \(\mathcal{S}_{\text{revert}}\). This enforces strictly monotonic expansion of the misclassified territory and eliminates counterproductive search cycles.

4. Progressive memory updating and policy reset: ensuring robust multi-step convergence

To prevent the policy network from getting trapped in local optima during long iterative exploration episodes, SegPAR maintains an external best-candidate memory tuple \((x^*, d^*, r^*)\), where \(r^* = \frac{1}{|\mathcal{K}|} \sum_{k \in \mathcal{K}} (R_d)_k\). At each step, the policy parameters \(\phi\) are optimized using REINFORCE on the class-isolated discrepancy rewards. When the improvement in mean reward stagnates over a patience window of 1–2 iterations within convergence thresholds \([10^{-2}, 3 \times 10^{-2}]\), the current search phase terminates. The algorithm then formally commits the best state \(x^{(t+1)} \leftarrow x^*\) and \(d^{(t+1)} \leftarrow d^*\), reinitializes the policy agent, and commences the next accumulation step up to a maximum budget of 100 steps. This staged restart mechanism anchors intermediate progress and systematically scales cumulative perturbation effectiveness.

Key Experimental Results

Main Results

Under a strict budget of 1,000 forward queries per image and an average sparsity target around 5%, SegPAR was comprehensively evaluated on Cityscapes (19 classes), ADE20K (150 classes), and Pascal VOC2012 (20 classes) across CNN architectures (DeepLabV3, PSPNet) and Vision Transformer architectures (SegFormer, SETR).

Model Dataset Attack Method Robust MIoU (R.MIoU) ↓ Sparsity (%) ↓ Average Queries ↓
DeepLabV3
(Clean MIoU: CS 0.798 / ADE 0.377 / VOC 0.861)		 Cityscapes PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.433
0.228
0.176
0.101		 5.72%
5.43%
4.51%
3.33%		 997.3
988.8
965.3
993.3
ADE20K PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.172
0.118
0.106
0.080		 6.73%
5.15%
4.50%
3.29%		 993.5
997.3
998.1
966.2
VOC2012 PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.669
0.469
0.466
0.238		 5.91%
5.15%
3.92%
2.24%		 952.5
998.8
986.3
680.4
PSPNet
(Clean MIoU: CS 0.793 / ADE 0.380 / VOC 0.860)		 Cityscapes PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.391
0.246
0.153
0.057		 5.78%
4.50%
4.61%
3.53%		 996.3
987.0
990.5
919.0
ADE20K PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.199
0.139
0.143
0.087		 6.32%
5.54%
4.36%
3.11%		 983.6
997.0
999.0
965.3
VOC2012 PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.704
0.488
0.529
0.240		 4.85%
4.66%
3.99%
2.29%		 929.6
998.7
984.3
679.1
SegFormer
(Clean MIoU: CS 0.802 / ADE 0.412)		 Cityscapes PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.594
0.602
0.566
0.341		 10.51%
5.30%
4.51%
3.96%		 997.9
994.3
958.5
960.4
ADE20K PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.316
0.293
0.300
0.209		 10.10%
7.03%
4.29%
3.60%		 973.2
996.0
912.0
987.0
SETR
(Clean MIoU: CS 0.780 / ADE 0.397)		 Cityscapes PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.607
0.602
0.527
0.481		 9.14%
5.71%
4.53%
3.89%		 997.1
997.3
962.9
940.4
ADE20K PointWise
SparseEvo
RFPAR
SegPAR (Ours)		 0.284
0.270
0.285
0.249		 8.31%
5.20%
3.90%
3.50%		 966.5
997.3
996.5
990.9

Ablation Study: Generalization of Discrepancy Reward to Other Accumulation Attacks

To isolate the standalone impact of the discrepancy reward \(R_d\), it was incorporated into multiple existing pixel-accumulation baselines (Pixle, Sparse-RS, and original RFPAR) and compared directly against standard transition reward optimization \(R_s\).

Model Objective Attack Framework Cityscapes R.MIoU (Sparsity) ADE20K R.MIoU (Sparsity) VOC2012 R.MIoU (Sparsity)
DeepLabV3 Standard (\(R_s\))
Discrepancy (\(R_d\))		 Pixle
Pixle + \(R_d\)		 0.394 (4.69%)
0.411 (4.04%)		 0.140 (4.79%)
0.111 (4.72%)		 0.447 (4.77%)
0.279 (4.59%)
Standard (\(R_s\))
Discrepancy (\(R_d\))		 Sparse-RS
Sparse-RS + \(R_d\)		 0.172 (4.87%)
0.152 (4.87%)		 0.100 (4.83%)
0.092 (4.83%)		 0.423 (4.84%)
0.258 (4.84%)
Standard (\(R_s\))
Discrepancy (\(R_d\))		 RFPAR
RFPAR + \(R_d\)		 0.176 (4.51%)
0.210 (3.56%)		 0.106 (4.50%)
0.090 (3.48%)		 0.466 (3.92%)
0.277 (3.43%)
PSPNet Standard (\(R_s\))
Discrepancy (\(R_d\))		 Pixle
Pixle + \(R_d\)		 0.369 (4.69%)
0.383 (4.13%)		 0.168 (4.79%)
0.131 (4.69%)		 0.374 (4.77%)
0.246 (4.59%)
Standard (\(R_s\))
Discrepancy (\(R_d\))		 Sparse-RS
Sparse-RS + \(R_d\)		 0.153 (4.87%)
0.139 (4.87%)		 0.121 (4.83%)
0.092 (4.83%)		 0.536 (4.84%)
0.304 (4.84%)
Standard (\(R_s\))
Discrepancy (\(R_d\))		 RFPAR
RFPAR + \(R_d\)		 0.153 (4.61%)
0.171 (3.92%)		 0.143 (4.36%)
0.133 (3.19%)		 0.529 (4.00%)
0.295 (3.13%)

Key Findings

  • Class-centric exploration drives extreme attack efficiency: Compared to image-centric exploration in RFPAR, partitioning the attack space across semantic classes allows SegPAR to discover vulnerable pixels rapidly. On VOC2012, SegPAR collapses DeepLabV3 from 0.861 to 0.238 MIoU using only 680 queries and 2.24% sparsity, whereas RFPAR consumes 986 queries to reach only 0.466 MIoU at 3.92% sparsity.
  • Discrepancy rewards virtually eliminate reversion failures: Monitoring curves of reverting pixels demonstrate that standard rewards induce thousands of oscillating reversion events where pixels toggle between correct and incorrect classifications. In contrast, \(R_d\) suppresses reversion almost completely, maintaining a steady, monotonic growth of misclassified territory.
  • Synergy between class cropping and discrepancy reward: In high-resolution domains like Cityscapes, applying \(R_d\) to global RFPAR leads to sparse reward signals and search stagnation. However, when paired with SegPAR's class-wise cropping and masked sampling, the search space shrinks dramatically, enabling \(R_d\) to unleash its full potential (driving PSPNet down to 0.057 MIoU).
  • Competitiveness with white-box baselines: When standardizing compute budgets (equating 1 black-box query to 1 white-box gradient update), SegPAR consistently outperforms classic white-box PGD0 and closely tracks advanced white-box methods like sPGD under low-sparsity constraints (1%–3%), despite possessing zero gradient information.

Highlights & Insights

  • Decoupling dense predictions into class-wise manifolds: Rather than treating an entire scene as a single optimization target, SegPAR recognizes that semantic segmentation represents an ensemble of heterogeneous decision surfaces, pioneering a class-centric reinforcement learning exploration pipeline.
  • Formal mathematical decomposition of step transitions: The authors categorize step-to-step pixel transitions into four mutually exclusive sets, pinpointing why standard transition rewards produce counterproductive positive reinforcement and providing an elegant, highly effective correction via initial prediction anchoring.
  • Mask-aware spatial sampling for irregular geometries: Using spatial Softmax masking based on predicted class contours guarantees zero cross-class action contamination within bounding boxes, resolving a long-standing mismatch between rectangular neural outputs and non-rectangular semantic objects.

Limitations & Future Work

  • Dependence on initial prediction completeness: SegPAR derives class masks and bounding boxes from the victim model's initial prediction \(y^{(0)}\). If the model fails entirely to detect small or obscure objects on the clean image, SegPAR cannot instantiate attack branches for those missing categories.
  • Computational overhead in dense many-class scenes: In complex datasets like ADE20K featuring dozens of co-occurring classes per image, maintaining cropped states and running policy inferences across all classes increases runtime latency and GPU memory requirements during exploration.
  • Extension to targeted semantic manipulation: SegPAR is formulated as an untargeted attack aiming to maximize global MIoU degradation. Expanding this class-centric framework to targeted manipulations (e.g., systematically disguising pedestrians as road surface in autonomous driving) represents a critical future research avenue.
  • vs RFPAR (NeurIPS 2024): RFPAR operates globally on full-size images and samples coordinates from continuous Gaussian distributions, suffering from cross-class contamination and frequent reversion failures; SegPAR replaces this with class-wise cropping, masked categorical sampling, and discrepancy reward tracking, dramatically improving sparsity efficiency and degradation severity.
  • vs PointWise & SparseEvo (ICLR 2019 / ICLR 2022): Both PointWise and SparseEvo rely on initial global misclassification followed by greedy reversion or evolutionary pruning, parameterized by rigid Success Ratio (SR) thresholds; in dense segmentation, they require high sparsity budgets (5%–10%), whereas SegPAR achieves superior degradation at 2%–3% sparsity.
  • vs White-Box Sparse Attacks (PGD0 & sPGD): White-box methods optimize \(L_0\) perturbations via gradients and top-\(k\) projections but remain susceptible to non-convex local minima; SegPAR proves that black-box decision queries guided by structured reinforcement learning can match or exceed gradient-based baselines under realistic query constraints.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering systematic study of decision-based black-box sparse attacks for semantic segmentation, featuring elegant class-centric formulation and discrepancy rewards.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across CNN and ViT backbones, three benchmarks, adversarially defended models, and white-box baselines with in-depth ablation studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, insightful motivation analysis, and exemplary structural logic throughout.
  • Value: ⭐⭐⭐⭐⭐ Establishes a formidable and realistic black-box security benchmark for dense visual perception, providing critical insights for the defense of autonomous driving systems.