Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning¶

Conference: ECCV 2026
arXiv: 2606.26973
Code: https://github.com/JiaheChen2002/GGR
Area: Self-Supervised / Representation Learning
Keywords: Open-Set Semi-Supervised Learning, Gradient Rectification, Optimization Geometry, Pseudo-Label Noise, Non-Adversarial Update

TL;DR¶

This paper proposes Geometric Gradient Rectification (GGR), a plug-and-play gradient-space rectification framework. Using the supervised gradient as an anchor, GGR projects conflicting unsupervised auxiliary gradients onto a safe half-space. This ensures that auxiliary updates remain first-order non-adversarial compared to the supervised direction on selected parameter blocks, simultaneously enhancing closed-set generalization and open-set robustness without relying on accurate OOD detection.

Background & Motivation¶

The realistic scenario of Open-Set Semi-Supervised Learning (OSSL) is that labeled data only cover known classes (in-distribution, ID), while the unlabeled pool contains out-of-distribution (OOD) samples. Existing methods generally fall into two main paradigms: filter-based methods (e.g., OpenMatch, ProSub) filter out suspicious samples using thresholds, entropy, or angles to avoid OOD contamination, but over-filtering can filter out hard ID samples, causing feature starvation; usage-based methods (e.g., IOMatch, MTCF) classify OOD samples into a unified unknown class to maximize data utilization, but erroneous pseudo-labels can push hard ID samples toward incorrect targets, generating auxiliary gradients that directly oppose the supervised learning direction (\(\langle g_s, g_u \rangle < 0\)).

Key Challenge: Sample-level ID/OOD separation is inherently unreliable—OOD and hard ID samples are difficult to distinguish in low-label regimes, and erroneous separation decisions manifest as destructive interference at the gradient level. This paper introduces a crucial perspective shift: instead of debating whether a sample should be used, the method directly checks whether the gradient generated by this sample harms supervised learning. From a gradient geometry perspective (as shown in the right panel of Figure 1 in the original paper), OOD gradients are mostly approximately orthogonal to the supervised direction (resembling random noise), whereas gradients from misclassified hard ID samples are truly directional adversaries, exhibiting a high negative correlation with the supervised direction. Key Insight: The robustness of OSL should not rely on fragile sample filtering; instead, geometric control can be applied directly in the gradient space to rectify only conflicting components while preserving orthogonal, useful signals.

Method¶

Overall Architecture¶

GGR is a pure gradient-space post-processing framework that does not modify any forward propagation or loss function definitions. Given a base OSSL method (e.g., FixMatch, IOMatch, DAC), in each iteration, the supervised loss \(\mathcal{L}_s\) and unsupervised auxiliary loss \(\mathcal{L}_u\) are computed as usual to obtain two gradients, \(g_s\) and \(g_u\). Then, geometric rectification is applied to \(g_u\) to obtain \(\tilde{g}_u\). Finally, the parameters are updated using the rectified combined gradient \(g_s + \lambda_u \tilde{g}_u\). The entire pipeline can be summarized in three steps: computing dual gradients, detecting conflicts and projecting, and combined updating. Because it is purely post-processing on gradients, GGR makes no assumptions about the pseudo-labeling mechanism, confidence thresholds, or data augmentation strategies of the base method and can be integrated as a plug-in into any OSR/OSSL method.

Key Designs¶

1. Gradient Conflict Diagnosis and First-Order Non-Adversarial Principle: From Sample Filtering to Gradient Control

The authors first formalize the source of gradient conflict in OSSL. Under the standard composite objective \(\min_\theta \mathcal{L}_s + \lambda_u \mathcal{L}_u\), when the pseudo-label of an unlabeled sample is incorrect, the gradient \(g_u\) generated by \(\mathcal{L}_u\) on this sample may have a negative inner product with \(g_s\), i.e., \(\langle g_s, g_u \rangle < 0\). This destructive interference weakens the ID discriminative features—which is more fatal than simple variance noise because its direction directly opposes the supervised signal. Thus, the first-order non-adversarial requirement is proposed: the rectified auxiliary direction \(\tilde{g}_u\) must satisfy \(\langle g_s, \tilde{g}_u \rangle \geq 0\), meaning that the auxiliary update must not oppose the supervised descent direction at the current step. This defines a closed half-space \(\mathcal{H}_{\text{safe}}(g_s) = \{d \in \mathbb{R}^D \mid \langle d, g_s \rangle \geq 0\}\) normal to \(g_s\), within which all rectification operations are geometrically constrained. The asymmetry of this principle is crucial—the supervised gradient acts as an immutable anchor verified by ground-truth labels, and only the auxiliary gradient is rectified. This distinguishes GGR from symmetric gradient surgery methods like PCGrad.

2. Vector-level Rectifier (VLR): Closed-form Solution for Half-space Projection

VLR is the most basic rectifier, directly using the current mini-batch's \(g_s\) as the anchor. Projecting \(g_u\) Euclideanly onto \(\mathcal{H}_{\text{safe}}(g_s)\) yields the closed-form solution:

\[\tilde{g}_u = \begin{cases} g_u - \frac{\langle g_u, g_s\rangle}{\|g_s\|_2^2} g_s, & \text{if } \langle g_u, g_s\rangle < 0 \text{ and } \|g_s\|_2 > 0 \\ g_u, & \text{otherwise} \end{cases}\]

When a conflict is detected (negative inner product), this formula precisely removes the components in \(g_u\) pointing toward \(-g_s\) while preserving the orthogonal component \(g_u - \frac{\langle g_u, g_s\rangle}{\|g_s\|_2^2} g_s\). This orthogonal component may still carry useful representation learning signals (such as learning ID intra-class variance). VLR satisfies two elegant mathematical properties: (1) Alignment identity \(\langle g_s, \tilde{g}_u \rangle = \max(0, \langle g_s, g_u \rangle) \geq 0\), which ensures the rectified inner product is non-negative; (2) Principle of minimal intervention—among all directions satisfying the constraint, VLR minimizes the modification (Euclidean distance) to \(g_u\). The advantages of VLR are zero hyperparameters and a computational overhead of only \(O(D)\) (one inner product plus one vector subtraction), making it the default recommended configuration.

3. Subspace-Aware Rectification (OSR and CSR): Smoothing Anchor Noise with Historical Gradients

The problem with VLR is that the mini-batch \(g_s\) is highly noisy in low-label regimes, making the single-step anchor unstable. To address this, the authors maintain a low-dimensional orthogonal basis \(U \in \mathbb{R}^{D \times k}\) composed of recent supervised gradients (incrementally updated via Gram-Schmidt + periodic QR orthogonalization). Replacing the instantaneous \(g_s\) with the subspace \(\mathcal{S} = \text{span}(U)\) as the anchor, they propose two variants:

Orthogonal Subspace Rectification (OSR): Projects \(g_u\) onto the orthogonal complement of \(\mathcal{S}\), \(\tilde{g}_u^{\text{orth}} = (I - UU^\top) g_u\). This is a more conservative choice—completely removing the component of \(g_u\) within \(\mathcal{S}\), which ensures \(\langle g_s, \tilde{g}_u^{\text{orth}} \rangle = 0\) (when \(g_s \in \mathcal{S}\)). Experiments show that OSR is more stable on open-set BA (Balanced Accuracy) and insensitive to the dimension \(d\).
Cone Subspace Rectification (CSR): Projects \(g_u\) onto the convex cone defined by the columns of \(U\), \(\mathcal{C}(U) = \{d \mid \langle d, u_i \rangle \geq 0, \forall i\}\), yielding \(\tilde{g}_u^{\text{signed}} = g_u - U \min(U^\top g_u, 0)\). This is a more permissive choice—only trimming the negative components that conflict with the directions of each anchor vector, while preserving positive and orthogonal components. CSR can achieve higher peak closed-set accuracy (e.g., 58.47% on CIFAR-100 when \(d=10\)) but is more sensitive to the quality of the basis.

Both subspace variants reflect a "conservatism-utilization" trade-off: OSR is safer but may discard useful signals, whereas CSR utilizes signals more fully but carries higher risk. The choice of dimension \(d\) is also crucial—too small fails to provide sufficient anchoring, while too large over-constrains and suppresses beneficial auxiliary signals. VLR remains the default option for optimal efficiency, while subspace variants are suitable for more stable performance when labels are extremely scarce.

A Complete Example: Gradient Conflict and Rectification on Hard ID Samples¶

Consider the 6/4 open-set split of CIFAR-10 with 5 labels per class. A hard sample belonging to the "cat" class (ID), because of image blur and feature similarity to the "dog" class, is mislabeled as "dog" by the base method FixMatch's pseudo-labeling mechanism. Under standard Softmax cross-entropy:

\(g_s\) (the supervised gradient from correctly labeled cat/dog/... samples) mainly aligns with the direction of enhancing the cat-dog decision boundary.
\(g_u\) (the auxiliary gradient from this mislabeled sample) pushes the model to move this sample from cat to dog—exactly forming a negative inner product with \(g_s\), \(\langle g_s, g_u \rangle \approx -0.3\|g_s\|\|g_u\|\).

After detecting the conflict, VLR computes the rectification term \(\frac{\langle g_u, g_s\rangle}{\|g_s\|_2^2} g_s\) (approximately the projection length of \(g_u\) along the \(-g_s\) direction) and subtracts it from \(g_u\). The inner product of the rectified \(\tilde{g}_u\) with \(g_s\) becomes zero—it no longer opposes the supervised direction, but the part of \(g_u\) orthogonal to \(g_s\) (such as learning general visual features like texture and background of this sample) is completely preserved. The final parameter update \(g_s + \lambda_u \tilde{g}_u\) advances the supervised objective without losing the representation learning value from the unlabeled pool.

Loss & Training¶

GGR does not modify the loss function definition of the base methods; it only intervenes in the gradient aggregation phase. The parameter update formula is \(\theta_{t+1} \leftarrow \theta_t - \eta(g_s + \lambda_u \tilde{g}_u)\), where \(\tilde{g}_u\) is generated by the selected rectifier (VLR / OSR / CSR). The rectification scope \(\mathcal{P}\) can be flexibly specified as the backbone, task-specific head, or all parameters; by default, only the backbone is rectified, while the head is updated with the original gradient to isolate representation-level conflict mitigation. All hyperparameters of the base methods (learning rate, \(\lambda_u\), confidence threshold, etc.) remain unchanged, ensuring that performance gains are attributable to the gradient rectification itself. Since VLR is a homogeneous function, the effect of \(\lambda_u\) before and after rectification is equivalent—it can be multiplied by the weight first and then projected, or projected first and then multiplied.

Key Experimental Results¶

Main Results¶

On CIFAR-10/100 (various seen/unseen splits + label budget combinations) and ImageNet-30 (20/10 split, 1% and 5% label ratios), GGR (default VLR) is integrated as a plug-in into three representative OSSL baselines: OpenMatch, IOMatch, and DAC.

CIFAR-10/100 Closed-set Accuracy (Abridged Representative Settings):

Dataset	Split	Labels/Class	IOMatch	+GGR	DAC	+GGR
CIFAR-10	6/4	5	89.87	91.87	86.69	87.63
CIFAR-10	6/4	25	93.61	93.64	93.02	93.25
CIFAR-100	50/50	5	59.49	60.61	56.94	57.19
CIFAR-100	20/80	5	56.48	56.75	53.30	53.45

ImageNet-30 Closed-set/Open-set Accuracy:

Evaluation	Label Ratio	OpenMatch	+GGR	IOMatch	+GGR	DAC	+GGR
Closed-set	1%	58.55	59.15	85.18	85.90	86.40	87.40
Closed-set	5%	86.72	87.43	90.32	90.82	93.25	93.33
Open-set BA	1%	14.75	15.56	74.88	75.46	77.78	78.40
Open-set BA	5%	61.51	68.15	81.03	82.14	87.81	88.74

The trend is clear: GGR achieves the most significant improvement under low-label regimes (fewer labels lead to worse pseudo-label noise and more frequent gradient conflicts) and is effective for both filter-based (OpenMatch) and usage-based (IOMatch, DAC) methods, validating that gradient-level control is a complementary tool independent of sample filtering strategies.

Ablation Study¶

Ablation of Rectification Scope (CIFAR10-6-30, VLR):

Method	Rectification Scope	Closed-set Accuracy	Open-set BA
DAC	—	86.69	72.02
+GGR	backbone	87.63	73.72
+GGR	head	90.38	75.24
+GGR	both	88.47	75.06
IOMatch	—	89.87	72.50
+GGR	backbone	91.87	74.13
+GGR	head	90.34	72.61
+GGR	both	89.56	71.99

Key Finding: The optimal rectification scope varies depending on the base method and task difficulty. When DAC has multiple auxiliary heads, rectifying the heads yields the highest gain; for IOMatch, backbone rectification is the most stable. Rectifying the backbone or the entire model serves as a robust default choice.

Effect of Subspace Dimension \(d\) (CIFAR-100 50/50): OSR maintains stable open-set BA performance across the entire \(d \in [1, 50]\) range, while CSR achieves its peak closed-set accuracy around \(d=10\) but exhibits higher variance. A moderate \(d\) (e.g., 10) achieves the best balance between conservatism and utilization.

Efficiency Analysis (Based on DAC, single-step time/memory): Compared to the baseline, VLR only increases time by about 11% and memory by ~300MB; OSR (\(d=10\)) has an overhead of roughly 1%, and CSR (\(d=10\)) about 11%. As the subspace dimension increases to 50, overhead grows linearly, but accuracy does not scale proportionally. Thus, VLR or the \(d=10\) subspace variant is recommended by default.

Key Findings¶

Gradient conflict diagnosis shows that the original conflict rate of baselines is non-zero and persistent. GGR reduces the residual conflict rate after updates to nearly zero, making the cumulative conflict regret almost flat—directly validating the practical effect of the first-order non-adversarial principle.
GGR shows the most prominent improvement in settings with the scanning labels (CIFAR-10 with 5 labels per class, ImageNet-30 with 1% labels), where the pseudo-label error rate is highest and gradient conflicts are most frequent. This aligns with the expectation that "the performance gain is most significant when the core idea is correctly addressed".
In comparison experiments with general optimization controls (PCGrad symmetric projection, gradient clipping, conflict discarding), GGR wins comprehensively across all settings. PCGrad is unstable due to symmetrically modifying the supervised anchor, while clipping/discarding loses useful signals. Only GGR's design of "asymmetric anchoring + removing only conflicting components" successfully preserves the valuable parts of both the supervised anchor and the auxiliary signals.

Highlights & Insights¶

Perspective shift is the core contribution: Shifting from the discrete decision of "whether a sample should be used" to the continuous rectification of "whether a gradient is harmful" bypasses the fundamental difficulty of ID/OOD inseparability in OSSL. This idea can be extended to any multi-task/multi-loss scenario where one objective might interfere with another.
The principle of minimal intervention for half-space projection: VLR only removes precisely conflicting components and preserves all orthogonal signals. Mathematically, it is the scheme that minimizes the modification to the original gradient under safety constraints. This design philosophy of "only discarding the bad without touching the good" is far superior to simple gradient clipping or discarding, and its closed-form solution yields an extremely simple implementation.
The conservatism-utilization trade-off of subspace anchors: The designs of OSR (removing all components in the subspace) and CSR (only trimming negative components) naturally form a 'safety-utilization' spectrum. Users can select the appropriate level of conservatism based on the noise level of the task. This analysis of the design space itself provides valuable reference for other gradient surgery methods.

Limitations & Future Work¶

The theoretical guarantee is first-order local—it only ensures that the auxiliary update does not oppose the supervised direction at the current iteration step, but does not guarantee global optimality or monotonic descent of the composite objective. When the mini-batch supervised gradient itself is highly noisy or carries zero information, the VLR anchor might become conservative or unstable (which is partially alleviated by the subspace variants).
GGR is a complement to, rather than a replacement for, existing OSSL methods. Stronger OOD detection or uncertainty estimation can still further enhance the effectiveness of the rectified auxiliary signal. GGR solves the "conflicting gradient" problem but does not address the issue of "meaningless auxiliary objectives caused by inherently incorrect pseudo-labels".
The choice of rectification scope (backbone/head/both) currently requires manual selection, and the optimal scope varies by base method and task. An adaptive scope selection strategy (e.g., dynamically deciding based on conflict frequencies across different layers) is a natural next step.
The authors have not validated the method on larger-scale datasets (such as full ImageNet-1K) or non-classification tasks (such as OSSL object detection/segmentation), leaving the generalization boundaries of the method yet to be defined.

vs PCGrad: PCGrad takes a step back for both conflicting gradients, which weakens the supervised anchor in OSSL scenarios. The key difference in GGR is asymmetric anchoring—the supervised gradient remains unmodified, and only the auxiliary gradient is rectified. The appendix proves that when two gradients are nearly in opposite directions, the supervised progress of PCGrad approaches zero, whereas GGR maintains \(\|g_s\|_2^2\).
vs OpenMatch/IOMatch/DAC: The common assumption of these methods is that ID and OOD can be distinguished through better detection/allocation strategies. GGR's stance is that "all samples can be safely utilized without explicit separation"—this provides a more fundamental solution that does not conflict with any detection improvements and can be combined with them.
vs Gradient Clipping (GradClip): Clipping uniformly scales all components, weakening both the good and the bad; GGR only removes components in the conflicting direction, preserving orthogonal signals. Experiments confirm that clipping performs the worst.
vs ConfDrop (Conflict Discarding): This method drops the entire \(g_u\) when \(\langle g_s, g_u \rangle < 0\) is detected, which is equivalent to degrading to pure supervised learning for that step—wasting unlabeled data and the useful signals in the orthogonal component.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The perspective shift from sample selection to gradient control is an original contribution to the OSSL field. The asymmetric anchoring design of half-space projection is simple and effective.
Experimental Thoroughness: ⭐⭐⭐⭐ It covers CIFAR-10/100 and ImageNet-30 under various splits and label budgets, three types of base methods, general comparative baselines, and efficiency analysis. However, verification on larger-scale datasets is lacking.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation is exceptionally clear (the gradient geometry analysis in Figure 1 is highly convincing), mathematical derivations are rigorous (with full proofs in the appendix), and the theory-experiment mapping is tight (ingenious metric designs like conflict diagnosis and cumulative regret).
Value: ⭐⭐⭐⭐⭐ Highly practical—plug-and-play, zero hyperparameters (VLR), and no modification to the forward pass. The idea of geometric control in gradient space is transferable to broader scenarios such as multi-task learning, continual learning, and robust optimization.