Skip to content

Local Spacing-Aware Hungarian Matching for Stable Point-Supervised Crowd Counting

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/kaijiang77/SAHCC
Area: Object Detection
Keywords: Crowd Counting, Point Supervision, Hungarian Matching, Assignment Stability, Local Spacing Prior

TL;DR

Addressing severe assignment ambiguity and cross-epoch training oscillation caused by globally uniform geometric matching costs in dense point-supervised crowd counting, this paper proposes SAH-matcher, which dynamically rescales target geometric matching costs via a k-NN local spacing prior, combined with CAS, HR, and IR process-level diagnostic metrics to substantially boost counting and localization accuracy with zero inference overhead.

Background & Motivation

Point-supervised crowd counting and individual localization are foundational problems in computer vision, underpinning intelligent surveillance, public safety, and urban transit analytics. Compared to conventional density-map regression paradigms that depend on heuristic Gaussian kernels and empirical post-processing, point-based set prediction frameworks such as P2PNet directly cast counting and localization into an end-to-end set matching problem. They predict discrete point proposals and employ the Hungarian algorithm to establish bijective one-to-one correspondences with ground-truth target points, eliminating the need for Non-Maximum Suppression (NMS) or density peak detection. However, point annotations merely designate head center coordinates without explicit bounding box scales or physical boundary extent, forcing standard frameworks to adopt a globally uniform, fixed-scale Euclidean distance metric across all targets in the Hungarian cost matrix.

The core tension stems from severe perspective scale variation and extreme spatial density heterogeneity in real-world crowd scenes. In sparse regions, target points are well separated, allowing a fixed Euclidean metric to clearly determine proposal ownership. In heavily congested areas, however, adjacent ground-truth heads are tightly packed in pixel space, rendering spatial displacement differences between a proposal and multiple nearby targets virtually indistinguishable. Under a globally shared geometric penalty, standard Hungarian matching frequently suffers from neighbor-induced misassignment and severe neighbor hijacking, where a proposal assigned to target A is geometrically closer to an adjacent target B. This matching ambiguity injects corrupt supervision into both classification and localization branches, triggering persistent cross-epoch assignment oscillations that severely impede model convergence.

To resolve this limitation, this paper recognizes that although point annotations lack explicit scale boxes, the spatial topology of the ground-truth point set inherently encodes rich geometric cues: the distance from any target to its nearest neighbors naturally mirrors the local crowd density and physical head scale. The core idea is to introduce a k-nearest-neighbor local spacing prior into the Hungarian cost formulation to dynamically rescale the geometric matching cost on a per-target basis, enforcing strict spatial selectivity in congested areas while preserving tolerance in sparse scenes, thereby stabilizing point-supervised training without altering the global one-to-one constraint or inference pipeline.

Method

Overall Architecture

SAH-matcher operates as a lightweight, plug-and-play replacement for standard Hungarian assignment during training. The underlying architecture builds upon an adapted P2PNet framework, where deep image features feed into tightly coupled classification and regression heads to generate proposal confidence scores and coordinate offsets. During backward supervision construction, SAH-matcher takes proposal predictions alongside precomputed target-wise local spacing priors, computes target-specific geometric modulation factors, dynamically scales the cost matrix entries, and solves for the optimal one-to-one assignment via standard Hungarian matching.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Point Annotations"] --> B["k-NN Local Spacing Prior Estimation<br/>Compute mean neighbor distance di"]
    B --> C["Target-Adaptive Cost Rescaling<br/>Dynamically scale geometric factor αi"]
    C --> D["Hungarian Injective Assignment<br/>Global one-to-one optimal matching"]
    D --> E["Set-Based Classification & Regression<br/>Supervise heads and backbone network"]
    D -.-> F["Process Diagnostic Suite<br/>Track CAS ambiguity and HR hijacking rate"]

Key Designs

1. k-NN Local Spacing Prior Estimation: Capturing local density and physical scale via neighbor distances To compensate for the lack of explicit scale annotations, this design extracts continuous local density representations directly from the spatial topology of the annotated points. For each ground-truth target point \(\mathbf{g}_i\), its \(k\) nearest spatial neighbors \(\mathcal{N}_k(i)\) (excluding itself) are retrieved in Euclidean coordinate space. The local spacing prior \(d_i\) is computed as the average distance to these neighbors: $\(d_i = \frac{1}{k} \sum_{\mathbf{g} \in \mathcal{N}_k(i)} \|\mathbf{g} - \mathbf{g}_i\|_2\)$ A small \(d_i\) indicates an intensely crowded neighborhood where heads severely overlap and demand tight spatial selectivity, whereas a large \(d_i\) corresponds to an isolated target in an open background. These distances can be efficiently precomputed offline and synchronously transformed under geometric augmentations such as random cropping and scaling, providing a consistent spatial geometric reference throughout training.

2. Target-Adaptive Cost Rescaling: Dynamically penalizing competitive misassignments in dense regions To overcome the inability of globally shared weights to resolve local competition, this design modulates the geometric entry of the Hungarian cost matrix with an adaptive target-specific scaling factor \(\alpha_i\). In practice, matching costs are calculated against fixed grid anchor coordinates \(\mathbf{a}_j\) as the proposal reference \(\mathbf{u}_j = \mathbf{a}_j\), decoupling assignment from noisy, unconverged predicted coordinate wander in early epochs. The target-specific weight \(\alpha_i\) is formulated as: $\(\alpha_i = \max\left(\alpha_{\min},\, \frac{\kappa}{d_i + \epsilon}\right)\)$ where \(\kappa\) is a global scaling factor, \(\epsilon\) avoids division by zero, and \(\alpha_{\min}\) imposes a lower bound to prevent geometric constraints from vanishing in extremely sparse regions. The total assignment cost between proposal \(j\) and target \(i\) is thus: $\(C_{j,i} = \lambda_{\mathrm{cls}} C_{j,i}^{\mathrm{cls}} + C_{j,i}^{\mathrm{geo}} = -\lambda_{\mathrm{cls}} \hat{c}_j + \alpha_i \|\mathbf{u}_j - \mathbf{g}_i\|_2\)$ In congested clusters, minimal \(d_i\) causes \(\alpha_i\) to scale up sharply, penalizing spatial displacements heavily and restricting the effective matching radius to the target's immediate vicinity. Conversely, larger spacing in sparse regions relaxes geometric penalties and maintains matching tolerance. Because modulation acts purely on scalar matrix entries, the global one-to-one injective formulation is fully preserved.

3. Process Diagnostic Suite: Quantifying within-epoch ambiguity, hijacking rates, and cross-epoch instability Prior studies primarily evaluated matching quality indirectly through final validation MAE/RMSE, lacking fine-grained instrumentation for assignment dynamics during optimization. This work formalizes a three-dimensional diagnostic suite comprising the Competitive Ambiguity Score (CAS), Hijacking Rate (HR), and Instability Rate (IR). Given optimal mapping \(\hat{\pi}\), let the matched proposal-target distance be \(d_i^+ = \|\mathbf{u}_{\hat{\pi}(i)} - \mathbf{g}_i\|_2\), and the distance from that proposal to its nearest non-assigned competing target be \(d_i^- = \min_{k \neq i} \|\mathbf{u}_{\hat{\pi}(i)} - \mathbf{g}_k\|_2\). Target-level ambiguity and image-level CAS are defined as: $\(\mathrm{CAS}_i = \frac{2 d_i^+}{d_i^+ + d_i^- + \epsilon}, \qquad \mathrm{CAS} = \frac{1}{M}\sum_{i=1}^M \mathrm{CAS}_i\)$ \(\mathrm{CAS} \to 0\) signifies distinct, well-separated matching; \(\mathrm{CAS} \approx 1\) marks ambiguous boundary competition; and \(\mathrm{CAS}_i > 1\) identifies a severe neighbor hijacking event where proposal \(\hat{\pi}(i)\) is closer to neighbor \(\mathbf{g}_k\) than to its assigned target \(\mathbf{g}_i\). The Hijacking Rate summarizes the proportion of targets experiencing severe hijacking: $\(\mathrm{HR} = \frac{1}{M} \sum_{i=1}^M \mathbb{I}[d_i^+ > d_i^-]\)$ Coupled with the Instability Rate (IR) measuring cross-epoch assignment churn, this diagnostic suite renders matching competition transparent, quantifiable, and comparable throughout model training.

Loss & Training

Following optimal injective assignment \(\hat{\pi}\), matched proposals form the positive set \(\mathcal{Q}^+\), while unmatched proposals constitute background negatives \(\mathcal{Q}^-\). Binary cross-entropy (BCE) supervises foreground-background classification, and an L1 loss supervises point coordinate regression on matched pairs \(\hat{\mathbf{p}}_{\hat{\pi}(i)}\): $\(\mathcal{L} = \frac{1}{Q} \left( \sum_{j \in \mathcal{Q}^+} \mathrm{BCE}(\hat{c}_j, 1) + \lambda_{\mathrm{neg}} \sum_{j \in \mathcal{Q}^-} \mathrm{BCE}(\hat{c}_j, 0) \right) + \lambda_{\mathrm{loc}} \frac{1}{M} \sum_{i=1}^M \|\hat{\mathbf{p}}_{\hat{\pi}(i)} - \mathbf{g}_i\|_1\)$ Models are optimized using AdamW with a batch size of 8, using a backbone learning rate of \(10^{-5}\) and head learning rate of \(10^{-4}\). Default hyper-parameters are set to \(k=4\), \(\kappa=1.2\), \(\alpha_{\min}=0.03\), \(\lambda_{\mathrm{cls}}=1\), and \(\lambda_{\mathrm{loc}}=2 \times 10^{-4}\). At test time, the matcher is entirely discarded, and predictions are produced directly via confidence thresholding with zero added runtime cost.

Key Experimental Results

Main Results

Evaluation across five major crowd counting benchmarks (ShanghaiTech Part A/B, UCF-QNRF, JHU-Crowd++, and NWPU-Crowd) demonstrates clear counting performance gains:

Paradigm Method SHHA (MAE/RMSE) SHHB (MAE/RMSE) UCF-QNRF (MAE/RMSE) JHU-Crowd++ (MAE/RMSE) NWPU (Test) (MAE/RMSE)
Detection-based TopoCount 61.20 / 104.60 7.80 / 13.70 89.00 / 159.00 60.90 / 267.40 107.80 / 438.50
Map-based DM-Count 59.70 / 95.70 7.40 / 11.80 85.60 / 148.30 - / - 88.40 / 388.60
Map-based MAN 56.80 / 90.30 - / - 77.30 / 131.50 53.40 / 209.90 76.50 / 323.00
Point-based Set Pred. P2PNet 52.74 / 85.06 6.25 / 9.90 85.32 / 154.50 - / - 83.28 / 553.92
Point-based Set Pred. CLTR 56.90 / 95.20 6.50 / 10.60 85.80 / 141.30 59.50 / 240.60 74.30 / 333.80
Point-based Set Pred. PET 49.34 / 78.77 6.19 / 9.69 79.53 / 144.32 58.50 / 238.00 74.40 / 328.50
Point-based Set Pred. APGCC 48.80 / 76.70 5.60 / 8.70 80.10 / 136.60 54.30 / 225.90 71.40 / 284.40
Point-based Set Pred. Ours (SAH) 47.23 / 75.17 6.14 / 9.50 76.91 / 135.92 58.32 / 250.62 68.90 / 306.70

For localization under strict distance criteria, SAH-matcher demonstrates substantial precision and recall advantages. On SHHA with strict \(\sigma = 4\) px, SAH achieves 48.9% F1 (outperforming APGCC's 48.7% and P2PNet's 40.6%). On NWPU under the strict threshold \(\sigma_s = \min(w, h)\), SAH reaches 72.1% F1, surpassing APGCC (68.9%) and P2PNet (67.5%).

To test portability across different frameworks, SAH-matcher was integrated as a plug-in into four point-based counting architectures without additional hyperparameter retuning:

Architecture Variant SHHA (MAE/RMSE) SHHB (MAE/RMSE) QNRF (MAE/RMSE) JHU (MAE/RMSE) Avg. \(\Delta\text{MAE}\)
P2PNet Baseline
+ SAH		 54.70 / 89.06
50.30 / 81.86		 6.29 / 10.18
6.11 / 9.97		 93.81 / 164.21
85.81 / 151.43		 62.23 / 269.70
60.85 / 264.30		 -
-3.49
CLTR Baseline
+ SAH		 66.53 / 114.74
65.20 / 108.89		 7.53 / 13.16
7.49 / 12.62		 93.49 / 166.72
91.11 / 154.90		 64.17 / 257.45
62.74 / 258.86		 -
-1.29
PET Baseline
+ SAH		 51.90 / 83.64
49.76 / 80.99		 6.56 / 10.23
6.76 / 10.95		 95.66 / 162.57
88.99 / 164.32		 59.98 / 252.52
59.05 / 250.93		 -
-2.39
P2R Baseline
+ SAH		 51.02 / 79.68
50.67 / 78.50		 6.95 / 11.54
6.68 / 10.51		 93.50 / 168.48
88.75 / 154.74		 62.23 / 272.66
63.45 / 267.91		 -
-1.03

Ablation Study

Ablations confirm that gains stem specifically from spacing-aware modulation rather than coarse tuning of a global geometric weight:

Configuration Mechanism Description SHHA MAE SHHA RMSE
Fixed global weight \(w = 0.01\) Overly permissive geometric penalty 51.82 85.64
Fixed global weight \(w = 0.04\) Moderately increased geometric cost 50.44 79.87
Fixed global weight \(w = 0.05\) (Baseline) Optimal global constant weight from grid search 50.06 81.57
Fixed global weight \(w = 0.07\) Overly restrictive geometric penalty in sparse scenes 50.53 81.68
SAH Dynamic Per-Target Rescaling Adaptive per-target scaling via neighbor spacing \(d_i\) 47.23 75.17
SAH + predicted coordinates reference (\(\mathbf{u}_j = \hat{\mathbf{p}}_j\)) Dynamic scaling evaluated on regressed point positions 51.63 86.84
Baseline + anchor coordinates reference (\(\mathbf{u}_j = \mathbf{a}_j\)) Fixed global weight evaluated on fixed spatial anchors 50.70 81.76

Key Findings

  • High Sensitivity in Dense Spacing Regimes: In the most congested bins (\(d_i \in [0, 10]\) px), the baseline matcher exhibits a high CAS of 0.63 and an HR of 0.16 (16% severe neighbor hijacking). SAH reduces CAS to 0.44 and slashes HR to 0.02 (an 87.5% reduction in hijacking), while halving the cross-epoch Instability Rate from 0.123 to 0.059. In sparse regimes, both matchers behave identically, verifying the targeted selective design.
  • Anchor Reference Synergy: Using fixed anchors \(\mathbf{a}_j\) prevents matching from chasing wandering early predictions, but under a fixed global weight it yields only modest improvement (MAE drops from 50.89 to 50.70). Only when coupled with SAH adaptive rescaling does anchor matching realize its full power, dropping MAE further to 47.23.
  • Negligible Computational Overhead: Precomputing spacing priors offline restricts runtime addition to \(O(M)\) scaling factors and an \(O(QM)\) elementwise product. Training latency increases by only 10.50% end-to-end on SHHA (104.44 ms to 115.41 ms/iter). Profiling on UCF-QNRF shows that relative matcher overhead declines from 15.89% to 3.82% as target density per crop rises, while test-time latency is completely unaffected.

Highlights & Insights

  • Turning Unsupervised Point Topology into Adaptive Matching Attention: Instead of hand-engineering complex scale bounding boxes or multi-stage anchors, SAH leverages simple k-NN Euclidean distances to introduce asymmetric spatial exclusivity into the Hungarian cost matrix, neatly resolving congested misassignments.
  • Interpretable Process-Level Match Diagnostics: The introduced CAS and HR metrics demystify matching dynamics during training, providing rigorous mathematical tools to detect neighbor hijacking that can readily generalize to other set-matching tasks (e.g., dense object detection and DETR variants).
  • Zero-Inference Plug-and-Play Portability: The method preserves standard Hungarian assignment complexity and leaves inference graphs unmodified, providing immediate performance gains across diverse point-based baselines.

Limitations & Future Work

  • Sensitivity to Extreme Annotation Corruption: Robustness stress-testing reveals that while SAH remains resilient under 10% point dropping or \(\le 4\) px coordinate jitter, severe noise (e.g., 8 px jitter) causes erroneous local density estimates, leading \(\alpha_i\) to explode and degrading MAE to 65.50. Restricting \(\alpha_i \le 0.08\) caps the divergence, highlighting reliance on reasonably clean point topologies.
  • Isotropic Density Assumption: Formulating local spacing as a scalar Euclidean average assumes isotropic crowd density. In scenarios with strong perspective tilt or anisotropic foreshortening, extending the prior to covariance-based or directional elliptical distance metrics could further enhance matching fidelity.
  • vs P2PNet [ICCV 2021]: P2PNet pioneered point-based set prediction with Hungarian assignment but used a fixed global geometric weight that fails in dense crowds. SAH maintains P2PNet's simplicity while eliminating dense-region misassignment and hijacking via local spacing calibration.
  • vs APGCC [ECCV 2024]: APGCC introduced auxiliary guidance points and implicit feature interpolation, adding internal structural complexity. SAH operates directly on the external cost matrix, achieving superior assignment stability and localization with far less architectural overhead.
  • vs CLTR / PET: CLTR and PET rely on neighborhood context reasoning and quadtree decomposition, incurring heavier model complexity. SAH shows that simple geometric cost rescaling achieves comparable or superior gains with minimal computing cost.

Rating

  • Novelty: ⭐⭐⭐⭐ [Derives intuitive local spacing priors from point topology to adaptively modulate Hungarian matching costs]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across five benchmarks, four point-based frameworks, with comprehensive ablations, noise stress-testing, and dynamic diagnostics]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formalisms, tight narrative flow, and well-supported empirical claims]
  • Value: ⭐⭐⭐⭐⭐ [A practical, zero-inference plug-in that provides valuable insights into set matching stability for crowded scenes]