DualCount: Structurally Consistent Density and Point Modeling for Zero-Shot Object Counting¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Object Detection
Keywords: zero-shot object counting, density map estimation, point prediction, optimal transport, geometric consistency
TL;DR¶
Addressing spatial ambiguity and background leakage in zero-shot object counting caused by weakly regulated density regression, DualCount presents an instance-aware dual-decoder architecture that softly decomposes the density map using predicted instance points and enforces per-instance mass conservation and center-of-mass alignment, establishing new state-of-the-art results on FSC-147 and PUCPR+.
Background & Motivation¶
Object counting has historically relied on category-specific supervision for constrained domains such as crowds, vehicles, or biological cells, which severely degrades when deployed across open-world distributions. To overcome category dependency, few-shot class-agnostic counting was introduced using visual exemplars (such as bounding boxes provided at test time). However, requiring human exemplar selection at inference creates a major bottleneck for scalable, fully automated systems. Text-specified zero-shot object counting has emerged as a compelling alternative: by querying visual-language models with free-form text descriptions (e.g., "flamingos" or "cars"), models predict the number of target instances in an image without category-specific training or visual exemplars.
Existing zero-shot counting methods predominantly employ text-conditioned density map regression. In this paradigm, global counts are obtained by integrating continuous scalar density maps supervised through pixel-wise loss or optimal transport matching. Despite their promise, unconstrained density formulations suffer from a fundamental structural limitation: they provide little to no explicit regulation over how density mass should be partitioned among individual object instances. When scenes contain crowded overlaps, self-similar distractors, complex non-convex geometries, or cluttered backgrounds, density mass frequently leaks into correlated background textures or breaks into fragmented, part-biased clusters centered around high-contrast regions.
This failure mode does not stem merely from weak cross-modal semantic alignment, but from the lack of instance-level geometric constraints within continuous density representations. Physically, each distinct instance should contribute approximately one unit of density mass, compactly concentrated near its true physical center. The core idea is to recast continuous density estimation as a structured mass allocation problem over a latent set of instances, coupling density regression with discrete point modeling to enforce per-instance mass conservation and center-of-mass alignment constraints.
Method¶
Overall Architecture¶
DualCount is built on an end-to-end multimodal dual-decoder architecture comprising five key components: a visual encoder \(E_v\), a text encoder \(E_t\), a cross-modal feature enhancement module \(\mathcal{E}\), a feature fusion module \(\mathcal{F}\), and a dual-decoder head consisting of a density decoder \(D_\rho\) and a point decoder \(D_c\). Unimodal representations from the image and text encoders are first enhanced via image self-attention and bidirectional cross-attention. The enhanced features are then fused via channel-wise feature modulation into a text-conditioned spatial representation. A lightweight shared refinement and upsampling head reconstructs full-resolution representations from which two complementary branches operate: \(D_\rho\) produces a continuous density map for global counting, while \(D_c\) predicts a discrete set of candidate instance points. Soft instance decomposition then bridges these two branches to enforce instance-level geometric consistency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input image I and text query t"] --> B["Image encoder Ev and text encoder Et<br/>Extract visual features Fv and text embedding ft"]
B --> C["Cross-modal feature enhancement module<br/>Image self-attention + bidirectional cross-attention"]
C --> D["Feature modulation fusion module<br/>Text-generated affine parameters modulate visual channels"]
D --> E["Shared upsampling and refinement head<br/>Bilinear interpolation and convolution restore resolution"]
E --> F["Dual-decoder decoupled prediction<br/>Density map ฯ and candidate point set P"]
F --> G["Soft instance decomposition<br/>Gaussian kernel weights compute instance-specific density components ฯi"]
G --> H["Structural geometric consistency constraints<br/>Per-instance mass conservation + center-of-mass alignment + count consistency"]
H --> I["Final predicted count N"]
Key Designs¶
1. Cross-Modal Feature Enhancement and Feature-Wise Modulation: Strengthening Text-Grounded Spatial Representations Raw visual features from vision backbones lack fine-grained spatial discrimination conditioned on arbitrary vocabulary text queries, making background regions prone to false-positive activations. DualCount builds a comprehensive bidirectional cross-modal enhancement module \(\mathcal{E}\). It first applies multi-head self-attention on visual tokens \(V \in \mathbb{R}^{(H'W') \times C}\) to capture long-range scene topology. Next, visual tokens attend to text embedding \(f_t\) via image-to-text cross-attention, while text features attend back to visual tokens via text-to-image cross-attention. In the subsequent feature fusion module, text embeddings generate affine transformation parameters: $$ \gamma = W_\gamma f_t^{\text{enh}}, \quad \beta = W_\beta f_t^{\text{enh}} $$ $$ F(x) = \gamma \odot \tilde{F}_v(x) + \beta $$ This feature-wise modulation dynamically shifts and rescales visual feature channels, actively suppressing irrelevant background dimensions while accentuating queried semantic categories.
2. Soft Instance Decomposition: Differentiable Bridge Between Continuous Density and Discrete Points Standard density maps \(\rho \in \mathbb{R}^{H \times W}\) represent an unconstrained continuous field devoid of instance ownership. Meanwhile, the point decoder outputs a confidence map \(h(x) = \sigma(S(x))\), and applies non-maximum suppression (NMS) and a confidence threshold \(\tau\) to yield a set of discrete instance center proposals \(\hat{\mathcal{P}} = \{\hat{p}_i\}_{i=1}^M\). To endow the continuous field with instance awareness, DualCount introduces a normalized Gaussian spatial kernel to define soft assignment weights: $$ w_i(x) = \frac{\exp\left(-\frac{|x - \hat{p}i|^2}{2\sigma^2}\right)}{\sum $$ Because weights satisfy }^M \exp\left(-\frac{|x - \hat{p}_j|^2}{2\sigma^2}\right)\(\sum_{i=1}^M w_i(x) = 1\) everywhere, the global density map is decomposed into \(M\) distinct instance-specific components: \(\rho_i(x) = w_i(x)\rho(x)\), where \(\sum_{i=1}^M \rho_i(x) = \rho(x)\). This soft decomposition converts a monolithic density map into an ensemble of differentiable, instance-attributed mass clusters.
3. Structural Geometric Consistency: Per-Instance Mass Conservation and Center-of-Mass Alignment Building upon the decomposed components, DualCount enforces two geometric constraints derived from first principles. First, Per-Instance Mass Conservation evaluates the total integrated mass belonging to instance \(i\): \(m_i = \sum_{x \in \Omega} \rho_i(x)\). Since each genuine physical instance corresponds to exactly one unit of mass in counting, the model minimizes the mass conservation loss: $$ \mathcal{L}{\text{mass}} = \frac{1}{M} \sum^M (m_i - 1)^2 $$ This regularizer prevents mass over-accumulation in background distractors or severe imbalance among detected instances. Second, Center-of-Mass Alignment computes the empirical centroid of each component \(\bar{p}_i = \frac{1}{m_i} \sum_{x \in \Omega} x \rho_i(x)\) and aligns it with its corresponding predicted discrete center \(\hat{p}_i\): $$ \mathcal{L}{\text{align}} = \frac{1}{M} \sum_i|^2 $$ Together, these dual constraints penalize spatial drift and part-biased allocation, compelling each component to form a compact cluster centered tightly at the object location.}^M |\bar{p}_i - \hat{p
Loss & Training¶
The framework is optimized end-to-end under a multi-task objective: $$ \mathcal{L} = \lambda_{\text{ot}}\mathcal{L}{\text{ot}} + \lambda}}\mathcal{L{\text{cst}} + \lambda}}\mathcal{L{\text{focal}} + \lambda}}\mathcal{L{\text{consistent}} + \lambda}}\mathcal{L{\text{mass}} + \lambda $$ - }}\mathcal{L}_{\text{align}Optimal Transport Loss \(\mathcal{L}_{\text{ot}}\): Supervised via ground-truth point annotations, encouraging global density distribution matching with minimal transportation cost. - Focal Loss \(\mathcal{L}_{\text{focal}}\): Supervises the point center confidence heatmap. - Contrastive Point Loss \(\mathcal{L}_{\text{cst}}\): Matches predicted points to ground-truth points via the Hungarian algorithm. Multi-scale bilinear sampled features across ResNet-50 layers (layer1 through layer4) form an InfoNCE contrastive objective that pulls matched point representations closer while pushing away unmatched background proposals. - Count Consistency Loss \(\mathcal{L}_{\text{consistent}} = (\hat{N} - M)^2\): Enforces agreement between total integrated density \(\hat{N} = \sum_{x \in \Omega}\rho(x)\) and proposal count \(M\). - Geometric Losses \(\mathcal{L}_{\text{mass}}\) and \(\mathcal{L}_{\text{align}}\): Bound the instance mass and geometric centroids as formulated above. Training is conducted with the Adam optimizer at a learning rate of \(6.25 \times 10^{-6}\). Backbones include CLIP or Swin Transformer with BERT.
Key Experimental Results¶
Main Results¶
On the large-scale open-category benchmark FSC-147 and cross-dataset transfer benchmarks CARPK and PUCPR+, DualCount demonstrates substantial improvements over prior zero-shot counting methods, matching or outperforming several few-shot methods requiring 3 manual visual exemplars.
| Dataset / Benchmark | Method | Setting / Shots | Val MAE โ | Val RMSE โ | Test MAE โ | Test RMSE โ |
|---|---|---|---|---|---|---|
| FSC-147 | CountGD (NeurIPS'24) | Zero-shot (0-shot) | 12.14 | 47.51 | 12.98 | 98.35 |
| FSC-147 | T2ICount (CVPR'25) | Zero-shot (0-shot) | 13.78 | 58.78 | 11.76 | 97.86 |
| FSC-147 | GeCo (NeurIPS'24) | Zero-shot (0-shot) | 14.81 | 64.95 | 13.30 | 108.72 |
| FSC-147 | CLIP-Count (ACM MM'23) | Zero-shot (0-shot) | 18.79 | 61.18 | 17.78 | 106.62 |
| FSC-147 | DualCount (CLIP) | Zero-shot (0-shot) | 14.12 | 54.27 | 12.33 | 99.78 |
| FSC-147 | DualCount (SwinT+BERT) | Zero-shot (0-shot) | 11.21 | 42.77 | 11.32 | 97.16 |
| FSC-147 | LOCA (ICCV'23) | Few-shot (3-shot) | 10.24 | 32.56 | 10.97 | 56.97 |
| FSC-147 | BMNet (CVPR'22) | Few-shot (3-shot) | 15.74 | 58.53 | 14.62 | 91.83 |
In cross-dataset generalization experiments: - CARPK: DualCount achieves 7.95 MAE and 9.78 RMSE, surpassing recent text-specified methods including CounTX (8.13 / 10.87), T2ICount (8.61 / 13.47), and CLIP-Count (11.96 / 16.61), while trailing only CountGD (3.83 / 5.41). - PUCPR+: DualCount achieves 24.27 MAE / 28.91 RMSE, outperforming CountGD (24.31 / 32.16), VLCounter (48.94 / 69.08), and CLIP-Count (55.82 / 91.23).
Ablation Study¶
The ablation experiments quantify the exact impact of each proposed loss term and architectural component on the FSC-147 test set:
| Ablation Config | \(\mathcal{L}_{\text{ot}}\) | \(\mathcal{L}_{\text{cst}}\) | \(\mathcal{L}_{\text{consistent}}\) | \(\mathcal{L}_{\text{mass}}\) | \(\mathcal{L}_{\text{align}}\) | Test MAE โ | Test RMSE โ | Note |
|---|---|---|---|---|---|---|---|---|
| Full Model | โ | โ | โ | โ | โ | 11.32 | 97.16 | Full dual decoders with all geometric consistency losses |
| w/o Center Alignment | โ | โ | โ | โ | โ | 12.84 | 100.45 | Removing \(\mathcal{L}_{\text{align}}\) increases MAE by +1.52 |
| w/o Mass Conservation | โ | โ | โ | โ | โ | 13.05 | 100.98 | Removing \(\mathcal{L}_{\text{mass}}\) worsens MAE by +1.73 |
| w/o Count Consistency | โ | โ | โ | โ | โ | 14.25 | 102.17 | Disconnecting count agreement raises MAE by +2.93 |
| w/o Contrastive Loss | โ | โ | โ | โ | โ | 14.49 | 103.11 | Missing discriminative point features causes severe degradation (+3.17) |
Architectural component ablation highlights: 1. Single decoder variants: Density decoder only yields 14.12 MAE; Point decoder only yields 14.54 MAE. Both underperform the dual-decoder framework (11.32 MAE). 2. Feature enhancer: Removing \(\mathcal{E}\) degrades MAE from 11.32 to 13.11. 3. Backbone selection: SwinT+BERT achieves 11.32 MAE / 97.16 RMSE, outperforming CLIP (12.33 / 99.78) thanks to rich multi-scale visual tokens and stronger linguistic context.
Key Findings¶
- Complementarity of density and points: Relying solely on continuous density integration (14.12) or discrete point detection (14.54) leads to high errors; combining them through soft decomposition drops MAE to 11.32, demonstrating that continuous mass and discrete centers reinforce each other.
- Crucial role of point feature contrast: The contrastive point objective \(\mathcal{L}_{\text{cst}}\) is the most vital individual loss term (removing it causes MAE to spike from 11.32 to 14.49), proving that false-positive suppression via Hungarian-matched contrastive learning is essential for stable geometric regularization.
- Robustness in cluttered and non-convex scenes: Qualitative density visualizations confirm that center-of-mass alignment and mass conservation eliminate spurious density leakage in challenging cases like flamingos, polka-dot backgrounds, and overlapping packaging.
Highlights & Insights¶
- Differentiable Continuous-to-Discrete Bridge: The soft instance decomposition mechanism maps continuous density fields to discrete instance points through normalized spatial Gaussian weighting, allowing geometric regularization losses to back-propagate gradients seamlessly into the continuous density map.
- First-Principles Geometric Regularization: Without requiring expensive mask annotations or external segmentation prompts, enforcing unit mass conservation (\(m_i \approx 1\)) and centroid-to-point alignment (\(\bar{p}_i \approx \hat{p}_i\)) effectively suppresses background leakage and part-biased splitting.
- Broad Transferability Across Spatial Tasks: The principle of structurally coupling continuous heatmaps with discrete keypoint proposals can be directly extended to other vision tasks requiring dense spatial accounting, such as crowd analysis, human keypoint estimation, and repetitive video motion counting.
Limitations & Future Work¶
- Sensitivity to Initial Point Proposals: Soft decomposition relies directly on the set of proposal points filtered by NMS and confidence thresholding. Severe occlusions leading to missed points or repetitive textures producing false detections can introduce inductive biases into the decomposition.
- Fixed Gaussian Spread Parameter: The assignment weight utilizes a fixed standard deviation \(\sigma\). In scenes featuring severe perspective distortion where object sizes vary across orders of magnitude, a static kernel width may over-smooth small instances or truncate large instances.
- Future Directions: Developing an instance-adaptive spatial bandwidth parameter or integrating weakly-supervised bounding box regressors could further improve structural consistency across multi-scale scenes.
Related Work & Insights¶
- vs CountGD (NeurIPS 2024): CountGD addresses open-world counting through unified detection grounding. While it shows strong performance on vehicle datasets like CARPK (3.83 MAE), DualCount exhibits stronger generalizability on diverse open-category targets in FSC-147 (11.32 vs 12.98 Test MAE) and fixed-camera surveillance in PUCPR+ (24.27 vs 24.31).
- vs CLIP-Count / VLCounter (ACM MM 2023 / AAAI 2024): Previous methods predict scalar density maps from text embeddings without instance-level constraints, leading to severe background leakage. DualCount curbs this structural deficiency, cutting FSC-147 Test MAE from ~17.5+ down to 11.32.
- vs Few-Shot Baselines (LOCA / BMNet): Few-shot approaches require users to provide 3 accurate bounding-box exemplars at inference time. DualCount relies purely on natural text queries and reaches 11.32 Test MAE, coming within striking distance of 3-shot LOCA (10.97).
Rating¶
- Novelty: โญโญโญโญ [Reformulates density regression into structured mass allocation with an elegant soft instance decomposition and dual geometric constraints]
- Experimental Thoroughness: โญโญโญโญโญ [Evaluated thoroughly across open-set benchmarks and cross-domain surveillance datasets with complete loss and architectural ablations]
- Writing Quality: โญโญโญโญโญ [Clear motivation chain, rigorous mathematical formulation with intuitive physical meanings, and tightly aligned illustrations]
- Value: โญโญโญโญโญ [Addresses the long-standing density diffusion and background leakage problems in zero-shot counting with high practical applicability]