MV2GF: Multi-view Pedestrian Detection with a Visual Geometric Foundation Model¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://mv2gf.github.io
Area: Autonomous Driving
Keywords: multi-view pedestrian detection, visual geometric foundation model, bird's eye view, pointmap projection, cross-camera generalization

TL;DR¶

Addressing the failure of visual geometry reasoning and severe overfitting to perspective distortion patterns under unseen camera configurations in conventional multi-view pedestrian detection, MV2GF introduces Depth Anything 3 to fuse task-specific pedestrian semantics with foundational geometric representations and aggregates features into 3D voxels using metric 3D pointmaps, achieving state-of-the-art cross-camera generalization.

Background & Motivation¶

Multi-view pedestrian detection (MVPD) aims to localize pedestrians on a ground-plane bird's eye view (BEV) map from overlapping multi-camera streams, serving as an indispensable capability for video surveillance, autonomous driving, and embodied robotic navigation in dense, crowded environments. Prevailing deep learning architectures follow a bottom-up projection-and-aggregation paradigm: they extract 2D image features using per-view encoders, project these features into a 3D world space using calibrated camera intrinsics and extrinsics, fuse them into a unified BEV feature representation, and finally predict pedestrian occupancy. However, these frameworks exhibit fragile generalization when deployed to novel camera setups—such as unseen camera poses, mounting heights, or sensor geometries—suffering severe performance degradation.

This generalization bottleneck stems from two coupled technical limitations. First, existing methods struggle to capture reliable visual geometry across views under unseen camera configurations. Standard backbones (e.g., standard ResNet or single-view FPN) operate independently on each view and attempt to learn the 2D-to-3D geometric correspondence solely from the scarce camera arrangements available in small-scale MVPD datasets. When deployed to unseen camera geometries, this implicit mapping collapses, preventing the model from associating pedestrian features across disparate viewpoints. Second, existing projection pipelines induce strong model dependency on training-specific distortion patterns. Prevailing approaches rely on ground-plane perspective transformations (e.g., MVDet) or multi-plane homographies (e.g., SHOT), which force all image pixels onto predefined horizontal planes. This projection creates severe "shadow-like" distortions that smear pedestrian features outward along the projection rays. Prior networks implicitly rely on subsequent CNN layers to memorize and compensate for these specific distortion artifacts. Once testing camera configurations deviate, the distortion geometry shifts drastically, rendering the detector incapable of pinpointing pedestrian ground positions.

While modern query-based spatio-temporal Transformers in autonomous driving mitigate projection distortions, they presuppose identical camera configurations and fixed BEV canvas grids between training and testing, making them impractical for versatile surveillance and flexible multi-camera setups. This work tackles the challenge by tapping into feed-forward visual geometric foundation models pretrained on massive multi-view datasets. Core idea: leverage a visual geometric foundation model with real-world metric scaling to extract general-purpose cross-view geometric features and pixel-wise 3D pointmaps, fusing task-specific pedestrian cues with geometric priors via TGF and scattering features into 3D voxel space via FPA to eliminate shadow-like distortions and achieve robust cross-camera generalization.

Method¶

Overall Architecture¶

MV2GF takes synchronized multi-view images and calibrated camera parameters as input, producing a BEV pedestrian occupancy probability map along with sub-grid continuous coordinate offsets. The pipeline coordinates two core modules: the Task-specific and Geometric Information Fusion (TGF) module, which fuses task-specific foreground representations extracted by a trainable ResNet with cross-view geometric representations extracted by a frozen Depth Anything 3 (DA3) foundation model; and the Feature Pointmap Aggregation (FPA) module, which scatters multi-view features into a 3D voxel grid using metric 3D pointmaps from DA3, pools them across viewpoints without distortion, and compresses the vertical dimension to output the final BEV representation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Images & Camera Parameters"] --> B["Stage 1: Task-specific and Geometric Information Fusion<br/>Fusing ResNet pedestrian features & DA3 geometric tokens via FPN"]
    A --> C["Foundation Model Geometric Prediction<br/>Frozen DA3 predicts metric 3D pointmaps"]
    B --> D["Stage 2: Feature Pointmap Aggregation<br/>Projecting pixel features into 3D voxels via pointmaps & max pooling"]
    C --> D
    D --> E["Vertical Compression & Detection Head"]
    E --> F["Predicted BEV Occupancy Map & Offset Map"]

Key Designs¶

1. Task-specific and Geometric Information Fusion (TGF): Balancing Foreground Saliency and Cross-view Geometric Reasoning Standard MVPD architectures deploy isolated CNN backbones for each view, failing to model inter-view relationships early during representation learning. Conversely, fully fine-tuning a massive geometric foundation model is computationally prohibitive and risks catastrophic forgetting of its rich geometric priors, while the raw foundation model lacks task-specific sensitivity toward pedestrians. TGF resolves this dilemma through a decoupled dual-branch design. The foundation model (DA3 Giant) is kept strictly frozen, providing general-purpose geometric tokens across views from four selected Transformer blocks \(\{E_l^n\}_{l \in \{20,28,34,40\}}\) with channel dimension \(C_D = 1536\). Concurrently, a lightweight, trainable ResNet-18 is optimized end-to-end to extract multi-scale task-specific features \(\{F_2^n, F_3^n, F_4^n, F_5^n\}\), prioritizing foreground pedestrian regions. For each stage \(k \in \{2, 3, 4, 5\}\), the geometric feature \(E_{l_k}^n\) is bilinearly resized to match the spatial dimensions of \(F_k^n\) and concatenated along the channel axis:

\[F_k'^n = [F_k^n \,\|\, \text{Resize}(E_{l_k}^n)] \in \mathbb{R}^{(C_k + C_D) \times \frac{H}{2^k} \times \frac{W}{2^k}}\]

These hybrid representations are then processed by a top-down Feature Pyramid Network (FPN), progressively blending task-specific semantic activations with cross-view geometric context into a unified high-resolution feature map \(F'^n \in \mathbb{R}^{C \times \frac{H}{4} \times \frac{W}{4}}\) with \(C = 256\). This hybrid fusion enables the network to maintain reliable inter-view correspondences even under completely unseen camera poses and focal lengths.

2. Feature Pointmap Aggregation (FPA): Distortion-free Projection via 3D Pointmap Voxelization Conventional homography-based projection forces every pixel of a 2D image onto a flat plane, scattering background structures, building facades, and upper-body pedestrian features across the ground plane into radial streaks. This forces the detection head to overfit specific artifact patterns. FPA eliminates planar homographies entirely by exploiting the dense, metric 3D pointmaps \(P^n \in \mathbb{R}^{3 \times H_D \times W_D}\) predicted by DA3. For each pixel \((u, v)\) in view \(n\), \(P^n(u, v) = (X, Y, Z)\) provides its estimated metric coordinates in the 3D world reference frame.

FPA resizes the fused image features \(F'^n\) to match the pointmap resolution and flattens all viewpoints into an aggregate feature pool \(\dot{F} \in \mathbb{R}^{C \times (N \cdot H_D \cdot W_D)}\) paired with their spatial coordinates \(\dot{P} \in \mathbb{R}^{3 \times (N \cdot H_D \cdot W_D)}\). A 3D voxel grid \(V \in \mathbb{R}^{C \times \frac{X}{4} \times \frac{Y}{4} \times Z}\) is instantiated with horizontal voxel dimensions \(\theta_x = \theta_y = 10\text{ cm}\), vertical resolution \(\theta_z = 50\text{ cm}\), and vertical bin count \(Z = 4\), spanning the real-world height interval of \([0, 2.0\text{ m}]\). For each voxel \((x, y, z)\), FPA identifies the set of all projected points \(S(x, y, z)\) falling within its geometric bounds and performs channel-wise max pooling:

\[V(x, y, z) = \begin{cases} \underset{s \in S(x, y, z)}{\max} \dot{F}(s), & \text{if } S(x, y, z) \neq \emptyset \\ \mathbf{0}, & \text{if } S(x, y, z) = \emptyset \end{cases}\]

The voxel grid is subsequently reshaped to \(\dot{V} \in \mathbb{R}^{(C \cdot Z) \times \frac{X}{4} \times \frac{Y}{4}}\) and collapsed along the height axis by a convolutional block into a compact BEV feature map \(B \in \mathbb{R}^{C \times \frac{X}{4} \times \frac{Y}{4}}\). Because feature vectors are deposited exclusively into their true 3D spatial bins, non-ground background elements no longer project onto the ground grid, thoroughly immunizing the detector against camera-specific projection distortion patterns.

Loss & Training¶

The detection head predicts a low-resolution BEV pedestrian occupancy map \(M \in \mathbb{R}^{1 \times \frac{X}{4} \times \frac{Y}{4}}\) followed by a sigmoid activation, as well as a continuous sub-grid offset map \(O \in \mathbb{R}^{2 \times \frac{X}{4} \times \frac{Y}{4}}\). The composite training objective balances BEV detection and per-view auxiliary supervision:

\[\mathcal{L} = \mathcal{L}_{\text{det}} + \mathcal{L}_{\text{off}} + \frac{1}{N} \sum_{n=1}^N \mathcal{L}_{\text{view}}^n\]

Here, \(\mathcal{L}_{\text{det}} = \text{FocalLoss}(M, \bar{M})\) penalizes ground occupancy discrepancies against Gaussian-smoothed ground-truth heatmaps, while \(\mathcal{L}_{\text{off}} = \text{L1Loss}(O, \bar{O})\) supervises grid discretization offsets. To expedite pedestrian localization within the trainable ResNet, single-view occupancy heatmaps for pedestrian heads and feet are predicted from \(F'^n\) and supervised with per-view focal losses:

\[\mathcal{L}_{\text{view}}^n = \text{FocalLoss}(M_{\text{heads}}^n, \bar{M}_{\text{heads}}^n) + \text{FocalLoss}(M_{\text{feet}}^n, \bar{M}_{\text{feet}}^n)\]

The model is trained using the Adam optimizer with an initial learning rate of \(1.0 \times 10^{-3}\), decayed to \(1.0 \times 10^{-6}\) via a cosine annealing schedule, with an effective batch size of 16 maintained through gradient accumulation.

Key Experimental Results¶

Main Results¶

To evaluate cross-camera generalization, models were trained on the multi-scene, multi-camera GMVD dataset under the GMVD-D split (6 scenes with 11 distinct camera configurations) and tested in a zero-shot transfer manner on unseen camera geometries in the GMVD-D test split (1 scene, 2 configurations), the dense synthetic benchmark MVPerception, and the real-world dataset Wildtrack. Standard metrics include Multiple Object Detection Accuracy (MODA, primary), Precision (Prec.), Recall (Rec.), and Multiple Object Detection Precision (MODP) with a 0.5-meter evaluation threshold.

Training Set	Test Dataset	Metric	MV2GF (Ours)	Prev. SOTA (MSMVD)	Gain
GMVD-D	GMVD-D (unseen cameras)	MODA / MODP	84.8 / 83.1	80.2 / 81.3	+4.6 / +1.8
GMVD-D	GMVD-D (unseen cameras)	Prec. / Rec.	96.8 / 87.7	95.7 / 83.9	+1.1 / +3.8
GMVD-D	MVPerception (unseen cameras)	MODA / MODP	86.5 / 80.6	81.8 / 79.4	+4.7 / +1.2
GMVD-D	MVPerception (unseen cameras)	Prec. / Rec.	98.5 / 87.8	94.2 / 87.2	+4.3 / +0.6
GMVD-D	Wildtrack (real-world transfer)	MODA / MODP	87.9 / 79.6	85.7 / 79.3	+2.2 / +0.3
GMVD-D	Wildtrack (real-world transfer)	Prec. / Rec.	94.7 / 93.2	92.2 / 93.6	+2.5 / -0.4

Under the conventional in-domain setting where training and testing share identical camera configurations (GMVD-S, MVPerception, Wildtrack), MV2GF achieves 91.3, 96.5, and 94.7 MODA respectively, outperforming or matching previous highly specialized detectors (e.g., MSMVD at 91.1, 96.4, and 94.6).

Ablation Study¶

All ablation models were trained on the GMVD-D training split and evaluated on the GMVD-D unseen camera test split to assess component contributions:

Ablation Category	Configuration	MODA (%)	MODP (%)	Prec. (%)	Rec. (%)	Note
Incremental Modules (Tab. 3a)	Baseline (MVDet + ResNet18)	73.4	78.5	93.2	79.1	pure perspective homography baseline
Incremental Modules (Tab. 3a)	+ TGF (DA3 geometric features)	77.8	79.5	93.8	83.3	+4.4 MODA improvement over baseline
Incremental Modules (Tab. 3a)	+ FPA (DA3 pointmap projection)	81.3	82.6	95.5	85.4	+7.9 MODA improvement over baseline
Incremental Modules (Tab. 3a)	Full MV2GF (TGF + FPA)	84.8	83.1	96.8	87.7	cumulative gain of +11.4 MODA
Feature Sources (Tab. 3b)	ResNet only	81.3	82.6	95.5	85.4	lacks cross-view geometric foundation cues
Feature Sources (Tab. 3b)	DA3 only	77.5	77.9	95.1	81.7	lacks pedestrian task specialization
Feature Sources (Tab. 3b)	ResNet + DA3 (TGF)	84.8	83.1	96.8	87.7	complementary semantic-geometric synergy
Voxel Vertical Resolution (Tab. 3d)	\(\theta_z = 200\text{ cm}\) (\(Z=1\))	83.7	82.9	96.1	87.3	over-compression along height
Voxel Vertical Resolution (Tab. 3d)	\(\theta_z = 50\text{ cm}\) (\(Z=4\), default)	84.8	83.1	96.8	87.7	optimal balance of resolution & occupancy
Voxel Vertical Resolution (Tab. 3d)	\(\theta_z = 12.5\text{ cm}\) (\(Z=16\))	81.6	81.7	96.3	84.8	severe voxel sparsity degrades aggregation
Foundation Choice (Tab. 3f)	MapAnything	83.0	82.3	96.5	86.2	outperforms conventional SOTA (80.2)
Foundation Choice (Tab. 3f)	Pi3X	82.3	82.6	96.4	85.5	validates framework generality
Foundation Choice (Tab. 3f)	Depth Anything 3 (DA3)	84.8	83.1	96.8	87.7	superior metric geometry yields best results

Key Findings¶

FPA pointmap projection yields greater generalization gains than feature enhancement: Replacing planar homography with 3D pointmap voxel aggregation (FPA) boosts MODA by +7.9 points alone, demonstrating that shadow-like projection artifacts represent the primary bottleneck in cross-camera transfer. Adding TGF yields an additional +3.5 point gain, culminating in an overall +11.4 point leap.
Geometric foundation models decisively surpass generic visual self-supervision: Swapping DA3 in TGF with general visual foundation models of comparable scale such as DINOv2 (82.9 MODA) or DINOv3 (83.2 MODA) results in noticeable performance drops compared to DA3 (84.8 MODA), confirming that explicit 3D geometric awareness—rather than mere large-scale image representation learning—is essential.
High resilience against camera view dropouts: When the number of available camera views \(N\) is reduced from 6 to 2, the performance of previous SOTA MSMVD plummets from 82.7 to 34.0 MODA (a 48.7-point drop), whereas MV2GF maintains 57.6 MODA (only a 28.3-point decline), demonstrating strong robustness under sparse viewpoints and severe occlusions.

Highlights & Insights¶

From planar homography approximation to metric 3D pointmap scattering: Instead of projecting pixels onto heuristic flat planes and suffering from perspective streaks, MV2GF grounds feature projection in metric 3D point clouds predicted by a foundation model, eliminating projection distortion at its physical root.
Decoupled task-specific and geometric representation: Freezing the foundation model preserves universal multi-view geometric awareness while a lightweight ResNet specializes in pedestrian localization, providing an effective blueprint for adapting 3D foundation models to dense downstream tasks without catastrophic forgetting.
Broad transferability to multi-camera perception: The FPA mechanism can seamlessly replace manual grid sampling and rigid spatial cross-attention across broader multi-camera perception domains, including autonomous vehicle surround-view BEV perception and cooperative infrastructure sensing.

Limitations & Future Work¶

Inference latency overhead: Executing the 40-layer DA3 Giant backbone incurs additional latency, leading to an inference speed of 3.3 FPS on an Nvidia A100 GPU compared to 4.5~5.2 FPS for traditional CNN-based baselines. While sufficient for typical 2.0 FPS surveillance streams, future work could explore distillation or weight quantization for real-time edge deployment.
Sensitivity to severe geometric depth failures: In scenes characterized by specular reflections or untextured surfaces where the foundation model produces systematic 3D pointmap drift, features can be scattered into erroneous voxels. Introducing depth uncertainty weights into voxel pooling could mitigate this issue.
Minor localization precision gap in fixed-camera setups: Under strict in-domain evaluations with unchanged cameras, MV2GF slightly trails MSMVD in MODP due to MSMVD's multi-scale BEV feature hierarchy, suggesting a promising integration of multi-scale BEV grids with metric pointmap projection.

vs MVDet / SHOT / MVDeTr: Traditional methods rely on single- or multi-plane homographies that inherently generate shadow-like perspective streaks, forcing networks to overfit camera-specific distortion patterns. MV2GF replaces homographies with metric 3D pointmaps, boosting unseen-camera MODA by over 10~15 points.
vs MSMVD: While MSMVD achieves high MODP under fixed camera layouts using multi-scale BEV features, its reliance on homographies limits its unseen-camera performance on GMVD-D to 80.2 MODA. MV2GF achieves 84.8 MODA and outperforms MSMVD by 23.6 MODA in sparse 2-view settings.
vs Autonomous Driving BEV Transformers (BEVFormer, SurroundOcc): Autonomous driving architectures presuppose fixed sensor rigs and predefined query grids, failing to adapt dynamically to diverse surveillance cameras. MV2GF ingests explicit camera parameters into a general foundation model, operating without rigid query priors.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of a visual geometric foundation model into multi-view pedestrian detection, fundamentally overcoming the longstanding cross-camera generalization barrier.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-scene (GMVD), cross-domain (MVPerception), and real-world (Wildtrack) evaluations, paired with deep ablations covering modules, feature sources, block selections, voxel resolutions, foundation variants, view sparsity, and latency.
Writing Quality: ⭐⭐⭐⭐⭐ Structured, coherent, and mathematically rigorous presentation with clear causal logic.
Value: ⭐⭐⭐⭐⭐ Highly impactful for practical deployment of multi-camera surveillance and infrastructure-assisted autonomous driving systems across dynamically changing camera layouts.