WildProp: Visual Estimation of Wildlife Body Proportions at Scale

Venue: ECCV 2026
Paper: ECCV page
Code: github:cvl-umass/wildprop
Area: Self-Supervised / Representation Learning
Keywords: wildlife morphometrics, pose-aware retrieval, foundation-model features, geometric consistency, population proportion distributions

TL;DR

WildProp turns wildlife morphometrics into a retrieval-driven correspondence problem: the user provides a single annotated query image in a canonical pose, and the system uses DINOv3 for pose-aware retrieval, SAM2 to suppress background, RANSAC to filter geometrically inconsistent matches, and iterative keypoint-descriptor refinement to estimate population-level part-length-ratio distributions from iNaturalist-scale image collections. Across three physically measured datasets (AVONET birds, Shorebirds, Frogs) the median relative error is roughly 11-18%, and adapting to a new species or new body parts requires only one annotated query image, with no species-specific training.

Background & Motivation

Morphometrics is a foundational tool in ecology and evolution: bill length, tarsus length, and wing length ratios underpin macroevolution, community assembly, population-health assessment, and the empirical basis of classic evolutionary theory. Curated trait datasets such as AVONET demonstrate the value of large-scale morphological data, but collection remains stuck at two extremes: physically handling specimens or controlled photography (with rulers and standardized poses) is accurate but expensive, invasive, and yields only single-digit to low-teens individuals per species (AVONET averages fewer than 9 individuals per species); camera traps and aerial surveys target large mammals with narrow coverage. Meanwhile, citizen-science platforms like iNaturalist have accumulated massive image collections across taxa, geography, and seasons — seemingly an ideal data source. But direct geometric measurement on arbitrary photos is ill-posed: viewpoint change, articulation, occlusion, scale ambiguity, and perspective distortion mean a single image cannot even fix absolute length.

The key observation reframes the problem: most images are unsuitable for direct measurement, but a web-scale collection always contains a substantial number of images "close to the canonical pose", whose relative geometry is approximately preserved. Since geometry cannot be reconstructed from arbitrary viewpoints, the alternative is to selectively retrieve measurable instances and aggregate statistics — replacing "reconstruct geometry from ill-posed views" with "estimate distributions from a pose-friendly subset". Foundation models make this training-free: DINOv3 patch features handle pose-aware retrieval and keypoint matching, SAM2 with Grounding-DINO performs text-prompted segmentation to suppress background, and no per-species model is ever trained. Core idea: rather than solving 3D measurement under arbitrary viewpoints, use foundation-model features to retrieve a pose-aligned image subset, transfer annotations via dense patch matching with geometric-consistency filtering, and aggregate population-level proportion distributions — converting an ill-posed single-image measurement problem into a retrievable, filterable, aggregable statistical estimation problem.

Method

Overall Architecture

The inputs are threefold: one user-annotated canonical-pose query image, a keypoint set K, and a list of body parts P (each part is defined by two keypoints; at least two parts are required because ratios, not lengths, are estimated). The target set is a large uncurated image collection (typically same-species iNaturalist images, on the order of 10^4-10^5). The whole pipeline is training-free and runs in three serial stages: pose-aware retrieval finds target images with poses similar to the query; keypoint matching and geometric alignment transfer part endpoints from the query to each retrieved image; after iterative refinement, per-image part-length ratios are computed and aggregated into a population distribution (mean, median, variance). All ratios are computed relative to the longest part (e.g. wing length in birds) for numerical stability, and error is defined as the relative deviation between the estimated median and the physically measured median, E = |μ̂−μ|/μ.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Q["User query image<br/>keypoints K + parts P ≥ 2"] --> R["Pose-aware retrieval<br/>SAM2 background removal + DINOv3 descriptor"]
    C["Target collection 10⁴–10⁵ images"] --> R
    R -->|top 100| M["Keypoint matching & alignment<br/>patch-level nearest neighbor + RANSAC"]
    M -->|geometrically consistent images| I["Iterative refinement<br/>top-20 similarity-weighted descriptor update"]
    I -->|updated descriptors re-match| M
    I --> E["Per-image part-length ratios"]
    E --> S["Population distribution aggregation<br/>median / mean / variance"]

Key Designs

1. Pose-aware retrieval: turning "which images are measurable" into "which images look like the query pose"

Retrieving directly from iNaturalist mixes in many images that match the background but not the pose, so retrieval first segments the target with SAM2 (ViT-H) plus Grounding-DINO (text prompts "bird"/"frog") and zeroes the background, then extracts DINOv3 patch embeddings from the 518×518 segmented image and concatenates them into an image descriptor, taking the top-100 by cosine similarity. This stage only solves "rapidly find 100 candidates out of ~100k images"; fine-grained alignment is left to the next stage — a coarse-to-fine structure mirroring the classic SIFT retrieval pipeline's "global retrieval + local verification" division of labor.

The motivation is direct: monocular measurement is ill-posed under arbitrary viewpoints, but once poses are aligned, perspective distortion is controlled and 2D correspondences suffice. Although top-100 is a small fraction of the collection, it is often an order of magnitude larger than traditional morphological datasets in individual count (AVONET averages <9 individuals per species). The ablation (Table 4a) shows this is the precondition for measurability at all: without pose retrieval, randomly sampled images matched directly blow the AVONET relative error up from 12.2% to 216.1%.

2. Keypoint matching with RANSAC geometric filtering: keeping single-query annotations from drifting

For each retrieved image, a query keypoint (xᵢ,yᵢ) takes its nearest 16×16 patch's 768-dimensional DINOv3 embedding, finds the highest-cosine-similarity patch in the retrieved image, and uses that patch's center as the predicted correspondence. Single-point matching for symmetric parts (wing tips, legs, repeated segments) is inherently error-prone, so RANSAC fits a rigid transformation between the query keypoints and the predicted positions, discarding entire images whose outlier fraction exceeds a threshold. The thresholds are deliberately loose (inlier distance tolerance at 5% of the image diagonal, outlier-rate cap at 20%), because genuine morphological and postural variation between individuals also shows up as "inconsistency", and overly strict filtering would discard valid samples.

In implementation, retrieval and matching share the ViT-B/16 DINOv3; SAM2 is applied differently in the two stages — fully zeroing the background for retrieval (which wants a pure target-pose signal) versus halving background pixel intensity for matching (which wants patch-neighborhood context). The resolution ablation shows gains saturate beyond 518×518; the RANSAC fitting tolerance is 5% of the image diagonal.

3. Iterative refinement: adapting keypoint descriptors to the target population's visual diversity

A single query image's keypoint appearance features cannot cover the target population's visual diversity (lighting, plumage, subspecies differences), so direct transfer drifts systematically. The refinement step updates each keypoint's descriptor as a similarity-weighted average of "corresponding patch features from the top-20 retrieved images plus the original query features", then re-estimates correspondences with the updated descriptors, repeating for 3 rounds. This essentially reshapes the descriptor from "local appearance in one query image" into "a statistical representative of that part's appearance across the target population". It is especially effective when transferring across species or visually diverse subpopulations. The ablation shows iterative refinement contributes about 1 point of error reduction (12.2→11.3) — a modest but stable gain — and the hyperparameters (3 rounds, top-20, similarity weighting) were tuned once on the California Gull validation set and then frozen for all species and datasets, avoiding per-dataset overfitting.

4. Anchor strategy for cross-viewpoint ratios: indirect estimation through a shared anchor when parts need different views

The fronto-parallel assumption is the framework's soft spot: two parts of one ratio sometimes cannot both be fronto-parallel in the same view (e.g. bill length needs a side view, bill width a front view). The paper's handling is to introduce a common anchor part (such as wing or body length, visible in both views), estimate each part's ratio to the anchor separately, and compose them into the cross-viewpoint ratio. This softens the hard "one image, one view" constraint into a combinatorial "multiple query images + shared anchor" constraint, at the cost of requiring the user to annotate one query image per view and accepting that composition errors accumulate. Symmetric parts (left/right wings) naturally provide two measurements per instance, effectively doubling the retrieved sample size.

Loss & Training

There is no training in this paper: all components (DINOv3 ViT-B/16, SAM2 ViT-H, Grounding-DINO, RANSAC) are off-the-shelf inference, and the only "learning" happens at test time in the iterative refinement of keypoint descriptors (3 rounds, top-20 similarity-weighted averaging). Hyperparameters (retrieval top-100, RANSAC 5%-of-diagonal inlier tolerance, 20% outlier-rate cap, 3-round top-20 refinement) were selected once on the AVONET California Gull validation setup and then frozen globally, without per-dataset retuning. Experiments ran on a single NVIDIA A100. The CLIP variant uses ViT-B/16 and Stable Diffusion uses v3.5-medium.

Key Experimental Results

Main results

The three datasets correspond to birds (AVONET, five species), shorebirds (Alaska Science Center adult data, five species), and amphibians (University of Michigan Deep Blue frog data, five species). All experiments retrieve from research-licensed full-resolution iNaturalist images captured on or before 2025-12-31, with query images manually selected by the authors and annotated in Labelbox. Error is the relative deviation between the estimated median and the physically measured median.

AVONET species	Bill L./Wing L.	Tarsus/Wing L.	Wing W./Wing L.	Mean
Great Egret	16.7	8.6	9.0	11.4
Blue Jay	4.0	12.9	8.8	8.6
White-tailed Kite	15.9	15.7	7.7	13.1
Cape May Warbler	14.1	8.1	15.2	12.5
California Gull	0.4	14.9	18.0	11.1
Mean	10.2	12.0	11.7	11.3

Dataset	Mean part relative error	Notes
AVONET (birds)	11.3%	bill/tarsus/wing width relative to wing length
Shorebirds	11.0%	tarsus/culmen/head length/wing length; head length is worst at 14.5% due to a less distinctive back-of-head keypoint
Frogs	18.4%	eye-eye/eye-nostril/head width/nostril-nostril; head width worst at 21.8%

Data from Tables 1-3 of the paper. For frogs, Pacific Chorus Frog degrades across all parts (mean 27.3%), which the authors attribute to scarce canonical-pose images on iNaturalist for that species. The negative correlation between pose availability and error (Fig. 5) turns retrieval similarity into a usable performance predictor. One notable boundary: AVONET's "bill length from tip to skull base" is excluded because the skull base is covered by feathers and visually unmeasurable, so evaluation covers only the visually measurable protocol subset.

Ablations

Table 4a accumulates modules (AVONET relative error, %), while Tables 4b/4c compare feature choices. The causal structure of the accumulation is preserved rather than only quoting the final number:

Configuration (Table 4a cumulative)	AVONET relative error %
Query image only	12.2
Random sampled images, direct matching	216.1
+ pose retrieval	40.0
+ SAM background suppression	16.1
+ RANSAC filtering	12.2
+ iterative refinement (full)	11.3

Retrieval / matching variants (Tables 4b/4c)	AVONET relative error %
DINOv3-patch (default retrieval)	11.3
CLIP retrieval	17.1
BioCLIP retrieval	14.8
DINOv2-patch retrieval	10.1 (worse on validation, hence not adopted)
DINOv2 keypoint matching	13.5
Stable Diffusion matching	25.2
DINOv3+SD concatenated matching	11.2 (best)

• Direct measurement on random poses is entirely infeasible: 216.1% versus 40.0% shows that "picking the right images" is itself the dominant source of gains; retrieval is not an optional preprocessing step.
• Background suppression contributes the most: 40.0→16.1, because DINOv3 patch features are sensitive to background, making segmentation the precondition for reliable matching.
• RANSAC and iterative refinement are polish, not pillars: 16.1→12.2→11.3, each contributing 1-4 points.
• Semantic and geometric features are complementary: CLIP-style retrieval (17.1) is clearly worse than DINO (11.3), showing that purely semantic alignment loses pose information; yet SD features alone are poor (25.2) while concatenating them with DINOv3 is best (11.2), indicating semantic-appearance information has complementary value at the matching stage.
• Query keypoint noise tolerance is limited: stable below ~1% diagonal noise, but at 2-3% the tarsus ratio error jumps from about 10% to over 40% (Fig. 7b), making user annotation quality the practical bottleneck.
• Resolution gains saturate around 512; at low resolution small parts (the bill) are compressed into a single patch and error rises sharply (Fig. 7c).

Highlights & Insights

• Reframing "measurement" as "retrieval + aggregation": instead of solving an ill-posed problem on a single image, the method accepts that most images are unmeasurable and selects a pose-friendly subset for statistics — turning viewpoint variation from "a difficulty to solve" into "a condition to filter", shifting the problem from optimization to sampling.
• Training-free cross-taxon generalization: a new species or new parts need only one annotated query image; DINOv3 keypoint matching plus SAM2 segmentation spans birds to frogs to butterflies to deer antlers, whereas DeepLabCut-style methods require 100-200 labeled images per class for training.
• Practical value for stratified ecological analysis: the retrieved corpus can be stratified by geography and month (bill-length differences across Dunlin subspecies, and seasonal antler growth and shedding in Roe Deer, both reproduce known biological findings), which small-sample morphological datasets cannot support.
• Honest uncertainty characterization: Fig. 5 uses the negative correlation between retrieval similarity and error as a performance predictor, and Fig. 4 shows the estimated interquartile ranges are wider than the physical measurements — the authors do not spin "wider distributions" as an advantage, instead explicitly noting that small-sample physical measurements (n≤11) may equally underestimate true population variation.

Limitations & Future Work

• The authors acknowledge: error is higher than physical measurement (10-20% median relative error versus laboratory-grade precision), unsuitable for high-precision applications; only ratios are estimable, not absolute lengths (scale ambiguity); non-visual traits such as mass and skeletal dimensions are unmeasurable; citizen-science imagery carries sampling biases toward human-populated areas, observer preferences, and conspicuous species; some viewpoints (ventral views of small birds, in-flight poses) are inherently hard to photograph, limiting the corresponding ratios.
• Evaluation difficulty is bidirectional: physical datasets are small (n≤11), unevenly sampled, and follow measurement protocols not fully matched to image data, so direct comparison is itself imperfect; part of WildProp's advantage comes from estimating a population distribution while the physical side only offers a median from a tiny sample.
• No principled abstention mechanism: when retrieval fails or a part is missing, implausible keypoints or ratios are still output (the antlers-matched-to-ear-tips failure case); currently the only remedy is manually filtering antler-absent images.
• Self-observed boundaries: query images were manually selected and annotated by the authors, and whether real users can pick canonical-pose queries as reliably is unquantified; the fronto-parallel assumption excludes scenes with large perspective distortion, with no quantification of perspective residuals; case-study claims of "separating three species" and "consistent with expectations" are qualitative judgments without quantitative significance tests.
• Proposed directions (by the authors): dedicated animal pose/correspondence models replacing general foundation models, calibrated depth or explicit 3D keypoints to widen the measurable pose range, small amounts of physical measurement to calibrate scale, richer anatomical representations (part segmentation, curves, point clouds), and extension to mass inference and color traits.

Related Work & Insights

• vs traditional morphometric pipelines (AVONET etc.): physical measurement is precise but yields very few individuals per species (AVONET averages <9); WildProp trades 10-20% error for population distributions from hundreds of individuals per species with geographic/seasonal stratification. The two are precision-scale complements, not replacements.
• vs animal pose estimation (DeepLabCut etc.): fixed keypoint definitions plus 100-200 labeled images per class for training do not adapt to arbitrary user-defined parts; WildProp is training-free but sacrifices individual-level precision, delivering only population statistics.
• vs monocular depth / image-to-3D reconstruction: depth models target large rigid objects and background geometry and are unreliable for fine articulated structures; 3D animal reconstruction (e.g. SAM-3D-Animal) is unvalidated for precise morphometrics. WildProp does not solve 3D at all — it uses retrieval to constrain away viewpoint variation, sidestepping depth-scale ill-posedness.
• vs CLIP semantic retrieval: CLIP encodes semantic alignment but loses geometric pose, while DINOv3 patch features preserve geometry, explaining the retrieval-variant gap; SD features concatenated with DINOv3 are complementary at the matching stage, showing semantic and geometric features suit different stages.
• vs qualitative trait work in ecological CV (plumage color etc.): this paper fills in the quantitative morphometrics slot, and can be further stratified by age/sex/color/location attributes — the two lines complement rather than compete.

Rating

• Novelty: 4/5. Reframing morphometrics from "solving a measurement problem" into "retrieval-driven statistical estimation" with training-free, single-query adaptation to arbitrary species is rare in ecological CV.
• Experimental sufficiency: 4/5. Three physically measured datasets plus module/retrieval/matching/sample-size/noise/resolution ablations and multi-taxon case studies, but only five species per dataset and evaluation depends on small physical samples.
• Writing quality: 4/5. Assumptions, failure modes (ear-tip mismatches, Pacific Chorus Frog) and bias sources are stated explicitly; high candor.
• Value: 4/5. A practical paradigm for scaling morphological proxy studies, though precision (10-20%) and sampling bias limit direct use in high-precision research.