Region-Aware Multimodal Interleaving for Animal Re-Identification¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ML-4-SocialGood/RAMI.git
Area: Multimodal VLM
Keywords: animal re-identification, region modelling, multimodal interleaving, fine-grained discrimination, computer vision for social good
TL;DR¶
To overcome background overfitting and the limitations of expressing fine-grained animal coat patterns in plain text, this paper proposes RAMI, a framework that mines biologically representative foreground regions via lightweight statistical image processing and interleaves regional visual tokens with text placeholders in a shallow Transformer to achieve dense cross-modal interaction, substantially outperforming prior SOTA methods on benchmark and in-the-wild datasets.
Background & Motivation¶
Animal Re-Identification (Animal ReID) aims to recognise individual animals across diverse camera views and over extended time spans, providing an indispensable technological foundation for wildlife ecological monitoring, population estimation, and long-term behavioural tracking. However, compared with conventional person and vehicle re-identification scenarios that benefit from structured canonical poses and rigid geometries, wild animals exhibit drastic non-rigid postural deformations, severe mutual or vegetative occlusions, and pronounced cross-camera environmental shifts. Current mainstream approaches predominantly rely on holistic visual representations, which inadvertently cause networks to overfit to ambient surrounding backgrounds such as foliage, rocks, and mud. Consequently, models latch onto spurious environmental correlations rather than learning invariant, identity-discriminative animal traits; even pattern-based models often extract global generalised textures that remain fragile to background intervention and partial occlusion.
In parallel, foundation vision-language models such as CLIP have recently been introduced to Animal ReID to provide high-level semantic supervision. Nonetheless, existing methods typically restrict cross-modal integration to post-hoc global feature concatenation or coarse sentence-image contrastive alignment. Crucially, the fine-grained visual evidence that distinguishes individual animals—such as subtle stripe configurations, unique facial markings, and distinct fur contours—is inherently difficult to comprehensively verbalise in plain textual descriptions. While multimodal interleaving has achieved remarkable success in Large Multimodal Models (LMMs) for multi-image generative and sequential reasoning tasks, its potential for fine-grained discriminative representation learning within pretrained foundation vision-language architectures has remained completely unexplored.
This paper tackles these challenges by shifting multimodal semantic grounding from global image representations down to biologically salient local regions through an explicit visual-textual interleaving sequence. Core idea: develop an unsupervised, statistically controlled region mining pipeline to discover identity-relevant anatomical patches, project them into text placeholder tokens, and perform dense cross-modal self-attention inside a shallow Transformer, achieving accurate, interpretable individual animal re-identification with negligible parameter overhead.
Method¶
Overall Architecture¶
The RAMI framework reformulates Animal ReID as a dense cross-modal interaction process over biologically informative, localised image regions. The overall pipeline operates across four interconnected stages: first, an image-driven, unsupervised region mining process identifies salient foreground patches from the input image; second, these patches are encoded by a SigLIP vision encoder and projected into the textual token space via a lightweight TokenProjector; third, the regional visual tokens are interleaved into structured text prompts containing species and identity placeholders, prepended with a learnable pooling token; fourth, a single-layer Transformer Interleaver facilitates dense bidirectional cross-modal attention, yielding an identity-discriminative representation trained via a progressive multi-task objective. During inference, the region mining pipeline and textual modality are completely discarded, allowing query and gallery matching to rely exclusively on the fine-tuned SigLIP vision backbone for maximum deployment efficiency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image Sample"] --> B["Image-Driven Biological Region Mining"]
B --> C["Region-Aware Visual Tokenisation & Projection"]
C --> D["Region-Referenced Interleaved Token Grounding"]
D --> E["Shallow Transformer Dense Interleaved Attention"]
E --> F["Alignment Loss & Joint Discriminative Optimisation"]
A -.->|Inference: Vision Backbone Only| G["SigLIP Visual Embedding & Cosine Matching"]
Key Designs¶
1. Image-Driven Biological Region Mining: Parameter-Free and Statistically Controlled Pattern Extraction
To eliminate reliance on expensive manual keypoint or bounding box annotations while completely insulating the model from background interference, RAMI introduces an autonomous, lightweight image processing pipeline. The procedure begins by isolating the animal foreground using an off-the-shelf segmentation model (e.g., SAM 3), discarding environmental clutter. The segmented mask undergoes grayscale normalisation to remove colour bias, followed by Sobel gradient magnitude filtering to quantify local texture intensity variations corresponding to distinctive stripes, spots, or facial boundaries. To prevent false positive detections caused by wild illumination and posture variations, RAMI frames pixel importance selection as a rigorous hypothesis testing problem. Specifically, it computes a one-tailed empirical \(p\)-value for each candidate pixel score \(s_l \in S_i\): $\(p(s_l) = \frac{|\{s_k \in S_i : s_k \geq s_l\}|}{L}\)$ The Benjamini–Hochberg (BH) procedure is then applied to enforce a strict 5% False Discovery Rate (FDR) control at a significance threshold of 0.01%. By taking the dataset-level statistical mode of candidate region counts per image, RAMI reliably extracts a stable set of informative anatomical regions \(R_i = \{r_{i,1}, \dots, r_{i,N}\}\) without introducing any learnable parameters.
2. Region-Aware Visual Tokenisation and Region-Referenced Interleaved Token Grounding: Unified Cross-Modal Semantic Anchoring
Because complex and irregular natural patterns cannot be faithfully captured by generic language descriptions alone, RAMI explicitly grounds visual region patches into structured textual contexts. Each extracted region \(r_{i,n}\) is fed into the SigLIP vision encoder to yield local patch embeddings \(\mathbf{r}_i \in \mathbb{R}^{N \times pats \times d_v}\). A lightweight linear projection module, TokenProjector \(\mathcal{P}(\cdot)\), maps these visual embeddings into the shared text embedding space: $\(\mathbf{z}_i = \mathcal{P}(\mathbf{r}_i) \in \mathbb{R}^{N \times pats \times d_t}\)$ Simultaneously, a structured natural language prompt is instantiated using the sample's species and identity: "A photo of a {species} individual with identity {id}. This individual has \(\langle\text{reg}_1\rangle \dots \langle\text{reg}_N\rangle\) patterns." SigLIP's text encoder converts this prompt into text tokens \(\mathbf{z}_i^{(t)}\) containing explicit placeholder positions. The projected regional visual tokens \(\mathbf{z}_i\) are then inserted directly into their respective placeholder slots, producing a unified region-referenced interleaved sequence \(\kappa_i = \mathbf{z}_i^{(t)} \oplus \mathbf{z}_i\).
3. Shallow Transformer Dense Interleaved Attention: Deep Fusion of Global Context and Microscopic Patterns
Conventional multimodal approaches typically enforce semantic alignment via single-vector contrastive objectives, failing to support fine-grained token-level cross-modal reasoning. RAMI addresses this through a shallow Transformer-based Interleaver \(\mathcal{F}(\cdot)\) comprising a single multi-head self-attention layer. A learnable pooling token \(\mathbf{z}_i^{pool}\) is prepended to the interleaved sequence: $\(\boldsymbol{\kappa}_i = [\mathbf{z}_i^{pool}, \mathbf{z}_{i,1}^{(t)}, \mathbf{z}_{i,1}, \mathbf{z}_{i,2}^{(t)}, \mathbf{z}_{i,2}, \dots, \mathbf{z}_{i,N}, \mathbf{z}_{i,M}^{(t)}]\)$ Through bidirectional self-attention, the pooling token dynamically aggregates semantic constraints from textual descriptions and fine-grained visual cues across all biological patches, outputting a rich joint representation \(\bar{\kappa}_i = \mathcal{F}(\boldsymbol{\kappa}_i) \in \mathbb{R}^{d_t}\). This compact module adds only 15.94M parameters while enabling dense cross-modal interaction between visual patterns and semantic identity cues.
4. Progressive Multi-Task Optimization and Counterfactual Causal Regularization: Balancing Alignment Stability and Fine-Grained Discrimination
To ensure stable convergence of the newly introduced projection and interleaving modules, RAMI utilizes a two-stage progressive optimization strategy. In the warmup stage, the pretrained SigLIP backbone remains frozen while the TokenProjector and Interleaver are trained using SigLIP's pairwise sigmoid contrastive loss \(\mathcal{L}_{Sig}\): $\(\mathcal{L}_{Sig} = -\frac{1}{B}\sum_{i=1}^B \sum_{j=1}^B \log \frac{1}{1 + \exp(-\sigma_{i,j} \cdot s(v_i, t_j))}\)$ In the joint fine-tuning stage, the vision encoder is unfrozen and optimized alongside the interleaving modules under a combined objective encompassing cross-entropy identity classification loss with label smoothing (\(\epsilon=0.1\)) \(\mathcal{L}_{id}\), margin-based triplet loss (\(\gamma=0.7\)) \(\mathcal{L}_{tri}\), and the contrastive loss \(\mathcal{L}_{Sig}\). Furthermore, to causally verify the necessity of specific anatomical patches, the framework models counterfactual interventions by masking individual regions and measuring the paired similarity drop \(\Delta_{i,n}(x_i) = s(v_i, t_i) - s(v_i, t_i^{\setminus n})\), providing theoretical and empirical validation that local patches carry causal identity-discriminative signals.
Loss & Training¶
The overall training objective is formulated as: $\(\mathcal{L}_{overall} = \mathcal{L}_{id} + \mathcal{L}_{tri} + \mathcal{L}_{Sig}\)$ where \(\mathcal{L}_{id} = -\sum_{\phi=1}^\Phi q_\phi \log(p_\phi)\) incorporates label smoothing \(q_\phi = (1-\epsilon)\delta_{\phi, y_i^c} + \epsilon/\Phi\), and \(\mathcal{L}_{tri} = \max(d_{pos} - d_{neg} + \gamma, 0)\). The vision backbone uses SigLIP's SoViT-400m architecture. The model is trained on NVIDIA HGX H200 GPUs using the AdamW optimizer with a batch size of 32. The TokenProjector and Interleaver are warmed up for 10 epochs at an initial learning rate of \(1 \times 10^{-3}\), followed by 60 epochs of joint optimization with the vision encoder (learning rate \(1 \times 10^{-5}\) for vision backbone, \(1 \times 10^{-4}\) for projection and interleaver modules) under a cosine learning rate decay schedule.
Key Experimental Results¶
Main Results¶
RAMI was comprehensively evaluated across four established Animal ReID benchmarks (ATRW for Amur tigers, Giraffes, NyalaData, SealID for ringed seals) and a newly collected in-the-wild Stoat dataset against five state-of-the-art baselines. All reported numbers represent the mean and 95% confidence intervals over 10 independent experimental runs:
| Dataset | Metric | TransReID | CLIP-ReID | GloPER | MDReID | CARE (Prev. SOTA) | RAMI (Ours) | Gain vs. Prev. SOTA |
|---|---|---|---|---|---|---|---|---|
| ATRW (Amur Tigers) | mAP (%) Rank-1 (%) |
44.6±0.9 89.7±1.0 |
54.1±0.3 94.2±0.2 |
37.8±0.0 81.4±0.0 |
49.5±0.7 91.9±0.6 |
57.0±0.4 95.4±0.2 |
63.2±0.1 97.4±0.1 |
+6.2% +2.0% |
| Giraffes | mAP (%) Rank-1 (%) |
25.1±1.7 51.6±4.1 |
48.9±0.1 77.2±0.8 |
20.7±0.0 44.7±0.0 |
34.9±0.8 66.7±1.8 |
60.0±0.5 80.9±0.7 |
69.3±0.4 88.0±0.2 |
+9.3% +7.1% |
| NyalaData (Nyala Antelopes) | mAP (%) Rank-1 (%) |
9.0±0.1 12.9±0.5 |
13.4±0.1 21.9±0.3 |
9.3±0.0 12.2±0.0 |
6.4±0.2 7.7±0.8 |
16.6±0.1 29.0±0.2 |
35.3±0.2 58.8±1.1 |
+18.7% +29.8% |
| SealID (Ringed Seals) | mAP (%) Rank-1 (%) |
11.2±0.4 26.5±3.5 |
20.4±0.2 57.0±0.5 |
10.4±0.0 25.0±0.0 |
12.2±0.9 34.6±2.0 |
21.6±0.3 58.5±1.1 |
30.1±0.2 71.0±0.1 |
+8.5% +12.5% |
| Stoat (Wild Stoats) | mAP (%) Rank-1 (%) |
58.1±1.4 85.7±2.0 |
62.8±0.3 91.5±0.7 |
28.9±0.0 63.8±0.0 |
57.8±0.7 89.3±0.4 |
64.1±0.2 91.9±0.4 |
66.8±0.5 91.6±0.5 |
+2.7% -0.3% |
Ablation Study¶
To rigorously isolate the benefits of multimodal interleaving and localized region modeling, the authors compared RAMI against a standard text fine-tuning baseline (SigLIP-FT) and a variant using global pattern extraction (RAMI-Global):
| Dataset | Config (SigLIP-FT: Global Text Fine-Tuning) | Config (RAMI-Global: GloPER Pattern Interleaving) | Config (RAMI: Localized Region Interleaving) | Note |
|---|---|---|---|---|
| ATRW | mAP: 55.1±0.3 / R-1: 94.2±0.2 | mAP: 61.4±0.3 / R-1: 96.5±0.2 | mAP: 63.2±0.1 / R-1: 97.4±0.1 | Region interleaving yields +8.1% mAP over SigLIP-FT |
| Giraffes | mAP: 67.2±0.5 / R-1: 87.1±0.6 | mAP: 69.0±0.2 / R-1: 88.2±0.5 | mAP: 69.3±0.4 / R-1: 88.0±0.2 | Consistent gains on large regular patch markings |
| NyalaData | mAP: 25.8±0.8 / R-1: 41.8±1.5 | mAP: 34.1±0.3 / R-1: 56.6±0.8 | mAP: 35.3±0.2 / R-1: 58.8±1.1 | Major +9.5% mAP gain on dense, subtle stripe patterns |
| SealID | mAP: 20.2±0.4 / R-1: 48.8±0.4 | mAP: 28.8±0.2 / R-1: 69.6±0.4 | mAP: 30.1±0.2 / R-1: 71.0±0.1 | Localized modeling sharply suppresses water surface noise |
| Stoat | mAP: 61.7±0.6 / R-1: 91.3±0.4 | mAP: 64.8±0.4 / R-1: 90.4±0.2 | mAP: 66.8±0.5 / R-1: 91.6±0.5 | Robust improvements on challenging wild camera-trap data |
Additionally, counterfactual and computational analyses demonstrate: 1. Counterfactual Masking Analysis (RAMI-CF vs. RAMI): Masking the most influential biological patch caused mAP on NyalaData to decline from 35.3% to 34.4% and Rank-1 to drop from 58.8% to 56.9%, confirming that the mined regions provide indispensable causal evidence for individual discrimination. 2. Computational Overhead: Compared to the 878.01M total parameters and 428.27M trainable parameters of SigLIP-FT, RAMI's TokenProjector (2.66M) and Interleaver (15.94M) add merely 18.6M trainable parameters. Backward computation increases marginally from 1440.23G to 1504.84G FLOPs, representing negligible runtime overhead.
Key Findings¶
- Localized region interleaving is vital for fine-grained animal discrimination: On NyalaData, characterized by thin and ambiguous torso stripes, RAMI achieves an unprecedented 18.7% mAP and 29.8% Rank-1 boost over the prior SOTA. Ablations verify that local region interleaving provides 1.2% to 2.2% higher Rank-1 accuracy compared to global pattern interleaving.
- Attention maps match biological discriminative anatomy: Grad-CAM visualizations demonstrate that RAMI's attention coincides closely with mined biological patches across species (e.g., giraffe coat tesselations, tiger limb stripes, seal ring patterns, and stoat head/tail markings), effectively discarding surrounding foliage.
- Training-inference decoupling enables practical wild deployment: Because the region mining and textual interleaving pipelines are active only during training, testing requires only a single forward pass through the fine-tuned vision backbone, eliminating runtime latency bottlenecks.
Highlights & Insights¶
- Repurposing multimodal interleaving for discriminative tasks: Shifts the paradigm of interleaved visual-textual tokens from multi-image generative modeling and visual reasoning to fine-grained representation learning in foundation VLMs.
- Statistically grounded, unsupervised region mining: Combines SAM 3 masks with Sobel gradient magnitudes under Benjamini–Hochberg FDR control, producing robust, biologically representative local patches without human keypoint supervision.
- Counterfactual verification of visual evidence: Evaluates representation shifts upon systematically removing individual patches, moving beyond heuristic attention maps to establish causal interpretability for ecological monitoring.
Limitations & Future Work¶
- Dependence on initial segmentation quality: The pipeline relies on off-the-shelf segmentation (e.g., SAM 3) to isolate animal foregrounds. In severe infrared noise, extreme vegetative occlusion, or murky aquatic environments, segmentation failures may propagate noisy background patches into training.
- Template-based textual prompts: The current text sequence relies on fixed prompt templates and generic placeholders. Integrating domain-specific biological knowledge graphs or LLM-generated anatomical priors could provide richer contextual conditioning.
- Dataset-level uniform region counts: Region numbers are presently fixed per dataset using the statistical mode of detected patches. Developing sample-adaptive patch allocation could better accommodate varying camera distances and animal postures.
Related Work & Insights¶
- vs CARE (WACV 2026): CARE employs image-conditioned prompt tuning and global cross-modal alignment to handle animal posture diversity, but operates purely at the holistic image level. RAMI decomposes images into localized biological patches, resolving background overfitting on fine patterns.
- vs GloPER (ICCV 2025): GloPER uses unsupervised reconstruction to extract global animal patterns within a unimodal visual space. RAMI extends pattern extraction to localized regions and introduces text placeholders for rich multimodal interleaved attention.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering adaptation of multimodal interleaving and statistically controlled region mining to fine-grained animal re-identification.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across five datasets with full baselines, ablations, counterfactual causal analysis, and Grad-CAM visualizations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear narrative flow, rigorous mathematical formulation, and well-structured empirical analysis.
- Value: ⭐⭐⭐⭐⭐ Delivers substantial performance breakthroughs in wildlife re-identification with an open-source codebase, offering immediate impact for biodiversity conservation.