ANFI: Rethinking Neighbor Feature Interaction in Person Re-ID

Conference: ECCV 2026
Paper: ECCV official page · Paper PDF
Area: Human Understanding
Keywords: person re-identification, neighbor feature interaction, discrepancy relations, neighborhood similarity, noisy relation supervision

TL;DR

ANFI learns both affinity and discrepancy interactions between neighboring person images, trains their sample-wise fusion with noisy relation supervision, and achieves 88.1% mAP on CUHK03 in the same-backbone comparison, 1.2 percentage points above its strongest comparator.

Background & Motivation

Person re-identification retrieves images of the same person across cameras rather than assigning an image to a fixed set of deployment identities. Illumination, occlusion, viewpoint, and resolution changes make individual image representations unreliable. Neighbor-based methods such as NFormer and GCR therefore enrich an image feature with information from nearby features, implicitly assuming that these neighbors mostly depict the same identity. When that assumption fails, aggregation pulls different people together instead of consolidating matching observations.

The difficulty is a mismatch between training and deployment neighborhoods. Identity supervision makes training features unusually clean, whereas unseen test identities, ambiguous queries, and small galleries can produce many false-positive neighbors. A large standard gallery may hide this weakness because easy queries dominate aggregate scores. The paper consequently examines small galleries, difficult queries, and cross-domain transfer rather than treating performance under one favorable distribution as sufficient evidence.

ANFI does not attempt to solve the problem solely by filtering neighbors more accurately. It gives the model an alternative to positive aggregation: preserve what distinguishes the current image from its neighborhood, then learn when that information is useful. Core idea: treat neighboring images as sources of both shared appearance and contrasting identity information, and adapt their contributions separately for each sample.

Method

Overall Architecture

The input is a collection of backbone features for a query and gallery images; the output is an enhanced representation used for retrieval ranking. A nearest-neighbor graph, including self-neighbors, supplies the connectivity for two interaction branches: affinity aggregation and discrepancy extraction. A sample-wise fusion module combines them, while Noisy Relation Supervision (NRS) trains the model to operate beyond clean training neighborhoods.

NRS changes training inputs and supplies an auxiliary target; it is not an additional retrieval stage at deployment. The evaluation uses a single-query protocol that excludes interactions between different queries while retaining gallery information. ANFI should therefore be understood as a gallery-context feature enhancer, not an independent encoder operating on a query image alone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Backbone features<br/>and nearest neighbors"] --> B["Dual-relation interaction<br/>affinity and discrepancy"]
    B --> C["Sample-wise adaptive fusion"]
    C --> D["Enhanced features<br/>and retrieval ranking"]
    C -.-> E["Noisy relation supervision<br/>training only"]
    E -.->|Feature perturbation and relation targets| B

Key Designs

1. Dual-relation interaction: neighbors can supply shared information or preserve contrasts

The affinity branch follows conventional neighbor aggregation. It selects the top \(k\) neighbors per feature, including the sample itself, retains similarity scores on these edges, and normalizes each row into nonnegative weights. Aggregating projected neighbor features with these weights suppresses image-specific disturbances when the neighbors share an identity. However, the same operation reduces useful separation when visually similar people enter the neighborhood. The discrepancy branch addresses that failure by subtracting a learned neighborhood component from the current representation, rather than merely reversing the affinity weights.

Its relation estimator uses neighborhood similarity instead of relying only on the direct similarity between two images. For each shared third-party neighbor, it multiplies the two samples' similarities to that neighbor and sums those products across their shared neighborhood. The result contains information about both neighborhood overlap and connection strength. This branch has a separate projection; its neighborhood similarities are masked by the original nearest-neighbor graph and row-normalized into a discrepancy relation matrix. The existing neighborhood size determines how many shared neighbors participate, so there is no additional third-party sample-count parameter. Unlike reciprocal-neighbor filtering, the operation examines shared external references rather than only whether two samples select each other. Importantly, this structural similarity is not itself a probability that a neighbor has the wrong identity: it provides weights for the subsequent discrepancy operation.

Equation (12) expresses the central feature transformation:

\[ F^d=(I-\hat A^d)\varphi'(X). \]

Here, \(\hat A^d\) is the row-normalized discrepancy relation matrix, \(\varphi'\) is the separate projection, and \(I\) is the identity matrix. Because the normalized weights sum to one, this is also a weighted combination of differences between the current sample and its neighbors. Negative non-self coefficients arise from \(I-\hat A^d\); the entries of \(\hat A^d\) are not themselves negative. The branch thus preserves a neighborhood-relative residual rather than identifying and deleting each false-positive edge explicitly.

2. Sample-wise adaptive fusion: different queries need different responses to neighborhood noise

A clear query and a heavily occluded query can have very different neighbor reliability even within the same gallery. A single global affinity-to-discrepancy ratio would ignore this heterogeneity. ANFI sends the affinity feature and discrepancy feature through separate scalar-output linear layers and applies a two-way softmax to their scores. Following Equations (13)–(14), \(w_i\) weights the discrepancy branch and \(1-w_i\) weights affinity:

\[ f_i^{\mathrm{mix}}=(1-w_i)f_i+w_i f_i^d. \]

The affinity feature is \(f_i\), the discrepancy feature is \(f_i^d\), and the fusion weights are predicted from the branch outputs. This permits the representation to emphasize shared information in reliable neighborhoods and residual information in noisy ones. The architecture alone does not guarantee that behavior: a fusion predictor trained exclusively on clean neighborhoods could still over-rely on affinity. NRS supplies the training conditions and targets needed to encourage the intended adaptation.

The paper defines the Noise Neighbor Ratio (NNR) as the number of different-identity neighbors divided by neighborhood size, with the self-neighbor included in that denominator. Batch NNR averages this quantity across samples. Low NNR describes a setting where affinity should dominate; high NNR suggests a stronger discrepancy contribution. Computing true NNR requires identity labels, so it is an analysis and training quantity, not an oracle available at inference.

3. Noisy relation supervision: expose the model to noise and construct relation-aware targets

The first part of NRS simulates noisy neighborhoods during training. It linearly mixes a feature with another randomly selected feature from the same batch, applying stop-gradient to the sampled reference so that the perturbation path does not update that reference. The perturbation strength adapts to the original batch NNR: cleaner batches receive stronger permitted perturbations, while already noisy batches receive milder ones. This targets the gap between easy identity-supervised training graphs and difficult graphs built from unseen identities, rather than merely adding another image augmentation. The locally extracted text corrupts the uniform-distribution interval in Equation (16), so this note retains the supported adaptation rule without reconstructing the missing interval formula.

The second part constructs label-guided relationship targets. An identity classification loss can make the final representation classifiable without ensuring that its individual neighbor interactions are sensible. For a reference affinity feature, NRS masks different-identity neighbors; for a reference discrepancy feature, it masks same-identity neighbors except the sample itself. Each masked relation matrix is row-normalized again before computing its reference feature. Preserving the self term maintains the structure of discrepancy extraction rather than replacing it with an arbitrary negative-example embedding.

To construct the fused reference, NRS replaces the predicted discrepancy weight with the sample's true NNR. A KL divergence then compares the classifier output distributions of the actual fused feature and this reference feature. This is neither direct Euclidean feature regression nor an independent binary loss on every edge. Labels influence the relation model and fusion behavior through a target distribution generated by the label-guided interactions. The replacement by true NNR occurs only during target construction; inference still uses the learned fusion predictor.

Loss & Training

Training is end-to-end with three objectives: the backbone Re-ID loss, identity classification on the fused representation, and NRS relationship regularization. NRS changes both the training feature distribution and the supervision target, so it should not be reduced to an extra scalar penalty alone. The main implementation uses ResNet-50 with improvements from the AGW baseline; the cross-modal version adopts the CIFT baseline's Re-ID loss. Both projections are implemented as batch-normalization layers rather than unspecified deep projection networks.

Images are resized to \(384\times192\), with random cropping, horizontal flipping, and random erasing; cross-modal training also uses random channel enhancement. SGD runs for 120 epochs with an initial learning rate of 0.01, warm-up, and cosine decay. The batch size is 64 with 8 identities per batch, and both the temperature and maximum noise-strength parameter are 0.4, as stated in Section 5. The main paper does not enumerate every final neighborhood hyperparameter for every evaluation mode, so a single universal value of \(k\) should not be assumed.

Key Experimental Results

Main Results

The selected comparison is Table 2, which controls the backbone rather than attributing differences between complete representation-learning systems solely to the relation module. All entries are mAP percentages, higher is better, under the paper's standard single-query evaluation. Detailed dataset split descriptions are deferred to the supplement and are not expanded in the local main-paper cache. Raw features do not use gallery enhancement; all other columns use the backbone specified in that row and gallery information.

Dataset / backbone	Raw	CA-Jaccard	GCR	ANFI	Gain over strongest comparator
CUHK03 / TransReID	75.4	86.9	84.0	88.1	+1.2 pp
MSMT17 / CLIP-ReID	73.4	86.0	83.5	86.5	+0.5 pp
SYSU-MM01 / DEEN	71.8	77.2	76.3	79.3	+2.1 pp

These are selected columns from Table 2; the strongest-comparator calculation also checks the omitted K-reciprocal and Cheb-GR columns. A shared backbone limits feature-extraction confounding, but ANFI learns a relation module whereas re-ranking methods are generally training-free. The table therefore does not establish equal training cost.

Ablation Study

Table 4 reports Market1501 mAP percentages, higher is better, under standard, 1-shot, and Hard Query evaluation. The 1-shot protocol reduces the gallery to one image per identity; Hard Query uses the baseline's lowest-performing 10% of queries, selected without consulting ANFI. The columns represent different gallery sizes or query difficulty, so their absolute values are not interchangeable measures of performance at a common difficulty.

Configuration	Standard	1-shot	Hard Query
Backbone baseline	90.3	93.5	65.5
Affinity only	94.2	92.5	63.1
Affinity + discrepancy	94.8	93.5	65.3
Also add noise simulation	95.1	93.8	65.9
Full ANFI + NRS	95.7	94.4	66.1

Affinity alone gains 3.9 percentage points under standard evaluation but loses 1.0 and 2.4 points under 1-shot and Hard Query. Adding discrepancy recovers 1.0 and 2.2 points relative to affinity alone in those difficult modes, although Hard Query remains slightly below the backbone baseline. Full NRS then adds 0.9, 0.9, and 0.8 points over the dual-relation model, supporting complementarity between the architecture and training strategy.

Key Findings

• Table 3 reports positive baseline-relative mAP changes for ANFI in all seven settings, but Market1501-to-SYSU-MM01 transfer gains only 0.2 percentage points; this is not evidence that cross-domain retrieval is solved.
• Neighborhood hyperparameters are adjusted separately for each evaluation mode and method. The robustness claim concerns tuned configurations, not one fixed configuration that works unchanged everywhere.
• Table 5 reports 24.8 seconds for the full Market1501 all-query test, including 0.3 seconds beyond shared backbone extraction. This is a hardware- and implementation-specific batch measurement, not per-query latency.

Highlights & Insights

• A false-positive neighbor need not only be filtered out. Its neighborhood-relative contrast can help preserve distinctions, providing an alternative to continually refining neighbor selection.
• Relation-aware targets are more specific than output classification alone. Label-guided interactions connect neighborhood noise to a desired fusion behavior through classifier-distribution matching.
• Average performance can hide harmful interactions. Affinity aggregation helps the standard setting while hurting difficult queries, motivating evaluation stratified by neighborhood reliability.

Limitations & Future Work

• Author-stated evidence boundary: Dataset details, hyperparameter analysis, adaptive-weight analysis, and further visualizations are placed in supplementary material. The local cache contains only the 18-page main paper and references, so those additional results cannot be verified here.
• Reader assessment: Neighborhood structure still depends on initial features and nearest-neighbor selection. Separate projections and residual interaction mitigate noisy propagation but do not guarantee recovery from an arbitrarily poor initial graph.
• Reader assessment: Mode-specific tuning and the absence of repeated-run uncertainty leave the stability of small gains, such as 0.2 percentage points, unresolved. Point estimates alone do not establish statistical significance.
• Authors' ethics statement: The work uses public benchmarks, collects no new personal data, and deploys no system. Real-world use still requires legal authorization, privacy protection, access controls, and bias auditing; retrieval accuracy is not a deployment approval criterion.

Related Work & Insights

• Versus NFormer / GCR / Cheb-GR: These methods exploit neighbors to improve representations. ANFI explicitly retains neighborhood-relative discrepancy and learns how much to combine it with affinity information.
• Versus K-reciprocal / CA-Jaccard: These methods refine distances using ranking and neighborhood structure, whereas ANFI propagates information at the representation level. Shared-neighborhood reasoning is related, but the outputs and relation-training requirements differ.
• Versus single-image Re-ID backbones: ANFI is a relation module compatible with multiple feature extractors. Same-backbone comparisons better expose its incremental value, with gallery-dependent inference as the corresponding requirement.

Rating

• Novelty: 4/5. Joint discrepancy and affinity interaction addresses a concrete failure, while building on established graph-interaction and residual ideas.
• Experimental Thoroughness: 4/5. Same-backbone, cross-modal, cross-domain, difficult-neighborhood, and component studies are included, with supplementary and uncertainty evidence still unverified.
• Writing Quality: 3/5. The progression from analysis to method is clear, but some locally extracted equations are damaged and important protocol details require the supplement.
• Value: 4/5. Useful for gallery-context retrieval, especially as a warning against judging interaction modules only on reliable neighborhoods.