Divide and Align: Disentangled Vision-Language Learning for Text-Based Person Anomaly Search

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ErgastiAlex/SeDA
Area: Multimodal VLM
Keywords: text-based person anomaly search, semantic disentanglement, vision-language representation, semantic token projection, feature decoupling loss

TL;DR

To tackle representation entanglement among appearance, action, and background in text-based person anomaly search, this paper proposes SeDA, a disentangled vision-language retrieval framework that projects embeddings into factor-specific subspaces via Semantic Token Projection and enforces semantic separation using LLM-decomposed caption supervision, significantly improving anomaly search and out-of-distribution generalization.

Background & Motivation

Retrieving pedestrians based on natural language descriptions (Text-Based Person Search, TBPS) holds tremendous practical value for intelligent surveillance and safety applications. Traditional benchmarks in TBPS, such as CUHK-PEDES and ICFG-PEDES, predominantly focus on static visual appearance, including clothing colors, hairstyles, and carried accessories. However, in real-world scenarios, pedestrians frequently share similar visual appearances. Relying solely on static appearance fails to reliably disambiguate targets, creating a pressing need to exploit dynamic behavioral cues and contextual scene backgrounds for fine-grained semantic differentiation. To address this, the community introduced Text-Based Person Anomaly Search (TPAS), which requires retrieving individuals performing both normal and anomalous activities, substantially expanding the semantic complexity and operational utility of text-driven search.

Nevertheless, existing TPAS methods largely inherit standard vision-language architectures or incorporate auxiliary pose priors without architectural decoupling. Their fundamental bottleneck is that appearance, action, and environmental background are conflated within a single joint embedding space. Lacking explicit disentanglement mechanisms, models readily succumb to shortcut learning—over-relying on identity priors or static background cues while suffering from poor alignment for action-centric descriptions. Consequently, models struggle to distinguish subtle behavioral nuances, such as mistaking an individual falling or suffering an accident for another person wearing identical clothing who is simply walking.

Breaking this representation entanglement requires structurally factorizing multimodal embeddings into complementary, non-overlapping semantic components across both visual and textual modalities, allowing the model to capture fine-grained attribute semantics while retaining a unified cross-modal metric space. Core idea: propose a disentangled vision-language retrieval framework named SeDA, which dynamically factorizes global features into appearance, action, and background factors via Semantic Token Projection and recomposes them into a compact retrieval embedding, guided by an LLM-supervised Feature Decoupling Loss and multi-granularity matching objectives.

Method

Overall Architecture

SeDA is built upon a dual-branch cross-modal architecture consisting of an image encoder, a text encoder, and a cross-modal interaction encoder. The visual input combines an RGB image \(I\) with a rendered 2D skeleton pose map \(P\), processed by a Swin-B transformer to extract visual sequence embeddings \(f_I\). The textual input is a natural language caption \(T\), processed by the first 6 layers of a BERT transformer to generate text embeddings \(f_T\). During training, an LLM (Qwen3-8B) decomposes the original caption into appearance, action, and background sub-captions as supervisory anchors. The image and text embeddings are then fed into the Semantic Token Projection (STP) module to extract disentangled factor tokens and recompose discriminative global \([CLS]\) tokens. Finally, the system is jointly optimized using the Feature Decoupling Loss (FDL), contrastive loss, fine-grained cross-modal matching objectives (ITM and ITM-sem) across the upper 6 cross-modal BERT layers, and masked language modeling (MLM).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input Data<br/>RGB Image + Pose Map / Caption"] --> B["Modality Encoders<br/>Swin-B Visual Encoder / BERT Text Encoder (6 Layers)"]
    B --> C["Semantic Token Projection STP<br/>Query Generation → Soft Weight Aggregation → Token Recomposition"]
    C --> D["Feature Decoupling Loss FDL<br/>LLM Sub-caption Anchors Enforcing Factor Orthogonality"]
    C --> E["Semantic ITM Matching ITM-sem<br/>Cross-Modal Encoder Aligning Disentangled Component Pairs"]
    D --> F["Unified Retrieval Objectives<br/>Contrastive + ITM + ITM-sem + MLM Optimization"]
    E --> F
    F --> G["Compact Retrieval Embeddings<br/>Efficient and Discriminative Cross-Modal Search"]

Key Designs

1. Semantic Token Projection (STP): Cooperative Factor Decoupling and Adaptive Embedding Recomposition

Global pooling in standard transformers inevitably conflates disparate visual and linguistic attributes into a single vector. The STP module addresses this by isolating semantic factors without forfeiting global contextual information. For modality \(M \in \{I, T\}\), STP first projects the global \([CLS]\) token \(x_{[CLS]}^M\) through a learnable projection matrix \(W_d^M \in \mathbb{R}^{d \times 3d}\) and splits it into three semantic queries: appearance query \(q_M^{App}\), action query \(q_M^{Act}\), and background query \(q_M^{Bg}\):

\[[q_M^{App}, q_M^{Act}, q_M^{Bg}] = \mathrm{Split}(x_{[CLS]}^M W_d^M)\]

For each factor \(k \in \{App, Act, Bg\}\), STP computes the dot-product similarity between the query and all sequence tokens \(x_i^M\), followed by a Sigmoid activation \(\alpha_M^{i,k} = \sigma(x_i^M \cdot q_M^k)\). Unlike Softmax, which enforces a winner-take-most competition across tokens, the Sigmoid formulation allows multiple relevant words or image regions to contribute cooperatively to the aggregated representation \(\tilde{x}_M^k = \sum_{i=1}^L \alpha_M^{i,k} x_i^M\). Concatenating the query with the aggregated features and projecting through \(W_m^M\) yields the disentangled token \(z_M^k\). To obtain a compact retrieval embedding, the visual branch concatenates the three semantic tokens and projects them via \(W_p^I\) to form \(z_I^{[CLS]}\) (excluding the original pooled \([CLS]\) token to prevent reintroducing entangled representations), whereas the text branch preserves sentence-level semantics by concatenating the original text \([CLS]\) token with the three factor tokens via \(W_p^T\) to produce \(z_T^{[CLS]}\).

2. Feature Decoupling Loss (FDL): Explicit Semantic Orthogonality via LLM-Decomposed Anchors

Architectural projection alone cannot guarantee that the extracted tokens align with their intended human-interpretable concepts. To provide explicit semantic supervision, an offline pre-processing step prompts Qwen3-8B to decompose each raw caption into appearance (\(T^{App}\)), action (\(T^{Act}\)), and background (\(T^{Bg}\)) sub-captions without adding extraneous facts. These decomposed descriptions are passed through the shared text encoder and STP to produce factor anchor tokens \(z_{T^k}^k\). FDL applies an InfoNCE contrastive objective that pulls each modality-specific factor token toward its corresponding text anchor while repelling in-batch mismatched samples:

\[\mathcal{L}_{FD} = \sum_{k \in \{App, Act, Bg\}} \left( - \log \frac{\exp(\mathrm{sim}(z_I^k, z_{T^k}^k) / \tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(z_{I_j}^k, z_{T^k}^k) / \tau)} - \log \frac{\exp(\mathrm{sim}(z_T^k, z_{T^k}^k) / \tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(z_{T_j}^k, z_{T^k}^k) / \tau)} \right)\]

where \(\tau = 0.07\). This loss forces the factor tokens to isolate designated information while implicitly suppressing cross-factor interference and identity bias.

3. Semantic-Aware Image-Text Matching (ITM-sem) and Factor-Level Hard Negative Mining

To ensure semantic consistency beyond coarse global alignment, the cross-modal encoder evaluates fine-grained matching between the image and each decomposed sub-caption. The semantic-aware matching loss \(\mathcal{L}_{ITM-sem}\) computes binary cross-entropy on \((I, T^{App})\), \((I, T^{Act})\), and \((I, T^{Bg})\) pairs, reinforcing granular correspondences across appearance, action, and background. Furthermore, in-batch hard negative mining is extended from the global \([CLS]\) similarity to individual factor spaces, exposing the model to challenging impostors that match on appearance but diverge on anomalous behaviors or environmental contexts.

Loss & Training

The overall training loss is an unweighted sum of the complementary objectives:

\[\mathcal{L} = \mathcal{L}_{FD} + \mathcal{L}_{C} + \mathcal{L}_{ITM} + \mathcal{L}_{ITM-sem} + \mathcal{L}_{MLM}\]

Training is conducted on 4 NVIDIA L40S GPUs with a per-GPU mini-batch size of 22 for 30 epochs. The learning rate is initialized at \(1 \times 10^{-4}\) and decays linearly to \(1 \times 10^{-5}\) using AdamW (weight decay 0.01). The backbone is initialized from X-VLM weights. Crucially, LLM caption decomposition is required exclusively during training; test-time retrieval uses raw text queries directly, preserving baseline inference latency and memory footprints.

Key Experimental Results

Main Results

SeDA is evaluated on the large-scale Person Anomaly Benchmark (PAB, 1.0M image-text pairs) and an out-of-distribution benchmark derived from UCF-Crime (UCC). SeDA significantly surpasses the previous state-of-the-art CMP and general vision-language baselines.

Dataset	Method	Training Data	R@1 (%)	R@5 (%)	R@10 (%)	mAP (%)
PAB	MRA	0.1M	70.53	94.69	97.47	81.59
PAB	APTM	0.1M	72.14	95.30	97.17	82.78
PAB	CLIP	0.1M	77.60	98.84	99.75	87.35
PAB	X-VLM	0.1M	81.95	98.84	99.19	89.86
PAB	CMP (Prev. SOTA)	0.1M	83.06	98.89	99.49	90.41
PAB	SeDA (Ours)	0.1M	85.14	99.24	99.80	91.83
PAB	CMP (Prev. SOTA)	1.0M	84.93	99.09	99.75	91.66
PAB	SeDA (Ours)	1.0M	86.45	99.44	99.85	92.59
UCC (OOD)	CMP (Prev. SOTA)	1.0M	55.23	71.67	77.99	44.35
UCC (OOD)	SeDA (Ours)	1.0M	58.97	73.37	79.08	44.57

Under challenging multi-weather conditions spanning 10 adverse scenarios (such as wind, heavy rain, snow, and low light), SeDA achieves a mean R@1 of 69.55% (+2.43 over CMP's 67.12%) and a mean mAP of 79.34% (+1.90 over CMP).

Ablation Study

Ablations on the PAB 0.1M split dissect the architectural choices within STP, the supervisory loss terms, and the semantic factor combinations:

Exp. ID	Config / Variant	R@1 (%)	R@5 (%)	R@10 (%)	mAP (%)	Note
(0)	CMP (Baseline)	83.06	98.89	99.49	90.41	Entangled baseline
(I)	\(\text{STP}_{\text{softmax}}\)	83.82	99.04	99.65	90.95	Softmax normalization in aggregation
(II)	\(\text{STP}_{\text{CA}}\)	83.92	98.94	99.44	90.92	Standard cross-attention module
(III)	STP (Sigmoid)	84.78	99.04	99.59	91.54	Full STP without \(\mathcal{L}_{ITM-sem}\)
(IV)	STP w/o \(\mathcal{L}_{FD}\)	83.15	99.09	99.39	90.52	Removing decoupling loss degrades to baseline
(V)	Full Model (STP + \(\mathcal{L}_{ITM-sem}\))	85.14	99.24	99.80	91.83	Complete proposed framework
(VI)	Appearance only	83.67	98.43	98.69	90.64	Isolated appearance factor
(VII)	Action only	83.92	97.42	97.92	90.39	Isolated action factor
(VIII)	Background only	82.10	97.07	97.52	89.33	Isolated background factor
(IX)	Appearance + Action	84.50	98.94	99.34	91.41	Pairwise combination improves over single
(X)	All Three Factors	85.14	99.24	99.80	91.83	Full complementary integration

Key Findings

• Necessity of Explicit Supervision: Removing \(\mathcal{L}_{FD}\) drops R@1 by 1.63% (from 84.78% down to 83.15%), causing performance to collapse back toward the entangled CMP baseline. This confirms that factor projection layers cannot self-organize into semantic subspaces without anchor supervision.
• Cooperative Weighting vs. Competition: Sigmoid soft-weighting outperforms Softmax (83.82%) and cross-attention (83.92%) by nearly a full percentage point in R@1. Softmax induces inter-token competition that suppresses co-occurring descriptive attributes across sentences and visual regions.
• Data Efficiency and Robustness: SeDA trained on only 0.1M samples (85.14% R@1) outperforms CMP trained on the full 1.0M dataset (84.93% R@1). On the out-of-distribution UCC dataset, SeDA gains +3.74% R@1, demonstrating superior resistance to environmental distribution shifts.

Highlights & Insights

• Training Disentanglement without Inference Overhead: By utilizing decomposed captions only as training anchors and deploying the STP module directly on global features, SeDA achieves deep semantic factor separation during learning while maintaining identical inference speed and memory footprint as standard models.
• Sigmoid-Driven Token Aggregation: Replacing competitive Softmax normalization with unconstrained Sigmoid weights prevents mutual suppression among multiple co-occurring descriptive tokens, providing a cleaner aggregation mechanism for multi-attribute retrieval.
• Context-Action Interplay in Anomaly Identification: Quantitative ablations confirm that background context is not merely passive noise; in anomaly detection, environmental cues provide indispensable constraints that help determine whether an action is normative or anomalous.

Limitations & Future Work

• Dependency on Upstream Pose Detection: The visual branch relies on pre-computed 2D body joints. Severe occlusions, extreme lighting variations, or heavy motion blur that cause pose estimation failures can impair action token extraction.
• Rigid Tri-Factor Decomposition: The prompt-based LLM decomposition currently assumes an invariant triplet (appearance, action, background). For highly complex queries involving sequential transitions or multi-person social dynamics, this rigid taxonomy may prove overly restrictive.
• Future Directions: Exploring end-to-end implicit pose reasoning without explicit joint rendering, as well as extending the framework to video-based continuous anomaly surveillance with temporal causal reasoning.

Related Work & Insights

• vs CMP [ICCV 2025]: CMP established the PAB benchmark and incorporated pose maps, but maps all modalities into an entangled feature space vulnerable to appearance shortcut learning. SeDA explicitly disentangles features into factorized subspaces via STP and FDL.
• vs Conventional TBPS (e.g., IRRA, RaSa): Traditional TBPS models assume pedestrian actions are predominantly static or mundane walking, overlooking environmental and behavioral context. SeDA demonstrates that isolating action and background factors yields marked gains in anomaly detection while simultaneously improving standard identity search.

Rating

• Novelty: ⭐⭐⭐⭐☆ An elegant and effective disentanglement strategy that combines token-level projection with LLM-decomposed semantic anchoring for person anomaly retrieval.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across 1.0M benchmark data, multi-weather robustness suites, out-of-distribution surveillance sets, and detailed factor ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from the core entanglement dilemma to modular architectural designs and empirical verifications.
• Value: ⭐⭐⭐⭐☆ Provides a practical and data-efficient paradigm for text-driven anomaly monitoring and surveillance search.

Related Papers

• [ECCV 2026] Same Person, Different Depiction: Counterfactual Evaluation of Vision-Language Models on Individuals with Limb Deficiencies
• [ECCV 2026] Incentivizing Vision Language Models to Search for Long Video Question Answering
• [ECCV 2026] AdaBoosting Text Prompts for Vision-Language Models
• [ECCV 2026] T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability
• [CVPR 2026] Quota-Calibrated Fine-Grained Alignment with Context-Aware Marginals for Text-based Person Retrieval