Skip to content

Disentangling and Reusing Interaction Cues for Zero-Shot HOI Detection

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: zero-shot learning, human-object interaction detection, vision-language models, feature disentanglement, background shortcut bias

TL;DR

To resolve the two-fold bottleneck where fine-grained interaction cues are overshadowed by global semantics and deep layers fall prey to background shortcut bias, DRIC decouples interaction-specific residuals via context-orthogonal projection and aggregates multi-depth features via stratified local fusion to recover essential contact details.

Background & Motivation

Human-Object Interaction (HOI) detection aims to localize human-object pairs within an image and identify their interaction relationships, formally represented as a triplet \(\langle\text{human}, \text{object}, \text{action}\rangle\). Serving as a cornerstone of embodied intelligence perception, augmented reality, and robotic assistance, traditional approaches have predominantly relied on fully supervised training paradigms in closed-set settings. However, these methods suffer from heavy annotation costs for predefined interaction categories and struggle to generalize to unseen verbs or novel human-object combinations. Recently, Vision-Language Models (VLMs) such as CLIP have shown impressive zero-shot transfer capabilities through open-vocabulary joint embedding spaces. While leveraging text prompts has advanced zero-shot HOI detection, current models still encounter severe performance degradation on novel interactions due to inherent representational bottlenecks.

This generalization limitation fundamentally stems from a two-fold bottleneck at both the feature and decision levels. At the feature level, pre-trained visual representations in foundation models are dominated by macroscopic object identities and global scene layouts. Consequently, fine-grained interaction cues—such as subtle physical contacts, hand manipulations, and local body poses—are easily overshadowed by high-level global semantics. For instance, distinguishing "ride-bike" from "repair-bike" relies almost entirely on local hand/body contact details, which are smoothed out by global representations. At the decision level, models exhibit strong background shortcut learning biases, relying on spurious environmental co-occurrences rather than interaction-centric evidence (e.g., predicting "ride" whenever an outdoor scene is recognized). As visual encoder depth increases, feature saliency drifts away from local interaction contacts toward background shortcuts, severely undermining cross-scene transferability and generalization to unseen classes.

Neither end-to-end global fine-tuning nor standard image-text distillation can fundamentally restore these lost fine-grained interaction cues. Core idea: Disentangling and Reusing Interaction Cues (DRIC) explicitly isolates fine-grained interaction residuals in the orthogonal complement of a global prototype subspace via Context-Orthogonal Residual Disentangling (CORD), while employing Stratified Local Fusion (SLF) to aggregate multi-depth contact-level details from the visual encoder, guided by a progressive two-stage training strategy to eliminate background shortcuts.

Method

Overall Architecture

DRIC adopts a standard two-stage HOI detection pipeline: the first stage localizes humans and objects, while the second stage classifies interaction verbs. For an input image \(I\), candidate bounding boxes, categories, and feature representations are extracted via a pre-trained DETR detector. After passing through a lightweight MLP, valid candidate human-object pairs are transformed into human-object (HO) tokens \(T \in \mathbb{R}^{N_{\text{pair}} \times D}\), accompanied by spatial prior embeddings \(p\). Prior to each layer of the CLIP image encoder, the CORD module establishes a global semantic prototype bank and performs closed-form orthogonal projection, decoupling patch features into global common components and orthogonal interaction residuals. The interaction residuals are reintegrated into visual patch tokens via gating modulation, and sparsely routed to HO tokens via cross-attention. Concurrently, the SLF mechanism collects HO tokens across all encoder layers, aligns inter-layer feature distributions using lightweight adapters, and adaptively fuses multi-depth features to produce an auxiliary interaction score that is combined with the zero-shot similarity score.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image + DETR Candidate Pairs"] --> B["Construct Candidate HO Tokens & Spatial Priors"]
    B --> C["Context-Orthogonal Residual Disentangling<br/>Form global prototypes & extract orthogonal residuals"]
    C --> D["Sparse Interaction Routing<br/>Cross-attention directs pure interaction cues to HO tokens"]
    D --> E["Stratified Local Fusion<br/>Adapter alignment & multi-depth weighted aggregation"]
    E --> F["Joint Scoring & Zero-Shot Classification<br/>Combine shoi & sfused for final prediction"]

Key Designs

1. Context-Orthogonal Residual Disentangling: Isolating interaction details in the orthogonal complement of the prototype subspace

To prevent fine-grained interaction cues from being overwhelmed by global semantics, CORD introduces explicit geometric decomposition. A global prototype bank is first constructed by pooling features from five semantic areas: the [CLS] token, background, human region, object region, and union region, yielding \(G = [g_{\text{cls}}; g_{\text{bg}}; g_h; g_o; g_{\text{union}}] \in \mathbb{R}^{K \times d}\). Features projected into this low-dimensional space \(\tilde{F} \in \mathbb{R}^{HW \times d}\) are decomposed using the orthogonal projection operator: $\(P = G^\top (G G^\top + \varepsilon I)^{-1} G\)$ Each patch feature \(\tilde{f}\) is explicitly separated into a global common component \(\tilde{f}_\parallel = P \tilde{f}\) and an orthogonal residual component \(\tilde{f}_\perp = \tilde{f} - \tilde{f}_\parallel\). Here, \(\tilde{f}_\parallel\) captures shared scene-level layouts, while \(\tilde{f}_\perp\) is orthogonal to global semantics and retains local discriminative action details. A gating coefficient \(m = \sigma(W_m \tilde{f}_\parallel)\) adaptively modulates the residual injection: \(\tilde{f}' = \tilde{f} + \alpha (m \odot \tilde{f}_\perp)\), selectively amplifying local interaction variance while safeguarding global vision-language alignment.

2. Sparse Interaction Routing: Directing orthogonal interaction cues to candidate HO tokens

To ensure candidate human-object pairs capture targeted interaction evidence rather than diffused scene noise, HO tokens query the decoupled residual space. With query vector \(Q_i = W_q T_i\) derived from HO token \(T_i\), the token first updates against the detector's geometric prior \(p\) via cross-attention to produce \(T_i^{(1)}\). Subsequently, it queries the orthogonal residual feature map \(\tilde{F}_\perp\) through a second cross-attention step: $\(T_i^{(2)} = \text{CrossAttn}(T_i^{(1)}, \tilde{F}_\perp)\)$ Because the memory bank strictly consists of orthogonal residuals free from dominant global object/background semantics, attention weights are guided to concentrate on physical contact regions and subtle pose configurations. The HO token is then updated via residual connection and layer normalization: \(T_i' = T_i + \gamma \text{Norm}(T_i^{(2)} + Q_i)\), before proceeding to subsequent Transformer layers.

3. Stratified Local Fusion: Multi-depth aggregation to counteract deep background shortcuts

To eliminate the tendency of deep encoder layers to rely on background shortcuts, SLF establishes a hierarchical multi-depth aggregation pathway. HO tokens \(T_i^{(l)}\) are extracted across selected layers \(l \in \mathcal{L}\) of the ViT backbone. Because feature distributions shift substantially across depths, lightweight adapters \(\text{Adapter}^{(l)}\) map features into an aligned space, yielding \(\tilde{T}_i^{(l)}\). A learnable layer-wise weighted aggregation computes the fused HO representation: $\(\tilde{T}_i^{\text{fused}} = \sum_{l \in \mathcal{L}} \beta_l \tilde{T}_i^{(l)}\)$ The deepest layer is initialized with a larger weight to preserve high-level semantic alignment, while shallow and intermediate layers supply contact-level geometric evidence. The fused representation passes through an MLP to generate an auxiliary score \(s^{\text{fused}}\), yielding the final interaction prediction \(s = s^{\text{hoi}} + s^{\text{fused}}\).

Loss & Training

Because hierarchical fusion depends heavily on the quality of disentangled local cues, joint end-to-end training causes fusion gradients to corrupt the orthogonal subspace. DRIC therefore employs a two-stage training paradigm: 1. Stage 1 (CORD Optimization): CORD is activated while SLF is disabled. The network is optimized purely on the zero-shot cosine similarity score \(s^{\text{hoi}}\) using Focal Binary Cross-Entropy loss: $\(\mathcal{L}_{\text{stage1}} = \mathcal{L}_{\text{FocalBCE}}(s^{\text{hoi}}, y)\)$ This ensures a solid foundation of disentangled interaction cues. 2. Stage 2 (SLF Integration): With CORD parameters frozen, the SLF module is trained. The objective combines the final fused classification loss with an L1 similarity consistency regularizer: $\(\mathcal{L}_{\text{stage2}} = \mathcal{L}_{\text{FocalBCE}}(s, y) + \lambda_{\text{sim}} \|s^{\text{hoi}} - s^{\text{fused}}\|_1\)$ where \(\lambda_{\text{sim}} = 1 \times 10^{-4}\) prevents the fused prediction from diverging from the vision-language alignment manifold.

Key Experimental Results

Main Results

Evaluation across five zero-shot benchmark settings on HICO-DET against leading SOTA methods (unified using ResNet-50 object detector and ViT-B/16 visual encoder):

Dataset / Setting Metric DRIC (Ours) LAIN (CVPR 2025) Prev. SOTA Gain
HICO-DET RF-UC Unseen mAP / HM 32.22 / 33.74 31.86 / 33.36 31.86 / 33.36 +0.36 / +0.38
HICO-DET NF-UC Unseen mAP / HM 38.59 / 35.28 36.41 / 34.31 36.45 / 34.31 (VDRP) +2.14 / +0.97
HICO-DET UV (Unseen Verb) Unseen mAP / HM 29.49 / 31.87 28.96 / 31.19 28.96 / 31.19 +0.53 / +0.68
HICO-DET UO (Unseen Object) Unseen mAP / HM 38.51 / 36.04 37.88 / 35.58 37.88 / 35.58 +0.63 / +0.46
HICO-DET UC (Unseen Composition) Unseen mAP / HM 32.58 / 34.08 31.64 / 33.25 31.64 / 33.25 +0.94 / +0.83
Fully Supervised HICO-DET Full mAP 36.30 36.02 36.02 +0.28
Fully Supervised V-COCO \(AP_{\text{role}}^{S2}\) 65.7 65.1 65.1 +0.6

Ablation Study

Ablations conducted under the challenging HICO-DET Unseen Verb (UV) setting:

Config / Strategy Unseen mAP Seen mAP Full mAP Note
Baseline (two-stage CLIP) 24.88 31.06 30.19 Lacks local decoupling and stratified fusion
+ CORD only 28.58 33.95 33.20 Decouples interaction cues (+3.70 Unseen)
+ SLF only 27.19 32.87 32.07 Suppresses background shortcuts (+2.31 Unseen)
DRIC Full Model (CORD + SLF) 29.49 34.67 33.95 Strong complementarity (+4.61 Unseen)
CORD: Feature Concatenation 28.41 33.46 32.75 Heuristic concatenation without orthogonality
CORD: Feature Subtraction 28.05 32.91 32.23 Subtraction alters representation geometry
CORD: Cross-Attention 28.86 34.09 33.36 Attention remains vulnerable to global context
SLF: Deepest Layer Only (L12) 28.58 33.95 33.20 Misses shallow contact details
SLF: Full Multi-Depth (L1-L12) 29.49 34.67 33.95 Effectively aggregates contact-level evidence
One-Stage Joint Training 28.78 33.90 33.18 Fusion gradients disrupt disentanglement
Two-Stage Progressive (Ours) 29.49 34.67 33.95 Solidifies feature foundation prior to fusion

Key Findings

  • Orthogonal subspace projection is critical: Replacing the closed-form orthogonal projection with heuristic concatenation (28.41) or subtraction (28.05) causes noticeable performance drops. Orthogonal decomposition mathematically guarantees the removal of shared global semantics, leaving pure local interaction variance.
  • Hierarchical fusion counteracts background shortcuts: Aggregating features from all encoder layers boosts Unseen Verb mAP from 28.58 to 29.49. Under severe background perturbations where test scene backgrounds are swapped, the baseline model plunges by 12.26% and LAIN drops by 3.56%, whereas DRIC only drops by 2.18% (retaining 33.21 Full mAP), confirming its resilience against scene co-occurrence bias.
  • Two-stage optimization avoids gradient interference: Decoupling CORD optimization from SLF training delivers a 0.71 mAP gain on unseen verbs compared to naive joint optimization, proving the necessity of establishing a stable orthogonal feature space first.

Highlights & Insights

  • Orthogonal complement purification: Leveraging background, human, object, and union prototypes to define an analytical projection operator effectively isolates subtle interaction nuances without disrupting pre-trained global vision-language representations.
  • Mitigating visual saliency drift: Empirical analysis uncovers that CLIP encoder features systematically drift toward background shortcuts in deeper layers; stratified multi-depth aggregation elegantly restores contact-level evidence back into final decisions.
  • Transferable design pattern: The combination of prototype-based orthogonal residual disentanglement and stratified hierarchical feature aggregation offers a general blueprint for fine-grained open-vocabulary tasks, including dense audio-visual action localization and robotic manipulation affordance modeling.

Limitations & Future Work

  • Dependency on upstream detector quality: DRIC assumes accurate human and object bounding box proposals from DETR; missed detections in dense crowds or severe occlusions bottleneck downstream interaction reasoning.
  • Computational and memory overhead: Calculating prototype projections at every ViT layer and routing multi-depth HO tokens across all 12 layers increases memory bandwidth demands relative to vanilla CLIP.
  • Future directions: Exploring single-stage detector-free architectures with integrated orthogonal residual learning and designing low-rank subspace approximations to lower deployment latency on edge devices.
  • vs LAIN (CVPR 2025): While LAIN aggregates neighboring image patches and spatial priors to bolster locality awareness, it remains vulnerable to deep-layer background correlations; DRIC explicitly purges global context biases through orthogonal projection and counters shortcut drift via multi-layer aggregation.
  • vs VDRP (NeurIPS 2025): VDRP injects visual variance and Gaussian noise into context embeddings to perturb representation distributions, representing an implicit regularization strategy; DRIC directly operates on geometric feature subspaces and architectural depths for interpretable evidence extraction.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant mathematical formulation of orthogonal residual disentanglement combined with multi-depth stratified fusion to tackle background shortcuts.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation over five zero-shot configurations, full supervision benchmarks, background replacement stress tests, and Grad-CAM visualizations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous motivation, precise mathematical notation, clear architectural figures, and structured narrative.
  • Value: ⭐⭐⭐⭐⭐ Establishes a new state of the art for zero-shot HOI detection and provides actionable insights into mitigating VLM shortcut learning.