Single-Query Person-Centric Bimanual Hand-Object Interaction Detection

Conference: ECCV 2026
Paper: ECCV Official
Code: https://lgecto-ail-vil.github.io/SingleQuery-BHOI/
Area: Human Understanding
Keywords: bimanual interaction detection, hand-object interaction, single-query framework, part-aware deformable attention, pose estimation

TL;DR

Addressing the ownership ambiguity and missing object semantics of conventional hand-centric HOI approaches in multi-person scenes, this paper proposes a person-centric single-query Transformer framework that couples part-aware multi-reference deformable attention with a hand-to-query relationship matrix to jointly predict human boxes, body poses, bimanual hands, and their interaction targets end-to-end.

Background & Motivation

Hands serve as the primary interface through which humans interact with the physical world. Fine-grained understanding of which person uses which hand to manipulate which object is vital for procedural activity recognition, demonstration learning in embodied robotics, and assistive perception systems. In recent years, large-scale hand contact datasets such as 100DOH and Hands23 have spurred substantial advances in hand detection and contact estimation. However, most existing approaches adopt a hand-centric formulation: each hand is treated as an isolated instance whose box, side (left vs. right), and contact state are regressed independently.

In crowded scenes with multiple interacting people, this hand-centric paradigm suffers from severe ownership ambiguity. A model can detect multiple hands and contacting objects yet fail to determine which two hands belong to the same person. This fundamentally reflects a mismatch between the prediction unit (isolated hands) and the true semantic unit (a person wielding two hands to interact with objects). Furthermore, directly regressing interacting object boxes fails to retrieve semantic object categories and is fragile under heavy occlusion.

To address these limitations, the authors advocate making the human instance the primary prediction unit and explicitly attaching left and right hands as dedicated slots. Core idea: formulate bimanual interaction parsing as a single-query person-centric structured prediction task, where one query jointly predicts human bounding boxes, body pose, bimanual hand slots, and interaction targets, using part-aware multi-reference deformable attention and a relationship matrix with an off token to directly retrieve target boxes and classes from the detected query set.

Method

Overall Architecture

The framework is built upon an end-to-end DETR-style set prediction detector (specifically the RT-DETR-R50-m architecture). Input images are first processed by a backbone and a Transformer encoder to extract multi-scale feature maps. The top-K candidate queries are selected and fed into a deformable decoder. Instead of spawning separate queries for instances, joints, and hands, each person query natively encapsulates the full structural hierarchy: human box, body skeleton keypoints, left/right hand boxes, and hand-specific interaction targets.

To enable a single query to jointly capture coarse whole-body context and fine-grained articulated hand/joint details, the decoder employs a part-aware deformable attention mechanism with multiple reference points. During the interaction reasoning phase, each hand embedding computes a relation distribution over all detected entity queries augmented with a learnable off token, eliminating the need for an auxiliary interacting-object regression head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image x"] --> B["Backbone + Transformer Encoder"]
    B --> C["Top-K Candidate Queries<br/>(Entity Query Set + Learnable off Token)"]
    C --> D["Part-Aware Deformable Decoder<br/>(Human / Hands / Joint Multi-References)"]
    D --> E["Multi-Head Structured Prediction<br/>Human Box / Keypoints / Bimanual Hand Boxes"]
    D --> F["Hand-to-Query Relation Matrix<br/>(Bimanual Embeddings vs All Entity Queries)"]
    E --> G["Person-Centric Structured Tuples<br/>(Human-Hand-Object Classes & Bounding Boxes)"]
    F --> G

Key Designs

1. Explicit Bimanual Slots and Single-Query Person Representation: Resolving Ownership Ambiguity

Prior hand-centric detectors break bimanual coordination into disparate entities that require heuristic post-processing to group back into a person. In contrast, this approach anchors left and right hands directly as dedicated slots within the human query representation. For each query \(i \in \{1, \dots, N\}\), the network predicts a class distribution \(\hat{c}_i\) and a human box \(\hat{b}_i\). When classified as a human instance, specialized prediction heads output \(J\) joint keypoints \(\hat{K}_i \in [0, 1]^{J \times 2}\), left and right hand boxes \(\hat{b}^L_i, \hat{b}^R_i \in [0, 1]^4\), and corresponding hand embeddings \(h^L_i, h^R_i \in \mathbb{R}^d\). This structural binding guarantees that bimanual ownership is explicitly preserved even in dense crowds with overlapping bodies.

2. Part-Aware Deformable Attention with Multiple Reference Points: Bridging Multi-Scale Articulations

Standard deformable attention decoders associate each query with a single 2D reference point (the box center), which easily washes out small extremities like hands and wrist joints within large human bounding boxes. This work distributes decoder attention heads across structured part references: the main human box center \(p^H_i = \phi(\hat{b}_i)\), left hand reference \(p^L_i = \phi(\hat{b}^L_i)\), and right hand reference \(p^R_i = \phi(\hat{b}^R_i)\). Furthermore, \(J\) joint-specific reference regions are established by scaling the predicted joint locations with learnable dimensions relative to the human box:

\[p^j_i = (\hat{k}^j_i, s^j_i), \quad s^j_i = (\gamma^j_w \hat{w}_i, \; \gamma^j_h \hat{h}_i)\]

Attention heads sample multi-scale feature maps around the human body, hands, and joints simultaneously, enabling a single query to jointly refine global torso context and fine-grained hand localization across layers.

3. Hand-to-Query Relationship Matrix with Off Token: Regression-Free Object Semantic Retrieval

Directly regressing an interacting-object bounding box from a human query fails to classify the object category and frequently hallucinates bounding boxes in cluttered backgrounds. Instead, this framework leverages the fact that the detector already outputs an entity set of \(N\) queries covering all detected persons and objects. The final query embeddings are projected into relation features \(r_i \in \mathbb{R}^d\), alongside a learnable off embedding \(r_0 \in \mathbb{R}^d\) indicating no contact. For each hand side \(s \in \{L, R\}\), dot-product matching is performed over the \((N + 1)\) target candidates:

\[\hat{t}^s_i = \arg\max_{j \in \{0, \dots, N\}} \text{Softmax}(h^s_i \cdot r_j)\]

If \(\hat{t}^s_i = 0\), the hand is marked as no-contact; if \(\hat{t}^s_i = i\), it indicates self-contact; if \(\hat{t}^s_i = j \; (j \neq i)\), the hand is in contact with another detected entity, and directly inherits that entity's bounding box \(\hat{b}_j\) and semantic class \(\arg\max \hat{c}_j\). This completely bypasses explicit interacting-object regression and naturally unifies detection and interaction semantics.

Loss & Training

The framework is trained end-to-end via Hungarian bipartite matching. The multi-task loss integrates classification cross-entropy, GIoU and \(\ell_1\) box losses, body pose OKS loss, bimanual hand box regression losses, and the cross-entropy loss over interaction target selection. While all prediction heads are evaluated during forward inference, hand and pose losses are supervised exclusively on queries matched to ground-truth human instances.

Key Experimental Results

Main Results

The model is evaluated on a manually verified COCO bimanual validation set (~2K images) and the ContactHands benchmark. Evaluation follows three progressive criteria: - Acc_soft: Measures hand contact state accuracy (hold vs. none) given a matched human instance. - Acc_mid: Requires both correct contact state and accurate interacting-object localization (IoU \(\ge 0.5\)). - Acc_hard: Further enforces that the predicted hand bounding box also overlaps with ground truth (IoU \(\ge 0.5\)).

Method Variant	COCO Det. mAP	COCO Pose AP	COCO Acc_soft (%)	COCO Acc_mid (%)	COCO Acc_hard (%)	ContactHands Acc_soft (%)
DirectBox (Direct Regression)	49.1	N/A	81.5	44.9	12.6	80.7
DirectBox + Pose	47.7	61.8	83.0	57.6	30.1	81.3
Relation (Ours)	48.8	N/A	80.0	60.8	28.9	81.7
Relation + Pose (Ours Full)	47.1	62.3	83.8	64.6	31.3	82.2
Relation + Pose (+ 100DOH data)	48.6	63.2	84.0	66.2	32.7	81.2

Note: Results cited from Table 2 and Table 3 of the original paper, using RT-DETR-R50-m.

Ablation Study: Part-Aware Reference Allocation

Config	Part-Aware	Det. mAP	Pose AP	Acc_soft (%)	Acc_mid (%)	Acc_hard (%)	ContactHands Acc_soft (%)
DirectBox + Pose	✗	48.7	43.4	80.4	43.1	11.5	80.5
DirectBox + Pose	✓	47.7	61.8	83.0	57.6	30.1	81.3
Relation + Pose	✗	48.6	42.8	83.0	63.6	10.3	81.7
Relation + Pose	✓	47.1	62.3	83.8	64.6	31.3	82.2

Note: Results cited from Table 4 of the original paper.

Key Findings

• Relation modeling decisively outperforms direct object regression: Without pose, switching from DirectBox to Relation boosts Acc_mid from 44.9 to 60.8 (+15.9) and Acchard from 12.6 to 28.9 (+16.3). Re-routing target selection over detected entity queries avoids noisy box regression and guarantees correct semantic attribution.
• Part-aware reference allocation is critical for extremity localization: In Table 4, removing part-aware sampling causes Pose AP to crash from 62.3 to 42.8 (-19.5), and slashes the strict Acchard score from 31.3 down to 10.3 (-21.0), verifying that dedicated part reference points are essential for small extremities.
• Joint pose learning provides strong structural regularization: Adding body pose supervision to the Relation model improves Acc_mid from 60.8 to 64.6 (+3.8) and Acchard from 28.9 to 31.3 (+2.4), demonstrating that kinematic posture context aids hand reasoning.
• Robustness degradation in dense crowding: On scenes partitioned by person count, performance remains strong for single-person (Acc_soft / Acc_mid / Acc_hard at 90.28 / 79.42 / 54.00) and moderate groups of 2–4 people, but falls to 22.02 on Acchard when person count \(\ge 5\), showing that severe occlusion remains an open challenge.

Highlights & Insights

• Bimanual Hands as First-Class Slots within Human Queries: Bypasses heuristic post-hoc hand-to-person grouping by modeling left and right hands directly inside the person representation, fundamentally preventing cross-person hand confusion.
• Detection Query Pool as Interaction Topology: Repurposes the detector's existing object queries as candidate interaction targets, unifying contact categorization (no-contact / self / object / person) with an elegant dot-product formulation and a single off token.
• Unified Multi-Scale Sampling in a Single Decoder: Instead of maintaining separate queries for human boxes, keypoints, and hands, head-wise multi-reference attention handles coarse torso context and fine-grained hand localization inside a single query.

Limitations & Future Work

• Performance drop under severe visual crowding: In scenes with \(\ge 5\) overlapping individuals, overlapping hands and severe occlusion still degrade the strictest tuple localization metric.
• Coarse interaction semantics: The current model targets contact entity identification (whether hands touch self, another person, or a specific object) rather than fine-grained interaction verbs or 3D grasp types.
• Pseudo-target reliance for unannotated objects: Interacting objects not present in COCO's closed vocabulary are assigned generic pseudo-object boxes, which may introduce minor supervisory noise.

Related Work & Insights

• vs. 100DOH / Hands23: Prior hand-contact benchmarks treat hands as independent entities without explicit human ownership; this work introduces a person-centric bimanual formulation evaluated on complete human-hand-object tuples.
• vs. HOTR / HOI Transformer: Traditional HOI detectors predict interactions at the holistic person-object pair level; this paper zooms into fine-grained left vs. right hand manipulation.
• vs. GroupPose / RelTR: Multi-entity or relation networks typically utilize multiple queries (human queries, keypoint queries, relation queries); this work achieves end-to-end multi-task parsing using a streamlined single-query design.

Rating

• Novelty: ⭐⭐⭐⭐ [Clean single-query person-centric formulation and dot-product interaction routing over detected entities]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Constructed and verified a new COCO bimanual benchmark with tiered soft/mid/hard tuple metrics and exhaustive ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Well-structured narrative with crisp motivation, formal problem setup, and transparent ablation analysis]
• Value: ⭐⭐⭐⭐ [Provides an effective unified baseline for embodied manipulation understanding and video action analysis]

Related Papers

• [ECCV 2026] HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching
• [ECCV 2026] ECHO: Ego-centric Modeling of Human-Object Interactions
• [CVPR 2026] HandX: Scaling Bimanual Motion and Interaction Generation
• [ECCV 2026] JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation
• [ICCV 2025] Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation