Context-Interactive Reasoning for Group Activity Detection

Conference: ECCV 2026
Paper: ECCV Official
Area: Video Understanding
Keywords: Group Activity Detection / Social Interaction Understanding / Spatio-temporal Reasoning / Scene Context Modeling

TL;DR

To tackle occlusion-induced spurious associations and context noise in crowded multi-view videos, this paper proposes a Context-Interactive Reasoning framework that softly constrains spatial topology via STIR and adaptively conditions on global scene semantics via gated LRCC, establishing new state-of-the-art performance on GAD benchmarks.

Background & Motivation

Group Activity Detection (GAD) aims to jointly infer who is with whom (group memberships) and what they are doing together (collective activities) from videos. Unlike conventional Group Activity Recognition (GAR), which typically assumes a single group per clip and pre-specified participants, realistic surveillance and social scenarios—exemplified by the multi-camera Café benchmark—exhibit dense crowds, severe occlusions, diverse camera viewpoints, and a large proportion of outliers. Prevailing approaches adopt an actor-centric modeling paradigm, relying predominantly on dot-product attention over appearance features to model inter-actor interactions. However, in cluttered scenes with visually similar actors, unconstrained attention easily forms spurious cross-group connections that violate the fundamental spatial topology of physical social interactions, causing group boundaries to blur.

To enforce locality, prior works have occasionally applied hard distance thresholds to prune distant links. Yet fixed heuristic cutoffs are brittle across camera view scales and varying crowd densities, frequently suppressing valid interactions within wide-spanning groups. At the same time, collective activity recognition heavily depends on the macro-environment and actor-scene dependencies (for instance, distinguishing "eating" from "studying" is visually ambiguous from local poses alone and requires tabletop and background contextual cues). Existing methods either overlook global scene context or resort to static fusion via naive concatenation or pooling, which injects background noise when scene semantics are weakly relevant. Furthermore, simply scaling visual backbones (e.g., DINOv2 or VideoMAE) fails to resolve these structural grouping and contextual reasoning failures.

The core angle of attack in this work is to combine soft spatial topology priors with adaptive context modulation, addressing structural grouping ambiguity and semantic activity ambiguity in tandem. Core idea: A context-interactive reasoning framework that regularizes short-term inter-actor reasoning via a Short-Term Interaction Regularizer (STIR) using learnable soft spatial distance penalties to suppress spurious connections, while selectively incorporating global scene semantics via a Long-Range Context Conditioner (LRCC) with decoupled actor/group scalar gates before temporal trajectory aggregation.

Method

Overall Architecture

The overall architecture comprises dual-stream feature representation, short-term interaction regularization (STIR), a set-prediction group decoder, long-range context conditioning (LRCC), a temporal encoder, and multi-task prediction heads. The input video is sampled in a dual-stream manner: a sparse local stream extracts actor RoI tokens from selected keyframes using a partially fine-tuned DINOv2 encoder, while a denser global stream extracts clip-level spatiotemporal scene semantics using a frozen VideoMAE-v2 encoder. Prior to group decoding, STIR operates on each local frame independently, constructing pairwise geometric descriptors and adding soft distance penalties to attention logits to filter out spurious edges. Refined actor tokens and learnable group queries are processed by the group decoder to decouple actor-group interactions. Decoupled tokens are then adaptively modulated by LRCC using decoupled scalar gating on the global scene feature. Finally, a temporal encoder aggregates cross-frame trajectories to output individual action logits, group assignment probabilities, and collective activity classes.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Dual-Stream Video Sampling<br/>Sparse local keyframes + Dense global clip"] --> B["Dual-Stream Feature Extraction<br/>DINOv2 (local RoI tokens) + VideoMAE-v2 (global pooled)"]
    B --> C["Short-Term Interaction Regularizer (STIR)<br/>Pairwise geometry descriptor + Soft distance penalty"]
    C --> D["Group Reasoning Decoder<br/>Actor tokens & learnable group queries cross-attention"]
    D --> E["Long-Range Context Conditioner (LRCC)<br/>Decoupled actor/group scalar gating on scene prior"]
    E --> F["Temporal Modeling<br/>Residual TCN + Temporal Self-Attention (TSA) + Attn-Pooling"]
    F --> G["Multi-Task Output Heads<br/>Individual actions + Group memberships + Collective activities"]

Key Designs

1. Short-Term Interaction Regularizer (STIR): Learnable soft spatial topology constraints against spurious links

In dense crowds, pure appearance attention easily links unrelated actors separated by clutter. STIR constructs explicit frame-level pairwise geometric descriptors to inject a soft, learnable spatial inductive bias into attention logits rather than relying on non-differentiable hard distance cutoffs. For actor bounding boxes \(b_i = (x_i, y_i, w_i, h_i)\) and \(b_j = (x_j, y_j, w_j, h_j)\) in normalized coordinates, STIR extracts center offsets, log relative scales, and Euclidean center distance into a descriptor vector \(s_{ij} = [\Delta x_{ij}, \Delta y_{ij}, \log(w_i/w_j), \log(h_i/h_j), \text{dist}_{ij}]\). In multi-head attention, the raw attention logit for head \(h\) is formulated as:

\[\mathrm{score}_{ij}^h = \frac{(W_q^h a_i)(W_k^h a_j)^\top}{\sqrt{d_k}} + \phi^h(s_{ij}) - \frac{\mathrm{dist}_{ij}^2}{\tau}\]

where \(\phi^h(s_{ij})\) is a head-specific spatial bias produced by a lightweight MLP, and \(-\frac{\mathrm{dist}_{ij}^2}{\tau}\) acts as a soft Gaussian-like distance suppression penalty. The temperature parameter \(\tau = \exp(\rho)\) is strictly positive and initialized to 1.0, enabling the network to learn the effective interaction radius directly from data in an end-to-end differentiable manner. This preserves necessary cross-distance interactions while penalizing cross-group clutter.

2. Group Reasoning Decoder and Temporal Trajectory Modeling: Cross-frame aggregation

After STIR refinement, per-frame actor tokens are fed into a DETR-style group decoder together with a fixed pool of learnable group queries. Alternating self-attention and cross-attention over frame visual features jointly decouples actor-group assignments. To resolve single-frame occlusions and transient viewpoint variations, decoded token trajectories \(U \in \mathbb{R}^{T \times d}\) pass through a 1D residual temporal convolution network (TCN) to capture local dynamics, followed by \(L\) stacked layers of temporal multi-head self-attention (TSA) with learnable temporal position embeddings \(P\):

\[U^{(0)} = U + \mathrm{TCN}(U) + P, \qquad U^{(l+1)} = \mathrm{TSA}^{(l)}(U^{(l)})\]

Finally, a lightweight attention pooling layer aggregates frame-level states into a clip-level compact representation for classification.

3. Long-Range Context Conditioner (LRCC): Decoupled adaptive scalar gating for global scene semantics

Recognizing context-dependent collective activities requires scene-level cues, but indiscriminate fusion introduces severe background noise when the scene is uninformative. LRCC projects the clip-level VideoMAE-v2 representation \(f_{\mathrm{mae}}\) into the shared token embedding space as \(z_{\mathrm{global}} \in \mathbb{R}^d\). Because actor tokens (capturing individual poses) and group tokens (capturing collective intent) have different functional roles, LRCC modulates them via decoupled non-shared scalar gates:

\[\alpha_i = \sigma\big(W_a [a_i; z_{\mathrm{global}}]\big), \quad \tilde{a}_i = a_i + \alpha_i z_{\mathrm{global}}\]

\[\beta_k = \sigma\big(W_g [g_k; z_{\mathrm{global}}]\big), \quad \tilde{g}_k = g_k + \beta_k z_{\mathrm{global}}\]

Here, scalar gates \(\alpha_i, \beta_k \in (0, 1)\) dynamically regulate the magnitude of global semantic injection on a per-token basis. Compared to high-dimensional channel-wise modulation, scalar gating is lightweight, highly interpretable, and less prone to overfitting on limited training video clips.

Loss & Training

The framework is optimized end-to-end under a multi-task objective. Hungarian bipartite matching pairs predicted group slots with ground-truth groups. The total training objective combines individual action loss \(\mathcal{L}_{\mathrm{ind}}\), collective activity loss \(\mathcal{L}_{\mathrm{group}}\), group membership binary cross-entropy \(\mathcal{L}_{\mathrm{mem}}\), and an InfoNCE-style contrastive regularizer \(\mathcal{L}_{\mathrm{con}}\):

\[\mathcal{L} = \lambda_{\mathrm{ind}} \mathcal{L}_{\mathrm{ind}} + \lambda_{\mathrm{group}} \mathcal{L}_{\mathrm{group}} + \lambda_{\mathrm{mem}} \mathcal{L}_{\mathrm{mem}} + \lambda_{\mathrm{con}} \mathcal{L}_{\mathrm{con}}\]

The contrastive term \(\mathcal{L}_{\mathrm{con}}\) pulls actor embeddings of the same group together while pushing apart actors from different groups. For training efficiency, only the final 2 transformer blocks of DINOv2-Base are fine-tuned with a learning rate factor of \(0.1\times\), while the VideoMAE-v2 Giant backbone remains fully frozen with offline pre-extracted features.

Key Experimental Results

Main Results

The proposed method is comprehensively evaluated on the challenging Café dataset (across both split-by-view and split-by-place protocols) and on Social-CAD. Primary evaluation metrics include Group mAP at IoU thresholds of 1.0 and 0.5 (Group mAP1.0, mAP0.5) and Outlier mIoU.

Dataset / Split	Method	Year	Backbone	Group mAP1.0	Group mAP0.5	Outlier mIoU	Gain (vs. Prev. SOTA)
Café (Split by view)	Practical GAD	2024	ResNet-18	18.84	37.53	67.64	+11.63 / +16.97 / +8.34
	VLCMTN	2025	ResNet-18	19.67	41.06	68.29	+10.80 / +13.44 / +7.69
	ProGraD (frozen)	2026	DINOv2	16.85	43.42	70.42	+13.62 / +11.08 / +5.56
	Ours	2026	DINOv2+VideoMAE	30.47	54.50	75.98	+10.80 / +11.08 / +5.56
Café (Split by place)	Practical GAD	2024	ResNet-18	10.85	30.90	63.84	+12.97 / +17.07 / +8.13
	VLCMTN	2025	ResNet-18	11.58	30.98	65.48	+12.24 / +16.99 / +6.49
	ProGraD (full FT)	2026	DINOv2	20.42	44.14	67.71	+3.40 / +3.83 / +4.26
	Ours	2026	DINOv2+VideoMAE	23.82	47.97	71.97	+3.40 / +3.83 / +4.26

On the Social-CAD benchmark, using only 1 local frame and 16 global frames, our model achieves 72.8% social activity accuracy and 90.2% individual action accuracy, outperforming ProGraD (70.0%) and STIRM (69.6% / 83.1%).

Ablation Study

Controlled component ablations conducted on the more demanding Café split-by-place setting isolate the individual and joint contributions of STIR and LRCC:

Config	Backbone	STIR	LRCC	Group mAP1.0	Group mAP0.5	Outlier mIoU	Note
Baseline	ResNet-18	✗	✗	8.75	29.65	59.86	Standard set prediction baseline
Baseline	DINOv2	✗	✗	10.45	31.76	65.75	Feature upgrading yields marginal reasoning gain
+STIR Only	ResNet-18	✓	✗	15.86	36.71	68.44	Spatial regularization boosts grouping
+STIR Only	DINOv2	✓	✗	15.69	35.16	69.71	Eliminates spurious crowd links
+LRCC Only	ResNet-18	✗	✓	12.71	34.83	66.97	Context helps activity discrimination
+LRCC Only	DINOv2	✗	✓	18.98	35.58	69.21	Adaptive gating prevents noise over-injection
Full Model	ResNet-18	✓	✓	17.14	34.45	69.64	Consistent dual-module improvement
Full Model	DINOv2	✓	✓	23.82	47.97	71.97	Optimal combination across all metrics

Internal design variants of STIR and LRCC:

Variant Category	Design Choice	Group mAP1.0	Group mAP0.5	Outlier mIoU	Analysis
STIR Spatial Design	w/o STIR	18.98	35.58	69.21	Unconstrained attention blurs group boundaries
	Spatial bias	22.64	45.10	71.39	Learns orientation but lacks distance attenuation
	Hard mask	20.70	43.66	70.00	Fixed cutoffs hurt intra-group distant links
	STIR (Soft penalty + Bias)	23.82	47.97	71.97	Differentiable soft decay fits diverse densities
LRCC Context Design	w/o LRCC	15.69	35.16	69.71	Missing global cues confuses fine activities
	Concat	17.57	35.81	70.36	Indiscriminate injection introduces noise
	Add	16.59	38.31	68.21	Linear blending degrades outlier handling
	Global pooling	22.90	43.28	70.93	Uniform broadcast lacks token adaptability
	Shared gate	16.45	32.41	68.31	Actor and group tokens compete under single gate
	Two-branch gates	23.82	47.97	71.97	Decoupled gating yields best activity conditioning

Key Findings

• Spatial topology regularizer sets grouping upper bound: Switching backbone from ResNet-18 to DINOv2 alone only improves Group mAP1.0 from 8.75 to 10.45. Adding STIR increases performance by 5~7 percentage points regardless of backbone, proving that unstructured visual features cannot solve dense crowd occlusion.
• Differentiable soft penalties outperform heuristic masks: Hard distance thresholding lags behind the soft design by 3.12 mAP1.0 (20.70 vs. 23.82), validating the benefit of data-driven adaptive interaction radii across camera angles.
• Decoupled gating is critical: A shared gate for both actor and group tokens drops mAP1.0 drastically by 7.37 (16.45 vs. 23.82). Instance-level pose cues and group-level intent cues require distinct degrees of environmental contextualization.

Highlights & Insights

• Differentiable Spatial Interaction Inductive Bias: Mapping relative bbox geometries into multi-head biases and an exponential decay penalty replaces rigid heuristic pruning, retaining end-to-end learnability across complex camera views.
• Decoupled Lightweight Gating for Context Modulation: Employing separate non-shared scalar gates for actors and groups modulates global scene features with minimal parameter overhead while preventing context over-injection.
• Seamless Plug-and-Play Integration: Operating purely on input token attention and decoded tokens, STIR and LRCC can be retrofitted onto arbitrary set-prediction or transformer-based group activity pipelines.

Limitations & Future Work

• Reliance on Ground-Truth Bounding Boxes: Evaluated under standard GT bounding box protocols, potential box drift and detector false negatives in open-world deployment may perturb STIR geometric descriptors.
• Offline Frozen Global Feature Extraction: VideoMAE-v2 features are pre-extracted offline; fine-tuning or end-to-end multi-rate visual training could allow deeper cross-modal adaptation.
• Temporal Trajectory Structure: Temporal modeling relies on 1D convolutions and self-attention on decoupled tokens; explicit spatiotemporal graph tracking across time could further clarify evolving group dynamics.

Related Work & Insights

• vs. Practical GAD [ECCV 2024]: Practical GAD pioneered realistic multi-group set prediction but utilized unconstrained inter-actor attention. Our work keeps its group decoder intact while complementing it with front-end topology filtering (STIR) and back-end context gating (LRCC), boosting cross-view mAP1.0 by +11.63.
• vs. ProGraD [CVPRW 2026]: While ProGraD investigated prompt-guided foundation model adaptation, it still suffers from crowd misgrouping due to lack of explicit physical constraints. Our approach confirms that structured spatial and context interactions are essential on top of VFM backbones.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Combines physical spatial soft decay with decoupled context gating, directly targeting core GAD bottlenecks]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous multi-split evaluations on Café and Social-CAD with extensive module-level ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically rigorous formulations, and consistent structural presentation]
• Value: ⭐⭐⭐⭐☆ [A clean, plug-and-play blueprint for structured interaction reasoning in crowded video understanding]

Related Papers

• [ECCV 2026] CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation
• [CVPR 2026] IPR-1: Interactive Physical Reasoner
• [ECCV 2026] PASR: Pattern-Aware Scene-Conditioned Reasoning for Camouflaged Object Detection
• [ECCV 2026] SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection
• [ECCV 2026] REVEAL: Reasoning-Enhanced Forensic Evidence Analysis for Explainable AI-Generated Image Detection