Decoupling Moment from Event for Video Temporal Grounding

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/zouyuda220/ME-DETR
Area: Object Detection
Keywords: Video Temporal Grounding, Dynamic Query Specialization, Synergistic Supervision, Hungarian Matching, Temporal Fluidity

TL;DR

Addressing the forced semantic suppression problem caused by directly adopting rigid 2D object detection one-to-one matching in video temporal grounding, ME-DETR decouples sub-event moment semantic exploration from full-event boundary regression via dynamic query role specialization and dual-head synergistic supervision, delivering a +2.43% test mAP gain on QVHighlights without requiring NMS.

Background & Motivation

Video temporal grounding (VTG) aims to locate the precise start and end timestamps of a target activity in untrimmed video given an open-ended natural language query. Inspired by the success of DETR-style direct set prediction in 2D object detection, recent VTG detectors—such as Moment-DETR, QD-DETR, CG-DETR, and Flash-VTG—have increasingly transitioned to end-to-end transformer architectures. By formulating grounding as one-to-one bipartite matching via the Hungarian algorithm, these detectors directly produce temporal spans from learnable queries and eliminate the need for heuristic post-processing such as Non-Maximum Suppression (NMS). However, directly porting the matching paradigm of spatial 2D images to 1D temporal video sequences overlooks a fundamental domain discrepancy between spatial objects and temporal processes.

In 2D image detection, object semantics are tightly bounded by spatial extents: a proposal box covering merely a "cat torso" is not semantically equivalent to "a cat", meaning that one-to-one matching based on Intersection over Union (IoU) appropriately penalizes partial boxes as background negatives. In sharp contrast, temporal video events exhibit strong temporal fluidity and intrinsic internal redundancy. A brief sub-span (a moment, such as a specific dance pose) can be strongly and correctly aligned with the prompt ("a girl is dancing") even though it covers only a fraction of the entire annotated event duration. Under rigid temporal IoU (tIoU) one-to-one matching, exactly one query with high tIoU is designated as positive, while all other queries that capture valid, salient sub-events receive low tIoU and are consequently penalized as negatives by the classification loss. This training conflict penalizes the detector for detecting correct semantic evidence, impairing multi-granularity representation learning—a dilemma identified here as "forced semantic suppression."

To resolve this conflict while strictly preserving end-to-end NMS-free inference, the guiding principle is to embrace the natural part-whole hierarchy of temporal events rather than fighting it. Core idea: decouple sub-event moment semantic exploration from full-event boundary regression using a dual-head architecture and dynamic query specialization, where the event head retains strict one-to-one matching for localization while the moment head mines high-containment sub-spans as classification-only positive supervision without regression penalties.

Method

Overall Architecture

ME-DETR maintains an end-to-end set prediction design while fundamentally restructuring query initialization, decoder prediction heads, and supervision assignment. Given an untrimmed video and a natural language prompt, the framework first extracts clip-level visual features and token-level textual features using a frozen multi-modal backbone (such as InternVideo or CLIP+SlowFast) projected into a hidden dimension \(d=256\). A cross-modal interaction module constructs a multi-scale temporal feature pyramid \(\{M_l\}_{l=0}^{L-1}\). Multi-scale reference spans generated densely from this pyramid initialize the query set. Inside an \(N=4\) layer Transformer decoder, queries are iteratively updated via self-attention, deformable cross-attention, and feed-forward networks (FFN). After each decoder layer, every query branches into two parallel 3-layer MLP heads: an "event head" predicting full-event classification confidence along with temporal boundary adjustments, and a "moment head" predicting semantic alignment confidence alone without boundary regression. During training, dynamic role assignment supervises both heads synergistically; at inference time, only the event head outputs are retained, preserving standard single-pass forward inference without NMS.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Video Clip Features + Text Prompt Tokens"] --> B["Pyramid Temporal Prior Initialization<br/>Multi-Scale Reference Spans + Zero Content Vectors"]
    B --> C["Dual-Head Decoupled Decoder<br/>Deformable Cross-Modal Attention Iterations"]
    C --> D["Event Head Branch<br/>One-to-One Hungarian Matching & Boundary Regression"]
    C --> E["Dynamic Moment Mining & Role Assignment<br/>High Containment & High Confidence Sub-Span Mining"]
    D --> F["Synergistic Supervision Objective<br/>Full Event Localization Loss + Moment Classification Loss"]
    E --> F
    D -.->|NMS-Free Inference| G["End-to-End Final Temporal Span Predictions"]

Key Designs

1. Pyramid temporal prior initialization: constructing multi-scale geometric anchors for varied event durations Standard DETR implementations rely on unconstrained learnable query embeddings, which frequently suffer from slow convergence and unpredictable temporal spans. Because real-world video events vary dramatically in length (from brief actions lasting two seconds to extended sequences spanning dozens of seconds), ME-DETR introduces a multi-scale reference span prior tied to the temporal feature pyramid. Each query \(q_i\) comprises a content embedding \(z_i \in \mathbb{R}^d\) and a reference span \(r_i = (c_i, w_i)\) parameterized by center and temporal duration. Candidate centers are populated across temporal positions of each feature level \(M_l\), where higher-resolution layers receive narrower initial widths and lower-resolution layers receive wider initial spans. Content embeddings \(z_i\) are initialized to zero, allowing the decoder to progressively absorb multimodal context and refine boundary offsets across successive layers.

2. Dual-head decoupled decoder: separating precise boundary regression from broad semantic exploration To prevent boundary regression penalties from corrupting semantic feature representations, the decoder branches into two separate 3-layer MLP heads at each layer. The event head produces a full-event classification score \(c_i^e \in [0, 1]\) and a boundary offset \(\Delta s_i^e\), which updates reference spans to fit complete ground-truth intervals. In parallel, the moment head outputs a semantic classification score \(c_i^m \in [0, 1]\) strictly without regression outputs. This separation isolates the dual objectives: the event head maintains the sharp extremum suppression needed to eliminate NMS, while the moment head establishes an unpenalized optimization channel for prompt-consistent local temporal features.

3. Dynamic moment mining and role assignment: converting suppressed sub-spans into qualitative semantic gains During training, the event head performs standard one-to-one Hungarian matching against ground-truth spans \(\hat{s}_j\) using matching costs balancing classification and boundary distances; the matched queries form the primary event query set \(\mathcal{S}_e\). For the remaining unmatched queries, the framework conducts dynamic moment mining. For any unmatched query \(q_k \notin \mathcal{S}_e\) with predicted span \(s_k\), its containment relative to the matched event span \(s_i\) is evaluated as: $$ \text{Contain}(s_k, s_i) = \frac{|s_k \cap s_i|}{|s_k|} $$ The model filters candidates meeting \(\text{Contain}(s_k, s_i) \ge \tau_{\text{in}}\) (default 0.7) and moment-head confidence \(c_k^m \ge \tau_{\text{cls}}\) (default 0.3), retaining up to Top-\(K\) (default \(K=2\)) highest-confidence queries per event to establish the auxiliary moment set \(\mathcal{S}_m\). Unlike 2D one-to-many schemes (such as Group DETR or H-DETR) that force multiple auxiliary queries to regress the full object box ("quantitative enrichment"), ME-DETR assigns positive labels \(y_k^m=1\) to moment queries exclusively within the moment head and exempts them from any regression loss, while keeping them labeled as negatives in the event head. This role specialization allows sub-events to explore fine-grained semantics freely without interfering with global deduplication.

Loss & Training

ME-DETR is trained end-to-end by optimizing a joint objective that combines event-head and moment-head losses: $$ \mathcal{L} = \mathcal{L}{\text{event}} + \lambda_m \mathcal{L} $$ where }\(\lambda_m=0.2\). The event loss \(\mathcal{L}_{\text{event}}\) consists of a focal classification loss across all queries alongside L1 distance and tIoU losses applied solely to the matched event set \(\mathcal{S}_e\): $$ \mathcal{L}{\text{event}} = \frac{1}{N_q}\sum}^{N_q}\lambda_{\text{cls}}\mathcal{L{\text{cls}}(c_i^e, y_i^e) + \frac{1}{|\mathcal{S}_e|}\sume}\Big(\lambda}|s_i-\hat{s{\sigma(i)}|_1 + \lambda}}\mathcal{L{\text{tIoU}}(s_i,\hat{s})\Big) $$ with loss weights \(\lambda_{\text{cls}}=30\), \(\lambda_{L1}=10\), \(\lambda_{\text{tIoU}}=1\), and ground-truth assignment \(y_i^e=\mathbb{I}(i \in \mathcal{S}_e)\). The moment loss \(\mathcal{L}_{\text{moment}}\) applies focal classification loss over all queries with positive targets defined on the union \(\mathcal{S}_e \cup \mathcal{S}_m\): $$ \mathcal{L}{\text{moment}} = \frac{1}{N_q}\sum_m) $$ The model is trained for 200 epochs using the AdamW optimizer with a base learning rate of }^{N_q}\mathcal{L}_{\text{cls}}(c_i^m, y_i^m), \quad y_i^m = \mathbb{I}(i \in \mathcal{S}_e \cup \mathcal{S\(1 \times 10^{-4}\) and weight decay of \(1 \times 10^{-4}\).

Key Experimental Results

Main Results

ME-DETR was benchmarked extensively across QVHighlights, Charades-STA, and TACoS. On QVHighlights, evaluations categorized by feature layer depth and feature backbone demonstrate that ME-DETR sets new state-of-the-art results using only a single feature layer.

Dataset / Split	Backbone / Feature Layers	[email protected]	[email protected]	[email protected]	[email protected]	Avg. mAP
QVHighlights Val	Clip & SlowFast (1-layer)	65.61	52.58	66.80	52.99	51.62 (+1.77 vs Flash-VTG)
QVHighlights Val	InternVideo (1-layer)	70.71	57.16	71.18	56.88	55.41 (+2.57 vs Flash-VTG)
QVHighlights Test	Clip & SlowFast (1-layer)	66.17	51.22	68.30	52.09	50.02 (+2.43 vs Flash-VTG)
QVHighlights Test	InternVideo (1-layer)	70.10	55.13	71.41	55.52	53.87 (+1.87 vs Flash-VTG)

Across other benchmarks, ME-DETR exhibits strong generalizability: on Charades-STA with InternVideo features, it delivers 61.9% mIoU (outperforming 4-layer SDST at 61.2%); on the complex cooking videos of TACoS, it achieves 43.0% mIoU (surpassing SDST at 42.2% and 1-layer Flash-VTG at 37.6%).

Ablation Study

The core ablation on supervision mechanisms and hyperparameter sensitivities was conducted on the QVHighlights validation set using the InternVideo backbone:

Supervision & Matching Scheme	[email protected]	[email protected]	[email protected]	[email protected]	Avg. mAP	Description & Mechanism
(a) Standard (1-to-1 matching)	70.45	56.22	69.53	53.68	52.83	Traditional DETR; salient sub-events penalized as negative
(b) One-to-Many (2D detection style)	70.91	56.69	69.92	54.21	53.35	Auxiliary positives forced to regress full span (+0.52%)
(c) ME-DETR (Synergistic supervision)	70.71	57.16	71.18	56.88	55.41	Moment head trains semantics without regression (+2.58%)

Hyperparameter Analysis	Values & Results (Avg. mAP)	Design Insight
Moment confidence threshold \(\tau_{\text{cls}}\)	0.2 (55.10) / 0.3 (55.41) / 0.4 (55.09)	Low values introduce noise; high values restrict valid sub-spans
Temporal containment threshold \(\tau_{\text{in}}\)	0.5 (55.09) / 0.6 (55.28) / 0.7 (55.41) / 0.8 (55.16)	0.7 ensures mined moments strictly reside within event bounds
Max moment queries per event \(K\)	0 (52.83) / 1 (55.08) / 2 (55.41) / 3 (55.17) / 5 (54.86)	\(K=2\) offers optimal balance between semantic cues and noise
Moment loss weight \(\lambda_m\)	0.0 (52.83) / 0.1 (55.14) / 0.2 (55.41) / 0.5 (54.98) / 1.0 (54.47)	Excessive weight slightly diverts decoder attention

Key Findings

• Qualitative enrichment triumphs over quantitative scaling: Direct transfer of 2D one-to-many matching yielded a modest +0.52% gain because forcing sub-spans to regress distant global boundaries generated conflicting gradients. In contrast, ME-DETR's regression-exempt semantic supervision produced a +2.58% boost, confirming that respecting temporal part-whole structure is vital.
• Substantial leaps in strict localization metrics: On high-precision thresholds like [email protected], ME-DETR improved performance from 53.68% to 56.88% (+3.20%) on validation and reached 52.09% on the test set with Clip & SlowFast (far higher than Flash-VTG's 48.70%). This reveals that liberating sub-event moments serves as rich contextual anchors that ultimately sharpen full-event boundary predictions.
• Zero test-time computational overhead: Because dual-head role assignment and moment mining operate exclusively during training, inference executes strictly via the event head without NMS or multi-layer feature fusion, maintaining pure real-time efficiency.

Highlights & Insights

• Formulates the concept of "forced semantic suppression" in video temporal grounding, highlighting why the spatial boundary-semantic coupling of 2D detection breaks down under video temporal fluidity.
• Replaces naive target duplication with a query role-specialization philosophy: full-event localization is retained in one head while sub-event semantic discovery is nurtured in another, resolving the fundamental tension between boundary alignment and semantic alignment.
• Fully backwards-compatible and inference-free: achieves new state-of-the-art results across three benchmarks without modifying the deployment-time model graph or adding post-processing.

Limitations & Future Work

• Static thresholding for moment mining: the selection of auxiliary moment queries relies on fixed containment (\(\tau_{\text{in}}=0.7\)) and confidence (\(\tau_{\text{cls}}=0.3\)) thresholds, which may not dynamically adapt to actions with heterogeneous velocities or temporal densities.
• Unmodeled compositional grammar: while auxiliary moments provide strong semantic anchors, the model currently treats multiple moments as an unordered set without explicitly modeling procedural dependencies (e.g., action A preceding action B).
• Promising directions include extending dynamic role specialization to spatio-temporal action localization and grounding-guided alignment in Video Large Language Models (Video-LLMs).

Related Work & Insights

• vs Moment-DETR / Flash-VTG: Baseline DETR architectures penalize all non-matched proposals despite high semantic relevance; ME-DETR removes this suppression via auxiliary moment supervision, producing an Avg. mAP gain of +1.8% to +2.5% under identical 1-layer feature setups.
• vs Group DETR / H-DETR: Image-based one-to-many methods force all positive queries to predict full ground-truth boxes; ME-DETR decouples classification from regression, exempting sub-event queries from boundary losses to eliminate geometric training conflicts.
• vs SDST / R2-Tuning: Prior methods often deploy complex 4-layer multi-scale feature pyramids or rely on post-processing NMS to capture dense events; ME-DETR outperforms them comprehensively using only single-layer features and end-to-end NMS-free decoding.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Provides a profound insight into the paradigm discrepancy between 2D spatial detection and 1D temporal grounding; the regression-free role specialization design is exceptionally clean and conceptually elegant.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across QVHighlights, Charades-STA, and TACoS under multiple backbones, accompanied by thorough ablations against 2D one-to-many paradigms.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation, and convincing visual and empirical justifications.]
• Value: ⭐⭐⭐⭐⭐ [Establishes state-of-the-art benchmarks while maintaining zero inference latency increase, providing an essential template for future VTG research.]