DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding¶

Conference: ECCV2026
Paper: ECCV Paper
Project: DART
Area: Video Understanding
Keywords: zero-shot video temporal grounding, determinantal point process, spectral entropy, difficulty-adaptive routing, temporal markup prompting

TL;DR¶

DART uses one query-conditioned DPP kernel for both keyframe selection and difficulty estimation, activates structured temporal reasoning only for complex events, and achieves 48.93 mIoU on Charades-STA and 39.89 on ActivityNet Captions without task-specific fine-tuning.

Background & Motivation¶

Video temporal grounding must find the beginning and end of an event in an untrimmed video from a natural-language query, rather than merely recognize relevant objects or actions. Zero-shot methods commonly use pretrained vision-language models to place frame and query representations in a comparable feature space, then merge high-scoring frames into a temporal segment. This works naturally for a single action such as sitting down, but can reduce a query about flipping, dismounting, and landing on a mat to the most easily matched landing frames. On 100 simple and 100 complex queries sampled from ActivityNet Captions val_2, the paper reports that the strongest feature-matching method drops from 42.2 to 30.6 mIoU, a gap of 11.6 percentage points. Complexity here primarily involves temporal ordering, causal relations, or conditions; it is not simply sentence length or video length.

Asking a large vision-language model to use chain-of-thought reasoning on every query does not automatically close this gap. When the input omits a critical action stage, reasoning operates on incomplete evidence; when it contains redundant or irrelevant frames, intermediate steps can be distracted. Simple events, meanwhile, do not require stage-by-stage explanations, and additional reasoning can increase generation time while introducing incorrect boundaries. The paper therefore addresses two concrete questions together: which frames suffice for localization, and which video-query pairs warrant structured reasoning.

The authors connect these questions through one relation matrix that encodes frame-query relevance, visual differences between frames, and temporal distance, then examine its significant spectral components. Intuitively, evidence for a single-stage event is concentrated, whereas relevant evidence for a multi-stage event occupies several distinct modes. This estimates difficulty from visual evidence structure rather than training an additional query classifier or asking the language model to report its confidence. Core Idea: use a query-conditioned DPP to supply both compact temporal evidence and a spectral-entropy difficulty signal, then route events requiring reasoning to temporal markup prompting with explicit intermediate outputs.

Method¶

Overall Architecture¶

The input is an untrimmed video and a natural-language query; the output is a segment represented by start and end timestamps. The video is first downsampled to 3 FPS, frame and query features are extracted, and the resulting representations enter Query-Conditioned Evidence Selection. Rather than simply taking the most similar frames, this stage balances relevance and complementarity through a DPP and adapts the frame count to the information contributed by additional frames. Spectral-Entropy Difficulty Routing then reads the eigenvalue distribution of the complete query-conditioned kernel instead of merely counting the selected frames. Finally, Dual-Path Temporal Localization passes the same timestamped keyframes to either fast direct prediction or slow temporal markup prompting. The slow path performs global event analysis, per-frame role annotation, and boundary extraction within a single generation, not three separate calls.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video and query<br/>Downsampling and feature extraction"] --> Evidence["Query-Conditioned<br/>Evidence Selection"]
    Evidence --> Routing["Spectral-Entropy<br/>Difficulty Routing"]
    Routing -->|Low entropy: direct prediction| Localize["Dual-Path<br/>Temporal Localization"]
    Routing -->|High entropy: temporal markup prompting| Localize
    Evidence -.->|Timestamped keyframes| Localize
    Localize --> Output["Start and end timestamps"]

Key Designs¶

1. Query-Conditioned Evidence Selection: put relevance and cross-stage complementarity in one kernel

Similarity ranking can repeatedly select the same salient action stage, while uniform sampling can skip a brief but important transition. The determinant of a DPP subset can be interpreted as the volume spanned by its evidence: a valuable subset contains important frames that are not redundant with one another. DART first multiplies a visual Gaussian kernel by a temporal Gaussian kernel, then weights the kernel's rows and columns by query relevance. Visually similar frames that are far apart receive lower pairwise similarity through the temporal kernel, allowing both to be retained. This is useful for repeated actions and temporally separated event stages, preventing similar appearance from automatically implying redundant evidence. Writing frame features as \(f_i\), the query feature as \(q\), and frame timestamps as \(t_i\), the paper's core kernel construction is:

\[ w_i=\sigma(f_i^{\top}q/\tau),\qquad L_{ij}=k_v(f_i,f_j)k_t(t_i,t_j),\qquad \widetilde L_{ij}=\sqrt{w_i}\,L_{ij}\sqrt{w_j}. \]

The sigmoid weights suppress frames unrelated to the query, while the two Gaussian kernels represent visual and temporal similarity. The weighted subset determinant equals the base subset determinant multiplied by the product of the selected frame weights, so relevance and diversity jointly affect selection. The algorithm repeatedly adds the frame with the largest marginal gain rather than exhaustively evaluating every subset. Early selections generally cover the most informative content, while later additions become redundant, motivating a stopping criterion based on declining relative gain. The paper states the condition \(\Delta_k/\Delta_1<\varepsilon\), with default \(\varepsilon=0.05\), instead of allocating the same frame count to all queries. The final \(K\) keyframes and their timestamps are passed to the localization model; \(K\) is adaptive to each video-query pair.

Equations (1)–(2) on cached pages 6–7 are visibly corrupted by text extraction, so the full marginal-gain expression cannot be reliably recovered from them. The prose describes a log-determinant gain and the joint contribution of diversity and query relevance; this note retains that mechanism and the readable stopping condition without guessing the damaged formulas. In particular, reproducing the method requires checking the gain's sign, its handling near zero, and the stopping implementation against the original formulas or code. The paper's broad statement about greedy approximation guarantees should not be read as establishing identical guarantees for every kernel and stopping rule.

2. Spectral-Entropy Difficulty Routing: assess evidence structure rather than frame count

Walking from a living room to a kitchen can require many frames because the scene changes, yet the action remains straightforward. Picking up a phone and answering it can be covered by few frames, but involves distinct sub-actions with a temporal relationship. DART therefore does not equate its adaptive frame count with difficulty; it eigendecomposes the complete query-conditioned kernel. A few dominant eigenvalues indicate that relevant evidence occupies a few modes, whereas distributed energy indicates several independent relevant modes. The authors interpret the latter as richer event-stage structure and use it to decide whether to activate the slow path. For \(M\) candidate frames, the eigenvalues are normalized and used to compute normalized spectral entropy:

\[ q_i=\frac{\lambda_i}{\sum_j\lambda_j},\qquad H_{\mathrm{spectral}}=-\frac{1}{\log M}\sum_{i=1}^{M}q_i\log q_i. \]

Here \(q_i\) denotes a spectral probability, distinct from the query vector \(q\) used earlier. The entropy lies in \([0,1]\): concentrated energy gives low values, while comparable energy across multiple eigenvalues gives higher values. Queries follow the fast path when \(H_{\mathrm{spectral}}\leq\theta\) and the slow path otherwise, with default \(\theta=0.45\). Reusing the kernel makes both evidence selection and routing query-conditioned rather than sensitive only to general visual variation throughout the video. However, visual similarity and temporal distance do not explicitly encode causal direction; the correspondence between spectral modes and event stages is the authors' interpretation and empirical hypothesis. The subsequent difficulty-bin experiment supports correlation, not a claim that spectral entropy identifies actual causal chains.

3. Dual-Path Temporal Localization: generate constrained event structure only when needed

The fast path receives keyframes, timestamps, and a localization instruction specifying the output format, then directly produces the start and end times without intermediate reasoning. It remains an LVLM prediction, so its implementation should not be equated entirely with a manually specified similarity-threshold algorithm. The slow path uses Temporal Markup Prompting (TMP), not merely an instruction to think step by step. Its first step is global temporal reasoning: decompose the query into key actions, establish their sequential, causal, or conditional relationships, and outline a rough temporal range. This asks the model to determine the stages belonging to the complete event before making frame-level judgments, avoiding premature boundaries around salient frames. The second step annotates every keyframe with one of before, start, during, end, or after. These labels describe positions relative to the complete target event, not a separate interval for every sub-action. The labels must also be monotonically non-decreasing over time, so the sequence cannot return to an event start after declaring its end. This structure connects the relationship between flipping, dismounting, and landing directly to interval prediction rather than merely producing explanatory prose.

The third step extracts boundaries: the first frame labeled start supplies the start timestamp, and the last frame labeled end supplies the end timestamp. Unlike direct continuous boundary regression, this makes slow-path boundaries depend on the temporal resolution of the selected keyframes. The paper argues that diversity selection preserves context outside the event, but query weighting can still suppress background frames, so such context is not guaranteed. If downsampling or DPP selection drops a critical stage, label constraints cannot recover visual information that was never observed. Moreover, the paper describes prompting constraints without specifying a constrained decoder for invalid label sequences or a fallback for missing start or end labels. Monotonic annotation is therefore a requirement of the method, not a demonstrated property of every generated sequence. This also explains why evidence selection is a prerequisite: the slow path organizes and interprets available evidence rather than replacing the need for appropriate evidence.

A Worked Example¶

Consider the gymnastics event in Figure 1: the query covers flipping, then dismounting and landing on a mat, rather than only the instant of landing. The following illustrates the mechanism; it does not reproduce measured frame labels or timestamps from the figure. Candidate frames may contain many similar preparation and landing images, so the query-conditioned DPP should retain complementary evidence for flipping, dismounting, and landing. If these relevant stages spread the spectral energy enough to exceed the entropy threshold, the pair enters the slow path. Global analysis first organizes the actions into one target event; frame annotation then identifies positions before, at the start of, during, at the end of, and after that event. Relevant flipping frames should inform the start decision, while post-landing frames help delimit the end; the interval is then extracted from the start and end labels. For a simple single-stage event with low entropy, the same type of keyframe input can instead follow the fast path without generating these labels.

Loss & Training¶

DART introduces no task loss and performs no task-specific fine-tuning on grounding data; its capabilities come from pretrained models and inference-time selection, routing, and prompting. The backbone is LLaVA-1.6-7B, and all experiments use a single NVIDIA A100 80 GB GPU. The visual kernel bandwidth is \(\sigma_v=0.5\), the temporal bandwidth is \(\sigma_t=2.0\), and the relevance temperature is \(\tau=0.1\). The paper explicitly selects these three parameters by grid search on a Charades-STA validation split, so training-free does not mean validation-free hyperparameter selection. It also sets \(\varepsilon=0.05\) and \(\theta=0.45\); inference uses greedy decoding with a maximum generation length of 512 tokens. The main text does not further explain how that validation split is constructed, leaving the parameter-selection protocol to be clarified by reproduction materials. Frame and query features are described as coming from the same LVLM's visual and text encoding components, but the precise layers, pooling, and feature alignment are not fully specified. This note does not present these missing details as author-provided implementation steps.

Key Experimental Results¶

Main Results¶

Table 1 on page 11 evaluates Charades-STA test and ActivityNet Captions val_2; the following excerpt includes three zero-shot methods, with all metrics in percent. Charades-STA test contains 3,720 video-query pairs, and ActivityNet Captions val_2 contains 17,031 pairs. [email protected] and [email protected] are the proportions of predicted intervals exceeding the respective IoU thresholds; mIoU averages IoU over all predictions.

Dataset	Method	[email protected]	[email protected]	mIoU
Charades-STA	TFVTG	49.97	24.32	44.51
Charades-STA	TAG	48.58	26.67	45.69
Charades-STA	DART	52.04	29.45	48.93
ActivityNet Captions	TFVTG	27.02	13.39	34.10
ActivityNet Captions	TAG	28.91	15.07	36.55
ActivityNet Captions	DART	32.14	18.11	39.89

DART improves mIoU over TAG by 3.24 and 3.34 percentage points, respectively; these are not relative percentage gains of the same magnitudes. Its advantage does not cover every zero-shot metric: GranAlign reaches 34.0 [email protected] on ActivityNet Captions in the same table, above DART's 32.14. Out-of-distribution evaluation covers temporal-location shifts, Charades-CD, Charades-CG, and cross-dataset evaluation. Table 2 on page 12 reports ActivityNet Captions OOD-1 mIoU of 39.7 for DART and 36.2 for TAG, a difference of 3.5 percentage points. Table 3 on page 12 reports DART mIoU of 45.30 and 51.09 for novel compositions and novel words; however, its novel-word [email protected] of 55.61 trails TFVTG's 56.26.

Ablation Study¶

Table 6 on page 13 changes individual parts of the full system on Charades-STA IID rather than retraining models under different supervision regimes.

Config	[email protected]	[email protected]	mIoU	Change
Full DART	52.04	29.45	48.93	Adaptive evidence and routing
w/o DPP	46.79	25.55	44.70	Uniform sampling; routing and TMP retained
w/o Adaptive K	49.01	26.73	47.26	Fixed K=8
w/o TMP	48.65	25.92	46.13	Generic CoT replaces TMP
Fast only	46.71	25.34	44.34	Direct prediction for every query
Slow only	49.80	26.78	47.53	TMP for every query

Removing DPP costs 4.23 mIoU; uniform sampling with routing and TMP reaches 44.70, only slightly above Fast only at 44.34, indicating that reasoning gains depend on evidence quality. Generic CoT trails TMP by 2.80 mIoU, fixing the keyframe budget costs 1.67 mIoU, and adaptive routing exceeds the all-slow configuration by 1.40 mIoU. The selection comparison in Table 5 on page 13 additionally reports 47.38 mIoU for Top-k + NMS versus 48.93 for DPP, a difference of 1.55 percentage points.

Table 7 on page 13 measures efficiency on Charades-STA with one A100; time is seconds per query, and frames denote the reported per-query processing amount rather than the raw video's total frame count.

Method	Frames	Time (seconds/query)
TFVTG	86	4.7
TAG	86	3.4
DART (Slow)	12	5.1
DART (Adaptive)	12	3.9

Key Findings¶

Adaptive DART reduces latency from 5.1 to 3.9 seconds relative to the all-slow configuration, a reported reduction of 23.5%, while mIoU rises from 47.53 to 48.93.
Processing 12 rather than 86 frames substantially compresses the visual input, but DART's 3.9 seconds remains slower than TAG's 3.4 seconds; it is not a speed improvement over TAG.
Table 8 on page 14 samples 200 ActivityNet Captions validation queries in each of five equal-width spectral-entropy bins, totaling 1,000 queries, and evaluates only the fast path.
Across bins from \([0,0.2)\) to \([0.8,1.0]\), mIoU is 49.2, 43.7, 37.1, 31.8, and 27.4, supporting a correlation between entropy and direct-localization difficulty; this balanced bin sample does not represent the natural query-difficulty distribution.

Highlights & Insights¶

Reuse evidence structure for routing. Difficulty estimation reads a signal from the kernel already constructed for keyframe selection instead of invoking another model; query conditioning prevents routing from reflecting only background changes.
Make intermediate representations determine boundaries. TMP labels are not disconnected explanatory text: start and end times are extracted from them, making it easier to inspect which stage failed.
Test whether more reasoning is beneficial. Fast only, Slow only, and adaptive routing show that universal reasoning is not optimal, although this remains an empirical result for the evaluated backbone and datasets.

Limitations & Future Work¶

Sparse evidence and boundary precision. Slow-path boundaries use keyframe timestamps, potentially limiting short-action and fine-boundary accuracy; dense local sampling after coarse localization is a reader suggestion, not a validated result here.
Entropy is not a reasoning-capability detector. Complex backgrounds, repeated events, or encoding biases can affect the spectrum; the paper provides no systematic cross-backbone difficulty calibration or routing-error statistics.
Reproducibility details remain incomplete. The cached gain formulas are damaged, and the main text does not fully specify prompts, invalid-label fallbacks, feature extraction, or numerical handling of near-zero kernels.
Cross-dataset results contain an unexplained difference. Table 4 on page 13 reports DART's top-choice [email protected] as 53.17 and [email protected] as 27.84, whereas Table 1 on the same Charades-STA benchmark reports 52.04 and 29.45; the text does not sufficiently explain the zero-shot configuration difference, so the tables should not be merged as one setting.
Validation and cost scope. Evidence mainly comes from two datasets, one backbone, and one A100, without confidence intervals across random seeds; the claim that eigendecomposition cost is negligible also lacks separate timing at long-video scale.

Versus TFVTG. TFVTG decomposes queries into sub-events, matches them, and aggregates results; DART additionally decides when structured temporal reasoning is needed and selects complementary evidence.
Versus TAG and GranAlign. TAG is the main mIoU baseline, while GranAlign aligns multi-granularity queries with video descriptions; DART differs through query-conditioned evidence selection and difficulty routing, not universal superiority on retrieval metrics.
Versus generic CoT. DART requires global relation analysis and a complete monotonic label sequence, constraining intermediate reasoning; its ablation supports this task structure over open-ended step-by-step explanation.
Transferable insight. Video question answering and long-video retrieval could allocate reasoning by first estimating evidence dispersion, but routing calibration, missing-evidence risk, and actual end-to-end cost need separate validation.

Rating¶

Novelty: 4/5. DPP is established; the contribution lies in shared-kernel evidence selection and entropy routing, together with grounding-specific intermediate labels.
Experimental Thoroughness: 4/5. IID, multiple OOD settings, component comparisons, and efficiency are covered, but cross-backbone validation and routing-error analysis are missing.
Writing Quality: 3/5. Motivation and flow are clear, while implementation details and cross-table differences require clarification; damaged cached formulas are a separate extraction limitation.
Value: 4/5. An interpretable reasoning-budget allocation approach for video grounding without task-specific fine-tuning, relevant to complex-event understanding and deployment costs.