Open-Vocabulary Long Term Action Anticipation

Conference: ECCV 2026
Paper: ECCV Official
Area: Video Understanding
Keywords: Long-term Action Anticipation, Open-Vocabulary, Egocentric Vision, Multi-Horizon Cross-Attention Decoder, Bottleneck Adapters

TL;DR

Addressing the limitation of conventional egocentric action anticipation confined to closed vocabularies and disjoint multi-horizon predictions, this paper introduces the first open-vocabulary long-term action anticipation evaluation framework and proposes a Multi-Horizon Cross-Attention Decoder (MHCAD) integrated with 4D Rotary Position Embeddings and lightweight text bottleneck adapters, substantially outperforming existing adapted models and foundation baselines in zero-shot cross-dataset transfer.

Background & Motivation

Egocentric action anticipation aims to forecast future actions from past video observations, serving as a fundamental capability for intelligent assistive devices, human-robot collaboration, and proactive AI agents. In long-range procedural tasks—such as the long-term anticipation (LTA) challenge in the Ego4D benchmark—models are required to observe \(O=8\) consecutive video segments and predict a sequence of \(Z=20\) future actions (represented as verb-noun pairs) in chronological order. Despite architectural improvements in recent years, prior work has operated exclusively within a closed-set setting where the training and test action classes are identical. This closed-set assumption severely limits real-world deployment, where human-object interactions are open-ended, compositional, and cannot be exhaustively covered during offline training.

However, formulating and solving open-vocabulary long-term action anticipation presents fundamental structural challenges. Existing anticipation paradigms either employ independent multi-head decoders branching from a single pooled video representation or directly fine-tune multi-billion-parameter multimodal LLMs (e.g., AntGPT and Video-LLaVA). Empirical evaluations reveal that these methods struggle significantly under open-vocabulary conditions: foundation LLM/VLM models easily overfit to the label semantics and linguistic distribution of the training set, while multi-head architectures treat distinct future horizons independently, lacking temporal coherence and failing to model sequential inter-dependencies. Furthermore, forecasting near-term versus long-range actions requires attending to different levels of visual granularity, which cannot be adequately captured by a single static summary feature.

To overcome these limitations, this paper introduces the first open-vocabulary evaluation protocol for long-term action anticipation, training exclusively on Ego4D v2 and evaluating in a zero-shot cross-dataset manner on EPIC-Kitchens-100 and EGTEA+, where the action vocabulary overlap is minimal. Core idea: replace disjoint single-feature multi-head decoders with a unified Multi-Horizon Cross-Attention Decoder that conditions horizon-specific learnable queries via causal self-attention, progressively attends to multi-scale visual features encoded with 4D RoPE, and performs open-vocabulary matching against bottleneck-adapted text encoders.

Method

Overall Architecture

The framework builds upon a frozen pretrained visual transformer backbone (such as LaViLa) and dual text encoders for verbs and nouns. Given \(O\) observed video segments, the vision backbone extracts segment-level CLS tokens along with dense multi-scale spatial feature maps from its final \(L_d\) layers. The overall pipeline operates across three primary stages: first, 4D Rotary Position Embeddings (RoPE) are applied to decouple and encode the hierarchical spatio-temporal structure across segments, frames, and 2D spatial coordinates; second, observed CLS tokens and \(Z\) horizon-specific learnable queries are concatenated and fed into the Multi-Horizon Cross-Attention Decoder (MHCAD), where queries progressively interact via causal self-attention and cross-attend to encoder visual features layer by layer; finally, the refined horizon queries are projected through shared linear heads and matched against class embeddings produced by dual text encoders equipped with lightweight bottleneck adapters via cosine similarity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Observed Segments S1...SO"] --> B["4D Rotary Position Embeddings<br/>Segment-Time-Space Decoupling"]
    B --> C["Horizon Query Interaction<br/>Causal Self-Attention"]
    C --> D["Layer-wise Cross-Attention<br/>Multi-Scale Feature Aggregation"]
    D --> E["Shared Projections & Bottleneck Adapters<br/>Open-Vocabulary Cosine Matching"]
    E --> F["Output: Z Future Actions (Verb, Noun)"]

Key Designs

1. 4D Rotary Position Embeddings: Factored Spatio-Temporal Structure Encoding

Prior long-term anticipation models commonly rely on global segment pooling or 1D learned position embeddings, discarding fine-grained object interaction locations and struggling with temporal extrapolation across varied sequence lengths. To preserve essential spatial and sequential cues for novel action recognition, the visual feature extraction retains dense spatial feature maps \(F_i^\ell \in \mathbb{R}^{T \times P \times d}\) from the last \(L_d\) layers of the frozen encoder. Each spatial token's position is decomposed into a 4D tuple \((i, t, h, w)\), representing the segment index \(i \in \{1, \dots, O\}\), frame index within the segment \(t \in \{1, \dots, T\}\), and 2D spatial coordinates \((h, w)\).

The feature dimension \(d\) is split into four equal sub-vectors of dimension \(d' = d/4\), with independent rotary position embeddings applied to each respective sub-vector:

\[\tilde{F}^\ell = \left[ \text{RoPE}_i(f_1^\ell); \, \text{RoPE}_t(f_2^\ell); \, \text{RoPE}_h(f_3^\ell); \, \text{RoPE}_w(f_4^\ell) \right]\]

By leveraging the inner product properties of rotary matrices in the complex domain, 4D RoPE naturally injects relative spatio-temporal positional awareness into cross-attention while maintaining computational factorization and superior length generalization for long-horizon temporal extrapolation.

2. Horizon-Specific Learnable Queries with Causal Self-Attention

Unlike previous models that decode all future steps from a single aggregated representation using separate heads at the final step, MHCAD introduces explicit horizon tokens at the decoder input. The initial query sequence \(Q^{(0)} \in \mathbb{R}^{(O+Z) \times d}\) is composed of \(O\) observed CLS tokens \(\{c^1, \dots, c^O\}\) representing past context, concatenated with \(Z\) horizon-specific learnable queries \(\{q^1, \dots, q^Z\}\) assigned to future time horizons.

To enforce temporal causality and facilitate coherent sequence rollout, each decoder layer first applies causal self-attention across the query sequence:

\[Q_{\text{self}}^{(m-1)} = \text{CausalSelfAttention}(Q^{(m-1)})\]

Under the causal mask, each query token at position \(j\) can only attend to earlier positions \(\le j\) (i.e., observed history and preceding future actions). This strictly prevents future queries from leaking information backwards, ensuring autoregressive consistency and allowing downstream action predictions to build contextually upon upstream decisions.

3. Layer-wise Cross-Attention Feature Aggregation

Different action horizons demand distinct semantic abstraction: immediate next actions rely heavily on low-level hand-object physical contacts, whereas distant future actions require high-level intentional cues. MHCAD realizes this through a progressive, multi-layer cross-attention mechanism across \(L_d\) decoder layers. At decoder layer \(m\), rather than repeatedly attending to a single static visual representation, the queries cross-attend to features extracted from encoder layer \(\ell = L_e - L_d + m\).

The self-attended queries act as Query vectors, while the position-encoded tokens \(\tilde{F}^\ell\) serve as Key and Value vectors:

\[Q'^{(m)} = \text{CrossAttention}(Q_{\text{self}}^{(m-1)}, \tilde{F}^\ell)\]

This is followed by residual connections with stochastic depth (DropPath) and a feed-forward network (FFN). This cascading design allows shallow decoder layers to focus on fine-grained spatial evidence while deeper layers capture broader procedural transitions, progressively refining horizon-specific representations.

4. Shared Projection Heads and Lightweight Bottleneck Text Adapters

To generalize to open vocabularies without discrete category classification heads, the final horizon queries \(\{q^1, \dots, q^Z\}\) from \(Q^{(L_d)}\) are mapped through linear projection heads \(h_v, h_n \in \mathbb{R}^{d \to d_e}\) into visual embeddings \(v_v^j, v_n^j\). Crucially, these projection heads are shared across all \(Z\) future horizons, constraining the model to learn a unified semantic manifold and preventing the severe overfitting and representation drift observed when using per-horizon independent heads.

On the text side, dual BERT-style text encoders for verbs and nouns remain frozen, and task-specific adaptation is achieved by inserting lightweight bottleneck adapters after each transformer block:

\[h_{\text{out}} = h + \alpha \cdot W_{\text{up}} \left( \text{GELU}(W_{\text{down}} h) \right)\]

where \(W_{\text{down}} \in \mathbb{R}^{d_e/4 \times d_e}\), \(W_{\text{up}} \in \mathbb{R}^{d_e \times d_e/4}\), and \(\alpha\) is a learnable scaling factor. Normalized dot products scaled by a learnable temperature \(\tau\) compute logits against candidate text embeddings, enabling zero-shot inference over arbitrary new action vocabularies at test time.

Loss & Training

The entire model is trained directly from scratch on the long-term anticipation objective, avoiding pre-training on action recognition that often introduces task distribution mismatches. The model minimizes standard cross-entropy loss over verb and noun predictions across all \(Z\) future horizons, averaged over time to ensure balanced optimization:

\[\mathcal{L} = \frac{1}{Z} \sum_{j=1}^Z \left[ \mathcal{L}_{\text{CE}}(s_v^j, y_v^j) + \mathcal{L}_{\text{CE}}(s_n^j, y_n^j) \right]\]

Training employs the Adam optimizer with a learning rate of \(5 \times 10^{-4}\) and batch size 16 for 50 epochs. Decoder depth is set to \(L_d = 2\), with DropPath regularization probability \(p_{\text{drop}} = 0.1\), extracting features from 32 uniformly sampled frames per clip.

Key Experimental Results

Main Results

Evaluation employs edit distance (ED ↓, lower indicates better sequential matching). The main results report performance under both closed-set settings (Ego4D splits) and open-vocabulary cross-dataset transfer (trained on Ego4D v2, tested zero-shot on EPIC-Kitchens-100 and EGTEA+).

Dataset / Setting	Method	Params (Frozen / Trained)	Verb ↓	Noun ↓	Action ↓
Ego4D v2 Test (Closed-Set)	Video-LLaVA	303M / 7.2B	-	-	-
Ego4D v2 Test (Closed-Set)	AntGPT	13.3B / 246M	0.650	0.650	-
Ego4D v2 Test (Closed-Set)	SlowFast	63.4M / 181M	0.717	0.736	-
Ego4D v2 Test (Closed-Set)	LaViLa-MultiHead	489M / 91M	0.720	0.727	-
Ego4D v2 Test (Closed-Set)	LaViLa-MHCAD (Ours)	489M / 157M	0.689	0.695	-
EK100 (Open-Vocabulary)	Video-LLaVA	303M / 7.2B	0.925	0.945	0.991
EK100 (Open-Vocabulary)	AntGPT	13.3B / 246M	0.933	0.958	0.993
EK100 (Open-Vocabulary)	XCLIP-MultiHead (Large)	575M / 91M	0.868	0.971	0.985
EK100 (Open-Vocabulary)	LaViLa-MultiHead (Large)	489M / 91M	0.849	0.968	0.982
EK100 (Open-Vocabulary)	LaViLa-MHCAD (Ours Large)	489M / 157M	0.728	0.936	0.969
EGTEA+ (Open-Vocabulary)	Video-LLaVA	303M / 7.2B	0.959	0.970	0.994
EGTEA+ (Open-Vocabulary)	AntGPT	13.3B / 246M	0.961	0.974	0.998
EGTEA+ (Open-Vocabulary)	XCLIP-MultiHead (Large)	575M / 91M	0.912	0.966	0.981
EGTEA+ (Open-Vocabulary)	LaViLa-MultiHead (Large)	489M / 91M	0.904	0.958	0.979
EGTEA+ (Open-Vocabulary)	LaViLa-MHCAD (Ours Large)	489M / 157M	0.864	0.927	0.968

Ablation Study

Ablations systematically evaluate decoder architectures, positional encoding designs, text adapter configurations, and query/head combinations under the open-vocabulary protocol.

Category	Variant Configuration	EK100 Verb ↓	EK100 Noun ↓	EGTEA+ Verb ↓	EGTEA+ Noun ↓	Note
Full Model	LaViLa-MHCAD (Standard)	0.728	0.936	0.864	0.927	4D RoPE + Causal SA + 2-layer Cross-Attn + Adapters
Decoder Design	No self-attention	0.805	0.931	0.869	0.932	Loses query temporal interaction; cross-domain drop
Decoder Design	Non-causal self-attention	0.821	0.939	0.895	0.941	Violates temporal order; causes sequence confusion
Decoder Design	Swap self/cross-attention	0.769	0.937	0.886	0.941	Performing cross-attn before self-attn degrades cue routing
Decoder Design	LaViLa-FUTR decoder	0.836	0.979	0.902	0.982	Single-layer absolute PE + unconstrained SA underperforms
Position Embedding	Dynamic PE (DPE)	0.746	0.962	0.885	0.952	Lacks relative coordinate modeling, weaker localization
Position Embedding	Absolute PE	0.783	0.969	0.904	0.974	Cannot extrapolate well over long multi-clip sequences
Text Adaptation	Adapter + Prompts	0.734	0.938	0.871	0.929	Soft prompts increase risk of in-domain overfitting
Text Adaptation	Prompts Only	0.736	0.937	0.876	0.929	Modifying only input embeddings is insufficient
Query & Heads	Single query + Per-horizon heads	0.759	0.992	0.901	0.970	Single representation cannot span multi-horizon dynamics
Query & Heads	20 queries + Per-horizon heads	0.837	0.947	0.879	0.946	Independent heads overfit to training distributions
Query & Heads	20 queries + Shared heads (Ours)	0.728	0.936	0.864	0.927	Horizon queries + shared head preserves unified semantics

Key Findings

• Foundation LLMs/VLMs Degrade Drastically on Out-of-Distribution Actions: Billion-scale generative models (e.g., AntGPT at 13.3B and Video-LLaVA at 7.2B) achieve strong closed-set scores but fail under zero-shot cross-dataset evaluation, with edit distance exceeding 0.92~0.99. Their open-ended autoregressive decoding suffers from distribution collapse and severe domain hallucination when vocabularies diverge sharply.
• Causal Constraint and Shared Heads Counteract Overfitting: Disabling the causal self-attention mask or adopting per-horizon prediction heads sharply degrades open-vocabulary edit distance (EK100 verb error increases from 0.728 to 0.821 and 0.837, respectively). Causal self-attention enforces ordered temporal reasoning, while shared heads regularize the prediction space into a single coherent visual-text embedding manifold.
• Shallow Decoder Depth Strikes the Optimal Capacity Balance: Increasing decoder depth from 0 (direct classification) to 2 layers yields dramatic gains (EK100 verb error drops from 0.849 to 0.728). However, scaling beyond 2 layers leads to slight performance degradation, indicating that a lightweight 2-layer cross-attention decoder optimally balances temporal representation capacity with out-of-distribution generalization.

Highlights & Insights

• First Open-Vocabulary Long-Term Anticipation Benchmark: Bridges the gap between academic closed-set evaluations and practical deployment by formalizing zero-shot cross-dataset evaluation across Ego4D, EPIC-Kitchens, and EGTEA+.
• Factored 4D Rotary Position Embeddings: Elegantly partitions feature channels across segment index, intra-segment frame order, and 2D spatial coordinates, providing a parameter-free relative spatio-temporal encoding with robust sequence length extrapolation.
• Decoupled Query Specialization and Unified Projection: Allocates 20 distinct learnable queries to capture horizon-specific temporal dependencies, while enforcing semantic invariance through a shared linear head matching against bottleneck-adapted text encoders.

Limitations & Future Work

• Admitted Limitations: Despite significant gains over baselines, open-vocabulary noun prediction error remains relatively high (e.g., 0.936 on EK100), reflecting the steep challenge of recognizing and forecasting novel fine-grained objects purely from egocentric visual cues.
• Identified Limitations: The model assumes a fixed anticipation horizon length (\(Z=20\)), lacking support for dynamic or variable-length temporal planning. Additionally, cross-dataset testing is currently concentrated in culinary settings (EPIC-Kitchens, EGTEA+), leaving generalizability to outdoor, assembly, or clinical environments to be verified.
• Future Directions: Integrating explicit 3D spatial object priors or open-vocabulary detection-tracking backbones (such as Grounding DINO) could provide stronger object-centric representations to improve noun forecasting accuracy.

Related Work & Insights

• vs AntGPT / Video-LLaVA: Generative foundation models rely on multi-billion-parameter LLM backbones that suffer from vocabulary bias and hallucination in open-vocabulary settings; MHCAD requires only 157M trainable parameters and significantly outperforms them via structured visual decoding and bottleneck text adaptation.
• vs SlowFast / VCLIP / LaViLa-MultiHead: Conventional methods pool video features into a single global vector and decode through disjoint classification heads; MHCAD introduces continuous temporal causal modeling via horizon queries and enables zero-shot open-vocabulary transfer through text similarity matching.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes the first open-vocabulary long-term action anticipation protocol with novel 4D RoPE and horizon-specific decoding.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three major benchmarks, spanning closed-set and open-vocabulary setups, varied backbone scales, and exhaustive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, well-structured methodology, and comprehensive empirical analyses.
• Value: ⭐⭐⭐⭐⭐ Offers crucial benchmarking guidelines and architectural insights for deploying proactive egocentric AI assistants in open-world environments.