Skip to content

Efficient Few-Shot Action Recognition via Multi-Level Post-Reasoning

Conference: ECCV 2024
Code: https://github.com/cong-wu/EMP-Net
Area: Video Understanding
Keywords: Few-Shot Action Recognition, CLIP, Post-Reasoning Mechanism, Multi-Level Representation, Efficient Fine-Tuning

TL;DR

EMP-Net proposes an efficient multi-level post-reasoning network. It reduces the domain alignment overhead of CLIP in few-shot action recognition by avoiding most gradient backpropagations through a post-reasoning mechanism. Meanwhile, it leverages multi-level representations (global, patch, and frame levels) to enhance feature discriminativeness, achieving an optimal balance between efficiency and performance.

Background & Motivation

Background: Few-Shot Action Recognition (FSAR) aims to recognize novel action categories using very few labeled videos, representing an important research direction in video understanding. In recent years, the introduction of large-scale vision-language pre-trained models such as CLIP (Contrastive Vision-Language Pre-training) has significantly updated the accuracy leaderboards of FSAR. By leveraging the powerful visual and textual representation capabilities of CLIP, FSAR methods can achieve better generalization performance under few-shot conditions.

Limitations of Prior Work: When adapting CLIP to the FSAR task, the primary challenge lies in the training overhead of domain alignment. Since CLIP is pre-trained on image-text pairs, applying it directly to video-level action recognition introduces a domain gap, necessitating alignment through fine-tuning or adapters. However, ensuring domain alignment between CLIP and FSAR typically requires substantial gradient backpropagation, which leads to two issues: (1) high computational cost, as backpropagation must be performed across the entire CLIP encoder; and (2) risk of overfitting, as fully fine-tuning large models under extremely few-shot conditions easily leads to overfitting.

Key Challenge: The contradiction between efficiency and effectiveness—fully exploiting the representation capability of CLIP requires deep fine-tuning (computationally intensive, prone to overfitting), while maintaining efficiency limits alignment to shallow layers (insufficient performance). Moreover, most existing methods only utilize the features from the final layer of CLIP, ignoring the rich multi-granularity information contained in the intermediate layers.

Goal: (1) How to effectively achieve domain alignment between CLIP and FSAR while avoiding massive gradient backpropagation? (2) How to fully utilize the multi-stage features of CLIP to improve the discriminativeness of action recognition? (3) How to design an FSAR framework that is both efficient and effective?

Key Insight: The authors propose the concept of "Post-Reasoning"—instead of performing domain adaptation during the forward/backward propagation of the CLIP encoder, a lightweight reasoning module is used to complete domain alignment and spatiotemporal modeling after the CLIP feature extraction is finished. In this way, the CLIP encoder can be completely frozen, requiring no gradient updates, which significantly reduces the computational overhead.

Core Idea: Freeze the CLIP encoder and cache multi-stage features, and then perform multi-level (global, patch, frame) spatiotemporal reasoning and matching via a lightweight post-reasoning module after feature extraction, achieving efficient few-shot action recognition.

Method

Overall Architecture

The pipeline of EMP-Net consists of three stages: (1) Skip-Fusion: extracting and caching multi-stage intermediate features from the frozen CLIP vision encoder, and fusing them through skip connections to obtain rich multi-level representations; (2) Multi-Level Decoupling & Spatiotemporal Reasoning: decoupling the fused features into three representations: global-level, patch-level, and frame-level, and performing spatiotemporal reasoning separately to generate discriminative features; (3) Joint Text-Visual & Support-Query Matching: combining text-visual contrast and support-query matching to make final classification decisions. Throughout the entire process, the CLIP encoder is completely frozen, and only the lightweight post-reasoning module needs to be trained.

Key Designs

  1. Skip-Fusion Module:

    • Function: Extract and fuse features from multiple intermediate layers of the CLIP encoder to construct more informative video representations.
    • Mechanism: The ViT encoder of CLIP contains multiple Transformer layers, where different layers capture semantic information of different granularities—shallow layers focus on low-level features such as textures and edges, while deep layers focus on high-level features such as semantics and categories. The Skip-Fusion module fuses intermediate features from multiple stages into a unified multi-level feature through weighted summation or concatenation. Since the CLIP encoder is frozen, these intermediate features can be pre-computed and cached, avoiding the need for repeated forward passes at each iteration.
    • Design Motivation: Using only the final layer output of CLIP discards rich intermediate layer information. Multi-stage fusion preserves the complete feature spectrum from low-level to high-level, providing a better foundation for subsequent multi-level reasoning.

  2. Multi-Level Decoupling & Spatiotemporal Reasoning:

    • Function: Decompose the fused features into three complementary representation levels and perform targeted spatiotemporal reasoning for each.
    • Mechanism: The fused features are decoupled into three levels—(a) Global-Level: performing global pooling over all tokens to obtain the holistic semantic representation at the video level; (b) Patch-Level: retaining spatial patch tokens to capture local action details (e.g., hand movements, object interactions); (c) Frame-Level: organizing tokens along the temporal dimension to capture the temporal dynamics of the action. Features of each level are processed by their respective spatiotemporal reasoning modules (lightweight Transformer layers or MLPs) to learn level-specific spatiotemporal patterns. The reasoning results of the three levels are finally combined through weighted fusion to obtain the final video features.
    • Design Motivation: Action recognition requires simultaneous attention to global semantics ("what action it is"), local details ("what the hand is doing"), and temporal variations ("the phases and rhythm of the action"). A single-level representation struggles to capture all three aspects concurrently.

  3. Joint Matching:

    • Function: Combine two complementary matching signals to provide more accurate few-shot recognition.
    • Mechanism: During the matching stage, two contrastive signals are integrated—(a) Text-Visual Contrast: utilizing CLIP's text encoder to encode category names/descriptions into text features, computing the cosine similarity with video features; (b) Support-Query Contrast: directly calculating the feature distance between the query video and the support set videos. The two contrastive scores are combined into final classification probabilities using learnable weighting coefficients. Text-visual contrast provides language prior constraints (i.e., the semantic information carried by the category names themselves), while support-query contrast provides a direct criterion for visual similarity.
    • Design Motivation: Pure visual matching might be limited by the lack of representativeness under few-shot scenarios. Text-visual matching utilizes the pre-trained cross-modal alignment knowledge of CLIP. Combining the two provides more robust classification.

Loss & Training

Standard cross-entropy loss is used for few-shot classification training. Since the CLIP encoder is completely frozen, only the parameters of the post-reasoning module and the fusion layer need to be optimized, which requires far fewer trainable parameters than end-to-end fine-tuning schemes. Training adopts an episodic training strategy (i.e., randomly constructing N-way K-shot tasks for each episode), corresponding to the standard paradigm of few-shot learning.

Key Experimental Results

Main Results

Dataset Setting EMP-Net Prev. SOTA Gain / Comparison
SSv2 (5-way 1-shot) temporal-heavy Best CLIP-FSAR, etc. Improved accuracy + significantly boosted training efficiency
SSv2 (5-way 5-shot) temporal-heavy Best Ditto Consistent improvement
Kinetics (5-way 1-shot) appearance-heavy Best or comparable Ditto Comparable accuracy but obvious efficiency advantages
Kinetics (5-way 5-shot) appearance-heavy Best Ditto Superior in both accuracy and efficiency
HMDB51 Small-scale Best Ditto Consistent improvement
UCF101 Large-scale Best Ditto Consistent improvement

EMP-Net achieves state-of-the-art or comparable accuracy across multiple standard FSAR benchmarks, while incurring significantly lower training overhead compared to end-to-end fine-tuning schemes.

Ablation Study

Configuration Accuracy (SSv2 5w1s) Training Cost Note
Full fine-tune CLIP Highest (slight margin) Highest End-to-end fine-tuning is effective but extremely expensive
Global-level only Baseline Lowest Single-level discriminativeness is insufficient
Global + patch +2.1% Slightly increased Patch-level supplements spatial details
Global + patch + frame +3.5% Medium Three-level combination is optimal
Without text-visual contrast -1.8% Same Lacks language prior
Without skip-fusion -2.3% Same Using only the final layer features is not rich enough

Key Findings

  • Efficiency Advantages of Post-Reasoning: Freezing the CLIP encoder combined with lightweight post-reasoning reduces training costs several-fold with minimal or no loss in accuracy. This indicates that the pre-trained features of CLIP are inherently of high quality, requiring no intensive fine-tuning.
  • Complementarity of Multi-Level Representations: Global-level provides the semantic foundation, patch-level provides spatial details, and frame-level provides temporal dynamics, all of which are indispensable. Frame-level representations yield the largest gain on temporally sensitive datasets (e.g., SSv2).
  • Practical Value of Multi-Stage Feature Caching: Pre-computing and caching intermediate features of CLIP avoids repeated forward propagation, which saves considerable resources for large-scale experiments.
  • Stable Gains from Text-Visual Contrast: Incorporating the category semantic information from the CLIP text encoder yields a stable 1-2% improvement across almost all datasets.

Highlights & Insights

  1. The concept of "Post-Reasoning" is highly generalizable—placing domain adaptation after feature extraction rather than during the process is a universal and efficient paradigm for leveraging large pre-trained models.
  2. The multi-level decoupling scheme is distinct; selecting global, patch, and frame granularities covers the main information dimensions required for action recognition.
  3. The source code is open-sourced (GitHub) with strong reproducibility, presenting a direct value contribution to the community.

Limitations & Future Work

  1. The design of the post-reasoning module is relatively simple (MLPs or lightweight Transformers). More complex spatiotemporal reasoning structures (e.g., graph networks or causal reasoning) might yield further improvements.
  2. Text prompts currently use simple category names; richer prompt engineering (e.g., action description templates) can be explored to enhance text-visual matching.
  3. Under extreme few-shot (e.g., 1-shot) scenarios, there is still substantial room for performance improvement. Incorporating meta-learning or data augmentation strategies could be considered.
  4. The current processing of frame-level representations is basic, which might be insufficient for long videos and actions with complex temporal structures.
  5. Comparisons with other parameter-efficient fine-tuning methods (e.g., LoRA, Adapters) are not sufficiently comprehensive.
  • CLIP-based FSAR Methods: Methods like CLIP-FSAR and AIM adapt CLIP to few-shot video tasks, but require training adapter modules on top of CLIP. EMP-Net proposes a cleaner "freeze-and-post-reason" scheme.
  • Efficient Model Adaptation: Methods such as Prompt Tuning, LoRA, and Adapters focus on adapting pre-trained models efficiently. The post-reasoning in EMP-Net shares similarities with these methods but differs fundamentally in its operational stage (post-processing vs. inline).
  • Multi-Level Video Representations: Classical methods like TSN and SlowFast also focus on multi-scale/multi-level video features. EMP-Net introduces this concept into the CLIP-based few-shot framework.
  • Insights: The efficient paradigm of post-reasoning can be extended to other downstream tasks requiring large model adaptation, such as few-shot object detection and few-shot image segmentation.

Rating

  • Novelty: ⭐⭐⭐⭐ The post-reasoning mechanism is a novel and practical contribution, providing a fresh perspective on the efficient utilization of large pre-trained models.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive evaluation across multiple benchmarks, detailed ablation studies, and efficiency comparisons are provided.
  • Writing Quality: ⭐⭐⭐⭐ Clear motivation of the methodology, and logical cohesion in the pipeline design.
  • Value: ⭐⭐⭐⭐ Balancing efficiency and performance is a key pain point in practical applications, and this paper presents a valuable solution.