Skip to content

OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization

Conference: ECCV 2026
arXiv: 2607.00289
Code: None (unreleased due to NDA, but detailed implementation descriptions and pseudocode are provided)
Project Page: https://sakibreza.github.io/OnPoint/
Area: Video Understanding
Keywords: Point-Supervised Temporal Action Localization, Online Learning, Offline-to-Online Distillation, Actionness-Calibrated Attention, Sliding Window Detection

TL;DR

OnPoint proposes an offline-to-online multi-level distillation framework. It utilizes an offline teacher model trained with only single-frame point annotations to generate pseudo-segment labels, frame-level Class Activation Sequences (CAS), and window-level action anticipation signals. These are injected into a strictly online student model through three levels of distillation (instance-level, frame-level, and window-level), complemented by actionness-calibrated attention decoding and anchor-level raw point supervision to stabilize training. OnPoint consistently outperforms strong baselines across five datasets (with an average mAP improvement of 4.8% and a maximum of +7.0% on THUMOS), initiating the novel task of Point-supervised Online Temporal Action Localization (POTAL).

Background & Motivation

Background: Mainstream Temporal Action Localization (TAL) methods rely on two strong assumptions—requiring complete action start and end boundary annotations during training (segment-level supervision) and access to all frames of the entire video during inference (offline inference). Recently, Online Temporal Action Localization (OnTAL) has relaxed the inference assumption to allow frame-by-frame output in streaming videos but still requires full annotations for training. Meanwhile, Point-Supervised Temporal Action Localization (PS-TAL) has relaxed the annotation assumption by needing only one timestamp annotated per action instance during training but still relies on offline full-video inference.

Limitations of Prior Work: The relaxation of both directions has never been simultaneously satisfied. OnTAL methods (e.g., OAT, HAT, MATR) require expensive full-segment annotations, making the annotation cost prohibitively high in continuously recorded real-world scenarios. Conversely, PS-TAL methods (e.g., HR-Pro, TSASPC, LACP), despite improving annotation efficiency by up to 6 times, rely on the global context of the entire video to infer boundaries from sparse points, making them unsuitable for streaming inference. Simply combining the two leads to a collapse in annotation assumptions or breaks the online constraint. Training an online model directly from point annotations yields extremely poor performance due to the lack of boundary information and future context (the distillation-free baseline achieves only 33.3% average [email protected]:0.5 on THUMOS).

Key Challenge: Point annotations only provide "roughly where the action is" without starting or ending boundaries, while online inference deprives the model of the ability to look forward. The combination of these factors means the model neither knows where actions start or end, nor can it infer boundaries by observing the complete video. This is the fundamental reason why POTAL is significantly more challenging than PS-TAL or OnTAL alone.

Goal: (1) Formally define the new POTAL task and establish evaluation protocols and strong baselines; (2) Propose a method that works effectively on this task, narrowing the performance gap compared to fully-supervised online methods and offline point-supervised methods.

Key Insight: The authors observe that while point annotations themselves carry minimal information, an offline model can "amplify" them into high-quality pseudo-segment labels and frame-level activation sequences by leveraging full-video context. If an offline teacher first digests the entire video to generate structured intermediate representations, and then distills this knowledge into a strictly causal online student, the difficulty of "online models learning directly from sparse points" can be bypassed. While this idea of offline-to-online distillation has precedents in video instance segmentation and spatio-temporal action detection, it has never been explored in TAL.

Core Idea: Utilize the full-video inference capability of a point-supervised offline TAL teacher. Through three distillation paths—pseudo-segment labels, frame-level CAS alignment, and window-level action anticipation—the "knowledge only available in full videos" is injected into an online student that only sees a sliding window.

Method

Overall Architecture

OnPoint is a "teacher-student" distillation framework. The core mechanism is to allow an offline TAL model trained with point supervision to observe the entire video first to generate high-quality pseudo-labels and intermediate representations, and then train a strictly online student model using multi-level distillation signals. The teacher model is frozen during training and discarded during inference; only the student model is deployed.

The input is a feature sequence of a streaming video, processed window-by-window by the student model. At each time step \(t\), a feature window \(X_t\) consisting of the current frame and the preceding \(W-1\) frames is extracted. Window features \(F_t\) are obtained via a Transformer encoder and passed to three outputs: the CASS predictor outputs frame-level class scores, the action anticipation head outputs a multi-hot vector indicating potential actions in the future window, and the anchor decoder generates anchor features combined with actionness-calibrated attention for classification and regression to yield action proposals. Finally, Online NMS (ONMS) is applied for duplication removal and final proposal output.

The teacher model provides three distillation signals: (1) instance-level pseudo-segment labels to supervise anchor classification and regression; (2) frame-level CAS to supervise the CASS predictor; (3) binarized CAS of future windows to supervise the anticipation head. Additionally, the raw point annotations are injected into the student via an auxiliary anchor-level point prediction head, providing reliable supervision independent of the teacher.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Full Video Features"] --> B["Offline Teacher<br/>HR-Pro / TSASPC"]
    B --> C["CAS (Frame-Level Class Activation)"]
    B --> D["Pseudo-Segment Labels"]
    E["Sliding Window Features"] --> F["Online Student<br/>Transformer Encoder"]
    F --> G["CASS Predictor<br/>Frame-Level Class Scores"]
    F --> H["Action Anticipation Head<br/>Future Window Action Multi-Hot Vector"]
    F --> I["Anchor Decoder<br/>Actionness-Calibrated Cross-Attention"]
    I --> J["Instance Prediction<br/>Classification + Regression"]
    G --> K["Anchor-Level Point Prediction<br/>Classification + Regression"]
    C -->|"L_cass Alignment"| G
    C -->|"L_ant Binarized Supervision"| H
    D -->|"L_ins Pseudo-Segment Supervision"| J
    L["Raw Point Annotations"] -->|"L_pnt Auxiliary Supervision"| K
    J --> M["ONMS Online Duplication Removal"]
    M --> N["Output Action Proposals"]

Key Designs

1. Class Activation Subsequence (CASS) Distillation and Actionness-Calibrated Attention Decoding: Compressing the Teacher's Full-Video Frame-Level Knowledge into Online Windows and Guiding Anchor Decoding with Actionness Prior

Directly training an online model to perform boundary regression is difficult because point annotations lack boundary information. OnPoint addresses this by having the teacher model first generate the Class Activation Sequence CAS \(\in \mathbb{R}^{T \times (C+1)}\) for the entire video, where each frame has a \(C+1\)-dimensional class score vector (including the background channel). This is a dense, temporally structured representation rich in context.

The student's CASS predictor is a two-layer MLP (with ReLU activation) that maps window features \(F_t\) to class scores for each frame in the window, \(\hat{A}_t \in \mathbb{R}^{W \times (C+1)}\). It is aligned with the corresponding segment of the teacher's CAS using an L2 loss: \(\mathcal{L}_{\text{cass}} = \frac{1}{W}\sum_{i=1}^{W}\|\hat{A}_t[i] - A_t^{\text{teacher}}[i]\|_2^2\). This signal enables the online backbone to learn temporally precise, class-discriminative frame-level representations, capturing subtle action transitions even without access to future frames.

Ingeniously, the outputs of CASS are also used to calibrate the cross-attention of the anchor decoder. Specifically, an actionness score \(r = 1 - \sigma(A_t^{(C+1)})\) is computed by taking the complement of the background channel score. A transformation \(\bar{r} = r + \log(r)\) maps this score to symmetric positive/negative biases (positive bias for high-actionness frames, negative bias for low-actionness frames), which is directly added to the logit of the Transformer decoder's cross-attention: \(\text{Softmax}(\frac{\mathcal{Q}\mathcal{K}^\top}{\sqrt{D}} + \bar{r}^\top)\mathcal{V}\). This naturally encourages anchor queries to aggregate features from action frames while suppressing background frames, without requiring explicit positional encodings or extra gating mechanisms. In comparative experiments, this composite transformation \(\bar{r}=r+\log(r)\) outperformed variants (such as suppression-only, enhancement-only, or direct multiplication with the attention map; avg mAP 61.9% vs. 61.5%/61.3%/58.3%/59.8%) because it achieves bidirectional biasing.

2. Window-Level Action Anticipation Distillation: Compensating for the Lack of "Look-Ahead" Capability in Online Inference Using Future Window Action Presence Information

Since the online model cannot observe frames beyond time \(t\) during inference, it lacks foresight regarding upcoming or ongoing actions. OnPoint compensates for this with a window-level action anticipation head: window features \(F_t\) are dimensionally reduced and passed through a two-layer FC network (ReLU + Sigmoid) to output a multi-hot vector \(\hat{y} \in [0,1]^{C+1}\) representing whether each action class will appear in the next \(W'\) frames.

The supervision signal comes from the future segment of the teacher's CAS. The CAS of the future \(W'\) frames is binarized using a threshold of 0.5 to form a multi-hot target vector \(y\), which is trained using binary cross-entropy: \(\mathcal{L}_{\text{ant}} = -\sum_{c=1}^{C}[y_c \log(\hat{y}_c) + (1-y_c)\log(1-\hat{y}_c)]\). The key design here is the choice of \(W'\)—too small, and the anticipation window heavily overlaps with the current action, yielding limited info; too large, and distant events dilute the supervision signal. On THUMOS, \(W'=16\) achieves the best performance (61.9% avg mAP), whereas alternative strategies (e.g., multi-scale windows, adaptive windows) fail to outperform this simple fixed-window scheme. This design allows the online model to "know" in advance what actions might occur next, enabling a more timely response at boundaries.

3. Anchor-Level Auxiliary Point Supervision: Using Raw Point Annotations as Teacher-Independent "Anchors" to Stabilize Training and Suppress Noise Propagation

An inherent risk of distillation frameworks is that the teacher's pseudo-labels are not always perfect. If the teacher produces incorrect pseudo-segments or noisy CAS for a video, these errors propagate to the student via \(\mathcal{L}_{\text{ins}}\) and \(\mathcal{L}_{\text{cass}}\). OnPoint tackles this by introducing an anchor-level point prediction module that directly injects raw point annotations into the student's training as a weak but fully reliable auxiliary supervision signal.

This module contains a classification head and a regression head (each a two-layer MLP) connected to the anchor features of the anchor decoder: (1) the classification head determines whether an anchor contains an annotated point and predicts its class using cross-entropy; (2) the regression head estimates the normalized distance from the anchor center to the annotated point using an L1 loss: \(\mathcal{L}_{\text{pr}} = \frac{1}{N_p}\sum_{j=1}^{N_p}\left|\hat{d}_j - \frac{|c_a - p_j|}{l_a}\right|\), where \(c_a\) is the anchor center, \(p_j\) is the annotated point position, and \(l_a\) is the anchor length. This utilizes the common observation [ma2020sf] that human-annotated points tend to cluster near the midpoints of actions, following a Gaussian-like distribution, thereby encouraging the model to favor anchors near the midpoints and suppress outliers.

The value of this design is validated in noise robustness experiments. When synthetic uniform noise is injected into the teacher's CAS (simulating a drop in teacher quality), the performance of the student trained with auxiliary point supervision degrades much more gracefully than the student trained without it. Removing the point prediction module entirely results in a 3.5% drop in avg mAP on THUMOS (61.9% \(\rightarrow\) 58.4%), while removing only the classification or regression head drops performance to 58.8% and 59.5% respectively.

Loss & Training

The total loss is a weighted sum of four components: \(\mathcal{L}_{\text{total}} = \alpha\mathcal{L}_{\text{ins}} + \beta\mathcal{L}_{\text{cass}} + \gamma\mathcal{L}_{\text{ant}} + \delta\mathcal{L}_{\text{pnt}}\).

Here, \(\mathcal{L}_{\text{ins}}\) is the anchor-level instance prediction loss (classification cross-entropy + regression L1, supervised by teacher pseudo-segments), \(\mathcal{L}_{\text{cass}}\) is the CASS alignment L2 loss, \(\mathcal{L}_{\text{ant}}\) is the anticipation head BCE loss, and \(\mathcal{L}_{\text{pnt}}\) is the point prediction loss (classification CE + regression L1).

Experiments show that setting \(\alpha=\beta=\delta=1\) is the most robust default configuration. Only \(\gamma\) needs to be tuned slightly (\(\gamma=1.0\) on THUMOS, \(\gamma=0.8\) on EGTEA/HOI4D-O), which significantly reduces the hyperparameter tuning overhead. The optimizer used is Adam with \(lr=1e-4\) and \(weight\ decay=1e-4\).

During inference, only the predictions from the instance prediction head are used, processed by ONMS to remove duplicates and filter out proposals whose predicted end times exceed the current time step (preventing the suppression of potentially more accurate future predictions). The CASS predictor and anticipation head do not participate in inference and only serve as representation learning signals during training.

Key Experimental Results

Main Results

OnPoint is evaluated on five datasets: THUMOS'14 (sports actions), EGTEA (first-person kitchen), HOI4D-O (first-person office), FineAction (dense, fine-grained actions), and EPIC-Kitchens-100 (large-scale first-person kitchen). All methods are trained with point annotations only.

THUMOS'14 Main Results (mAP@tIoU):

Method 0.3 0.4 0.5 AVG[0.1:0.5] AVG[0.1:0.7]
Distillation-Free Baseline* 31.8 18.4 10.3 33.3 24.8
HR-Pro + OAT-ONMS* 58.7 49.3 40.1 55.7 46.3
HR-Pro + HAT* 48.9 39.3 28.2 46.0 36.2
HR-Pro + MATR-ONMS* 56.5 48.2 36.7 54.2 44.5
OnPoint (Ours) 63.9 56.3 45.2 61.9 51.1
Reference: Fully Supervised Online MATR-ONMS 70.3 62.7 52.1 - 49.5
Reference: Offline Point-Supervised HR-Pro 74.3 64.3 52.2 71.6 60.3

OnPoint consistently outperforms the distillation-free baseline across all tIoU thresholds. The AVG[0.1:0.7] is 4.8 percentage points higher than the strongest baseline, HR-Pro+OAT-ONMS, even surpassing some early fully-supervised online methods (e.g., CAG-QIL, 2PESNet, SimOn).

EGTEA and HOI4D-O Results (avg mAP@[0.1:0.5]):

Method EGTEA HOI4D-O
Distillation-Free Baseline* 14.3 20.6
TSASPC + OAT-ONMS* 19.7 42.3
TSASPC + HAT* 16.6 42.0
TSASPC + MATR* 14.5 41.6
OnPoint (Ours) 23.1 44.6
Reference: Fully Supervised Online OAT-ONMS 23.7 48.8

On EGTEA, the performance increases by 3.4 percentage points, and on HOI4D-O, by 2.3 percentage points. On EPIC-Kitchens-100, it improves from 8.5% to 10.5%, and on FineAction, from 5.3% to 7.4%.

Ablation Study

Ablation of Multi-Level Distillation Components (THUMOS'14):

Configuration AVG mAP@[0.1:0.5] Description
OnPoint Full Model 61.9 All components included
w/o Window Anticipation Distillation (WAD) 60.0 Remove future window anticipation, drop 1.9
w/o Actionness-Calibrated Attention (ASAC) 59.9 Remove attention bias, drop 2.0
w/o CASS Distillation + ASAC 58.5 Remove both frame-level supervision and attention calibration, drop 3.4
w/o ASAC + WAD 58.0 Remove both distillation branches, drop 3.9
w/o All Three 57.5 Only instance-level distillation and point supervision remaining, drop 4.4

Ablation of Anchor-Level Point Supervision (THUMOS'14):

Configuration AVG mAP@[0.1:0.5] Description
Full Model 61.9 Includes point classification + point regression
w/o Point Classification Head 58.8 Remove point classification, drop 3.1
w/o Point Regression Head 59.5 Remove point regression, drop 2.4
w/o Both 58.4 Pure distillation without point supervision, drop 3.5
Point Supervision Only (Distillation-Free) 33.3 No teacher, trained purely on point annotations

Key Findings

  • CASS+ASAC Combination Contributes the Most: Removing both CASS distillation and actionness-calibrated attention leads to a 3.4 percentage point drop, as ASAC directly relies on the outputs of CASS as its source of actionness signal, making them deeply coupled.
  • The Core Value of Point Supervision Lies in Robustness, Not Absolute Accuracy: Removing point supervision alone only drops performance by 3.5 percentage points (compared to ~2 percentage points for WAD or ASAC). However, in teacher noise experiments, the model with point supervision shows a much more gradual performance decline under high noise levels—indicating that point supervision acts as a "stabilizer" rather than a performance booster.
  • ONMS vs. OSN Exhibits an Accuracy-Latency Trade-off: ONMS provides superior localization accuracy (61.9% vs. 51.6% avg mAP), whereas OSN offers lower detection latency (AEDT -1.53 vs. -0.17 seconds). The choice depends on whether the application prioritizes precision or real-time responsiveness.
  • The Framework Generalizes Well to Different Offline Teachers: Using HR-Pro, SMBD, LACP, and TSASPC as teachers, OnPoint components consistently yield significant improvements over their respective baseline distillations (ranging from 2.6 to 6.5 percentage points), demonstrating that the framework is teacher-agnostic.
  • Inference Efficiency is Acceptable: The student model has 93M parameters, 2.88 GFLOPs, and runs at 312 FPS (on an RTX 4090), which is comparable to OAT (92M/2.75 GFLOPs/355 FPS) and significantly better than HAT (248M/7.09 GFLOPs/161 FPS) and MATR (191M/7.49 GFLOPs/206 FPS).

Highlights & Insights

  • The "Two-Birds-with-One-Stone" Design of CASS: The same CASS prediction serves both as a distillation target (aligning with the teacher's CAS) and as an actionness signal source driving attention calibration, without requiring extra modules. This idea of "reusing an intermediate representation for two different tasks" is simple yet highly efficient.
  • Actionness Transformation Formula \(\bar{r}=r+\log(r)\): Instead of using simple linear scaling or threshold truncation, this logarithmic function elegantly models an asymmetric bias where "low-value regions decay rapidly, while high-value regions grow gradually." This fits the intuition that "background should be strongly suppressed, while actions should be moderately enhanced." Ablation studies showing that suppress-only (\(\log r\)) and enhance-only (\(r\)) perform worse than the combined transform validate the importance of both ends.
  • Window-Level Anticipation Distilled "What Actions Are in the Future" Rather Than "Where the Boundaries Are": The anticipation head only predicts the presence of action classes in future windows (as a multi-hot vector) instead of precise boundaries. This is a pragmatic choice, as predicting presence is much easier than predicting boundaries, and this coarse-grained information is sufficient for the model to "prepare" its detection of relevant classes ahead of time. In the anticipation strategy ablation, a simple fixed window performed best, showing that "stable and predictable supervision" is more valuable than "clever adaptive mechanisms" in distillation scenarios.
  • The Metaphor of Point Supervision as an "Anchor": Although raw point annotations contain limited information, they are human-annotated, independent of the teacher, and free of noise propagation. In a distillation framework, they act like an "unbiased weak reference" that prevents the student from being led astray by the teacher's systematic biases. This concept can be transferred to any task that utilizes pseudo-label distillation by keeping a small fraction of raw weak annotations as anchor signals.

Limitations & Future Work

  • No Open-Source Code: Due to NDA constraints, the authors could not release the full source code. While they provide detailed architecture descriptions and pseudocode, reproducing the model remains challenging.
  • Heavy Reliance on a Strong Offline Teacher: The performance ceiling of OnPoint is determined by the offline teacher. If the teacher performs poorly on certain action categories or scenarios (e.g., HR-Pro achieves only 12.5% mAP on the dense-action EGTEA dataset), the student's performance will also be limited. A more systematic analysis of sensitivity to teacher quality is required.
  • Centrality Assumption of Point Annotations: The anchor-level point regression head relies on the assumption that annotated points lie near the centers of actions. If annotators habitually mark action starts or ends, the effectiveness of this module may decrease. The authors only validated this assumption on THUMOS, and its generalizability to other datasets' annotation distributions remains unclear.
  • Anticipation Window Size Needs Tuning for Each Dataset: Although the authors claim only \(\gamma\) needs tuning, the optimal \(W'\) on THUMOS is 16. For datasets with vastly different average action lengths, this parameter may require a grid search, indicating a lack of adaptive mechanisms.
  • Lack of Multi-Layer Actionness Calibration Analysis: The paper only visualizes the attention maps of the last layer, but the \(\bar{r}\) bias is added to the cross-attention of every decoder layer. A layer-by-layer ablation to check if the calibration and dependency on this bias are consistent across deep and shallow layers is missing.
  • vs. PS-TAL (HR-Pro, TSASPC, LACP): These methods perform TAL under the offline setting using point annotations, relying on full-video context for pseudo-label generation and boundary refinement. OnPoint adopts them as teachers to distill offline capabilities into an online student, essentially "borrowing PS-TAL's full-video reasoning capabilities without running them online."
  • vs. OnTAL (OAT, HAT, MATR): These methods serve as the foundation for OnPoint's student. The anchor decoder from OAT and ONMS from MATR are reused in OnPoint. The distinction is that while these methods require full-segment annotations for training, OnPoint's student obtains equivalent supervision from point annotations indirectly via distillation.
  • vs. General Offline-to-Online Distillation (DSTA [patel2026distilling], Video Instance Segmentation [kim2024offline]): These works also carry out offline-to-online knowledge transfer, but they distill spatial attention or RoI features, and the teacher and student share similar architectures. The uniqueness of OnPoint lies in the completely different architectures of the teacher (a full-video TAL model) and student (a sliding-window detector), and that it distills "temporal knowledge" (pseudo-segments, CAS, anticipation) rather than "representation alignment," providing more referential value for TAL.
  • vs. Multimodal Large Language Model Annotation (Gemini 2.5 Flash): Auxiliary experiments show that full-segment pseudo-labels generated by Gemini on THUMOS yield only 29.2% mAP, and point labels yield only 30.9% mAP—both far below task-specific models (90.3%/81.6%). This indicates that MLLMs are still unreliable for precise temporal localization, and training specialized, lightweight models remains the superior choice for specific tasks.

Rating

  • Novelty: ⭐⭐⭐⭐ Establishes the POTAL task and provides a systematic solution for the first time. Offline-to-online distillation is a new paradigm in TAL. The design combination of CASS+Actionness calibration and window anticipation distillation shows originality; however, the individual components (distillation, CAS, anticipation) are not entirely new concepts in their respective fields.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive and in-depth experimental design covering five datasets, ablations of the three distillation components, attention calibration variants, point supervision, teacher noise robustness analysis, comparisons of loss functions (6 CASS losses), hyperparameter sensitivity, inference efficiency, online post-processing, and anticipation strategies.
  • Writing Quality: ⭐⭐⭐⭐ Clearly defines the task and illustrates the methodology using detailed block diagrams and equations. However, some key details (such as the offline teacher's post-processing pipeline and pseudo-segment quality metrics) are relegated to the supplementary materials, which may require experienced readers to look up.
  • Value: ⭐⭐⭐⭐ POTAL is a practical task setting (low annotation costs + streaming deployment). Although OnPoint does not outperform fully-supervised methods, it finds a working balance between labeling efficiency and online inference, establishing a reliable baseline and framework for future studies.