Bayesian Uncertainty Attribution-Guided Fine-Tuning for Open-Set Action Recognition¶

Conference: ECCV2026
Paper: Official page · PDF
Authors: Shehan Senavirathna, Hongji Guo, Qiang Ji
Area: Video Understanding
Keywords: Open-set action recognition, epistemic uncertainty, deep ensembles, uncertainty attribution, knowledge distillation

TL;DR¶

The paper turns ensemble uncertainty from a test-time rejection score into a representation-refinement signal, using three-stage fine-tuning and two-phase distillation to reach 88.50 AUROC with a single TPN student on UCF101/HMDB51, compared with 88.62 for the refined ensemble.

Background & Motivation¶

Open-set action recognition requires a model to recognize actions seen during training while rejecting unseen categories, rather than assigning a known label to every video. The problem starts before threshold selection: a classifier that recognizes golf from grass may confidently call an unfamiliar lawn-mowing video golf. Once these contextual shortcuts are embedded in the representation, changing the rejection rule alone may not repair the underlying evidence.

Existing methods address this from different directions. DEAR uses evidential learning to formulate uncertainty, SOAR combines scene debiasing with reconstruction, and deep ensembles estimate epistemic uncertainty through predictive disagreement. However, a sample-level score only indicates that the model is uncertain; it does not tell the training procedure which input evidence to change. This paper therefore first reduces static-context bias with motion guidance, stabilizes ensemble predictions on known examples, and then converts input gradients of uncertainty into training attention.

Uncertainty attribution does not mean that the most uncertain region should be removed. Occluded hands, subtle motion, and actor-object contact can be both uncertain and important for recognition. Core idea: retain the original signal in high-attribution regions, softly amplify lower-attribution evidence, and distill both classification and rejection behavior into one RGB model.

Method¶

Overall Architecture¶

Training uses known action classes only. The method trains 10 video recognition models with independent sources of variation, then refines the same ensemble through motion-prior alignment, epistemic regularization, and attribution residual attention. Once refinement finishes, the teacher's mean class distribution and epistemic uncertainty are cached to supervise a student with a classification branch and an auxiliary uncertainty head.

Deployment takes an RGB clip and returns known-class probabilities plus a scalar rejection score. It requires no optical flow, Grad-CAM, attribution maps, or execution of the 10 teachers. The paper also evaluates the full ensemble separately to establish whether the training strategy improves over ordinary ensembling under matched conditions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Known-class RGB clips"] --> Motion["Motion-prior alignment<br/>RAFT and FlowI-SAM"]
    Motion --> Regularize["Epistemic regularization<br/>Stabilize known predictions"]
    Regularize --> Attention["Attribution residual attention<br/>Refresh masks each epoch"]
    Attention --> Distill["Two-target staged distillation<br/>Class distribution and uncertainty"]
    Distill --> Output["One RGB student<br/>Classification and rejection"]

Key Designs¶

1. Motion-prior alignment: move classification evidence toward motion without adding a test-time modality

Stage 1 estimates optical flow with RAFT, then obtains moving-object segmentations using FlowI-SAM, which combines flow with SAM. The segmentation mask is normalized between 0 and 1 and resized to the saliency-map resolution, producing a weak motion prior. In parallel, Grad-CAM identifies the regions actually supporting the RGB classifier's class decision. Training aligns these maps through mean squared error while retaining ordinary cross-entropy supervision for the action label.

The two maps serve different purposes: flow indicates where motion occurs, whereas Grad-CAM indicates where the classifier obtains its decision evidence. Their alignment discourages static background shortcuts; it neither trains the classifier to predict flow nor creates a two-stream deployment model. Camera motion and moving backgrounds can corrupt the prior, which is why the guidance remains weak. Different random seeds, classifier-head initializations, minibatch orders, and stochastic augmentations provide diversity among ensemble members for the later disagreement estimate.

2. Epistemic regularization: reduce disagreement on known clips without treating all ambiguity as novelty

Each member predicts probabilities over known classes, and the ensemble averages these probabilities. Total uncertainty is the entropy of the mean prediction; aleatoric uncertainty is the mean of the members' predictive entropies; epistemic uncertainty is their difference. Several models agreeing that a video is ambiguous is different from several confident models disagreeing with one another. The framework primarily uses the latter kind of uncertainty for refinement and rejection.

The central relation explicitly stated in the cached prose is:

\[ U_e = U_t - U_a. \]

Stage 2 computes the current ensemble's average epistemic uncertainty over each minibatch and optimizes it alongside each member's cross-entropy. All training samples belong to known classes, so lower disagreement is desirable there, but the objective does not directly specify the uncertainty of unknown examples. It could also reduce useful ensemble diversity, so improved open-set detection requires experimental support rather than following automatically from lower known-set uncertainty. The reported analysis finds that total, aleatoric, and epistemic uncertainty all decrease on held-out known clips after this stage, rather than disagreement merely being transferred into aleatoric uncertainty.

3. Attribution residual attention: preserve difficult evidence while amplifying more stable regions

The scalar Stage 2 regularizer does not localize uncertainty within a video. At the start of every Stage 3 epoch, the current ensemble computes epistemic uncertainty on training clips and backpropagates it into input space. The gradient magnitude is normalized into an uncertainty-attribution map. High attribution indicates strong local sensitivity of the uncertainty scalar to the input; it does not prove that a region is background or give a calibrated probability that the region is unknown.

UAM-IA converts this map into smooth residual attention: high-attribution regions stay close to their original scale, while lower-attribution regions receive additional amplification. Attention lies in \([1, 1+\alpha]\). This differs from suppressing high attribution, which can erase occluded hands or subtle interactions, and from amplifying high attribution, which can emphasize unstable evidence. The attention map is multiplied elementwise with the video, and each ensemble member is fine-tuned using classification cross-entropy. Stage 3 does not require all earlier stage objectives to remain active simultaneously.

Attribution maps are refreshed once at the beginning of an epoch, detached, and held fixed throughout that epoch. Parameter updates therefore do not differentiate through attribution generation, avoiding repeated higher-order differentiation through that step. Recomputing at the next epoch still allows the masks to adapt to the evolving model. At inference, the model receives unmodified RGB and does not generate attention maps; the intended effect is a change in the learned representation, not additional per-test-example preprocessing.

4. Two-target staged distillation: transfer the rejection scalar as well as class probabilities

After refinement, the ensemble produces two cached training targets: its mean class distribution and its epistemic uncertainty. In the first student phase, Kinetics-pretrained weights initialize the backbone, which learns together with the classification branch using ground-truth cross-entropy and KL distillation from the teacher distribution. Soft labels preserve class relationships, but the mean class distribution does not uniquely determine disagreement among the teachers. Classification distillation alone therefore cannot fully specify the epistemic target.

The second phase freezes the backbone and classification branch and trains only an auxiliary uncertainty head. It receives concatenated student features and class probabilities and predicts the teacher scalar. Supervision combines numerical matching in log(1+uncertainty) space, within-minibatch pairwise margin ranking based on teacher targets, and matching batch means and standard deviations. These terms address complementary failures: correct ordering with a drifting scale, or a correct average with incorrect sample ordering. The output is a learned approximation to teacher uncertainty, not ensemble disagreement recomputed inside a single model and not an exact Bayesian posterior.

A Worked Example¶

Consider the paper's motivating distinction between known golf and unseen lawn mowing. Training uses known-class clips only. Motion guidance first moves golf-classification evidence away from grass and toward motion-relevant regions; regularization stabilizes the members' known-video predictions; attribution attention retains potentially occluded hand evidence while emphasizing other, more stable regions.

For TPN, \(\alpha=1.00\), so training attention ranges from 1 to 2 and never zeroes out an input region. After distillation, a test clip is rejected as unknown if its predicted uncertainty exceeds the 0.047 threshold selected from known validation data; otherwise, the classifier returns a known label. This example explains the information flow only. The paper does not provide stepwise scores for these particular videos, so it does not establish that any specific lawn-mowing clip is successfully rejected.

Loss & Training¶

The evaluated backbones are TPN, TSM, SlowFast, and I3D, with teacher members initialized from Kinetics-400 pretraining. Optimization uses Adam at an initial learning rate of \(10^{-3}\). The three teacher stages run for 30, 10, and 10 epochs, totaling 50 epochs. Each of the two student phases runs for 15 epochs.

For TPN, the motion-alignment weight, epistemic-regularization weight, and attention strength are 0.65, 0.55, and 1.00. The stages are applied sequentially rather than optimizing every objective jointly from the beginning. Consequently, attribution is generated from an ensemble that has already received motion guidance and uncertainty stabilization. Hyperparameters and rejection thresholds are selected using a held-out subset of the UCF101 training partition; unknown sources are reserved for final evaluation.

Several equations in the cache have extraction damage, including attention parentheses, operators in the distillation objectives, and fractions. This note explains the loss components and attention range from the accompanying intact prose without reconstructing damaged formulas. Batch sizes, distillation temperatures and weights, and the precise threshold-selection rule are deferred to supplementary material by the paper and are not guessed here.

Key Experimental Results¶

Main Results¶

The UCF101 split-1 training partition contains 9,537 videos from 101 classes, with a stratified validation subset held out. The official test partition provides known test clips. After removing semantic overlap with the known classes, the unknown sets contain 41 HMDB51 classes with 1,230 videos and 277 MiT-v2 classes with 27,700 videos.

The following results come from Tables 1 and 3 and all use TPN. AUROC summarizes the known/unknown separation curve; FPR@95 is the false-positive rate at a 95% true-positive rate; TPR@10 is the true-positive rate at a 10% false-positive rate. Open maF1 treats unknowns as one additional class and measures open-set macro F1. Accuracy is measured only on known UCF101 clips. Values retain the paper's percentage scale, with lower-is-better columns marked.

Unknown source	Method	AUROC ↑	FPR@95 ↓	Open maF1 ↑	Known accuracy ↑
MiT-v2	SOAR	91.45	30.96	74.48 ± 0.21	92.63
MiT-v2	Student	93.10	24.64	78.04 ± 0.18	95.98
HMDB51	SOAR	86.67	61.02	85.43 ± 0.14	92.63
HMDB51	PSL, rerun under the same protocol	88.04	55.82	Not reported	95.73
HMDB51	Student	88.50	53.85	88.97 ± 0.17	95.98

Against SOAR on HMDB51, the TPN student improves AUROC by 1.83 percentage points and reduces FPR@95 by 7.17 points. Against the protocol-matched PSL rerun, the AUROC gain is only 0.46 points. The latter provides a more restrained comparison with a closer purely visual baseline.

Ablation Study¶

The following Table 4 results use the full TPN ensemble setting, with UCF101 known and HMDB51 unknown; they must not be conflated with the single-student results above. H is the Hellinger distance between known and unknown uncertainty distributions, with larger values indicating greater separation. ECE measures the discrepancy between confidence and empirical accuracy, and NLL is negative log-likelihood; both are evaluated on known data and are better when lower.

Enabled stages	AUROC ↑	FPR@95 ↓	Known accuracy ↑	H ↑	ECE ↓	NLL ↓
None, vanilla ensemble	84.39	62.64	93.24	0.67	0.025	0.19
Stage 1 only	84.78	61.44	94.33	0.69	0.023	0.17
Stages 2+3, without Stage 1	87.76	55.28	95.69	0.74	0.016	0.12
Stages 1+2	86.21	56.97	95.02	0.77	0.018	0.13
Stages 1+2+3	88.62	54.16	96.17	0.80	0.014	0.10

Stage 1 alone adds only 0.39 AUROC points, whereas adding UAM-IA after Stages 1+2 adds 2.41 points. Removing motion guidance still yields 87.76, making it a useful initialization rather than the primary explanation for the improvement. These increments depend on stage ordering and do not constitute a strict decomposition of independent module contributions.

Table 6(a) fixes the Stage 1+2 starting point and tests which attention form matters:

Stage 3 attention form	AUROC ↑	FPR@95 ↓	Known accuracy ↑
No mask	86.21	56.97	95.02
Frame-difference mask	87.18	56.04	95.47
Amplify high attribution	87.05	56.31	95.38
Suppress high attribution	87.42	55.71	95.61
UAM-IA	88.62	54.16	96.17

Key Findings¶

UAM-IA exceeds the frame-difference mask by 1.44 AUROC points and high-attribution suppression by 1.20 points, supporting a benefit beyond ordinary motion emphasis. In the attention-strength sweep, \(\alpha=1.00\) reaches 88.62, while 2.00 falls to 87.26; stronger amplification is not always better.
After Stage 2, mean known-set total uncertainty decreases from 0.391 to 0.357, aleatoric uncertainty from 0.355 to 0.331, and epistemic uncertainty from 0.036 to 0.026. In Table 5(a), epistemic regularization yields 86.21 AUROC, compared with 85.94 for total-uncertainty regularization and 85.38 for aleatoric regularization.
Reported deployment cost decreases from 322.525 ms / 505.42 GFLOPs for the 10-member ensemble to 39.240 ms / 52.542 GFLOPs for the student. These are measurements under the paper's conditions; the cached main text does not supply complete hardware and timing details, so they are not platform-independent latency guarantees.
The student does not lead on every backbone. On HMDB51, TSM achieves 84.24 AUROC versus SOAR's 85.96, SlowFast achieves 88.38 versus 90.72, and I3D achieves 86.31 versus 88.10. Improving a matched vanilla ensemble does not imply that the distilled student universally outperforms single-model baselines.

Highlights & Insights¶

Move from scalar diagnosis to input-level training. Backpropagating uncertainty into an attribution map provides a way to change evidence use instead of merely moving a rejection threshold. Stabilizing the estimate before generating attribution is an important part of the sequence.
Residual preservation suits ambiguous action details. High-attribution regions can remain informative, so retaining their original content while amplifying lower-attribution evidence is a more cautious inductive bias than deleting every apparently unreliable region.
Distillation targets should match deployment decisions. Transferring a rejection score in addition to classification soft labels, with both scale and ordering constraints, can inform other ensemble-based OOD systems. Its fidelity must still be tested on unseen domains.

Limitations & Future Work¶

The authors explicitly identify expensive offline training. Ten members, flow and saliency computation, epoch-wise attribution refreshes, and two subsequent distillation phases move cost away from deployment rather than eliminating overall computation.
Gradient sensitivity is not causal relevance. A low-attribution region may contain a stable but incorrect background shortcut. Residual attention preserves information but cannot guarantee that amplified evidence belongs to the action itself; camera and background motion can also corrupt the Stage 1 prior.
The uncertainty head learns teacher behavior from known examples only. Rejection on unknown domains depends on student generalization and is not guaranteed by known-set regression loss. HMDB51 regressions on some backbones reinforce the need to inspect ranking, tail false positives, and fixed-threshold behavior separately.
Evaluation scale and accessible reproduction details remain limited. Most controlled mechanism studies use TPN/UCF101/HMDB51. Filtered class lists, detailed threshold selection, and failure examples are assigned to supplementary material absent from this cache. Larger studies with Transformer and foundation-model video backbones remain future work identified by the authors.

DEAR / SOAR: DEAR builds an evidential rejection mechanism, and SOAR additionally addresses scene bias. This paper instead uses ensemble disagreement and feeds it back into training inputs, changing both representation learning and deployment rather than only replacing the test-time score.
PSL: PSL strengthens instance- and class-specific information and is rerun here as a purely visual single-label baseline. The protocol-matched AUROC gap is only 0.46 points, motivating a careful comparison between training complexity and practical gain rather than relying only on weaker older baselines.
Gradient-based uncertainty attribution: Prior work already localizes sources of uncertainty. The contribution here is to turn attribution into epoch-wise residual input attention within a staged action-recognition training pipeline.
Open-vocabulary action recognition: ActionCLIP and Open-VCLIP use vision-language priors to recognize categories outside the supervised vocabulary. This work instead returns a known category or an unknown decision; it does not name novel actions or use external textual semantic supervision.

Rating¶

Novelty: 4/5. The ingredients have precedents, but combining epistemic attribution, residual input attention, and two-target distillation provides a clear task-specific contribution.
Experimental Thoroughness: 4/5. Multiple backbones, matched ensembles, a PSL rerun, stage ablations, and attention controls are included; most controlled analysis still concentrates on one backbone and one unknown source.
Writing Quality: 4/5. Stage roles and unknown-data isolation are clearly explained, while reproduction relies on supplementary details. Damaged cached formulas are an extraction issue and do not establish errors in the original manuscript.
Value: 4/5. The method offers a practical route from expensive training-time uncertainty estimates to inexpensive RGB rejection, but offline cost and uneven cross-backbone gains limit its generality.