CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition¶

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Authors: Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoît Savary, Alessandro Lameiras Koerich, Marco Pedersoli, Eric Granger
Code: https://github.com/osamazeeshan/CLIP-AUTT
Area: Human Understanding
Keywords: Action Units, fine-grained emotion recognition, test-time adaptation, video personalization, temporal-window selection

TL;DR¶

CLIP-AUTT first learns expression classification from similarities between Action Unit text and videos, then selects the lowest-prediction-entropy contiguous segment in an unseen subject's video and updates only the AU prompt embeddings, improving BioVid WAR from 78.0 for the non-personalized CLIP-AU to 81.5.

Background & Motivation¶

Subtle expressions cannot be recognized reliably simply by presenting CLIP with broad labels such as happiness or sadness. Pain, stress, or hesitation may appear only as brief eye tightening or changes around the mouth, with informative frames occupying only part of a video. Existing CLIP expression methods often use class templates or descriptions generated by large language models. The former offer coarse semantics; the latter do not necessarily describe local muscle changes merely because they are longer. Full CLIP fine-tuning can exploit emotion labels, but increases training cost without directly resolving these fine-grained cues.

Action Units (AUs) provide an alternative representation: instead of asking directly which emotion is present, first measure correspondence between the video and a set of local facial-action descriptions, then learn how those correspondences combine into emotion categories. Individual differences remain important. Similar brow, eye, or mouth movements need not indicate the same state for different subjects. A generic mapping trained on source subjects does not automatically address changes in a target subject's expression intensity, appearance, and motion patterns.

The paper first builds CLIP-AU while retaining the CLIP backbone, then uses CLIP-AUTT to address individual differences at test time. Rather than requiring emotion labels for each new subject, it adjusts AU representations using relatively reliable contiguous expression evidence within a video. Core Idea: replace coarse class prompts with muscle-level textual semantics, and let the lowest-prediction-entropy temporal window drive per-video AU prompt-embedding updates instead of retraining the entire recognition model.

Method¶

Overall Architecture¶

The input is a video containing a single target expression, and the output is its emotion-class probability distribution. During source training, “AU Semantic and Temporal Alignment” forms CLIP-AU: CLIP's image and text encoders are frozen, while a lightweight AU adapter, temporal module, and classifier learn from source-subject emotion labels. At test time, CLIP-AUTT adds “Low-Entropy Temporal-Window Selection” followed by “AU Prompt Personalization,” without requiring source training videos or target labels.

Personalization here does not maintain a continuously updated model for a person. The paper explicitly states that each target video is adapted independently and adaptation parameters are reset afterward; one video's updates therefore do not carry over to the next. The final reset in the diagram represents this isolation mechanism and distinguishes per-video adaptation from continual learning across videos.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source videos and emotion labels<br/>AU text descriptions"] --> B["AU Semantic and Temporal Alignment"]
    B --> C["Low-Entropy Temporal-Window Selection"]
    T["Video from an unseen subject"] --> C
    C --> D["AU Prompt Personalization"]
    D --> E["Emotion prediction<br/>Then reset adaptation parameters"]

Key Designs¶

1. AU Semantic and Temporal Alignment: classify through combinations of facial actions

The method uses 46 local facial-action descriptions as text prompts. A frozen CLIP text encoder produces their embeddings, and an AU adapter consisting of a two-layer MLP specializes them for emotion recognition. This does not require a language model to generate descriptions for individual videos, nor does it require AU labels in the training set. AUs serve as structured semantic references. Because one emotion can involve multiple actions, recognition cannot be reduced to selecting the single most similar AU.

The video branch extracts frame features with the frozen image encoder, then combines a lightweight one-dimensional temporal module, a gated linear unit (GLU), and average pooling into a video representation. The temporal module captures local changes across contiguous frames, while the GLU applies learned gating rather than treating every frame identically. The method section calls this a one-dimensional convolutional temporal encoder, whereas the implementation section describes a lightweight 1D CNN containing a single linear layer. This note retains the consistently supported sequence of temporal encoding, gating, and pooling without inventing convolution depths or kernel sizes.

The aggregated video representation is compared with all 46 AU embeddings using cosine similarity. The resulting vector records correspondence to the complete set of AUs, and a two-layer emotion-classification MLP maps it to class probabilities. Video-level emotion labels supervise the AU adapter, temporal module, and classifier rather than CLIP's two backbone encoders. This preserves the pretrained cross-modal space while learning which AU combinations distinguish the current task, without turning AU detection into a separate supervised task.

These similarities can help explain a prediction, but they are not calibrated probabilities of muscle activation. The paper provides qualitative comparisons with an external OpenFace detector rather than ground-truth AU labels verifying every coordinate. “AU semantic-similarity representation” is therefore more accurate than claiming reliable measurements of AU intensity.

2. Low-Entropy Temporal-Window Selection: adapt only on relatively reliable contiguous evidence

The beginning and end of a test video may be nearly neutral, with the strongest expression confined to a short interval. Adapting on the entire video could dilute useful evidence with ambiguous frames. CLIP-AUTT slides a contiguous window of length \(L\) over frame features with unit stride; a video of \(T\) frames therefore produces \(T-L+1\) candidate windows. Each window passes through the trained temporal module, is compared with the AU embeddings, and produces emotion probabilities through the fixed classifier.

Selection uses the Shannon entropy of the emotion-class distribution, not the entropy of directly normalized AU similarities. Let \(p_i(c)\) denote window \(i\)'s predicted probability of emotion class \(c\), with \(C\) classes in total. The paper's selection rule can be written clearly as:

\[ h_i=-\sum_{c=1}^{C}p_i(c)\log p_i(c),\qquad i^*=\arg\min_i h_i. \]

A more concentrated distribution has lower entropy; only the lowest-entropy window is retained for subsequent AU prompt updates. Contiguous windows avoid stitching distant high-confidence frames into a segment that never occurred as an actual motion sequence. Temporal encoding also makes candidate comparison sensitive to local dynamics. However, low entropy indicates confidence, not necessarily correctness. Calling the selected window the “most expressive” reflects the authors' interpretation, not an additional manually annotated expression peak.

3. AU Prompt Personalization: move textual semantic references while fixing the visual and classification paths

After selection, the model retains the window's temporal visual representation, recomputes cosine similarities to the AU embeddings, and obtains class probabilities from the fixed classifier. Adaptation reduces the entropy of this prediction distribution so that AU representations provide more decisive emotion evidence for the target video. The updated variables are continuous AU prompt embeddings, not regenerated natural-language descriptions or a classifier trained on target labels. CLIP's encoders, the temporal module, and all other modules remain frozen.

This locates personalization at the visual–textual interface. Visual features and the classifier remain unchanged, while the relationship between the target expression and muscle-level semantic references is adjusted. Source training supplies a generic mapping of AU combinations; test-time adaptation calibrates it only for the current video. The text describes entropy minimization, but the cached adaptation-loss equation has missing extracted symbols. This note therefore does not treat the damaged equation as a reproducible implementation: its clearly stated intent is to minimize the same type of class-prediction entropy used for window selection.

At the end of each video, the updated AU embeddings return to their pre-adaptation state. This limits error accumulation across videos but does not eliminate confirmation bias within a video. If the initial prediction is confident yet wrong, reducing its entropy may reinforce the error. The paper establishes empirical benefits from these unlabeled updates, not a guarantee of correct adaptation.

A Worked Example¶

Consider a pain video from an unseen subject. The system extracts frame features and constructs contiguous candidates using 16-frame windows, the best setting in the BioVid window-length ablation. All candidates share the same temporal module, initial AU embeddings, and classifier. The lowest-entropy comparison thus uses identical model conditions rather than separately adapting each window before comparing them.

Only the lowest-entropy candidate then drives prompt updates. Its visual representation remains fixed; changing the AU embeddings changes the similarity vector and emotion probabilities. The classifier uses the full collection of AU cues without requiring a predetermined rule that a particular AU necessarily implies pain. After predicting the current video, the AU embeddings are reset, so the next video starts from the source model's corresponding state. This example illustrates the algorithm's order without inventing a sample outcome or adaptation-step count.

Loss & Training¶

Source training is supervised by emotion labels. The cached supervised-loss equation also has missing characters, so its complete algebraic form is not asserted here. Test-time adaptation uses only unlabeled entropy minimization on the selected window. “No labels needed at test time” must not be expanded into “the entire method is zero-shot and requires no source training.”

The implementation section reports AdamW, a learning rate of \(1\times10^{-3}\), and weight decay of \(1\times10^{-4}\). CLIP-AU uses a batch size of 8; CLIP-AUTT processes videos individually with a batch size of 1. The cached main paper does not sufficiently specify details such as the number of test-time gradient steps or stopping criterion, so these are not presented as established settings.

Key Experimental Results¶

Main Results¶

BioVid uses 77 source subjects and 10 target subjects; the corresponding splits are 44/10 for StressID and 143/10 for BAH. Every result below is averaged over 10 target subjects per dataset in the original Table 1. WAR is recall weighted by class frequency, which the paper equates with overall classification accuracy. F1 combines precision and recall, but the cached text does not specify its averaging convention, so it should not be labeled macro-F1. Higher is better for both metrics, and the table retains the percentage-scale scores.

Method and setting	BioVid WAR ↑	BioVid F1 ↑	StressID WAR ↑	StressID F1 ↑	BAH WAR ↑	BAH F1 ↑
CLIP-ViT-B/32, full fine-tuning	69.7	66.6	67.0	44.5	60.4	39.8
CLIP-AU, lightweight source training	78.0	74.8	66.5	58.5	68.3	40.3
PromptAlign, TTA	75.3	71.6	74.6	53.2	67.1	39.7
T3AL, TTA adapted for classification	76.1	72.9	75.9	59.4	67.9	40.7
CLIP-AUTT, TTA	81.5	78.0	80.8	77.9	69.8	41.1

Relative to T3AL in this table, CLIP-AUTT improves WAR by 5.4, 4.9, and 1.9 percentage points and F1 by 5.1, 18.5, and 0.4 percentage points, respectively. These are within-dataset score differences, not relative percentage gains. The methods do not share identical source-training architectures: the paper specifies a source-subject-fine-tuned CLIP-ViT-B/32 for TTA baselines, whereas CLIP-AUTT builds on CLIP-AU. Consequently, the main table alone cannot isolate the contribution of an individual adaptation component.

Ablation Study¶

The original Table 5 isolates the two test-time components on BioVid. Every configuration starts from CLIP-AU, and metrics follow the main experiment's definitions.

Config	WAR ↑	F1 ↑	Note
CLIP-AU	78.0	74.8	No window selection or AU prompt updates
Low-Entropy Temporal-Window Selection only	78.7	75.2	Changes only the input segment
AU Prompt Personalization only	80.0	76.4	No low-entropy window selection
Both components, CLIP-AUTT	81.5	78.0	full model

AU prompt updates alone add 2.0 WAR points, while window selection alone adds 0.7 WAR points. Together they improve CLIP-AU by 3.5 WAR points and 3.2 F1 points. This supports the importance of both what is adapted and which video evidence drives adaptation, but does not establish that statistical significance was tested.

Key Findings¶

Longer windows are not always better. In the original Table 4, BioVid WAR is 78.8, 81.5, 79.8, 80.0, and 79.0 for window lengths of 8, 16, 32, 64, and 72 frames. The 16-frame setting is best in this ablation, not necessarily across all video tasks.
AU semantics do more than increase the number of prompts. A control using the same number of generic class prompts, 46, achieves StressID WAR of 61.3 after source training and 44.1 after test-time adaptation, versus 66.5/80.8 for CLIP-AU/CLIP-AUTT. The generic-prompt control drops 17.2 percentage points; this note follows the table rather than the text's rough “nearly 15%” description.
Benefits vary substantially across datasets. Final BAH F1 is only 41.1 and exceeds T3AL by just 0.4 percentage points. The authors associate its difficulty with device, lighting, head-motion, and speaking-related variation; the overall ranking does not establish reliability in every setting.

Highlights & Insights¶

Semantic granularity is separated from prediction granularity: prompts describe local facial actions, while output labels remain emotions. This allows the model to learn combinations of AUs without forcing one prompt to represent one emotion class.
Window selection is not merely inference-time cropping; it determines which visual evidence generates test-time gradients. The component ablation shows why the source of adaptation evidence should be considered alongside the updated parameters.
Freezing a large model does not mean avoiding training. The method concentrates source supervision in lightweight modules and target adaptation in AU embeddings, distinguishing general knowledge, task mapping, and per-video calibration.

Limitations & Future Work¶

A single expression and a contiguous expressive peak are important assumptions. Mixed expressions, missing peaks, or incorrect confidence on neutral segments can make the lowest-entropy window unrepresentative of the target state; the main paper provides insufficient failure-rate analysis.
Without ground-truth AU labels, interpretability rests mainly on textual priors and qualitative agreement with an external detector, not validated muscle-action measurements. Subtle facial cues should not be treated as definitive evidence of a person's internal emotional state.
The main table averages only 10 target subjects per dataset. Although an expanded-target-pool analysis is included, the main results lack variance estimates or confidence intervals. Low BAH F1 indicates a substantial remaining gap in natural settings.
Reproducibility gaps remain: cached equations are damaged, some figures contain inconsistent AU identifiers and action descriptions, and test-time update counts and the F1 averaging convention are unclear. Reproduction should check the original PDF and code rather than constructing a full AU vocabulary from figure labels.

vs EmoCLIP / Exp-CLIP: These methods use expression-related text or descriptions for cross-modal learning. This work instead represents videos through combinations of AU-text similarities, emphasizing local actions rather than only enriching emotion-class descriptions.
vs TPT / PromptAlign: The methods share lightweight test-time prompt adaptation, but CLIP-AUTT handles contiguous videos, selects a low-entropy temporal window, and adjusts AU semantic embeddings. Its source stage requires emotion labels and should not be presented as purely zero-shot prompt learning.
vs T3AL: The original method targets test-time temporal action localization and is adapted here as an expression-classification baseline. These scores describe that modified protocol, not performance on the original localization task.

Rating¶

Novelty: 4/5. Combining AU semantic representations with per-video low-entropy prompt adaptation is well targeted, although the underlying components are established.
Experimental Thoroughness: 3/5. Three datasets, major baselines, and component ablations are covered, but uncertainty reporting and reproduction details are incomplete.
Writing Quality: 3/5. The two-stage logic is clear, although baseline names and numerical descriptions are inconsistent in places, and cached equations should not be used directly.
Value: 4/5. A clear approach to lightweight personalization for fine-grained video expressions, with reliability in real-world settings still limited.