Learning Probabilistic Embeddings for Unsupervised Action Segmentation

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/derkbreeze/PEOT
Area: Video Understanding / Unsupervised Temporal Action Segmentation
Keywords: probabilistic embeddings, optimal transport, graph convolutional networks, pseudo-labels, heteroscedastic uncertainty

TL;DR

PEOT predicts Gaussian frame embeddings with a local temporal graph, trains with multiple sampled optimal-transport pseudo-labels and cross-entropy losses, and uses only the means at test time, improving Desktop Assembly MoF from 59.0 to 71.2 and F1 from 63.7 to 75.8 over a code-reproduced ASOT baseline without leading on every metric of every dataset.

Background & Motivation

Unsupervised temporal action segmentation divides an untrimmed video into action segments while discovering which segments belong to the same action, without frame-level action labels during training. Unlike assigning one category to an entire video, this requires handling sequences in which placing a teabag and pouring water can exchange order, and the same pouring action can recur. Earlier methods learned representations through tasks such as relative-time prediction before performing separate clustering and decoding, so representation learning did not necessarily serve the final segmentation objective directly. Methods such as TOT and ASOT alternate between these operations: current frame embeddings and action prototypes determine an optimal-transport assignment, which then supervises the network as pseudo-labels. ASOT uses structural costs to encourage neighboring frames to share an action and allows unequal action proportions, reducing restrictions associated with fixed ordering and equally long actions.

This loop retains a weakness: if current embeddings misassign an action segment, cross-entropy encourages the network to become more confident in that error, and the next OT step can reproduce a similar assignment. A deterministic frame vector offers the optimizer only one representation location and cannot explicitly express that a frame might plausibly lie at nearby alternatives. The authors therefore propose changing the representations entering clustering instead of adding more elaborate OT hierarchies. Uniformly scaled noise is insufficient because action interiors, boundaries, and ambiguous images differ in uncertainty, while a single-frame estimate may lack stabilizing context. The method consequently combines input-dependent Gaussian distributions with a local temporal graph, using neighboring frames to estimate both the representation center and its range of variation.

Probabilistic modeling here is primarily a training strategy, not a requirement to repeatedly sample many segmentations during deployment. During training, different samples produce different visual costs and pseudo-labels, potentially allowing representations to escape a prematurely fixed incorrect partition; testing still produces one deterministic action sequence. The central question is therefore not how to replace classifier confidence with variance, but how to preserve opportunities to revise action assignments early in self-training. Core Idea: predict data-dependent Gaussian frame embeddings from temporal context, expose OT to multiple plausible representations and pseudo-labels during training, and segment with the learned means at inference.

Method

Overall Architecture

The input consists of pre-extracted per-frame video features rather than raw pixels used to train a new video backbone. A single-hidden-layer MLP projects these features into the frame embedding space, after which Temporal Graph Gaussian Encoding predicts a mean and diagonal covariance for each frame. The training branch uses Sampling-Driven OT Assignment to compute frame-to-prototype costs and independently estimate soft pseudo-labels for each Gaussian sample. Multi-Sample Training and Mean Inference averages the corresponding cross-entropy losses to update the encoder and action prototypes; its inference branch bypasses random sampling and applies OT only to the means. The output is an action-cluster assignment for each frame and the resulting temporal segments; matching clusters to true action names is an evaluation operation, not training supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Pre-extracted frame features"] --> MLP["MLP frame projection"]
    MLP --> Gaussian["Temporal Graph<br/>Gaussian Encoding"]
    Gaussian -->|Training: Gaussian samples| OT["Sampling-Driven<br/>OT Assignment"]
    OT -->|Pseudo-labels for each sample| Strategy["Multi-Sample Training<br/>and Mean Inference"]
    Gaussian -->|Inference: means only| Strategy
    Strategy -.->|Training: cross-entropy updates| Gaussian
    Strategy -->|Inference: means through OT| Output["Action clusters and segments"]

The feedback denotes representation-network updates from the training objective; the MLP and action prototypes are also updated, but gradients do not pass through the OT pseudo-label solver. The probability distribution is not a Bayesian posterior defined directly over action classes: it is defined over continuous frame embeddings and subsequently converted into assignments through OT. This distinction explains why the probabilistic embedding approach can also be attached to VASOT rather than being restricted to one new clustering objective.

Key Designs

1. Temporal Graph Gaussian Encoding: estimate representation centers and variation from neighboring frames

Each sampled frame becomes a graph node connected by default to its predecessor, itself, and its successor, so the 3-frame connectivity includes the self-connection rather than 3 neighbors on each side. Edges are weighted by feature similarity; after adding self-connections, symmetric degree normalization lets each node aggregate neighborhood information while retaining its own features. One GCN branch predicts means and another predicts log variances, allowing each frame and embedding dimension to have a different variance. Log-variance parameterization ensures positive variances, and their input-dependent variation represents heteroscedastic uncertainty rather than a fixed noise scale shared across all videos. Graph convolution provides local context for more stable estimates within an action, while input-dependent edge weights avoid treating every neighboring frame equally.

\[ p(\mathbf z\mid\mathbf x_i)=\mathcal N\!\left(\boldsymbol\mu_i,\operatorname{diag}(\boldsymbol\sigma_i^2)\right), \qquad \mathbf z_i=\boldsymbol\mu_i+\boldsymbol\epsilon\odot\boldsymbol\sigma_i, \quad \boldsymbol\epsilon\sim\mathcal N(\mathbf0,\mathbf I). \]

The distribution follows Eq. (5), and the sampling relation follows the reparameterization description on page 6; multiplication is elementwise. A diagonal covariance means that the model does not explicitly estimate full correlations between embedding dimensions, rather than representing an arbitrary joint distribution. Unlike a TCN with fixed convolution weights, the GCN adapts its aggregation to neighboring-frame content; unlike a per-frame MLP, it has temporal evidence for estimating variance. A wider neighborhood is not necessarily better, since aggregation across an action boundary can blur short actions, motivating the paper's connectivity ablation. The extracted Eq. (6) contains an anomaly in its norms, while the surrounding text explicitly describes cosine-similarity edge weights; this explanation follows the prose instead of treating the damaged equation as implementation-ready.

2. Sampling-Driven OT Assignment: re-estimate pseudo-labels for each representation sample

Each Gaussian sample is compared with learnable action prototypes to produce visual costs for assigning frames to actions. The costs also incorporate a temporal prior based on relative frame and prototype positions; this is a soft preference, not a requirement that every video follow an identical action order. OT itself is inherited from ASOT rather than introduced here: the Kantorovich term measures individual assignment costs, while the Gromov-Wasserstein term supplies temporal structure. The latter penalizes combinations that assign nearby frames within a temporal neighborhood to different actions, without applying the same penalty outside that neighborhood. This encourages continuity at the assignment stage, whereas the GCN smooths local information at the representation stage; the two operations should not be conflated into one filtering step.

Every frame must distribute a fixed total mass, but the action marginal is only softly regularized toward a uniform distribution through a KL term, so actions need not receive equal numbers of frames. This matters for actions with unequal durations, repetition, or partial absence; entropy regularization further controls assignment smoothness. The paper solves this nonconvex OT problem with mirror descent and reports \(O(NK)\) complexity per solver iteration, where \(N\) is the number of frames and \(K\) is the number of prototypes. Sampling changes visual evidence and the resulting pseudo-labels rather than merely perturbing a fixed target label. It can therefore change which action cluster explains which frames, with the motivation of reducing mutual reinforcement between wrong assignments and wrong representations, not guaranteeing escape from every local optimum.

3. Multi-Sample Training and Mean Inference: explore assignments during learning and deploy one representation

For each sampled embedding, temperature-scaled dot products with action prototypes yield action probabilities, and that sample's soft OT assignment becomes the cross-entropy target. Sampling a video \(M\) times produces \(M\) embedding sets, \(M\) pseudo-label sets, and \(M\) losses whose average approximates the expected loss under the distribution. This is not averaging all sampled embeddings into one vector before solving OT once, nor is it ensemble training across multiple models. Each sample supplies gradients to the mean and variance branches through reparameterization, while pseudo-label estimation and network updating remain alternating operations.

\[ \mathcal L_{\mathrm{uncer}}\approx -\frac{1}{MN}\sum_{m=1}^{M}\sum_{i=1}^{N}\sum_{j=1}^{K} t_{ij}^{(m)}\log p_{ij}^{(m)}. \]

This is the Monte Carlo training objective in Eq. (11), where \(t_{ij}^{(m)}\) and \(p_{ij}^{(m)}\) denote the pseudo-label and action probability for sample \(m\). The paper describes the alternation as EM-like: estimate targets using the current representation, then optimize the representation, without human action labels. Its stated objective does not add a VAE-style KL loss to a standard Gaussian prior, so Gaussian distributions and reparameterization alone do not make the method a VAE. At inference, the mean branch supplies the final frame embeddings for OT costs, while the variance branch and Monte Carlo samples are not used for the final segmentation. The precise claim is therefore absence of repeated-sampling overhead at inference, not measured identical runtime relative to every MLP baseline.

A Worked Example

Consider the tea-making videos discussed in the paper: placing a teabag and pouring water may change order, and the current model may merge a short action into an adjacent longer one. The MLP first projects the frame features, and the local GCN predicts Gaussian distributions from neighboring sampled frames, potentially allowing greater representation variation around the short action. With the default \(M=3\), the same video produces 3 OT assignments from different samples, each providing its own cross-entropy supervision. When these assignments disagree in ambiguous regions, updating no longer reinforces only the initial partition, and both means and variances can change during training. Testing uses only the learned means to obtain the corresponding segmentation, without voting among the 3 training samples. This is an illustrative walkthrough of the mechanism, not a claim that the authors reported this tea video's exact frame variances or 3 sampled label sequences.

Loss & Training

The implementation uses a single-hidden-layer MLP with ReLU and a one-layer GCN, sampling 256 frames from uniformly spaced intervals in each video. The defaults are \(M=3\), Adam with learning rate \(10^{-3}\) and weight decay \(10^{-4}\), and action prototypes initialized by K-means. The prototype count \(K\) equals the ground-truth number of action classes per activity; absence of frame-level supervision does not mean the action vocabulary size is unknown. All four datasets use existing pre-extracted features, including IDT for Breakfast and 50Salads, with the same feature type used for training and inference. The paper writes training complexity as \(O(MTNK)\) versus \(O(TNK)\) for the baseline; here \(T\) denotes gradient-descent steps rather than the transport matrix used earlier. More detailed timings and hyperparameters are referred to supplementary material, and this note does not infer unprovided measurements from the available main-paper text.

Key Experimental Results

Main Results

Evaluation uses activity-level Hungarian matching: predicted clusters and true actions are aligned jointly across all videos of an activity rather than separately optimized for each video. MoF is the percentage of correct frame predictions; segment-level F1 counts a matched segment as a true positive when correct frames exceed 50% of the ground-truth segment length; mIoU averages action-class IoUs. F1 and mIoU are subsequently averaged across activities, and 50Salads uses the 12-class Eval granularity, preventing direct comparison with other granularities.

The following values are selected from Tables 1 and 2 on page 10 and retain the paper's percentage scale; published and reproduced baselines are different sources and must not be mixed.

Dataset	Method and source	MoF	F1	mIoU
Breakfast	ASOT, Table 1 published	56.1	38.3	18.6
Breakfast	ASOT, Table 2 reproduced	56.4	35.7	17.2
Breakfast	PEOT, Tables 1/2	60.7	40.5	19.0
YTI	ASOT, Table 1 published	52.9	32.1	24.7
YTI	ASOT, Table 2 reproduced	49.0	34.1	23.7
YTI	PEOT, Tables 1/2	55.4	37.4	22.9
50Salads (Eval)	ASOT, Table 1 published	59.3	53.6	30.1
50Salads (Eval)	ASOT, Table 2 reproduced	59.4	56.7	25.6
50Salads (Eval)	PEOT, Tables 1/2	64.9	58.9	30.2
DA	ASOT, Table 1 published	70.4	68.0	45.9
DA	ASOT, Table 2 reproduced	59.0	63.7	40.6
DA	PEOT, Tables 1/2	71.2	75.8	51.7

On DA, MoF increases by 12.2 percentage points over the code reproduction, approximately 20.7% relative; F1 increases by 12.1 percentage points, approximately 19.0% relative. Against the published baseline, MoF on the same dataset rises by only 0.8 percentage points, so the abstract's maximum gain must not be read as the margin over previously published results. PEOT leads in 9 of the 12 dataset-metric combinations in Table 1, but CLOT achieves 63.2/38.8 F1/mIoU on 50Salads, above PEOT's 58.9/30.2. Another counterexample is YTI, where VASOT's mIoU of 25.2 exceeds PEOT's 22.9, showing that frame-accuracy gains do not automatically imply gains in class IoU. Source-consistency note: the prose on page 9 gives a YTI MoF gain of +1.5, but Table 1 values 55.4 and 52.9 imply +2.5; this note preserves the table values and flags the conflict.

Ablation Study

The following Breakfast and 50Salads (Eval) subset comes from Table 3 on page 11, under the paper's training and evaluation setup; probability only predicts distributions with an MLP, while GCN only retains deterministic embeddings.

Config	BF MoF	BF F1	BF mIoU	FS MoF	FS F1	FS mIoU
Neither probability nor GCN	56.4	35.7	17.2	59.4	56.7	25.6
Probability only	59.2	34.9	15.1	60.1	56.3	23.7
GCN only	58.2	40.7	18.5	50.4	48.7	17.2
Probability + GCN	60.7	40.5	19.0	64.9	58.9	30.2

Probability alone increases BF MoF but lowers F1 from 35.7 to 34.9; GCN alone even reduces FS MoF from 59.4 to 50.4. Combining them yields more consistent overall benefits on these datasets, although deterministic GCN still has slightly higher BF F1 at 40.7 versus the full model's 40.5. This supports learning distributions with temporal conditioning rather than a claim that arbitrary noise or graph convolution always helps.

Table 7 on page 11 varies the Monte Carlo sample count on Breakfast with the other settings held fixed; \(M=1\) still samples and is not a deterministic baseline.

Training samples	MoF	F1	mIoU
\(M=1\)	60.0	39.4	18.4
\(M=2\)	60.3	40.1	18.8
\(M=3\)	60.7	40.5	19.0
\(M=5\)	60.6	40.3	18.9

Key Findings

• Benefits saturate after \(M=3\): moving to \(M=5\) does not improve any of the three metrics, supporting a small sampling budget.
• In Table 5 on page 11, BF MoF is 60.7 with a 3-frame graph and 58.9 with a 5-frame graph; the authors suggest oversmoothing as a possible explanation, not a proven exclusive cause.
• In Table 8 on page 12, fixed Gaussian noise, Dropout, and learned uncertainty achieve DA MoF of 61.9, 65.3, and 71.2 respectively, indicating benefits beyond uniform noise regularization.

Highlights & Insights

• Uncertainty participates in generating pseudo-labels rather than merely displaying confidence after training. Changing the source of self-training targets can interrupt reinforcement of incorrect assignments.
• GCN smooths representations while OT constrains assignments, applying temporal inductive biases to different objects. The ablations emphasize their cooperation with probabilistic modeling rather than the presence of a named module alone.
• Training with distributions and testing with means concentrates the exploration budget in learning. This interface could transfer to other pseudo-label-based clustering pipelines, although such gains require dedicated validation.

Limitations & Future Work

• The method uses the true number of action classes per activity and existing pre-extracted features; the experiments do not establish equivalent benefits with unknown class counts or end-to-end learning from raw video.
• Uncertainty is not a reliable framewise error detector: Figure 6 and the discussion on pages 14-15 show that some incorrect frames retain low uncertainty, so low variance does not guarantee correctness.
• Training repeatedly solves OT, and sampling-free inference does not make training free; the main-paper tables lack error bars across repeated runs, so the stability of small differences cannot be established from them alone.
• YTI mIoU and some 50Salads segment metrics remain weaker, indicating that class coverage, short segments, and boundary precision are not all solved simultaneously.
• The authors suggest increasing sampling early in training when uncertainty is higher and making pseudo-label generation differentiable; these are future directions rather than validated existing features.

Related Work & Insights

• ASOT supplies the unbalanced fused OT framework reused here. PEOT mainly changes frame representations and training-time sampling rather than designing a more complicated transport objective.
• VASOT jointly addresses video alignment and action segmentation through matching video pairs from the same activity. Adding probabilistic embeddings in Table 2 raises FS MoF from 53.4 to 62.2 but lowers mIoU from 26.7 to 26.4, so transfer is not uniformly beneficial across metrics either.
• CLOT improves segmentation with frame-segment feedback and multiple OT levels, benefiting short segments. PEOT avoids such hierarchical complexity, but its weaker 50Salads segment metrics illustrate a performance tradeoff accompanying simplicity.
• Probabilistic representation learning has previously used supervised matching for word, image, face, and pose embeddings; this paper brings distribution modeling into unsupervised temporal clustering. An extension is to adapt sample counts to training stage or frame uncertainty while testing whether the learned variances are calibrated.

Rating

• Novelty: 4/5. Applying probabilistic embeddings to the OT self-training loop is a clear contribution, although Gaussian modeling, GCN, and the solver build on established components.
• Experimental Thoroughness: 4/5. Four datasets, two baseline integrations, and structural and sampling ablations provide broad evidence, while statistical stability and unknown-class-count settings remain underexplored.
• Writing Quality: 4/5. Training and inference are clearly distinguished, but some prose values conflict with tables and require careful reading.
• Value: 4/5. The method offers a relatively simple response to premature pseudo-label fixation, with practical use requiring separate checks of frame-level and segment-level gains.