LogFA: Efficient Feature-Space Data Augmentation for Egocentric Temporal Action Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ZijiaLewisLu/ECCV2026-LogFA
Area: Segmentation
Keywords: temporal action segmentation, data augmentation, long video understanding, vision-language models, generalized DAG

TL;DR¶

LogFA introduces a highly efficient feature-space data augmentation framework for egocentric temporal action segmentation, combining prompt-enhanced cross-modal textual features for intra-action variations with a Generalized DAG for inter-action sequence diversification without pixel synthesis.

Background & Motivation¶

Egocentric AI assistants aim to perceive environments from a user's first-person perspective, infer ongoing actions and intentions, and provide proactive procedural guidance across daily activities and complex industrial operations. To support these interactive systems, Temporal Action Segmentation (TAS) partitions continuous, untrimmed long videos into sequential, non-overlapping semantic action segments. However, identical procedural tasks in the real world exhibit substantial variations in tool appearances, object textures, spatial layouts, and operational sequences. Because collecting and densely annotating long egocentric videos is labor-intensive and costly, public benchmarks often contain only a handful of demonstrations per task. Consequently, contemporary TAS models suffer severe performance degradation when deployed to unseen environments or alternative workflows.

While data augmentation represents an intuitive solution to this scarcity, current paradigms present an impractical trade-off. Conventional image transformations (such as random flipping, affine cropping, and color jittering) provide only superficial visual perturbations without introducing meaningful semantic variations—such as substituting utensils or altering background contexts—and excessive distortions risk destroying essential action cues. Conversely, recent generative approaches that employ video or image diffusion models can synthesize diverse object- and scene-level modifications, but they incur prohibitive computational overhead. For instance, editing a single 400-frame video segment with state-of-the-art diffusion editors demands over 13.3 GPU hours, making training-time scaling impractical. Furthermore, standard TAS architectures do not intake raw RGB pixels during sequence modeling; they operate directly on pre-extracted visual embeddings from frozen encoders, rendering pixel-level rendering and subsequent re-encoding deeply redundant.

The fundamental insight of this work is to eliminate pixel-level synthesis entirely and perform semantic data synthesis directly within the visual feature space. Core idea: develop LogFA, a unified local-global feature-space augmentation framework that leverages contrastive vision-language models equipped with Prompt-based Feature Enhancement (PFE) to approximate intra-action variations, coupled with a Generalized Directed Acyclic Graph (GDAG) to sample topologically valid, diversified task action sequences.

Method¶

Overall Architecture¶

LogFA decouples data augmentation into two complementary dimensions: local augmentation (intra-action variations in objects, tools, and visual appearance) and global augmentation (inter-action variations in procedural workflow and execution ordering). The entire pipeline operates in two distinct stages: offline feature augmentation and TAS model training. In the first stage, given a small collection of annotated video clips, multimodal and textual LLMs produce granular action captions and semantic variants. These textual descriptions are projected through a frozen contrastive VLM text encoder and refined by clip-specific learnable prompts to approximate target visual features. In parallel, task-level execution sequences are structured into a GDAG using LLM reasoning. In the training stage, the TAS backbone samples augmented features via stochastic interpolation and dynamic sequence reordering, exposing the downstream model to rich procedural distributions at minimal computational cost.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Few Annotated Video Clips v"] --> B["Stage 1: LLM-Driven Action Modification<br/>Generate Base Caption t0 & M Variants tm"]
    B --> C["Stage 2: Prompt-based Feature Enhancement PFE<br/>Optimize Learnable Token Sequence Ev"]
    C --> D["Local Augmented Features fm_hat<br/>Align with Visual Representation Space"]
    A --> E["Stage 3: Generalized DAG Construction<br/>Encode Dependencies, Optionality & Routines"]
    E --> F["GDAG to DAG Conversion & Topological Sort<br/>Sample Diverse Valid Global Action Sequences"]
    D --> G["Stage 4: Stochastic Feature Interpolation & Training<br/>Ratio r1 Local Interpolation + Ratio r2 Global Reordering"]
    F --> G
    G --> H["Downstream TAS Backbone (FACT / MSTCN++)"]

Key Designs¶

1. Local Feature Augmentation: Bridging the Multimodal Gap via Learnable Prompt Enhancement Rather than synthesizing pixels using diffusion architectures, local augmentation generates feature representations directly from textual descriptions of action variants. In contrastive vision-language models such as SigLIP2, visual and textual feature spaces are jointly aligned. LogFA extracts an action-centric caption $t_0$ for clip $v$ using a multimodal LLM (Qwen2.5-VL), and prompts a text LLM (Qwen3) to synthesize $M$ plausible action modifications $t_1, \dots, t_M$ (e.g., swapping a plastic case for a ceramic bowl, or using a spoon).

However, raw textual features $f_m = \Omega(E_m)$ exhibit a non-negligible modality gap when compared to true visual features $g = \Psi(v)$ (with an empirical cosine similarity of only 0.18), because concise text captions omit background ambiance, viewpoint idiosyncrasies, and scene clutter. To bridge this gap, LogFA introduces Prompt-based Feature Enhancement (PFE). For each clip $v$, a set of learnable token embeddings $\widehat{E}_v$ is appended to the tokenized caption embeddings prior to encoding: $$\tau(E_m) = \text{concat}(E_m, \widehat{E}_v), \quad \widehat{f}_m = \Omega(\tau(E_m))$$ By freezing the text encoder $\Omega$ and optimizing only $\widehat{E}_v$, the learnable tokens inject the missing visual context into the modified textual representations, raising the cross-modal cosine similarity to 0.72.

2. Preservation and Alignment Dual-Objective Optimization To prevent the learnable prompt $\widehat{E}_v$ from overfitting solely to the visual feature $g$ and completely ignoring the modified caption tokens $E_m$, LogFA formulates a balanced optimization objective. The alignment loss pulls the enhanced base caption feature $\widehat{f}_0$ toward the ground-truth visual feature: $$\mathcal{L}_{\text{align}} = -\cos(\widehat{f}_0, g) = -\frac{\widehat{f}_0 \cdot g}{\|\widehat{f}_0\|\,\|g\|}$$ Simultaneously, a preservation loss $\mathcal{L}_{\text{preserve}}$ enforces that each modified feature $\widehat{f}_m$ maintains maximum mutual information with its unenhanced counterpart $f_m$ relative to other variants $f_n$, anchored by a soft regularizer against drifting too far from $g$: $$\mathcal{L}_{\text{preserve}} = \frac{1}{M}\sum_{m=1}^{M} \left[ -\log \frac{\exp(\cos(\widehat{f}_m, f_m))}{\sum_{n=1}^{M}\exp(\cos(\widehat{f}_m, f_n))} + w (1 - \cos(\widehat{f}_m, g)) \right]$$ This dual objective guarantees that $\widehat{f}_m$ accurately reflects the semantic transformation specified in $t_m$ while faithfully inhabiting the target video's visual manifold.

3. Global Procedure Augmentation: Procedural Diversification with Generalized DAG Standard data augmentation techniques operate frame-by-frame and fail to introduce procedural diversity. While standard Directed Acyclic Graphs (DAGs) can model basic prerequisite constraints, they cannot naturally represent real-world procedural complexities such as optional steps (e.g., adding seasoning), repetitive actions (e.g., repeatedly slicing vegetables), or unbroken multi-step routines.

LogFA proposes the Generalized Directed Acyclic Graph (GDAG). A GDAG introduces: (i) hierarchical nodes that bundle tightly coupled routines into sub-GDAGs executed contiguously, and (ii) an execution count random variable $C \sim \text{Uniform}(c_1, c_2)$ per node, where $C \in \{0, 1\}$ signifies optionality and $C \ge 2$ signifies repetition. Constructed automatically using LLM world knowledge (GPT-5o or Qwen3-30B) prompted with training examples, the GDAG is sampled by instantiating $C$, removing zero-count nodes, duplicating repetitive nodes, and applying topological sorting to obtain a new procedural sequence. Video action segments are subsequently spliced according to this ordering, with boundary features linearly interpolated to ensure smooth temporal transitions.

Loss & Training¶

During downstream TAS training, the system dynamically activates local feature augmentation with probability $r_1 = 0.3$. For an augmented clip, an index $m \in \{1, \dots, M\}$ is sampled, and the augmented feature is convexly interpolated with the original visual feature $g$: $$\widehat{f}_m^{(\alpha)} = (1 - \alpha) g + \alpha \widehat{f}_m, \quad \alpha \sim \text{Uniform}(0, 1)$$ This continuous blending exposes the temporal backbone to gradual semantic shifts (e.g., smooth color transitions). With independent probability $r_2 = 0.3$, the clip sequence is rearranged following an execution trajectory sampled from the GDAG. Evaluated downstream on architectures like FACT and MSTCN++, LogFA operates exclusively during training; test videos are evaluated directly on pristine visual features without any test-time augmentation.

Key Experimental Results¶

Main Results¶

Evaluations were conducted on the EgoPER benchmark (comprising normal and abnormal error executions across five recipes) and the complex multi-task EgoProceL benchmark. In both setups, training is restricted to only 3 videos per task to simulate low-data deployment regimes.

Dataset / Setting	Method Configuration	Edit	F1@10	F1@25	F1@50	Acc (%)
EgoPER (In-Domain)	Base (3 videos, no aug)	77.5	80.9	77.3	64.1	65.3
EgoPER (In-Domain)	Base + Image Transform	79.3	82.2	78.9	66.4	67.7
EgoPER (In-Domain)	Base + Generative Model (Qwen-Edit)	80.5	82.4	79.1	67.0	69.2
EgoPER (In-Domain)	Base + LogFA (Ours)	82.8	86.1	83.0	70.7	74.0
EgoPER (Out-of-Domain)	Base (3 videos, no aug)	60.2	67.8	62.9	47.7	53.8
EgoPER (Out-of-Domain)	Base + Image Transform	62.0	69.1	64.8	50.0	55.5
EgoPER (Out-of-Domain)	Base + Generative Model (Qwen-Edit)	63.2	69.9	65.7	51.3	56.8
EgoPER (Out-of-Domain)	Base + LogFA (Ours)	68.6	76.0	71.9	57.0	66.0
EgoProceL (Multi-Task)	Base (3 videos, no aug)	45.9	46.6	41.1	28.8	44.3
EgoProceL (Multi-Task)	Base + Image Transform	46.7	48.2	44.6	32.4	46.3
EgoProceL (Multi-Task)	Base + LogFA (Ours)	52.0	51.2	46.9	35.0	54.7

(Data sourced from original paper Table 3 and Table 5; metrics include Segmental Edit distance, F1@{10, 25, 50}, and framewise Accuracy)

Ablation Study¶

The ablations dissect the individual impacts of caption modification types, VLM encoder backbones, and the topological flexibility of GDAG.

Experiment Group	Configuration	Edit	F1@10	F1@25	F1@50	Acc (%)
EgoPER Coffee Local Mod Types	Base (no aug)	47.1	52.7	47.5	32.2	36.7
	+ Type 1 (Paraphrasing)	51.9	55.8	50.3	34.1	43.7
	+ Type 2 (Background & Table Context)	53.7	56.6	51.9	35.8	44.7
	+ Type 3 (Object Texture & Color)	53.1	55.4	51.5	36.8	45.6
	+ Type 4 (Tool & Action Execution Variant)	53.2	55.5	51.3	37.0	45.4
	+ All (Types 1-4 combined, SigLIP2)	55.8	57.6	53.5	38.4	45.0
	+ All (EgoVLP video backbone)	53.5	56.9	52.5	36.7	44.9
EgoProceL Graph Modeling	Base (3-video baseline)	45.9	46.6	41.1	28.8	44.3
	Base + Standard DAG (Dependencies only)	46.2	46.1	41.1	28.8	46.1
	Base + GDAG (Repetitions & Sub-routines)	50.9	50.4	45.4	32.2	53.0

(Data sourced from original paper Table 1 and Table 2)

Key Findings¶

Substantial Out-of-Domain Generalization: On out-of-domain error videos in EgoPER, LogFA enhances F1@50 by +9.3% over the unaugmented baseline (57.0% vs. 47.7%), significantly outperforming conventional image transformations (+2.3%) and pixel-level diffusion editing (+3.6%).
Order-of-Magnitude Efficiency Gain: While generative editing requires 13.3 GPU hours per video augmentation (demanding ~3,480 GPU days for EgoProceL), LogFA's local PFE optimization completes in approximately 6 seconds per augmentation on a single GPU. The entire training pipeline achieves equivalent or superior diversity with over $10\times$ less computational wall-clock time.
Superiority of GDAG over Vanilla DAG: Modeling flexible task structures via GDAG provides a notable +3.4% F1@50 gain and +8.7% Acc improvement on EgoProceL, whereas a standard DAG exhibits stagnant performance (28.8% vs. 28.8% F1@50) due to its inability to capture optional actions and execution repetitions.

Highlights & Insights¶

Feature-Space Bypass of Generative Bottlenecks: Rather than using computationally cumbersome diffusion pipelines to synthesize pixels only to discard them after feature extraction, LogFA demonstrates that text-guided feature perturbation in aligned VLM latent spaces is sufficient to achieve high-diversity data synthesis.
Continuous Latent Trajectories via Convex Blending: Linearly interpolating between raw visual features and prompt-enhanced textual approximations exposes temporal action segmentation networks to continuous semantic spectrums without requiring synthetic rendering.
Formalizing Real-World Procedural Flexibility: By introducing hierarchical nodes and stochastic repeat counters into directed acyclic graphs, GDAG bridges the gap between rigid graph priors and messy real-world procedural execution.

Limitations & Future Work¶

Domain Drift in Specialized VLM Embeddings: Off-the-shelf vision-language models like SigLIP2 excel on common kitchen and household objects, but may exhibit degraded text-to-feature alignment when deployed to specialized industrial assembly lines or medical surgical video.
Heuristic Boundary Smoothing: Reassembled action segments currently rely on linear feature interpolation across boundaries; explicit latent flow smoothing or learned temporal transition modeling could further enhance boundary realism.
Third-Person Viewpoint Evaluation: The current evaluation focuses primarily on egocentric manipulation; validating LogFA on exocentric action benchmarks such as 50Salads or Breakfast remains an important future milestone.

vs. Generative Video Editing (Qwen-Edit / DiffAct): Diffusion-based editors yield realistic frames but struggle with temporal coherence and require massive GPU budgets; LogFA operates natively in latent space, achieving comparable or superior downstream accuracy at a fraction of the computation.
vs. Classical Image Transformations & MixUp: Pixel-level affine transformations fail to teach models invariant representations for alternative tools or altered execution routines; blindly applying temporal MixUp corrupts action transitions, whereas LogFA's GDAG guarantees logically compliant sequences.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Pioneering feature-space text-guided augmentation and GDAG topological sequencing for temporal action segmentation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluations across in-domain, out-of-domain, and few-shot adaptation regimes with thorough ablation and token semantics analysis]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, transparent computational complexity analysis, and intuitive figures]
Value: ⭐⭐⭐⭐⭐ [Directly resolves the prohibitive data collection and compute costs hindering edge deployment of egocentric procedural assistants]