Skip to content

Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition

Conference: ECCV 2026
arXiv: 2606.22416
Paper: https://prajwalgatti.github.io/gen2balance/
Code: None
Area: Video Understanding
Keywords: Long-Tailed Learning, Video Action Recognition, Generative Data Augmentation, Text-to-Video, Two-Stage Training

TL;DR

Gen2Balance utilizes a text-to-video generative model (WAN 2.1) to synthesize tail-class samples for long-tailed video action recognition. It ensures the diversity and semantic correctness of generated videos through a VLM (Gemini 2.5 Pro) driven three-tier prompting pipeline (diversification + Action Profile disambiguation + In-Context Exemplars anchoring). Combined with a two-stage training strategy (learning features on mixed synthetic and real data, and calibrating on purely real data), it surpasses the strongest long-tailed baselines by 7.0% on K100-LT and 5.1% on UCF-LT, while achieving a 31.9% improvement on RareAct rare actions.

Background & Motivation

The long-tailed distribution is a persistent challenge in visual recognition. Even on ImageNet-LT, the state-of-the-art (SOTA) is only 80.6%, whereas the balanced version of ImageNet reaches 91.0% using the same backbone. Video action recognition suffers similarly: UCF-101 performance drops sharply from 92% under a balanced setting to 64% under a long-tailed setting. Existing long-tailed methods are categorized into three classes: re-sampling (over-sampling the tail easily leads to overfitting), re-weighting (amplifying tail gradients often harms head classes), and data augmentation (implicit feature interpolation cannot generate novel information, while explicit web retrieval is limited by the scarcity of online data for tail classes and associated annotation noise). These methods operate entirely in the feature space or logit space, failing to truly inject new appearance and motion diversity for the tail classes.

Key Challenge: Training samples for tail classes are extremely scarce, preventing models from learning sufficient visual and temporal features. However, video understanding inherently requires rich variations in scenes, people, and actions—feature-level manipulations alone cannot compensate for the lack of pixel-level diversity. Meanwhile, text-to-video generative models (such as WAN 2.1 and Sora) can generate high-quality video clips, and successful precedents of using generative data for long-tailed augmentation already exist in the image domain (SYNAuG, Fill-Up, LTGC). However, in the video domain, this path is almost entirely unexplored.

Core Idea: This work designs a carefully structured VLM prompting pipeline to guide a text-to-video model in synthesizing diverse and semantically accurate videos for each tail class, "filling up" the imbalanced training set into a balanced one. It then employs a two-stage training strategy (first learning features on mixed synthetic and real data, and then fine-tuning/calibrating on purely real data) to mitigate synthetic domain shift, thereby significantly improving tail recognition rates without sacrificing performance on head classes.

Method

Overall Architecture

Gen2Balance consists of two sequential components: (1) Synthesizing videos for tail classes to scale up the training set to a target size using a VLM-driven prompting pipeline combined with a text-to-video model; (2) Learning a classifier on the mixed data using a two-stage training strategy. Given the original long-tailed training set \(\mathcal{D}_{train}\), for each class \(c\) with the number of samples \(N_c < B\), \(N'_c = B - N_c\) synthetic videos are generated to obtain the generative set \(\mathcal{D}_{gen}\) and the augmented set \(\mathcal{D}_{aug} = \mathcal{D}_{train} \cup \mathcal{D}_{gen}\). By default, \(B\) is set to the sample count of the largest head class (full balancing), but it can also be set to a smaller value for partial balancing. The classifier \(f_\theta\) utilizes a pre-trained VideoMAE (ViT-B) backbone, with only the last encoder layer and classification head fine-tuned (7.4M/86M parameters).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long-Tailed Training Set<br/>D_train (Many Head, Few Tail)"] --> B["VLM-Driven Prompting Pipeline<br/>Diversity → Action Profile → In-Context Exemplars"]
    B --> C["Text-to-Video Model<br/>WAN 2.1-14B Generation"]
    C --> D["Balanced Augmented Set<br/>D_aug = D_train ∪ D_gen"]
    D --> E["Stage 1: Mixed Data Training<br/>BSCE Loss + Real Frequency Prior"]
    E --> F["Stage 2: Purely Real Data Fine-Tuning<br/>Low LR BSCE + Real Frequency"]
    F --> G["Classifier f_θ<br/>VideoMAE ViT-B"]

Key Designs

1. Three-Tier Prompt Generation Pipeline: From Diversity to Semantic Disambiguation

Directly applying templated prompts (e.g., "A video depicting the action of [class name]") to text-to-video models presents two critical issues: (1) The class names themselves can be semantically ambiguous—for example, "Robot Dancing" in Kinetics refers to a street dance style where a human imitates a robot, but a templated prompt generates an actual mechanical robot dancing; (2) Templated prompts yield repetitive, stereotypical samples, lacking the visual diversity required for robust training.

The Gen2Balance prompting pipeline disambiguates progressively in three tiers. The first tier, Diversification Prompting, employs an MLLM (Gemini 2.5 Pro) to generate variations along nine diversity axes—scenery/environment, camera composition, video quality, actor demographics, relevant props, action intensity, lighting conditions, background density, and social context—ensuring that the generated videos cover rich visual conditions. The second tier, Action Profile Disambiguation, prompts the MLLM to generate an Action Profile \(\mathcal{A}_c\) for each class, which consists of three components: (a) action definition, (b) positive constraints (visual features that should appear), and (c) negative constraints (common misconceptions, e.g., "not a mechanical robot"). This step is equivalent to reverse-engineering the dataset annotation process—encoding the prior knowledge human annotators use to judge whether a video belongs to a class. The MLLM then generates prompts conditioned on \(\mathcal{A}_c\), defined as \(\mathcal{T}_c = \mathcal{M}(\mathcal{A}_c, c)\). However, relying solely on textual descriptions can still lead to incorrect interpretations (e.g., a real robot still appearing in Figure 2 Col. 3). The third tier, In-Context Exemplars Anchoring, feeds 5 real training video clips \(\mathcal{S}_c\) to the MLLM as context examples, prompting it to view real samples before generating the Action Profile, formulated as \(\mathcal{A}_c = \mathcal{M}(\mathcal{S}_c, c)\) and \(\mathcal{T}_c = \mathcal{M}(\mathcal{A}_c, c)\). Key design: the exemplars only influence the Action Profile and are not directly injected into the prompt generation stage—this ensures that the diversity driven by the axes is not constrained by the narrow distribution of a few real samples.

Ablation experiments (Table 6) verify the independent contributions of each tier: naive prompting achieves only a 37.3% average accuracy on 10 tail classes, which rises to 44.9% with diversification prompting, further increases to 47.2% with Action Profiles, and reaches 49.8% with the complete pipeline (including In-Context Exemplars), while the FVD drops consistently from 1277.1 to 836.1.

2. Two-Stage Training: Synthetic Domain Learning + Real Domain Calibration with Real Frequency as Prior

Directly mixing synthetic and real videos to train with standard CE not only fails to improve performance but actually leads to a 4% drop (Table 5, (a) vs (b)), because the model learns shortcuts from synthetic data rather than generalizable features, and the synthetic-to-real domain shift pulls representations away from the real data manifold.

The two-stage paradigm of Gen2Balance solves this issue. Stage 1: Train on \(\mathcal{D}_{aug}\) using Balanced Softmax (BSCE) loss, formulated as \(\mathcal{L}_{BS} = -\log\left(\frac{N_c e^{\eta_c}}{\sum_{j=1}^{C} N_j e^{\eta_j}}\right)\), where \(\eta_c\) represents the logit and \(N_c\) represents the sample count of class \(c\). BSCE adjusts the loss margin according to class frequency, imposing larger gradient penalties on minority classes. Critical detail: the frequency \(N_c\) used is the sample count from the original training set, rather than the post-augmentation total \(N_c + N'_c\). If the latter were used, under a fully balanced dataset where \(N_c + N'_c\) is identical for all classes, BSCE would degenerate into standard CE, causing the re-weighting effect to disappear entirely. Utilizing real frequencies assigns larger loss margins to minority classes, allowing the model to learn features using a large volume of generative data while keeping the decision boundary calibrated with the prior of the real classes. Stage 2: Fine-tune solely on \(\mathcal{D}_{train}\) using the same BSCE loss but with a lower learning rate (\(5 \times 10^{-4}\) vs. \(5 \times 10^{-3}\) in Stage 1) to make minor corrective updates. This eliminates the domain shift introduced by synthetic data without forgetting the learned representations.

The ablation in Table 5 deconstructs each design choice: training with CE on the augmented set drops performance by 4% ((b) vs. (a)); adding BSCE with augmented frequencies raises it to 61.3% ((c)); incorporating Stage 2 fine-tuning raises it to 70.6% ((d)); switching to real frequency as the BSCE prior further elevates it to 70.9% ((g))—with few-shot classes improving from 61.0% to 62.1% while head classes maintain 87.9%. Replacing Stage 2 with extended Stage 1 training (135 epochs without Stage 2; 135 vs. 100+35) yields only 67.9% ((f)), demonstrating that the improvement stems from the two-stage strategy itself rather than additional training volume.

Loss & Training

Both stages employ the Balanced Softmax loss. Stage 1: 100 epochs, learning rate \(5 \times 10^{-3}\), AdamW, weight decay 0.05, batch size 84, cosine decay + 5-epoch linear warm-up, and input resolution of \(224 \times 224\). Stage 2: 35 epochs, learning rate \(5 \times 10^{-4}\), with other parameters kept identical. The real training set sample count \(N_c\) is consistently used as the frequency in BSCE.

Key Experimental Results

Main Results

Method K100-LT Few K100-LT Tail K100-LT Head K100-LT Avg UCF-LT Few UCF-LT Tail UCF-LT Head UCF-LT Avg
CE (Full Dataset) 74.9 81.7 88.4 80.4 92.7 89.4 94.0 92.0
CE 23.4 63.3 94.7 54.8 45.7 79.6 95.7 64.0
BSCE 48.7 68.5 86.7 64.5 74.3 83.0 88.5 79.3
Logit Adj. 55.5 68.4 78.5 65.6 81.2 83.4 88.6 83.1
LMR (Video SOTA) 51.5 68.7 83.1 65.1 79.6 84.6 94.4 83.6
Sariyildiz et al. (Gen) 26.2 58.8 93.2 52.8 73.1 86.4 94.3 80.6
Li et al. (Gen) 23.8 62.6 93.4 54.4 75.2 88.1 95.9 82.5
Gen2Balance 62.2 75.0 88.4 72.6 86.7 90.3 93.4 88.9

K100-LT is a long-tailed version constructed by selecting 100 temporally dependent classes from Kinetics-400 (imbalance ratio 198, with 30% few-shot classes). UCF-LT is a long-tailed version of UCF-101 (imbalance ratio 24, with 55% few-shot classes). Gen2Balance achieves the highest overall accuracy on both benchmarks, with particularly significant gains on few-shot classes (+6.7% on K100-LT and +5.5% on UCF-LT compared to Logit Adj.), while head class accuracy remains comparable to the CE baseline (88.4% vs. 94.7% on K100-LT, which is a much smaller gap compared to Logit Adj.'s 78.5%). Notably, both generative baselines (Sariyildiz et al. and Li et al.), despite using the same set of synthetic data, underperform even non-generative long-tailed baselines, demonstrating that merely having generative data is insufficient and that the training strategy is equally crucial.

RareAct Rare Actions: Appending 22 cleaned rare actions from RareAct (e.g., cut keyboard, drill phone) as few-shot classes to K100-LT, Gen2Balance achieves 59.7% on these 22 classes, while CE delivers only 11.3% and Logit Adj. only 27.8%—proving that generative balancing is highly effective for truly scarce real-world actions.

Ablation Study

Ablation of Training Strategy (Table 5, B=330)

Config Data Loss Frequency Source Epochs Stage 2 Few Tail Head Avg C/A
(a) \(\mathcal{D}_{train}\) CE - 100 No 23.4 63.3 94.7 54.8
(b) \(\mathcal{D}_{aug}\) CE - 100 No 27.3 55.2 91.2 50.8
(c) \(\mathcal{D}_{aug}\) BSCE \(N+N'\) 100 No 41.8 65.7 90.5 61.3
(d) \(\mathcal{D}_{aug}\) BSCE \(N+N'\) 100+35 Yes 61.0 72.4 87.7 70.6
(e) \(\mathcal{D}_{aug}\) BSCE \(N\) 100 No 61.9 68.3 82.7 67.8
(f) \(\mathcal{D}_{aug}\) BSCE \(N\) 135 No 54.6 70.7 89.7 67.9
(g) \(\mathcal{D}_{aug}\) BSCE \(N\) 100+35 Yes 62.1 72.3 87.9 70.9

The key conclusions step forward in a three-fold progression: (1) Adding synthetic data and training with standard CE degrades performance ((b) < (a)), indicating that a class-balanced loss is mandatory; (2) Stage 2 fine-tuning ((d) vs. (c)) delivers across-the-board gains, effectively correcting domain shift; (3) Utilizing real frequencies as the BSCE prior ((g) vs. (d)) further boosts minority classes while maintaining performance on head classes.

Ablation of Generation Pipeline (Table 6, 10 Tail/Few-Shot Classes)

Pipeline Stage FVD↓ ViCLIP↑ 10-Class Avg↑ Overall Avg↑
Naive Prompting 1277.1 0.1635 37.3 68.9
+ Diversification Prompting 886.8 0.1585 44.9 69.5
+ Action Profile 842.8 0.1649 47.2 70.0
+ In-Context Exemplars 836.1 0.1774 49.8 70.9

With each added layer of the pipeline, FVD (Fréchet Video Distance, lower is better) consistently drops, ViCLIP semantic similarity peaks at the final stage, and classification accuracy steadily increases, validating the progressive necessity of the three-tier design.

Key Findings

  • Core contributions stem from the training strategy rather than pure data volume: The two comparison generative methods use the exact same synthetic dataset but perform significantly worse than Gen2Balance (even underperforming non-generative baselines), indicating that the two-stage training paired with real-frequency BSCE is the key differentiator.
  • Partial balancing is highly cost-effective: On K100-LT, \(B=330\) (filling up to only 1/3 of the size of the largest class) achieves 70.9% Avg C/A using 2.5K GPU hours, capturing 79% of the performance gains of full balancing (\(B=990\), 9.2K GPU hours, 72.6%) while incurring only 27% of the computational cost.
  • Spillover benefits of generative filling on small datasets: On UCF-LT, even when over-sampling beyond the size of the largest head class (\(B > 121\)), performance on few-shot classes continues to improve (from 74.4% to 88.8%), indicating that for small-scale benchmarks, generative filling not only rectifies the long tail but also acts as an all-distribution data augmentation.
  • Robust to web popularity re-ranking: Re-ranking K100-LT classes based on their frequencies in WebVid10M (pushing classes that are truly rare in the generative model's training data to be the few-shot classes), Gen2Balance still outperforms Logit Adj. by 19% on these few-shot classes, demonstrating that the method does not rely on the generative model's pre-trained memorization of specific classes.
  • Generative quality validated by humans: A user study (500 tasks, 2500 judgments) shows that the semantic accuracy of generated videos is 87%, whereas the real Kinetics training videos themselves have a semantic accuracy of only 92% (with errors on both sides being false negatives and no false positives)—resulting in a mere 5% annotation noise gap between synthetic and real data.
  • Backbone and methodology are complementary: After switching to a larger V-JEPA 2 (375M) backbone, Gen2Balance remains the strongest method (76.3% vs. BSCE 71.3%), and the improvement margin is consistent with that on VideoMAE (+5.0% vs. +6.4%), proving that the gains from generative data and the proposed training strategy are orthogonal to backbone scaling.

Highlights & Insights

  • The progressive disambiguation logic of the three-tier prompting pipeline is exceptionally clear: Moving from "diversification" to "definition disambiguation" to "exemplar anchoring", each tier resolves the issues left by the previous one. Crucially, a protective boundary is maintained between tiers (exemplars are not directly injected into prompt generation, preventing diversity collapse). This "layered isolation" design philosophy is highly transferable to any data synthesis scenario using LLMs or generative models.
  • "Using real frequency as the BSCE prior" is a subtle implementation choice with massive impact: On a fully balanced augmented set, if the total sample size is used as the frequency, BSCE degenerates to standard CE, and all re-weighting effects vanish. Using real frequencies mathematically encodes the intuition: "data augmentation can be freely added, but class importance should still reflect the real-world distribution." This trick can be widely generalized to any combination of "synthetic data + re-weighted loss".
  • Directly comparing the annotation quality of generated videos and real videos via a user study is a highly rigorous approach: Rather than evaluating "how good the generated videos are" in isolation, the authors tested the real training set using the same standard (Kinetics annotation protocol). They found that real data also carries 8% annotation noise, meaning the 87% accuracy of generated data is only 5 percentage points behind—this benchmarking strategy is exemplary for all papers working on generative augmentation.
  • The scaling analysis of partial vs. full balancing provides practical guidance: Achieving 79% of the gains at only 27% of the cost is immensely valuable for industrial deployment, demonstrating that one does not need a "fully filled" training set to reap most of the benefits.

Limitations & Future Work

  • Computational cost remains high: Fully balancing K100-LT requires 9.2K H100 GPU hours for video generation. Although this is a one-time offline cost (analogous to dataset collection), it remains a significant burden for larger datasets (such as full Kinetics-400). Partial balancing (\(B=330\), 2.5K hours) offers a pragmatic trade-off.
  • Generated video quality is constrained by model capabilities: Failure cases showcased in the paper include objects appearing in the frame without corresponding actions, or actions being performed but missing required props. As video generation models iterate (from WAN 2.1 to more powerful future models), generation quality and downstream recognition performance are expected to improve synchronously.
  • The specific filtering of temporally sensitive classes may limit the generalizability of the conclusions: The 100 classes in K100-LT were intentionally chosen to be temporally dependent (e.g., breakdancing, robot dancing), where the conclusion "video is superior to static images" is strongest (shown in Table 4, where Qwen-Image generation underperforms WAN video generation). For action classes that can be sufficiently distinguished by static features (e.g., "drinking" vs. "smoking", which might rely heavily on object recognition), the marginal utility of generating videos could diminish.
  • Alternative single-modal VLMs are not comprehensively tested: Table 0.B.3 evaluated Qwen3-VL as a replacement for Gemini and showed good consistency, but this was limited to only 10 classes. How prompt differences generated by various VLMs affect large-scale (100+ classes) training remains unclear.
  • The Stage 2 "rehearsal" is essentially mitigating catastrophic forgetting: However, the paper does not directly measure how much forgetting occurred during the two stages or how much was recovered in Stage 2. Quantitative forgetting analysis would have made the necessity of the two-stage design more compelling.
  • Directions for improvement: (1) Incorporating automatic quality filtering within the generation pipeline to discard synthetic videos with large semantic deviations could further improve training stability; (2) Exploring lightweight domain adaptation techniques (e.g., adversarial training or feature alignment) to replace Stage 2 could yield similar performance with fewer fine-tuning steps on real data; (3) Extending this framework to other video understanding tasks such as temporal action localization, video QA, etc.
  • vs. SYNAuG / Fill-Up (Image Long-Tailed Generative Augmentation): Both of these methods use textual inversion or simple prompts for long-tailed augmentation in the image domain, paired with two-stage training. The core differentiators of Gen2Balance are: (a) introducing the additional challenge of temporal consistency in the video domain; (b) the three-tier prompting pipeline is far more scalable than textual inversion (inversion requires per-class training, which directly exceeds H100 VRAM limits on WAN 2.1-14B); (c) substituting training-intensive inversion with MLLM-based "glance" (In-Context Exemplars), representing a clever and practical engineering reduction.
  • vs. LTGC (LLM-Driven Image Long-Tail Generation): LTGC also uses an LLM to generate diverse prompts for tail-class image synthesis but directly applies them to improve VLMs. Built upon this, Gen2Balance adds Action Profile and In-Context Exemplars for two extra tiers of disambiguation, which is particularly vital for the video domain (where semantic ambiguity is much higher than in the image domain—e.g., "robot dancing" in an image might just show a robot, whereas in video, one must evaluate the motion patterns to distinguish). This approach of "determining prompting pipeline depth based on the degree of modality ambiguity" is highly inspiring.
  • vs. LMR / MOVE / MEDC (Video Long-Tailed Recognition): These methods operate entirely in the feature space—LMR performs linear combinations of head-tail features, MOVE executes feature extrapolation and interpolation, and MEDC utilizes multi-expert branches. Gen2Balance represents the first work to execute video long-tailed augmentation in the pixel space, synthesizing new videos directly rather than recombining existing features. These two routes are inherently complementary: feature-space methods have low computational costs but cannot inject truly new information, while pixel-space methods inject novel information at high computational costs. A natural next step is their combination—using generated videos for backbone pre-training, and then applying fine-grained augmentation in the feature space.
  • vs. Li et al. (Video Generation for Few-Shot): Li et al. first pre-train on synthetic videos and then fine-tune on real data. While structurally similar to the two-stage training in Gen2Balance, the key differences are: (a) they rely on simple templated prompts, whereas Gen2Balance implements a three-tier disambiguating prompting pipeline; (b) their Stage 2 employs uncertainty-based label smoothing, while Gen2Balance utilizes BSCE combined with real frequencies—experiments demonstrate that the latter is far superior under long-tailed scenarios (Li et al. achieves only 54.4% vs. Gen2Balance's 72.6% on K100-LT).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Represents the first study to apply generative pixel-space augmentation to video long-tailed recognition. The three-tier prompting pipeline (containing Action Profile and In-Context Exemplars) is entirely novel in the video domain, simultaneously addressing the orthogonal issues of "diversity" and "semantic correctness."
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated on two main benchmarks, RareAct rare action extrapolation, and 6 sets of ablation studies (training strategy, generative pipeline, data source, generation volume scaling, backbone generalization, VLM generalization, web search popularity re-ranking, test-set leakage checking, and a user study). The coverage is comprehensive and provides clear causal conclusions for each component.
  • Writing Quality: ⭐⭐⭐⭐⭐ Figure 2's "Robot Dancing disambiguation process" serves as an outstanding illustrative diagram—using a running example to visually demonstrate the effect of each pipeline tier side-by-side. The methodology section features precise formulations, tables are clearly formatted, and the supplementary material is substantial (including appendices for the user study interface, failure cases, and full prompts).
  • Value: ⭐⭐⭐⭐⭐ Open-sources a 140K synthetic video dataset (covering 223 classes across three benchmarks), providing an immediately usable public resource for subsequent video long-tailed and generative augmentation research. The cost-effectiveness analysis of partial balancing and the "real frequency as BSCE prior" trick have direct guiding significance for industrial applications.