Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition¶

Conference: ECCV 2026
arXiv: 2606.22416
Paper: https://prajwalgatti.github.io/gen2balance/
Code: None
Area: Video Understanding
Keywords: Long-Tailed Learning, Video Action Recognition, Generative Data Augmentation, Text-to-Video, Two-Stage Training

TL;DR¶

Gen2Balance utilizes a text-to-video generative model (WAN 2.1) to synthesize tail-class samples for long-tailed video action recognition. It ensures the diversity and semantic correctness of generated videos through a VLM-driven (Gemini 2.5 Pro) three-tier prompting pipeline (diversity expansion + Action Profile disambiguation + In-Context Exemplars anchoring). Combined with a two-stage training strategy (learning representations on mixed synthetic/real data, and calibrating the classifier using only real data with real-frequency priors), Gen2Balance outperforms the strongest long-tailed baselines by 7.0% and 5.1% on K100-LT and UCF-LT respectively, and achieves a 31.9% improvement on RareAct rare actions.

Background & Motivation¶

The long-tailed distribution is a persistent bottleneck in visual recognition. Even on ImageNet-LT, the SOTA is only 80.6%, whereas the balanced version of ImageNet achieves 91.0% with the same backbone. Video action recognition suffers from the same issue: performance on UCF-101 drops sharply from 92% under a balanced setting to 64% under a long-tailed setting. Existing long-tailed methods fall into three categories—re-sampling (over-sampling the tail classes often leads to overfitting), re-weighting (amplifying tail-class gradients frequently harms head-class performance), and data augmentation (implicit feature interpolation cannot generate new information, while explicit web retrieval is constrained by the scarcity and labeling noise of online tail-class data). All of these operate in the feature or logit space, failing to truly inject novel appearance and motion diversity into tail classes.

Key Challenge: Training samples for tail classes are extremely scarce, preventing models from learning sufficient visual and temporal features. However, video understanding inherently requires rich variations in scenes, actors, and actions—feature-level manipulations alone cannot compensate for the lack of pixel-level diversity. Meanwhile, text-to-video generative models (such as WAN 2.1 and Sora) have become capable of generating high-quality video clips, and the image domain has seen successful precursors using generative data for long-tailed augmentation (SYNAuG, Fill-Up, LTGC). In the video domain, however, this pathway remains largely unexplored.

Core Idea: Use a carefully designed VLM prompting pipeline to guide text-to-video models to synthesize diverse and semantically precise videos for each tail class, "flattening" the imbalanced training set into a balanced one. This is followed by a two-stage training strategy (first learning features on a hybrid of synthetic and real data, and then fine-tuning/calibrating on purely real data) to alleviate the synthetic domain shift, which significantly improves tail recognition accuracy without sacrificing head performance.

Method¶

Overall Architecture¶

Gen2Balance consists of two sequential components: (1) a VLM-driven prompting pipeline coupled with a text-to-video model to generate synthetic videos for tail classes, filling the training set to a target scale; and (2) a two-stage training strategy to learn the classifier on the hybrid data. Given the original long-tailed training set $\mathcal{D}_{train}$, for each class $c$ with sample count $N_c < B$, the framework generates $N'_c = B - N_c$ synthetic videos to construct the generative set $\mathcal{D}_{gen}$, resulting in the augmented set $\mathcal{D}_{aug} = \mathcal{D}_{train} \cup \mathcal{D}_{gen}$. By default, $B$ is set to the sample count of the largest head class (perfect balance), but it can also be configured to a smaller value for partial balancing. The classifier $f_\theta$ employs a pre-trained VideoMAE (ViT-B) backbone, fine-tuning only the last encoder layer and the classification head (7.4M/86M parameters).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Long-Tailed Training Set<br/>D_train (Many Heads, Few Tails)"] --> B["VLM-Driven Prompting Pipeline<br/>Diversity → Action Profile → In-Context Exemplars"]
    B --> C["Text-to-Video Model<br/>WAN 2.1-14B Generation"]
    C --> D["Balanced Augmented Set<br/>D_aug = D_train ∪ D_gen"]
    D --> E["Stage 1: Hybrid Data Training<br/>BSCE Loss + Real Frequency Prior"]
    E --> F["Stage 2: Real-Only Fine-Tuning<br/>Low LR BSCE + Real Frequency"]
    F --> G["Classifier f_θ<br/>VideoMAE ViT-B"]

Key Designs¶

1. Three-Tier Prompt Generation Pipeline: From Diversity to Semantic Disambiguation

Directly applying templated prompts (e.g., "A video depicting the action of [class name]") to text-to-video models presents two critical issues: (1) class names themselves are often semantically ambiguous—for example, "Robot Dancing" in Kinetics refers to a street dance style where humans mimic robots, but a templated prompt will yield actual mechanical robots dancing; (2) templated prompts produce repetitive, stereotypical samples that lack the visual diversity required for robust training.

Gen2Balance's prompt pipeline resolves ambiguity hierarchically through three tiers. The first tier, Diversity Prompts, uses an MLLM (Gemini 2.5 Pro) to generate variations along nine diversity axes—scene setting, camera framing, video quality, actor demographics, relevant props, action intensity, lighting conditions, background density, and social context—ensuring that generated videos cover diverse visual conditions. The second tier, Action Profile Disambiguation, prompts the MLLM to generate an Action Profile $\mathcal{A}_c$ for each class, which comprises three parts: (a) action definition, (b) positive constraints (visual features that should be present), and (c) negative constraints (common misconceptions, such as "not a mechanical robot"). This step acts as reverse-engineering the dataset annotation process, encoding the prior knowledge used by human annotators. The MLLM then generates prompts conditioned on the Action Profile: $\mathcal{T}_c = \mathcal{M}(\mathcal{A}_c, c)$. However, text descriptions alone can still lead to incorrect interpretations (e.g., real robots still appearing in Fig 2 Col. 3). The third tier, In-Context Exemplars Anchoring, feeds 5 real training videos $\mathcal{S}_c$ to the MLLM as in-context exemplars, letting the MLLM observe real samples before generating the Action Profile: $\mathcal{A}_c = \mathcal{M}(\mathcal{S}_c, c)$, $\mathcal{T}_c = \mathcal{M}(\mathcal{A}_c, c)$. A critical design choice: exemplars only influence the Action Profile and are not directly injected into the final prompt generation phase—ensuring that prompt diversity driven by the diversity axes is not constrained by the narrow distribution of a few real samples.

Ablation studies (Table 6) validate the contribution of each tier: naive prompting yields an average accuracy of only 37.3% on 10 tail classes, which increases to 44.9% with diversity prompts, 47.2% with Action Profiles, and reaches 49.8% with the complete pipeline (including In-Context Exemplars), while FVD consistently drops from 1277.1 to 836.1.

2. Two-Stage Training: Synthetic Representation Learning & Real Domain Calibration with Real Frequency Priors

Simply mixing synthetic and real videos to train using standard CE not only fails to boost performance but degrades it by 4% (Table 5, (a) vs (b)). This occurs because the model learns shortcuts specific to synthetic data instead of generalizable features, and the synthetic-to-real domain shift pulls representations away from the real data manifold.

The two-stage protocol of Gen2Balance resolves this bottleneck. Stage 1: Train on $\mathcal{D}_{aug}$ using the Balanced Softmax (BSCE) loss, formulated as: $$\mathcal{L}_{BS} = -\log\left(\frac{N_c e^{\eta_c}}{\sum_{j=1}^{C} N_j e^{\eta_j}}\right)$$ where $\eta_c$ represents the logit for class $c$, and $N_c$ is the class sample count. BSCE adjusts the loss margin based on class frequencies, penalizing minority classes with larger gradients. Crucial Detail: The frequencies $N_c$ are derived from the original training set ($\mathcal{D}_{train}$), not the augmented count $N_c + N'_c$. If the latter were used, they would be identical across all classes under a perfectly balanced dataset, causing BSCE to degenerate into standard CE, completely nullifying the re-weighting effect. Utilizing real frequencies assigns larger loss margins to minority classes, allowing the model to learn representations from abundant synthetic data while keeping decision boundaries calibrated to the real class prior. Stage 2: Fine-tune exclusively on $\mathcal{D}_{train}$ with the same BSCE loss using a lower learning rate ($5 \times 10^{-4}$ vs. $5 \times 10^{-3}$ in Stage 1) for minor corrective updates. This eradicates the domain shift introduced by synthetic data without discarding the learned representations.

The ablation in Table 5 dissects each choice: training on the augmented set with CE drops performance by 4% ((b) vs (a)); incorporating BSCE with augmented frequencies raises it to 61.3% ((c)); adding Stage 2 fine-tuning improves it to 70.6% ((d)); and using real frequencies as the BSCE prior further elevates performance to 70.9% ((g))—boosting few-shot classes from 61.0% to 62.1% while maintaining head performance at 87.9%. Replacing Stage 2 fine-tuning with a prolonged Stage 1 (135 epochs without Stage 2, 135 vs. 100+35) yields only 67.9% ((f)), proving that the gains stem from the two-stage strategy rather than just extended training duration.

Loss & Training¶

Both stages utilize the Balanced Softmax loss. Stage 1: 100 epochs, learning rate $5 \times 10^{-3}$, AdamW optimizer, weight decay 0.05, batch size 84, cosine decay with a 5-epoch linear warm-up, input resolution $224 \times 224$. Stage 2: 35 epochs, learning rate $5 \times 10^{-4}$, keeping other hyper-parameters identical. In both stages, BSCE uniformly uses the sample count $N_c$ from the original real training set.

Key Experimental Results¶

Main Results¶

Method	K100-LT Few	K100-LT Tail	K100-LT Head	K100-LT Avg	UCF-LT Few	UCF-LT Tail	UCF-LT Head	UCF-LT Avg
CE (Full Dataset)	74.9	81.7	88.4	80.4	92.7	89.4	94.0	92.0
CE	23.4	63.3	94.7	54.8	45.7	79.6	95.7	64.0
BSCE	48.7	68.5	86.7	64.5	74.3	83.0	88.5	79.3
Logit Adj.	55.5	68.4	78.5	65.6	81.2	83.4	88.6	83.1
LMR (Video SOTA)	51.5	68.7	83.1	65.1	79.6	84.6	94.4	83.6
Sariyildiz et al. (Gen)	26.2	58.8	93.2	52.8	73.1	86.4	94.3	80.6
Li et al. (Gen)	23.8	62.6	93.4	54.4	75.2	88.1	95.9	82.5
Gen2Balance	62.2	75.0	88.4	72.6	86.7	90.3	93.4	88.9

K100-LT is a long-tailed subset of 100 temporally dependent classes selected from Kinetics-400 (imbalance ratio 198, 30% few-shot classes). UCF-LT is a long-tailed variant of UCF-101 (imbalance ratio 24, 55% few-shot classes). Gen2Balance achieves the highest overall accuracy on both benchmarks, with particularly remarkable gains in few-shot classes (+6.7% on K100-LT and +5.5% on UCF-LT compared to Logit Adj.), while keeping the head-class accuracy on par with the standard CE baseline (88.4% vs 94.7% on K100-LT, which is a much smaller drop than Logit Adj.'s 78.5%). The two pure generation baselines (Sariyildiz et al. and Li et al.), despite using the same set of synthetic data, fail to match even non-generative long-tailed baselines, demonstrating that synthetic data alone is insufficient and proper training strategies are equally crucial.

RareAct Rare Actions: Appending 22 cleaned rare action classes from RareAct (e.g., cut keyboard, drill phone) as few-shot classes to K100-LT, Gen2Balance achieves 59.7% on these 22 classes, whereas CE yields only 11.3% and Logit Adj. 27.8%—demonstrating that generative balancing is highly effective for truly scarce real-world actions.

Ablation Study¶

Training Strategy Ablation (Table 5, B=330)

Config	Data	Loss	Frequency Source	Epochs	Stage 2	Few	Tail	Head	Avg C/A
(a)	$\mathcal{D}_{train}$	CE	-	100	No	23.4	63.3	94.7	54.8
(b)	$\mathcal{D}_{aug}$	CE	-	100	No	27.3	55.2	91.2	50.8
(c)	$\mathcal{D}_{aug}$	BSCE	$N+N'$	100	No	41.8	65.7	90.5	61.3
(d)	$\mathcal{D}_{aug}$	BSCE	$N+N'$	100+35	Yes	61.0	72.4	87.7	70.6
(e)	$\mathcal{D}_{aug}$	BSCE	$N$	100	No	61.9	68.3	82.7	67.8
(f)	$\mathcal{D}_{aug}$	BSCE	$N$	135	No	54.6	70.7	89.7	67.9
(g)	$\mathcal{D}_{aug}$	BSCE	$N$	100+35	Yes	62.1	72.3	87.9	70.9

Three progressive conclusions emerge: (1) mixing synthetic data and training with CE degrades performance ((b) < (a)), highlighting the necessity of class-balanced losses; (2) Stage 2 fine-tuning ((d) vs (c)) brings massive gains, successfully correcting domain shift; (3) employing real frequencies as the BSCE prior ((g) vs (d)) further elevates minority-class performance while preserving head-class performance.

Prompting Pipeline Ablation (Table 6, 10 Tail/Few-Shot Classes)

Pipeline Stage	FVD↓	ViCLIP↑	10-Class Avg↑	Overall Avg↑
Naive Prompting	1277.1	0.1635	37.3	68.9
+ Diversity Prompts	886.8	0.1585	44.9	69.5
+ Action Profile	842.8	0.1649	47.2	70.0
+ In-Context Exemplars	836.1	0.1774	49.8	70.9

With each added pipeline tier, the Fréchet Video Distance (FVD, lower is better) consistently decreases, overall semantic similarity evaluated via ViCLIP reaches its peak at the final tier, and classification accuracy steadily climbs, validating the necessity of the proposed three-tier design.

Key Findings¶

Core contributions stem from training strategies rather than simply data volume: The two baseline generative methods leverage the same synthetic datasets but perform significantly worse than Gen2Balance (even falling behind non-generative methods), proving that the two-stage training scheme combined with real-frequency BSCE is the decisive factor.
Highly cost-effective partial balancing: On K100-LT, setting $B=330$ (scaling tail classes to 1/3 of the maximum head class size) requires only 2.5K HW GPU hours but achieves 70.9% Avg C/A, retaining 79% of the performance gains of complete balancing ($B=990$, 9.2K GPU hours, 72.6%) while cutting computational costs by 73%.
Spillover benefits of generative balancing on small datasets: On UCF-LT, even when $B$ exceeds the largest head class size ($B > 121$) to over-sample, minority-class performance continuously improves (from 74.4% to 88.8%), indicating that on smaller benchmarks, generative balancing acts as an effective distribution-wide data augmentation rather than just correcting long-tails.
Robustness to Web-popularity reordering: To eliminate potential bias from web-pretraining data, classes in K100-LT were reordered according to their occurrence frequency in WebVid10M (pushing truly rare classes in the pretraining dataset to few-shot classes). Gen2Balance still outperforms Logit Adj. by 19% on the rescheduled few-shot classes, demonstrating that the method does not rely solely on the generative model's memory of Web pretraining data.
Human-validated generation quality: A user study spanning 500 tasks and 2,500 trials shows that the semantic accuracy of the generated videos is 87%, which is exceptionally close to the 92% accuracy of real Kinetics training videos (both errors were false negatives with zero false positives). This highlights a marginal 5% gap in labeling noise between synthetic and real-world data.
Orthogonality to backbone scale: When evaluated using a larger V-JEPA 2 (375M) backbone, Gen2Balance maintains its superior performance (76.3% vs. BSCE's 71.3%), and the gain remains consistent with that on VideoMAE (+5.0% vs. +6.4%), suggesting the benefits from synthetic data and training strategies scales orthogonally to backbone capacity.

Highlights & Insights¶

Clear progressive disambiguation logic in the three-tier prompting pipeline: Moving from "diversity expansion" to "definition disambiguation" to "exemplar anchoring", each tier elegantly addresses the pitfalls of the preceding one. The isolated design boundaries (preventing exemplars from directly leaking into the prompt generation phase to avoid diversity collapse) offer a solid template for using LLMs/generative models in data synthesis tasks.
"Using real frequencies as the BSCE prior" is a subtle yet highly impactful design choice: On a fully balanced augmented set, if the augmented sample count is used, BSCE degrades to CE and all re-weighting effects vanish. Using real frequencies encodes the intuition that "synthesized data can be added arbitrarily, but class weights should remain bound to the real-world distribution." This trick is widely applicable to any "synthetic data + reweighting loss" paradigms.
Benchmarking synthetic labeling quality directly against real data in user studies is exceptionally rigorous: Instead of assessing synthesized videos in isolation, comparing them closely with real training sets under identical standards reveals that real data has an 8% annotation noise level, while the synthetic noise level is 13%—a mere 5 percentage point gap. This comparative framing provides a great blueprint for validating generative data augmentation.
The scaling analysis of partial vs. full balancing offers practical engineering guidelines: Capturing 79% of the gains at only 27% of the cost is critical for industrial deployment, demonstrating that full dataset balancing is not always necessary to unlock maximum utility.

Limitations & Future Work¶

High computational footprint: Full balancing on K100-LT requires 9.2K H100 GPU hours for video generation. Although this is a one-time offline cost (analogous to dataset collection), it remains a heavy burden for scaling up to larger datasets (e.g., full Kinetics-400). Partial balancing ($B=330$, 2.5K hours) presents a practical compromise.
Synthesized video quality is bounded by generative model capacity: Failure cases still exist, such as objects appearing but failing to perform the target action, or actions being performed without necessary props. As video generation models (e.g., WAN 2.1) continue to evolve, downstream recognition will scale accordingly.
Selective focus on temporally active classes might limit generalizability: The 100 classes in K100-LT were intentionally chosen to be temporally dependent (e.g., breakdancing, robot dancing), accentuating the advantage of video over static imagery (as shown in Table 4, where Qwen-Image generation falls behind WAN video generation). For classes where static features suffice (e.g., drinking vs. smoking), the marginal utility of generative video over cheaper image augmentation might shrink.
Alternative MLLMs are not exhaustively evaluated: Although Table 0.B.3 tests Qwen3-VL as a prompt generator and shows consistent trends, it is limited to 10 classes. The influence of different MLLMs on prompt variation and subsequent large-scale training (100+ classes) remains under-studied.
Stage 2 "rehearsal" essentially serves to mitigate catastrophic forgetting: Although effective, the paper does not quantitatively measure the exact degree of forgetting or recovery across the two stages. Quantitative forgetting analysis would further solidify the necessity of this two-stage design.
Future Directions: (1) Integrate automatic quality filtering into the generation pipeline to weed out semantically deviated syntheses and improve stability. (2) Explore lighter domain adaptation techniques (such as adversarial training or feature alignment) to replace Stage 2 fine-tuning, aiming to match performance with fewer real data updates. (3) Extend the paradigm to other video understanding tasks such as temporal action localization (TAL) and video QA.

vs. SYNAuG / Fill-Up (Generative Augmentation in the Image Domain): Both utilize textual inversion or simple prompts for long-tailed image augmentation alongside two-stage training. Gen2Balance differentiates itself by: (a) addressing additional challenges of temporal consistency in videos; (b) presenting a three-tier prompting pipeline that scales better than textual inversion (which demands per-class training and exceeds H100 VRAM on WAN 2.1-14B); (c) employing a simple "look once" strategy via MLLM (In-Context Exemplars) to replace training-heavy inversion, offering a smart engineering simplification.
vs. LTGC (LLM-guided Image Generation): LTGC uses LLM-generated diverse prompts for tail-class image synthesis to improve VLMs. Gen2Balance extends this by adding Action Profile and In-Context Exemplars to resolve semantic ambiguities, which are far more severe in videos (e.g., "robot dancing" can only be distinguished from a literal mechanical robot moving through temporal patterns). The idea of tailoring prompting depth to the ambiguity of the target data modality is highly instructive.
vs. LMR / MOVE / MEDC (Video Long-Tailed Recognition): These methods operate solely in the feature space (e.g., LMR combines head/tail features, MOVE performs feature extrapolation/interpolation, and MEDC uses multi-expert branches). Gen2Balance is the first to address long-tailed video recognition in the pixel space by synthesizing brand-new video instances. They are inherently complementary: feature-space methods are cheap but cannot inject new visual concepts, while pixel-space generation is expressive but computationally heavy. Combining both—pre-training backbones with generative video and then augmenting inside the feature space—is a natural next step.
vs. Li et al. (Video Generation for Few-Shot Learning): Li et al. pre-trains on synthetic videos and fine-tunes on real data, resembling Gen2Balance's two-stage setup. However, the differences are: (a) they use naive templated prompts, whereas Gen2Balance implements a three-tier disambiguation pipeline; (b) their second stage relies on uncertainty-based label smoothing, while Gen2Balance uses BSCE with real-frequency priors. Experiments prove the latter is vastly superior for long-tailed scenarios (Li et al. scores only 54.4% compared to Gen2Balance's 72.6% on K100-LT).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to apply generative pixel-space augmentation to video long-tailed recognition. The proposed three-tier prompting pipeline (Action Profile + In-Context Exemplars) is novel and simultaneously addresses the orthogonal challenges of "diversity" and "semantic correctness" in the video domain.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Two benchmarks + RareAct extension + 6 sets of ablation studies (training strategies, rendering pipelines, data scales, target size scaling, backbone generalization, MLLM generalization, frequency re-indexing, leak checks, and user studies) ensure thorough and well-explained causal conclusions.
Writing Quality: ⭐⭐⭐⭐⭐ Highly polished layout. The "Robot Dancing" pipeline diagram in Figure 2 is an exemplary visual explanation, showing progressive pipeline execution with a clear example. The main texts are mathematically sound, and the supplementary material features extensive prompt listings and failure analysis.
Value: ⭐⭐⭐⭐⭐ Releasing a dataset of 140K synthesized videos (spanning 223 classes across three benchmarks) provides a powerful asset for future generative augmentation or long-tail video tasks. Additionally, the insights on partial balancing and real-frequency BSCE priors provide direct engineering blueprints for shipping models.

Config	Data	Loss	Frequency Source	Epochs	Stage 2	Few	Tail	Head	Avg C/A
(a)	\(\mathcal{D}_{train}\)	CE	-	100	No	23.4	63.3	94.7	54.8
(b)	\(\mathcal{D}_{aug}\)	CE	-	100	No	27.3	55.2	91.2	50.8
(c)	\(\mathcal{D}_{aug}\)	BSCE	\(N+N'\)	100	No	41.8	65.7	90.5	61.3
(d)	\(\mathcal{D}_{aug}\)	BSCE	\(N+N'\)	100+35	Yes	61.0	72.4	87.7	70.6
(e)	\(\mathcal{D}_{aug}\)	BSCE	\(N\)	100	No	61.9	68.3	82.7	67.8
(f)	\(\mathcal{D}_{aug}\)	BSCE	\(N\)	135	No	54.6	70.7	89.7	67.9
(g)	\(\mathcal{D}_{aug}\)	BSCE	\(N\)	100+35	Yes	62.1	72.3	87.9	70.9