MoAKE: Toward Unified All-in-One Action Quality Assessment via Mixture of Action Knowledge Experts¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/XuHuangbiao/MoAKE
Area: Human Understanding
Keywords: Action Quality Assessment, All-in-One Unified Model, Mixture of Experts, Zero/Few-shot Learning, Multi-granularity Temporal Modeling
TL;DR¶
Addressing the severe negative transfer and deployment bottlenecks of conventional per-action assessment pipelines, MoAKE introduces the first unified all-in-one action quality assessment framework that couples segment-aware prototype aggregation, adaptive multi-granularity relationship modeling, and label-free soft routing to achieve positive cross-action transfer, outperforming existing methods on both unified benchmarks and zero/few-shot generalization.
Background & Motivation¶
Action Quality Assessment (AQA) aims to objectively evaluate human motion performance from video recordings, playing a crucial role across competitive sports scoring, surgical training, and medical rehabilitation. However, existing literature overwhelmingly adheres to a fragmented "one-by-one" paradigm, in which dedicated models are trained and deployed independently for each distinct action category. This isolated setup poses severe barriers to real-world applications: open-domain video platforms host heterogeneous user-generated content that lacks action-type metadata; automated content-agnostic screening services cannot afford manual action routing; and low-resource or rare action classes suffer from prohibitive data collection and model retraining costs.
Naively consolidating diverse actions into a single monolithic model for all-in-one training triggers severe "negative transfer." Because distinct movement types exhibit vastly different kinematic structures, execution cadences, and grading criteria, cross-action gradients interfere in a shared parameter space. As a result, prominent AQA baselines suffer an average drop of 10.17% in Spearman's Rank Correlation Coefficient (SRCC) when trained in an all-in-one manner. Prior architectures lack mechanisms to adaptively isolate conflicting signals or extract the underlying motor aesthetics that could otherwise serve as complementary knowledge.
The core opportunity is to transform this semantic discrepancy from an impediment into a source of cross-domain synergy. Core idea: develop a lightweight Mixture of Action Knowledge Experts (MoAKE) framework that unifies heterogeneous temporal lengths via shared adapters and segment-aware prototypes, models multi-granularity dynamics via adaptive intra- and inter-segment relations, and dynamically mixes complementary expert knowledge via a label-free visual soft router.
Method¶
Overall Architecture¶
MoAKE processes variable-length video feature sequences through a single unified architecture to evaluate heterogeneous actions without requiring test-time category labels. The processing pipeline advances through four sequential stages: first, snippet-level visual features extracted by a pretrained backbone are dimensionally reduced and projected into a unified semantic space via a two-layer MLP adapter shared across all experts; second, each expert employs specialized segment-aware prototypes (SATA) to compress variable temporal lengths into a standardized sequence of segment tokens via cross-attention; third, the Adaptive Intra- and Inter-Segment Relationship Modeling (AIISRM) module processes these tokens across three parallel pathways targeting intra-segment global feature evolution, inter-segment multi-scale local correlations, and inter-segment long-range consistency; finally, an input-dependent soft router dynamically infers normalized expert weights from visual features to produce a mixed multi-expert representation, which a grade-aware decoder translates into discrete score distribution expectations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input Video Feature Sequence<br/>X in R^(T x D), variable length T"] --> SATA["Shared Adaptation & Segment-Aware Temporal Aggregation<br/>Align shared semantic space + standardize length to N"]
SATA --> AIISRM["Adaptive Intra- and Inter-Segment Relationship Modeling<br/>Tri-branch intra-global, inter-local, and inter-global modeling"]
In -.->|Lightweight Soft Router R| Router["Visual-Guided Soft Routing & Dynamic Expert Fusion<br/>Label-free dynamic weighting across K action experts"]
AIISRM --> Router
Router --> Decoder["Grade-Aware Decoding & Diversity Regularization<br/>M learnable queries predict discrete grade expectation"]
Decoder --> Out["Predicted Quality Score y_hat"]
Key Designs¶
1. Shared Adaptation & Segment-Aware Temporal Aggregation: Unifying action semantics and resolving variable temporal durations Heterogeneous action sequences exhibit dramatic variations in temporal length (for instance, rhythmic gymnastics spans 68 snippets, figure skating spans 124 snippets, while short-term diving contains only 10 snippets). To build a shared foundation across these divergent regimes, MoAKE passes reduced snippet features \(X \in \mathbb{R}^{T \times d}\) through a two-layer MLP adapter shared across all experts, obtaining unified representations \(X' \in \mathbb{R}^{T \times d}\). Within each expert \(E_k\), the framework assigns \(N\) learnable segment-aware prototypes \(P_k \in \mathbb{R}^{N \times d}\) (\(N=64\)). Following the cross-attention formulation of Q-Former, the prototypes serve as queries \(Q = W_{seg}^Q P_k\) to aggregate temporal information from keys and values \(K = W_{seg}^K X', V = W_{seg}^V X'\): $$ F_{seg}^k = \text{Softmax}\left(\frac{W_{seg}^Q P_k (W_{seg}^K X')^T}{\sqrt{d}}\right) W_{seg}^V X' \in \mathbb{R}^{N \times d} $$ This mechanism compresses arbitrary temporal lengths \(T\) into a fixed token length \(N\), standardizing downstream tensor dimensions while allowing each expert's prototypes to capture specialized temporal patterns.
2. Adaptive Intra- and Inter-Segment Relationship Modeling: Disentangling multi-granularity dynamics and execution coherence While prototype aggregation standardizes temporal lengths, evaluating subtle motion execution requires modeling temporal dynamics across multiple granularities. The AIISRM module refines \(F_{seg}^k\) using three specialized parallel branches: - Intra-Segment Global Relationship Branch: Transposes the segment representation to \((F_{seg}^k)^T \in \mathbb{R}^{d \times N}\) and applies self-attention along the feature dimension \(d\). Rather than computing dependencies between temporal steps, this branch captures how individual kinematic attributes (such as posture, acceleration, and balance) evolve synchronously across the entire sequence. - Inter-Segment Local Relationship Branch: Because complex actions consist of structured sub-movements, three parallel 1D convolutions with kernel sizes 1, 3, and 5 capture multi-scale local transition dynamics. A lightweight convolutional gating block adaptively predicts dynamic weights \(w_i\) to blend these scales: $$ F_{local}^k = \sum_{i \in {1, 3, 5}} w_i \cdot \text{Conv1D}i(F^k) $$ - Inter-Segment Global Relationship Branch: Applies standard self-attention across the \(N\) segment tokens, connecting distant execution phases to evaluate overall rhythm, symmetry, and pacing. The outputs of the three branches are combined via element-wise summation and residual addition with the input: \(F_k = F_{seg}^k + F_{intra}^k + F_{local}^k + F_{inter}^k\), yielding a refined feature that balances micro-level execution details with holistic choreography.
3. Visual-Guided Soft Routing & Dynamic Expert Fusion: Mitigating negative transfer via label-free adaptive integration Conventional MoE architectures frequently employ hard routing or token-level sparse selection, which strictly isolates sub-networks and impedes cross-domain synergy. MoAKE operates at the macro action level using a lightweight soft router \(R(\cdot)\). During inference, the router inspects the unlabelled visual features \(X\) and dynamically generates continuous expert weights \(\{w_k\}_{k=1}^K\) satisfying \(\sum_k w_k = 1\). The final representation is obtained by a linear combination: $$ F_{mix} = \sum_{k=1}^K w_k F_k $$ By avoiding discrete hard gating, the model dynamically blends complementary proficiencies—such as posture precision from gymnastics and fluid continuity from skating—without requiring ground-truth action labels during testing. This lightweight design (8.01M total parameters) effectively converts cross-action competition into positive transfer.
4. Grade-Aware Decoding & Diversity Regularization: Enhancing fine-grained score discrimination Direct linear score regression often fails to capture non-linear grading boundaries. MoAKE employs a Transformer decoder where \(M=4\) learnable grade queries interact with \(F_{mix}\) to produce \(M\) discrete grade-aware vectors \(\{r_m\}_{m=1}^M\). A shared MLP predicts a probability distribution over fixed grade anchors \(G_m = \frac{m-1}{M-1}\), computing the final score as the expectation \(\hat{y}\). A triplet diversity loss penalizes cosine similarity between distinct grade vectors, ensuring that the model maintains sharp sensitivity to subtle execution errors.
Loss & Training¶
MoAKE is trained end-to-end under a composite objective: $$ \mathcal{L}{total} = \lambda_1 \mathcal{L}} + \lambda_2 \mathcal{L{route} + \lambda_3 \mathcal{L} $$ with loss weights set to \(\lambda_1 = 6, \lambda_2 = 1, \lambda_3 = 1\). - Score Regression Loss \(\mathcal{L}_{score}\): Optimizes Mean Squared Error (MSE) on both the unified prediction \(\hat{y}\) and an auxiliary prediction \(\hat{y}_k\) generated by the corresponding action expert: \(\mathcal{L}_{score} = \text{MSE}(\hat{y}, y) + \text{MSE}(\hat{y}_k, y)\), reinforcing expert specialization during training. - Soft Routing Loss \(\mathcal{L}_{route}\): Employs label-smoothed cross-entropy (\(\epsilon = 0.3\)) supervised by ground-truth action labels \(c_k\): $$ \mathcal{L}{route} = -\sum\right) \log(w_k) $$ Label smoothing prevents the router from collapsing into a rigid categorical switch, encouraging the discovery of shared cross-action patterns. - }^K \left(c_k (1 - \epsilon) + \frac{\epsilon}{KGrade Diversity Loss \(\mathcal{L}_{div}\): Applies a triplet margin loss across grade representations \(r_m\): $$ \mathcal{L}{div} = \sum(r_m, r_j) + \alpha\right]_+ $$ The network is optimized using Adam for 380 epochs with an initial learning rate of } \left[\max_{j \neq m} \text{cos}(r_m, r_j) - \min_{j \neq m} \text{cos\(3.4 \times 10^{-4}\) decayed via cosine annealing to \(3.4 \times 10^{-6}\), accompanied by a cross-action snippet replacement augmentation strategy (probability 0.6, replacing 10% snippets).
Key Experimental Results¶
Main Results¶
The authors established unified benchmarks across three long-term actions (Rhythmic Gymnastics, Figure Skating, Artistic Swimming) and zero/few-shot transfer benchmarks on three short-term actions (Diving, Skiing/Snowboarding, Surgical Robotics).
Table 1: All-in-One unified assessment comparisons on three long-term benchmarks (Metrics: SRCC \(\rho \uparrow\) and R-\(\ell_2 \downarrow\); bold denotes best)
| Paradigm | Methods | Rhythmic Gymnastics (RG) SRCC / R-\(\ell_2\) | Figure Skating (Fis-V) SRCC / R-\(\ell_2\) | Artistic Swimming (LOGO) SRCC / R-\(\ell_2\) | Average SRCC / R-\(\ell_2\) |
|---|---|---|---|---|---|
| One-by-One | GDLT (CVPR'22) | 0.765 / 2.401 | 0.685 / 3.717 | 0.647 / 4.148 | 0.703 / 3.422 |
| One-by-One | CoFInAl (IJCAI'24) | 0.807 / 4.028 | 0.716 / 2.875 | 0.698 / 4.019 | 0.744 / 3.641 |
| One-by-One | ASGTN (TCSVT'25) | 0.799 / 3.027 | 0.703 / 3.039 | 0.704 / 5.695 | 0.739 / 3.920 |
| All-in-One | GDLT (CVPR'22) | 0.702 / 2.716 | 0.377 / 4.681 | 0.447 / 5.499 | 0.525 / 4.299 |
| All-in-One | CoFInAl (IJCAI'24) | 0.741 / 2.834 | 0.496 / 3.032 | 0.603 / 6.580 | 0.624 / 4.149 |
| All-in-One | ASGTN (TCSVT'25) | 0.767 / 3.558 | 0.602 / 3.790 | 0.545 / 4.809 | 0.649 / 4.052 |
| All-in-One | Cond-Base (Baseline) | 0.748 / 2.891 | 0.621 / 3.900 | 0.769 / 4.924 | 0.718 / 3.905 |
| All-in-One | MCMoE (AAAI'26) | 0.771 / 3.426 | 0.625 / 3.608 | 0.763 / 4.816 | 0.726 / 3.950 |
| All-in-One | MLAVL (CVPR'25) | 0.755 / 2.655 | 0.632 / 2.919 | 0.765 / 5.043 | 0.722 / 3.539 |
| All-in-One | MoAKE (Ours) | 0.801 / 1.822 | 0.647 / 2.771 | 0.790 / 2.708 | 0.753 / 2.434 |
Table 2: Generalization comparisons on unseen short-term benchmarks under zero-shot and few-shot protocols
| Protocol | Methods | Diving (MTL-AQA) SRCC / R-\(\ell_2\) | Skiing (AQA-7) SRCC / R-\(\ell_2\) | Surgery (JIGSAWS) SRCC / R-\(\ell_2\) |
|---|---|---|---|---|
| Zero-Shot | GDLT | 0.088 / 4.564 | -0.024 / 23.615 | 0.133 / 15.917 |
| Zero-Shot | ASGTN | 0.198 / 8.793 | 0.143 / 37.896 | -0.055 / 28.787 |
| Zero-Shot | MLAVL | 0.211 / 5.017 | 0.148 / 18.249 | 0.122 / 16.053 |
| Zero-Shot | MoAKE (Ours) | 0.319 / 3.928 | 0.210 / 12.640 | 0.219 / 12.367 |
| Few-Shot | GDLT | 0.605 / 2.319 | 0.463 / 10.426 | 0.602 / 12.568 |
| Few-Shot | ASGTN | 0.577 / 2.199 | 0.443 / 10.094 | 0.576 / 12.082 |
| Few-Shot | MLAVL | 0.618 / 2.244 | 0.458 / 11.247 | 0.620 / 13.047 |
| Few-Shot | MoAKE (Ours) | 0.664 / 2.058 | 0.513 / 9.138 | 0.651 / 11.168 |
Ablation Study¶
Table 3 presents component ablations evaluated on the average of the three long-term datasets under the all-in-one setup.
Table 3: Ablation study of MoAKE components on long-term all-in-one benchmark
| Config | Average SRCC \(\uparrow\) | Rel. Change | Average R-\(\ell_2 \downarrow\) | Rel. Change | Note |
|---|---|---|---|---|---|
| Simple Baseline (Single Branch) | 0.506 | - | 5.245 | - | Linear head with temporal self-attention |
| + SATA (Single Branch) | 0.607 | +20.0% | 4.057 | -22.7% | Introduces segment prototypes |
| + SATA & AIISRM (Single Branch) | 0.655 | +7.9% | 3.439 | -15.2% | Adds multi-granularity modeling |
| + MoAKE w/ Specific Adapter | 0.742 | +13.3% | 2.628 | -23.6% | Multi-expert expansion with separate adapters |
| Full Model (MoAKE w/ Shared Adapter) | 0.753 | - | 2.434 | - | Complete proposed framework |
| Full Model w/o AIISRM | 0.720 | -4.4% | 2.905 | +19.3% | Removes multi-granularity refinement |
| Full Model w/o SATA & AIISRM | 0.683 | -9.3% | 3.109 | +27.7% | Removes prototype & multi-granularity |
| Full Model w/o Shared Adapter | 0.737 | -2.1% | 2.648 | +8.8% | Replaces shared adapter with distinct MLPs |
| Full Model w/o Soft Router | 0.529 | -29.7% | 6.081 | +149.8% | Replaces soft router with uniform average |
| Full Model w/o \(\mathcal{L}_{route}\) | 0.730 | -3.1% | 2.952 | +21.3% | Router trained without category guidance |
| Full Model w/o \(\mathcal{L}_{div}\) | 0.718 | -4.6% | 2.888 | +18.6% | Drops grade query separation loss |
| Full Model w/o \(\mathcal{L}_{route}\) & \(\mathcal{L}_{div}\) | 0.706 | -6.2% | 3.131 | +28.6% | Drops both auxiliary regularizers |
Key Findings¶
- The soft router is the single most critical safeguard against negative transfer: Removing the soft router collapses average SRCC from 0.753 down to 0.529 (-29.7%) and balloons R-\(\ell_2\) error by 149.8% (from 2.434 to 6.081). This empirically verifies that unmoderated expert feature blending causes fatal representational interference across conflicting action patterns.
- Unified training achieves positive transfer over per-action models: MoAKE improves upon its own One-by-One baselines across all datasets (+3.0% on RG, +1.1% on Fis-V, and +9.1% on LOGO in SRCC), yielding an overall +4.7% SRCC gain and a 34.4% drop in R-\(\ell_2\) (Table 3 of the paper). Unlike baseline architectures that degrade under multi-task setups, MoAKE successfully leverages cross-action complementarity.
- Remarkable parameter efficiency: Computational analysis (Table 5) reveals that MoAKE requires only 8.01M parameters and 1.44G FLOPs, far lighter than multimodal architectures like PAMFN (18.06M / 2.562G), while outperforming conditional routing baselines (0.753 vs 0.718 in SRCC).
Highlights & Insights¶
- Macro-Action MoE avoids token-level parameter explosion: Rather than expanding parameter counts through token-level sparse routing in every layer, MoAKE deploys a compact set of \(K=3\) action-level experts combined via feature-level soft aggregation, keeping total parameters at an ultra-compact 8.01M.
- Feature transposition captures kinematic attribute trajectories: The intra-segment global branch transposes the token sequence to \((F_{seg}^k)^T \in \mathbb{R}^{d \times N}\) to perform attention across feature channels, offering a mathematically clean way to track how coordinate and velocity dynamics evolve across time.
- Label-free zero-shot inference: Because the soft router relies purely on input video visual features at inference time, MoAKE eliminates the need for an external action classification front-end, making it directly deployable for in-the-wild video quality inspection.
Limitations & Future Work¶
- Absolute correlation in zero-shot regimes remains modest: While MoAKE achieves substantial relative improvements over prior art in zero-shot assessment (0.319 SRCC on diving), absolute predictive correlations for completely unseen domains like surgical robotics (0.219 SRCC) remain relatively low for standalone automated deployment.
- Coupling of expert count with primary training categories: The current configuration links \(K=3\) experts directly to the three training datasets. Scaling the framework to hundreds of sports or clinical activities will require investigating dynamic expert allocation and parameter sharing.
- Absence of multimodal audio and textual regulation: MoAKE operates strictly on visual snippets, omitting auditory rhythm cues and judging rubrics (Code of Points), which could provide explicit semantic constraints.
Related Work & Insights¶
- vs Conventional Per-Action Models (GDLT, CoFInAl, ASGTN): Prior models achieve strong metrics in isolated training but degrade severely when pooled into all-in-one datasets (e.g., GDLT drops from 0.703 to 0.525 average SRCC). MoAKE introduces shared adaptation and soft routing to resolve feature interference, turning cross-action variety into a positive transfer asset.
- vs Multimodal Missing-Expert Models (MCMoE): MCMoE applies MoE routing across missing sensory modalities (RGB, flow, audio). In contrast, MoAKE targets the cross-action domain gap within the visual modality, introducing segment-aware prototype queries and multi-granularity intra/inter-segment temporal operators.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first systematic all-in-one action quality assessment formulation and benchmarks with an elegant action-level MoE architecture.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation spanning 6 diverse benchmarks across long-term all-in-one, zero-shot, few-shot, and exhaustive module ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough theoretical justifications for every architectural choice, and informative visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational milestone and reproducible blueprint for deploying generalized, content-agnostic AQA systems.