DisentangledTMR: Privacy-Preserving Skeleton Motion Retargeting via Factorized Transformers

Conference: ECCV2026
Paper: Official paper page / PDF
Area: Video Understanding / Skeleton Action Analysis
Keywords: Skeleton anonymization, motion retargeting, disentangled representations, factorized attention, privacy-utility trade-off

TL;DR

DisentangledTMR separately encodes a source person's action and a reference person's skeletal identity, then recombines them through a factorized Transformer; at its recommended partial-retargeting setting on NTU60, pre-trained re-identification falls from 75.4% to 18.1%, with 75.8% pre-trained action recognition and 87.1% action recognition after retraining.

Background & Motivation

Removing faces and textures does not remove identity. Bone lengths and limb proportions provide static biometric cues, while gait, rhythm, and execution style provide dynamic ones. Anonymization therefore cannot simply change body shape or add joint noise: the former may leave recognizable gait intact, whereas the latter may destroy the information needed for action recognition first.

Motion retargeting offers an intuitive objective: preserve what the source person is doing, but have a different body structure perform it. However, shared-encoder approaches such as DMR and PMR compress action and identity into the same representation. Reconstruction rewards retaining information, while adversarial identity suppression rewards removing it, so the objectives compete over the same features. This paper moves part of that conflict into architectural design: dynamic action receives a high-bandwidth temporal pathway, while reference identity receives a low-bandwidth static pathway. Still, the action stream sees positions and the mean pose may depend on the action, so these are inductive biases rather than a proof of complete separation.

Deployment introduces another difficulty: downstream recognizers are often already trained on raw skeletons. An anonymized sequence may still contain action information while changed body proportions make an existing model fail. Information retention and immediate compatibility must therefore be measured separately. Core idea: reduce opportunities for action-identity mixing through asymmetric encoders and separate attention streams, remove residual leakage with staged training, and use a partial-retargeting ratio fixed during training to control compatibility with existing recognizers.

Method

Overall Architecture

The inputs are source and reference single-person 3D skeleton sequences, each normalized to 64 frames and 25 joints. The action encoder extracts time-varying motion from the source, while the identity encoder extracts a global skeletal descriptor from the reference. A factorized decoder generates motion autoregressively, followed by optional blending with the source sequence. At inference, the reference subject is sampled uniformly from a pool excluding the source; training also includes self-reconstruction with the same source and target person.

The action representation has size \(64\times768\), whereas the identity representation has only 256 dimensions, giving a total representation capacity ratio of 192. This is a ratio of representation elements, not encoder parameter counts. The identity vector is broadcast across decoding steps to condition body structure, rather than supplying a reference action trajectory to copy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    source["Source skeleton sequence"] --> action["Dual Action-Identity Encoding<br/>Temporal action / static identity"]
    target["Reference skeleton sequence"] --> action
    action --> decoder["Factorized Decoding<br/>Separate attention and gating"]
    training["Staged Disentanglement Training<br/>Encoding / reconstruction / joint"] -.-> action
    training -.-> decoder
    decoder --> blend["Partial Retargeting<br/>Fixed-ratio residual blend"]
    source --> blend
    blend --> output["Anonymized skeleton sequence"]

Key Designs

1. Dual Action-Identity Encoding: restrict interference through input access and capacity

The action stream receives joint positions, velocities from adjacent frames, and accelerations from velocity differences, giving 9 channels per joint. One branch applies parallel temporal convolutions with kernel sizes 3, 5, and 7, combining short- and longer-term changes. It then uses an intermediate dimension of 256 and 2 layers of 8-head temporal self-attention for longer-range dependencies, before projecting to 768 dimensions. A second branch uses a modified Skeleton MixFormer to produce per-frame skeleton features. A learned sigmoid gate fuses the two branches elementwise. MixFormer uses Kaiming normal initialization and is trained jointly from scratch, without pre-trained weights.

The identity stream root-centers skeletons at the pelvis, canonicalizes orientation, and averages over time. The mean pose enters a spatial GCN: an input projection maps 3 coordinates to 64 channels, followed by three graph-convolution layers with channels 64 to 128 to 256 to 256, residual connections, and 8-head spatial attention. Globally pooled 256-dimensional features are concatenated with a bone-length representation: 24 lengths computed from the mean pose pass through an MLP with dimensions 24 to 128 to 128. A fusion MLP with dimensions 384 to 512 to 256 produces the final identity vector. The text computes bone lengths from the mean pose; this should not be silently rewritten as the mean of framewise bone lengths. Removing time prevents the identity stream from directly carrying a complete trajectory, but also excludes dynamic identity cues, whose leakage must be addressed in the action stream.

2. Factorized Decoding: read action and identity separately before mixing them

The decoder has 6 layers and model dimension 320. Each layer performs causal self-attention, then separate cross-attention to action and identity representations. Instead of immediately concatenating both factors into one key-value set, it first produces separate action-conditioned and identity-conditioned features, combines them with a learned elementwise sigmoid gate, and applies normalization and a feed-forward network with dimensions 320 to 2048 to 320. A final projection produces joint coordinates. Temporal motion and body structure thus remain distinguishable pathways until fusion.

Identity supplies only one global token, so its cross-attention effectively broadcasts a conditioning signal rather than selecting among reference frames. The methodology defines the gate elementwise over time and hidden channels. Although the abstract and conclusion describe per-joint gating, the formula should not be read as explicitly containing a separate joint dimension. At inference, each output frame becomes the input for the next step, yielding fully autoregressive generation; teacher forcing reduces early error accumulation during training.

3. Staged Disentanglement Training: establish specialization before reconstruction and output supervision

Stage 1 freezes the decoder and trains the encoders with auxiliary heads. Action cross-entropy keeps the action stream informative about the activity, while identity cross-entropy preserves reference structure in the identity stream. Supervised contrastive learning uses the same action performed by different people as positive pairs, with temperature 0.07, discouraging grouping by person. An identity adversary first learns to identify people from stop-gradient action features; the action encoder then pushes its predicted identity distribution toward uniformity. This uses separate optimization steps, not a gradient-reversal layer.

The prose defines the adversarial objective clearly enough to express it as follows, where \(N_{\mathrm{id}}\) is the identity count and \(U_{N_{\mathrm{id}}}\) is the uniform distribution over those identities:

\[ \mathcal L_{\mathrm{adv}}=\mathrm{KL}\!\left(D_{\mathrm{adv}}(\cdot\mid H_{\mathrm{action}})\,\|\,U_{N_{\mathrm{id}}}\right). \]

An orthogonality penalty between normalized features and a Frobenius-norm penalty on cross-correlation provide additional separation. Both operate at a shared dimension of 256, but their preprocessing differs: the orthogonality term temporally averages action features before truncation, whereas the cross-correlation term flattens time and retains the first 256 elements. Consequently, these terms do not directly constrain every action-feature dimension. Decorrelation also does not establish full statistical independence.

Stage 2 introduces decoder reconstruction while continuing to update the encoders at 1% of the base learning rate, rather than freezing them. Framewise joint MSE, target bone-length consistency, acceleration smoothing, and velocity matching constrain geometry and jitter. Stronger position and velocity supervision covers 7 end-effectors, including hands, feet, and head; joint-range penalties and foot-speed penalties on contact frames discourage implausible poses and sliding. Stage 3 jointly optimizes all modules, retains both earlier sets of losses, and adds output action classification, enhanced end-effector supervision, a distribution discriminator, and an output identity adversary. Output supervision matters because separated encoders do not prevent the decoder gate from reintroducing identity-related signals. Auxiliary heads are discarded at inference.

4. Partial Retargeting: adapt to existing recognizers through a training-time residual connection

Full retargeting can cause substantial downstream distribution shift. The paper therefore blends decoder output with the source sequence. Following the text accompanying Equation (16), the relationship is:

\[ \hat{\mathbf s}_{\beta}=\beta D\!\left(E_A(\mathbf s_{\mathrm{src}}),E_I(\mathbf s_{\mathrm{tgt}})\right)+(1-\beta)\mathbf s_{\mathrm{src}}. \]

At the recommended \(\beta=0.2\), source coordinates receive 80% of the blend weight and decoder output receives 20%. This does not mean privacy changes by only 20%, or that exactly 80% of identity leakage remains. Each \(\beta\) is fixed before training, with the corresponding residual connection active throughout, so the decoder learns output complementary to the source. The sweep therefore compares separately trained models, not arbitrary inference-time settings of one model. At \(\beta=1.0\), the source residual disappears, but this still provides no mathematical guarantee of complete anonymity.

A Worked Example

Suppose the source person waves while the reference sequence comes from a different identity and need not depict waving. This is an illustrative walkthrough, not an additional experiment. After normalization to 64 frames, the action stream preserves the temporal structure of raising and moving the arm. The reference sequence becomes a 256-dimensional body descriptor through the mean-pose and bone-length pathways, without directly passing its action trajectory to the decoder.

The decoder generates a wave frame by frame, conditioned on the reference structure. For a configuration trained specifically with \(\beta=0.2\), generated and source coordinates are blended with weights 20% and 80%, then passed to an SGN trained on raw data. A separate evaluation trains a new action classifier on anonymized data to test whether the action remains learnable. These protocols answer different questions; the higher retrained result cannot substitute for direct-deployment performance.

Loss & Training

Training uses Adam, batch size 128, and 20, 15, and 20 epochs for the three stages, totaling approximately 28 GPU-hours. Stage 3 uses a learning rate of \(5\times10^{-5}\), one tenth of the base rate. Teacher-forcing probability decreases linearly from 1.0 to 0.5 in Stage 2 and from 0.5 to 0.3 in Stage 3; inference uses 0. Teacher forcing versus fully autoregressive decoding is selected per batch.

Stage-1 weights for action classification, identity classification, contrastive learning, adversarial suppression, orthogonality, and cross-correlation are respectively 3.0, 1.0, 1.0, 0.5, 0.1, and 0.01. Stage-2 weights for MSE, end-effectors, bones, velocity, smoothness, feet, and joint constraints are respectively 1.0, 2.0, 1.0, 0.2, 0.1, 1.0, and 1.0. The final weights were not copied directly from Optuna's best trial: the authors report that trial underperformed at full scale and manually set the weights using importance rankings as guidance.

The model has 22.7M parameters: 4.9M in the action encoder, including 2.3M in MixFormer; 0.8M in the identity encoder; and 17.0M in the decoder. Paired training combines two identities and two actions into quadruplets, with 50k quadruplets for NTU60. Equations in the cached extraction contain formatting damage. This note normalizes only the two key relationships unambiguously explained in the prose, without inventing missing loss derivations or supplementary hyperparameters.

Key Experimental Results

Main Results

All datasets use Kinect v2 skeletons with 25 joints and cross-view protocols, excluding multi-person actions. The original NTU60 dataset has 56,880 sequences, 60 actions, and 40 subjects, with 49 single-person classes after filtering. NTU120 originally has 114,480 sequences, 120 actions, and 106 subjects, with 94 classes after filtering. ETRI originally has 112,620 sequences, 55 actions, 100 subjects, and 8 viewpoints. These sequence counts describe the original datasets, not filtered training-set sizes. NTU60 trains on cameras 2 and 3 and tests on camera 1; ETRI trains on cameras 2, 3, 5, and 7 and tests on the remaining views.

AR is action-classification accuracy, where higher is better; RI is source-identity re-identification accuracy, where lower is better. The primary protocol applies raw-trained SGN classifiers directly to anonymized output. The secondary protocol retrains only the action classifier. Chance RI on NTU60 is approximately 2.5%. The following results come from Table 1; accuracies are percentages. Proposed-method pre-trained AR/RI are means and standard deviations over 5 runs, while other entries follow the source reporting. Geometric metrics compare against target ground truth; their numerical scale is preserved without inventing millimeter units.

Method	Pre-trained AR (higher)	Pre-trained RI (lower)	Retrained AR (higher)	MSE (lower)	MPJPE (lower)	Bone error (lower)	Smoothness term (lower)	Velocity error (lower)
Raw skeleton	89.1	75.4	89.1	N/A	N/A	N/A	N/A	N/A
DMR	49.1	25.7	43.1	0.047	0.202	0.029	0.023	0.001
PMR	35.7	7.8	19.9	0.050	0.272	0.051	0.094	0.002
DisentangledTMR, beta=0.2	75.8 +/- 1.1	18.1 +/- 1.8	87.1	0.030	0.113	0.016	0.029	0.0005

Compared with raw skeletons, RI falls by 57.3 percentage points, but pre-trained AR also falls by 13.3 points. Compared with PMR, pre-trained AR improves by 40.1 points while RI is 10.3 points higher, so the method is not best on every metric. It also does not achieve the lowest smoothness term: DMR's 0.023 is better than its 0.029. Strong overall geometric quality should not be rewritten as dominance on all physical metrics.

Ablation Study

The following genuine parameter analysis comes from Table 2, with single-run pre-trained SGN accuracies in percent. NTU120 and ETRI use earlier checkpoints without the complete output-supervision suite, so this is not a comparison of one uniform final-model configuration across all datasets.

beta	NTU60 AR (higher)	NTU60 RI (lower)	NTU120 AR (higher)	NTU120 RI (lower)	ETRI AR (higher)	ETRI RI (lower)
0.00, raw skeleton	89.1	75.4	88.2	67.6	90.8	45.6
0.10	82.7	52.0	85.3	62.4	90.0	43.0
0.20	76.8	17.3	81.4	39.4	87.8	35.7
0.30	42.8	14.3	73.5	22.8	84.0	26.0
0.50	27.7	3.1	47.1	9.9	67.3	10.4
1.00	2.0	4.7	2.6	0.9	3.7	1.0

The NTU60 beta=0.20 entries of 76.8/17.3 in Table 2 differ from 75.8/18.1 in Table 1. Although the caption says this row matches Table 1, the former is a single-run result and the latter a 5-run aggregate; this note preserves both instead of silently reconciling them. The prose additionally reports that moving beta from 0.20 to 0.25 changes pre-trained AR from 76.8% to 38.8%. Retrained AR at full retargeting is 55.4%, showing that retraining recovers some usability but does not fully preserve raw-data utility.

Section 4.5 provides only directional component-ablation findings: removing either MixFormer or temporal convolutions causes the largest utility drops; replacing the identity GCN with fixed bone lengths worsens privacy; feeding a full temporal sequence into the identity stream retains low RI but harms utility; and removing Stage-1 adversarial or orthogonality losses worsens the trade-off. The cache contains no complete numerical table for these configurations, so exact drops cannot be supplied.

The stage ablation reports approximately 82% pre-trained AR for configurations including Stage 3, versus 57.3% AR and 15.3% RI for Stage 1 alone. The approximately 82% value differs from the main table's 75.8%, and the prose does not provide enough settings to align them. It supports the local conclusion that joint fine-tuning matters, but not an exact improvement claim comparing the full curriculum with each isolated stage.

Key Findings

• Action learnability differs from compatibility with an existing recognizer: the recommended setting achieves 87.1% retrained AR and 75.8% pre-trained AR.
• Recommended-setting RI remains 18.1%, well above 2.5% chance. Empirical risk reduction does not establish that re-identification is impossible.
• NTU60 RI is not strictly monotonic in beta: it is 3.1% at beta=0.50 but rises to 4.7% at beta=1.00. The paper's smooth-decrease description needs this exception.

Highlights & Insights

• Information pathways become privacy design variables. The action stream retains time while the identity stream sees compressed static structure, providing clearer specialization than simply adding an adversary after a shared representation. Its effectiveness still requires attack-based validation.
• Decoded output is checked explicitly. Classification and decorrelation of latent representations do not guarantee anonymous joint sequences; Stage 3 places identity and distribution discriminators on output to address leakage reintroduced during decoding.
• Deployment constraints are part of the evaluation. Reporting both pre-trained and retrained AR reveals cases where action information exists but an old model cannot use it. This dual protocol is transferable to other privacy-preserving data-release tasks.

Limitations & Future Work

• Author-acknowledged: no formal privacy guarantee. Metric-learning, model-inversion, or stronger identity attacks may achieve higher RI. This is not a differential-privacy method and provides no composable privacy budget.
• Attack evidence has a restricted scope. The main table does not report retrained RI. The text says the supplement tests attackers trained directly on anonymized data under full retargeting, with results within +/-0.3 percentage points of the standard evaluator. The local cache does not contain that supplement, and the statement does not replace adaptive-attack testing at the recommended beta=0.2 setting.
• Reader observation: mean poses and decorrelation do not imply factor independence. Mean pose may retain action-dependent structure, and the action stream can access raw positions. In particular, contrastive and decorrelation objectives do not cover every possible nonlinear identity cue.
• Data coverage remains narrow. Evaluation is limited to one sensor family, 25 joints, and single-person actions. Transfer to OpenPose, COCO, or SMPL topologies is unproven. Table 2 is a within-dataset sweep on each benchmark, not an explicit experiment training on one domain and testing on an unseen domain.
• Resource and compatibility costs remain substantial. Training with batch size 128 requires approximately 40 GB of GPU memory, while pre-trained AR remains 13.3 points below raw data. Next steps include adaptive attacks at the recommended beta, cross-sensor evaluation, and domain adaptation, while checking whether recovered utility reintroduces identity leakage.

Related Work & Insights

• vs DMR / PMR (source reference 7, ICCV2025): These share a CNN encoder and decoder; DMR omits adversarial training and PMR adds privacy adversaries. The present method replaces this with complementary encoders and separate cross-attention. PMR has lower RI but much lower AR, so comparisons must retain both metrics.
• vs Skeleton MixFormer (source reference 35): MixFormer is a skeleton action-recognition backbone used here as one trainable action-encoder branch. The contribution is primarily the privacy-retargeting architecture and training combination, not the introduction of that backbone.
• vs FiLM and Barlow Twins: Broadcasting a single identity token resembles FiLM-style conditioning. The cross-stream correlation penalty borrows the redundancy-reduction idea from Barlow Twins, but does not reproduce its original same-view objective and is not a direct mutual-information estimator.
• vs direct perturbation and differential-private anonymization: Output forms, target-identity conditioning, and attack protocols differ, and the paper does not numerically rank them under a unified protocol. A useful direction is to align the released object and adversary capabilities before comparing utility and risk, instead of ranking RI alone.

Rating

• Novelty: 4/5. Asymmetric encoding and factorized decoding are well motivated for private retargeting, although their building blocks are largely established techniques.
• Experimental Thoroughness: 3/5. Three datasets, two utility protocols, and repeated runs are useful, but adaptive attacks at the recommended setting and numerical component-ablation evidence remain incomplete.
• Writing Quality: 3/5. The architecture and deployment problem are clear, but stage ablations, main results, single-run sweeps, and monotonicity claims need more careful alignment.
• Value: 4/5. Relevant to skeleton-data release with existing action recognizers, while practical deployment still requires validation against the intended adversary.