SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation¶

Conference: ECCV2026
arXiv: 2603.14152
Project Page: https://sk-adapter.github.io/
Code: To be confirmed
Area: 3D Vision
Keywords: 3D skeleton control, native 3D generation, Adapter fine-tuning, Graph Relative Position Encoding, skeleton-guided generation

TL;DR¶

SK-Adapter proposes 3D skeletons as a first-class control signal, injecting them into a frozen 3D flow matching generative backbone (Trellis) via a lightweight adapter network (GRPE topological encoding + skeletal cross-attention). This approach achieves precise structural control in native 3D space while preserving pre-trained generative priors, and is accompanied by the construction of Objaverse-TMS, a dataset containing 24k text-mesh-skeleton triplets.

Background & Motivation¶

Native 3D generative models (e.g., Trellis, Hunyuan3D) have recently made leaps in progress, enabling high-fidelity 3D asset generation from text or images within seconds. However, these models suffer from a key limitation: the inability to specify precise structural poses. Text prompts can describe "a standing dog" but fail to precisely convey specific joint-level topological constraints like "knees bent at 60 degrees" or "left front leg extended forward, right hind leg kicking back." Similarly, image prompts provide view-specific appearance cues rather than complete structural specifications. For animation and game pipelines, explicit structural control is a prerequisite for downstream workflows—generated assets must be re-riggable and animatable, yet current native 3D generation models lack this level of structural controllability.

In 2D generation, skeleton guidance is highly mature: ControlNet and T2I-Adapter achieve precise layout and pose control by injecting 2D human pose skeletons. Inspired by this, 3D works like SKDream attempt to project 3D skeletons onto a 2D plane via a "2D lifting" strategy to condition multi-view diffusion models, followed by reconstruction and UV refinement to obtain 3D assets. However, this approach faces a fundamental dimensional mismatch: depth information is completely lost after compressing 3D structures into 2D planes, and self-occlusion causes the model to misunderstand complex topological relationships. Furthermore, multi-stage reconstruction pipelines degrade texture quality and introduce geometric artifacts, drastically reducing the structural fidelity and visual quality of the final output.

The key insight of this paper is that precise structural control requires the control signal to be isomorphic to the generation space—skeletal information should be directly injected into native 3D space without passing through a lossy 2D projection bottleneck. However, adapting large-scale 3D Transformers to strict skeletal structure constraints without catastrophic forgetting remains a key challenge. Core Idea: Encode 3D skeletons as topology-aware sparse spatial tokens and inject them into a frozen Trellis generation backbone through a cross-attention layer in a lightweight adapter, achieving precise skeletal structure control with zero forgetting of pre-trained priors.

Method¶

Overall Architecture¶

SK-Adapter adopts the Sparse Flow Transformer of Trellis as a frozen backbone, inserting a lightweight adaptation module designed specifically for skeletal control into each of its DiT blocks. The input consists of three parts: a 3D skeleton \(\mathcal{S} = \{J, G\}\) (joint coordinates \(J \in \mathbb{R}^{N \times 3}\) + topological graph \(G\)), a text prompt, and Gaussian noise. The output is a 3D asset consistent with the skeletal structure and text semantics (SLAT latent → Mesh/Gaussian). The entire pipeline operates through two core modules: a GRPE topological encoder that parses the skeleton tree structure into structure-aware token sequences, followed by skeletal cross-attention within each backbone block to allow voxel features to dynamically attend to the most relevant skeletal joints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["3D Skeleton S={J,G}"] --> B["GRPE Topological Encoder<br/>Distance Encoding + Relation Encoding"]
    B --> C["Skeleton Feature Tokens<br/>(N x F)"]
    C --> D["Skeletal Cross-Attention<br/>(x24 backbone blocks)"]
    E["Trellis Voxel Features"] --> D
    D -->|Zero-init Projection + Residual| F["Modulated Voxel Features"]
    F --> G["Flow Matching Denoising → SLAT → Mesh"]
    H["Text Prompt"] --> I["CLIP Text Features"]
    I --> D

Key Designs¶

1. GRPE Topological Encoding: Encoding skeletal geometric positions and hierarchical topology into structure-aware tokens

Directly using joint coordinates as control signals is insufficient—two joints that are spatially close but topologically distant (e.g., left hand and left foot) should play vastly different roles in generation. SK-Adapter employs Graph Relative Position Encoding (GRPE) to quantize and encode the two dimensions of the skeletal tree structure into attention biases. The first dimension is topological distance \(D_{ij}\): the shortest path length between two joints, truncated to \(d_{\text{max}}=5\) for a total of 6 levels (self-loop / 1-hop / 2-hop / 3-hop / 4-hop / \(\ge 5\)-hop or disconnected). Three sets of embedding matrices (one for query, key, and value respectively) are learned to inject into the attention computation. The second dimension is edge relation \(R_{ij}\): six types of directed semantic relations, namely self-loop (with children), parent, child, sibling (same parent but different joints), distant relative (no direct kinematic relationship), and end-effector (leaf nodes like fingertips, toes, top of head). Among these, end-effectors are encoded separately rather than as general self-loops because they are the most commonly manipulated points by users in inverse kinematics, warranting separate structural priors. The relation matrix is asymmetric: if joint i is the parent of j, then \(R_{ij}=\)Parent while \(R_{ji}=\)Child, whereas sibling relationships are symmetric. After GRPE encoding, the skeleton embedding matrix \(\mathbf{f}_{\text{skel}}\) of size \(N \times F\) is obtained, with each joint token simultaneously carrying both its spatial coordinates and its structural identity within the full joint tree.

2. Skeletal Cross-Attention: Allowing voxel features to dynamically attend to the most relevant skeletal joints

Once skeletal tokens are obtained, the key challenge is injecting them into the 3D generation process. SK-Adapter achieves this by inserting a dedicated cross-attention layer following the standard self-attention in each DiT block of the frozen backbone. The Query for this layer is derived from the backbone's intermediate voxel features \(\mathbf{h}_{\text{base}}\), while the Key and Value come from the GRPE-encoded skeleton tokens \(\mathbf{f}_{\text{skel}}\). This enables each spatial voxel to dynamically focus on the joints that exert the strongest constraints on it: high attention weights focus on leg joints when generating legs, spine joints when generating the torso, and end-effectors when generating extremities. The output of the cross-attention passes through a zero-initialized linear projection layer \(\mathbf{W}_o\) and is added to the original voxel features via residual connection to form the final hidden state \(\mathbf{h}' = \mathbf{h} + \mathbf{W}_o \mathbf{f}_{\text{attn}}\). The elegance of zero-initialization is that at the beginning of training, the signal contributed by the adapter is zero, keeping the output distribution of the entire backbone identical to its frozen state, which guarantees zero loss of the generative priors. As training progresses, the projection layer gradually learns to modulate voxel latents based on skeletal constraints. During training, the entire Trellis backbone remains frozen, with only the GRPE encoder, cross-attention layers, and zero-initialized projection layers being trainable (approx. 151M parameters, which is a small fraction of the backbone), structurally preventing catastrophic forgetting.

3. Objaverse-TMS Dataset: Filling the gap in skeleton-text-mesh triplet data

Training skeleton-guided 3D generation requires aligning three modalities simultaneously: description (text), geometry (mesh), and structure (skeleton). Existing datasets either contain skeletons without text descriptions (Rig-XL, Articulation-XL) or have text descriptions with automatically generated skeletons (e.g., Objaverse-SK in SKDream, which suffers from anatomical inconsistency). SK-Adapter extracts data from the intersection of Anymate (expert-annotated skeletons) and CAP3D (text descriptions), yielding 24k high-quality text-mesh-skeleton triplets after filtering. Expert-annotated skeletons provide more logical joint locations and bone structures than automatically generated skeletons, allowing the model to learn more accurate joint topology priors. The dataset covers diverse poses across three categories: humanoid, animal, and other objects, providing a reliable foundation for training.

Loss & Training¶

Training follows the Latent Flow Matching paradigm. For a real 3D asset, its sparse latent \(\mathbf{z}_0\) is first obtained using the frozen voxel encoder. A linear interpolation path from Gaussian noise \(\mathbf{z}_1\) to \(\mathbf{z}_0\) is defined as \(\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\mathbf{z}_1\). The SK-Adapter-enhanced model \(v_\theta\) is trained to predict the velocity field that transforms noise toward the target conditioned on both skeleton and text:

\[ \mathcal{L}_{FM} = \mathbb{E}_{t,\mathbf{z}_0,\mathbf{z}_t} \| v_\theta(\mathbf{z}_t, t, \mathbf{c}_{\text{text}}, \mathbf{f}_{\text{skel}}) - \mathbf{u}_t(\mathbf{z}_0) \|^2 \]

The model is trained on Objaverse-TMS for 200 epochs, with a batch size of 16 and a learning rate of \(1\times 10^{-5}\). A 10% dropout is applied to the text condition to support classifier-free guidance, while no dropout is applied to the skeletal condition.

Key Experimental Results¶

Main Results¶

The test set, TMS-eval, consists of 140 instances (54 humanoid, 63 animal, 23 other objects). Structural alignment is measured by the ReRigging Score (Chamfer Distance between the resurrected skeleton after re-rigging the generated mesh and the conditioning skeleton, with a ground truth mesh oracle reference of 0.2073). Visual quality is evaluated using PickScore and KD-DINO.

Metric	SKDream	SpaceControl	SK-Adapter (Ours)
ReRigging Score ↓ (Overall)	0.2818	0.2740	0.2228
ReRigging ↓ (Humanoid)	0.2385	0.2282	0.1740
ReRigging ↓ (Animal)	0.2730	0.2898	0.2415
ReRigging ↓ (Other)	0.4075	0.3386	0.2861
CLIP Score ↑	25.65	25.66	26.16
PickScore ↑	20.46	20.55	21.01
KD-DINO ↓	1.3809	1.7821	0.7778
Generation Time	~40s	<15s	<15s

Ablation Study¶

Ablation results trained on an 8k subset for 200 epochs (the Full Model metrics are slightly lower than the main results of 0.2228 on the full dataset due to the reduced training set, but the relative relationships remain consistent).

Configuration	ReRigging ↓ (Overall)	CLIP ↑	PickScore ↑
Full Model	0.2355	26.11	20.94
w/o Dedicated Cross-Attention	0.5049	24.71	20.54
w/o Topological Encoding (Joint Coords Only)	0.2527	26.14	20.90

Key Findings¶

Dedicated cross-attention is the most crucial module for structure: removing it doubles the ReRigging Score to 0.5049, demonstrating that skeletal control requires a dedicated attention pathway isolated from traditional text/image conditions. Mixing them causes structural constraints to be overwhelmed by semantic signals.
Topological encoding delivers stable and consistent gains; removing it degrades ReRigging from 0.2355 to 0.2527, with more pronounced degradation observed in complex joint structures (e.g., multi-limbed animals).
Generative quality (PickScore / KD-DINO) is not sacrificed by introducing structural control—the adaptation strategy on the frozen backbone successfully preserves Trellis's high-frequency details and texture generation capabilities.
The performance is best on the humanoid category (ReRigging of 0.1740, which is close to the oracle of 0.2073), while the 'other' category is the most challenging due to high topological diversity.

Highlights & Insights¶

Zero-initialized cross-attention injection: The adapter output passes through a zero-initialized linear layer before being residually added to the backbone. At the start of training, the control signal is "empty," and the backbone distribution is completely undisturbed. This design enables stable training even for large-scale adapters with 151M parameters.
Delicate design of GRPE relation encoding: Rather than merely encoding distance, it distinguishes four directed relations (parent, child, sibling, and end-effector), enabling the model to learn distinct physical constraints such as "parent joint pulling child joint vs sibling joints coordinating balance"—a feat unreachable by coordinate position encoding alone.
Natural extension to editing capabilities: The decoupled nature of skeletal control enables region-level editing—replacing local skeletons and regenerating voxel latents under masked regions using inpainting strategies, allowing addition and re-posing operations without any fine-tuning.
Crucial role of training data: The comparison between expert-annotated skeletons and automatically generated skeletons (SKDream) is key—ablation results suggest that superior skeleton quality is one of the pillars supporting SK-Adapter's significant lead.

Limitations & Future Work¶

Generative quality is bounded by the capabilities of the Trellis base model; distortion still occurs in detailed regions such as faces and fine textures, which is an inherent ceiling of the adapter paradigm.
When the input skeleton is overly complex (e.g., dense intersecting finger topology), structural guidance becomes ambiguous, dropping local geometric quality. The truncation distance \(d_{\text{max}}=5\) in GRPE may limit the expression of ultra-long-distance joint pairs.
Objaverse-TMS contains only 24k samples and is biased towards humanoids and animals, leaving its generalization capability on abstract topologies (mechanical devices, architectural structures) questionable.
Future directions: Integrating higher-resolution 3D foundation models, stronger text encoders, and training on larger-scale data.

vs SKDream (CVPR 2025): SKDream projects 3D skeletons to 2D to condition multi-view diffusion, followed by 3D reconstruction, suffering from spatial ambiguity and multi-view inconsistency. SK-Adapter directly injects skeleton tokens into native 3D space, reducing the structural alignment ReRigging Score from 0.2818 to 0.2228, while accelerating generation by 2.5x.
vs SpaceControl: SpaceControl is a training-free spatial guidance method on Trellis that converts skeletons to voxel grids and injects them during inference. This training-free paradigm limits its structural alignment accuracy (0.2740), and over-constraint can lead to topological tearing. SK-Adapter achieves more precise and robust control via a learned adapter.
vs ControlNet / T2I-Adapter: An extension of the 2D adapter paradigm into 3D voxel space—the core difference being that skeleton tokens replace 2D pose images, and cross-attention injects features into the frozen 3D Transformer rather than a 2D UNet.

Rating¶

Novelty: ⭐⭐⭐⭐☆ First to introduce a skeleton adapter into native 3D generation voxel space, with a simple and effective GRPE topological encoding + cross-attention adapter design.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Quantitative structural alignment and visual quality metrics are comprehensive, with ablation studies clearly validating the contributions of each component; editing capabilities are only qualitatively shown, lacking quantitative evaluation.
Writing Quality: ⭐⭐⭐⭐⭐ Clear chain of motivation, with step-by-step reasoning on the dimensional mismatch from 2D to 3D, followed by complete method descriptions and precise mathematics.
Value: ⭐⭐⭐⭐⭐ Fills a crucial gap in structural control for native 3D generation, providing a simple and highly efficient benchmark solution for controllable 3D generation via the adapter paradigm.