GKDT: General Keypoint Detection Transformer¶

Conference: ECCV 2026
arXiv: 2607.00752
Code: https://github.com/AlanLuSun/General-Keypoint-Detection
Area: Multimodal VLM
Keywords: General Keypoint Detection, Multimodal Prompting, DINOv3, Kernel Generation Transformer, Large-scale Dataset

TL;DR¶

This paper constructs a large-scale unified keypoint dataset, MegaKPT, containing 1.3 million instances, and proposes GKDT, a General Keypoint Detection Transformer based on DINOv3. Through a Kernel Generation mechanism, visual/textual keypoint prototypes are dynamically converted into convolutional kernels. GKDT achieves 90%+ [email protected] accuracy on most categories across 22 test sets using a single model.

Background & Motivation¶

Keypoint detection is a foundational task in computer vision, providing fine-grained part understanding for applications such as human pose estimation, animal behavior analysis, and robotic manipulation. Over the past decades, this field has evolved from traditional methods like Harris corners and SIFT to the deep learning era based on CNNs and Transformers, yielding high-precision human pose models like ViTPose++ and HRNet, as well as domain-specific animal models like AP-10K and Animal Kingdom. However, these specialized models are essentially closed-set detectors—they can only recognize specific keypoint classes defined during training (e.g., 17 human skeletal keypoints). Once they encounter objects with entirely different anatomical structures, such as fish or sofas, their generalization capability drops to zero. The field of keypoint detection has long lacked a general model capable of working across categories and domains.

Recent research has attempted to break this limitation through prompt learning, enabling models to detect keypoints of arbitrary objects based on a few annotated images (visual prompts) or textual descriptions of keypoints (textual prompts). Works like OpenKD and X-Pose have demonstrated the viability of multimodal prompts, but their training datasets are limited in scale (UniKPT contains only about 418k instances), which restricts cross-domain knowledge transfer and generalization to unseen categories. A deeper contradiction is that supporting true general keypoint detection requires both a large-scale, high-quality, and unified keypoint dataset to cover an extensive span of categories from insects to furniture and from humans to medical imaging, and a detection model that can flexibly handle visual and textual prompts while training efficiently on such large-scale and heterogeneous data.

This paper presents systematic solutions from both data and model perspectives. Regarding data, the authors manually unified the annotation formats of 29 existing datasets and supplemented them with textual descriptions, constructing the MegaKPT dataset that spans 10 supercategories and covers over 1.3 million instances. Regarding the model, using DINOv3 as the visual backbone, they designed the Kernel Generation Transformer to dynamically transform multimodal keypoint prototypes into detection convolutional kernels. Core Idea: Leverage the strong representation capabilities of DINOv3 self-supervised pre-training to dynamically convert visual and textual keypoint prompts into convolutional detection kernels via a Kernel Generation mechanism. By training a single model on the large-scale unified MegaKPT dataset, general keypoint detection is achieved across 22 test sets (including unseen categories).

Method¶

Overall Architecture¶

GKDT is a prompt-based general keypoint detection model that supports visual prompts (one or several annotated support images), textual prompts (direct descriptions of keypoint names, e.g., "left eye", "right front wrist"), or both simultaneously. The model uses DINOv3 Visual Transformer as the backbone and performs full-parameter fine-tuning to unleash its self-supervised representation capability; the text branch leverages a frozen DINOv3 text encoder combined with a lightweight text adapter. The core pipeline is as follows: first, visual keypoint prototypes (VKP) and textual keypoint prototypes (TKP) are extracted from visual and textual prompts, respectively; then, these prototypes are input as tokens into the Kernel Generation Transformer for cross-modal interaction and query context aggregation, eventually generating a set of convolutional kernels; finally, these kernels are dynamically convolved channel-wise with the upsampled query feature map to produce the keypoint heatmaps.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Query Image"] --> B["DINOv3 Visual Backbone<br/>Full parameter tuning to extract dense features"]
    B --> C["Feature Map Upsampling + L2 Normalization"]
    C --> D["Channel-wise Convolution with Kernels<br/>→ Keypoint Heatmap"]

    E["Support Image +<br/>Keypoint Annotation"] --> F["Extract Visual Keypoint<br/>Prototype VKP"]
    F --> G["Kernel Generation<br/>Transformer"]

    H["Keypoint Text Description"] --> I["DINOv3 Text Encoder<br/>(Frozen) + Text Adapter"]
    I --> J["Textual Keypoint Prototype TKP"]
    J --> G

    G --> K["Generate Convolutional Kernels<br/>+ K2I Channel Projection"]
    K --> D

    D --> L["Multimodal Fusion<br/>Output Final Heatmap"]

Key Designs¶

1. Multimodal Keypoint Prototype Construction: Extracting Unified Keypoint Representations from Visual and Textual Prompts

For visual prompts, the model encodes each annotated keypoint coordinate on the support image into a Gaussian heatmap (with standard deviation \(\sigma\) controlling the spread). It then performs a linearly weighted summation over the spatial dimensions of the DINOv3 visual feature map using the heatmap weights, yielding the visual embedding of the keypoint. When \(K\) support images are provided, the \(K\) visual embeddings for the same keypoint type are averaged to obtain the Visual Keypoint Prototype (VKP). This averaging operation significantly reduces noise caused by annotation errors in individual images. For textual prompts, each keypoint text is processed by a frozen DINOv3 text encoder and a trainable text adapter, and its classification token is taken as the Textual Keypoint Prototype (TKP). Finally, a T2I projector aligns the channel dimension of the text features with the visual features, ensuring both prototypes can share the subsequent Transformer processing. The core advantage of this construction is that Gaussian heatmap weighting focuses the model on semantic areas around keypoints rather than relying on precise single-point coordinates, providing a unified and robust representation base for diverse keypoints across various objects.

2. Kernel Generation Transformer: Cross-Modal Interaction and Adaptive Kernel Generation

This is the most critical design of this paper. The VKP and TKP are concatenated into a token sequence and input to the KG Transformer. The internal mechanism of this module consists of two layers: first, self-attention allows VKPs and TKPs to thoroughly exchange information—when the prompt of one modality is weak (e.g., vague textual description or poor visual support image perspective), the strong features of the other modality can propagate via attention to help improve the representation of the weak modality, which is a direct manifestation of multimodal fusion at the representation level; second, cross-attention enables prototype tokens to aggregate feature contexts from the query image, making the generated convolutional kernels not only contain the prior semantic information of the keypoints but also sense the specific pose, scale, and appearance of objects in the query image. After multiple layers of self-attention and cross-attention, the Kernel Expander upsamples the prototype tokens layer-by-layer via deconvolution into spatial convolutional kernels (supporting sizes like \(1\times1\), \(3\times3\), \(5\times5\)). Compared to traditional methods that learn fixed detection parameters for each keypoint (such as Conv1x1 weights in ViTPose++), the convolutional kernels generated by the KG Transformer are dynamic—they are generated specifically for each query image and each keypoint class in that scene, granting the model flexibility to handle extremely diverse objects from cars to flies.

3. Non-parametric Detection and Multimodal Heatmap Fusion: Convolutional Kernels as Direct Keypoint Detectors

The generated visual convolutional kernel \(W^v\) and textual convolutional kernel \(W^t\) are convolved channel-wise with the upsampled L2-normalized query feature map, producing two independent keypoint heatmaps \(H^v\) and \(H^t\). In the evaluation phase, the two heatmaps are fused via a weighted mechanism guided by validity indicators \(\mathbb{I}^v\) and \(\mathbb{I}^t\)—a heatmap is included in the fusion only when its corresponding modality prompt is valid. If both prompts are valid, their average is taken as the final output. The brilliance of this design is that it seamlessly supports three inference modes: visual-only, text-only, and multimodal prompts. Furthermore, multimodal fusion bypasses the risk of users choosing the wrong prompt modality—experiments show that for 21 out of 22 test sets, the multimodal results differ from the optimal single-modality results by less than 2%, meaning that even if users are unsure which prompt is better, the multimodal mode consistently approaches the optimal performance.

Loss & Training¶

Mix-Modal Prompted Training: In real-world usage, users can freely choose visual, textual, or multimodal prompts. However, if the model is trained only on multimodal data, its single-modality performance significantly degrades during testing. To address this, each training episode randomly selects a prompt mode from \(\{\text{visual}, \text{text}, \text{multimodal}\}\) and masks the keypoint validity indicators accordingly. Experiments show that training with this three-mode mixture achieves the best performance across all inference styles, whereas training with only \(\{\text{multimodal}\}\) severely harms single-modality performance, and training with only \(\{\text{visual}, \text{text}\}\) degrades multimodal performance.

Dynamic Importance Sampling: MegaKPT exhibits a severe long-tail distribution across its 10 supercategories—the clothing category (DeepFashion2) has nearly 500k instances, while the medical category has only around a thousand. Simple importance sampling would quickly exhaust tail-class samples, causing catastrophic forgetting, whereas uniform sampling would lead to underfitting on tail classes. The core mechanism of the proposed dynamic importance sampling is as follows: based on the remaining sample count of each supercategory, it dynamically determines whether the currently sampled category belongs to the "head" (categories in the top 50% of sample counts). If so, the sample is removed from the data pool; otherwise, it is put back. When a supercategory is completely emptied, it is deleted from the data pool. As iterations progress, head-class samples are gradually depleted, naturally increasing the relative proportion of tail classes and dynamically balancing the overall distribution. Experiments show that this strategy yields significant performance gains on tail classes (+1.03% on HInt and +0.40% on Cephalometric) with almost no loss in head-class performance.

The training loss is the mean squared error (MSE) between the predicted heatmaps and the ground-truth Gaussian heatmaps. Losses are calculated separately for the visual and textual outputs and then weighted by the validity indicators: only when a modality prompt is valid is the corresponding heatmap included in the loss calculation. Invalid keypoints are treated as negative samples (with a target of an all-zero heatmap).

Key Experimental Results¶

Main Results¶

Below are the [email protected] comparison results in single-object scenarios (for representative datasets; unseen categories are marked with ▲):

Prompt Mode	Dataset	Ours	OpenKD	X-Pose	Gain
Visual	Animal Pose (△)	81.11	49.41	-	+31.70
Visual	CUB (▲)	97.71	85.51	-	+12.20
Visual	Hand X-ray (▲)	99.28	68.67	-	+30.61
Text	CUB (▲)	98.58	89.82	67.07	+8.76
Text	AwA (△)	92.80	81.80	28.57	+11.00
Text	Hand X-ray (▲)	99.62	82.24	2.41	+17.38
Multimodal	NABird (▲)	96.86	91.81	-	+5.05
Multimodal	HInt	81.13	50.94	-	+30.19
Multimodal	Cephalometric	99.49	99.18	-	+0.31
Multimodal	Hand X-ray (▲)	99.60	85.31	-	+14.29

COCO Human Pose Estimation AP results in multi-object scenarios:

Model	Detector	AP	AP_50	AP_75
GKDT-H† (Fine-tuned)	G-DINO	78.1	91.7	85.1
GKDT-H	G-DINO	77.2	91.6	84.3
GKDT	G-DINO	75.0	90.8	82.3
X-Pose	End-to-End	71.8	88.9	78.3
ViTPose++-H (Expert Model)	Faster R-CNN	79.1	91.6	85.7

Ablation Study¶

Configuration	Animal(visual)	Animal(text)	CUB(visual)	CUB(text)
Full GKDT	81.11	73.27	97.71	98.58
Frozen Backbone + No KG	53.86	54.00	79.55	87.11
Fine-tuned Backbone + No KG	79.08	68.29	96.78	96.81
KG w/o Cross-Attention (Self-Attention Only)	80.23	73.15	-	-
KG w/o Self-Attention (Cross-Attention Only)	65.90	82.94	-	-
Uniform Sampling \(\rightarrow\) Dynamic Importance Sampling	Head classes +0.05	Head classes -0.05	Tail class HInt +1.03	Tail class Cephalo +0.40

Key Findings¶

KG Transformer is the core driver of performance improvement: Completely removing KG leads to a substantial drop in text-prompted performance (CUB decreases from 98.58 to 96.81). Self-attention (SA) contributes far more to visual prompting than cross-attention (CA), while CA slightly benefits textual prompting.
Fine-tuning the DINOv3 visual backbone is critical: Completely freezing the backbone severely degrades performance across all modes (e.g., Animal-visual drops from 81.11 to 53.86), demonstrating that general keypoint detection requires thorough feature adaptation.
Dynamic Importance Sampling significantly improves tail-class performance (up to +6.36% on OpenKD) while keeping head-class performance virtually unaffected, validating the efficacy of the "depletion-based removal of head classes" strategy.
Multimodal fusion effectively mitigates the risk of weak single-modality prompts: In 21 out of 22 test sets, the difference between multimodal results and the optimal single-modality results is \(<2\%\), and \(<1\%\) in 18 out of 22.
Even with only 10% of the training data, the model still achieves 90%+ accuracy on Cephalometric, reflecting the robust knowledge transfer capabilities of DINOv3 pre-training.

Highlights & Insights¶

The design philosophy of Kernel Generation is elegant: By framing keypoint detection as a response of a set of convolutional kernels on a feature map, the prompt learning objective is reformulated as dynamic kernel generation. This avoids rigid, category-specific head designs, allowing the model to handle 740 different keypoint classes.
Self-attention enables mutual enhancement between visual and textual prototypes: When a visual support image is low-quality or a textual description is incomplete, representation of the weak modality is augmented via attention propagation from the stronger modality. This implicit cross-modal enhancement is far more elegant than explicit modal fusion.
Dynamic Importance Sampling is an efficient engineering remedy for long-tailed distributions: Without relying on complex resampling distributions or modified loss formulations, this "consuming head classes while retaining tail classes" mechanism balances the distribution naturally, remaining simple to implement yet highly effective.
The quality control efforts in MegaKPT are exemplary: Manually unifying annotation formats from 29 distinct sources, standardizing category names via LLMs, and providing expert-labeled keypoint texts for medical imaging—these meticulous data engineering endeavors establish a solid infrastructure for future general keypoint detection (GKD) research.

Limitations & Future Work¶

Large model size: GKDT-L possesses 349M learnable parameters, and GKDT-H reaches 898M, posing deployment challenges on edge devices or real-time scenarios. Exploring knowledge distillation or structural pruning represents a valuable future direction.
High pipeline complexity of two-stage inference (object detection followed by keypoint detection): Although flexible, it introduces system overhead. While the end-to-end competitor X-Pose is more efficient, GKDT's decoupled strategy of "freely deploying better detectors" is academically sound but complicates the engineering pipeline.
Heavy reliance on the DINOv3 feature space: Whether these designs smoothly transfer to subsequent, stronger vision foundation models remains to be verified, although the KG Transformer, as a generic component, is theoretically decoupled from the backbone.
Substantial room for improvement in cross-supercategory transfer (e.g., from humans to quadrupeds): The drastic drop from COCO (92.82%) to AP-10K (58.87%) indicates that anatomical structural differences remain a primary challenge in GKD.
Lack of systematic analysis on challenging scenarios: Robustness under occlusion, extreme poses, and small objects is not fully discussed, though these are unavoidable hurdles in actual deployments.

vs OpenKD [Lu et al. 2024]: OpenKD pioneered multimodal prompted keypoint detection, but its dataset UniKPT contains only 418k instances, and its model architecture is relatively simple. GKDT achieves comprehensive performance superiority using MegaKPT (3x dataset scale) and the DINOv3 + KG Transformer paradigm, demonstrating the combined benefits of scale and a strong backbone.
vs X-Pose [Yang et al. 2024]: X-Pose unifies object and keypoint detection in an end-to-end framework, but the dual tasks interfere with each other, limiting keypoint accuracy. GKDT adopts a two-stage approach to flexibly pair with superior object detectors (like G-DINO), lifting the COCO AP from 71.8 to 77.2 and verifying the design choice of decoupled detection and keypoint estimation.
vs ViTPose++ [Xu et al. 2023]: ViTPose++ is an expert model for human pose estimation, scoring 79.1 AP on COCO but limited to 17 predefined human keypoints. As a general model, GKDT achieves 78.2 AP (with GT boxes) on COCO, approaching expert-level performance while simultaneously supporting keypoint detection for hundreds of different object categories.

Rating¶

Novelty: ⭐⭐⭐⭐ Applying DINOv3 + Kernel Generation to general keypoint detection is pioneering; the curation of the MegaKPT dataset represents a significant contribution to the community.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive experiments across 22 test sets, 3 prompting modes, multi-object scenarios, and extensive ablation studies.
Writing Quality: ⭐⭐⭐⭐ Clear methodological descriptions and strong motivations. However, formula density is slightly high, and certain symbol definitions could be more intuitive.
Value: ⭐⭐⭐⭐⭐ The MegaKPT dataset and GKDT model provide both data and model baselines for general keypoint detection, with broad application potential in robotics, medical imaging, and animal behavior analysis.