DETRPose: Real-Time End-to-End Multi-Person Pose Estimation via Modified Transformer Decoder and Novel Denoising Keypoints¶
Conference: ECCV2026
Paper: ECCV
Code: https://github.com/SebastianJanampa/DETRPose
Area: Human Understanding / Multi-Person 2D Pose Estimation
Keywords: multi-person pose estimation, real-time inference, DETR, denoising training, keypoint similarity
TL;DR¶
DETRPose ports D-FINE's real-time detection architecture to multi-person 2D pose estimation, reworking the GroupPose decoder with a keypoint-similarity (KS, i.e. COCO OKS) driven denoising keypoint strategy, a KSVF loss, and a lightweight Pose-LQE head, yielding the first end-to-end real-time transformer pose model: 73.3 AP for DETRPose-X on COCO val, and 71.2 AP / 4.66 ms for DETRPose-L on test-dev, an order of magnitude faster than GroupPose-R50 with 37% fewer parameters.
Background & Motivation¶
Multi-person 2D pose estimation locates 17 body joints for every person in an image, and it serves as the preprocessing stage of activity recognition, 2D-to-3D human pose estimation, and VR β so latency matters as much as accuracy. The methods that currently run fast fall into two families. Top-down methods detect each person first, crop the region, and then estimate the pose inside it; they are usually more accurate, but their inference time grows linearly with the number of people. Single-stage methods (YOLOv8-Pose, YOLO11-Pose, RTMO) produce people and keypoints in a single forward pass and offer a good speed-accuracy trade-off, at the cost of relying on non-maximum suppression (NMS) at inference β and since NMS cost fluctuates with the number of detected boxes, its latency is unstable.
DETR-based methods (PETR, QueryPose, ED-Pose, GroupPose) replace NMS with queries plus Hungarian matching, giving constant latency and strong accuracy, yet all of them rank accuracy above speed: on COCO test-dev, ED-Pose-Swin-L takes 94.11 ms and GroupPose-Swin-L 96.77 ms, whereas RTMO-L needs only 4.89 ms β nearly a twenty-fold gap. Meanwhile object detection already has real-time DETR detectors such as RT-DETR, D-FINE, and DEIM, while pose estimation still has none, which keeps transformer pose models out of real-time applications. The bottleneck in GroupPose's decoder is group self-attention: it must transpose tensors repeatedly to alternate between the "keypoints within one person" level and the "same joint type across people" level, breaking memory contiguity β an overhead invisible in FLOPs but very real in wall-clock time.
This paper approaches the problem as one of adapting a real-time DETR architecture to pose rather than as one of stacking yet another stronger decoder: the backbone and encoder are taken from D-FINE and initialized with its Object365 weights, and the decoder removes projection layers from GroupPose while integrating D-FINE's pre-pose and FDR refinement layers. More importantly, it moves two training-time techniques from boxes to keypoints. The first is denoising: DN-DETR/DINO construct positive and negative queries by shifting and scaling ground-truth boxes to stabilize Hungarian matching, but a keypoint is a single point with no width or height, so there is no natural scale for how far it should be displaced; ED-Pose therefore applies denoising only in its detection layers and discards the denoising queries before the keypoint layers, while GroupPose does no denoising at all. The second is classification score: a pose model has no "objectness" to lean on and needs a new quality measure. Core idea: treat COCO's keypoint similarity (KS/OKS) as a single ruler running through the whole method β use it to invert how far each denoising query should be displaced, use it as the soft label of a KSVF classification loss, and add a lightweight Pose-LQE head that folds image-content evidence into the classification score, which together let the number of instance queries drop from 100 to 60 and directly cut the decoder computation that every keypoint pays for.
Method¶
Overall Architecture¶
DETRPose takes an image as input and outputs the 17 keypoints of every person together with their classification scores, in a single forward pass with no NMS and no post-hoc grouping. Architecturally it is D-FINE's three-part stack β a hierarchical HGNetv2 backbone, a hybrid encoder, and a transformer decoder β with the backbone and encoder reused as-is and initialized from D-FINE's Object365 pretrained weights. What is actually rewritten is the decoder: it keeps GroupPose's group self-attention (the reason no dedicated detection decoder layers are needed to produce people and keypoints together), removes the in-layer projection layers, integrates D-FINE's pre-pose and FDR refinement heads, and attaches a Pose-LQE classification head in every layer.
Queries are built as follows. A linear layer reads the encoder feature maps and selects the top-N pixels most likely to contain a person; for each selected pixel another linear layer generates K keypoints, and the mean of those K keypoints becomes the instance query for that person. The decoder therefore receives NΒ·(K+1) queries in total, and because every keypoint is its own query, this is exactly why reducing the number of predicted people later translates directly into FLOPs savings. Each decoder layer performs three operations in sequence: within-instance self-attention among the keypoints of one person, across-instance self-attention among the same joint type across people (where an attention mask keeps the denoising negative, denoising positive, and prediction groups isolated from one another), and deformable attention that lets all queries sample the encoder feature maps. During training, an additional set of denoising keypoint queries sampled through KS is injected and supervised by the KSVF loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input image"] --> B["HGNetv2 backbone<br/>+ hybrid encoder"]
B --> C["Query selection<br/>top-N pixels β K keypoints"]
G["Denoising keypoints<br/>sample Β± queries via KS"] -->|injected at training| D["Modified GroupPose decoder<br/>group self-attention + FDR"]
C --> D
D --> E["Pose-LQE head"]
E --> F["Instances + keypoints + scores"]
D -->|training supervision| H["KSVF loss<br/>OKS as soft label"]
Key Designs¶
1. Denoising keypoint strategy: inverting how far a keypoint should be displaced via KS
Hungarian matching in DETR is notoriously unstable early in training, and DN-DETR/DINO solve this by feeding the decoder a batch of "noised ground-truth queries", teaching the model to accept queries close to the ground truth and reject ones far from it, which provides a stable gradient signal for matching. Every existing denoising technique, however, operates on boxes β a box has four corners, and shifting or scaling it by the ground-truth width and height yields both positive and negative samples, with the scale given by the box itself. A pose instance is a single keypoint: it has no width or height, and neither the direction nor the magnitude of the displacement is defined. Worse, the spatial tolerance of the 17 joint types differs enormously β an eye off by 10 pixels is a severe error, a hip off by 10 pixels is irrelevant. ED-Pose therefore applies DINO-style denoising boxes only in its detection layers and discards the denoising queries before the keypoint layers, while GroupPose dispenses with denoising entirely.
This paper makes the tolerance explicit by borrowing COCO's OKS measure, defining the per-keypoint similarity as
where \(d\) is the distance between a sampled keypoint and the ground-truth keypoint, \(\kappa>0\) is a per-joint fall-off constant (taken from the official COCO and CrowdPose evaluation code: small for eyes, large for hips), and \(s^{2}\) is the person's segmented area with \(0\le s^{2}\le HW\). Writing the sampled point as a displacement along a random unit vector \(\hat n\), i.e. \(\hat p=p+\alpha_{pose}\hat n\), and noting that \(d(p,\hat p)=\alpha_{pose}\), substituting back gives the displacement magnitude
Query construction thus becomes a two-step procedure of first sampling a similarity and then inverting it into a distance: positive queries sample KS from \(U(0.5,1)\), negative queries from \(U(0.1,0.5)\), the direction \(\hat n\) is uniform on the unit circle, and training uses a fixed 100 positive plus 100 negative queries. The advantage of this sampling scheme is that one set of hyper-parameters rescales automatically across joints and body sizes: for a larger instance \(s\) is larger, so the same KS value maps to a larger absolute displacement, while for joints with a small \(\kappa\) even a tiny offset already drops KS. In other words, the perturbation scale of a denoising query is not an arbitrary hyper-parameter but a reuse of the tolerance definition of the evaluation metric itself, which is why it is more stable than manually fixing a pixel offset. Denoising and prediction groups are separated by an attention mask: denoising groups may attend to the prediction group, but not vice versa, and denoising groups are masked from each other, so noisy queries never contaminate the real inference path. Since CrowdPose has no mask annotations, the authors approximate \(s^{2}=A_{box}\cdot 0.53\), where 0.53 comes from CrowdPose's evaluation code.
2. Modified GroupPose decoder: dropping projection layers and refining keypoints with FDR
GroupPose's decoder has two sources of latency. The first is structural to group self-attention itself: it transposes tensors back and forth between the keypoint level and the instance level, breaking memory contiguity, so even when FLOPs are close to the YOLO family the wall-clock latency stays higher β the reason the authors give in the conclusion for DETRPose remaining slower than YOLO models with comparable FLOPs. The second is the in-layer projection layers that shuttle features between the two levels; they add no new expressive power and merely maintain GroupPose's original tensor-shape conventions.
DETRPose's recipe is subtraction plus a head swap: remove those projection layers entirely, then integrate two D-FINE components. The pre-pose layer produces a coarse initial keypoint prediction before refinement, serving as the starting point for subsequent refinement; the FDR (Fine-Grained Distribution Refinement) layer changes keypoint coordinate regression from "regress a point directly" to "regress a distribution over offsets and refine it layer by layer" β judging from the caption of Fig. 3, the FDR heads emit vertical and horizontal offsets per keypoint. The effect is not only accuracy: a distributional representation means each layer only needs to make small incremental corrections, so shallow decoders remain usable, which explains why cutting the decoder from 6 layers to 4 in the later Table 8 costs only 1.2 AP while reducing latency by 13.5%. β οΈ The paper does not spell out the exact parameterization of FDR for keypoints (number of distribution bins, the precise refinement operator); the description here follows D-FINE's original design and the Fig. 3 caption, and the details should be checked against the paper or the official code.
3. Pose-LQE head: pulling content features at the predicted keypoints into the classification score
DETR detectors can lean on objectness or localization quality estimation (LQE, as in GFLv2 and the equivalent layer in D-FINE's decoder) to judge whether a query is trustworthy, but a pose model has no such handle: it does not predict boxes, it predicts a set of points, and how much a set of points "looks like a person" cannot be read off the regression outputs directly. Classification scores are therefore prone to inflation, and spuriously high-scoring queries both pollute Hungarian matching and crowd out genuine instances at inference.
Pose-LQE's approach is to gather content evidence at the keypoints. On the encoder's highest-spatial-resolution feature map, it samples a feature at each predicted keypoint location to obtain a content vector; rather than using the whole vector, it keeps only the top-k channels with the strongest response β letting the network itself pick the few channels most discriminative for that joint type β and passes them through a linear layer whose output is added to the conventional classification branch to produce the final score. The score thus sees three things at once: the query itself, the keypoint locations it predicted, and the evidence that something actually exists at those locations in the image. The design is deliberately kept extremely light β sampling only on a single highest-resolution feature map and keeping only top-k channels β so it barely adds latency (in the ablation, adding it moves latency from 4.58 ms to 4.66 ms). This matters because the paper's whole premise is real-time inference, and any component that buys accuracy with latency is unacceptable here.
4. Keypoint Similarity VariFocal loss: making the classification score mean "how accurate this pose is"
Because it predicts no bounding boxes, DETRPose cannot use an IoU-based localization-quality loss, and the soft label of the Varifocal loss used by RT-DETR/D-FINE is precisely the IoU. The authors replace it with the OKS: the classification target is no longer 0/1 but the keypoint similarity between the prediction and the ground truth, giving
where \(q\) is the OKS between prediction and ground truth, \(\tilde c\) is the predicted classification score, and \(\alpha=0.25\), \(\gamma=2.0\) control the weight of low-similarity cases. The positive term is weighted by \(q\) itself: a pose that matches the ground truth only halfway carries a classification target of only 0.5, so the model is pulled toward aligning "score" with "how accurate the position is" rather than pushing every matched query to 1; the negative term damps the gradient of already low-scoring negatives through \(\tilde c^{\gamma}\), preventing the flood of background queries from dominating training. The most direct benefit is a sharp reduction of false positives β and a useful side effect of scores no longer being inflated is that fewer people can be requested: the number of instance queries drops from 100 to 60, and since every keypoint is a query, this multiplies straight into decoder FLOPs (in the ablation, FLOPs fall from 121.6 G to 107.1 G and latency from 5.24 ms to 4.58 ms). The chain "loss quality β reliable classification scores β query count β computation" is the most reusable element of this paper.
A Worked Example¶
A concrete sampling of denoising keypoints: suppose a person has segmented area \(s^{2}=20000\) pixels (\(s\approx141.4\)), and take \(\kappa=0.1\) for ease of mental arithmetic (β οΈ real \(\kappa\) values come from the official COCO/CrowdPose per-joint constants, not 0.1; it is used here only to illustrate the magnitude). Then:
- drawing \(KS=0.9\) gives \(\alpha_{pose}=141.4\times0.1\times\sqrt{-2\ln 0.9}\approx 6.5\) pixels β a positive query, with the sampled point within 6.5 pixels of the ground truth;
- drawing \(KS=0.2\) gives \(\alpha_{pose}=141.4\times0.1\times\sqrt{-2\ln 0.2}\approx 25.4\) pixels β a negative query, pushed 25 pixels away from the ground truth;
- \(KS=0.5\) is exactly the positive/negative boundary, corresponding to a displacement of \(\approx16.6\) pixels.
The direction \(\hat n\) is drawn uniformly on the unit circle, so the same wrist joint is sampled into positive and negative queries in all directions and at all distances during training; for a larger person, the same set of KS values scales the displacement up proportionally, while for a joint with smaller \(\kappa\) (an eye, say) the displacement corresponding to the same KS threshold shrinks noticeably. That is the intuitive meaning of "using the metric as a ruler".
Loss & Training¶
Training uses AdamW with batch size 16 on 4 Tesla V100 GPUs. Data augmentation follows the DEIM protocol β horizontal flipping, color jittering, mosaic composition, mix-up, and multi-scale sampling β but without DEIM's cosine annealing schedule. The backbone and encoder follow RTMO's transfer-learning strategy and are initialized with D-FINE's Object365 pretrained weights. Hyper-parameters for the four variants are: 3/4/6/6 decoder layers, hidden dimension 256/256/256/384, backbone HGNetv2-B0/B2/B4/B5, 96/60/48/48 training epochs, a global learning rate of 1e-4 (backbone learning rates 1e-4/1e-5/1e-5/5e-5), and a fixed 60 detected people. Matching still uses Hungarian matching, but the matching cost contains no boxes and can only be built from keypoint-side similarity and coordinate regression error (β οΈ the paper does not detail the exact composition of the matching cost; refer to the paper or the code). At inference, inputs are resized to 640Γ640, and latency is measured with TensorRT FP16 on an RTX A5500 using Roboflow's benchmarking tool, which reports the median rather than the mean β an important detail, since latency fluctuation of NMS-based methods is exactly what this paper sets out to avoid.
Key Experimental Results¶
Main Results¶
Comparison on COCO (all latencies are TensorRT FP16 on an RTX A5500; entries marked * use an 800Γ1300 input size, so their latency is not directly comparable to the 640Γ640 models):
| Model | Backbone | Params | Latency (ms) | FLOPs (G) | AP (val) | AP (test-dev) |
|---|---|---|---|---|---|---|
| YOLOv8-X-Pose | - | 69.4M | 5.23 | 263 | 67.3 | - |
| YOLO11-X-Pose | - | 58.8M | 4.93 | 203 | 67.2 | - |
| RTMO-L | CSPDarknet | 44.7M | 4.89 | 68 | 72.4 | 71.6 |
| ED-Pose-R50* | ResNet-50 | 48.0M | 68.10 | 591 | 71.6 | 69.8 |
| GroupPose-R50* | ResNet-50 | 52.4M | 70.44 | 614 | 71.5 | 70.2 |
| ED-Pose-Swin-L* | Swin-L | 218.4M | 94.11 | 1954 | 74.2 | 72.7 |
| GroupPose-Swin-L* | Swin-L | 219.6M | 96.77 | 1977 | 68.1 | 74.8 |
| DETRPose-S | HGNetv2-B0 | 11.5M | 2.39 | 33 | 67.0 | 66.5 |
| DETRPose-M | HGNetv2-B2 | 20.8M | 3.47 | 67 | 69.4 | 68.5 |
| DETRPose-L | HGNetv2-B4 | 32.8M | 4.66 | 107 | 72.7 | 71.2 |
| DETRPose-X | HGNetv2-B5 | 73.3M | 7.36 | 240 | 73.3 | 72.2 |
Cross-dataset results (CrowdPose test / OCHuman test; on OCHuman every model is trained only on the COCO train set, making it a pure out-of-distribution test):
| Dataset | Model | Params | Latency (ms) | FLOPs (G) | AP |
|---|---|---|---|---|---|
| CrowdPose | RTMO-L | 44.5M | 4.68 | 68 | 73.2 |
| CrowdPose | ED-Pose-R50* | 48.0M | 66.91 | 578 | 69.7 |
| CrowdPose | ED-Pose-Swin-L-5S* | 218.6M | 367.45 | 3048 | 76.5 |
| CrowdPose | DETRPose-L | 32.7M | 4.52 | 104 | 73.3 |
| CrowdPose | DETRPose-X | 73.3M | 7.11 | 232 | 75.1 |
| OCHuman | RTMO-L | 44.7M | 4.43 | - | 41.6 |
| OCHuman | GroupPose-R50* | 52.4M | 70.44 | - | 36.6 |
| OCHuman | ED-Pose-Swin-L* | 218.4M | 94.11 | - | 31.2 |
| OCHuman | DETRPose-S | 11.5M | 2.39 | - | 42.2 |
| OCHuman | DETRPose-X | 73.3M | 7.36 | - | 45.8 |
Ablation Study¶
Component-wise ablation (DETRPose-L, COCO val, all trained from scratch):
| Config | KSVF | Pose-LQE | Denoising keypoints | Pretraining | AP | FLOPs (G) | Latency (ms) |
|---|---|---|---|---|---|---|---|
| β RT-GroupPose (100 queries) | - | - | - | none | 69.9 | 121.6 | 5.24 |
| β‘ detections reduced to 60 | - | - | - | none | 68.9 | 107.1 | 4.58 |
| β’ β‘ + KSVF | β | none | 69.9 | 107.1 | 4.58 | ||
| β£ β‘ + Pose-LQE | β | none | 69.5 | 107.1 | 4.66 | ||
| β€ β‘ + denoising keypoints | β | none | 70.0 | 107.1 | 4.58 | ||
| β₯ KSVF + Pose-LQE | β | β | none | 69.6 | 107.1 | 4.66 | |
| β¦ KSVF + denoising | β | β | none | 70.9 | 107.1 | 4.58 | |
| β§ Pose-LQE + denoising | β | β | none | 70.5 | 107.1 | 4.66 | |
| β¨ all three | β | β | β | none | 71.5 | 107.1 | 4.66 |
| β© β¨ + COCO weights | β | β | β | COCO | 71.1 | 107.1 | 4.66 |
| βͺ β¨ + Obj2Coco weights | β | β | β | Obj2Coco | 72.4 | 107.1 | 4.66 |
| β« β¨ + Object365 weights | β | β | β | Object365 | 72.7 | 107.1 | 4.66 |
Ablation on the prior information used by denoising keypoints:
| Choice of ΞΊ | Choice of sΒ² | AP |
|---|---|---|
| constant ΞΊ=1/(#keypoints) | box approximation \(A_{box}\cdot0.53\) | 71.1 |
| official COCO ΞΊ | box approximation | 71.2 |
| constant ΞΊ | segmentation mask | 71.3 |
| official COCO ΞΊ | segmentation mask | 71.5 |
Key Findings¶
- Denoising keypoints is the largest single contributor: taking Model 2 (60 detections, 68.9 AP) as the baseline, adding KSVF alone reaches 69.9 (+1.0), Pose-LQE alone 69.5 (+0.6), denoising keypoints alone 70.0 (+1.1), and all three together 71.5 (+2.6 AP). Note that going from Model 1 (100 queries, 69.9) to Model 2 (60 queries, 68.9) costs 1.0 AP but buys 14.5 G of FLOPs and 0.66 ms of latency β this trade of one point for real-time performance is the starting point for every later gain.
- Combinations are not monotonic: KSVF pairs well with denoising (70.9) and with Pose-LQE (70.5), but denoising together with Pose-LQE falls back to 69.6 β below denoising alone (70.0) and roughly level with Pose-LQE alone (69.5). The authors offer no explanation, which makes this a phenomenon worth probing.
- Pretraining is not required: training with no pretrained initialization already reaches 71.5 AP, higher than loading D-FINE's COCO weights (71.1); only Object365 weights bring a clear gain (72.7). DETRPose can therefore be trained directly on custom 2D pose datasets that lack large-scale pretraining data.
- The denoising prior degrades gracefully: even with a constant ΞΊ and a box-area approximation of the segmented area β no mask annotations at all β the model still reaches 71.1 AP, only 0.4 below the 71.5 obtained with official COCO ΞΊ plus masks. This means the method can be fine-tuned on custom datasets without segmentation labels, and optionally adding SAM-style masks recovers roughly another 0.2.
- The model can be trimmed without retraining: reducing decoder layers or queries on a trained DETRPose-L gives 72.7/72.6/71.8/70.6 AP at 4.66/4.25/4.03/3.69 ms for 6/5/4/3 layers, and 72.7/72.6/72.4/71.9/71.5 AP for 60/50/40/30/25 queries. Even 25 queries with 4 layers still yields 70.7 AP at 3.53 ms, beating YOLOv8/11-X in accuracy while running faster than YOLOv8/11-L. The authors attribute the small effect of query count to the fact that both the mean and the maximum number of people per image in COCO are well below 40.
- Out-of-distribution generalization is the standout result: on OCHuman, DETRPose-S reaches 42.2 AP, at least 0.6 above every compared method and 1.85Γ faster than RTMO-L (41.6 AP), while the far larger ED-Pose-Swin-L manages only 31.2 AP, indicating those large models overfit COCO. On model scaling, going from S to L adds only 0.7 AP for RTMO but 3.0 AP for DETRPose, showing that DETRPose scales better on out-of-distribution data.
Highlights & Insights¶
- Turning the evaluation metric into the ruler for training signals: KS/OKS is normally an evaluation similarity, yet this paper uses it to solve two seemingly unrelated problems at once β inverting Ξ± for how far a denoising query should move, and supplying the soft label q of the KSVF loss. This "the metric is the prior" idea transfers to any task with a standardized similarity: 3D keypoints, mesh vertices, point-cloud registration β whenever the evaluation protocol defines a tolerance scale, it can be reused to build denoising perturbations.
- Trading loss-function quality for computation: KSVF makes classification scores reliable, which permits reducing instance queries from 100 to 60, and since every keypoint is a query, this saves 12% of FLOPs directly. Explicitly converting "better prediction quality" into "computation that can be cut" is a design rationale that is rarely spelled out, and it is worth reusing in other DETR variants.
- Keypoint denoising is a gap others walked around: DN-DETR/DINO denoising boxes are mature, yet pose models either skip denoising (GroupPose) or use it only in detection layers (ED-Pose). This paper identifies the seemingly trivial obstacle β a box has four corners to scale, a keypoint is a single point β as the reason for the gap, then sidesteps it with a similarity measure. The selection strategy of "find the pit others walked around and supply a principled fix" is itself instructive.
- A trained model doubles as an inference knob: layers and queries can be trimmed at inference without retraining, yielding a family of accuracy-latency operating points that naturally supports elastic and anytime inference.
Limitations & Future Work¶
- The limitation the authors acknowledge is latency still above the YOLO family: with comparable FLOPs DETRPose is slower, because group self-attention requires multiple transpose operations that break tensor contiguity. In the real-time comparison of Fig. 1 this shows up as curve position rather than in the FLOPs comparison.
- The paper never gives the complete loss list (the keypoint regression term besides KSVF, and the weights of the individual matching-cost terms, are all left out), so reproduction depends on the official code.
- Two ablation observations go unexplained: the degradation when denoising is paired with Pose-LQE, and COCO-pretrained weights scoring 0.4 lower than no pretraining at all. These are analytical gaps.
- Experiments cover only 17-keypoint human pose; face, hand, and animal keypoint settings are untested, and their ΞΊ values would have to be recalibrated β whether the same pipeline transfers is unknown.
- A concrete improvement direction: if the two-level interaction of group self-attention could be rewritten into an equivalent operator that preserves memory contiguity (for instance, reordering the layout by joint type and running a batched attention), latency could potentially be brought level with YOLO models of comparable FLOPs.
Related Work & Insights¶
- vs GroupPose: GroupPose uses group self-attention to eliminate dedicated detection decoder layers and is the direct base of this work, but it does no denoising and keeps many projection layers in its decoder, measuring 70.44 ms in practice. This paper keeps its group self-attention, removes the projection layers, and integrates FDR and Pose-LQE to bring latency down to 4.66 ms.
- vs ED-Pose: ED-Pose also uses DETR for pose, but with serial decoder stages β early layers detect, later layers estimate keypoints β and denoising is applied only to the detection layers and then discarded. This paper applies denoising directly to keypoints and avoids explicit detection-layer supervision, giving a simpler architecture.
- vs D-FINE / RT-DETR: These are real-time DETR detectors; this paper reuses D-FINE's backbone, encoder, and FDR refinement layers together with its Object365 weights, essentially porting "real-time DETR detection" into "real-time DETR pose", with the difference that KS replaces IoU as the quality measure for classification and matching.
- vs RTMO / YOLOv8-Pose / YOLO11-Pose: These single-stage methods rely on NMS and their latency fluctuates with the number of people. In accuracy, DETRPose-L is essentially level with RTMO-L (71.2 vs 71.6 AP on test-dev) while using 27% fewer parameters and gaining 4% in AR; DETRPose-S matches YOLOv8-X-Pose (67.3 AP) and YOLO11-X-Pose (67.2 AP) at 67.0 AP with 11.5M parameters, 81% fewer parameters and 52% faster.
Rating¶
- Novelty: ββββ Individually the three components β denoising keypoints, KSVF, and Pose-LQE β are ports and rewrites of existing techniques, but unifying denoising and classification quality through KS genuinely fills the gap of real-time DETR pose estimation.
- Experimental Thoroughness: ββββ Three datasets, four model scales, and three ablation groups (components, prior information, trim-ability), with disciplined reporting of latency and parameters; marks deducted for the missing full loss formulation and two unexplained ablation anomalies.
- Writing Quality: βββ The method and formulas are clear, but there are internal inconsistencies (the CrowdPose AP of ED-Pose-SwinL-5S is 76.6 in the abstract but 76.5 in Table 4; the text claims DETRPose-S/-M need "4.5Γ and 9.7Γ more" epochs than their RTMO counterparts, whereas the table's 700 RTMO epochs versus 156/72 actually means fewer), leaving readers to cross-check.
- Value: ββββ It demonstrates that DETR-based pose models can be both real-time and constant-latency, ships open-source code, and is directly relevant to real-time pose applications that need stable latency (VR, activity-recognition front ends).