ECCV 2026 3D Vision Multimodal VLM Robotics human-robot interaction 3D human keypoint localization egocentric view vision-language model reinforcement fine-tuning

TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Tencent/TAIHRI
Area: 3D Vision / Multimodal VLM / Robotics
Keywords: human-robot interaction, 3D human keypoint localization, egocentric view, vision-language model, reinforcement fine-tuning

TL;DR¶

TAIHRI recasts task-relevant 3D human localization under a robot's egocentric camera as next-token prediction in a vision-language model: the interaction space is quantized into voxel tokens, the model reasons over 2D keypoints before inferring depth, and a pose-aware reward drives GRPO reinforcement fine-tuning — cutting global MPJPE on upper-body keypoints from 124.91 mm to 93.83 mm on Harmony4D and EgoBody close-range settings.

Background & Motivation¶

For robots to collaborate with people through physical contact — collaborative manufacturing, service robotics, assistive care — they must know where a person's hands, elbows, and shoulders are in the robot's own coordinate frame. What these applications need is not a pretty body mesh but metric-scale absolute positions in the camera frame: a handshake needs the palm, object handover needs the wrist, and lifting someone out of a wheelchair needs the underarm and torso. Mainstream 3D human pose and shape estimation (3D HPE/HMR) follows a different regime: SMPL-family parametric models regress a full-body mesh from an image, and errors are always measured relative to the root joint (Multi-HMR, PARE and similar methods), with root alignment applied before scoring. Recent methods such as CameraHMR, PromptHMR, and SAM 3D Body do feed camera intrinsics into the network and recover a global pose with absolute translation in camera coordinates, but they still optimize whole-body reconstruction quality and only generalize after training across many camera parameters and datasets.

Moving this machinery to close-range HRI breaks in two places. First, the evaluation frame does not match: a root-relative representation discards absolute translation and depth by construction — exactly what a robot needs for motion planning. Once errors are measured in the global camera frame (no root alignment, no rigid transformation), these methods degrade by multiples, running at 120–180 mm on Harmony4D. Second, the body parts of interest do not match: HRI typically happens at 0.5–3 m, where the robot's field of view is narrow and the person is only partially visible. Truncation, self-occlusion, and close-range perspective distortion disproportionately corrupt distal keypoints far from the root (wrists, ankles). Those are precisely the parts interaction depends on — in the visualizations of Figure 5, CameraHMR, PromptHMR, and SAM 3D Body show errors of hundreds of millimeters at wrists and ankles while torso joints stay within tens of millimeters.

The paper therefore changes the question: instead of reconstructing the whole body first and then picking out the few joints a task needs, it asks the model directly "which points does this task need, and where are they?" The mechanism is to turn localization into next-token prediction in a VLM: a natural-language instruction ("give the person a hug", "assist him to stand from left") prompts the model to name the task-relevant body parts itself, and 3D coordinates are quantized into discrete voxel tokens that the model emits one by one. Task awareness is thus not an add-on attention module but a direct consequence of language conditioning. Two obstacles had to be cleared first: no public dataset offers close-range egocentric views, so the authors synthesized CloseHRI; and monocular depth ambiguity makes direct depth regression unreliable, so 3D localization is decomposed into chained 2D-then-depth reasoning, followed by reinforcement fine-tuning with a pose-aware reward to push metric errors down. Core idea: write task-relevant 3D keypoint localization as an autoregressive sequence of "task instruction → keypoint names → 2D pixel coordinates → discretized 3D voxel coordinates", teach the format and the 2D–3D spatial correspondence with SFT, and calibrate accuracy with a GRPO reward computed only over visible joints, so the model outputs robot-ready body-part positions directly in the camera frame.

Method¶

Overall Architecture¶

The input is an RGB image from a monocular camera mounted on the robot, one natural-language instruction describing the interaction task, and known camera intrinsics; the output is the metric-scale 3D positions of N task-relevant keypoints in the robot's camera coordinate system. The pipeline has three parts. First, imaging normalization: different cameras are mapped onto one shared intrinsic assumption by fixing the focal length at 1000, rescaling the image resolution accordingly, and simulating principal-point offsets with random crops. Second, representation discretization: the interaction space in front of the robot is cut into a \(W\times H\times D\) voxel grid and coordinates are quantized to integers between 0 and 999, so "regressing a 3D coordinate" becomes "reporting three integers". Third, autoregressive localization: the VLM emits, in order, the names of the task-relevant keypoints, their 2D pixel coordinates, and their 3D voxel coordinates, and a decoder maps the voxel tokens back to metric coordinates. The model is built on Qwen3-VL, trained with SFT and then GRPO reinforcement fine-tuning; the predicted keypoints serve either directly as end-effector targets or as anchors that lift a root-relative human mesh into the global coordinate system.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Egocentric RGB + instruction<br/>+ camera intrinsics K"] --> C["Discretized interaction space<br/>focal length fixed at 1000, coords to 0–999 tokens"]
    B["CloseHRI dataset<br/>0.5–3 m synthetic egocentric views + instruction bank"] -.->|SFT supervision| D["2D keypoint reasoning<br/>name task parts and 2D first, then depth to 3D"]
    C --> D
    D --> E["Voxel token decoding<br/>back to metric camera coordinates"]
    E --> F["Downstream use<br/>global mesh alignment / IK robot control"]
    D -.->|sampled responses| G["Pose-aware reinforcement fine-tuning<br/>Huber + PCK success reward"]
    G -.->|updated policy| D

Key Designs¶

1. CloseHRI dataset: synthetic data fills the close-range egocentric gap

No existing dataset offers the view that HRI needs. Human3.6M and MPI-INF-3DHP are third-person full-body captures, and while AGORA and BEDLAM offer crowded scenes, their cameras sit far away. Views at 0.5–3 m from a head-mounted camera, with the truncation and occlusion that a narrow field of view produces, are a distribution that public data simply does not contain — and it is exactly the distribution the model must learn.

The construction pipeline follows the WildHuman line of work and splits "geometrically exact" from "photorealistic" across two tools. Motion sequences are sampled from AMASS and rendered as normal maps with the SMPL-X model in Blender, giving poses with exact geometry. A virtual camera is then placed 0.5–3 m from the subject with randomized height and orientation, approximating the viewpoint of a humanoid robot's head camera. Using the normal maps as control signals, SDXL synthesizes photorealistic images with complex backgrounds under diverse text prompts (e.g. "a person wearing a red shirt standing in a living room"). Two automatic filters follow: SAM3 discards samples whose mask IoU falls below 0.9, and VitPose re-verifies geometry by computing 2D keypoint reprojection errors, keeping only images below 15 pixels — in effect using a 2D detector to check the generated imagery backwards for geometric consistency. This yields over 1 million close-range egocentric images. To widen the distribution further, BEDLAMv1/v2 and PDhuman are mixed in, uniformly filtered so that the average depth of all keypoints stays within 3 m, for a final training set of roughly 1.2 million images.

An equally important second product is a prompt bank of over 6,000 interaction-centric instructions (e.g. "lift the person from the wheelchair."). These deliberately name the body regions that matter for HRI tasks, and during training they replace the default instruction with 50% probability so the model binds linguistic semantics to specific anatomical regions; the other 50% keeps a generic prompt asking for 1 to 6 keypoints, preserving whole-body understanding and preventing overfitting to template phrasing. Forcing the camera distance distribution into 0.5–3 m aligns the training distribution with the HRI test distribution — the precondition for transferring directly to the two real egocentric datasets, Harmony4D and EgoBody.

2. Discretized interaction space: turning 3D coordinates into integer tokens a VLM can say

VLMs excel at predicting discrete tokens and remain weak at continuous coordinate regression: in Table 2, GPT-5.2 does not support 3D coordinate output from a monocular image at all, and Qwen3-VL-235B reaches a 1298.3 mm global error. Following the Pix2seq line, the authors assume the robot operates inside a predefined interaction cuboid (width \(W\), height \(H\), depth \(D\)) and quantize each axis to integers between 0 and 999, so a keypoint at \((x_i,y_i,z_i)\) becomes three voxel tokens:

\[X_i=\left\lfloor \frac{x_i}{W}\times 999\right\rfloor,\quad Y_i=\left\lfloor \frac{y_i}{H}\times 999\right\rfloor,\quad Z_i=\left\lfloor \frac{z_i}{D}\times 999\right\rfloor\]

(⚠️ The subscripts and multipliers of this equation are corrupted in the extracted text; 999 and the quantization direction are restored from the prose statement that coordinates are quantized to values between 0 and 999. The offset convention for the origin of the interaction volume should be checked against the original paper.) Quantization replaces "emit a real number" — something VLMs do badly — with "emit a token from a 0–999 vocabulary", at the cost of a precision ceiling set by the grid resolution.

Discretization solves how to output, but not where depth comes from: monocular depth ambiguity means the model must know what kind of camera it is looking through. Rather than adding a camera-parameter encoder or learning a ray embedding, the authors adopt the conclusion of DepthLM — intrinsic-conditioned augmentation is more effective than building an adapter for camera parameters. During both training and inference the focal length is fixed at 1000 and the image resolution is scaled accordingly; principal-point offsets are simulated with random crops so the model generalizes across image shapes and sizes at inference. The ablation prices this choice: a model with no intrinsic conditioning at all sits around 425 mm on all four body-part configurations of Harmony4D, swapping in a learnable ray embedding still leaves 380–424 mm, whereas the full model with unified intrinsics reaches 93.83 mm (upper body). The authors' explanation is that new parameters like a ray embedding break the VLM's pretrained visual encoding, while changing the resolution provides a geometric prior without touching the architecture at all.

3. 2D keypoint reasoning: see first, then measure distance

Asking the model to emit a 3D coordinate in one shot forces it to answer "which point is this" and "how far away is it" simultaneously, and the latter is the least reliable part of monocular vision. The starting observation is that 2D locations are evidence directly visible in the image and are far easier to estimate than depth, so writing them explicitly into the output sequence turns them into a chain-of-thought for depth reasoning.

The model is therefore trained to emit three segments in order: first a sentence naming the task-relevant keypoints ("I should focus on the left shoulder, right shoulder, left elbow and right elbow."), then their 2D pixel coordinates inside a <kpt2d> tag, and finally their 3D voxel coordinates inside a <kpt3d> tag, each entry written as <part name><x><y> (or <part name><x><y><z> for 3D) token by token. One design detail is worth noting: the names of the task-relevant parts sit at the very front of this autoregressive sequence, so "which points" and "where they are" share a single generation — the part names the model utters are both the list of coordinates to emit next and a cue that directs attention to the corresponding image regions. The ablation confirms the chain matters: removing 2D reasoning and predicting 3D directly raises the upper-body error from 93.83 mm to 126.67 mm, degrading all four configurations, so "2D first" is not a dispensable intermediate step but the scaffold on which the model builds the 2D–3D spatial correspondence.

4. Pose-aware reinforcement fine-tuning: reward only the visible joints

SFT can only imitate coordinate sequences found in the training data; at the edges of that distribution — truncation, self-occlusion — it still emits wildly misplaced numbers. After SFT, the authors add a GRPO stage that optimizes the whole response with a reward designed for pose error. The reward carries one key constraint: it is computed only over the set of visible joints \(\mathcal{V}\) — truncated or occluded keypoints have no reliable supervision signal, and including them would only push the policy in the wrong direction. The reward sums two terms: a soft term that passes each joint error through a Huber function, averages over visible joints, and takes a negative exponential; and a PCK-style success term measuring the fraction of joints whose error falls below a threshold \(\kappa\), giving a hard signal of "how many points are actually accurate enough":

\[r=\lambda\exp\!\left(-\frac{1}{|\mathcal{V}|}\sum_{j\in\mathcal{V}}\rho_\delta(d_j)\Big/\tau\right)+(1-\lambda)\frac{1}{|\mathcal{V}|}\sum_{j\in\mathcal{V}}\mathbb{I}(d_j<\kappa)\]

Here \(d_j=\lVert \hat{y}_j-y_j^{gt}\rVert_2\) is the prediction error of joint \(j\), and \(\rho_\delta(d)=\frac{1}{2}d^2\) for \(d\le\delta\) and \(\delta(d-\frac{1}{2}\delta)\) otherwise is the Huber function. The same formulation is instantiated once for 3D joints and once for 2D keypoints (\(y\in\{x,u\}\)), choosing the corresponding \(d_j\) and keeping the units of \((\delta,\kappa,\tau)\) consistent. (⚠️ The normalization inside the exponential is corrupted in the extracted text; the form above is reconstructed from the legible parts, and the exact placement of \(\tau\) should be checked against the original paper.)

Why this works: Huber keeps the few grossly mispredicted joints from dominating the gradient, and the PCK term directly rewards "all the points this task needs land inside the threshold". The ablation is unusually decisive — replacing the reward with plain RMSE sends the four configurations to 795–831 mm, an order of magnitude worse than skipping reinforcement learning entirely, showing that outlier joints in the pose error distribution skew the gradient of a naive regression objective; dropping only the RFT stage costs a mild 93.83 mm → 101.82 mm.

A Worked Example¶

Take the flow of Figure 4. The input is an image from the robot's viewpoint, the camera intrinsics, and the human instruction "Give the person a hug." The model first resamples the image to the unified focal length of 1000, then generates its answer autoregressively:

Naming the task parts: "I should focus on the left shoulder, right shoulder, left elbow and right elbow." — the model decides by itself that a hug needs both shoulders and elbows, not palms or knees;
Reporting 2D: <kpt2d><l_shoulder><283><764><r_shoulder>…</kpt2d>, giving pixel-level x and y per keypoint;
Reporting 3D: <kpt3d><l_shoulder><367><865><r_shoulder>…</kpt3d>, where the numbers are already integer tokens in 0–999 on the interaction voxel grid.

The decoder maps the voxel tokens back to metric coordinates in the camera frame using the width, height, and depth of the interaction volume (the triplets of the form \((-0.15,-0.16,0.67)\) in Figure 1 are of this magnitude, in meters). Those coordinates go two ways: directly as the task's human affordance region, retargeted to robot end-effector poses through inverse kinematics to close a perception–action loop; or as anchors that lift the root-relative mesh produced by SAM 3D Body into the robot's camera frame, yielding a global human mesh.

Loss & Training¶

Training has two stages. The SFT stage builds on Qwen3-VL at 4B and 2B scale and fits the "part names → 2D → 3D" sequence with next-token prediction: batch size 4, 5 epochs, learning rate 2e-5. On the data side, each training image draws an instruction from the interaction bank with 50% probability and otherwise uses a default prompt asking for 1–6 keypoints. The RFT stage continues with GRPO: for one prompt, \(K=8\) responses are sampled, the group mean reward serves as the baseline for a group-relative advantage, and the policy is updated with a clipped importance-ratio objective (the GRPO equation is corrupted in the extracted text so it is not reproduced here; ⚠️ refer to the original paper). The KL penalty coefficient is \(\beta=0.01\), batch size is 32, learning rate 1e-6, 5 epochs, with 10,000 images sampled from the training set per run. All experiments run on 4 NVIDIA H20 GPUs.

Key Experimental Results¶

Main Results¶

The metric is G-MPJPE (Global Coordinate Mean Per Joint Position Error, in millimeters): the average Euclidean distance between predicted and ground-truth joints computed directly in the camera coordinate frame, without root alignment and without any rigid transformation, so it penalizes both relative structural error and absolute translation/depth error. This differs from prior work reporting MPJPE in root-relative coordinates; the authors argue that absolute error is what reflects HRI's real requirement for metric localization in a shared physical workspace, where a millimeter-level deviation can cause unsafe contact or faulty motion planning.

Two test sets are used, both egocentric interaction data: Harmony4D-Test (6,389 frames of close human–human interaction captured by a head-mounted camera, with accurate 3D keypoint annotations from multi-view triangulation across 20 external cameras) and EgoBody (5,000 close-range frames sampled within 3 m out of 62,155). Evaluation covers four body-part configurations: Upper (both shoulders plus both elbows), Lower (both hips plus both knees), L-Upper (left shoulder, elbow, wrist), and R-Upper (right shoulder, elbow, wrist) — deliberately spanning symmetric (upper vs. lower body) and asymmetric (left vs. right upper limb) groupings to test whether the model's attention shifts to different semantic regions.

Method	Intrinsics	Harmony4D Upper	Harmony4D Lower	Harmony4D L-Upper	Harmony4D R-Upper	EgoBody Upper	EgoBody Lower	EgoBody L-Upper	EgoBody R-Upper
CameraHMR	yes	167.50	165.72	179.22	169.26	94.92	118.96	101.27	99.45
PromptHMR	yes	158.70	158.44	158.25	157.18	84.45	127.26	91.82	87.48
SAM 3D Body (3DB-H)	yes	127.72	129.73	143.41	118.77	88.23	110.58	93.12	94.06
SAM 3D Body (3DB-DINOv3)	yes	124.91	127.80	143.13	123.58	89.87	107.75	92.36	93.30
TAIHRI-2B	yes	97.15	118.82	119.86	103.16	85.62	113.94	93.56	91.37
TAIHRI (4B)	yes	93.83	114.98	107.81	98.23	75.77	101.58	81.42	81.27

Beyond 3D human pose methods, the authors compare against general-purpose VLMs and vision foundation models. Since these cannot take HRI-style instructions, they are given a blunt question instead: "Given the input image, please provide the 2D and 3D coordinates of the following keypoints of the person: left shoulder, right shoulder, left elbow, right elbow. Each 2D keypoint should have x, y values in pixel level. Each 3D keypoint should have x, y, and z values in mm." That comparison uses 50 samples from Harmony4D-Egocentric under the Upper configuration, so the full model's 97.2 mm and the 93.83 mm of Table 1 are not measured on the same subset.

Method	2D	3D	G-MPJPE (mm)
GPT-5.2	✓	✗	–
Qwen3-VL-235B-A22B-Instruct	✓	✓	1298.3
Gemini-2.5-Pro	✓	✓	436.9
Rex-Omni	✓	✗	–
VitPose + Depth Anything 3	✓	✓	352.2
VitPose + DepthLM	✓	✓	282.3
TAIHRI (full model)	✓	✓	97.2

Ablation Study¶

Ablations run on the Harmony4D-Egocentric test set, again in G-MPJPE (mm). Two questions are asked: whether intrinsic conditioning is needed, and which part of the two-stage training strategy carries the gain.

Config	Upper	Lower	L-Upper	R-Upper
w/o camera intrinsic conditioning	425.13	433.59	429.47	428.11
learnable ray embedding instead	380.29	423.55	400.09	382.53
w/o 2D keypoint reasoning	126.67	138.39	134.50	126.71
w/o RFT	101.82	121.26	110.01	110.79
RFT with RMSE reward	795.24	831.09	820.75	798.63
Full model	93.83	114.98	107.81	98.23

Key Findings¶

Intrinsic conditioning is the single biggest pillar: removing all intrinsic handling takes the error from 93.83 mm to 425.13 mm (Upper), the worst degradation in the whole ablation; swapping in a learnable ray embedding only recovers to 380–424 mm. The point is not "feeding camera parameters in" but injecting geometric priors without disturbing the pretrained architecture.
The chained 2D reasoning contributes consistently: removing it sends the four configurations to 126.67 / 138.39 / 134.50 / 126.71 mm, a uniform degradation — clean evidence that the format itself carries capability.
The RFT reward must be robust: switching the reward to RMSE blows up to 795–831 mm, an order of magnitude worse than skipping RFT altogether (101.82 mm); outlier joints dominate. The gain from RFT itself is comparatively modest (93.83 vs. 101.82 mm), a refinement rather than the main source of accuracy.
General-purpose models cannot reach this task: the best, Gemini-2.5-Pro, still sits at 436.9 mm, and VitPose + DepthLM reaches 282.3 mm; GPT-5.2 and Rex-Omni do not support 3D coordinate output at all. Depth pipelines fail because per-pixel prediction does not model articulated human structure and is least reliable exactly where self-occlusion occurs — they have pixels, not the prior that "this is a person's arm".
Model scale mainly affects cross-dataset generalization: TAIHRI-2B scores 97.15 mm versus the 4B model's 93.83 mm on Harmony4D Upper, a small gap; on EgoBody the gap widens (85.62 vs. 75.77 mm), and the 2B model even falls slightly behind PromptHMR (84.45 mm) on EgoBody Upper — the smaller model cannot hold accuracy out of domain.
The four body-part configurations differ in difficulty: on Harmony4D, Lower (both hips and knees) is hardest at 114.98 mm, whereas on EgoBody Upper, L-Upper, and R-Upper all land in the 75–81 mm range. This tracks differences in camera mounting height, field of view, and interaction type between the two datasets, so absolute numbers should not be compared naively across them.

Applications: Global Mesh Alignment and Real-Robot HRI¶

The predicted 3D keypoints are also used as anchors to align a root-relative human mesh: given a mesh \(M\) in normalized space and the set of predicted keypoints \(\{K_i\}\), a transformation \(T\) is computed that aligns the mesh's keypoints with the predicted ones and is then applied to the whole mesh, giving the global mesh \(M_{global}=T(M)\). Taking a handshake as the example, 1 to 3 anchors are placed on the right arm and alignment quality is evaluated at the right wrist:

Anchor configuration	Alignment error (mm)
1 anchor	15.79
2 anchors	32.52
3 anchors	22.07

All three anchor counts improve clearly over the root-relative baseline, showing that anchoring with task keypoints is enough to pin the mesh into the robot's coordinate system. The lower half of Figure 6 additionally shows anchor choices for other interaction tasks (left arm, pelvis, with errors of 121.45 mm and 65.89 mm). (⚠️ The layout of that figure is scrambled in the extracted text; the mapping between anchor count/region and specific tasks should be read off Figure 6 of the original paper.)

For the real robot, the authors build a closed-loop controller for a bimanual robot: an Orbbec Femto Bolt camera mounted on the robot captures egocentric RGB at 720p and is calibrated to the robot's coordinate system; image, interaction instruction, and intrinsics go into TAIHRI to obtain task-relevant 3D keypoints, while a root-relative human mesh from SAM 3D Body is aligned to global space using the predicted anchors. For each task a corresponding human affordance region is predefined, and the robot's end-effectors are retargeted to the predicted body parts via inverse kinematics, forming a perception–action loop. Contact-rich tasks such as shaking hands and shoulder massage are demonstrated. This part is qualitative — no success rate or error statistics are reported — and the authors' claim is that localization is accurate and stable enough to guide precise, reliable physical interaction.

Highlights & Insights¶

Task awareness falls out of language conditioning rather than being bolted on as a part-selection module: because the part names sit at the very front of the output sequence, selection and localization share one autoregressive generation, and the mapping from task to keypoint set is controlled directly by instruction text. The same "say what you need, then say where it is" pattern transfers to any instruction-driven sparse perception task, such as localizing graspable regions or selecting assembly contact points on demand.
Intrinsic conditioning by changing resolution instead of adding a module: fixing the focal length at 1000, rescaling the image, and simulating the principal point with random crops all happen in the data layer, leaving every network parameter untouched. It beats a ray embedding by an order of magnitude in the ablation, suggesting that when injecting geometric priors into a VLM, "don't touch the pretrained weights" often beats "add an adapter that can learn geometry".
Rewarding only the visible joints: truncation and occlusion leave some keypoints with no reliable ground truth, and excluding them from the reward lets reinforcement learning focus on what is learnable. The idea applies to any supervision or reward design under partial observability.
2D as chain-of-thought, depth as the conclusion: the least reliable quantity in monocular 3D localization is placed at the end of the sequence, with visible evidence (2D coordinates) in front; the degradation curve (93.83 → 126.67 mm) shows the chain is doing reasoning rather than decorating the format.
Keypoints as anchors is a zero-cost downstream reuse: instead of retraining a global HMR model, a few anchors lift an existing root-relative model's output into the camera frame, bringing right-wrist error down to 15.79 mm for the handshake case.

Limitations & Future Work¶

Dependence on known, fixed intrinsics: inference requires the intrinsics, and the image must be resampled after normalizing the focal length to 1000. On a real robot with a zoom lens, unknown intrinsics, or calibration drift, this precondition needs an extra calibration step.
The interaction volume is a preset cuboid: quantizing to 0–999 means the precision ceiling is set by grid resolution — the larger the volume, the coarser a single token becomes — and scenes beyond the volume (room-scale navigation, multi-person settings) fall outside the scope.
Training data is mostly synthetic: SDXL generation plus automatic filtering preserves geometric consistency, but the appearance gap to real imagery and whether the AMASS motion distribution covers contact-rich interactions such as hugging or lifting are not analyzed. Both real test sets are human–human interaction, still a distribution away from real human–robot interaction.
The real-robot experiment is qualitative only: the handshake and shoulder-massage demos report no success rate, error statistics, or user study, so how much TAIHRI's localization error affects end-task success in the loop is unquantified — precisely what an HRI paper should supply.
Task awareness lacks a quantitative metric: the four body-part configurations are assigned by hand, and "given 6,000+ instructions, can the model pick the right part set" is shown only as illustrative examples in Figure 6. A selection accuracy/recall for instruction → part set would make the task-awareness claim much firmer.
Accuracy is still short of contact-rich manipulation: G-MPJPE averages over all joints while distal joints err more, and 75–115 mm may be just enough for a handshake but not for fine manipulation such as handing over a cup or fastening a button.
Possible improvements: replace the fixed interaction volume with a data-driven, robot-centered non-uniform quantization (fine near, coarse far), or add hierarchical tokens (coarse cell plus residual refinement) to break the 0–999 resolution ceiling; replace 2D reasoning with explicit geometric constraints (limb-length consistency, projection consistency) as intermediate supervision; and evaluate on real robots with task success rates so localization error can be tied to task outcomes.

vs global 3D human pose estimation (CameraHMR / PromptHMR / SAM 3D Body): they also output absolute poses in camera coordinates and accept intrinsics, but optimize whole-body reconstruction and are still scored on root-relative or reconstruction error; under close-range truncation their distal joints (wrists, ankles) err by 120–180 mm. TAIHRI predicts task keypoints directly, concentrating capacity on the parts that matter, and leads by 30–50 mm on the camera-frame metric.
vs VLM-based 3D human understanding (ChatPose / ChatHuman / PoseLLaVA): these let MLLMs manipulate 3D pose through tools or dialogue, but their output remains root-relative pose and "task-relevant parts" is not a first-class object. TAIHRI's voxel quantization plus chained 2D reasoning is a targeted redesign for millimeter-level task-part localization.
vs VLM 3D grounding (LocateAnything3D / Rex-Omni / SeeGround): they infer 3D boxes from 2D cues for generic objects; Rex-Omni cannot even output 3D on this task (the 3D column in Table 2 is ✗), showing that object-level 3D localization and body-part-level localization demand different precision and structural priors.
vs depth back-projection pipelines (Depth Anything 3 / DepthLM + VitPose): the two-stage route is the simplest to engineer, but per-pixel depth does not model articulated structure and is worst under self-occlusion, measuring 282–352 mm. The comparison even compensates for the roughly 2 cm systematic offset between surface depth and anatomical joint centers, giving that baseline a fair shot.
vs synthetic data work (WildHuman / AGORA / BEDLAM): the synthesis pipeline is inherited largely unchanged; the increment is placing the camera at 0.5–3 m egocentric poses, refiltering by "average depth ≤ 3 m", and pairing the data with an interaction instruction bank — a reminder that the viewpoint dimension of a data distribution can matter more for downstream usability than its size.

Rating¶

Novelty: ⭐⭐⭐⭐ First to make task-relevant body-part 3D localization an explicit VLM task; the combination of voxel tokens, chained 2D reasoning, and pose-aware GRPO is clean and self-consistent.
Experimental Thoroughness: ⭐⭐⭐⭐ Two egocentric datasets, four body-part configurations, three families of baselines, and two ablations give good coverage; it loses a star for the purely qualitative real-robot study and the missing quantitative measure of task awareness.
Writing Quality: ⭐⭐⭐⭐ The motivation and the root-relative vs. camera-coordinate contrast are explained clearly; some equations and figures are poorly typeset, and a few application results must be inferred from the figure.
Value: ⭐⭐⭐⭐ A confident answer to "can a VLM do millimeter-level robot perception", and CloseHRI plus the instruction bank are directly reusable by the embodied HRI community.