FingerCap: Fine-grained Finger-level Hand Motion Captioning¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: finger-level motion captioning, hand-object interaction, pose augmentation, sparse video sampling, HandJudge
TL;DR¶
FingerCap establishes finger-level captioning using gesture and hand-object interaction videos, and injects dense 2D hand poses into a sparse-RGB video model through FiGOP, improving the same fine-tuned Qwen3-VL-8B baseline from 107.28 to 155.27 CIDEr and from 2.13 to 2.74 HandJudge.
Background & Motivation¶
A video model saying “stirring with a spoon” does not establish that it has resolved the hand motion: which fingers hold the handle, how the other hand stabilizes the container, and when the hands exchange the object can all disappear behind an action label. Conventional gesture recognition often predicts vocabulary items or categories, while human motion captioning emphasizes the body and whole hands. Such supervision teaches intent recognition without necessarily requiring an account of individual finger flexion, contact, and coordination. FingerCap therefore narrows the target from “what the person is doing” to “how particular fingers carry out the action,” retaining natural language as the output rather than joint coordinates or sign labels.
The missing evidence concerns both data and observation. Sign dictionaries can offer precise motion semantics, but dominant-hand references, instructional phrasing, and non-motor content are unsuitable as direct video captions. Natural hand-object interactions contain richer contact relations but lack corresponding finger-level textual supervision. Even with better captions, sparse RGB sampling can skip brief taps, finger configuration changes, and grip transitions; increasing the frame count then increases video tokens and memory use.
The paper addresses these issues through FingerCap-40K and FiGOP: the former supplies explicit descriptions of hand side, finger identity, motion order, and contact, while the latter adds structured motion evidence between keyframes. Evaluation must change as well, because a long caption sharing many reference words can still identify the wrong finger or invent a contact. Core Idea: use dense hand keypoints to fill the motion gaps in sparse imagery, then assess generated captions through finger identity, trajectories, contact, and sequence completeness instead of treating language similarity as motion understanding.
Method¶
Overall Architecture¶
The work spans data, modeling, and evaluation, with Qwen3-VL-8B as the model backbone. Given a video, FiGOP groups each sparse RGB keyframe with a neighboring sequence of 2D hand poses. The visual branch retains appearance and object context, the pose branch encodes finger topology and short-term changes, and a motion-aware projector passes their information to the language model for caption generation. Training uses FingerCap-40K reference captions; test-time generation needs no reference, and HandJudge only compares against it after generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
DATA["Finger-level Data Construction"] -.->|Videos and training supervision| STREAM["FiGOP Dual-Stream Encoding"]
VIDEO["Test video"] --> STREAM
STREAM --> FUSION["Motion-Aware Fusion"]
FUSION --> CAPTION["Language model generates caption"]
CAPTION --> JUDGE["HandJudge Evaluation"]
DATA -.->|Reference text only during evaluation| JUDGE
JUDGE --> SCORE["Four-dimensional scores"]
Reference captions in the diagram are not inputs to test-time generation, and no reinforcement learning loop feeds judge scores back into the model. Keeping RGB sparse does not mean the entire system reads only those RGB frames: pose estimation still needs intermediate imagery, while the savings concern dense pixel representations passed to the video-language model.
Key Designs¶
1. Finger-level Data Construction: ground language supervision in observable joint movements
Gesture sources span ASL, CSL, Auslan, and BSL; Table 1 lists BSL as an out-of-distribution source rather than a regular training source. Annotation starts from motion descriptions in the corresponding sign dictionaries, with humans resolving “dominant/non-dominant hand” into the left and right hands visible in the video and removing emotions, analogies, and other non-motor content. GPT-4.1 then polishes the wording under instructions not to change motion semantics or introduce facts, with sign language experts involved in review and refinement. This order matters: the model edits language rather than guessing finger movements from a sign's lexical meaning. The resulting captions describe the visible realization of signing, which is not equivalent to translating its linguistic meaning.
Hand-object interaction data primarily come from GigaHands and OakInk2, selecting views in which finger movements are clearly visible from multi-view recordings. Humans describe finger flexion, contact targets, and bimanual coordination before the same textual refinement; HOI4D and MotionBench provide out-of-distribution evaluation data. Table 1 reports 21,055 gesture clips and 19,922 interaction clips, totaling 40,977; “40K” is an approximate name. The subsets contain 1,922,471 and 4,251,389 frames, respectively, and the authors use an 8:1:1 training, validation, and test ratio. Selecting clear views improves fine-grained supervision, but also means the data do not represent arbitrary occlusions and cannot establish coverage of all real-world difficulty.
2. FiGOP Dual-Stream Encoding: recover between-keyframe changes with low-dimensional motion sequences
Each unit contains an RGB keyframe and a neighboring hand-keypoint sequence instead of passing every image through the vision encoder. DWPose extracts keypoints organized by hand joints; the paper gives 2D coordinates and confidence as examples of joint features without specifying the final channel configuration in the main text. The visual branch uses a frozen pretrained vision encoder to produce spatial appearance tokens. These tokens retain object context such as spoons and bowls, which keypoints alone cannot provide. The pose branch uses a two-layer ST-GCN followed by a lightweight temporal Transformer, exploiting joint connections for local motion before aggregating cross-frame dependencies.
The division is not intended to replace images with poses: images establish where an interaction occurs and what it involves, while poses describe the intermediate changes. For a brief touch or finger extension, keypoint sequences can preserve temporal evidence missed by sparse RGB keyframes. The main setting uses RGB at 2 fps, and the text attaches the following 8 pose frames to each keyframe, with pose-window ablations denoted by \(T_p\in\{4,8,16\}\). However, Table 7 also labels the pose stream as 8 fps; “8 frames per keyframe” and “poses at 8 fps” do not automatically specify the same temporal span. Exact resampling, window overlap, and boundary padding therefore cannot be inferred from the main text and require implementation verification.
3. Motion-Aware Fusion: let visual tokens retrieve relevant motion
Fusion occurs inside the multimodal projector rather than converting keypoints into text appended to the prompt. Cross-attention uses visual features as queries and pose motion features as keys and values, allowing visual tokens to retrieve relevant dynamics. The fused representation is then projected into the language model's embedding space and supports autoregressive caption generation together with the text instruction. Motion evidence thus enters the visual representation before language generation instead of being imposed by post-generation wording rules. For an identical hand configuration, RGB object context can still determine whether the action involves grasping an object, touching another hand, or making an object-free gesture.
The fusion operator in Equation (2) is corrupted in the text extraction, preventing a reliable reconstruction of how visual features and attention outputs are combined. This note retains only the query/key/value direction supported by Equation (3) and the prose, without inventing residual addition or elementwise multiplication as the authors' formula. The module gains information without increasing RGB sampling density, but it is not free: it adds pose estimation, graph convolution, temporal encoding, and projector fusion. The main text does not report complete end-to-end latency or added parameter counts, so the lightweight claim is supported mainly by the architecture and memory-constrained comparisons.
4. HandJudge Evaluation: inspect identity, motion, contact, and completeness separately
HandJudge uses GPT-4.1 to compare predicted and reference captions, provide reasoning, and assign scores from 0–5 on four dimensions. FHI checks left/right hand and individual finger identity; a vague reference to “fingers” cannot replace the essential participants. MT checks motion type and directional trajectory, distinguishing tapping, circling, and raising. CI checks physical contact and interaction relations, particularly penalizing hallucinated or physically implausible contacts. CMS checks coverage of semantically necessary phases in the reference annotation rather than demanding every observable state in the video.
Overall summarizes the four dimensions in the reported tables, but interpretation should examine the individual dimensions as well. Omitting a contact target and missing an action phase are different errors that a single lexical-overlap metric struggles to separate. This is a reference-conditioned language judge, not a sensor independently verifying geometric truth from the video. If the reference omits a contact or covers only part of an action, that annotation boundary also affects the score. The paper therefore supplements it with human model-ranking checks rather than establishing that judge outputs are infallible physical facts.
A Worked Example¶
The target action in Figure 5 is the right index finger tapping the left thumb. Sparse RGB can help identify the positions of both hands, but a model observing only a few images may misdescribe the approach and tap as circular motion. FiGOP's pose branch adds the index finger's approach trajectory, while the visual branch retains appearance cues for hand side and the contact region. Motion-aware fusion passes this evidence to the language model so the caption can specify the participating fingers and contact location. The zero-shot caption in the figure receives FHI, MT, CI, and CMS scores of 2.0, 1.0, 0.0, and 1.0; the FiGOP caption receives 5.0 on all four. This is one qualitative example selected by the authors, not a substitute for test-set averages or evidence that 2D poses recover true contact under every occlusion.
Loss & Training¶
Stage 1 freezes the vision encoder and language model, training only the pose encoder and projector for 1 epoch at a learning rate of \(10^{-4}\). This first aligns the new motion representation with the existing multimodal representation space without simultaneously updating the whole language model. Stage 2 unfreezes the language model; the text explicitly trains the projector and language model for 3 epochs using next-token prediction at a learning rate of \(10^{-5}\). The vision encoder remains frozen; whether the pose encoder continues updating in Stage 2 is not explicit in the main text and is not assumed here. Training uses 8 NVIDIA A100 GPUs with a reported batch size of 1, without clarifying whether this is global or per device in the main text. There is no additional HandJudge reward optimization, and inference requires no annotated finger trajectories: poses come from the estimator.
Key Experimental Results¶
Main Results¶
The following selects Average columns from Table 2 on page 11 and Table 3 on page 12, evaluated on the FingerCap-40K test set with RGB inputs uniformly sampled at 2 fps. CIDEr follows the authors' reporting scale, multiplied by 100 after evaluation with Jury; HandJudge dimensions range from 0–5, and higher is better throughout. Zero-shot and task-fine-tuned systems have different training conditions, so the last two rows provide the primary comparison for FiGOP's module-level gains.
| Model / Setting | CIDEr | FHI | MT | CI | CMS | Overall |
|---|---|---|---|---|---|---|
| GPT-4o, zero-shot | 21.29 | 2.54 | 2.03 | 1.70 | 2.81 | 2.27 |
| Gemini-2.5-Pro, zero-shot | 32.37 | 2.73 | 2.28 | 2.17 | 3.36 | 2.64 |
| Qwen3-VL-8B-Instruct, zero-shot | 29.14 | 2.38 | 1.69 | 1.52 | 2.79 | 2.09 |
| Qwen3-VL-8B + MM Proj + SFT | 107.28 | 2.42 | 1.84 | 1.80 | 2.47 | 2.13 |
| Qwen3-VL-8B + FiGOP + SFT | 155.27 | 2.96 | 2.50 | 2.45 | 3.06 | 2.74 |
Standard fine-tuning raises CIDEr from 29.14 to 107.28, but HandJudge only changes from 2.09 to 2.13, while CMS falls from 2.79 to 2.47. This directly demonstrates that matching reference wording and describing an action more completely are not equivalent, rather than showing that standard fine-tuning is entirely ineffective. FiGOP adds 47.99 CIDEr and 0.61 HandJudge over standard fine-tuning with the same backbone, improving all four fine-grained dimensions. Its Overall exceeds Gemini-2.5-Pro by 0.10, but CMS remains below the latter's 3.36, so it does not lead on every dimension.
Ablation Study¶
Table 7 on page 13 compares modalities and video sampling strategies; HJ denotes HandJudge Overall, while OOM means insufficient memory in that experimental setting, not impossibility on all hardware. Pose-only input has no RGB resolution; FiGOP's pose frequency follows Table 7, with the ambiguity relative to the textual window definition discussed in Method.
| Setting | Modality | RGB Resolution | RGB / Pose fps | CIDEr | HJ |
|---|---|---|---|---|---|
| Pose-only | Pose | Not applicable | Not applicable / 8 | 62.10 | 1.25 |
| Base MLLM | RGB | 512 × 512 | 2 / Not applicable | 107.28 | 2.13 |
| Medium Sampling | RGB | 512 × 512 | 4 / Not applicable | 124.00 | 2.29 |
| Dense Sampling | RGB | 512 × 512 | 8 / Not applicable | OOM | OOM |
| Low-Resolution | RGB | 256 × 256 | 8 / Not applicable | 118.00 | 2.25 |
| FiGOP | RGB + Pose | 512 × 512 | 2 / 8 | 155.27 | 2.74 |
Increasing RGB frame rate helps, but reducing resolution to fit more frames weakens finger details, while a pose-only stream lacks object semantics. These comparisons support complementarity between the two information sources, not an inherent superiority of pose features over visual features. Table 5 on page 13 further controls pose-window length under the remaining FiGOP settings, with standard caption metrics again following the authors' reporting scale.
| Pose Window \(T_p\) | BLEU-4 | ROUGE-L | METEOR | CIDEr |
|---|---|---|---|---|
| 4 frames | 14.12 | 36.10 | 37.95 | 147.50 |
| 8 frames | 15.36 | 37.92 | 38.89 | 155.27 |
| 16 frames | 15.15 | 37.75 | 38.50 | 153.80 |
Key Findings¶
- An 8-frame window beats 4 frames, but extending it to 16 does not help further; the authors attribute this to redundancy, jitter, and accumulated occlusion errors, an explanation not backed by separate noise measurements in the table.
- In the out-of-distribution evaluation of Table 6 on page 13, FiGOP scores 2.04 HandJudge versus 1.61 for standard fine-tuning; it still falls substantially below its in-distribution 2.74, leaving generalization unresolved.
- Table 4 on page 12 separately samples 100 clips rated by 5 independent human judges; human averages are 1.10 for zero-shot, 3.01 for standard fine-tuning, and 3.74 for FiGOP, with corresponding GPT-4.1 scores of 1.55, 3.18, and 3.61.
- This human subset supports agreement in the ranking of the three systems, but differs from the main test set: its 3.61 must not be conflated with the main table's 2.74, nor does it establish high per-example correlation.
Highlights & Insights¶
- The task turns correct finger identity from a descriptive detail into an explicit supervision target, separating recognition of intent from understanding how an action is executed.
- Multimodal budgets can be allocated by information type: sparse high-resolution images retain appearance, while structured poses add high-frequency motion, extending beyond a simple trade-off between frame count and resolution.
- Evaluation exposes a blind spot of standard fine-tuning: large language-metric gains need not improve contact relations and phase coverage, a useful lesson for fine-grained video caption evaluation.
Limitations & Future Work¶
- The authors acknowledge remaining out-of-distribution gaps and note that long pose windows can accumulate estimation noise; 2D keypoints also have limited ability to represent occlusion, depth, and true contact.
- From an evaluation perspective, GPT-4.1 participates in both caption polishing and judging, introducing potential style and reference biases; the human subset supplements this evidence without excluding every judge bias.
- Data selection favors clearly visible hand motion, and the main text gives split ratios without specifying isolation by person, original video, or action category, so neither a cross-subject split nor leakage should be asserted.
- The authors claim that prohibiting new facts during polishing completely avoids hallucinations, but a procedural constraint is not proof of zero annotation errors and should be distinguished from actual review quality.
- The available full text ends with references and lacks the appendix cited in the paper; the precise fusion operator, sampling-window implementation, Stage 2 pose-encoder status, and complete inference overhead still require verification.
- Future studies could examine confidence-aware pose fusion, auxiliary 3D signals under occlusion, and independent judge calibration; these are reader suggestions, not experiments completed in this paper.
Related Work & Insights¶
- Compared with GigaHands / OakInk2: these provide interaction videos and motion resources, while FingerCap builds finer finger-level language supervision on top rather than collecting every video from scratch.
- Compared with MotionLLM / ChatMotion: both belong to the broader motion-language research direction; FingerCap emphasizes individual fingers and contact phases, adding denser pose evidence to sparse video models.
- Compared with Video-LaVIT / Efficient Motion-Aware Video MLLM: the paper follows the separation of appearance and motion modeling, but uses structured hand joints as the motion representation for a finger-level task.
- Research insight: measuring omitted contacts separately from hallucinated contacts, then relating pose confidence to each error type, may locate bottlenecks better than continuing to optimize a single text-similarity score.
Rating¶
- Novelty: 4/5. The main contribution is the finger-level task, data, and evaluation framework, with FiGOP providing a task-focused combination of existing structures.
- Experimental Thoroughness: 4/5. Covers open and closed models, pose windows, modality and sampling, distribution shifts, and human evaluation, but lacks some reproduction details and end-to-end costs.
- Writing Quality: 3/5. The motivation is clear, but pose frequency and window length are not described consistently, and extracted fusion equations limit verification.
- Value: 4/5. Useful for fine-grained video-language understanding and hand-interaction captioning, but not yet a reliable physical-contact understanding system.