MotionAtlas: Detailed Region Captioning for Motion-Centric Videos¶

Conference: ECCV 2026
arXiv: 2606.29531
Code: https://kagura-0001.github.io/projects/MotionAtlas (Project page, including benchmark / data / code)
Area: Multimodal VLM / Video Understanding
Keywords: Region-level motion captioning, Video-MLLM, Fine-grained motion, Bootstrap refining, Evaluation benchmark

TL;DR¶

MotionAtlas narrows down the task of "describing the motion of an entire video"—which is notoriously difficult to evaluate quantitatively—to "only describing the motion within a target region given a video and a spatio-temporal mask." Along with this formulation, it delivers a diagnostic benchmark with 2,073 MCQs, a bootstrapped data pipeline capable of scalably generating 159k high-fidelity motion descriptions, and a training recipe that consistently boosts Qwen3-VL and Molmo2 across eight motion understanding benchmarks.

Background & Motivation¶

Video Multimodal Large Language Models (Video-MLLMs) have become remarkably strong at general video understanding. However, when tasks require precise spatio-temporal reasoning over local regions—such as determining which direction a badminton player moves when returning to position, how their torso leans back during a jump, or how their footwork tempo changes—existing models almost collectively fail. They tend to capture high-level semantics ("a person playing badminton") but fail to articulate fine-grained kinematic details of "how" they move. Even more troublesome is the data aspect: prevailing approaches rely on global motion captioning (describing the motion of the entire scene or salient objects). This paradigm suffers from an almost intractable weakness—the non-evaluability of the evaluation itself. Free-form global motion captions make it extremely difficult to verify either completeness (whether key motions are omitted, i.e., recall) or faithfulness (whether hallucination occurs, i.e., precision/anti-hallucination). Consequently, annotation pipelines cannot perform reliable quality control, hindering scalability.

The authors attribute the root of this impasse to the "inherently divergent task definition" of global captioning. In global captions, visual clutter and multi-object motion entanglement are intertwined, dispersing the annotator's attention and leaving the model's output without a verifiable anchor to align with. Instead of banging one's head against the wall on "how to evaluate divergent free-form text," it is far better to switch to a more controllable task formulation: restricting the scope of description using a spatio-temporal mask, demanding the model to only describe the motion within this designated region. Once the region is bounded, visual clutter and motion entanglement are naturally decoupled. Annotators can focus solely on local motion, yielding annotations that are finer, more comprehensive, and highly consistent—establishing the exact prerequisite for making evaluation "quantifiable and scalable."

Following this intuition, this work decomposes region-level motion evaluation into an event–motion–fact annotation hierarchy: a video is first segmented into several events, each described with dense motion details, which are then distilled into independently verifiable atomic facts. These facts are further converted into a Multiple Choice Question (MCQ) checklist. Thus, the challenging problem of "scoring a piece of free-form description" is reformulated into a dense, diagnostic verification process where "a judge model checks the list question-by-question." Core Idea: Narrow down motion captioning from non-evaluable "global free-form text" to "region-level description constrained by spatio-temporal masks"; use the "event \(\to\) atomic fact \(\to\) MCQ checklist" hierarchy to turn evaluation into quantifiable question-by-question verification; and replace manual auditing with "dual-rollout bootstrap differential judgment" to scale high-fidelity motion annotation to the hundred-thousand level.

Method¶

Overall Architecture¶

MotionAtlas is a trinity system of "benchmark + data + model," but it is fundamentally powered by two parallel annotation pipelines and a training recipe. On the evaluation side, MotionAtlas-Bench is constructed by selecting entities whose "motions are inherently difficult to describe" from four motion-rich data sources. Following event segmentation, six-dimensional motion ontology densification, atomic fact extraction, MCQ generation, and two-level quality control, it yields 107 videos and 2,073 MCQs, accompanied by a checklist-based judging protocol to evaluate any caption on Accuracy / Recall / Precision. On the data side, to scale up the high-quality "AI proposal + manual refinement" workflow to 159k samples, the key challenge is to eliminate the manual auditing step without sacrificing quality. To this end, a three-stage segment–caption–bootstrap–summarize pipeline is designed, using dual-rollout bootstrap differential judgment to replace human correction and multi-source narrative synthesis to resolve temporal inconsistencies between events. On the training side, region-level motion data is blended with general motion and image-text QA in proportionate ratios for a unified SFT to produce the MotionAtlas-4B/8B model series.

The diagram below illustrates the data pipeline (the evaluation and data sides share the designs of event segmentation and spatial cropping; the difference lies in human involvement on the evaluation side vs. bootstrapping on the data side):

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Video V + frame-wise masks {m_t}"] --> B["Event Segmentation<br/>One Event = One Core Motion"]
    B --> C["Merged Focal Crop<br/>Union of full-sequence bbox, locking target"]
    C --> D["Dual Local+Global Description<br/>Local captures details, Global provides timeline"]
    D --> E["Bootstrap Refinement<br/>Dual-rollout discrepancy -> Blind referee -> Correction"]
    E --> F["Multi-Source Narrative Synthesis<br/>Unify timeline based on global caption"]
    F --> G["MotionAtlas-Data 159k samples"]
    G --> H["Unified SFT Mixed Training"]
    H --> I["MotionAtlas-4B / 8B"]

Key Designs¶

1. Region-Level Motion Captioning Task + Event–Fact–MCQ Evaluation Protocol: Turning "Unscoreable Free Descriptions" into "Question-by-Question Checklist Verification"

The fatal flaw of global motion captioning is not that models cannot describe motion, but that "there is no way to objectively score the descriptions." The authors break this bottleneck by first constraining the task within a spatio-temporal mask, and then decoupling evaluation into three layers: a video is first segmented into events (each corresponding to a core motion); each event is densified along six motion dimensions—Kinematics (direction/trajectory/speed change/rotation), Body Parts (independent movements of limbs), Spatial (position relative to landmarks), Interaction (contact/grasp/release), State (posture and occlusion changes), and Camera (lens movement)—into dense descriptions, which are then distilled into atomic facts that isolate a single verifiable attribute. The referee protocol is thus defined: given the model-generated caption \(C\) and video \(V\), a judge model answers each MCQ checklist item one by one, outputting three complementary metrics. Notably, the definition of Recall is ingenious—it includes both "correctly answered" and "incorrectly answered but at least mentioned" in the numerator, specifically measuring the model's "willingness to describe the target motion" rather than evading with "not mentioned":

\[\text{Recall} = \frac{N_{\text{correct}} + N_{\text{error}}}{N_{\text{total}}}, \qquad \text{Precision} = \frac{N_{\text{correct}}}{N_{\text{correct}} + N_{\text{error}}}\]

Precision calculates the accuracy rate only when the motion is "indeed mentioned." This MCQ checklist explicitly introduces three types of diagnostic negative options: not mentioned / mentioned but no value (used to penalize missed observations and explicitly evaluate recall), and mentioned but different value (used to catch hallucinations). Consequently, the otherwise divergent evaluation of motion descriptions is re-engineered into a dense, diagnostic, reproducible check-off, which is highly robust to the choice of the judge model (even when replacing judges with Gemini 3 Pro / GPT-5.2 / Qwen3-VL-235B, the Spearman's \(\rho\) of model ranking remains between 0.89–0.98).

2. Merged Focal Crop: Locking Targets with Large Displacements Using the Union of the Entire Sequence's Bounding Boxes Instead of Frame-by-Frame Cropping

Fine-grained motion understanding requires VLM to anchor its attention on the target entity. However, targets often undergo huge physical displacement in videos; frame-by-frame cropping causes spatial jitter and varying crop sizes, leading to unstable effective resolution. The authors' approach is to take the union of the bounding boxes of the entity across all frames within the event interval, constructing a stationary merged focal crop area. This way, even if the target moves significantly, the cropped region remains fixed, eliminating spatial jitter, lifting the effective resolution over the target, and suppressing background distractors. Ablations reveal that this step is absolutely critical: removing spatial cropping and feeding raw video segments directly drops annotation Accuracy from 39.9 to 32.7 (\(-7.2\)), representing one of the largest performance drops among the three components—proving that "zooming in on the target" is the prerequisite for guiding VLMs to look at the right place.

3. Bootstrap Refinement: Localizing Hallucinations via Discrepancies between Dual Independent Rollouts, Leveraging "Discriminative Superiority over Generative Ability" for Self-Correction

This is the core mechanism that replaces the manual auditing of the benchmark with automation while maintaining high quality. It is grounded in the observation that, given the same visual input, discrepancies between two independently generated descriptions are highly likely to be hallucinations, whereas consistencies are likely true visual evidence. The process proceeds in four steps: first, independently generate two descriptions for the same crop segment (Dual Rollout); then, employ an LLM to extract the set of conflicting assertions between the two descriptions (Differential Extraction)—these conflicts bear the highest uncertainty and require verification; next, for each conflicting assertion, randomly shuffle option A, option B, and a distractor option, feeding them to a VLM for a three-way blind-choice judgment (Visual Grounded Judgment). Here, the distractor acts as a calibration anchor; if the referee selects it, the judgment is deemed unreliable and immediately discarded to avoid error propagation. Finally, reliable judgments are integrated back into the original description to yield the refined result. This works effectively because it substitutes "generation" (a relatively weak capability of VLMs) with "discrimination/verification" (a relatively strong capability of VLMs; a motion-adapted counterpart to self-consistency / SelfCheckGPT). Removing bootstrap refinement in the ablation study drops annotation Accuracy from 39.9 to 36.4 (\(-3.5\)), leaving more uncorrected hallucinations in a single-turn generation.

4. Multi-Source Narrative Synthesis + Entity-Centric Motion Score: One for Cross-Event Temporal Coherence, One for Sample Selection

The final step of the four-stage pipeline is Multi-Source Narrative Synthesis: merging the refined local descriptions \(\{ \hat{c}_k \}\) of each event with the global caption \(c_{\text{full}}\) of the full video. The global timeline of \(c_{\text{full}}\) is used to sequence local narratives, eliminating speculative descriptions at event boundaries and redundant details between adjacent events, with conflicts resolved by prioritizing "motion details > scene context." Ablations reveal that omitting the global caption and relying solely on local captions drops Accuracy by 6.7 to 33.2—as local descriptions focus strictly on their own scopes, causing temporal collisions between events; the global caption provides the unifying timeline. Another key, often overlooked design is the Entity-Centric Motion Score in the data filtration stage. Unlike full-video global scoring, it solely measures the motion intensity of the specified entity while suppressing camera motion and global background common-mode movement. It is calculated by taking the optical flow intensity within the mask \(c_t\), subtracting the optical flow intensity of the surrounding band and background area \(\beta_t\), and adding a small absolute term to prevent over-suppressing the true motion of small targets:

\[f_t = \max(0,\; c_t - \beta_t) + \alpha \cdot c_t, \qquad \alpha = 0.3\]

The term \(\alpha c_t\) acts as a critical "safety net signal"—a pure \(c_t - \beta_t\) formulation would completely wipe out the already subtle motion of small targets. The final entity motion score also jointly considers motion intensity (at the 90th percentile) and motion persistence (percentage of frames exceeding the threshold, \(p^\gamma\)) to suppress occasional motion spikes. This filtering suite successfully screens the original 220k samples down to 159k high-quality samples that are "truly challenging to describe."

Loss & Training¶

The training features a straightforward, unified SFT without bells and whistles. It mixes four data types (Table 3): 159k MotionAtlas region-level motion descriptions (to inject entity-centric fine-grained motion capability), 417k Tarsier2-Recap + 130k MotionSight (to preserve general motion understanding), and 320k LLaVA-OneVision-1.5 image-text QA (to prevent catastrophic forgetting). The four sources are concatenated and proportionally sampled by dataset size, running for 1 epoch. The base models are Qwen3-VL(4B/8B) and Molmo2(4B/8B). Full-parameter fine-tuning is conducted with 16 uniformly sampled frames per video, a maximum sequence length of 16384, using AdamW with a peak learning rate of \(1\times10^{-5}\), a cosine learning rate scheduler, and 3% warmup.

Key Experimental Results¶

Main Results¶

On the self-built MotionAtlas-Bench, region-level motion captioning is shown to be extremely challenging, with even the strongest closed-source models struggling to score high on fine-grained details; after fine-tuning with MotionAtlas-Data, all base models consistently achieve substantial gains. The table below lists the Overall Accuracy (%) under the Single-Frame Grounding setting:

Model	Before	Ours	Gain
Molmo2-4B	13.7	21.6	+7.9
Molmo2-8B	19.2	24.4	+5.2
Qwen3-VL-4B	19.3	27.7	+8.4
Qwen3-VL-8B	24.3	31.6	+7.3

A minor detail highlighting the data quality is that the model fine-tuned on Qwen3-VL-8B (31.6 / 34.1) outperforms the un-tuned Qwen3-VL-235B (30.5 / 33.7), validating that small model + high-quality data > large model. Furthermore, Full-Sequence Grounding (which overlays precise masks on every frame) consistently outperforms Single-Frame Grounding (which only provides the first-frame mask and requires the model to track), indicating that continuous spatial prompts indeed bolster fine-grained motion captioning capabilities.

On 7 public motion/video understanding benchmarks, despite the training data being entirely region-level motion descriptions, general non-regional motion tasks are consistently improved (taking Qwen3-VL-4B as an example):

Benchmark	Qwen3-VL-4B	+ MotionAtlas-Data	Gain
MotionBench	55.9	61.9	+6.0
TOMATO	27.4	35.2	+7.8
FAVOR-Bench	47.0	55.0	+8.0
TempCompass	69.6	74.2	+4.6
DREAM-1K (F1)	35.6	38.9	+3.3

The most significant increases occur in TOMATO and FAVOR, which emphasize temporal dynamics and action attribute recognition, aligning perfectly with the fine-grained motion coverage in the data. Importantly, no performance degradation is observed on any of the benchmarks, confirming that no catastrophic forgetting was induced.

Ablation Study¶

Ablation of the three data pipeline components (judged on caption quality over MotionAtlas-Bench, using Accuracy as the metric):

Configuration	Accuracy	Recall	Precision	Description
MA Pipeline (full)	39.9	68.2	58.5	Full Pipeline
w/o Self-Bootstrap	36.4	64.1	56.8	Remove dual-rollout bootstrap, drop 3.5, uncorrected hallucinations
w/o Full-Video Caption	33.2	58.9	56.4	Remove global caption, drop 6.7, temporal conflicts between events
w/o Spatial Crop	32.7	60.9	53.6	Remove spatial crop, drop 7.2, attention distracted by background

Data composition ablation studies (Qwen3-VL-4B, incrementally adding data) are also revealing: adding only general captions already brings gains (e.g., FAVOR 47.0 \(\to\) 52.2), but adding "regional details" yields noticeably larger benefits (FAVOR \(\to\) 55.7, TOMATO 27.4 \(\to\) 31.9). Adding explicit visual regional cues further boosts TOMATO to 35.2. This proves that the performance gains are not mere byproducts of "introducing more action verbs or generic descriptions," but rather that region-grounded supervision is genuinely functioning.

Key Findings¶

Among the three components, spatial cropping leads to the most severe drop (\(-7.2\)), followed by the global caption (\(-6.7\)). This indicates that "zooming in on the target" and "unifying local narratives with a global timeline" are the dual pillars of this automated pipeline; bootstrap refinement (\(-3.5\)) mainly serves to suppress hallucinations.
Region-level supervision cross-transfers to general tasks: The authors explain that MotionAtlas-Data contains highly dense fine-grained cues that are "answerable at 32 frames but unanswerable at 16 frames." This supervision inherently forces the model to digest temporally dense visual evidence. Consequently, the trained model relearns to "consistently benefit from increasing frames from 8 \(\to\) 16 \(\to\) 32," whereas the original un-tuned Qwen3-VL-8B/32B actually degrades at 32 frames (where dense spatio-temporal noise overloads their ability to stably track target regions).
Irreplaceability: Replacing MotionAtlas-Data with an equal volume of TarsierRecap yields almost zero gains on MotionAtlas-Bench (hovering around ~12–13% across all scales), proving that its fine-grained details have no off-the-shelf substitutes. While external benchmarks can improve without this specific data, the gains are consistently smaller.

Highlights & Insights¶

Designing the task by treating "unevaluability" as a first-class citizen: The smartest move of this paper is not modeling, but realizing that "the non-evaluability of global motion description" is the systemic bottleneck holding back data pipelines. Thus, using "region mask constraints + MCQ checklist verification" makes evaluation quantifiable—a classic instance of "changing the exam question being more cost-effective than solving a hard math problem."
Bootstrap refinement converts generative challenges into discriminative tasks: Pinpointing discrepancies between two rollouts as high-risk candidates, then applying a three-way blind referee (augmented with a calibration distractor to verify judgment confidence), takes full advantage of the VLM's characteristic of "having stronger discriminative than generative capabilities." This is a highly transferable trick for any synthetic data pipeline requiring high quality without manual auditing.
The tiny design of Merged Focal Crop (bbox union) accounts for the largest drop in the ablation study, reminding us that crop box stability is far more important than usually assumed in regional video understanding—jitter in frame-by-frame cropping under large displacements severely damages annotation quality.
Pure regional data feedback boosts general capabilities: The authors elucidate why this transfer works through their "answerable at 32 frames vs. unanswerable at 16 frames" diagnostic analysis, rather than settling for a shallow "it just works."

Limitations & Future Work¶

The authors concede that the current setup is confined to single-target scenarios; multi-object interactions and identity correspondence are not yet covered. Upgrading to multi-objective interaction is a natural next step.
Large VLM backbones (Qwen3-VL-235B, Gemini 3 Pro) are solely used for offline data construction, while deployment relies on distilled 4B/8B models. This implies that this high-fidelity annotation capability currently hinges on giant models and cannot be replicated end-to-end online.
Criticisms observed: The benchmark contains only 107 videos. Although the authors assert statistical stability using 2,073 MCQs and subset ranking robustness (\(\rho > 0.90\) at \(1/4\) size), the video-level domain coverage (sports/daily/animals/objects) is relatively narrow; the inter-annotator agreement on aspect-level tags is somewhat low (\(\kappa = 0.47\)), which is why the paper treats it merely as a diagnostic slice rather than an isolated conclusion.
Hyperparameters such as motion scores and clipping bounds (\(\alpha=0.3, \lambda=0.2,\) etc.) are empirically defined, and whether they require retuning during cross-domain deployment is not thoroughly discussed.

vs. Global Motion Captioning (DREAM-1K / Tarsier / ShareGPT4Video): These methods describe the entire scene or the motion of salient objects, whereas this work only describes the motion in masked regions. The core difference lies in leveraging spatio-temporal masks to peel away visual clutter and motion entanglement, securing a quantifiable, hallucination-resistant evaluation at the cost of requiring extra mask annotations.
vs. Fine-Grained Motion Benchmarks (MotionBench / FAVOR-Bench / MotionSight): These also focus on fine-grained motion but mostly employ global annotations, leaving evaluation plagued by the "unscoreable free text" issue. MotionAtlas structures evaluation into a dense diagnostic process using the event \(\to\) fact \(\to\) MCQ checklist, and stands out as the only benchmark that annotates multi-dimensional motion attributes, averaging 19.4 questions per video.
vs. Region-Level Understanding (Osprey / VideoRefer / PixelRefer / Sa2VA): These extend region-level captioning into the spatio-temporal video domain but primarily depict appearance and static attributes. The ablation on data sources directly proves that appearance-heavy data like VideoRefer even underperforms the baseline on motion tasks, accentuating that the "region-level + motion-centric" void is precisely what MotionAtlas bridges.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Neatly bypasses the non-evaluability of global motion captioning through "region mask constraints + MCQ checklists," presenting a clean redefinition of both task formulation and evaluation protocols.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensively verified across 8 benchmarks \(\times\) 4 base models. Multidimensional ablations cover pipeline stages, data composition, scale, judges, and frame counts; the appendix even presents subset ranking robustness runs and annotation funnel statistics.
Writing Quality: ⭐⭐⭐⭐☆ Clear logical chain and informative diagrams. Symbols are slightly dense (particularly the derivation of the motion score in the appendix), but the main text details the mechanism of the four-stage bootstrapping exceptionally well.
Value: ⭐⭐⭐⭐⭐ Fully open-sourced benchmark, data, and models. The finding that pure regional data back-propagates to boost general model capability holds immediate reference value for researchers in video motion understanding.