Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

Conference: ECCV2026
Paper: Official page · PDF
Project: https://cfg-bench.github.io/
Area: Robotics and Embodied Intelligence
Keywords: Fine-grained action understanding, cognitive evaluation, counterfactual QA, visual grounding, embodied transfer

TL;DR

CFG-Bench decomposes action understanding into physical interaction, temporal-causal relations, intention, and evaluation, using hybrid QA and counterfactual gating to expose the gap between description and execution knowledge; fine-grained data SFT raises Qwen2.5-VL-7B's average EB-ALFRED result from 4.7 to 9.7.

Background & Motivation

An embodied agent needs more than the next symbolic instruction, such as picking up a cup. It must infer contact locations, coordination between hands, why an action occurs at a particular moment, and whether the current technique is effective. Existing evaluations cover these requirements unevenly: VSI-Bench emphasizes spatial relations, online embodied benchmarks measure planning and control through task outcomes, and action benchmarks such as FAVOR-Bench primarily test the fidelity of an observer's description. Recognizing cup cleaning does not imply noticing that one hand continuously rotates the cup while the other cleans its inner surface, let alone understanding the purpose of that coordination.

The missing piece is not simply another large video question set, but a way to turn executable knowledge into verifiable questions. Multiple-choice questions enable objective scoring of contact points, motion directions, and temporal order. For intentions and strategies, however, options can expose the reasoning path, allowing a model to answer through textual plausibility rather than video evidence. Making everything open-ended introduces answer diversity and evaluation noise, so task-specific formats and reliability checks need to be designed together.

The paper combines existing multi-source videos with human-AI annotation, organizes action knowledge into four cognitive tiers, and assigns multiple-choice, open-ended, and counterfactual questions according to task properties. Core idea: evaluate not only whether a model can describe what happened, but whether it can ground how and why an action is performed in visual evidence and correct a false premise instead of inventing an answer that complies with it.

Method

Overall Architecture

CFG-Bench is a dataset and evaluation protocol, not a new visual encoder or robot control module. The input is an action video and a question; the output is an option or a natural-language answer. On the construction side, Four-Tier Cognitive Tasks specify the required execution details, Human-AI Construction produces annotations and questions, and Hybrid QA and Counterfactual Gating scores model responses. A separate validation route uses the data for SFT to test whether fine-grained knowledge transfers to established embodied tasks.

The benchmark contains 1,368 videos and 19,562 QA pairs, with 9,165 multiple-choice and 10,397 open-ended questions, not an exactly equal split. Its four tiers contain 11 tasks, and not every video supports every task. Video duration ranges from 1.83 to 107.8 seconds, with a mean of 26.3 seconds. The diagram represents benchmark construction and use rather than the internal architecture of a single model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Videos["Multi-source action videos"] --> Taxonomy["Four-Tier Cognitive Tasks"]
    Taxonomy --> Dataset["Human-AI Construction"]
    Dataset --> Evaluation["Hybrid QA and<br/>Counterfactual Gating"]
    Evaluation --> Scores["Task scores and error analysis"]
    Dataset --> Transfer["Fine-Grained Supervision Transfer"]
    Transfer --> Tasks["Planning and manipulation evaluation"]

Key Designs

1. Four-Tier Cognitive Tasks: extending description to execution, intention, and feedback

Inspired by Norman's Action Cycle, the framework organizes action knowledge into four connected tiers. Physical Interaction contains Factual Action Understanding, FAU, and Counterfactual Interaction, CIA. FAU covers not just object identity but left/right hands, contact parts, tools, operation types, and motion dynamics. CIA inserts a nonexistent physical operation into a question and requires correction. Temporal-Causal Relation contains Temporal Relation, TR, Causal Relation, CR, and Counterfactual Relationship, CRS. TR tests ordering, whole-sequence verification, and concurrent actions; CR focuses on an operation's direct consequence; CRS tests whether the model accepts a relationship inconsistent with the video.

Intentional Understanding distinguishes local Functional Intention, FI, Global Intention, GI, linking an entire activity, and Counterfactual Intention, CIT, asking how actions should change under a different overall goal. Evaluative Judgment contains Process Monitoring, PM, Strategy Evaluation, SE, and Counterfactual Evaluation, CEU. These assess whether progress is obstructed or off-target, whether the technique is reasonable and efficient, and how changing an operation would affect its outcome. The progression moves from describing events to judging them, but it is a task taxonomy, not an experimentally established causal hierarchy of abilities.

2. Human-AI Construction: making questions depend on actual video details

Videos come from five datasets: 329 from EgoTaskQA, 191 first-person and 298 third-person videos from Charades-Ego, 394 from EgoExo4d, 104 from Something-Something-V2, and 52 from FineAction. First-person daily activities provide hand-object interactions and goal-directed workflows; third-person views supplement temporal and strategy analysis; the remaining sources broaden precise interaction and complex activity coverage. The first-person emphasis concerns both viewing perspective and execution-oriented questions. It does not mean that every video is egocentric.

GPT-4o drafts action annotations, eight annotators refine them, and two reviewers verify them over an approximately month-long process. For FAU, TR, and CR, GPT-4o initially generates 19,372 multiple-choice questions. Blind and single-frame filtering removes questions answerable from common sense or a static image alone. Qwen3-Max then uses questions and video captions to flag ambiguity or unanswerable items for targeted human review. About 35% are discarded during automated filtering, followed by multiple rounds of checking by six annotators, leaving 9,165 questions; the final retention count should not be attributed entirely to the earlier filter. Open-ended construction starts from 10,397 separately generated multiple-choice seeds, retains only each question and correct answer, and asks ten annotators to expand, revise, or replace them before cross-checking. Anchoring the process in a definite answer before opening up its expression helps constrain ambiguity.

3. Hybrid QA and Counterfactual Gating: limiting option cues and compliant hallucination

Only FAU, TR, and CR use standard multiple-choice evaluation; the other eight tasks are open-ended. Intentional and evaluative tasks do not provide candidate explanations because a model might select a linguistically plausible option without independently constructing visually grounded reasoning. DeepSeek-R1 assists open-ended evaluation by separately scoring Correctness and Detailedness from 0 to 10 and averaging them. Detailedness is not a substitute for factual correctness: a richly worded answer can still describe the wrong action, so both dimensions matter.

CIA and CRS add a prerequisite: the answer must explicitly reject or implicitly correct the false premise, or both dimensions receive zero. CIT and CEU introduce legitimate hypothetical changes and do not use this false-premise gate. The following equation formalizes the prose protocol for clarity; it is an equivalent presentation in this note, not a separate optimization objective introduced by the paper:

\[ S = g\,\frac{C+D}{2},\qquad g = \begin{cases} 0, & \text{CIA/CRS with an uncorrected false premise},\\ 1, & \text{otherwise}. \end{cases} \]

Here \(C\) denotes Correctness and \(D\) denotes Detailedness. The rule separates two superficially fluent behaviors: correcting video facts before answering and assuming that the question must be factually sound. It tests the former rather than generic refusal. Merely saying that a question is wrong does not guarantee a high subsequent score for grounded facts or details.

4. Fine-Grained Supervision Transfer: testing knowledge beyond the question set

The authors apply CFG data SFT to Qwen2.5-VL-7B and InternVL3-8B, then evaluate high-level planning on EB-ALFRED and low-level control on EB-Manipulation within EMBODIEDBENCH. They state that these downstream tasks do not overlap with CFG-Bench and include FAVOR data SFT as a control for Qwen2.5-VL-7B. This is more informative than reporting improvements only on the new question set: it tests whether execution-oriented details and intentions offer more useful supervision than general motion descriptions. It still does not isolate the contribution of each cognitive tier.

For the LIBERO extension, the authors first apply CFG-Bench SFT to Qwen3-VL-4B, then jointly train it with an action expert under the same recipe as their reproduced StarVLA-π baseline. Natural-language answers are therefore not directly used as executable trajectories, and QA fine-tuning alone does not produce a complete controller. The result supports fine-grained knowledge as a useful initialization for VLA training, not a claim of validated real-robot deployment.

A Worked Example

Consider the chain-cleaning scene in the paper's Fig. 2: the person brushes the chain back and forth with a toothbrush in the right hand while rotating the pedal counterclockwise with the left hand. FAU should recover the tool, hand, and motion details. If CIA instead asks how the person turns the pedal with a foot while brushing, the response must correct the foot premise and identify the actual left-hand operation. A detailed but plausible description of foot-powered pedaling still receives a final score of 0 under the gate. The same visual evidence thus plays different roles in ordinary action description and counterfactual verification.

Loss & Training

The main contribution is not a new loss function. The main text specifies supervised fine-tuning and subsequent joint VLA training. It refers to Appendix B3 for SFT settings, but the local cache ends with the references and does not include that appendix. Training mixture proportions, learning rates, epochs, and parameter-freezing policies cannot be verified from the available material and are not supplied here. Zero-shot evaluation follows officially recommended sampling strategies: most models use 4 fps, while GPT-5 uses 1 fps. The main comparison consequently does not impose an identical input budget.

Key Experimental Results

Main Results

The following rows are selected from the paper's Table 2. Avgc is the reported average multiple-choice accuracy (%), and Avgo is the average open-ended score (0–10); they should not be merged into an undefined overall score. Human evaluation does not have a model input setting.

Model	Input	Avgc ↑	Avgo ↑
Human	Not applicable	95.85	9.05
Gemini-3-Pro	4 fps	65.60	5.67
Gemini-2.5-Pro	4 fps	59.52	5.40
GPT-5	1 fps	55.44	4.13
Qwen3-VL-30B-A3B-Instruct	4 fps	57.68	5.09
Qwen2.5-VL-7B	4 fps	40.71	4.38
RoboBrain2.0-7B	4 fps	44.53	4.72

Gemini-3-Pro remains 30.25 accuracy percentage points and 3.38 open-ended score units below humans. The embodied-specialized RoboBrain2.0-7B outperforms its Qwen2.5-VL-7B base, but this is not the paper's CFG data SFT result and must not be confused with the transfer table below.

Ablation Study

The paper does not report module ablations removing individual cognitive tiers or disabling the gate. Verifiable controlled analyses concern frame rate, QA format, and training data. The following is the same-model frame-rate comparison from Table 2.

Qwen2.5-VL-7B input	Avgc ↑	Avgo ↑	FAU accuracy ↑	TR accuracy ↑	CR accuracy ↑
4 fps	40.71	4.38	43.93	38.58	39.63
8 fps	42.40	4.39	45.16	39.51	42.52

Doubling the sampling rate improves the averages by 1.69 accuracy percentage points and 0.01 open-ended score units. Additional frames do not resolve the main difficulty in this setting, but this does not prove that temporal resolution is unimportant for all models.

Table 3 also changes the format of the same tasks: Gemini-2.5-Pro obtains 96.14% on multiple-choice CIT but 5.31/10 on open-ended CIT, while Qwen3-VL-8B-Instruct obtains 90.30% and 5.00/10 on SE. These results suggest that high option-selection performance can conceal difficulty constructing explanations. They are not same-scale percentage drops or network-module ablations.

The training-data comparison from Table 4 follows. Values are average downstream task results (%, higher is better). FAVOR data SFT is reported only for Qwen2.5-VL-7B; there is no corresponding InternVL3-8B row to fill in.

Model and training data	EB-ALFRED Avg ↑	EB-Manipulation Avg ↑
Qwen2.5-VL-7B original base	4.7	9.6
Qwen2.5-VL-7B + FAVOR SFT	4.3	8.2
Qwen2.5-VL-7B + CFG SFT	9.7	15.3
InternVL3-8B original base	10.3	11.5
InternVL3-8B + CFG SFT	14.0	15.8

CFG SFT improves Qwen2.5-VL-7B by 5.0 and 5.7 percentage points, and InternVL3-8B by 3.7 and 4.3 percentage points. Additional LIBERO results are Spatial 98.2→98.2, Object 97.8→98.0, Goal 98.6→99.4, and Long 94.4→95.2. Three of the four splits improve and one remains unchanged, rather than all four improving.

Key Findings

• Reliability has quantitative support: three annotators assess 1,000 samples, yielding Krippendorff's α of 0.95 for multiple-choice answers and 0.82/0.84 for open-ended correctness/detail. Across 4,200 human-LLM scoring samples, the corresponding α values are 0.74/0.75. Replacing the evaluator with Claude-Sonnet-4 yields a model-ranking Spearman correlation of 0.92, supporting relative stability rather than error-free scoring.
• Recurring failures include missing dynamic details, treating coordinated hand movements as sequential actions, inferring goals from simplistic heuristics, and praising inefficient execution simply because the final task succeeds. This last positivity bias shows why task success and execution quality need separate assessment.
• The prose describes local intention outperforming global intention as a trend, but not every model follows it: Gemini-3-Pro's FI/GI scores are 5.29/6.48. This note does not turn that interpretation into a universal empirical claim.

Highlights & Insights

• Counterfactual gating makes correcting a false question a prerequisite for factual scoring. The protocol can inform visually grounded QA audits, but should target false factual premises rather than legitimate hypothetical reasoning.
• Question formats follow cognitive task properties rather than a uniform multiple-choice template. Measurement design helps reduce language shortcuts while preserving objective comparisons for factual tasks.
• The FAVOR-versus-CFG transfer difference suggests that supervision granularity and purpose matter beyond simply adding action text. Stronger follow-up studies should match data volume and training budgets before separating dynamics, causality, intention, and evaluation supervision.

Limitations & Future Work

• Incomplete reproduction material: the current cache lacks appendices with training settings, scoring prompts, and further frame-rate studies, and some figure text is truncated during extraction. Readable main-text tables support the results above but do not enable complete reproduction of SFT.
• QA understanding is not closed-loop execution: embodied transfer and LIBERO provide additional evidence, but the available main text does not report real-robot deployment. Absolute EB-ALFRED and EB-Manipulation results remain low after base-model fine-tuning.
• Prior data exposure is not ruled out: the authors report no direct known overlap for the new questions, but source videos come from public datasets and closed-model pretraining corpora cannot be comprehensively checked. A novel annotation perspective does not itself establish that a model has never seen the videos.
• Evaluator bias remains possible: intention and strategy questions can admit multiple reasonable explanations. Agreement statistics reduce concern without eliminating it; response length, style, and refusal tendencies deserve targeted scoring audits.
• Mechanistic attribution remains limited: there are no reported tier-removal, gate-toggle, or budget-matched repeated-run controls. Future evaluations could combine uncertainty estimates and format-specific scores with robotic failure-recovery rates to test when more detailed explanations yield more reliable execution.

Related Work & Insights

• FAVOR-Bench / MotionBench: these emphasize action and temporal detail, while CFG-Bench extends execution-oriented knowledge to contact techniques, purposes, and process evaluation. FAVOR's transfer control is evidence for this particular setup, not proof that motion descriptions are universally useless for embodied tasks.
• VSI-Bench / ECBench: these target spatial and embodied cognition, while CFG-Bench contributes action-execution knowledge. The benchmarks are complementary: correct spatial relations do not guarantee correct execution techniques.
• EMBODIEDBENCH / LIBERO: these directly test planning or manipulation outcomes, whereas CFG-Bench provides interpretable intermediate diagnostics and supervision. Combining them can help distinguish failures in perception, action understanding, and control.
• Norman's Action Cycle: the cycle organizes goals, execution, and feedback. Its value here is guiding coverage; dependencies between the measured abilities still require intervention studies.

Rating

• Novelty: 4/5. Four cognitive tiers, hybrid formats, and selective gating form a targeted evaluation contribution rather than a model-architecture innovation.
• Experimental Thoroughness: 4/5. Multiple models, human agreement, and downstream transfer provide substantial evidence, but component-level causal ablations and repeated-run statistics are missing.
• Writing Quality: 3/5. The task organization is clear, although some trend statements have table-level exceptions and appendix details are unavailable in the current cache.
• Value: 4/5. Useful for diagnosing action-knowledge gaps and designing fine-grained supervision, but not a replacement for execution-success evaluation.