EatVid-Bench: A Multimodal Fine-Grained Eating Behavior Video Dataset

Conference: ECCV 2026
Paper: Official page · PDF
Area: Video Understanding
Keywords: Eating behavior, temporal grounding, video question answering, traceable annotation, annotation-guided reasoning

TL;DR

EatVid-Bench turns 682 real-world eating videos into traceable hierarchical behavioral annotations and 8,184 QA pairs, shows that strong health-related descriptions can coexist with weak objective perception, and uses AGCoT-LoRA to improve Qwen2.5-VL-7B's objective weighted score from 0.1623 to 0.3214 on the 64-frame Hard subset.

Background & Motivation

Recognizing that someone is eating is different from understanding how many bites they took, when their pace increased, and whether pauses and expressions support a particular behavioral interpretation. General benchmarks such as Video-MME, MLVU, and MVBench cover diverse activities but do not isolate short-timescale localization and repetitive event counting during eating. Ego4D and EPIC-Kitchens emphasize activities, object interactions, and action semantics rather than a complete evidence chain from intake events to behavioral interpretations. For models potentially used in dietary monitoring, a professional-sounding answer does not establish accurate visual understanding.

Eating makes this distinction especially easy to obscure with language ability. A hand-to-mouth movement is brief but repeats throughout a video lasting several minutes. Food categories need cross-frame aggregation, pace depends on intervals between intake events, and health-related descriptions combine these observations. If evaluation only scores the final paragraph, generic advice about slower eating or a balanced diet may earn credit without identifying individual bites. Manually annotating all geometric, behavioral, and semantic signals is expensive, while unconstrained automatic generation can package upstream errors as convincing answers.

The paper therefore connects annotation and evaluation: specialized perception tools produce inspectable signals, behavioral modules quantify them, semantic interpretation is constrained by those records, and QA pairs are generated from the resulting annotations. Existing pose, segmentation, depth, and multimodal models make this combination practical, but evaluation labels still require human review. Core idea: trace both QA labels and supervised reasoning outputs back to concrete actions, timestamps, and statistics, and separate objective from open-ended scores to test whether models actually rely on video evidence.

Method

Overall Architecture

EatVid-Bench is a dataset and evaluation system; AGCoT-LoRA is its domain-adapted baseline, not a new video encoder. Inputs are curated web videos or voluntarily recorded eating sessions. The pipeline performs foundational perception, behavioral quantization, and traceable semantic interpretation, constructs hierarchical QA, and uses annotation traces from the training portion to supervise explicit reasoning outputs.

Approximately 2,000 raw sessions yield 856 valid sessions, with final statistics of 682 videos and 8,184 QA pairs. Videos average 4.3 minutes and total nearly 3,000 minutes. Selection favors one main participant, visible facial regions, and visible hand-food interactions, with complete eating episodes captured where possible. Scenes are home 60%, cafeteria or restaurant 30%, and other settings 10%; utensils are chopsticks 42%, spoons 34%, forks 19%, and hands 5%. These choices improve annotatability but also limit what the dataset's real-world coverage represents.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real-world eating videos"] --> B["Foundational Perception"]
    B --> C["Behavioral Quantization"]
    C --> D["Traceable Semantic<br/>Interpretation"]
    D --> E["Hierarchical QA<br/>and Scoring"]
    E -->|Training annotations| F["Annotation-Guided<br/>Adaptation"]
    E -->|Evaluation portion| G["Capability-specific diagnosis"]
    F --> G

Key Designs

1. Foundational Perception: retain physical signals that can be inspected

Tier 1 does not immediately ask a large model whether the meal is healthy. It extracts the structures needed to support later judgments. MediaPipe supplies 33 body keypoints in 3D, 21 keypoints per hand, and 468 facial landmarks. Grounding DINO detects and categorizes food, SAM refines food masks, and MiDaS supplies monocular depth cues. The food taxonomy contains 9 coarse categories and finer subcategories; uncertain fine-grained predictions fall back to a coarse category instead of forcing a specific food identity.

Whisper-v3 transcribes audio synchronized with the visual stream, and the pipeline also uses swallowing-related acoustic peaks. Transcription, acoustic peak detection, and synchronization are distinct operations. The main text does not provide a complete acoustic detection implementation, so Whisper itself should not be presented as a validated swallowing detector. This tier makes downstream claims traceable to inputs; it does not make predicted poses, categories, masks, or depth automatically trustworthy ground truth.

2. Behavioral Quantization: convert repetitive micro-actions into event timelines and statistics

Tier 2 locates intake events from the changing Euclidean distance between wrist and nose-tip landmarks. A local distance minimum below an adaptive threshold anchors a candidate bite, with its timestamp and hand laterality recorded. This converts a generic hand-near-mouth observation into a queryable event, but face touching, occlusion, and unusual viewpoints can still introduce errors. The main text does not specify a deduplication window or threshold parameters, so these details cannot be filled in from assumption.

Chewing frequency is estimated from mandibular oscillations reflected in facial landmarks and combined with swallowing-related audio cues. The distribution of intervals between consecutive bites describes eating pace. By the definition of this concept, the interval between adjacent timestamps can be written as follows; this is explanatory notation in the note, not a new learning objective introduced by the paper:

\[ \mathrm{IBI}_j=t_{j+1}-t_j. \]

Bite density further divides sessions into preparation, active, peak, and rest phases. Food predictions are aggregated over time, and temporal category proportions feed a Shannon-entropy-based Nutritional Diversity Index. This measures diversity of category occurrence, not consumed mass or nutrient balance. Equation (1) is corrupted in the cached source, so its expression and logarithm base are not reconstructed here. Before-and-after food-mask changes combined with depth provide only a coarse intake-volume proxy, not precise calorie estimation.

3. Traceable Semantic Interpretation: constrain higher-level descriptions with lower-level evidence

Tier 3 uses high-capacity multimodal models to integrate Tier 1-2 signals, action and expression segments, and a compact knowledge base drawn from behavioral psychology research. Instead of freely producing a health report, each interpretation is linked to provenance metadata. The text illustrates this with an average IBI below 15 seconds supporting an elevated-pace statement. This is the paper's provenance example, not a universal clinical threshold. Expressions and pace can at most support constrained behavioral interpretations; video alone cannot establish psychological or metabolic diagnoses.

Automatic annotation is followed by quality control. All Tier 2-3 annotations used in evaluation splits are manually checked and corrected, and Tier 1 signals affecting QA construction undergo source-level audits. Three trained annotators perform double-blind review with third-annotator arbitration, achieving \(\kappa=0.85\) for Tier 2 and \(\kappa=0.82\) for Tier 3. Semantic outputs also undergo structural parsing and a second GPT-4o review. Provenance helps locate the origin of an error, but does not eliminate shared biases across perception and semantic models.

4. Hierarchical QA and Scoring: measure perceptual correctness separately from fluent answers

A template engine queries annotation modules and maps events, statistics, and categories into inspectable questions and answers. The seven capability dimensions are bite detection, chewing analysis, action timeline, facial expression, dietary composition, eating pace, and behavioral health. Five formats cover single choice (SC), true/false (TF), multiple choice (MC), fill-in-the-blank (FB), and short answer (SA). L1 concerns an individual attribute or short action, L2 aggregates multiple timestamps, and L3 combines multimodal and long-range evidence. Their approximate count ratio is 9:2:1. L3 implies a broader evidence scope, not necessarily greater difficulty under the current scorer.

SC, TF, and FB use exact or regular-expression matching; MC uses F1 between predicted and ground-truth option sets. GPT-4o scores SA responses from 1-10 on temporal grounding, nutritional completeness, reasoning validity, and fluency. The paper reports raw Acc, difficulty-weighted WAcc, and separate SA-WAcc and Non-SA-WAcc. The main text does not fully specify difficulty weights, four-axis aggregation, or normalization. Consequently, WAcc cannot be recomputed from this cache alone and should not be treated as ordinary accuracy.

To examine judge dependence, the authors rescore short answers with Gemini, Claude, and humans, obtaining Spearman correlations of 0.89, 0.87, and 0.82 with GPT-4o. Objective questions do not use an LLM judge. Table 2 also reports a chance Non-SA-WAcc anchor of 22.0, text-only SA-WAcc of 58.2, and human WAcc of 72.0 with Non-SA-WAcc of 88.0; the human anchor is labeled \(n=30,k=3\). These anchors warn that language priors contribute to scores, although their complete sampling details cannot be recovered from the main text.

5. Annotation-Guided Adaptation: supervise evidence derivation rather than answers alone

AGCoT-LoRA starts from Qwen2.5-VL-7B and constructs explicit reasoning supervision from Tier 1-2 annotations in the training portion. A target can first list bite timestamps, then calculate intervals, and finally answer a pace or counting question. Unlike answer-only supervision, this asks the output to preserve intermediate evidence connecting observations to conclusions. LoRA provides parameter-efficient domain adaptation. The method constrains an observable output process using external annotations; it does not prove that the model's internal reasoning is faithful.

The important comparison is plain LoRA trained on the same data without annotation-guided reasoning targets. This separates the benefit of exposure to eating-domain data from the benefit of learning annotation traces, making it more informative than a before-versus-after fine-tuning comparison alone. Nevertheless, generating timestamps does not establish that those timestamps came from visual evidence. Objective questions and additional evidence interventions are still needed.

Loss & Training

The main text identifies Qwen2.5-VL-7B, LoRA, and annotation-derived reasoning supervision, but does not supply a fully reproducible loss formulation, LoRA rank, learning rate, epoch count, or training-split size. These details are referred to supplementary material. The local cache contains only the main paper and references, so no hyperparameters or separate localization loss are assumed here.

Evaluation is zero-shot by default; AGCoT-LoRA and the plain LoRA control are fine-tuned exceptions. The main evaluation uses 32 uniformly sampled frames and excludes sessions longer than 5 minutes. The main Hard-subset table uses 64 frames. The authors also describe controls with 32/64 frames and 1/2/4 FPS, stating that denser sampling offers only limited improvement. Their complete numerical tables are not in this cache, so this statement cannot establish that every model receives equivalent temporal evidence.

Key Experimental Results

Main Results

The tables retain the original 0-1 scale, with higher values better. The following selection comes from Table 3: \(N=6,300\) evaluation questions and 32 frames. This is not the dataset-wide total of 8,184 QA pairs, and the main text does not fully document the filtering correspondence between them.

Model	Acc	WAcc	SA-WAcc	Non-SA-WAcc
InternVL2-8B	0.3787	0.6537	0.8462	0.2843
Qwen2.5-VL-7B	0.3310	0.6300	0.8323	0.2415
VideoLLaMA2-7B	0.3332	0.5530	0.7101	0.2516
VideoLLaMA3-7B	0.3868	0.4678	0.5404	0.3284

InternVL2 has stronger WAcc, whereas VideoLLaMA3 has a higher objective score. Aggregate rankings therefore cannot replace capability-specific diagnosis. Qwen2.5-VL achieves 0.8323 SA-WAcc but only 0.2415 Non-SA-WAcc. This separation conveys the paper's research problem more accurately than a claim of strong overall understanding.

The next selection is from Table 4: the Hard subset has \(N=376\), uses 64 frames, and includes fine-tuned models. It is constructed from task instances receiving consistently low scores across an initial model group, not from an independent random sample of the overall test population.

Model	Acc	WAcc	SA-WAcc	Non-SA-WAcc
Qwen3.5-35B-A3B, API	0.4388	0.7359	0.9667	0.2982
InternVL2-8B	0.2606	0.5868	0.8000	0.1823
Qwen2.5-VL-7B	0.2500	0.5798	0.8000	0.1623
AGCoT-LoRA, 7B	0.4069	0.6130	0.7667	0.3214

Relative to the base model, AGCoT-LoRA increases Acc by 0.1569, or 15.69 percentage points, and Non-SA-WAcc by 0.1591, or 15.91 percentage points. However, SA-WAcc decreases by 0.0333. Its objective score exceeds the API model's score in this table, but its WAcc does not; this is not evidence of uniformly outperforming the larger model.

Ablation Study

Section 5.4 reports the following matched Hard-subset comparison at 4 FPS, in percent. This is not the 64-frame protocol above, so the difference between 36.4% and 32.14% must not be attributed to reasoning supervision.

Configuration	Non-SA-WAcc (%)	Comparison meaning
Qwen2.5-VL base	14.1	No domain adaptation
Plain LoRA	22.3	Same training data, without annotation-guided reasoning targets
AGCoT-LoRA	36.4	Adds annotation-guided reasoning supervision

Plain LoRA improves on the base by 8.2 percentage points. AGCoT-LoRA adds another 14.1 percentage points over plain LoRA and gains 22.3 percentage points over the base. This supports additional value from explicit annotation-trace supervision, but does not separately ablate pose, audio, depth, or the knowledge base.

Key Findings

• Strong SA scores do not establish accurate perception. Both the full evaluation and Hard subset show substantial separation between short-answer and objective performance, while the text-only anchor further indicates the influence of language priors.
• Supervision does not improve every metric simultaneously. On the 64-frame Hard subset, improved objective performance accompanies lower SA performance; this trade-off must remain visible rather than being replaced by one favorable gain.
• Difficulty comes from both input coverage and evaluation design. Sparse frames can miss brief bites while fluent L3 answers still receive high scores. The reported results do not fully separate model capability, sampling coverage, and scorer preferences.

Highlights & Insights

• Provenance records become supervision resources. The same event timelines support label audits and reasoning targets, reducing the scope for correct-looking answers with unsupported explanations.
• Open-ended and objective questions are reported side by side. Aggregate scores may reward linguistic completeness, whereas objective questions expose count and category errors more directly; both should inform model assessment.
• Plain LoRA is the essential control. Domain data itself improves performance, so matched-data ablation is necessary to establish an additional contribution from annotation-guided supervision.

Limitations & Future Work

• The authors acknowledge limited cultural and dietary coverage, and exclude precise calorie or nutrient-gram estimation from scope. Behavioral and affective interpretations are not clinically validated diagnoses.
• Automatic perception and semantic integration can still propagate errors. Human review and judge correlations reduce some risks, but correlation does not establish clinical validity or exclude shared language biases.
• Long-video conclusions are constrained by the main protocol. The dataset includes sessions exceeding 20 minutes, but the primary evaluation excludes videos longer than 5 minutes. Its main tables therefore do not validate full understanding of extended meals.
• Reproducibility details remain incomplete. The mapping from 8,184 QA pairs to 6,300 evaluation questions, WAcc weights, training splits, and hyperparameters are not fully specified. Table 2 lists category proportions for only six categories despite the seven-dimensional taxonomy; chewing is not separately accounted for.
• Future work should include evidence masking, temporal shuffling, event-level localization, and cross-cultural validation, and document participant- or source-disjoint splitting. These checks would test visual dependence more directly than higher SA scores alone; the main text does not provide complete quantitative results for such interventions.

Related Work & Insights

• Compared with OREBA and EatSense: prior eating research emphasizes intake events, action localization, or movement quality. EatVid-Bench extends the task to provenance-linked video QA and behavioral interpretation, gaining a diagnostic chain at the cost of more complex annotation and quality control.
• Compared with Video-MME, MLVU, and MVBench: general benchmarks offer broader coverage. EatVid-Bench concentrates on eating-specific details and domain transfer, but cannot replace general capability evaluation.
• Compared with Ego4D, EPIC-Kitchens, and temporal grounding benchmarks: the distinctive contribution is connecting event timelines to dietary questions, not simply increasing video duration. The transferable idea is to preserve explicit low-level evidence behind higher-level questions.

Use the official page and PDF linked above for resources. The paper states that code and benchmark are available on GitHub and HuggingFace, but the cache does not preserve specific addresses and the manifest supplies no project link. No repository or dataset URL is guessed here.

Rating

• Novelty: 4/5. Combining traceable eating-domain QA with annotation-guided supervision is useful, although the underlying tools and LoRA are not new algorithms.
• Experimental Thoroughness: 3/5. Multiple models, disaggregated metrics, human audits, and a plain LoRA control are useful, but some protocol details and causal evidence interventions are missing from the main text.
• Writing Quality: 3/5. The central problem is clear, but dataset scale, question distributions, and evaluation-set accounting need further clarification.
• Value: 4/5. A useful diagnostic benchmark for domain-specific video understanding and evidence dependence, not a ready-to-deploy clinical assessment system.