VIRA-Bench: Diagnosing Pedagogical Contingency Understanding in Online Tutoring Videos¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Multimodal VLM / Educational Dialogue
Keywords: Pedagogical contingency, Multimodal video benchmark, IRF exchange structure, Video-language models, Perturbation diagnostics
TL;DR¶
This paper introduces VIRA-Bench, a diagnostic benchmark of 1,005 clips and 1,322 multiple-choice questions grounded in the IRF exchange structure and contingent scaffolding theory, exposing a 29.7-point gap between leading MLLMs (Gemini 3 Pro at 61.3%) and human experts (91.0%), with paired video-removal and temporal-shuffle controls systematically isolating visual perception from order-sensitive pedagogical contingency reasoning.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have made tremendous strides in static image understanding, general video comprehension, and perceptual question answering. However, contemporary video benchmarks predominantly assess physical event recognition, spatial-temporal object localization, or generic social perception. In parallel, educational NLP has actively developed datasets and benchmarks around teacher-student conversations (such as NCTE Transcripts, MathDial, and MathTutorBench), yet these resources almost entirely rely on textual speech transcripts. This creates a critical computational blind spot: real-world instructional dialogues—particularly in modern online tutoring—are deeply multimodal, where spoken utterances are tightly coupled with live teacher actions on a shared screen, real-time student responses, and subsequent pedagogical scaffolding.
The core tension lies in the gap between isolated event recognition and interactional contingency understanding. A model can readily detect that a teacher circled a word at one timestamp and that a student typed an answer seconds later. Yet, true pedagogical comprehension requires discerning whether that visual action genuinely elicited the learner's response, whether the student's uptake reflects comprehension or confusion, and whether the teacher's follow-up appropriately adapts support based on that specific response. Existing video QA suites rarely test whether models recover these directional dependencies, while standard evaluation metrics often conflate static visual recognition shortcuts, speech-only cues, and true temporal causality into an uninterpretable aggregate accuracy score.
To resolve this limitation, this work grounds benchmark construction in two foundational educational frameworks: the Initiation–Response–Follow-up (IRF) discourse structure and contingent scaffolding theory. By extracting authentic 15–60 second interaction clips from over 1,000 hours of professional online English tutoring, the authors design a controlled multi-stage construction pipeline with paired evidence perturbations. Core idea: decompose multimodal pedagogical contingency understanding into a three-layer diagnostic hierarchy—Observable Evidence Grounding (OEG), Interaction Contingency Reasoning (ICR), and Support-Response Modeling (SRM)—and employ paired video-removal and temporal-shuffling interventions to rigorously isolate visual grounding from order-dependent instructional causality.
Method¶
Overall Architecture¶
VIRA-Bench focuses on 15–60 second clips extracted from authentic one-to-one online tutoring sessions featuring synchronized shared-screen content and webcam feeds. The pipeline integrates coarse-to-fine clip extraction, multi-stage question generation with disjoint construction models, rigorous shortcut and difficulty filtering, and an expert-adjudicated diagnostic evaluation protocol across three distinct cognitive layers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Tutoring Sessions & Audio<br/>1000+ hours 1-on-1 tutoring"] --> B["Coarse-to-Fine Clip Extraction<br/>ASR intervals + VCE visual events"]
B --> C["Structured Evidence Auditing<br/>GPT-4o 5-dimension draft + human audit"]
C --> D["Three-Layer Diagnostic Construction<br/>OEG grounding / ICR contingency / SRM modeling"]
D --> E["Dual Adversarial Filtering<br/>STEM+OPTIONS shortcuts + diagnostic calibration"]
E --> F["Expert Verification & Stratification<br/>360-item double-blind audit"]
F --> G["Paired Perturbation Evaluation<br/>Full / ASR-Only / Visual-Shuffle"]
G --> H["Final VIRA-Bench: 1,005 clips & 1,322 MCQs"]
Key Designs¶
1. Coarse-to-Fine Clip Extraction: Anchoring Interaction Windows to Visual Content Events
To locate pedagogically meaningful exchanges from unconstrained 1–2 hour tutoring sessions, the authors deploy a three-stage extraction pipeline. First, Whisper-large-v3 performs ASR transcription, identifying instructional speech acts (explanations, questions, corrections, elicitation) to narrow each session down to 15–30 minutes of active interaction. Second, Visual Content Event (VCE) detection samples the shared screen at 2 fps, tracking structural similarity (SSIM) and changed-pixel ratios: a drop in SSIM below 0.88 with a pixel difference exceeding 3% triggers a local event (drawing, highlighting, circling, dragging), while differences exceeding 30% flag slide transitions. Cursor-only jitter and duplicate events within 2 seconds are strictly filtered. Third, clip boundaries are dynamically expanded 3 seconds prior to the VCE onset and 12 seconds post-onset (aligned to conversational turn boundaries), yielding 15–60 second windows (mean 32.4 s) centered on a complete teacher intervention, student response, and teacher follow-up exchange.
2. Theory-Grounded Three-Layer Diagnostic Framework: Decoupling Cognitive Operations
Rather than treating pedagogical reasoning as a monolithic skill, VIRA-Bench builds on the ICAP cognitive framework and the IRF cycle to separate questions into three diagnostic layers spanning eight targeted subtasks: - Observable Evidence Grounding (OEG): Evaluates direct perceptual grounding on screen content and participant behavior, comprising Action Identification (AI; recognizing teacher tool usage on screen), Behavior Extraction (BE; identifying student verbal and bodily reactions), and Content Recognition (CR; identifying instructional text or visual objects). - Interaction Contingency Reasoning (ICR): Evaluates the recovery of directional causal relationships across the interaction, comprising Sequence Recognition (SR; determining event order across modalities), Causal Attribution (CA; verifying whether a student response was provoked by a teacher intervention), and Contingent Feedback (CF; identifying whether teacher feedback addresses the preceding student answer). - Support-Response Modeling (SRM): Evaluates predictive and evaluative pedagogical adaptation, comprising Outcome Prediction (OP; predicting learner uptake given instructional cues) and Support Calibration (SC; judging whether instructional assistance is excessive, insufficient, or appropriately scaffolding the learner).
3. Separation of Construction and Dual Adversarial Filtering: Eliminating Unimodal Shortcuts
To prevent models from exploiting superficial text artifacts, question generation strictly isolates construction models from evaluation models. GPT-4o first drafts structured evidence across five typed JSON fields (screen events, student behaviors, teacher speech acts, temporal dynamics, cognitive demand), which researchers audit against raw video to eliminate hallucinations. Next, GPT-5 generates four-option MCQs under strict length parity (\(\le 20\%\) variance from the mean), concrete distractor specificity, and tone consistency constraints. Two held-out validators (GLM-5V-Turbo and Claude-Haiku-4.5) then perform sequential adversarial filtering: under STEM+OPTIONS (stem and choices only, no video or audio), any item solved correctly by both validators is eliminated as a shortcut-exploitable question (10.6% rejected); under the full evidence setting, ambiguous or trivially non-discriminative items are purged (34.4% rejected). Finally, three applied linguistics experts audit 360 stratified items, achieving a 94.4% direct acceptance rate and bounding residual defects to under 2.8%.
4. Paired Modality and Temporal Shuffling Protocols: Isolating Evidence Dependencies
To uncover the exact failure modes of video-language models, the benchmark introduces paired intervention conditions alongside the standard Video+ASR benchmark:
- ASR Text Only: Completely strips visual video streams while preserving verbatim speech transcripts, quantifying language-alone comprehension and isolating the incremental contribution of visual grounding.
- Visual Shuffle with Fixed ASR: Permutes temporally coherent visual event segments while retaining the original ASR transcript, disrupting the chronological sequence of intervention, response, and adjustment while preserving intra-frame visual fidelity.
- Synchronized Video+ASR Shuffle: Permutes visual segments together with their corresponding timestamped ASR spans, maintaining cross-modal local alignment while breaking the global IRF sequence.
Benchmark & Evaluation Protocol¶
The benchmark evaluates 11 commercial and open-source MLLMs: Gemini 3 Pro, Gemini 3 Flash, Qwen3.6-Plus, Qwen3.6-Flash, Qwen3.5-Flash, Kimi-K2.5, MiMo-V2.5, and four variants of Qwen3-VL (235B-A22B and 32B in Instruct and Thinking modes). Videos are processed at native 720p resolution sampled at 1 fps with Whisper-large-v3 transcripts provided as text. A human reference upper bound was established via majority voting across three domain experts on a stratified 200-question subset (25 items per subtask). For every model-item evaluation, option order is randomized with gold labels remapped, eliminating positional selection bias.
Key Experimental Results¶
Main Results¶
Performance across the three diagnostic layers and overall accuracy under the full Video+ASR condition (Accuracy %, 95% Wilson Confidence Intervals):
| Model | Type | OEG Avg | ICR Avg | SRM Avg | Overall Accuracy (All) [95% CI] |
|---|---|---|---|---|---|
| Human Reference† | Expert Majority | 92.0 | 88.0 | 94.0 | 91.0 [86.2, 94.2] |
| Gemini 3 Pro | Proprietary | 63.8 | 57.7 | 63.9 | 61.3 [58.7, 63.9] |
| Gemini 3 Flash | Proprietary | 62.0 | 55.0 | 64.2 | 59.7 [57.0, 62.3] |
| Kimi-K2.5 | Open Source | 60.8 | 50.1 | 64.9 | 57.4 [54.7, 60.1] |
| MiMo-V2.5 | Open Source | 57.7 | 53.7 | 61.1 | 56.9 [54.2, 59.5] |
| Qwen3.6-Plus | Proprietary | 55.1 | 54.3 | 60.1 | 55.9 [53.2, 58.6] |
| Qwen3-VL-235B-Thinking | Open Source | 45.1 | 39.3 | 45.9 | 43.0 [40.3, 45.7] |
| Qwen3.6-Flash | Proprietary | 43.9 | 36.7 | 43.6 | 40.9 [38.3, 43.6] |
| Qwen3-VL-235B-Instruct | Open Source | 39.4 | 34.6 | 41.9 | 38.0 [35.5, 40.7] |
| Qwen3.5-Flash | Proprietary | 21.9 | 24.0 | 30.4 | 24.7 [22.4, 27.1] |
| Qwen3-VL-32B-Thinking | Open Source | 21.3 | 22.7 | 22.3 | 22.1 [19.9, 24.4] |
| Qwen3-VL-32B-Instruct | Open Source | 19.3 | 20.4 | 22.0 | 20.3 [18.3, 22.6] |
Note: †The human reference is based on a stratified 200-question subset with majority voting; random chance baseline is 25.0%.
Evidence Dependency Ablation Study¶
Ablations on representative models across modalities and temporal structures (Reporting accuracy and relative degradation \(\Delta\) in percentage points against the Full setting):
| Model | Setting | OEG Avg (\(\Delta\)) | ICR Avg (\(\Delta\)) | SRM Avg (\(\Delta\)) | Overall (All) |
|---|---|---|---|---|---|
| Gemini 3 Flash | Full (Video+ASR) | 62.0 | 55.0 | 64.2 | 59.7 |
| ASR Text Only | 40.0 (\(-22.0\)***) | 48.0 (\(-7.0\)***) | 56.1 (\(-8.1\)**) | 46.8 | |
| Visual Shuffle (Fixed ASR) | 52.9 (\(-9.1\)***) | 35.9 (\(-19.1\)***) | 59.8 (\(-4.4\) n.s.) | 47.7 | |
| Kimi-K2.5 | Full (Video+ASR) | 60.8 | 50.1 | 64.9 | 57.4 |
| ASR Text Only | 42.9 (\(-17.9\)***) | 47.1 (\(-3.0\) n.s.) | 62.5 (\(-2.4\) n.s.) | 48.9 | |
| Visual Shuffle (Fixed ASR) | 61.8 (\(+1.0\) n.s.) | 36.9 (\(-13.2\)***) | 55.1 (\(-9.8\)***) | 50.3 | |
| MiMo-V2.5 | Full (Video+ASR) | 57.7 | 53.7 | 61.1 | 56.9 |
| ASR Text Only | 43.3 (\(-14.5\)***) | 44.8 (\(-8.9\)***) | 58.4 (\(-2.7\) n.s.) | 47.3 | |
| Visual Shuffle (Fixed ASR) | 54.3 (\(-3.4\) n.s.) | 35.9 (\(-17.8\)***) | 60.1 (\(-1.0\) n.s.) | 48.3 | |
| Qwen3.6-Plus | Full (Video+ASR) | 55.1 | 54.3 | 60.1 | 55.9 |
| ASR Text Only | 38.2 (\(-16.9\)***) | 49.5 (\(-4.7\)*) | 57.8 (\(-2.4\) n.s.) | 47.1 | |
| Visual Shuffle (Fixed ASR) | 54.3 (\(-0.8\) n.s.) | 36.9 (\(-17.4\)***) | 57.1 (\(-3.0\) n.s.) | 48.0 | |
| Qwen3-VL-235B-Think | Full (Video+ASR) | 45.1 | 39.3 | 45.9 | 43.0 |
| ASR Text Only | 32.4 (\(-12.7\)***) | 33.6 (\(-5.7\)**) | 44.9 (\(-1.0\) n.s.) | 35.7 | |
| Visual Shuffle (Fixed ASR) | 43.1 (\(-2.0\) n.s.) | 27.4 (\(-11.9\)***) | 44.9 (\(-1.0\) n.s.) | 37.2 |
Note: Significance evaluated via paired two-sided McNemar tests with Holm correction; \(p < .05\), ** \(p < .01\), *** \(p < .001\), n.s. = not significant.*
Key Findings¶
- Substantial Human-Model Capability Chasm: The best-performing model, Gemini 3 Pro, attains 61.3% overall accuracy, trailing the 91.0% human reference by 29.7 percentage points. Even the human lower confidence bound (86.2%) surpasses the best model by ~25 points.
- ICR is the Universal Bottleneck: Across all competitive models, layer performance consistently exhibits the hierarchical ordering
SRM > OEG > ICR. Models struggle most with Interaction Contingency Reasoning (50.1%–57.7% for the top tier), proving that connecting teacher actions to student uptake and subsequent adaptation is vastly more challenging than static visual detection. - Double Dissociation via Paired Perturbations: Removing the video modality (ASR Text Only) severely degrades Observable Evidence Grounding (dropping 12.7–22.0 points, all \(p < .001\)), while affecting ICR to a significantly lesser extent. Conversely, disrupting visual event order (Visual Shuffle) leaves OEG virtually unharmed (no significant difference in 4 of 5 models) but collapses ICR accuracy (plummeting 11.9–19.1 points, with Sequence Recognition dropping 16–30 points).
- Shortcut Controls Confirm Construct Validity: Under the blind
STEM+OPTIONScondition (no video, no audio), Gemini 3 Flash and Qwen3.6-Plus drop from ~58% to 29.7% and 29.2% (chance baseline 25%), demonstrating that benchmark items cannot be solved using linguistic priors or option-length heuristics.
Highlights & Insights¶
- Theoretically Principled Decomposition: Grounding the benchmark in the IRF exchange structure and contingent scaffolding ensures that questions measure verifiable educational interactions rather than ad-hoc video QA categories.
- Diagnostic Modality Dissociation: The paired perturbation framework provides a methodology for determining whether an MLLM error stems from perceptual blindness, transcript over-reliance, or broken temporal-causal reasoning.
- Robust Quality Engineering: Disjoint construction and validation models, length-variance constraints, and multi-stage expert adjudication restrict residual defective items to fewer than 37 across the entire dataset.
Limitations & Future Work¶
- Domain Scope and Setting: Clips are currently restricted to 15–60 second excerpts from one-to-one online English tutoring, which may not directly generalize to multi-party classrooms, physical STEM laboratories, or longitudinal scaffolding across weeks.
- Format Constraints: Using four-option MCQs facilitates clean statistical significance analysis and benchmarking at scale, but does not measure open-ended pedagogical utterance generation or agentic adaptive interventions.
- Future Directions: Extending the framework to multi-turn conversational agents with interactive rollouts in simulated or live learning environments.
Related Work & Insights¶
- vs Causal-VidQA / NExT-QA / CLEVRER: Prior causal video benchmarks predominantly probe physical mechanics or third-person event causality; VIRA-Bench pioneers interactional causality between human participants in communicative contexts.
- vs Video-MME / MVBench: General video benchmarks are susceptible to static frame shortcuts and temporal pooling; VIRA-Bench's synchronized shuffling proves that temporal order is strictly necessary for solving ICR.
- vs NCTE Transcripts / MathTutorBench: Educational dialogue datasets have historically ignored visual whiteboard manipulations and nonverbal engagement; VIRA-Bench unites visual grounding with dialogic exchanges.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Groundbreaking formulation of multimodal pedagogical contingency understanding with rigorous theory-driven decomposition.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 11 MLLMs, expert human baselines, paired modality/temporal perturbations, and McNemar statistical tests.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, precise mathematical and experimental formulation, and transparent documentation.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable diagnostic benchmark for training and evaluating next-generation multimodal educational agents.