CausalChapter: Improving Long-Form Video Chaptering with Interventional Dependency Modeling¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Video Understanding
Keywords: Video Chaptering, Causal Intervention, Dependency Shift, Cross-Segment Support, Dense Video Captioning
TL;DR¶
To tackle boundary error propagation and cross-segment context fragmentation in scalable segment-then-caption long-form video chaptering, CausalChapter introduces intervention-inspired predictive dependency modeling via Local Dependency Shift (LCDS) to detect smooth boundary transitions and Cross-Segment Support Selection (CSSE) to rerank historical contexts by output KL divergence shift, substantially outperforming prior state-of-the-art systems on AVLecture and VidChapters-7M.
Background & Motivation¶
Long-form instructional and educational videos, such as academic lectures, meeting recordings, and online tutorials, contain dense knowledge structures and intricate pedagogical reasoning. Partitioning these untrimmed videos into semantically coherent temporal chapters paired with accurate, navigable descriptions is vital for content browsing, indexing, and knowledge retrieval. Driven by rapid progress in long-context language models, recent approaches like Chapter-Llama convert multi-hour videos into timestamped Automatic Speech Recognition (ASR) transcripts and frame-level captions, delegating boundary localization and title generation to a long-context LLM in a single holistic forward pass. However, for information-dense lecture videos characterized by lengthy transcripts, gradual topic transitions, and detailed multi-sentence chapter descriptions, full-video joint modeling quickly strains token capacity and video memory budgets, forcing practical deployments to rely on aggressive truncation or sparse downsampling that discards fine-grained evidence essential for boundary and description precision.
To ensure computational scalability across hour-long recordings, the field has increasingly adopted a decoupled segment-then-caption paradigm: a boundary detector first partitions the long video into short local segments, and a generative language model subsequently describes each segment independently. While this decomposition prevents memory explosion and aligns segment inputs with chapter-level generation, it introduces two acute structural bottlenecks: first, boundary localization errors propagate monotonically into description generation, where slight temporal offsets blur adjacent topic boundaries or misplace key derivations; second, isolated segment-level generation severs long-range semantic dependencies, depriving the generator of prerequisite definitions, formal assumptions, or experimental setups introduced earlier in the lecture. Standard retrieval-augmented generation (RAG) pipelines attempt to retrieve relevant history using lexical similarity, semantic embeddings, or temporal proximity, but in technical lectures, surface-level similarity often retrieves repetitive restatements while missing logically indispensable foundations that share minimal vocabulary.
This observation exposes a fundamental misalignment: temporal units in long-form videos should not be judged merely by surface similarity or temporal closeness, but by whether intervening on them meaningfully alters the model's predictive behavior. Core idea: introduce intervention-inspired dependency modeling into the segment-then-caption chaptering pipeline, using Local Dependency Shift (LCDS) to capture sharp drops in predictive dependency between adjacent temporal windows for smooth boundary localization, and Cross-Segment Support Selection (CSSE) to rerank historical contexts based on the KL divergence shift of output predictive distributions under leave-one-out removal interventions.
Method¶
Overall Architecture¶
CausalChapter operates on a scalable two-stage segment-then-caption pipeline enhanced with intervention-based predictive signals. Given a long video paired with sentence-level ASR transcripts \(U = \{u_i\}_{i=1}^N\), the model extracts 1 FPS video frames within each sentence boundary, projects visual CLIP embeddings and textual tokens into a shared space, and fuses them into multimodal semantic units \(z_i\). In the boundary localization stage, a base classifier predicts transition probabilities, while the Local Dependency Shift (LCDS) module applies sequential masking interventions across preceding temporal windows to estimate local dependency drops, which are adaptively added to the boundary logits to pinpoint smooth transitions. Following temporal segmentation, an LLM generator drafts preliminary chapter descriptions. The Cross-Segment Support Selection (CSSE) module builds a compact candidate context pool from temporal neighbors and semantic retrieval, performs leave-one-out removal interventions, and measures the output distribution change via KL divergence to identify the most supportive historical segments. These selected segments are then fed into a second-pass generation prompt to produce the final refined chapter descriptions.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Long Video & Timestamped ASR Transcripts"] --> B1["Semantic Unit Encoding<br/>Fuse sentence ASR with frame CLIP into z_i"]
B1 --> B2["Base Boundary Probability Prediction<br/>Lightweight classifier outputs base score p_base"]
B1 --> D1["Local Causal Dependency Shift (LCDS)<br/>Sliding-window masking intervention detects dependency drop"]
B2 & D1 --> B3["Enhanced Boundary Decision<br/>Combine logits via learnable gamma to obtain segments S_k"]
B3 --> G1["First-Pass Draft Generation<br/>Segment multimodal prompt produces initial draft y_k~"]
B3 & G1 --> G2["Candidate Context Pool Construction<br/>Retrieve compact set from temporal neighbors and similarity"]
G2 --> D2["Cross-Segment Support Selection (CSSE)<br/>Leave-one-out removal intervention evaluates KL shift"]
D2 --> D3["Two-Pass Refinement Generation<br/>Refine chapter description using top supportive contexts"]
D3 --> OUT["Final Grounded Chapter Sequence {s_k, e_k, y_k}"]
Key Designs¶
1. Local Causal Dependency Shift (LCDS): Detecting Boundary Dependency Drops under Smooth Transitions In academic lectures and structured tutorials, instructors transition between topics with smooth, conversational phrasing and persistent slide templates, causing conventional visual difference thresholds or text contrastive loss objectives to overlook genuine topic boundaries. Targeting this failure mode where surface continuity masks semantic transitions, LCDS operationalizes chapter boundaries as positions where predictive dependency between adjacent temporal windows exhibits an abrupt local cliff. Concretely, for a candidate boundary after unit \(u_i\), LCDS defines a preceding window \(A_i = [z_{i-W+1}, \dots, z_i]\) and a following window \(B_i = [z_{i+1}, \dots, z_{i+W}]\) with default window size \(W=5\). An MLP-based dependency predictor \(R_\phi\) reconstructs the following window from the preceding one: \(\hat{B}_i = R_\phi(A_i)\). LCDS then systematically masks the \(r\)-th unit in \(A_i\) to yield \(A_i^{\backslash r}\) and computes the perturbed prediction \(\hat{B}_i^{\backslash r} = R_\phi(A_i^{\backslash r})\). The overall predictive dependency is quantified by the average cosine distance shift across all units: $\(d_i = \frac{1}{W^2} \sum_{r=1}^W \sum_{j=1}^W D_{rep}(\hat{b}_{i,j}, \hat{b}_{i,j}^{\backslash r})\)$ Within a continuous chapter topic, preceding units provide strong predictive support for subsequent units, sustaining a high \(d_i\); at true boundaries, topic divergence triggers an abrupt drop in \(d_i\). Comparing \(d_i\) against its local neighborhood average \(\bar{d}_i\), the dependency-shift score is normalized as \(s_i^{dep} = \text{Norm}(\max(0, \bar{d}_i - d_i))\) and directly injected into the boundary classifier logit \((o_i^{enh})_{c_b} = (o_i^{base})_{c_b} + \gamma s_i^{dep}\) via a learnable parameter \(\gamma\), effectively preventing boundary under-segmentation without incurring heavy sequence modeling overhead.
2. Cross-Segment Support Selection (CSSE): Estimating Context Utility via Leave-One-Out Intervention Segment-level generation isolates chapters from their broader narrative arc, yet naรฏve similarity-based retrieval often pulls lexically redundant paragraphs or misleading entities that provoke hallucinations. To address the discrepancy between surface similarity and actual generation utility, CSSE uses leave-one-out removal interventions to directly quantify how much each historical segment influences the language model's generation distribution. First, a lightweight candidate context pool \(C_k = C_k^{near} \cup C_k^{sem}\) is formed by combining adjacent temporal segments with top-retrieved semantic segments (budget \(M=5\)), restricting interventions to a compact subset. The generator LLM \(G_\theta\) initially produces a reference sequence \(y_k^{ref} = G_\theta(I_k, \tilde{y}_k, C_k)\) under the full candidate set. For each candidate \(S_j \in C_k\), CSSE constructs an ablated context \(C_k^{-j} = C_k \setminus \{S_j\}\) and computes the teacher-forced token-level KL divergence over the reference prefix: $\(s_{j \to k} = \frac{1}{T} \sum_{t=1}^T D_{KL}\left(p_\theta(\cdot \mid y_{k,<t}^{ref}, I_k, \tilde{y}_k, C_k) \,\parallel\, p_\theta(\cdot \mid y_{k,<t}^{ref}, I_k, \tilde{y}_k, C_k^{-j})\right)\)$ A larger divergence signifies that removing \(S_j\) drastically alters the predictive distribution of the generator, establishing that \(S_j\) provides genuine causal support for the current chapter's content. The candidate segments are reranked by \(s_{j \to k}\), and the top-\(K\) most supportive segments \(R_k\) are retained, ensuring that retrieved history delivers foundational premises rather than superficial vocabulary overlap.
3. Two-Pass Refinement Generation: Grounded Description Polishing Anchored by Initial Drafts Naively concatenating retrieved long-range historical text into an LLM generation prompt frequently leads to context distraction, where the model incorporates distant details at the expense of describing the immediate segment. To prevent context contamination while preserving critical technical terminology, CausalChapter introduces a structured two-pass prompting protocol. In Pass 1 (Segment-Level Generation), the LLM processes the current segment transcript, visual summaries, and temporal prompts in isolation to generate an initial draft title \(\tilde{y}_k\) focused entirely on the local timeline. In Pass 2 (Temporal Refinement), the model receives the prompt augmented with both the supportive historical segments \(R_k\) and the initial draft \(\tilde{y}_k\). Explicit system guidelines instruct the model to treat retrieved history strictly as background clarification for topic transitions, forbid direct copying of past titles, and enforce the preservation of verified technical terms from \(\tilde{y}_k\) unless a fundamental conceptual omission is detected. This anchored refinement strategy balances local descriptive precision with global narrative coherence.
Loss & Training¶
The overall training objective of CausalChapter is optimized end-to-end via a multi-task loss formulation: $\(\mathcal{L} = \mathcal{L}_{bd} + \mathcal{L}_{rec} + \mathcal{L}_{gen} + \mathcal{L}_{time}\)$ where \(\mathcal{L}_{bd} = -\sum_i \log p_i^{enh}(b_i^*)\) optimizes the enhanced boundary classification cross-entropy; \(\mathcal{L}_{rec} = \sum_i D_{rec}(\hat{B}_i, B_i)\) supervises the MLP dependency predictor \(R_\phi\) to accurately reconstruct subsequent temporal windows from preceding units; \(\mathcal{L}_{gen}\) represents the standard autoregressive cross-entropy loss over the reference chapter descriptions given ground-truth boundaries and supportive contexts; and \(\mathcal{L}_{time} = -\sum_{t \in \Omega_{time}} \log p_\theta(y_t \mid y_{<t}, I_k)\) is an auxiliary timestamp reconstruction loss designed to strengthen temporal grounding within the LLM backbone.
Key Experimental Results¶
Main Results¶
CausalChapter is evaluated on the primary long-form instructional benchmark AVLecture (350 lecture videos split into 245 train / 35 validation / 70 test, augmented with human-verified reference descriptions) and the large-scale open-domain benchmark VidChapters-7M. On AVLecture, CausalChapter establishes new state-of-the-art performance across all evaluation metrics, substantially outperforming fine-tuned Chapter-Llama and conventional dense video captioning (DVC) models:
| Method Type | Model | Generation Backbone | CIDEr | SODA_c | F1@30 (%) | tIoU (%) |
|---|---|---|---|---|---|---|
| DVC | PDVC | โ | 6.11 | โ | 9.07 | 56.44 |
| DVC | Vid2Seq | T5 | 61.22 | 9.79 | 9.84 | 53.77 |
| Video LLM | VTimeLLM | Vicuna-7B (ZS) | 0.02 | 3.04 | 47.53 | 46.72 |
| Chaptering | Chapter-Llama | LLaMA-3.1-8B | 99.78 | 8.15 | 63.93 | 62.18 |
| Chaptering | Chapter-Llama | Qwen2.5-7B | 95.36 | 6.91 | 57.73 | 59.62 |
| Chaptering | CausalChapter (Ours) | LLaMA-3.1-8B | 110.12 | 11.73 | 72.97 | 69.95 |
| Chaptering | CausalChapter (Ours) | Qwen2.5-7B | 116.88 | 13.80 | 72.97 | 69.95 |
| Proprietary LLM | GPT-4o | Zero-Shot Prompting | 1.82 | 0.08 | 24.89 | 35.23 |
| Proprietary LLM | Gemini-2.5-Pro | Zero-Shot Prompting | 2.40 | 1.64 | 26.72 | 36.12 |
| Proprietary LLM | Gemini-2.0-Flash | Zero-Shot Prompting | 0.28 | 0.18 | 11.71 | 13.54 |
On the open-domain VidChapters-7M benchmark, CausalChapter with Qwen2.5-7B reaches 109.2 CIDEr, 18.8 SODA, 46.2% F1, and 72.0% tIoU, surpassing Chapter-Llama (100.9 CIDEr / 45.3% F1). This confirms that interventional dependency modeling generalizes effectively from lecture recordings to broad user-generated YouTube content.
Ablation Study¶
The core modules are systematically ablated on AVLecture using a standardized Qwen2.5-3B backbone under predicted boundaries:
| Experimental Variant | CIDEr | METEOR | SODA_c | F1 (%) | BS@30 (%) | tIoU (%) | Note |
|---|---|---|---|---|---|---|---|
| Backbone (Base Seg-then-Cap) | 88.39 | 16.83 | 10.84 | 52.95 | 59.61 | 66.36 | Standard segment-then-caption model |
| + LCDS | 91.36 | 17.37 | 11.90 | 56.72 | 63.02 | 69.95 | Boundary localization isolated boost |
| + CSSE | 98.40 | 17.15 | 12.63 | 52.95 | 59.61 | 66.36 | Generation isolated boost with fixed bounds |
| Full Model (CausalChapter) | 104.59 | 17.87 | 12.73 | 56.72 | 63.02 | 69.95 | Optimal synergy across all metrics |
In dedicated module comparisons, LCDS outperforms representation similarity drop (Sim. Drop: F1=54.51%, tIoU=65.68%) and contrastive learning objectives (CL Loss: F1=53.28%, tIoU=65.23%) by achieving F1=56.72% and tIoU=69.95%. Similarly, CSSE surpasses temporal proximity (Previous-K: CIDEr=90.85), semantic retrieval (Sim. Top-K: CIDEr=95.97), and LLM direct relevance scoring (LLM Scoring: CIDEr=96.11) with a peak CIDEr of 104.59, verifying the superiority of leave-one-out output divergence over surface-level similarity metrics.
Key Findings¶
- Predictive Dependency Cliffs Counteract Boundary Smoothing: Integrating LCDS elevates boundary tIoU from 66.36% to 69.95% and exact-boundary F1 from 52.95% to 56.72%. While adjacent audio-visual features remain continuous across gradual transitions, the ability of preceding frames to predict subsequent representations drops sharply at true thematic turning points.
- Surface Similarity Induces Mechanism Drift: Naive semantic retrieval (Sim. Top-K) achieves only 95.97 CIDEr, lagging behind CSSE's 104.59. Qualitative audits reveal that similarity matching frequently pulls later segments repeating identical technical phrases, which distracts the model into confusing current derivations with earlier steps. CSSE accurately selects antecedent definitions and setup phases.
- Favorable Latency and Memory Trade-Off: Profiling on an NVIDIA A100 GPU demonstrates that intra-video micro-batching (batch size 8) reduces total end-to-end inference latency to 10.50 seconds per lecture (peak memory 19.35 GiB), where CSSE accounts for 4.59 seconds (43.7% of total runtime). Under 4-bit NF4 quantization, peak memory drops to 8.28 GiB, establishing high deployability on consumer-tier hardware.
Highlights & Insights¶
- Pragmatic Operationalization of Causal Influence: Instead of attempting unfeasible real-world causal DAG discovery from complex continuous video streams, the authors ground causality as observable prediction changes under input masking and removal interventions.
- Decomposition without Fragmentation: CausalChapter bridges the longstanding trade-off between the low computational overhead of segment-then-caption pipelines and the rich holistic context of whole-video attention models, achieving the advantages of both paradigms through intervention-driven context routing.
Limitations & Future Work¶
- Intervention Latency Overhead: Computing leave-one-out KL divergences across candidate contexts requires multiple forward passes, accounting for over 40% of generation time. Accelerating support estimation via implicit gradient approximations or amortized meta-predictors represents a promising future avenue.
- Synthetic Style Artifacts in Benchmark References: The dense chapter annotations on AVLecture rely on GPT-4o initialization followed by human review. Although extensive sensitivity audits confirm consistent gains across Claude and human phrase variations, developing fully human-authored large-scale lecture corpora remains desirable.
Related Work & Insights¶
- vs. Chapter-Llama: Chapter-Llama feeds entire textualized transcripts into a long-context LLM in a single pass. For multi-hour lectures, this risks token truncation and high VRAM usage. CausalChapter decouples segmentation from description while bridging the context gap via interventional support estimation, yielding a 17โ21 point CIDEr margin on AVLecture.
- vs. Memory-Augmented Video Captioning (HICM2, Video Recap): Prior memory-augmented video architectures store past representations and retrieve them based on inner product similarity. CausalChapter demonstrates that in technical video reasoning, surface similarity fails to capture true predictive utility, establishing interventional sensitivity as a more reliable context selection criterion.
Rating¶
- Novelty: โญโญโญโญโญ Formulates video chaptering through the lens of interventional dependency drop and leave-one-out support scoring, neatly avoiding real-world causal graph complexity.
- Experimental Thoroughness: โญโญโญโญโญ Thoroughly evaluated across two distinct benchmarks with rigorous sensitivity checks covering window sizes, candidate pool scaling, micro-batching efficiency, and 4-bit quantization profiling.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, clean mathematical formulation, and well-structured diagrams and tables.
- Value: โญโญโญโญโญ Offers a highly practical, computationally viable blueprint for structuring long-form lecture videos, MOOC platforms, and enterprise meeting archives.