VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection¶

Conference: ECCV2026
Paper: ECCV Paper
PDF: Full Paper
Project: VDC-Agent
Area: Video Understanding / Multimodal Reasoning
Keywords: video detailed captioning, self-reflection, prompt optimization, preference data, curriculum DPO

TL;DR¶

VDC-Agent uses one multimodal model to generate, evaluate, and reflect on video captions, then applies easy-to-hard DPO to trajectory-derived preference pairs, turning iterative improvements into single-pass generation that achieves 49.08% average accuracy on VDC and 37.4% F1 on DREAM-1K.

Background & Motivation¶

Video detailed captioning must go beyond recognizing that people are tending plants: it should explain their collaboration, object arrangement, action progression, and camera movement. General vision-language models (VLMs) already produce fluent summaries, but fluency does not guarantee coverage of details or consistency with visual evidence. A caption may omit tray arrangements or identify an indistinct plant as a tomato based on prior expectations. Existing methods therefore fine-tune on high-quality caption data: Cockatiel uses human feedback, while ShareGPT4Video and AVC-DPO draw supervision from stronger models. These approaches improve fine-grained descriptions but shift the cost of scaling data to human annotation or external teachers.

The paper addresses a mismatch: a model may already be able to detect some omissions without fully using that ability during direct generation. Assigning the same model an evaluator role could expose missing information through the video and explicit principles, enabling a better prompt and a better candidate caption. However, self-improvement is not monotonic: requesting additional details can encourage speculation or divert attention from an initially correct emphasis. Simply repeating a request for more detail is therefore insufficient; the process must recognize when a prompt update has made the result worse.

Even effective iterative evaluation and revision also impose deployment costs. The paper treats online reflection as a way to construct training data, not as a mandatory procedure for every future request. It retains comparisons between good and bad captions of the same video and encodes their relative preferences into model parameters. Core Idea: generate preference pairs with quality gaps through principle-guided self-evaluation and failure reflection, then train from large to small gaps so that one caption-generation call benefits from iterative refinement.

Method¶

Overall Architecture¶

Inputs are videos without caption annotations, an initial task prompt, and textual quality principles; outputs are an offline preference dataset and a trained detailed captioner. The offline pipeline follows four designs: Principle-Guided Self-Evaluation, Degradation-Triggered Reflection, Trajectory Preference Construction, and Curriculum Preference Internalization. One multimodal large language model (MLLM) uses different instructions to act as generator, evaluator, and prompt refiner; these roles do not introduce three independent teacher models. Candidate captions, quality scores, revision suggestions, and the previous prompt-refinement reasoning connect the roles as intermediate state.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Video and initial prompt"] --> Evaluate["Principle-Guided<br/>Self-Evaluation"]
    Evaluate --> Reflect["Degradation-Triggered<br/>Reflection"]
    Reflect -->|Update prompt until termination| Evaluate
    Reflect -->|Retain full trajectory| Pairs["Trajectory Preference<br/>Construction"]
    Pairs --> Train["Curriculum Preference<br/>Internalization"]
    Train --> Model["Trained captioner"]
    Test["New video and task prompt"] --> Model
    Model --> Output["Single-pass caption generation"]

The loop belongs to offline data generation; deployment sends a new video directly to the trained captioner without repeating scoring and reflection. Here, single-pass means one autoregressive caption-generation call, not one token-level forward computation for the entire caption. The paper also evaluates a test-time reflection variant, which has a different deployment path from the final VDC-Agent-7B.

Key Designs¶

1. Principle-Guided Self-Evaluation: turn a vague request for detail into actionable feedback

The initial prompt first elicits a candidate caption from the video. The evaluator then reads the video, that caption, and quality principles rather than judging textual fluency alone. The principles cover aspects such as camera motion, background, and main objects, specifying what the current task should inspect. Evaluation returns a quality score from 0 to 100 and a natural-language suggestion for revising the next prompt. The score determines whether to stop or detect degradation, while the suggestion provides a concrete direction, such as adding spatial relations or emphasizing the main object. These outputs are not interchangeable: a scalar ranks candidates but does not explain where the next generation should focus.

The evaluator and generator share a backbone, so data generation does not require a larger external scoring model. This does not remove all human knowledge: people still specify the initial prompt and textual principles. The precise claim is that this data-construction process requires neither per-video human caption annotations nor a stronger teacher to supply preference labels. The main text only illustrates categories of principles; complete principles and system prompts are assigned to an appendix absent from the supplied full-text cache. The available material therefore does not support reproducing the exact evaluation prompt or interpreting its score as a calibrated probability of correctness.

2. Degradation-Triggered Reflection: diagnose harmful prompt updates instead of blindly expanding captions

The loop stops as soon as the current score reaches the threshold, which defaults to 90. Below that threshold, a score at least as high as the previous one leads the prompt refiner to use the current suggestion directly. Only a score decrease activates the additional self-reflection branch to inspect why the previous update was harmful. The reflection context includes the current prompt and caption, the previous prompt and caption, and the previous prompt refiner's chain-of-thought (CoT). This connects failure to a particular revision decision rather than merely indicating that the output was inadequate. The next round uses the revised prompt to generate a caption from the same video again, rather than merely polishing the old caption as text.

An important boundary is that changes in self-assigned scores trigger reflection, not an external correct answer. If the scoring model misses a hallucination, the loop may still accept a harmful direction. The system is therefore not a formal verifier of correctness; it exploits differences in the same model's capabilities under different task instructions. The main text sets the maximum iteration parameter to \(T=4\) but indexes steps from 0 to \(T\); without the appendix pseudocode, this should not be converted into an asserted total number of model calls. After reaching the threshold or iteration cap, it retains every candidate and score rather than only the final caption.

3. Trajectory Preference Construction: extract supervision from successes and failures on the same video

The source is Cockatiel-4K's 4,008 high-resolution videos; the paper uses the videos but not their provided captions. Each video is processed for five tasks: camera, short, background, main object, and detailed, covering camera properties, summaries, backgrounds, main entities, and detailed events. These dimensions exercise complementary capabilities instead of making every preference pair express a preference for greater length. They produce 20,040 raw trajectories, each corresponding to one video-task combination rather than a distinct video. Of these, 1,078 reach the threshold at the first step and lack a comparative refinement trajectory, so they are discarded. Another 76 are removed for JSON formatting or parsing errors, leaving 18,886 sets, called VDC-Agent-19K.

Within each retained trajectory, the highest-scoring caption becomes the preferred item and the lowest-scoring caption becomes the dispreferred item. The highest-scoring caption need not be the last, and the lowest-scoring caption need not be the first; intermediate failures can therefore provide training signal. Their quality gap is defined as:

\[ \Delta s = s^{+} - s^{-}. \]

Here, \(s^{+}\) and \(s^{-}\) are the highest and lowest scores; the gap represents relative preference strength and supplies the subsequent curriculum ordering. These are not human-certified quality labels, so preference quality requires separate validation, which the paper provides through blind evaluation. The main text does not fully specify how tied candidates are handled; it does not establish that every retained pair has a strictly positive gap.

4. Curriculum Preference Internalization: learn clear preferences before subtle distinctions

Training uses direct preference optimization (DPO) to make the current policy favor the preferred caption over the dispreferred one relative to a reference policy. It directly uses relative probabilities of paired texts without training a separate reward model or running an online reinforcement learning sampling loop. Compared with ordinary randomly ordered DPO, the method changes only sample order: it sorts gaps from largest to smallest and forms sequential mini-batches. Large gaps represent easier preferences for establishing coarse behavior, while later small gaps refine subtler differences between captions. Difficulty is therefore a proxy derived from self-evaluation, not human-labeled difficulty or the intrinsic complexity of a video.

The gap is not multiplied into the loss as a new sample weight, and no new reward function is introduced. The contribution consequently lies in data construction and training scheduling rather than a mathematical replacement for the DPO objective. The authors combine this curriculum with cosine learning-rate decay, assigning larger early updates to clearer preferences and smaller later updates to ambiguous ones. The cached DPO equation has extraction damage; this note retains the mechanism confirmed by prose without presenting a guessed reconstruction as the original equation. The outcome is best understood as parameter-level internalization of preference behavior, not proof that the model internally reproduces the same explicit reflection chain.

A Worked Example¶

Consider the greenhouse video in Figure 4: a caption should cover collaboration, measurement, and tray arrangement without assigning an unsupported plant species. Figure 4 compares final-model outputs rather than showing a full reflection trajectory; the following illustrates the method, not a recorded round-by-round trace. If an initial caption omits tray arrangements, the evaluator can suggest spatial organization, prompting the refiner to direct the next generation toward visible layout. If that revision adds an unreliable species inference and lowers the self-score, the reflector can examine the previous revision rationale to diagnose the change. After termination, actual scores select the best and worst captions for a preference pair; this example invents neither scores nor a guarantee of successful correction. At deployment, the new model directly generates a caption without rerunning these data-construction steps for the greenhouse video.

Loss & Training¶

The base model is Qwen2.5-VL-7B-Instruct; LoRA adapts only the LLM backbone while all other parameters remain frozen. LoRA rank is 16, alpha is 32, and dropout is 0.1; training lasts 3 epochs with a global batch size of 16. The initial learning rate is \(5\times10^{-5}\), warmup covers 10% of total steps, and cosine decay follows; training uses 4 NVIDIA A800 GPUs. The setup samples at most 32 frames with at most 128 visual tokens per frame; decoding uses beam size 1, temperature 0.1, and a maximum caption length of 1024. These settings are explicit in the main text; the DPO beta value, inter-epoch ordering details, and full prompts are not specified in the supplied body.

Key Experimental Results¶

Main Results¶

VDC contains 1,027 videos and follows AuroraCap's VDCscore, reporting Accuracy/Score across five dimensions and their average; average accuracy is not the training-time self-score from 0 to 100. DREAM-1K contains 1,000 videos and reports event-level Precision, Recall, and F1: correctness among generated events, coverage of reference events, and their harmonic relationship, respectively. The table selects results from Table 1 (page 9) and Table 2 (page 10), under each benchmark's evaluation protocol; higher is better.

Model	VDC Average Acc. (%)	VDC Average Score	DREAM F1 (%)	Precision (%)	Recall (%)
Qwen2.5-VL-7B	43.95	2.23	30.1	30.5	29.7
OwlCap-7B	46.90	2.40	34.7	34.1	35.3
VDC-Agent-7B	49.08	2.50	37.4	38.5	36.3

Relative to the same Qwen backbone, average VDC accuracy increases by 5.13 percentage points and DREAM F1 by 7.3 percentage points; these are absolute differences, not relative percentage increases. AVC-DPO-7B reports 47.70%/2.47 in Table 1, so the gains are 1.38 percentage points and 0.03 score, without implying superiority on every subdimension.

Ablation Study¶

Table 4 (page 12) compares principle prompting, test-time reflection, and training-based internalization using the same Qwen backbone; runtimes are in seconds under the paper's timing setup.

Config	VDC Average Acc. (%)	VDC Average Score	Inference Time (s)
Baseline	43.95	2.23	15.5
Baseline + Principle	46.47	2.35	22.3
Baseline + Principle + Reflection	47.10	2.38	164.9
VDC-Agent	49.08	2.50	15.5

Table 5 (page 12) compares training strategies on VDC: random DPO shuffles data, anti-curriculum DPO orders gaps ascending, and curriculum DPO orders gaps descending.

Training Strategy	VDC Average Acc. (%)	VDC Average Score
SFT	47.54	2.41
Random DPO	48.03	2.42
Anti-Curriculum DPO	47.65	2.40
Curriculum DPO	49.08	2.50

Key Findings¶

The internalized model exceeds test-time reflection by 1.98 accuracy percentage points while reducing reported runtime from 164.9 s to 15.5 s; this does not make offline data generation free.
Curriculum DPO exceeds random DPO by 1.05 accuracy percentage points, supporting ordering in this setting without establishing that arbitrary self-score gaps reliably measure difficulty.
Table 7 (page 13) reports data-construction costs for the default \(T=4\): 70.7 s per sample and 49.2 h for the dataset on 8 A800 GPUs.
At \(T=6\) in the same table, VDC accuracy reaches 49.55%, but DREAM F1 falls to 36.8%; additional reflection does not monotonically improve every metric.

Highlights & Insights¶

Failed trajectories need not be discarded. Pairing the best and worst captions of the same video turns harmful revisions into preference-learning signal.
The score supports stopping, degradation detection, and curriculum ordering. This reuse simplifies the pipeline while making self-evaluation reliability a shared dependency.
Separating data generation from deployment is the practical benefit. Offline computation can explore captions while the online interface retains a single generation call.

Limitations & Future Work¶

Self-evaluation may reinforce shared model blind spots. In the 500-pair blind evaluation of Table 3 (page 11), human evaluators prefer the higher-scoring caption in 75.8%, 69.0%, and 67.6% of cases, supporting overall usefulness rather than noiseless labels.
GPT-5.1 preference rates in that table are 84.4% and 95.2% under reversed input orders. The authors consider the conclusion robust to ordering, but the 10.8-percentage-point difference shows that position bias remains.
Overlapping length distributions in Figure 3 cannot alone exclude length bias. Stricter length-matched evaluation and independent visual verification are reader suggestions, not completed experiments in this paper.
The supplied cache ends with references and lacks the appendix, with extraction damage in some equations; full prompts, reflection-trigger rates, and several reproduction details require the original supplementary material.
The authors plan larger backbones and video question answering; present captioning benchmarks do not directly establish broad improvements in general video reasoning.

Compared with Cockatiel: the method reuses its video resources without caption annotations; supervision comes from self-evaluated trajectories rather than a newly collected video corpus.
Compared with AVC-DPO and VideoCap-R1: these methods use stronger teachers or reward providers, whereas this method removes that data-generation dependency; this does not mean the pretrained backbone never received external supervision.
Compared with Best-of-N: Table 8 (page 14) reports 45.12%/2.30 and 186.0 s for 12-candidate BoN versus 47.10%/2.38 and 164.9 s for test-time reflection, supporting directed revision over undirected sampling in this configuration.
Transferable insight: video question answering could retain positive and negative answer-revision trajectories, but would require newly designed, verifiable answer-quality principles rather than a preference for more descriptive detail.

Rating¶

Novelty: 4/5. A coherent integration of degradation-triggered prompt reflection, trajectory preferences, and curriculum ordering; DPO itself is not a new objective.
Experimental Thoroughness: 4/5. Two benchmarks, multiple backbones, human preferences, and curriculum ablations are substantial, while stricter independent error verification remains absent.
Writing Quality: 4/5. The main pipeline and tables are clear, but the available body omits implementation details and extracted equations should not be reused directly.
Value: 4/5. A concrete approach to reducing online reflection costs through offline self-improvement, useful for iterating detailed-video-captioning data and models.