CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/MICLAB-BUPT/CritiqueDriveVLM
Area: Autonomous Driving
Keywords: vision-language models, multidimensional verifier, multi-turn reinforcement learning, latent thought distillation, CoT-free inference

TL;DR¶

CritiqueDriveVLM trains a self-correcting teacher using verifier scores and textual critiques, then aligns the teacher's reasoning-end state with the student's answer-start state, enabling CoT-free inference at 68.59% MCQ and reducing mean latency from 3482 ms to 416 ms, without preserving the teacher's 76.54% accuracy.

Background & Motivation¶

Autonomous driving vision-language models (VLMs) must identify road entities and explain how those entities affect the current driving choice. DriveLMM-o1 formulates such tasks as visual question answering with step-by-step reasoning; the output remains a language answer or action option, not a continuous control trajectory. Supervised fine-tuning (SFT) can teach an answer format without ensuring reliable visual reasoning: missing a pedestrian can produce dangerous decisions, while reflexively slowing down can reflect conservative bias. Fluent answers and lengthy explanations therefore do not independently establish that a driving judgment is correct.

AgentThink and OmniDrive-R1 strengthen visual grounding through chain-of-thought (CoT) and external perception tools, but faulty tool outputs, interface dependencies, and long generations add failure modes and latency. This paper separates acquiring more reliable reasoning from removing verbose reasoning outputs at deployment. Rewarding only the final option can overlook a trajectory with an incorrect explanation and a coincidentally correct choice; imitating only the final answer can discard information used by the teacher to correct itself. The authors consequently focus on both the content of training feedback and the location of distillation supervision, rather than merely shortening prompts or responses.

The teacher can spend time receiving critiques and rechecking the scene, whereas the student should answer directly from the same image and question. System-2 and System-1 here describe slow explicit reasoning and fast direct output, not different network sizes or an empirical account of human cognition. Core Idea: train a corrective teacher with multidimensional critiques and a cost for additional turns, then align its completed reasoning representation with the student's answer-start representation to transfer part of the reasoning benefit into a CoT-free inference path.

Method¶

Overall Architecture¶

The inputs are driving-scene images and a question; the output is a tagged language answer or driving-action option. Both teacher and student use Qwen3-VL-8B, with three training stages: Format Warm-up and Verifier Construction, Critique-Driven Multi-Turn RL, and Latent Thought Distillation. The verifier reads the images, question, and model response; the teacher produces explicit reasoning, while the student produces only an answer. The teacher is not an additional component invoked during student inference: it supplies frozen hidden-state supervision during distillation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    input["Driving images and questions<br/>Training data with reasoning"] --> warmup["Format Warm-up and<br/>Verifier Construction"]
    warmup --> rl["Critique-Driven<br/>Multi-Turn RL"]
    rl -->|Frozen teacher reasoning-end states| distill["Latent Thought Distillation"]
    input -->|Images, questions, and answer supervision only| distill
    distill --> student["Student answers directly<br/>No CoT or verifier calls"]

The diagram represents supervision between training stages, not a requirement to execute the teacher before every student prediction. Teacher training and inference can involve verifier-guided correction; only the deployed student entirely bypasses the verifier and intermediate reasoning text. The paper's term tool-free primarily excludes external perception tools such as detectors and depth estimators, and should not be expanded to mean that the teacher needs no external model.

Key Designs¶

1. Format Warm-up and Verifier Construction: make reasoning parseable and errors assessable

Warm-up SFT uses a high-quality CoT subset of DriveLMM-o1 and requires the format <think>...</think><answer>...</answer>. The first block contains reasoning and the second contains the final answer, allowing process and outcome checks to be separated. Its primary role is to establish the format required by subsequent reinforcement learning, rather than serve as the final training stage for maximizing reasoning ability. If exploration causes the model to omit these tags, format rewards, option extraction, and process verification can lose their consistent input structure. An independent verifier is initialized from a separate base model instead of directly using the policy being optimized as its own judge. Positive examples come from ground-truth annotations and receive maximum perception, logic, and safety scores with empty critiques. Hard negatives come from a baseline GRPO model trained only with format and answer-accuracy rewards, exposing visual hallucinations and logical contradictions.

The stronger Qwen3-VL-235B assigns discrete per-dimension scores and targeted textual critiques to these flawed responses; data filtering and human verification precede verifier training. Perception checks whether entities are visually grounded, logic checks causal reasoning and internal consistency, and safety evaluates driving-safety reasoning and conservative bias. Scalar scores enter the reinforcement learning objective, whereas textual critiques tell the policy what to examine next: the two feedback channels have different roles. For example, identifying a missed crossing pedestrian offers more specific guidance than an aggregate score, although this still depends on the verifier detecting the error. The trained verifier is frozen to avoid simultaneous policy and judge drift; freezing does not itself establish reliability. The main text does not specify the exact base-model size of the verifier, and the available cache does not supply its full discrete scoring scale, so it should not be described as a 235B verifier. In the described pipeline, the 235B model provides data annotations and is not a confirmed identity for the final frozen verifier.

2. Critique-Driven Multi-Turn RL: reward corrected outcomes while charging for extra turns

The teacher first generates complete reasoning and an answer from the images and question; the verifier then returns scores for the three dimensions and a textual critique. The interaction ends if every dimension receives a perfect score; otherwise, the previous prompt, response, and critique are appended to the dialogue history before another complete response is generated. This does not replace the policy's answer with a verifier answer: the policy reasons again under specific feedback. The implementation sets \(K=2\), permitting an initial answer and at most one correction rather than unlimited reflection. Reaching the turn limit also terminates the loop even if the verifier remains dissatisfied, so termination is not equivalent to correctness. The terminal response determines the final reward, combining output-format assessment, agreement with the ground-truth answer, and perception, logic, and safety process scores. Table 1 distinguishes these process dimensions to prevent a correct option from concealing an unreliable rationale.

A second-turn correction additionally incurs a fixed step-decay penalty, encouraging the policy to complete the necessary reasoning on its first attempt. The intended effect is to retain critique as a learning aid while discouraging a strategy of answering carelessly and waiting for reminders. However, first-attempt pass rates are not separately reported, so improved terminal MCQ does not establish that the teacher has become fully independent of critiques. Optimization uses Group Relative Policy Optimization (GRPO), sampling \(G=4\) candidates per question and deriving relative advantages from within-group reward statistics. A clipped importance-sampling ratio limits update magnitude, while KL regularization constrains deviation from the reference model obtained through warm-up. Compared with PPO using an equivalently sized value network, GRPO reduces the need for an additional Critic network, but the verifier still has its own computational cost. The cached reward-combination and GRPO equations have missing operators and subscripts; this note retains mechanisms confirmed by the prose without guessing exact reward algebra or penalty constants.

3. Latent Thought Distillation: align the end of reasoning with the start of answering

The third stage freezes the teacher and initializes the student from the original base VLM, rather than simply continuing to train the RL teacher. Student examples retain only images, questions, and answers, removing explicit CoT text so that reproducing the teacher's rationale cannot satisfy the objective. For the same input, the teacher can still generate a multi-turn reasoning trajectory to supply supervision. The method extracts the teacher's deep hidden state at the last </think> token of the final trajectory and the student's corresponding state at <answer>. These positions mark the boundary between completed reasoning and answer generation; supervision does not require matching the entire CoT token by token. Because teacher and student use the same 8B base architecture, corresponding hidden spaces can be aligned; the reported efficiency gain is not obtained by reducing parameter count. The chosen target represents the teacher after inspecting evidence and correcting its response, without adding intermediate output steps to the student.

Alignment uses cosine distance, constraining representation direction without requiring equal vector magnitudes. The following is a normalized rendering of the cosine loss defined in Section 3.3 and described by Equation 8; the cached equation itself has damaged formatting.

\[ \mathcal{L}_{\mathrm{align}}=1-\frac{\mathbf{h}_{S}^{\mathrm{answer}}\cdot\mathbf{h}_{T}^{\mathrm{think}}}{\|\mathbf{h}_{S}^{\mathrm{answer}}\|_2\|\mathbf{h}_{T}^{\mathrm{think}}\|_2}. \]

The student also receives autoregressive cross-entropy supervision on answers, so it learns answer generation rather than only matching a hidden vector. The prose explicitly combines cross-entropy with weighted alignment, using an alignment coefficient of \(\lambda=0.5\). Hidden-state similarity is an optimizable proxy, not proof that the student performs the teacher's logical steps. The empirical evidence for its usefulness is improved CoT-free student MCQ, not direct observation of latent thoughts.

A Worked Example¶

Figure 3 on page 14 shows a pedestrian-crossing question asking which action among stopping, sudden braking, acceleration, and lane changes avoids a collision. The baseline incorrectly describes the road as having no pedestrians and selects F, meaning that none of the listed actions is needed. The teacher also misses the pedestrian on its first turn, demonstrating that higher aggregate performance does not make every initial perception correct. The verifier asks it to recheck the front view for a crossing pedestrian or stop sign; the second-turn teacher identifies the pedestrian and selects A, coming to a complete stop. The student directly outputs A in this example without displaying the initial error, critique exchange, or full corrected rationale. The example illustrates how corrective experience during training can supervise rapid output, but one case cannot establish reliable transfer across all hazardous scenes. The figure's baseline and teacher rationales state 20.3 km/h and 16.17 km/h respectively, an internal textual inconsistency; this note does not treat them as a verified common scene speed.

Loss & Training¶

All stages use the same DriveLMM-o1 training split, containing over 18,000 visual question-answer pairs derived from nuScenes. The vision encoder remains frozen; warm-up SFT and distillation use LoRA, while the RL stage uses GRPO. Learning rates for the three stages are \(1\times10^{-4}\), \(2\times10^{-6}\), and \(2\times10^{-5}\) respectively. Each stage runs for 2 epochs with a global batch size of 128, using 8 NVIDIA A100 GPUs for training. Inference time is measured on a single A100 and should not be extrapolated to an onboard vehicle platform. The main text refers to metric details in Appendix A and hyperparameters in Appendix B, but the supplied cache ends with references and contains neither appendix. The exact scoring implementation, penalty coefficient, and full reproduction configuration therefore require the original appendices or code for further verification.

Key Experimental Results¶

Main Results¶

The following values come from Table 2 on page 11, under the official DriveLMM-o1 protocol; all are percentage-scale metrics for which higher is better. MCQ denotes Multiple Choice Quality and Overall Reasoning is the aggregate reasoning score; detailed calculation is deferred to the unavailable Appendix A, and the latter is not a real-world driving success rate.

Model	Overall Reasoning (%)	MCQ (%)	Rule Adherence (%)	Relevance (%)
Qwen3-VL-8B	77.76	55.54	84.32	84.43
DriveLMM-o1	75.24	62.36	81.56	79.42
AgentThink	79.68	71.35	84.98	84.99
OmniDrive-R1	80.35	73.62	85.42	82.58
CritiqueDriveVLM Teacher	80.48	76.54	86.21	85.46

Teacher MCQ exceeds the base model by 21.00 percentage points and OmniDrive-R1 by 2.92 points, while its Overall Reasoning advantage over OmniDrive-R1 is only 0.13 points. It does not lead every submetric: other baselines retain higher risk-assessment, scene-understanding, or missing-details scores.

Ablation Study¶

Table 3 on page 12 compares output modes; the selection below retains output tokens, mean single-A100 latency, and MCQ, without treating CoT and CoT-free operation as identical conditions.

Model and output mode	Mean output tokens	Mean latency (ms)	MCQ (%)
Qwen3-VL-8B, with CoT	390.95	4265	55.54
Qwen3-VL-8B SFT, with CoT	147.88	1675	62.86
AgentThink, with CoT and tools	515.01	5416	71.35
CritiqueDriveVLM Teacher, with CoT	223.32	3482	76.54
Qwen3-VL-8B, CoT-free	37.23	461	54.96
Qwen3-VL-8B SFT, CoT-free	23.65	343	61.73
CritiqueDriveVLM Student, CoT-free	28.83	416	68.59

The student reduces mean teacher latency by approximately 88% while losing 7.95 MCQ percentage points; the abstract's 28 tokens is abbreviated, whereas the precise table mean is 28.83. It exceeds CoT-free SFT by 6.86 points but takes 416 ms rather than 343 ms, so it is not faster in every comparison. The authors present Table 3 as evidence for alignment ablation, but it lacks a separately listed student with an otherwise identical training pipeline and only the alignment term removed, limiting causal isolation.

Table 4 on page 13 progressively adds RL components, with MT jointly introducing multi-turn interaction and the turn penalty.

Config	Overall Reasoning (%)	MCQ (%)
Base model	77.76	55.54
Add SFT	73.38	62.86
Add answer-accuracy GRPO	73.85	64.49
Further add multidimensional verifier rewards	75.10	67.46
Further add MT, full teacher	80.48	76.54

Key Findings¶

Verifier rewards raise MCQ from 64.49% to 67.46%; adding MT then raises it from 67.46% to 76.54%, the largest gain in this progressive ablation.
MT changes both interaction and penalties, so the table cannot separately quantify textual critiques, additional compute, and the turn penalty.
SFT improves MCQ while reducing Overall Reasoning from 77.76% to 73.38%, indicating that answer metrics and explanation quality should be examined separately.

Highlights & Insights¶

The verifier connects scores with actionable textual feedback. Training tells the teacher not just that it is wrong, but where to re-examine visual evidence.
Distillation targets the boundary between reasoning completion and answer generation. It retains an entry point for process supervision without making the student pay for token-by-token explanations at inference time.
Both models are 8B, so compression primarily concerns the computation path and output length. The contribution changes the inference mode rather than demonstrating a smaller parameter model.

Limitations & Future Work¶

Evaluation is limited to DriveLMM-o1, without closed-loop simulation, actual vehicle control, or cross-dataset generalization; a 416 ms question-answering response does not establish suitability for safety-critical control cycles.
The teacher still receives external verifier critiques, allowing verifier errors or poor advice to propagate to teacher and student; the paper does not provide a systematic verifier error analysis.
The cache lacks appendices and has damaged reward and GRPO equations; whether timing includes all verifier overhead is also unclear, requiring further checks for reproduction.
The student loses 7.95 MCQ points relative to the teacher and provides no explicit rationale for stepwise auditing; latent alignment does not establish complete preservation of safety properties or causal reasoning.
The authors plan to extend the framework to multi-camera video streams and temporal-logic distillation; further reader recommendations are to isolate interaction from penalties and report first-attempt pass rates and tail latency.

Compared with AgentThink and OmniDrive-R1: these methods use tool-augmented explicit reasoning, while this teacher substitutes verifier feedback for external perception-tool pipelines and the student additionally removes CoT; removing tools and removing the verifier occur at different levels.
Compared with AlphaDrive: both apply GRPO to driving VLMs, but this work adds process scoring, natural-language correction, and subsequent latent distillation instead of optimizing only the final response.
Compared with Distilling Step-by-Step and MiniLLM: this work emphasizes alignment at an internal-state boundary without requiring the student to reproduce teacher rationales, at the cost of reduced direct inspectability of reasoning.
Research direction: contrasting distillation from an incorrect initial state and a corrected terminal state for the same input could test whether gains arise from terminal representations or simply extra supervision; this is not an experiment completed in the paper.

Rating¶

Novelty: 4/5. The combination of critique-guided RL and answer-boundary hidden-state distillation is well targeted, while building on established GRPO and latent reasoning distillation ideas.
Experimental Thoroughness: 3/5. Main comparisons, efficiency measurements, and progressive ablations are included, but only one VQA benchmark is used and alignment and turn penalties lack strictly isolated ablations.
Writing Quality: 3/5. The three-stage narrative is clear, but tool-free can be misread, the qualitative example has inconsistent speeds, and cached equations and missing appendices constrain verification.
Value: 4/5. The work offers a concrete accuracy-latency trade-off for driving question answering, but does not establish readiness for real safety-critical deployment.