Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

Conference: ECCV 2026
Paper: ECCV Official Page · PDF
Area: Vision-Language Model Reasoning
Keywords: Continuous visual tokens, visual chain of thought, perceptual distillation, depth estimation, instance segmentation

TL;DR

CoVT teaches a VLM to generate continuous visual tokens supervised by segmentation, depth, and semantic experts before answering, replacing purely verbal intermediate observations with 16 latent perceptual states and raising Qwen2.5-VL-7B from 74.5 to 80.0 on CV-Bench, including a 14.0-percentage-point gain on its depth subtask.

Background & Motivation

Vision-language models can produce fluent explanations without accurately resolving instances, boundaries, or relative depth. A textual chain of thought can describe a sensible procedure, such as identifying objects before comparing their distances, yet a mistaken initial observation can invalidate every subsequent step. The issue is not the complete absence of visual encoding. Rather, supervision of the answer is predominantly textual, with limited direct pressure on intermediate states to preserve depth, contours, and regional structure.

External vision tools can supply segmentation or depth maps, but they add execution and orchestration costs. Describing their outputs in words can also discard the spatial detail needed for precise comparisons. Instead of generating longer explanations, this paper places continuous vectors carrying perceptual information inside the reasoning sequence itself. Unlike unsupervised thinking placeholders, these vectors must reconstruct targets supplied by vision experts through specialized decoding paths; merely extending the sequence before the answer is insufficient.

This creates a training challenge: the model must understand the new visual tokens, generate them, and ultimately make its answers depend on those states. Core idea: give a small set of continuous tokens perceptual meaning through dense visual supervision, then internalize expert capabilities through a four-stage curriculum of comprehension, generation, reasoning, and flexible use.

Method

Overall Architecture

The input remains an image and a question, and the output remains a textual answer. The change is in the intermediate sequence: CoVT interleaves text with continuous visual tokens, which condition subsequent answer generation. During training, these tokens pass through trainable projections and expert-specific decoding paths to receive supervision from segmentation, depth, edges, or DINO features. Ordinary inference can skip those dense outputs and retain only latent visual reasoning.

The diagram summarizes the relationship between training and deployment. Expert signals shape the tokens rather than acting as external tools that return results for every answer. The main configuration contains three categories, segmentation, depth, and DINO, totaling 16 tokens. Adding edges gives four categories and 20 tokens. Thus, the paper's table label "3 Visual Tokens" denotes three types, not three vectors in total.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and question"] --> Tokens["Compact Visual<br/>Thinking Slots"]
    Experts["SAM / depth / edges<br/>DINO expert signals"] --> Align["Granularity-Aware<br/>Alignment"]
    Tokens --> Align
    Align --> Curriculum["Four-Stage<br/>Visual Curriculum"]
    Curriculum --> Latent["Latent Reasoning and<br/>Optional Decoding"]
    Latent --> Answer["Textual answer"]
    Latent -->|When visualization is needed| Maps["Masks / depth / edges"]

Key Designs

1. Compact Visual Thinking Slots: preserve distinct kinds of perception in intermediate states

CoVT retains an autoregressive sequence framework, but not every state corresponds to a discrete word. Segmentation tokens represent instance location, shape, and two-dimensional layout; depth tokens represent pixel-level depth relationships; edge tokens capture contours and fine structure; and DINO tokens carry patch-level semantic representations. These are not depth estimates written out as sentences. They participate in subsequent answer generation as continuous vectors, without requiring inference to reconstruct an image from each vector and feed it back into the VLM.

The concrete allocation is 8 segmentation tokens, 4 depth tokens, and 4 DINO tokens, with an optional 4 edge tokens. The three-type main configuration already covers regions, geometry, and semantics; adding edges does not guarantee improvements on every task. Compactness here means a small set of decodable latent states, not a claim that those tokens independently store all information in the image. The training-time decoding paths also use dense features from expert encoders.

2. Granularity-Aware Alignment: decode task outputs but match representation features

Treating every expert as a feature vector to match with MSE overlooks the output structure that makes segmentation and depth useful. CoVT uses projection layers consisting of one multi-head attention layer and two fully connected layers. For task-oriented experts, it projects tokens into a prompt space that controls task decoding. For a representation model such as DINOv2, it instead projects them to the shape of patch features and matches those features directly. These are two supervision interfaces within one token framework, not a single undifferentiated objective imposed on all four experts.

The segmentation branch follows a LISA-style interface: each segmentation token acts as a prompt to the SAM decoder, which also receives dense image features from the SAM encoder. Eight tokens produce eight masks. Hungarian matching associates predictions with target masks, after which Dice and focal losses provide supervision. Matching avoids treating token order as a fixed instance numbering scheme. Producing eight masks should not be mistaken for an upper bound of eight on the model's numerical counting answers.

The depth branch uses four tokens as prompts that interact with intermediate dense features from DepthAnything v2 through batched matrix multiplication and softmax-related operations to reconstruct depth maps. The four predictions are averaged and aligned to the target using L1 loss. The edge branch treats four tokens as \(1\times1\) convolutional kernels applied to PIDINet dense features, producing four edge maps that are likewise averaged and trained with L1 loss. In both cases, tokens must influence spatial outputs rather than merely resemble a global image embedding.

The DINO branch does not require manually defining segmentation or depth labels. Its four tokens are projected to match DINOv2 patch features under an MSE objective, transferring semantic and local representation knowledge. The depth equation is corrupted in the cached text, so this note retains the operations and averaging described in the prose without inventing a softmax axis, tensor dimensions, or an unverified complete equation.

3. Four-Stage Visual Curriculum: learn to understand and generate states before using them to answer

The first stage is comprehension: visual tokens are inserted after the input image, and the model answers the original question, learning to read those new states as conditioning information. The second stage is generation: the question is reformulated to request segmentation, depth, edges, and patch features, and the model outputs the corresponding tokens under expert supervision. Comprehension alone does not ensure that the model can produce the states itself; generation alone does not ensure that those states help answer questions.

The third stage places visual tokens inside the thinking portion of the response before the final answer, teaching the model to answer using its own perceptual states. The fourth stage randomly drops entire token categories, with the number of dropped categories ranging from zero to the total number of types. This breaks dependence on a fixed format that always emits every category. The authors call this efficient reasoning. However, training with random category dropout should not be confused with demonstrating an optimal, explicit question-conditioned routing policy.

4. Latent Reasoning and Optional Decoding: separate answering from inspecting internal states

After training, the model answers through a chain of textual and continuous visual states without requiring SAM, DepthAnything, or PIDINet to return external tool feedback for each question. When interpretation is needed, the relevant tokens can be passed through decoding paths to inspect masks, depth, or edges. This preserves a visualization interface without making dense prediction a mandatory step in every question-answering call.

Interpretability here means that intermediate states can be associated with perceptual outputs, not that those outputs are necessarily causally faithful explanations. Task decoders also consume expert dense features, so a convincing visualization alone does not prove that the answer depends only on information encoded in the tokens. The paper additionally examines answer-to-image representation similarity and attended regions, but those analyses still provide correlational evidence.

A Worked Example

One example in Figure 5 asks which of two marked points is closer to the camera. The baseline answers A based on an apparent foreground position. CoVT's segmentation states locate the face region containing B, while depth states supply its depth relationship to the surrounding scene, leading to the answer B. If inspection is desired, the segmentation and depth tokens can be decoded into corresponding spatial outputs. Otherwise, the answer is generated directly from the latent states.

The point is not to repeat "B is closer" in a longer explanation. Region localization and geometric relationships enter the chain through two different kinds of continuous state. This is a qualitative example supplied by the paper, not evidence that every two-point depth comparison is solved, and no numerical pixel depths should be invented for it.

Loss & Training

The joint objective combines textual cross-entropy with a weighted sum of visual task losses. Based on the prose in Section 3.4, its structure can be summarized as follows. This is a structural restatement of the corrupted equation, not a symbol-for-symbol recovery:

\[ \mathcal{L}_{\mathrm{total}} =\mathcal{L}_{\mathrm{CE}}+ \gamma\sum_{t\in\{\mathrm{seg},\mathrm{depth},\mathrm{edge},\mathrm{dino}\}} \lambda_t\mathcal{L}_t. \]

Only visual tasks enabled in the current configuration contribute to the sum. Segmentation uses Dice and focal losses, depth and edges use L1, and DINO uses MSE. The paper sets \(\gamma\) and every \(\lambda\) to 1. These explicit descriptions take precedence over the generic CE labels in Figure 3; the visual branches should not all be described as cross-entropy training.

The main backbone is Qwen2.5-VL-7B, tuned with LoRA rank 16 and alpha 32. The LoRA learning rate is \(5\times10^{-5}\), and the projection learning rate is \(10^{-5}\). The four stages run for 4,000, 3,000, 3,000, and 5,000 steps, respectively, with batch size 4 on one A100 or four A6000 GPUs. Projection layers are trainable, while the standard visual encoder is frozen in the method's base formulation.

Training data comprises a vision-centric/real-world subset of LLaVA-OneVision plus spatial perception data from TallyQA and ADE20K-Depth. The cached body does not provide exact subset sizes or mixture ratios, nor a complete account of continuous-state feedback, start/stop handling, or freezing of every expert parameter. Those reproduction details therefore cannot be filled in by assumption. Evaluation uses VLMEvalKit.

Key Experimental Results

Main Results

The following results are selected from Tables 2 and 3. Scores follow the original reporting scale; gains are percentage-point differences, not relative percentages. The main CoVT configuration has 8 segmentation, 4 depth, and 4 DINO tokens, totaling 16.

Evaluation	Qwen2.5-VL-7B	Three-Type CoVT	Gain (pp)
CV-Bench overall	74.5	80.0	+5.5
CV-Bench Count	65.0	66.2	+1.2
CV-Bench Depth	72.8	86.8	+14.0
CV-Bench Distance	75.5	82.5	+7.0
MME-RealWorld	60.0	63.7	+3.7
BLINK	55.7	56.0	+0.3
RealWorldQA	68.6	71.6	+3.0
MMStar-P	67.1	69.2	+2.1
MMVP	56.0	58.7	+2.7
HRBench4K	68.6	72.9	+4.3
HRBench8K	64.9	69.4	+4.5

MMStar-P includes only Coarse Perception, Fine-grained Perception, and Instance Reasoning, and is not the full MMStar score. The strongest improvements concern depth and distance; counting and BLINK improve much less. Better visual reasoning does not imply equal gains across all perceptual capabilities.

Ablation Study

The next table comes from Table 5. When the segmentation count is 1, 8, or 32, depth and DINO remain fixed at 4 tokens each. Zero tokens means direct fine-tuning on the same training data, not the original backbone without that fine-tuning used in the preceding table. "16 empty" means 16 ordinary tokens without visual alignment.

Configuration	CV-Bench	BLINK	RealWorldQA	MMVP	HRBench4K
0 tokens, direct fine-tuning	76.6	55.5	70.7	55.3	68.6
16 empty, no visual alignment	75.7	56.0	70.3	56.7	68.1
1 segmentation token, 9 total	78.9	55.6	70.8	56.7	73.0
8 segmentation tokens, 16 total	80.0	56.0	71.6	58.7	72.9
32 segmentation tokens, 40 total	73.9	54.4	68.4	55.3	70.8

The full configuration gains 3.4 percentage points on CV-Bench over direct fine-tuning and 4.3 points over unaligned tokens. These comparisons separate additional training exposure from visually supervised intermediate states more clearly than quoting only the 5.5-point gain over the original backbone. They still do not provide a complete factorial separation of dense supervision and latent-chain structure.

The following selection from Tables 2 and 3 compares three and four types to test whether adding edges is uniformly beneficial.

Configuration	CV-Bench	CV-Bench Depth	CV-Bench Distance	MMVP	HRBench8K
Three types, 16 tokens total	80.0	86.8	82.5	58.7	69.4
Four types, 20 tokens total	79.8	89.2	80.5	56.7	69.9

Key Findings

• Adding edges improves depth by 2.4 points but reduces overall CV-Bench by 0.2 and MMVP by 2.0. Complementary expert information does not imply monotonically increasing performance as experts are added.
• Using 32 rather than 8 segmentation tokens reduces CV-Bench by 6.1 points and falls below direct fine-tuning. The authors suggest that aligning more segmentation tokens is harder, but the table provides no training variance, so the proposed mechanism is not established.
• Results also improve on LLaVA-v1.5-13B: Table 4 reports BLINK relative-depth scores of 75.8 for CoVT-depth and 62.9 for reproduced Aurora-depth. This is a same-backbone gain of 12.9 points, not a result to combine with the Qwen table as though both describe one model.
• Table 6 reports 48.7/77.5 on MMVP/V* for its depth-alignment configuration, whereas Table 3 reports 58.7/79.1 for the corresponding depth-only configuration. The current cache cannot resolve this discrepancy, so this note does not adopt the strong claim that decoder alignment improves every metric.

Highlights & Insights

• Supervision determines what latent states are used for. The same number of unaligned tokens does not reproduce the gains. The important factor is not simply additional computation positions, but pressure for those positions to carry decodable visual information.
• Experts need not remain online tools. Training-time perceptual interfaces can distill capabilities into reasoning states and reduce dependence on tools at deployment. This does not eliminate the cost of experts during training.
• Granularity should determine the alignment interface. Supervising task output structure for task experts and features for representation experts better reflects their distinct objectives than imposing feature MSE everywhere. The principle is transferable, but each new task still needs validation.

Limitations & Future Work

• Efficiency evidence is incomplete. The paper emphasizes a small token budget and normally skipping decoding, but the cached body lacks systematic end-to-end latency, memory, and throughput comparisons. Twenty visual tokens also does not mean the entire answer sequence contains only twenty tokens.
• Supervision gains and reasoning mechanisms are not fully separated. Direct fine-tuning and empty-token controls are useful, but a control with the same dense supervision and no visual chain is still needed, along with causal interventions that replace or perturb visual tokens.
• Expert knowledge has boundaries. Biases from SAM, monocular depth, and edge models may enter the latent states. The evidence mainly concerns image-based perceptual question answering, not video, action planning, or reliability in medical deployment.
• The curriculum and dynamic selection need further analysis. The body does not provide a complete quantitative stage-removal ablation or a systematic analysis of question difficulty versus token-selection cost. Whether random category dropout yields effective adaptation needs a dedicated evaluation.
• Reproduction and reporting have gaps. Data mixture ratios and continuous-token implementation details are not fully specified, some table values disagree, and cached extraction damages parts of figures and equations. This note does not fill in those numbers. Future verification should inspect the original layout and add repeated runs and confidence intervals.

Related Work & Insights

• Versus textual CoT: Textual CoT mainly improves symbolic organization, while CoVT additionally constrains perceptual information in intermediate states. The paper's textual CoT controls often degrade on visual tasks, but this does not show that textual CoT is useless for all multimodal problems.
• Versus Coconut and CCoT: These approaches share the use of continuous latent states. CoVT specifically binds its states to dense vision-expert targets rather than only compressing language reasoning or adding latent computation steps.
• Versus LISA: The prompt interface from segmentation tokens to the SAM decoder builds on prior work. CoVT incorporates it into multiple perceptual token types and a question-answering curriculum; the novelty is the multi-expert continuous visual reasoning framework, not the first use of language hidden states to drive SAM.
• Versus Aurora: Aurora uses VQ-VAE latents for perceptual information such as depth and detection, while CoVT uses continuous tokens with expert-granularity-aware alignment. The LLaVA comparison supports applicability beyond one backbone, but reproduced results should not be presented as an absolute ranking across every setting.

Rating

• Novelty: 4/5. Combining expert-constrained continuous visual states, multi-granularity alignment, and curriculum training addresses a perceptual bottleneck, although the interfaces and latent-reasoning paradigm have clear precedents.
• Experimental Thoroughness: 4/5. Multiple benchmarks, two backbones, and token controls provide useful coverage; runtime costs, stage ablations, repeated trials, and stronger causal isolation remain missing.
• Writing Quality: 3/5. The training logic is clear overall, but token categories and counts can be confused, some table entries disagree, and reproduction details are incomplete.
• Value: 4/5. The framework offers transferable ideas for perception-intensive VLM question answering, while deployment efficiency and explanatory faithfulness still require independent verification.