ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space

Conference: ECCV2026
Paper: ECCV Paper
Project: ProLaViT
Area: Multimodal VLM / Visual Reasoning
Keywords: latent visual thoughts, self-distillation, progressive reasoning, counterfactual verification, diversity constraints

TL;DR

ProLaViT distills programmatically synthesized visual-operation trajectories into four continuous latent states and constrains their differences according to step distance, enabling progressive reasoning in Qwen2.5-VL-7B-Instruct without inference-time visual tools and raising BLINK-Jigsaw accuracy from the base model's 59.33% to 76.00%.

Background & Motivation

A vision-language model (VLM) can describe an image fluently without preserving its spatial relationships correctly. For example, identifying the furniture supporting a television requires distinguishing the cabinet, floor, and occlusion boundaries; compressing these details into language before chain-of-thought reasoning can produce a coherent but incorrectly grounded answer. Generating intermediate images or invoking cropping and segmentation tools can return visual evidence to the reasoning process, but each generation, tool call, and re-encoding adds latency. Latent visual reasoning offers a different efficiency path by retaining intermediate states in continuous features without producing pixels.

However, compressing all localization and visual transformations into one latent prediction does not automatically create reliable intermediate derivation. The paper's one-step baseline must predict complex visual cues at once and can incorporate irrelevant background into its final representation. Adding more latent steps is also insufficient if they converge to almost identical vectors: the sequence then stores repeated information instead of progressively narrowing the evidence. The paper therefore addresses both what each step should learn and how to prevent these steps from degenerating into the same state during training.

The authors construct intermediate supervision from programmable visual operations: spatial tasks progress from a global grid to bounding boxes, crops, and segmentation, while puzzle tasks contrast incorrect and correct assemblies. These images provide target features only during training, and the teacher is the model's own frozen vision encoder, reducing incompatibility between different models' feature spaces. Here, structure refers both to ordered operation semantics and to latent states that should be neither universally similar nor universally orthogonal. Core Idea: teach a semantically ordered latent trajectory using native visual features, then apply stronger distinctiveness constraints to more distant steps so that additional steps carry new visual information.

Method

Overall Architecture

The input is an image and a textual question, and the output remains a textual answer; the added intermediate elements are latent visual thought tokens, not images or textual explanations that must be shown to the user. The framework uses Qwen2.5-VL-7B-Instruct with four reasoning steps and four special tokens per step, giving 16 latent tokens in total. On the training side, Endogenous Self-Distillation supplies visual targets; Progressive Latent Visual Thought organizes intermediate states; and the Distance-Weighted Diversity Constraint suppresses collapse across steps. At inference time, the input image is encoded once, after which learned latent states support derivation and answer generation without producing auxiliary images or executing visual tools at each step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Aux["Training auxiliary images"] --> Teacher["Endogenous<br/>Self-Distillation"]
    Teacher -.->|Training feature supervision| Thoughts["Progressive Latent<br/>Visual Thought"]
    Input["Input image and question"] --> Thoughts
    Thoughts --> Answer["Textual answer"]
    Thoughts -.->|Training state centroids| Diversity["Distance-Weighted<br/>Diversity Constraint"]
    Diversity -.->|Training regularization| Thoughts

Dashed edges denote training supervision or regularization, not inference-time calls to a teacher, segmenter, or evaluator. The auxiliary images are not correct-answer images supplied at test time; the model must learn to predict the corresponding latent visual information from the original input. The following explanation first establishes how supervision reaches the latent tokens, then describes the four states and the need for cross-step constraints.

Key Designs

1. Endogenous Self-Distillation: supervise compressed intermediate states in the same visual space

The lack of annotations for intermediate mental images is a direct obstacle to training latent visual reasoning. For each training sample, the authors programmatically synthesize four auxiliary images, such as a crop based on a target box or a view isolating an object using target information. These operations construct checkable visual trajectories rather than programs that the model must execute at inference time. The frozen native vision encoder then encodes each auxiliary image into a teacher feature sequence for that step. Using the same encoder brings the supervision targets closer to the input visual space on which the student ultimately depends, avoiding adaptation to an external model's preferred texture, semantic, or reconstruction space.

The student has only four tokens per step, whereas each teacher image has a visual sequence of length \(L\), so direct position-wise matching is impossible. The authors linearly project and normalize the hidden states of all latent tokens, then use them as keys and values in a single cross-attention operation. Learnable queries initially correspond to one image's length and are expanded to the total length of four images; the attention output is split into four segments and aligned with the respective teacher features using mean squared error. This decoding head expands compressed states into a representation suitable for training supervision; it does not generate auxiliary-image pixels. Importantly, §3.3 explicitly states that all queries access the same set of compressed latent tokens, rather than restricting each teacher step to its corresponding four student tokens. Feature reconstruction can therefore use information from the entire chain, and the reconstruction objective alone does not guarantee that each token group independently learns its assigned operation.

2. Progressive Latent Visual Thought: assign ordered visual tasks to four token groups

Spatial tasks use a coarse-to-fine causal chain summarized as Locate, Focus, and Isolate, but training implements four states. First, Grid View overlays a spatial grid to establish positional references; second, Bounding Box identifies the relevant target region and performs localization. Third, Crop enlarges and inspects the target region while reducing irrelevant background; fourth, Segmentation isolates the object through its boundaries, separating it from adjacent surfaces. These are not four equivalent data augmentations: they emphasize global references, region selection, local detail, and object boundaries, respectively. Breaking precise target identification into progressively narrower visual evidence is the authors' central rationale for improving over one-step prediction.

Puzzle tasks instead use a dialectical reasoning chain summarized as Hypothesize, Critique, and Verify, again instantiated through four visual operations. The first enhances edges to expose potentially matching contours; the second overlays arrows showing a proposed movement direction; the third constructs a plausible but incorrect assembly; and the fourth presents the correct assembly. The counterfactual example in the third step matters because showing only the correct result does not directly teach the distinction between subtle misalignment and a genuine match. However, this is a training-trajectory design, not evidence that deployment includes an additional search engine enumerating puzzle candidates and explicitly backtracking. The supplied main text does not detail how arbitrary new questions automatically select between the two paradigms, so the method should not be described as a validated general-purpose dynamic routing system.

Both trajectories use four groups of special tokens, with tokens within a group jointly representing an intermediate visual state. The 16-token budget is a fixed experimental configuration, not an adaptive thinking budget selected according to question difficulty. This makes supervision and state comparisons straightforward, but it also restricts trajectory length and visual operations to a predefined structure. Latent thoughts here are hidden representations constrained by training objectives, not necessarily readable and faithful evidence of a natural-language reasoning process.

3. Distance-Weighted Diversity Constraint: preserve nearby shared information while separating distant states

With visual reconstruction alone, successive steps can rely on common global information, leaving the later token groups highly similar. The authors compute a centroid for the tokens in each step and compare the cosine similarity of centroids from different steps. A basic margin-based diversity loss permits some sharing and penalizes only similarity above a learnable threshold, initialized to 0.8. Applying the same penalty to every step pair, however, can excessively separate adjacent states such as cropping and segmentation that should share object appearance.

The distance-weighted version therefore assigns penalties according to the distance between step indices: weaker for nearby states and stronger for distant states. The paper defines this distance as \(d(i,j)=|i-j|\), with maximum distance \(K-1\) and minimum weight 0.1. A learnable steepness parameter constrained to be at least 1 adjusts how the weights increase with distance. For the spatial chain, the global grid and final segmentation should differ substantially, whereas a bounding box and crop can retain common visual cues. This better reflects progressive visual operations than demanding orthogonality everywhere, while giving a concrete training signal for what another step should contribute. Operators and exponents in cached equations (7) through (9) are corrupted, so this note states only the threshold, distance, and penalty behavior supported by the prose rather than guessing the complete loss formula.

A Worked Example

In the television example from Figure 1, the answer to the furniture question is Cabinet, not the floor that also contains wood-like textures. During spatial-trajectory training, the grid view preserves the television's position relative to its surroundings, and the bounding box identifies the relevant furniture region. The crop directs inspection toward the cabinet, while segmentation further emphasizes its boundary against the floor. The frozen encoder converts these four views into teacher features, and the cross-attention reconstruction head uses them to supervise the 16 latent tokens. The diversity constraint discourages later states from copying the initial global representation while allowing the bounding-box and crop states to share features of the same furniture. At test time, the model receives only the original image and question, not these four target views, and uses the learned latent chain to generate Cabinet. This example explains the operations the authors intend to internalize; it does not establish that every test example has an internally generated trajectory verified step by step by humans.

Loss & Training

The overall objective combines language-modeling cross-entropy for the answer, visual distillation, and diversity regularization. Using the prose description of the three components and weights in §3.5, the total objective can be written as:

\[ \mathcal{L}=\mathcal{L}_{\mathrm{LM}}+1.0\mathcal{L}_{\mathrm{distill}}+0.2\mathcal{L}_{\mathrm{div}}. \]

Training starts with latent anchor initialization: the vision encoder and language-model backbone are frozen, and only special thought-token embeddings are trained to place them in a suitable semantic space. The second phase introduces endogenous knowledge distillation, prioritizing reconstruction of target visual features without the diversity constraint. The third phase adds the distance-weighted diversity loss to differentiate steps after reconstruction has been learned, rather than immediately separating states that have not yet acquired meaning. The fourth phase jointly trains on general visual question answering data to reduce forgetting of general instruction-following ability after specialized visual reasoning training. The teacher vision encoder receives no gradients; the main text does not fully specify later backbone-unfreezing choices, training sample counts, or curriculum durations, so these details cannot be inferred from the phase names. Experiments use 2 NVIDIA H20 GPUs; the authors report approximately 28 GPU-hours of training, but the main text does not provide enough detail to reproduce the complete cost accounting.

Key Experimental Results

Main Results

The following selection comes from Table 1 on page 10: all metrics are accuracy (%), the backbone is Qwen2.5-VL-7B-Instruct, and BLINK-J denotes BLINK-Jigsaw. Task-specific scores are retained rather than renamed as a unified capability metric, avoiding conflation of different evaluation types.

Method	MMVP	VisPuzzle	VStar	ChartQA	BLINK	CV-Bench	BLINK-J
Qwen2.5-VL-Instruct	77.33	34.75	76.44	78.45	54.49	73.61	59.33
CoT SFT	77.66	69.75	75.91	75.83	56.54	64.55	68.66
One-step Latent Pred.	79.33	72.00	78.01	63.02	56.28	76.10	66.00
ProLaViT (Base)	78.00	74.00	79.05	76.18	57.49	77.24	71.33
ProLaViT (Full)	79.00	74.25	80.11	78.99	57.23	77.66	76.00

Compared with one-step prediction, Full raises ChartQA from 63.02% to 78.99%, a gain of 15.97 percentage points, and BLINK-J from 66.00% to 76.00%, a gain of 10.00 percentage points. It does not win every metric: one-step prediction reaches 79.33% on MMVP versus Full's 79.00%, and Base reaches 57.49% on BLINK versus Full's 57.23%. Table 1 reports Full's Avg. as 75.11%, but the simple arithmetic mean of its seven displayed scores is approximately 74.75%, without an explanation of an alternative weighting scheme. Thus, 75.11% should be treated only as the paper-reported value, not an average verified from the displayed scores; this note does not silently replace the source value.

Ablation Study

The following reproduces Table 3 on page 13, comparing diversity losses within the progressive latent structure; metrics are accuracy (%). Its Avg. covers only the four listed tasks and is not directly comparable with the seven-column results in Table 1.

Config	VisPuzzle	VStar	ChartQA	BLINK-J	Avg.
No diversity loss	74.00	79.05	76.18	71.33	75.14
Margin-based constraint, no distance weights	75.25	79.05	75.66	67.33	74.32
Distance-weighted constraint, Full	74.25	80.11	78.99	76.00	77.34

Relative to no diversity constraint, a uniform margin penalty reduces BLINK-J by 4.00 percentage points, whereas the distance-weighted version improves it by 4.67 percentage points. The margin-based constraint nevertheless achieves the highest VisPuzzle result of 75.25%, showing that distance weighting improves overall performance rather than being optimal for every task. Additionally, Table 5 on page 14 shows that randomly permuting the steps reduces CV-Bench from 77.66% to 69.14%, supporting the usefulness of operation order without establishing complete latent-chain faithfulness.

Key Findings

Table 6 on page 15 compares explicit tool use with latent distillation; latency and compute are normalized to the base model and should not be read as seconds or unnormalized absolute TFLOPs.

Method	VStar accuracy (%)	BLINK-J accuracy (%)	Relative latency	Relative compute
Qwen2.5-VL (Base)	76.44	59.33	1.00×	1.00
Explicit Tool-Use SFT	79.60	74.00	3.40×	2.17
ProLaViT	80.11	76.00	1.21×	1.12

The efficiency benefit comes from avoiding repeated auxiliary-image encoding, not from eliminating additional reasoning: ProLaViT still adds 21% latency and 12% compute over the base model. Table 4 on page 14 also reports VisPuzzle accuracy of 74.25% with the native teacher, 73.75% with DINOv2, and 45.50% with SDXL-VAE; teacher representation compatibility matters more than merely adding an external vision model.

Highlights & Insights

• Supervision concerns how visual states change, not just the final answer. Programmable operations therefore become a resource for training intermediate representations without necessarily becoming deployment dependencies.
• Diversity regularization should not indiscriminately maximize decorrelation. Encoding step relationships into penalty strengths suggests that ordered state learning should preserve local continuity.
• A native teacher avoids forcing the student into another expert's feature space. Table 4 supports this choice in the tested setting, but does not prove that all external teachers must be inferior.

Limitations & Future Work

• The authors explicitly limit the current scope to spatial and logical reasoning on static images; temporal video reasoning and complex geometric transformations remain future directions.
• Four-step templates depend on synthesizable operations and target information; the supplied main text refers synthesis details to supplementary material and does not explain how boxes and masks are obtained for all real-world settings.
• Reader interpretation: centroid differences and similarity heatmaps support reduced collapse, but do not independently establish the causal role of each step, especially when the training decoder can read across groups.
• The text-trajectory comparison suggests benefits from latent representations, but the averaging scope in §4.4 is insufficiently documented; the authors' strong claim that the benefit arises purely from continuous-space reasoning should not be repeated as established causality.
• Corrupted extracted formulas, the inconsistent Table 1 average, and missing repeated-run variance limit exact reproducibility and statistical confidence; this note does not fill these gaps by guessing.

Related Work & Insights

• Compared with CoT SFT: textual reasoning compresses visual information into discrete language; ProLaViT adds continuous states supervised by visual features, but does not provide an equally readable reasoning trace.
• Compared with LVR and Mirage: the emphasis is on explicitly defined multi-step operation trajectories and distance-aware constraints, rather than only demonstrating that latent visual tokens can assist reasoning.
• Compared with CoVT: the differences concern teacher selection and trajectory structure; ProLaViT uses a frozen native encoder to reduce external-expert dependencies, not to eliminate visual supervision.
• Compared with explicit visual generation such as ThinkMorph: ProLaViT avoids pixel-level intermediate outputs at inference time for greater efficiency, but loses the direct observation channel provided by generated intermediate images.
• Research extension: masking or swapping latent groups under an identical token budget and measuring answer changes could test whether localization, focusing, and isolation have distinct roles; this is a proposed experiment, not one completed in the paper.

Rating

• Novelty: 4/5. Combines operation trajectories, native self-distillation, and distance-aware regularization, emphasizing structure rather than a new backbone.
• Experimental Thoroughness: 3/5. Includes structure, teacher, ordering, and tool-efficiency comparisons, but averaging conventions and reproduction details remain incomplete.
• Writing Quality: 3/5. The methodological intent is clear, but some causal language exceeds the evidence, and the current cache does not reliably preserve all formulas.
• Value: 4/5. Offers a useful training approach for low-overhead visual reasoning, particularly for tasks with well-defined intermediate visual operations.