LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/Svardfox/LaViT
Area: Vision-Language Reasoning
Keywords: latent visual thoughts, knowledge distillation, attention alignment, sensory gating, visual grounding

TL;DR¶

LaViT makes Qwen2.5-VL-3B generate four continuous visual tokens before answering, distills visual semantics and attended regions from a 32B teacher, and gradually opens the direct image-access path to prevent bypassing those tokens, improving BLINK Relative Depth from 61.29% to 78.23%.

Background & Motivation¶

Vision-language models (VLMs) can produce fluent chain-of-thought (CoT) without relying on the correct image evidence. Supervised fine-tuning a smaller model on a larger model's answers or reasoning text primarily constrains what to say. It does not directly ensure that both models attend to the same locations when judging color, spatial relationships, or depth. Earlier latent visual reasoning methods compress intermediate computation into continuous hidden states, but many rely on auxiliary images, bounding boxes, or static vision-encoder features, leaving the teacher's internal visual behavior during answering underused.

The paper makes this problem concrete with two observations. On 1,000 randomly sampled Visual-CoT examples, Qwen2.5-VL-32B has mean visual focusing scores of 18.78% for correct answers and 9.18% for incorrect answers. This score is the fraction of total image attention mass falling inside the target bounding box. On another 1,000 held-out examples, a text-only Naive SFT student has similar hidden-state cosine distances from its teacher across token categories, while attention KL divergence rises from 1.11 for functional tokens to approximately 1.39 for attribute tokens. Similar output representations and aligned visual attention are therefore distinct, although these observations remain correlational.

LaViT consequently does more than require answer imitation. It assigns continuous pre-answer states two specific responsibilities: retaining question-relevant visual content and reproducing the teacher's attended image locations. The difficulty is that answer tokens can still ignore these new states if they always have unrestricted access to image tokens. Core Idea: supervise latent visual thoughts with both teacher semantics and attention, initially suppress direct image access by answer tokens, and then restore it smoothly so that the latent states first become necessary information carriers.

Method¶

Overall Architecture¶

The inputs are an image and a question, and the output remains a textual answer. Training examples are filtered and augmented with visual features and attention from a white-box teacher. The student then autoregressively generates four continuous visual tokens before producing its answer. Teacher supervision constrains these latent states, while the answer loss teaches them which visual information must be retained.

Curriculum sensory gating controls the attention path through which answer tokens read the original image. It does not remove the image from the entire model input. Latent tokens still need to read the image to construct their representations. At inference time, direct image access is fully open, and neither the teacher nor the bounding boxes used for filtering are required.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training image, question,<br/>answer and boxes"] --> B["White-box Supervision Construction"]
    C["Image and question"] --> D["Dual Latent Visual Alignment"]
    B -.->|Training only: semantics and attention| D
    D -->|Four continuous visual tokens| E["Curriculum Sensory Gating"]
    C -->|Direct image access for answers| E
    E -->|Gradual opening in training; fully open at inference| F["Textual answer"]

Key Designs¶

1. White-box Supervision Construction: select teacher traces that depend on visual evidence

LaViT-SFT-15K contains 15K tuples of images, questions, answers, attention maps, and semantic features. The teacher is Qwen2.5-VL-32B. Filtering successively requires a correct answer, failure of a text-only model to solve the example, and at least 20% of aggregated attention mass inside the target boxes annotated in Visual-CoT. The final criterion excludes not just wrong answers but also supervision where an answer happens to be correct despite attending to the wrong locations. Boxes are used only for offline filtering, so the method is neither entirely box-annotation-free nor trained to directly fit box coordinates.

For retained examples, the authors average the teacher's attention from generated answer tokens to image patches across layers, heads, and answer length. The resulting patch saliency undergoes Min-Max normalization, and only the 8 highest-attention locations are retained as sparse supervision. Semantic targets come from image-token representations in the teacher's final Transformer layer. The authors interpret these as question-contextualized visual thoughts rather than static outputs of an independent vision encoder. Importantly, the resulting “trajectory” is an aggregated spatial map, not the full time-resolved sequence of the teacher's token-by-token attention. This distinction determines the granularity of the transferred behavior.

2. Dual Latent Visual Alignment: make intermediate states retain content and locate evidence

After the image and question, the student generates continuous latent tokens before entering text decoding. These are not verbal instructions such as “look left, then right,” nor must they decode into new images. They are hidden states that carry visual information. The paper calls this training Latent Teacher Forcing: instead of supervising only the final language output, it adds a pre-answer computation that can be constrained by the teacher's internal signals. The four states are generated sequentially, allowing later states to use information accumulated earlier.

The two targets have different roles. Semantic reconstruction uses an MLP projection to map student latent states into the teacher's feature space and increase cosine similarity. Attention alignment uses KL divergence to bring latent-token attention over image patches closer to the teacher target. The prose specifically states that attention supervision applies only to the last latent token, intending the preceding three to aggregate and compress information rather than requiring every step to independently reproduce the same map. The semantic loss constrains what the container holds; the attention loss constrains where its information comes from. Either objective alone leaves one of these requirements insufficiently specified.

3. Curriculum Sensory Gating: establish an information bottleneck before restoring detail access

If answers can always access the image directly, latent tokens may become a side branch optimized by auxiliary losses but unused by the answer itself. LaViT adds an attention bias of \(\ln\gamma(t)\) between answer queries and image keys. Training starts at \(\gamma=10^{-6}\), producing a strongly negative bias on this direct path. This does not delete image inputs. It suppresses the path before softmax, encouraging answers to depend primarily on evidence carried by latent states.

During the first 400 steps, a cosine-style schedule gradually opens the gate. The next 600 steps use \(\gamma=1\), removing the bias and allowing original-image access to act as a residual perception path for fine-grained details. Early training pressure discourages bypassing the latent states; later training teaches joint use of original images and latent states under the same visibility conditions as inference. The authors describe this as avoiding a training-inference topology mismatch, but it does not establish the absence of every distribution shift. Likewise, the paper's approximate relationship between gradients and gating should not be treated as an exact identity for all model gradients.

A Worked Example¶

Consider the relative-depth question in Figure 5: an image marks locations A and B, and the model must identify which is closer to the camera. During offline supervision construction, the teacher first answers. The example is retained only if the answer is correct, language alone cannot solve the problem, and attention passes the target-box coverage threshold. Its visual semantics and 8 strongest attention locations are then stored, without creating an additional explanatory image for the student.

The student reads the same image and question and generates four latent states. Early in training, answer tokens can barely look back at the original image, so information about the two locations and depth cues must pass through the latent states. The final state's attention must also match the teacher. As the gate opens, the answer can use both this compressed representation and original-image details. Inference retains the four latent computation steps but requires neither the teacher nor target boxes. This example does not assume that one token necessarily represents A and another represents B; the paper assigns no such interpretable labels to individual steps.

Loss & Training¶

The total objective combines next-token prediction for the answer with a fixed weight of 0.3 multiplying the sum of semantic reconstruction and attention alignment. Gating changes during training, but the three losses are optimized jointly rather than training one module fully before another. The paper's “two stages” primarily refer to how direct visual access opens.

The student starts from Qwen2.5-VL-3B and trains on LaViT-SFT-15K with 4 latent tokens, 8 retained attention locations, and 1,000 total steps. Teacher features are fixed targets, and the semantic projection head handles the difference between student and teacher representation spaces. The method introduces neither reinforcement learning rewards nor online visual tools.

Several equations have missing characters in the supplied text extraction. This note therefore retains only objective components and gating relationships confirmed by the prose, without reconstructing exact normalization, KL summation, or cosine-schedule equations. The body refers to an appendix for further training details, but the supplied cache ends with references. It does not provide the learning rate, batch size, hardware, or training time, so training cost cannot be estimated from it.

Key Experimental Results¶

Main Results¶

The following values are selected from Table 1 and are percentage-scale evaluation scores, with higher being better. Gains are absolute percentage points. BLINK uses the four listed tasks, and V* uses only its Attribute Recognition subset; these are not overall benchmark scores. The body does not explicitly list test split names or sample counts. LaViT and Naive SFT are both 3B models, with the latter learning only teacher text from the same dataset.

Evaluation Task	Qwen2.5-VL-3B	Naive SFT-3B	LaViT-3B	Gain over Base Model
MMVP	62.33	65.33	67.33	+5.00
V* Attribute Recognition	81.74	80.87	82.61	+0.87
BLINK Relative Depth	61.29	70.16	78.23	+16.94
BLINK IQ-Test	24.0	28.0	32.0	+8.00
BLINK Relative Reflectance	29.85	34.33	45.52	+15.67
BLINK Spatial Relation	81.12	81.82	81.82	+0.70
MMStar	50.2*	55.53	54.07	+3.87

The asterisk follows the original table and denotes a result reported by an earlier paper. LaViT improves all seven scores over the base model, but not over Naive SFT: Spatial Relation is tied, and MMStar is lower by 1.46 percentage points. Across model scales, its Relative Depth score exceeds LVR-7B's 76.61%, but this does not establish superiority over 7B models on every task or under matched training budgets.

Ablation Study¶

The following results come from Table 3 and remain percentage-scale scores. Except for the explicitly identified inference-time masking experiment, the variants test training objectives or gating settings. They should not all be interpreted as removing a network module and retraining.

Config	MMVP	Relative Depth	IQ-Test	Relative Reflectance	Spatial Relation
Full LaViT-3B	67.33	78.23	32	45.52	81.82
Without attention alignment loss	64.33	75	30.67	44.03	78.32
Without semantic reconstruction loss	65.33	75.81	30.67	42.54	76.92
Two-stage hard switching instead of curriculum gating	59.33	71.77	27.33	48.51	79.72
Latent tokens masked at inference	64.33	77.42	25.33	40.30	81.12
Single stage with direct visual access always open	65.67	71.77	28	38.81	79.02

Hard switching means a gate of 0 in the first phase and 1 in the second, not permanently removing gating. The always-open variant is the final row. Although hard switching lowers MMVP by 8.00 percentage points, it raises Relative Reflectance to 48.51%, above the full model. The benefit of curriculum gating is therefore task-dependent.

Key Findings¶

Both distillation signals contribute, but differently: removing semantic reconstruction lowers Spatial Relation by 4.90 percentage points, while removing attention alignment lowers Relative Depth by 3.23 percentage points.
Masking latent tokens at inference lowers IQ-Test from 32 to 25.33, supporting their use by the model. Relative Depth drops by only 0.81, so the experiment does not show equal reliance on the latent chain across tasks.
In Table 2, mean attention entropy on Relative Depth falls from 4.870 for the base model to 4.686, compared with 4.284 for the teacher. Entropy is computed over normalized patch weights: lower entropy means a more concentrated distribution, not necessarily attention to the correct regions.
Salient regions are patches with weights above 1.5 times the global mean, and the coefficient of variation of their count is standard deviation divided by mean. Table 2 reports coefficients of 0.392 for the teacher and 0.102 for LaViT, but some coefficients do not match the displayed means and standard deviations. This note does not silently correct them; the “more stable” conclusion warrants checking the underlying statistics.

Highlights & Insights¶

The distillation target expands from linguistic answers to both content and evidence locations. This makes failures where a student uses language priors to bypass visual evidence easier to expose than answer matching alone.
Gating changes the answer's information-access path rather than simply increasing an auxiliary loss. It provides a reusable structural intervention for intermediate states that train well but remain unused.
Restoring original-image access means latent states do not have to preserve every detail losslessly. Latent reasoning and direct perception can complement each other instead of requiring a permanent choice between them.

Limitations & Future Work¶

Conditions explicitly stated by the authors: the method requires white-box access to teacher representations and attention, plus Visual-CoT target boxes for offline filtering. It cannot be transferred directly to a closed API teacher.
Assessment in this note: averaging across layers, heads, and answer length removes observation order. The evidence therefore more directly supports aggregated visual-attention alignment than complete inheritance of a stepwise visual search strategy.
Assessment in this note: the body provides no multiple-seed results, confidence intervals, additional latency or memory measurements, or systematic sweeps of latent length and Top-N. A smaller parameter count does not necessarily imply lower end-to-end cost.
Assessment in this note: correlation between attention concentration and correct answers does not establish causal necessity, and hard switching performs better on Relative Reflectance. Evidence-region occlusion and task-stratified experiments could distinguish genuine evidence dependence from changes in attention metrics.

vs Naive SFT: text supervision from the same dataset already improves Relative Depth and MMStar. LaViT adds constraints on internal visual states and gains substantially on tasks such as depth, but its MMStar score is lower. Alignment is not a universal performance switch.
vs LVR / CoVT: as described in this paper's related work, these latent visual methods use signals such as boxes or external vision encoders. LaViT emphasizes internal attention and semantics during teacher answering, while retaining its own dependence on box-based filtering.
vs DistillVLM / MAD: attention and cross-modal feature distillation already exist. The more specific contribution here combines continuous pre-answer states, dual constraints, and progressively opened direct perception, rather than introducing attention distillation itself.

Rating¶

Novelty: 4/5. The combination of latent states, white-box distillation, and sensory gating is clear, although its component techniques are not entirely new.
Experimental Thoroughness: 3/5. Seven evaluations and objective, gating, and inference-dependence ablations are provided, but cost, variance, and broader backbone validation remain missing.
Writing Quality: 3/5. The perception-gap narrative is clear, but the distinction between trajectories and aggregated maps, statistical consistency, and the strength of some claims require careful reading.
Value: 4/5. The work offers useful guidance for training evidence dependence in small VLMs, within the constraints of white-box teachers and filtered data.