CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

Conference: ECCV2026
Paper: Official paper page · PDF
Code: https://github.com/hulianyuyy/CoLT
Area: Multimodal Reasoning
Keywords: latent chain of thought, forward decoding, backward alignment, step-level supervision, inference acceleration

TL;DR

CoLT compresses multimodal textual reasoning into 3 continuous latent states by default using training-time forward text decoding, backward semantic alignment, and adjacent-state prediction, achieving a 79.1 average over eight benchmarks and approximately 10.1x end-to-end speedup over text CoT on MMStar using a single H200.

Background & Motivation

Textual chain-of-thought in visual question answering has a direct cost: the model must not only recognize an image and find an answer, but also write its explanation token by token. Even when the answer is just a number, describing the chart, identifying evidence, and spelling out calculations consume autoregressive decoding time. In the paper's MMStar comparison, text CoT produces an average of 142.1 reasoning tokens and spends 7.24 seconds on generation, far more than its 0.47 seconds of input encoding. The target for compression is therefore the expression of reasoning, not merely the visual input.

Methods such as CoCoNut and CODI move reasoning into continuous hidden space, but replacing text tokens with vectors does not guarantee that those vectors preserve progressively useful semantics. Final-answer supervision alone does not identify which intermediate step went wrong, and the latent states can become difficult to interpret. Another family of latent visual reasoning approaches uses additional intermediate images for supervision, introducing auxiliary visual annotation requirements. CoLT instead reuses existing textual CoT: text need not be generated at deployment, but it can check whether hidden states contain reasoning information during training.

Core idea: use textual reasoning as a removable training-time semantic constraint, supervise latent-state content through bidirectional decoding, and constrain transitions through internal prediction so that deployment requires only a short latent chain and final-answer generation.

Method

Overall Architecture

The inputs are an image and a text question. A vision encoder produces visual tokens, which enter Qwen3-VL-8B-Instruct together with the question. The backbone generates 3 latent thought states by default, feeding each state back as the input embedding for the next position. The final answer can attend to the entire latent chain rather than depending only on its last vector.

Training has two stages: supervised fine-tuning with textual CoT, a bidirectional external decoder, and an internal prediction head; then outcome-reward optimization with GRPO after removing the auxiliary constraints. Standard inference retains only the backbone, completing the configured number of latent steps before generating the answer. The paper does not introduce an adaptive stopping criterion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and question"] --> Latent["Continuous-state feedback"]
    Latent --> Answer["Final-answer generation"]
    Latent --> External["Bidirectional external<br/>decoder supervision"]
    Text["Training textual CoT"] --> External
    Latent --> Internal["Internal adjacent-state<br/>prediction"]
    External --> Training["Constrain-then-release<br/>training"]
    Internal --> Training
    Answer --> Training
    Training --> Deploy["Deployment: backbone only<br/>latent chain and answer"]

Decoder supervision and adjacent-state prediction are parallel training branches, not additional models called sequentially at inference. Textual CoT also supplies training targets only; testing does not require reference answers, teacher reasoning, or auxiliary intermediate images.

Key Designs

1. Continuous-state feedback: one latent position carries a segment of reasoning

Ordinary language generation maps the final-layer hidden state into the vocabulary, selects a discrete token, and feeds its embedding back into the model. At latent reasoning positions, CoLT bypasses this discrete selection and directly feeds the final-layer hidden state back as the next position's input embedding. Self-attention still accesses the image, question, and all preceding latent states. The method therefore does not remove autoregressive computation; it compresses many textual generation positions into a small number of continuous positions. The paper states that it neither extends the vocabulary nor introduces new special tokens.

Training text is dynamically divided into \(K\) segments to supply semantic targets for the latent steps, with \(K=3\) by default. A state is thus a compressed representation of a reasoning segment, not a directly readable word. The paper describes this process in terms of both logical steps and consecutive text splits, but the cache does not provide a complete boundary-selection algorithm. It would be unjustified to assume sentence-boundary splitting or exactly three natural-language logical steps in every example. Preserving multiple possible paths in latent space is the authors' interpretation, not an explicit enumeration or execution of branching search.

2. Bidirectional external decoder supervision: check decodable semantics and learn text-to-latent correspondence

The external decoder is Qwen3-0.6B, sharing the backbone's tokenizer and vocabulary. In forward mode, it conditions on preceding latent states, autoregressively predicts the next textual reasoning segment, and receives a negative log-likelihood loss on the target text. The important constraint is not another prediction of the final answer: the compressed state must support an expansion of the next reasoning step. Decoder gradients propagate through the latent states into the backbone, providing finer supervision than whether the final answer is correct.

In backward mode, the decoder receives the preceding textual reasoning segment, extracts its final-layer hidden state, and aligns its normalized direction with the corresponding backbone latent state. Here, backward means a text-to-latent mapping, not reconstructing the entire reasoning chain from the final answer. Crucially, the backbone latent state is a stop-gradient target: this loss directly trains the decoder side rather than pulling the backbone toward potentially degenerate decoder outputs. Sharing that decoder with forward training creates an indirect coupling; the three losses should not all be described as directly updating the same latent vector. The norm and exponent in the backward-loss expression are incompletely extracted in the cache, so its exact algebraic form is not reconstructed here.

3. Internal adjacent-state prediction: constrain predictability between states

Even if individual latent states can each decode plausible text, adjacent states may fail to connect coherently. The authors add a two-layer MLP with GELU to predict the next latent state from the current one, using cosine distance to encourage a learnable transition along the chain. This is not another answer classifier, nor does it require adjacent states to be identical: the MLP can learn a transformation between them.

Using the prose definition and the recoverable content of Equation (9), the internal loss can be written as:

\[ \mathcal L_{\mathrm{int}}= \frac{1}{K-1}\sum_{k=1}^{K-1} \left[1-\frac{f_\theta(\mathbf h_k)^{\mathsf T}\operatorname{sg}(\mathbf h_{k+1})} {\|f_\theta(\mathbf h_k)\|_2\,\|\operatorname{sg}(\mathbf h_{k+1})\|_2}\right]. \]

Here, \(\operatorname{sg}\) denotes stop-gradient: this alignment does not move the target branch, while the prediction head and current-state branch receive training signals. The authors motivate it as reducing representation-collapse risk, not as a mathematical guarantee against collapse. There is also an implementation boundary at \(K=1\): no adjacent-state pair exists, and the expression cannot be evaluated directly. Although the paper reports a single-step experiment, the cache does not explain how the implementation handles this branch.

4. Constrain-then-release training: establish structure before outcome rewards refine the latent strategy

The first stage trains the backbone's language-model component and the external decoder together, allowing answer prediction, textual correspondence, and state transitions to shape the latent chain jointly. The second stage applies GRPO on 50K CoT samples, removing external decoder losses and internal prediction constraints and optimizing through outcome rewards only. The intention is to first avoid unstable training in an unconstrained latent space, then allow the model to move beyond intermediate trajectories prescribed by teacher text.

The external decoder and prediction head are both removed at deployment. Consequently, the small decoder affects training cost, not the cost of each deployed question-answering request. For qualitative analysis, the authors can reuse the forward decoder to read out latent content, but this is an interpretability procedure rather than part of the default low-latency inference path. The paper does not independently quantify the GRPO-stage gain or sufficiently explain how continuous states enter policy-probability calculations; a specific policy-gradient implementation cannot be inferred from the algorithm name alone.

A Worked Example

The MathVista place-value example in Figure 2 asks which number is represented by a collection of blocks. Its decoded text identifies 9 thousand-blocks, 5 hundred-flats, 6 ten-rods, and 7 unit-blocks, yielding \(9000+500+60+7=9567\) and a final answer of 9567.

In standard CoLT execution, the backbone first encodes the image and question, processes the configured 3 latent positions, and outputs the number. Only during interpretability analysis are the latent states passed to the forward decoder to obtain text about identifying place values, computing components, and summing them. The cache loses the original figure's color segmentation, so the exact sentence-to-state assignment cannot be recovered; those operations are not forcibly assigned to steps 1, 2, and 3 here. This example demonstrates decodable reasoning content, but not that the decoded text completely exposes the causal process behind the answer.

Loss & Training

Following the prose explanation of Equation (10), the supervised-stage objective is:

\[ \mathcal L=\mathcal L_{\mathrm{task}} +\alpha\mathcal L_{\mathrm{fwd}} +\beta\mathcal L_{\mathrm{bwd}} +\gamma\mathcal L_{\mathrm{int}}, \qquad \alpha=\beta=\gamma=0.2. \]

The task loss is final-answer next-token negative log-likelihood conditioned on the input and latent chain. The forward term supervises reasoning text, the backward term aligns directions, and the internal term predicts adjacent states. PDF extraction has scrambled symbols in some cached equations; the objective above only consolidates components supported by the prose and does not reconstruct the damaged backward-distance details.

Implementation uses the image subset of OneThinker for 1 training epoch, AdamW, a cosine learning-rate schedule, batch size 8, and 3 latent steps by default, followed by the aforementioned GRPO stage on 50K samples. The cache omits the actual learning rate, GRPO group size, reward function, and full training time, and does not sufficiently specify hidden-dimension adaptation between models. A shared vocabulary does not automatically imply equal hidden dimensions, so the claimed alignment without projection requires implementation verification rather than being treated here as an established fact.

Key Experimental Results

Main Results

The following same-backbone comparison is selected from the paper's Table 1. Scores retain the paper's percentage scale, with higher being better; each benchmark has its own evaluation protocol, and Avg. is the paper's aggregate score rather than pooled accuracy over all examples. The final column subtracts displayed values and is measured in score points.

Benchmark	SIM-CoT	Qwen3-VL textual reasoning	CoLT	Gain over textual reasoning
SeedBench	72.4	76.4	77.5	+1.1
MMBench	82.0	83.4	84.6	+1.2
ChartQA	66.7	65.1	74.7	+9.6
TextVQA	70.2	75.2	81.3	+6.1
ScienceQA	90.8	91.8	92.8	+1.0
MMStar	65.6	67.1	68.9	+1.8
AI2D	82.0	83.6	85.4	+1.8
MMT-Bench	62.4	63.3	67.4	+4.1
Avg. (reported)	74.0	75.7	79.1	+3.4

The main table prints an average improvement of +3.3, while the prose states +3.4; subtracting the already-rounded averages gives \(79.1-75.7=3.4\). Averaging individual benchmark scores before subtraction can produce a different rounding result. This note preserves the reported averages and makes its calculation convention explicit rather than mixing the two versions. The average gain over SIM-CoT is 5.1 score points, but this does not imply superiority over every open-source model: the original table reports 86.6 on ChartQA for InternVL3, above CoLT's 74.7.

Ablation Study

The paper's Table 2 uses a four-benchmark subset: SeedBench, MMStar, AI2D, and MMT. These averages are not directly comparable to the eight-benchmark Avg. above. No auxiliary supervision still retains task learning; it does not mean an untrained model.

Forward decoding	Backward alignment	Internal prediction	Four-benchmark Avg.
Off	Off	Off	61.4
On	Off	Off	65.8
Off	On	Off	67.2
Off	Off	On	63.6
On	On	Off	69.8
On	Off	On	68.1
Off	On	On	69.1
On	On	On	72.6

Removing backward alignment lowers the full configuration from 72.6 to 68.1, a loss of 4.5 score points; removing forward decoding or internal prediction loses 3.5 or 2.8 points, respectively. This supports complementary supervision. However, the full ablation row also has lower per-benchmark scores than the same benchmarks in the main table, and the cache does not clearly establish whether this reflects the RL stage or another setting difference. That gap cannot simply be attributed to GRPO.

Key Findings

• More steps are not necessarily better. In Table 4, the four-benchmark averages for \(K=1,2,3,4,6,8\) are 66.8, 70.3, 72.6, 72.2, 71.7, and 71.2. Matching training and testing at 3 steps performs best; additional latent computation does not automatically improve reasoning.
• A lightweight decoder provides most of the benefit. Table 3 reports averages of 72.6, 72.9, 73.1, and 73.2 for 0.6B, 1.7B, 4B, and 8B decoders. Large increases in decoder size yield limited improvements, but corresponding training costs are absent, preventing a precise cost-benefit calculation.
• Noise robustness is not a mechanism proof. For 30% occlusion on MMStar, Table 6 reports accuracy degradation of 19.0, 13.8, and 9.0 for Direct Answer, Text CoT, and CoLT, retaining the original table's percentage convention. This supports robustness in that setting, but does not directly prove that multipath search causes the benefit.

The speed measurements below come from Table 7, using Qwen3-VL-8B on a single H200 and per-sample averages. Reasoning positions mean textual tokens for text CoT and continuous latent states for CoLT, not final-answer token counts.

Benchmark	Method	Reasoning positions	Encoding time (s)	Generation time (s)	Score
MMStar	Direct Answer	0	0.45	0.14	58.6
MMStar	Text CoT	142.1	0.47	7.24	67.1
MMStar	CoLT	3	0.44	0.32	68.9
MMT-Bench	Direct Answer	0	0.46	0.15	55.6
MMT-Bench	Text CoT	138.5	0.48	7.38	63.3
MMT-Bench	CoLT	3	0.45	0.33	67.4

MMStar generation speedup is \(7.24/0.32\approx22.6\), while end-to-end speedup is \((0.47+7.24)/(0.44+0.32)\approx10.1\). CoLT remains slower than direct answering: its benefit is the combination of reasoning quality and latency relative to text CoT, not being the fastest strategy overall. Table 7's caption names MMStar and SeedBench, whereas its column headers and experimental prose name MMStar and MMT-Bench. This note follows the latter two and explicitly preserves the source discrepancy.

Highlights & Insights

• Move explainability capacity to training. The decoder requires latent states to carry textual semantics without making deployment pay for generating explanations. This is useful when final answers are short but detailed process annotations exist for training.
• Content and transition supervision address different layers of the problem. External decoding constrains what a state contains, while internal prediction constrains how states connect. The ablations make their non-redundancy clearer than final accuracy alone.
• Gradient direction matters more than remembering three loss names. Backward alignment stops gradients through the backbone target and is not simply a textual teacher directly distilling a student. The shared decoder's coupling is the mechanism worth transferring and testing further.

Limitations & Future Work

• Textual process data is still required. Avoiding auxiliary intermediate images does not eliminate extra supervision; CoT quality and dynamic segmentation can still determine what the latent states learn.
• Scope and cost evidence remain limited. Experiments mainly cover one 8B backbone and image tasks, without establishing performance on long videos, other model families, or longer multi-turn tasks. Inference speedup also does not substitute for training-cost, memory, or throughput evaluation.
• Reproduction details have gaps. Hidden-dimension adaptation, handling the single-step internal loss, GRPO policy definitions, and differences between main and ablation settings require further verification. The cached paper cannot justify filling in those implementation details.
• Cross-step robustness is overstated. In Table 5 on MMStar, training with 3 steps and testing with 1 lowers the score from 66.5 to 61.0; not every step-count mismatch causes only a minor drop. Adaptive step counts are an author-proposed future direction, not a mechanism already validated here.
• Decodability does not guarantee faithful explanations. The qualitative examples lack causal interventions or a systematic failure-rate analysis. Latent-state interventions, failure-chain diagnostics, and the authors' proposed hybrid latent-text reasoning could test when explicit verbal correction is necessary.

Related Work & Insights

• Compared with CoCoNut and CODI: all move from discrete text toward continuous reasoning. CoLT focuses on step-level supervision in multimodal tasks and the combination of forward, backward, and internal signals, rather than introducing latent chains of thought for the first time.
• Compared with SIM-CoT: both use auxiliary decoders and step-level supervision, so decoding latent states is not the whole novelty. The more informative question is whether backward semantic correspondence and adjacent-state prediction provide independent gains.
• Compared with LVR and MoNet: these methods use auxiliary image features to constrain latent visual reasoning, whereas CoLT substitutes textual process supervision. Its interpretation therefore depends more on semantics expressed in text; better preservation of fine-grained spatial information cannot be assumed.
• A transferable research question: use existing process annotations to train a removable checker, while evaluating information preservation, state transitions, and deployment cost separately. Separating SFT and RL contributions should take priority over simply increasing latent step counts.

Rating

• Novelty: 4/5. The three-way supervision combination and multimodal application have clear value, but latent chains and auxiliary decoders have precedents.
• Experimental Thoroughness: 4/5. Eight main benchmarks plus supervision, step-count, decoder, noise, and speed analyses provide broad evidence, but independent RL ablations and repeated-run uncertainty are missing.
• Writing Quality: 3/5. The main narrative is clear, but some table conventions, cross-step claims, and implementation details require clarification, and the cache contains damaged equation extraction.
• Value: 4/5. Short latent chains show practical potential for reducing textual reasoning latency, but deployment benefits should be reassessed after reproduction and cross-model validation.