StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning¶

Conference: ECCV2026
arXiv: 2607.00465
Code: https://yuanqing-ai.github.io/StochasT
Area: Multimodal VLM
Keywords: Visual Instruction Tuning, Multi-turn Dialogue, Contextual Robustness, Stochastic Depth, Training-Evaluation Mismatch

TL;DR¶

StochasT proposes randomly pruning the historical dialogue context during visual instruction tuning (VIT) of large visual-language models (LVLMs), unfolding a fixed chain into a dialogue tree. This trains an implicit multi-depth ensemble model that optimizes both single-turn and multi-turn evaluation performance. Along with this, a robustness evaluation framework based on Balanced Latin Squares and the CRA/CRA+ metrics are introduced.

Background & Motivation¶

Large Visual Language Models (LVLMs) typically require Visual Instruction Tuning (VIT) to activate their multimodal reasoning capabilities. In standard VIT practices, a common approach is to pack multiple question-answer pairs for the same image into a multi-turn dialogue for training. This "one-image-multiT" format efficiently reuses images and enriches training samples, being widely adopted by classic frameworks such as LLaVA. However, almost all current LVLM evaluation benchmarks (MMBench, MMMU, MME, Seed-Bench, etc.) employ a single-turn evaluation protocol, where each question is independently paired with the image, and the model answers without historical context.

This exposes a severe training-evaluation mismatch. Models perform well in multi-turn dialogues because previous Q&As provide rich contextual cues to assist the current question; however, the same models suffer a sharp decline in performance in single-turn scenarios. Data from one paper indicates that switching evaluation from single-turn to multi-turn dynamically boosts the performance of several SOTA models. Closer analysis reveals that standard multi-turn training itself amplifies this issue: the model learns to rely on textual context shortcuts rather than genuine visual understanding to answer, leading to visual attention decay and contextual overfitting. In other words, the fixed-length multi-turn context during training becomes a "crutch" for the model in single-turn scenarios.

The key insight of this paper is that single-turn and multi-turn capabilities are not irreconcilable. The key lies in breaking the fixed context depth during training, enabling the model to construct stable answers across various context lengths. Core Idea: Introduce Stochastic Turn Depth into VIT. Through a random backward traversal mechanism controlled by a Beta distribution, each multi-turn dialogue is dynamically unfolded into a dialogue tree. Different samples within the same batch possess varying context lengths, thereby training an implicit multi-depth ensemble model that naturally harmonizes single-turn and multi-turn capabilities without increasing training tokens.

Method¶

Overall Architecture¶

Inspired by Dropout and Stochastic Depth, the core idea of StochasT is not to discard neurons or residual blocks, but to randomly prune the historical context dependencies between dialogue turns. Given a standard multi-turn dialogue containing \(N\) turns of Q&As, StochasT performs backward traversal for each turn: starting from its previous turn, a dropout probability \(p_k\) is sampled from a Beta distribution to decide whether to "skip" this historical turn. The first retained turn encountered serves as its parent node. If all historical turns are discarded, it connects directly to the root node (image + system prompt). Consequently, the linear chain is unfolded into a directed tree structure.

Unlike a more intuitive "Turn Dropout" baseline (which directly deletes turns and removes their loss tokens), StochasT retains the loss tokens of all \(N\) turns for gradient calculation, only modifying their contextual connection relationships. This is a crucial distinction: it does not sacrifice any training data, only altering the organization of data under causal attention. Since the dialogue tree is randomly re-sampled in each epoch, the model indirectly experiences a "continuous spectrum" from single-turn zero-context to full deep multi-turn dialogue during training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Root Node<br/>Image + System Prompt"] -->|"Direct connection"| B["Turn 1"]
    A -->|"Skip Turn 1"| C["Turn 2"]
    B -->|"Keep Turn 1"| C
    A -->|"Skip Turns 1-2"| D["Turn 3"]
    B -->|"Keep Turn 1<br/>Skip Turn 2"| D
    C -->|"Keep Turns 1-2"| D
    D --> E["More turns<br/>Expanded according to rules"]

When the dropout probability approaches 1, StochasT degenerates into single-turn training (where every turn connects directly to the root); when it approaches 0, it degenerates into standard multi-turn training. The intermediate random states perfectly balance the advantages of both.

Key Designs¶

1. Causal Backward Traversal: Tree-Construction Algorithm from Chains

The core algorithm performs backward traversal starting from turn \(n-1\) for the \(n\)-th turn. For each historical turn \(k\), a dropout probability \(p_k\) is sampled from \(Beta(\alpha, \beta)\), and a retention flag \(m_k\) is sampled from \(Bernoulli(1-p_k)\). The backward traversal stops at the first \(m_k=1\) encountered, setting turn \(k\) as the parent of turn \(n\). If no historical turn is retained after traversing all predecessors (all \(m_k=0\)), the root node becomes the parent. The attention mask and position IDs are adjusted according to this tree structure—skipped turns no longer appear in the causal field of subsequent turns, but their own loss tokens still compute gradients in place.

The paper contrasts this stochastic process with the Chinese Restaurant Process (CRP): CRP clusters unordered elements via preferential attachment, whereas StochasT's tree construction is strictly causal and chronologically driven—more recent turns are more likely to become parent nodes, ensuring that chronological continuity is preserved. This design is better suited for sequence modeling than CRP.

2. Flexible Contextual Preference Control via Beta Distribution

The dropout probability is generated by a \(Beta(\alpha, \beta)\) distribution, where two hyperparameters control the preference for "short history" versus "long history". A symmetric setup \((2,2)\) is adopted by default, producing a distribution with a moderate preference—retaining some history without fixing the length. A setting of \((5,1)\) prefers short histories, approaching single-turn training, whereas \((1,5)\) prefers long histories, approaching standard multi-turn training. Experiments show that \((2,2)\) achieves the optimal balance on CRA and CRA+, and is relatively robust to hyperparameter choices. The beauty of this design is that it transforms "context depth" from a discrete hyperparameter into a continuous random variable, allowing the model to adapt to various depths during training without manually specifying truncation lengths.

3. Balanced Latin Square Evaluation Framework: Systematically Quantifying Contextual Robustness

Traditional evaluations only test under a single fixed setting (either single-turn or multi-turn), failing to expose the model's sensitivity to contextual changes. This paper proposes an evaluation paradigm based on the Balanced Latin Square (BLS): constructing an \(N \times N\) matrix where each \(N\)-turn dialogue requires \(N\) inferences. Each row of the matrix represents a permutation. Each permutation satisfies two conditions: every question appears exactly once at every position, and every pair of questions appears adjacent to each other exactly once. This evaluation covers single-turn (when appearing in the first position) and all context lengths simultaneously, while eliminating first-order carryover effects. Two new metrics are defined based on BLS: (1) Contextual Robustness Accuracy (CRA), which computes the average accuracy of a question across all \(N\) contexts; (2) Strict Contextual Robustness Accuracy (CRA+), which requires the question to be answered correctly in all \(N\) contexts to score 1—reflecting the model's true, context-independent mastery of visual knowledge. Odd-turn dialogues are padded with a dummy prompt to make \(N\) even; this dummy prompt is excluded from final scores.

Loss & Training¶

StochasT does not alter the standard autoregressive cross-entropy loss function, only modifying the attention mask and position IDs between tokens in each forward pass. Therefore, it can be seamlessly integrated into any VIT training pipeline, requiring zero data augmentation, no optimizer changes, and zero extra training overhead. Practical experiments employ LoRA parameter-efficient fine-tuning with a global batch size of 128, a warmup ratio of 0.03, and early stopping.

Key Experimental Results¶

Main Results¶

Model	Evaluation Setup	Training Strategy	4-Dataset Average	MMDU Score
LLaVA-1.5-7B	SingleT	MultiT (Original)	61.51	-
	SingleT	SingleT	68.46	-
	SingleT	StochasT	67.16	-
	MultiT	MultiT (Original)	68.52	12.68
	MultiT	SingleT	68.60	-
	MultiT	StochasT	70.63	13.10
Qwen2.5-VL-3B	SingleT	MultiT (Original)	71.20	-
	SingleT	SingleT	76.71	-
	SingleT	StochasT	77.28	-
	MultiT	MultiT (Original)	77.39	57.90
	MultiT	SingleT	77.75	-
	MultiT	StochasT	80.16	59.80

BLS Robustness Evaluation¶

Model	Training Strategy	CRA	CRA+
LLaVA-1.5-7B	Original (No FT)	45.73	27.66
	+MultiT	61.82	41.19
	+SingleT	67.33	53.15
	+StochasT	67.69	50.89
Qwen2.5-VL-3B	Original (No FT)	57.26	38.46
	+MultiT	73.63	54.30
	+SingleT	77.26	64.57
	+StochasT	77.53	61.76

Key Findings¶

Narrowing the Single-Turn vs. Multi-Turn Gap: StochasT narrows the gap between single-turn and multi-turn evaluations from 7.01% (LLaVA-1.5) and 6.19% (Qwen2.5-VL) under original MultiT to 3.47% and 3.33% respectively, approaching the baseline robustness of pretrained models.
Robustness of Beta Parameters: The differences in CRA across four configurations—\((0.5,0.5)\), \((2,2)\), \((1,5)\), and \((5,1)\)—do not exceed 1.34%, proving the method is robust to hyperparameters; the symmetric \((2,2)\) setting is optimal for CRA+.
vs. Random Cutoff: A random cutoff baseline with equivalent expected depth performs worse than MultiT (CRA 70.61 vs. 73.65), indicating that StochasT's improvements do not stem from merely "shortening context," but from the dynamically diverse context structures.
Training Efficiency: SingleT requires approximately \(2\times\) the training tokens to match the single-turn performance of StochasT, whereas StochasT introduces no additional tokens.
MME Category-wide Gains: Applying StochasT during the LLaVA-150K general fine-tuning stage yields a 19.36% improvement in perception and a 25.56% improvement in reasoning.
Model Scale Scaling: On Qwen3VL-32B, StochasT maintains an advantage of approximately 3 CRA points and 3.8 CRA+ points.

Highlights & Insights¶

The elegance of not introducing new loss functions: Merely changing how training data is organized within the attention mechanism achieves a balance between single-turn and multi-turn capabilities. This innovation in data arrangement is completely orthogonal to and compatible with training objective modifications like L2T, Vittle, and Ross.
Superb analogy from "layer depth" to "turn depth": Transferring Stochastic Depth from ResNet's layer level to the turn level of multi-turn dialogues represents a high-level cross-domain conceptual leap. In both contexts, "depth" behaves as a continuous hyperparameter, and randomization yields implicit ensembling effects.
Independent contribution of the BLS evaluation framework: It exposes a significant blind spot in LVLM benchmarking—the performance fluctuations between single-turn and multi-turn settings have never been systematically quantified. As an extremely stringent metric, CRA+ accurately measures whether a model possesses genuine visual knowledge rather than relying on contextual shortcuts.
Explicitly quantified token efficiency: While many papers claim to be "data-efficient" without direct evidence, this paper presents diminishing return curves comparing training tokens to performance, quantifying StochasT's approximately \(2\times\) token advantage over SingleT.

Limitations & Future Work¶

The paper specifies that StochasT is suited for dialogues with weak contextual dependencies—which comprises most VIT datasets. However, in strongly dependent scenarios (such as step-by-step reasoning or story comprehension), random pruning may disrupt semantic coherence.
StochasT slightly underperforms SingleT on the CRA+ metric, indicating that while single-turn training sacrifices substantial multi-turn capability, it still maintains an edge in extreme consistency. How to maintain contextual robustness without sacrificing consistency remains an open question.
The method was validated largely under LoRA fine-tuning (while the LLaVA-150K stage is full fine-tuning, it ran for only one epoch). Its behavior under full-parameter large-scale VIT warrants further investigation.
Although the Beta distribution is relatively robust, the optimal \((\alpha,\beta)\) may vary across datasets, and a strategy for automatic selection is currently lacking.

vs. L2T: L2T appends auxiliary losses on instruction tokens to penalize shortcut learning; StochasT does not alter the loss but shapes the context distribution through data structure, making them orthogonal and complementary.
vs. Stochastic Depth: Stochastic Depth randomly drops ResNet blocks to train a layer-ensemble network; StochasT masterfully maps "depth" from network layers to dialogue turn depths.
vs. Turn Dropout: A simple baseline that randomly drops entire turns (and their losses). StochasT retains all loss tokens, exhibiting higher data utilization and slightly superior performance.
vs. Existing VIT Data Strategies: Existing works (DRESS, Vittle, Ross) modify either the loss or the data quality; StochasT is the first to systematically target the multiT/singleT mismatch via data organization.

Rating¶

Novelty: ⭐⭐⭐⭐ [The first to systematically reveal the structural mismatch between multi-turn training and single-turn evaluation, offering an elegant data-level solution.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 5 cross-domain datasets, 2 model architectures, complete ablations (parameters, depth, attention) + the BLS evaluation framework + scaling to general fine-tuning stages and larger models.]
Writing Quality: ⭐⭐⭐⭐ [Clear and solid motivation, precise methodology analysis, and deep theoretical insight in comparison with CRP.]
Value: ⭐⭐⭐⭐⭐ [Targets a structural blind spot in the LVLM training pipeline. The proposed scheme requires zero extra computational overhead, is plug-and-play, easy to integrate, and the BLS evaluation framework stands as an independent methodological contribution.]