VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization¶

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: vision backbone, multi-task learning, post-training, depth estimation, referring segmentation

TL;DR¶

VersaViT jointly post-trains one MLLM vision backbone with language, geometry, and pixel-localization supervision, raising Qwen2-VL-ViT's frozen ADE20k linear-probing performance from 33.6 to 49.6 mIoU while improving the OpenCompass average from 64.8 to 66.4 under a shared evaluation configuration.

Background & Motivation¶

Multimodal large language models (MLLMs) typically combine a vision encoder, a vision-language projection layer, and a large language model (LLM). The vision encoder converts images into features that the language model uses to answer questions or produce descriptions, so its training quality is often represented by the entire system's question-answering performance. However, recognizing that an image contains a car does not ensure that its features preserve wheel boundaries, occlusion relationships, or relative object distances. Those local structures are essential for classic vision tasks such as semantic segmentation and monocular depth estimation. CLIP-style pretraining and text-centered instruction tuning provide strong semantic signals without necessarily enforcing spatial precision at every pixel. The paper therefore asks not only whether an MLLM answers well, but whether its vision backbone can serve as a general vision foundation model.

Frozen-backbone linear probing of the Qwen-VL family reveals that question-answering ability does not automatically translate into strong dense representations. A more discriminating result comes from the training-objective ablation: additional VQA and captioning training raises OpenCompass from 64.8 to 66.1, yet lowers ADE20k from 33.6 to 32.0 and increases NYUv2 RMSE from 0.541 to 0.557. This does not establish that architectural capacity can never be a bottleneck, but it demonstrates that continuing language supervision can exacerbate representational bias. Simply adding a powerful segmentation decoder could improve downstream scores without improving the backbone itself. The paper addresses this attribution problem by using specialized task heads to provide training gradients, then freezing the backbone to test whether its features transfer.

VersaViT retains the existing vision Transformer instead of attaching another complete vision encoder. Captioning and VQA preserve semantic expression, depth estimation encourages spatial geometry, and referring segmentation connects textual semantics to local pixels. The three supervision sources need not annotate the same image; independent data streams jointly update the shared backbone. Core Idea: use task losses at different granularities to shape one visual representation, recovering geometric and pixel-level detail insufficiently constrained by language-centered training instead of compensating with another vision backbone.

Method¶

Overall Architecture¶

Inputs are images from separate VQA/captioning, depth, and referring-segmentation streams, together with the text or supervision required by each task. The shared backbone is initialized from Qwen2-VL-7B's vision encoder; the language model used while constructing the method is Qwen3-1.7B. The backbone produces per-layer visual features: language and segmentation primarily use the final layer, while depth draws on three feature sets distributed across layers. Training starts with Projection Alignment Warm-Up, followed by Three-Granularity Task Supervision coupled through Alternating Accumulated Updates. The result is a vision backbone that can connect to different downstream heads, not a fixed system required to execute all three tasks at every inference call.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image-text pairs + initial backbone"] --> Warmup["Projection Alignment<br/>Warm-Up"]
    Warmup --> Supervision["Three-Granularity Task Supervision<br/>Language / depth / referring segmentation"]
    Streams["Independent data streams<br/>Text / pseudo-depth / masks"] --> Supervision
    Supervision --> Update["Alternating Accumulated<br/>Updates"]
    Update --> Backbone["Updated shared backbone"]
    Backbone --> Deploy["Inference: attach only the required head"]

Supervision and updates in the diagram occur during training; the models producing pseudo-depth and segmentation data are not mandatory inference components. One important distinction is that the main VQA evaluation separately uses Qwen3-8B and keeps the compared vision backbones frozen throughout adaptation. This holds the language-model configuration constant across backbones, but does not imply that the 1.7B language model used for post-training directly reproduces Table 2.

Key Designs¶

1. Projection Alignment Warm-Up: establish the visual interface to the language model

If a new vision-language interface is not aligned, immediately changing both the backbone and the language model can entangle interface adaptation with representation learning. The authors first freeze the vision backbone and Qwen3-1.7B, training only the intervening MLP vision projector. This projector converts final-layer visual features into image embeddings usable by the language model and concatenates them with text embeddings. Training uses autoregressive cross-entropy to predict the next token from the image and preceding text tokens. This is interface warm-up, not a stage claimed to directly improve the frozen backbone's spatial features.

Warm-up data combines natural image-text pairs with OCR, preventing the interface from adapting only to object descriptions in natural photographs. Table 1 lists 0.6M Pixmo-cap, 4.7M IDL-WDS, and 4.1M SA-1B-InternVL examples. After warm-up, the language branch continues to provide semantic constraints rather than being discarded when pixel-level tasks begin. Subsequent geometric learning thus supplements the established vision-language interface instead of replacing the original question-answering objective.

2. Three-Granularity Task Supervision: constrain the same features with semantics, geometry, and localization

The VQA and captioning branch passes final-layer visual features through the MLP projector into Qwen3-1.7B, retaining conventional generative language training. Its purpose is not merely to add another reported metric, but to prevent the backbone from abandoning language-readable semantics in pursuit of depth or contours. The depth branch uniformly samples three feature sets from the backbone's layer outputs and feeds them into a DPT head to predict a full-image depth map. Cross-layer features allow the decoder to use different abstraction levels rather than relying exclusively on final semantic representations. This is an auxiliary training path; frozen linear probing uses final-layer patch features and must not be confused with direct prediction by this DPT head. Depth targets are pseudo-labels generated by Depth Anything V2 rather than manually collected pixelwise depth ground truth. The depth loss combines scale- and shift-invariant supervision with multiscale gradient matching: the former handles the uncertain scale of relative depth, while the latter constrains local changes and edges. The cached gradient formula has layout corruption, and supplementary material is unavailable; this note retains the established mechanism without reconstructing an exact implementation.

The referring-segmentation branch requires an image, a target description, and its mask, teaching the backbone which pixels correspond to the described object. Qwen3-Embedding-8B extracts description embeddings offline, and a prompt encoder converts them into prompt features. A mask decoder combines final-layer visual features, prompt features, and an output token to predict the target mask, without asking a generative LLM to express every pixel as text. Text embedding extraction is offline; this 8B embedding model must not be mistaken for a language decoder jointly optimized in every training batch. Existing referring-segmentation datasets can be used directly; mask-only datasets receive descriptions from Describe Anything. For datasets containing only boxes and categories, SAM first generates masks from boxes, after which generated descriptions or existing class names supply the text. The segmentation loss combines pixelwise binary cross-entropy and DICE, supervising both local classification and overall region overlap. These heads accommodate different output formats, but absorbing task conflicts is the authors' interpretation, not evidence that the method includes explicit gradient-conflict removal.

3. Alternating Accumulated Updates: combine three gradient sources without requiring triple annotations

Actual training samples come from independent task streams, rather than every image having VQA, depth, and referring-mask annotations. The procedure takes one batch per task in turn, runs the shared backbone and corresponding head, computes that task's loss, and accumulates gradients. It executes one collective parameter update after processing all three tasks, rather than immediately letting each task supersede the preceding task's update. The vision backbone, corresponding task heads, vision projector, and generative LLM are unfrozen during this stage. The losses meet through shared parameters, while the specialized heads retain prediction formats appropriate to each task. Task weights control the supervision balance in the joint objective; they are not routing scores that dynamically select experts for an input.

The paper's joint objective is:

\[ \mathcal{L}_{\mathrm{all}}=\lambda_{\mathrm{cap}}\mathcal{L}_{\mathrm{cap}}+\lambda_{\mathrm{depth}}\mathcal{L}_{\mathrm{depth}}+\lambda_{\mathrm{seg}}\mathcal{L}_{\mathrm{seg}}. \]

The language term is the autoregressive VQA/captioning loss, the depth term combines scale/shift-invariant and gradient losses, and the segmentation term combines BCE and DICE. Table 5 lists 0.33 for each of the three weights in the balanced configuration; no elaborate online weight optimizer is introduced. Increasing the depth weight can improve depth without producing the best VQA or segmentation result, so a general backbone still requires choices among objectives. At downstream inference, only the required path is retained: projector and LLM for VQA, a depth head for depth, or a text-conditioned mask head for referring segmentation. A deployed segmentation service must still supply a representation for each new description; offline embeddings do not mean that text conditioning disappears.

A Worked Example¶

Consider successive training inputs consisting of a street image requiring a description, an indoor image with pseudo-depth, and an image annotated with the target phrase "the car on the left." This illustrates the data flow, not an additional quantitative experiment, and the three images need not depict the same content. The street-image batch uses final-layer visual features and the language model to compute captioning loss, preserving object and scene semantics. The indoor batch combines three feature levels with DPT and matches Depth Anything V2 pseudo-labels, emphasizing spatial structure. The vehicle batch supplies the description embedding and image features to the mask head, using the target mask to constrain object boundaries and language localization. The three batches accumulate gradients before a joint update; the model does not first generate depth, feed depth into segmentation, and then answer questions from the segmentation result. Afterward, the auxiliary training paths can be removed and the backbone frozen while a linear segmentation head is trained, testing whether spatial information entered the visual representation itself.

Loss & Training¶

Warm-up uses batch size 1024, learning rate \(10^{-3}\), and 1 epoch, updating only the vision projector. Joint training also lasts 1 epoch; learning rates are \(10^{-5}\) for the vision encoder, vision projector, and text decoder, and \(10^{-4}\) for other trainable parameters. The model supports dynamic resolutions, which does not conflict with the fixed \(560\times560\) segmentation linear-probing resolution in Table 3. Joint VQA/captioning training uses the 14.0M FineVision data listed in Table 1, while depth and segmentation have separate multisource streams. The supplied main text does not fully specify all internal loss coefficients, exact depth feature-layer indices, or supplementary evaluation recipes; these must not be filled in from customary practice. VQA evaluation separately adapts a Qwen3-8B interface: only the projector is trained first, then the projector and LLM, with the vision backbone frozen in both stages. The main results therefore support transferable visual representations, not the claim that every task works directly without downstream adaptation.

Key Experimental Results¶

Main Results¶

The following selection comes from Table 2 on page 9 and Table 3 on page 10, using Qwen2-VL-ViT as the baseline throughout. OpenCompass is the eight-benchmark average after adaptation with the same Qwen3-8B recipe; dense tasks use frozen-backbone linear probing, with \(560\times560\) segmentation inputs. Segmentation uses mIoU and depth uses RMSE; changes are absolute differences, not relative percentage improvements.

Evaluation	Metric and direction	Baseline	VersaViT	Absolute change
OpenCompass	Eight-benchmark average, higher is better	64.8	66.4	+1.6
ADE20k	mIoU, higher is better	33.6	49.6	+16.0
Cityscapes	mIoU, higher is better	57.6	74.5	+16.9
Pascal VOC	mIoU, higher is better	67.5	86.6	+19.1
NYUv2	RMSE, lower is better	0.541	0.473	-0.068
KITTI	RMSE, lower is better	3.735	3.136	-0.599

Not every VQA sub-benchmark improves: Table 2 reports MMB decreasing from 78.2 to 78.0 and AI2D from 79.4 to 79.1, while MMVet increases from 58.7 to 62.6. In Table 3, DINOv3-7B/16 reaches 55.9 on ADE20k and 0.309 NYUv2 RMSE, both better than VersaViT; its VOC score of 86.6 matches VersaViT. Model scales and pretraining differ, so this is not a strictly compute-matched ranking.

Ablation Study¶

Table 4 on page 11 incrementally adds supervision, reporting the OpenCompass average, NYUv2 RMSE, and ADE20k mIoU. It directly tests whether additional language training suffices, rather than merely showing that the full model improves over initialization.

Post-training tasks	OpenCompass, higher is better	NYUv2, lower is better	ADE20k, higher is better
No additional post-training	64.8	0.541	33.6
VQA and captioning only	66.1	0.557	32.0
VQA and captioning + depth	66.3	0.495	35.1
VQA and captioning + depth + referring segmentation	66.4	0.473	49.6

Adding referring segmentation on top of language and depth supervision raises ADE20k by 14.5 points and further lowers NYUv2 RMSE by 0.022, indicating benefits beyond the newly introduced task. However, this is not a complete factorial ablation: depth-only, segmentation-only, and language-plus-segmentation configurations are missing, preventing unique attribution of each task's independent contribution. In Table 5, setting the depth weight to 0.50 and the other weights to 0.25 yields 0.455 on NYUv2 but 65.7 on OpenCompass; balanced weights are a compromise, not optimal in every column.

Key Findings¶

Table 7 on page 12 reports 66.2 / 0.547 / 34.7 for the 35M single-task data baseline and 66.4 / 0.473 / 49.6 for the full model, ordered as OpenCompass / NYUv2 / ADE20k. This weakens a data-volume-only explanation without fully matching compute or annotation sources.
Table 8 on page 12 reports NAVI Recall increasing from 39.27 to 41.64 and SPair Recall from 17.05 to 26.99. The exact gains are 2.37 and 9.94; the main text's 2.3 and 10 are coarser descriptions.
Tables 9–10 on page 13 require attention to training conditions: DA-2K depth accuracy is 91.3% without task-specific fine-tuning and 94.2% afterward; RefCOCO results follow further fine-tuning on that dataset family and are not zero-shot segmentation.

Highlights & Insights¶

The strongest contribution separates question-answering quality from visual-feature quality. Frozen linear probing is more informative about retained dense information than simply attaching a stronger decoder.
Task heads inject constraints into shared representations without forcing every output into text generation. Depth and masks can therefore use spatial losses suited to their respective outputs.
Independent streams reduce the need for fully annotated samples. This principle can help supplement missing capabilities, but pseudo-label quality and data bias still require inspection.

Limitations & Future Work¶

The authors emphasize low cost and lightweight heads, but the supplied main text does not report complete GPU time, memory usage, or pseudo-label production cost. Qwen3-1.7B training and preprocessing with the two large models for depth labels and text embeddings also matter.
Task heads absorbing conflicts is an explanatory claim without direct gradient-conflict measurements in the main text. Comparing gradient similarity and parameter-update directions would strengthen the analysis; this is a reader suggestion, not a reported result.
The depth teacher, SAM, and Describe Anything introduce their own biases, and Table 7 does not fully isolate the contribution of additional teacher knowledge. The method should not be described as independent of teacher supervision.
The authors propose introducing the approach earlier in MLLM pretraining, but current evidence primarily concerns post-training. Stability and gains from large-scale early joint training remain to be established.
The cache contains the main paper and references, but no supplementary material; the depth formula layout, some implementation details, and the full VQA adaptation recipe cannot be completely verified from this material.

Qwen2-VL / Qwen2.5-VL, references [66] / [1]: provide the initial visual backbones and language-centered training context. VersaViT changes the granularity of representation supervision rather than introducing another VQA architecture.
DINOv3, reference [54]: supplies a strong dense-representation and frozen-probing reference. VersaViT closes some gaps but does not comprehensively outperform it on ADE20k or depth.
DUNE / UNIC, references [50] / [51]: primarily distill representations from frozen vision teachers; VersaViT shapes its backbone through task-output supervision, although its depth pseudo-labels still come from a teacher, so the approaches are not mutually exclusive.
Depth Anything V2 / SAM, references [78] / [29]: support depth pseudo-labeling and box-to-mask construction respectively. They are supervision sources, not extra vision backbones chained into downstream inference.

Rating¶

Novelty: 3/5. The contribution is a clear diagnosis and a multi-granularity post-training combination, rather than a new architecture or optimizer.
Experimental Thoroughness: 4/5. Evaluation covers VQA, dense probing, and transfer, but task combinations and cost attribution remain incomplete.
Writing Quality: 4/5. The narrative and ablations are clear, although some prose generalizes beyond the tables and implementation details depend on supplementary material.
Value: 4/5. A transferable training approach for visual backbones that need both language semantics and pixel precision.