Steerable Visual Representations

Conference: ECCV 2026
arXiv: 2604.02327
Code: https://jonaruthardt.github.io/project/SteerViT (Project Page)
Area: Multimodal Visual Representations / Referring Segmentation
Keywords: Steerable Visual Representations, Early Fusion, Gated Cross-Attention, Frozen ViT, Referring Segmentation

TL;DR

SteerViT interleaves lightweight gated cross-attention layers inside a frozen ViT (such as DINOv2), enabling language to intervene during the visual encoding process. Trained via a patch-level referring segmentation proxy task, it adds only 21M parameters to allow natural language to "steer" visual features and focus on arbitrary objects. Meanwhile, it maintains the representation quality of the original ViT, achieving zero-shot parity with or outperforming specialized methods on text-conditioned retrieval, personalized object discrimination, and industrial anomaly segmentation tasks.

Background & Motivation

Pre-trained ViTs like DINOv2, MAE, and SigLIP already generate generic image features that are highly effective for retrieval, classification, and segmentation. However, they suffer from a persistent limitation: they only focus on the most salient subject in the image. In an indoor photo containing a cat, a remote control, and a bookshelf, DINOv2 encodes almost exclusively a representation of the "cat", while the remote and bookshelf are entirely ignored. This largely stems from the "photographer's bias" and the saliency preferences learned from object-centric visual datasets. The challenge is that many downstream tasks (e.g., fine-grained localization, on-demand retrieval of small objects) require focusing on non-obvious concepts, yet purely visual encoders are query-agnostic—we have no means to instruct them to "look at the bookshelf."

An alternative path is Multimodal Large Language Models (MLLMs), which can indeed be guided by text prompts. However, the fusion occurs in the early layers of the language model, resulting in features that fall into the language space and dilute visual fidelity. Consequently, they perform worse than purely visual encoders on general visual tasks and incur high computational costs with billions of parameters. Cross-modal encoders like CLIP and SigLIP represent typical "late fusion"—image and text are encoded independently, where text only provides a supervisional signal during training but cannot alter the visual encoding at inference time. Posteriori adding text features to visual features yields virtually no change in retrieval accuracy (experimentally, only +0.02%). Although open-vocabulary localization models (SAM2/3, GroundingDINO) are steerable by text, their intermediate representations are specifically optimized for localization and lack the generalization required for generic visual tasks. The key challenge is that existing solutions are either steerable but have poor representation quality, or have high-quality representations but lack steerability. No single approach satisfies both needs.

This paper draws inspiration from human vision: when prompted with words, top-down task-driven attention alters the way humans parse scenes. The authors thus invert the MLLM paradigm: instead of feeding vision into language, they use language to modulate vision. Core Idea: Interleave lightweight, trainable gated cross-attention layers into the blocks of a frozen ViT, allowing visual patch tokens to "attend" to text tokens during encoding (early fusion). By training these added layers with a patch-level referring segmentation proxy task, they obtain a class of vision-centric multimodal representations that are steerable by text while fully preserving the representation quality of the original ViT.

Method

Overall Architecture

SteerViT aims to equip any pre-trained ViT with "text-steerable" capabilities without breaking its original representations. The backbone ViT and text encoder remain completely frozen throughout. To enable steering, a gated cross-attention layer is inserted every other ViT block, allowing visual tokens to query text tokens. During training, a patch-level referring segmentation task forces these cross-attention layers to route textual cues into the corresponding visual patches. At inference, no task-specific training is required; changing the text prompt steers the features to new concepts or domains.

Specifically, the model consists of four components: A. Visual Encoder (frozen ViT; DINOv2 ViT-B/14 is used in main experiments to output \(N\) patch tokens and an optional [CLS]); B. Text Encoder (frozen RoBERTa-Large, outputting token-level text embeddings); C. Multimodal Adapter (a trainable two-layer MLP projecting L2-normalized text embeddings into the visual alignment space); D. Gated Cross-Attention Layer (the newly inserted module that bears the actual steering capability, placed at every other ViT block, e.g., 6 layers inserted into a 12-layer ViT-B). Among these, A and B are frozen scaffold components, C is a generic step to align text and vision, and D along with the training objective represents the main innovation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image + Text Prompt"] --> B["Frozen ViT / Frozen RoBERTa<br/>Independent Encoding"]
    B --> C["Text Alignment<br/>2-Layer MLP Projection to Visual Space"]
    C --> D["Early Fusion Gated Cross-Attention<br/>Visual Tokens Query Text, tanh Gating into Residual Stream"]
    D --> E["Referring Segmentation Proxy Task<br/>Patch-level Soft Cross-Entropy Training of CA Layers"]
    E -->|Inference-time Prompt Swap| F["Steerable Visual Features<br/>Retrieval / Classification / Segmentation / Anomaly Detection"]

Key Designs

1. Early Fusion Gated Cross-Attention: Allowing Text to Rewrite the Visual Residual Stream During Encoding

This design directly addresses the pain point where "late fusion cannot steer frozen visual features." The authors invert the gated cross-attention direction of Flamingo (where language attends to vision) by letting the visual hidden states attend to language tokens: at layer \(\ell\), visual patch tokens act as queries (\(Q\)), and the adapter-aligned text tokens act as keys/values (\(K/V\)) in standard cross-attention. The critical ingenuity lies in how the attention output is incorporated into the residual stream—it is not added directly but is modulated by a layer-specific learnable scalar \(\alpha_\ell\) through a tanh gate, initialized to zero:

\[Z_v^{(\ell+1)} = Z_v^{(\ell)} + \tanh(\alpha_\ell)\cdot \hat{Z}_v^{(\ell)}\]

Since \(\tanh(0)=0\), the model starts exactly equivalent to the original frozen ViT, preventing pre-trained features from being disrupted at the start. Yet, it is highly plastic: the gradient of the gate with respect to \(\alpha_\ell\) contains \(\mathrm{sech}^2(\alpha_\ell)\), and since \(\mathrm{sech}^2(0)=1\), the gradient signal remains active even under zero initialization. Consequently, \(\alpha_\ell\) can gradually depart from zero during optimization to open the text injection route. This preserves the original ViT's representation quality while allowing text to progressively rewrite the visual representations layer by layer. Removing this tanh gate (using ungated cross-attention) causes FG-CLS, CORE, and PODS to drop by 4.2, 1.4, and 11.0 points respectively, demonstrating that ungated cross-attention directly corrupts frozen features. Additionally, the authors intentionally omit the gated FFN after the cross-attention block (which is present in Flamingo). Adding an FFN barely improves representation quality while hurting steerability and OOD transfer, while also inflating adapter parameters from 21.2M to 35.4M (+67%).

2. Patch-level Referring Segmentation Proxy Task: Forcing Cross-Attention to Genuinely Absorb Language via "Content Matching"

Merely inserting cross-attention layers is insufficient; a task that "cannot be solved without looking at the text" is required to force the visual encoder to use textual cues. The authors adopt referring segmentation: given an image and a prompt pointing to a target object, the model predicts which patches belong to the referred region. To bypass the complexity of pixel-level decoders, this task is performed directly on the \(n\times n\) patch grid of the ViT. The ground truth is the proportion of foreground pixels in each patch (\(y_i\)), derived from pixel-level binary masks. A linear classification head maps each patch representation to a mask probability \(p_i\), which is trained via soft cross-entropy:

\[\mathcal{L} = -\sum_{i=1}^{n\times n} y_i \log p_i\]

This soft label design is essential. The authors compared this to a simpler "pointing" baseline (where a Gaussian kernel centered on the bounding box is used as the target) and found that pointing only teaches the model "where the object is" while being insensitive to object shape and size, collapsing supervision into a single spatial location. Conversely, the segmentation objective activates all tokens overlapping with the object, teaching "content matching"—each patch must determine whether its content matches the described object rather than just its proximity to the center. Experimentally, the segmentation objective completely outperforms pointing (FG-CLS +7.3, ADE20k +8.0, PODS +12.4). It is this "content matching" supervision that forces cross-attention layers to route textual information precisely to corresponding visual patches, yielding steerable representations.

3. Gated Scalar as a Continuous Inference-Time Control Knob: Smooth Interpolation Between Original ViT and Fully Conditioned States

Because steering is governed by the tanh gate, the authors discovered a valuable emergent property: the gating scalar \(\alpha_\ell\) naturally serves as a continuous knob for text-conditioning strength. During inference, scaling the learned \(\alpha_\ell\) by a factor \(\omega\in[0,1]\) allows a smooth interpolation between the "intact ViT subspace" and the "fully text-conditioned state" (\(\omega=0\) reverts to the original ViT, while \(\omega=1\) offers full steering). Plotting this trajectory reveals a clear Pareto frontier: both DINOv2 and SigLIP achieve the optimal tradeoff between steerability and representation quality around \(\omega=0.6\), where representation quality slightly exceeds the original ViT while unlocking high steerability. The most striking result is found in MAE: its representation quality rises monotonically with \(\omega\), increasing from 40 points at \(\omega=0\) to over 50 points at \(\omega=0.6\). This suggests that text conditioning introduces semantic structures into MAE's under-developed features, making them more transferable. The knob converts "steerability vs. representation quality" from a binary choice into an adjustable continuum.

Loss & Training

The training objective is the patch-level soft cross-entropy referring segmentation loss detailed above. The training dataset consists of a mixture of referring segmentation and grounding datasets: RefCOCO/+/g, Visual Genome, LVIS, and Mapillary Vistas, spanning 162k unique images and 2.28M image-text pairs. The dataset covers indoor, outdoor, and street view domains, and text prompts range from two-word labels to multi-sentence descriptions. Bounding boxes are converted to binary masks using SAM2. The backbone is DINOv2 ViT-B/14 + RoBERTa-Large, trained at 336² resolution with a batch size of 12 for 500k steps (approx. 84 H100 GPU hours) using AdamW and a cosine scheduler (warmup to 3e-4, then decaying to 3e-5). Throughout this process, only the 21.2M cross-attention parameters are optimized, while the ViT and text encoder remain completely frozen.

Key Experimental Results

Main Results

The model is evaluated against four classes of baselines: purely visual encoders (DINOv2, MAE), cross-modal encoders (CLIP, SigLIP with late-fusion element-wise addition), MLLMs (InternVL3, Qwen3-VL, LFM-2.5-VL using the last-token pooling of E5-V), and open-vocabulary localization models (SAM3, GroundingDINO using intermediate multimodal states).

Task / Benchmark	Metric	SteerViT	DINOv2	Compared Baselines
CORE Conditioned Retrieval	acc@1	96.0	43.7	FLAIR: 81.3 / SAM3: ~96 / InternVL3-2B: 20 points lower
MOSAIC Directed Attention	PR-AUC	50.2	14.3	—
GeneCIS Focus Object (Zero-shot Real)	R@1	25.4	9.6	Specialized Baseline: 18.7
PODS Personalized Object Discrim.	PR-AUC	58.1	29.6	Fine-tuned DINOv2 Variant: 48.0 (requires 1 model/class)
MVTec AD Zero-shot Anomaly Seg.	PRO	82.1	—	SAM3: 54.5 / FADE (specialized): 84.5
VisA Zero-shot Anomaly Seg.	PRO	82.0	—	FADE (specialized): 79.3

Key Observations: On CORE, early fusion boosts DINOv2 from 43.7 to 96.0, whereas late-fusion posteriori text addition yields a negligible improvement (+0.02%), confirming that "late fusion cannot steer frozen features." In the extreme OOD scenario of anomaly segmentation, SteerViT matches specialized methods in a zero-shot manner. On personalized discrimination, a single SteerViT model (by varying the prompt) outperforms fine-tuned DINOv2 models trained individually for each of the 100 objects.

Ablation Study

Configuration	FG-CLS↑	ADE↑	CORE↑	PODS↑	Description
Full (Early Fusion + tanh Gating + MLP)	87.7	55.4	96.0	58.1	Complete model
w/o Early Fusion (Late Fusion instead)	91.8	55.5	93.3	36.6	Classification improves, but PODS plummets by 21.5
w/o tanh Gating (Ungated CA)	83.5	55.3	94.6	47.1	Ungated setup disturbs frozen features
w/o MLP (Single Linear Layer instead)	86.7	54.5	95.2	56.4	Text-vision alignment deteriorates
Training Objective: pointing (vs segmentation)	80.4	47.4	95.2	45.7	Only teaches "where", not "what"

Key Findings

• Early fusion is critical for fine-grained tasks: While late fusion yields slightly higher accuracy on coarse, super-category classification, it collapses on tasks requiring instance-level discrimination like PODS (36.6 vs 58.1). When using coarse class prompts, the difference disappears (26.5 vs 27.9), showing that early fusion shines precisely at fine-grained levels.
• Text details directly control feature granularity: On PODS, as prompts progress from super-categories ("mug") to instance names ("white ECCV mug") to detailed appearance descriptions generated by MLLMs, the PR-AUC increases steadily from 27.9% to 58.1%. Rather than just appending information, SteerViT style prompt specificity directly steers the semantic granularity of the visual features.
• Steering is genuinely text-driven: When conditioned on a randomly permuted class prompt, SteerViT's performance drops by 48.3 points (and FLAIR drops by 29.4 points), whereas CLIP/SigLIP remain unchanged. This proves that SteerViT's features are actively reshaped by the text, whereas CLIP/SigLIP only rely on unconditioned visual features.
• Weaker backbones benefit more: The CORE gain of early fusion over late fusion is +33.9 for MAE and +15.9 for SigLIP, which is much larger than DINOv2's +2.7. This indicates that the less semantically mature the underlying visual representations are, the more they benefit from language injection.
• Emergent multi-level / compositional steering: Conditioning on "animal" groups animal categories into a macro-cluster while preserving fine details. Conditioning on "eye" reorganizes space based on the compositional attribute "having eyes" (clustering humans and animals together), which was never explicitly supervised during training.

Highlights & Insights

• Inverting the MLLM paradigm is the most refreshing aspect of this work. Instead of feeding vision into language, the authors modulate vision with language. By adding only 21M parameters (two orders of magnitude fewer than MLLMs), they obtain vision-centric steerable representations, achieving both efficiency and high quality.
• Zero-initialized tanh gating is an elegant engineering design. It guarantees that the initial model is equivalent to the original frozen ViT (preventing damage to pre-trained representations). Because \(\mathrm{sech}^2(0)=1\), it preserves active gradients for optimization, while seamlessly serving as a continuous inference-time knob. This "zero-initialized gated residual connection" is widely applicable for safely introducing new modules into frozen backbones.
• The comparison of pointing vs. segmentation offers a valuable insight: the density of the proxy task's supervision determines what is learned. Sparse point supervision only teaches localization, whereas dense mask supervision teaches "content matching", yielding superior representations. This is highly instructive for designing proxy tasks in representation learning.
• Prompt detail as a granularity knob: A single frozen model, without retraining, can shift its semantic granularity from coarse categories to instance-level specifics merely by modifying the prompt. This offers excellent utility in personalized and long-tail scenarios, where one general model can replace an entire suite of specialized fine-tuned models.

Limitations & Future Work

• Under zero-shot anomaly segmentation settings, certain structural anomalies (e.g., flipped metal nuts) are not predicted accurately. Because the model is not trained on normal samples and lacks visible surface defects (like scratches), texture defects are handled well, but geometric/structural anomalies remain challenging.
• Representation quality is not entirely cost-free: on fine-grained classification (Birds, Cars), SteerViT scores slightly lower than DINOv2 (91.2 vs 94.7, 77.7 vs 83.8), though it retains 98.8% of DINOv2's average benchmark performance overall. Steerability is unlocked at a very minor cost to original quality.
• Inference overhead increases: SteerViT totals 465M parameters (including the frozen backbone), requiring 133.8 GFLOPS/img and 9.5 ms/img, which is heavier than vanilla DINOv2 (98.6 GFLOPS, 6.6 ms). While significantly lighter than SAM3 and MLLMs, it is not completely free.
• Training relies on referring segmentation/grounding data (with bounding boxes converted to masks via SAM2). As a result, annotation quality and domain coverage limit steering generalization; performance on extreme domains not seen in training still relies on zero-shot extrapolation.

Related Work & Insights

• vs. CLIP / SigLIP (Cross-modal Late Fusion): These models use text during training as supervision, but the visual encoder is query-agnostic at inference. Posteriori text features offer no steering effect (+0.02%). In contrast, SteerViT's early fusion allows text to intervene during visual encoding, raising CORE from 43.7 to 96.0.
• vs. MLLMs (InternVL3 / Qwen3-VL): MLLMs fuse modalities in the early layers of their language models, which dilutes visual fidelity and pushes features into the text space, while requiring billions of parameters. SteerViT adds only 21M parameters, keeps features in the visual space, and outperforms InternVL3-2B on CORE by 20 points.
• vs. Open-Vocabulary Localization (SAM3 / GroundingDINO): While highly steerable (with SAM3 retrieval approaching SteerViT), their intermediate features are specialized for localization and lack transferability, resulting in poor performance on general visual tasks. SteerViT preserves steerability while maintaining the generic quality of the original ViT's representations.
• vs. FLAIR: FLAIR applies text-conditioned attention pooling on a frozen SigLIP (retaining a late fusion style), offering suboptimal steerability (CORE 81.3) and underperforming compared to purely visual encoders on standard visual benchmarks. SteerViT's early fusion achieves 14.7 points higher steerability while preserving representation quality.
• vs. TIE / ELIP / TEVI (Parallel Conditioned Visual Feature Works): These focus on narrow pipelines (TIE optimizes MLLM visual tokens for document understanding, ELIP refines text-to-image retrieval reranking, and TEVI edits CLIP's final layer via SAEs). SteerViT, however, serves as a general-purpose, steerable visual representation framework transferable across various tasks.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Inverting the MLLM paradigm to modulate vision with language introduces a strong class of steerable visual representations. The design of the early fusion gated cross-attention is clean and highly effective.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ The authors introduce two new steerability benchmarks (CORE and MOSAIC) and evaluate across various retrieval, classification, segmentation, anomaly detection, and personalization tasks. Ablations on early vs. late fusion, pointing vs. segmentation, multiple backbones, scaling, data size, and FFNs are exceptionally rigorous.
• Writing Quality: ⭐⭐⭐⭐⭐ The five main findings progress logically. The taxonomy diagrams and Pareto frontiers clearly highlight how the model simultaneously satisfies both steerability and representation quality.
• Value: ⭐⭐⭐⭐⭐ With minimal parameter increases, SteerViT transforms any frozen ViT into a text-steerable encoder that transfers zero-shot to new domains, allowing a single model to substitute for many specialized fine-tuned models. Highly practical and inspiring.

Related Papers

• [AAAI 2026] Empowering DINO Representations for Underwater Instance Segmentation via Aligner and Prompter
• [CVPR 2026] Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera
• [NeurIPS 2025] Exploring Structural Degradation in Dense Representations for Self-supervised Learning
• [ECCV 2024] UniFS: Universal Few-Shot Instance Perception with Point Representations
• [NeurIPS 2025] TabRAG: Improving Tabular Document Question Answering for Retrieval Augmented Generation via Structured Representations