Steerable Visual Representations¶

Conference: ECCV 2026
arXiv: 2604.02327
Code: https://jonaruthardt.github.io/project/SteerViT (Project Page)
Area: Multimodal Visual Representation / Referring Segmentation
Keywords: Steerable Visual Representations, Early Fusion, Gated Cross-Attention, Frozen ViT, Referring Segmentation

TL;DR¶

SteerViT interleaves lightweight gated cross-attention layers inside a frozen ViT (e.g., DINOv2) to introduce textual intervention during the visual encoding process. Trained via a patch-level referring-segmentation proxy task, it adds only 21M parameters while enabling natural language "steering" of visual features to focus on arbitrary objects without degrading the representation quality of the host ViT. It achieves zero-shot performance competitive with or superior to specialized models in text-conditional retrieval, personalized object discrimination, and industrial anomaly segmentation.

Background & Motivation¶

Pre-trained ViTs like DINOv2, MAE, and SigLIP provide universal image features that excel in retrieval, classification, and segmentation. However, they suffer from an inherent limitation: their attention is predominantly captured by the most salient subject in the scene. In an indoor photo containing a cat, a remote control, and a bookshelf, the representation encoded by DINOv2 centers almost entirely on the "cat," while completely neglecting the remote and the bookshelf—primarily due to the "photographer bias" and saliency preferences inherited from training on object-centric visual datasets. The issue is that many downstream tasks (e.g., fine-grained localization, on-demand retrieval of small objects) require focusing on non-salient concepts. Because pure visual encoders are query-agnostic, there is no mechanism to instruct them to "focus on the bookshelf this time."

Another direction is Multimodal Large Language Models (MLLMs), which can indeed be guided by textual prompts. However, because their fusion occurs in the early layers of the language model, the resulting features map to the language space, which dilutes visual fidelity and compromises performance on general vision tasks compared to pure vision encoders. Furthermore, MLLMs require billions of parameters, which is computationally expensive. Cross-modal encoders like CLIP and SigLIP represent typical "late fusion"—image and text are encoded independently, where text only provides supervisory signals during training and cannot alter the visual encoding at inference time. Adding text features posteriously to visual features yields virtually no improvement in retrieval accuracy (experimentally, only +0.02%). Open-vocabulary localization models (e.g., SAM3, GroundingDINO), although steerable by text, specialize their intermediate representations for localization, lacking transferability to general vision tasks. The key challenge is that existing options are either steerable with poor representation quality, or exhibit high representation quality but lack steerability, with no single solution simultaneously satisfying both requirements.

Inspired by human vision—where a textual prompt triggers top-down, goal-directed attention that alters how a scene is parsed—this work reverses the MLLM paradigm: rather than feeding visual inputs into language models, text is used to modulate vision. Core Idea: Lightweight, trainable gated cross-attention layers are interleaved within the blocks of a frozen ViT, allowing visual patch tokens to attend to text tokens during the encoding process (early fusion). These added layers are trained using a patch-level referring segmentation proxy task, yielding a class of vision-centric multimodal representations that are highly steerable by text while preserving the original ViT representation quality.

Method¶

Overall Architecture¶

The objective of SteerViT is to equip any pre-trained ViT with "text-steerable" capabilities without disrupting its native representations. Conceptually, the backbone ViT and text encoder remain completely frozen. Highly lightweight gated cross-attention layers are inserted at every other layer inside the ViT, enabling visual tokens to query text tokens. During training, a patch-level referring segmentation task forces these cross-attention layers to route textual cues into the corresponding visual patches. At inference, no task-specific fine-tuning is required; merely changing the prompt redirects the features to novel concepts or domains.

Specifically, it consists of four components: A. Visual Encoder (frozen ViT, with DINOv2 ViT-B/14 used in main experiments, outputting N patch tokens and an optional [CLS]); B. Text Encoder (frozen RoBERTa-Large, outputting token-level text embeddings); C. Multimodal Adapter (a trainable two-layer MLP that projects L2-normalized text embeddings into the visual alignment space); D. Gated Cross-Attention Layer (the newly introduced module hosting the steering capability, inserted at every other ViT block, resulting in 6 layers for a 12-layer ViT-B). Components A and B are frozen scaffolding blocks, and C is a general alignment step. The core novelty lies in D and the training objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image + Text prompt"] --> B["Frozen ViT / Frozen RoBERTa<br/>Independent Encoding"]
    B --> C["Text Alignment<br/>Two-layer MLP Projection to Visual Space"]
    C --> D["Early Fusion Gated Cross-Attention<br/>Visual Tokens Query Text, tanh Gated Residual Flow"]
    D --> E["Referring Segmentation Proxy Task<br/>Patch-level Soft Cross-Entropy Trains CA Layers"]
    E -->|Change Prompt at Inference| F["Steerable Visual Features<br/>Retrieval / Classification / Segmentation / Anomaly Detection"]

Key Designs¶

1. Early-Fusion Gated Cross-Attention: Allowing Text to Rewrite the Visual Residual Stream during Encoding

This design directly addresses the pain point where "late fusion cannot steer frozen visual features." The authors invert the direction of Flamingo's gated cross-attention—instead of language attending to vision, visual hidden states query language tokens. At layer \(\ell\), visual patch tokens act as queries, while text tokens aligned by the adapter serve as keys/values in standard cross-attention. The elegant design resides in the residual stream gating: rather than directly adding the attention output back, it passes through a tanh gate with a layer-specific learnable scalar \(\alpha_\ell\), yielding:

\[Z_v^{(\ell+1)} = Z_v^{(\ell)} + \tanh(\alpha_\ell)\cdot \hat{Z}_v^{(\ell)}\]

Since \(\tanh(0)=0\), the initialization ensures the model is equivalent to the original frozen ViT, preventing pre-trained features from being disrupted at the start. However, the mechanism is dynamic: the gradient through the gate for \(\alpha_\ell\) contains \(\mathrm{sech}^2(\alpha_\ell)\), and since \(\mathrm{sech}^2(0)=1\), gradient signals remain active even at zero initialization. Therefore, \(\alpha_\ell\) can gradually shift away from zero during optimization, gradually "opening" the language infusion channel. This preserves original ViT representation quality while enabling text to progressively rewrite visual representations layer-by-layer. Ablation studies show that removing this tanh gate (using ungated cross-attention) drops FG-CLS, CORE, and PODS scores by 4.2, 1.4, and 11.0 points respectively, demonstrating that ungated cross-attention heavily disrupts frozen features. Additionally, the authors intentionally omit the gated FFN following cross-attention from the original Flamingo architecture. Adding the FFN offers negligible benefits to representation quality, hurts steerability and OOD transfer, and inflates adapter parameters from 21.2M to 35.4M (+67%), which is counterproductive.

2. Patch-level Referring Segmentation Proxy Task: Driving Cross-Attention to Absorb Language via "Content Matching"

Simply inserting cross-attention layers is insufficient; a task that cannot be solved without referring to text must be designed to compel the visual encoder to leverage textual cues. The authors selected referring segmentation: given an image and a prompt targeting a specific object, the model predicts which patches belong to the referred region. To bypass the complexity of pixel-level decoders, the task is performed on the \(n\times n\) patch grid of the ViT. The ground truth \(y_i\) represents the proportion of foreground pixels in each patch after patchifying the pixel-level binary mask. A linear classification head maps each patch representation to a mask probability \(p_i\), trained with soft cross-entropy:

\[\mathcal{L} = -\sum_{i=1}^{n\times n} y_i \log p_i\]

This soft-label design is crucial. The authors compared this with a simpler "pointing" baseline (placing a Gaussian kernel on the bounding box center as the target) and found that pointing only teaches the model "where the object is" while remaining insensitive to object shape and size, collapsing supervision to a single spatial position. In contrast, the segmentation target activates all tokens overlapping with the object, teaching "content matching"—each patch must determine whether it depicts the described object, rather than just how close it is to the center. Experimentally, the segmentation target consistently outperforms pointing: FG-CLS +7.3, ADE20k +8.0, and PODS +12.4. This "content matching" supervision enables cross-attention layers to accurately route text instructions into matching visual patches, producing highly steerable representations.

3. Gating Scalars as Continuous Control Knobs at Inference: Smoothly Interpolating between Original ViT and Fully Conditioned States

Because steering capacity is fully mediated by the tanh gates, the authors discovered a free benefit: the gating scalar \(\alpha_\ell\) naturally acts as a continuous dial for "textual conditioning strength." At inference, multiplying the learned \(\alpha_\ell\) by a scaling factor \(\omega\in[0,1]\) allows smooth interpolation between the "unaltered ViT subspace" and the "fully conditioned state"—where \(\omega=0\) reverts to the original ViT, and \(\omega=1\) enables full steering. Plotting this trajectory reveals a clear Pareto frontier: both DINOv2 and SigLIP achieve the best trade-off between steerability and representation quality around \(\omega=0.6\), where representation quality slightly exceeds the original ViTs while unlocking high steerability. The most dramatic shift is seen in MAE, where representation quality monotonically rises with \(\omega\), climbing from 40 points at \(\omega=0\) to 50 points at \(\omega=0.6\). This indicates that textual conditioning injects semantic structure into MAE's semantically immature features, enhancing their transferability. This knob transforms the "steerability vs. representation quality" trade-off from a binary choice into an adjustable, continuous spectrum.

Loss & Training¶

The training objective is the patch-level soft cross-entropy referring segmentation loss detailed above. Training data is compiled from a mixture of referring segmentation and grounding datasets: RefCOCO/+/g, Visual Genome, LVIS, and Mapillary Vistas, totaling 162k unique images and 2.28M image-text pairs. This dataset spans indoor/outdoor/street-view domains and ranges from two-word labels to multi-sentence descriptions. Bounding boxes are processed into binary masks using SAM2. The backbone consists of DINOv2 ViT-B/14 + RoBERTa-Large with a \(336^2\) resolution, a batch size of 12, trained for 500k steps (approx. 84 H100 GPU hours) using AdamW with a cosine schedule (warmup to 3e-4, then decaying to 3e-5). Throughout training, only the 21.2M cross-attention parameters are updated, while the ViT and text encoder remain fully frozen.

Key Experimental Results¶

Main Results¶

Four classes of baselines are compared: pure visual encoders (DINOv2, MAE), cross-modal encoders (CLIP, SigLIP, with late fusion via element-wise addition), MLLMs (InternVL3, Qwen3-VL, LFM-2.5-VL, pooling the final token similar to E5-V), and open-vocabulary localization models (SAM3, GroundingDINO, extracting intermediate multimodal features).

Task / Benchmark	Metric	SteerViT	DINOv2	Baselines
CORE Conditional Retrieval	acc@1	96.0	43.7	FLAIR 81.3 / SAM3 near 96 / InternVL3-2B 20 pts lower
MOSAIC Directed Attention	PR-AUC	50.2	14.3	—
GeneCIS Focus Object (Zero-shot Real)	R@1	25.4	9.6	Specialized Baseline 18.7
PODS Personalized Object Discrimination (detailed prompt)	PR-AUC	58.1	29.6	Fine-tuned DINOv2 variant 48.0 (requires one model per category)
MVTec AD Zero-shot Anomaly Segmentation	PRO	82.1	—	SAM3 54.5 / FADE (Specialized) 84.5
VisA Zero-shot Anomaly Segmentation	PRO	82.0	—	FADE (Specialized) 79.3

Key observation: On CORE, early fusion boosts DINOv2 from 43.7 to 96.0, whereas late-fusion posterior text addition yields a negligible +0.02% improvement, confirming that "late fusion cannot modify frozen features." Anomaly segmentation presents an extreme OOD setting where SteerViT matches specialized methods zero-shot. In personalized discrimination, a single SteerViT model (relying on prompt switching) outperforms fine-tuned DINOv2 structures that require training 100 separate models for 100 distinct objects.

Ablation Study¶

Configuration	FG-CLS↑	ADE↑	CORE↑	PODS↑	Description
Full (Early Fusion + tanh Gate + MLP)	87.7	55.4	96.0	58.1	Full model
w/o Early Fusion (changed to late fusion)	91.8	55.5	93.3	36.6	Classification increases, but PODS plummets by 21.5
w/o tanh Gate (ungated CA)	83.5	55.3	94.6	47.1	Ungated CA disrupts frozen features
w/o MLP (modified to single linear layer)	86.7	54.5	95.2	56.4	Worse alignment
Pointing Objective (vs. Segmentation)	80.4	47.4	95.2	45.7	Teaches "where" but not "what"

Key Findings¶

Early fusion is highly crucial for fine-grained tasks: While late fusion achieves higher performance on coarse superclass classification, its execution collapses on tasks requiring instance-level discrimination like PODS (36.6 vs 58.1). When using coarse superclass prompts, this performance gap disappears (26.5 vs 27.9), demonstrating that the value of early fusion lies specifically within fine-grained tasks.
Text granularity directly controls feature granularity: On PODS, upgrading prompts from coarse superclasses ("mug") to instance names ("white ECCV mug") and further to MLLM-generated detailed appearance descriptions increases the PR-AUC from 27.9% to 58.1%. Rather than merely adding information, SteerViT leverages prompt detail to precisely control the semantic granularity of visual features.
Steering is genuinely text-driven: When conditioned on false random classes, SteerViT drops by 48.3 points and FLAIR drops by 29.4 points, whereas CLIP/SigLIP show zero changes. This proves that SteerViT's features are actively reshaped by text, whereas late-fusion setups merely maintain unconditional visual representations.
Weaker backbones benefit more: The CORE gain of early fusion relative to late fusion is +33.9 on MAE and +15.9 on SigLIP, which is significantly larger than the +2.7 gain on DINOv2. The less semantically mature the underlying visual representation, the more useful the language injection.
Emergent multi-level and compositional steering: Conditioning on "animal" merges animal classes into a macro-cluster while retaining fine details. Conditioning on "eye" reorganizes spatial representations based on the compositional attribute of "having eyes," grouping humans and animals together—even though such behavior was not explicitly targeted during training.

Highlights & Insights¶

Inverting the MLLM paradigm is the key "aha" moment of this paper: Rather than the default multimodal pipeline of feeding vision into language, this paper reverses the flow to modulate vision with language. By adding only 21M parameters (two orders of magnitude fewer than MLLMs), it achieves highly steerable, vision-centric representations, securing both efficiency and quality.
Zero-initialized tanh gating is a highly elegant engineering trick. It ensures initial equivalence to the frozen ViT (hence protecting pre-trained features) while retaining gradient signals via \(\mathrm{sech}^2(0)=1\), which then naturally serves as a continuous inference-time scaling knob. This "zero-initialized gated residual infusion" paradigm heavily translates to any scenario seeking to safely add newly trained modules to frozen backbones.
The pointing vs. segmentation comparison provides a highly reusable insight: The density of supervision in a proxy task dictates what is learned. Sparse coordinate supervision merely teaches localization, whereas dense mask supervision forces "content matching," yielding high-quality representations. This is highly informative for designing proxy tasks in self-supervised representation learning.
Prompt detail acts as a granularity scale: A single frozen model, without retraining, can switch its semantic granularity from coarse categories to instance-level descriptions simply by changing the prompt. This offers massive practical value in personalized and long-tail scenarios, replacing multiple specialized models with a single unified framework.

Limitations & Future Work¶

Under zero-shot anomaly segmentation settings, certain physical defects (e.g., flipped metal nuts) cannot be accurately predicted because the model lacks training on normal-class behaviors or visible surface anomalies (like scratches). Although texture defects yield strong performance, structural or geometric anomalies remain challenging.
Representation quality is not entirely cost-free: On fine-grained classification datasets like Birds and Cars, SteerViT scores slightly lower than DINOv2 (91.2 vs. 94.7, and 77.7 vs. 83.8), maintaining an overall score of 98.8% of the original DINOv2 performance. Steering capabilities come at a minimal cost to raw representation quality.
Computational overhead increases: SteerViT's total parameter count is 465M (including the frozen backbone), consuming 133.8 GFLOPS/img and 9.5 ms/img, which is heavier than vanilla DINOv2 (98.6 GFLOPS, 6.6 ms). Although vastly lighter than SAM3 and MLLMs, it is not computationally free.
Training supervision relies heavily on referring segmentation/grounding data (utilizing SAM2 to convert bounding boxes to masks). The quality and domain coverage of these annotations impact steerable generalization; performance on highly out-of-distribution domains still relies on zero-shot extrapolation.

vs. CLIP / SigLIP (Cross-modal late fusion): These models use text-supervision during training, but visual encoding remains query-agnostic at inference, where adding text posteriously yields zero steering (+0.02%). In contrast, early fusion in SteerViT intervenes during visual encoding, pushing CORE from 43.7 to 96.0.
vs. MLLMs (InternVL3 / Qwen3-VL): MLLM fusion occurs within early language layers, causing features to drop into language space with degraded visual fidelity and requiring billions of parameters. This work adds only 21M parameters, keeps representations in the visual space, and exceeds InternVL3-2B on CORE by 20 points.
vs. Open-vocabulary localization (SAM3 / GroundingDINO): While highly steerable (with SAM3 retrieval approaching SteerViT), their systems are optimized solely for localization and lack transferability, scoring poorly on general vision tasks. This work achieves high steerability while preserving the performance of general-purpose visual representations.
vs. FLAIR: FLAIR applies text-conditioned attention pooling to a frozen SigLIP (still late fusion), yielding sub-optimal steerability (81.3 on CORE) and failing to equal pure visual encoders on standard tasks. This early-fusion approach achieves 14.7 higher points in steerability while preserving representation quality.
vs. TIE / ELIP / TEVI (Parallel text-conditioned visual feature works): These works focus on narrow pipelines (e.g., TIE optimizes MLLM vision tokens for document understanding, ELIP redirects text-image search reranking, TEVI edits CLIP final layers). This work offers a generalizable framework for producing steerable visual representations across diverse downstream tasks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Reversing the MLLM paradigm to modulate vision with language establishes a new class of steerable visual representations. The implementation of early-fusion gated cross-attention is clean and highly effective.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Introducing two new steerability benchmarks (CORE, MOSAIC) and demonstrating performance across retrieval, classification, segmentation, anomaly detection, and personalization. Ablations verify early vs. late fusion, pointing vs. segmentation, multiple backbones, scaling, data quantity, and FFNs.
Writing Quality: ⭐⭐⭐⭐⭐ The five main findings build step-by-step, and the taxonomy overview together with the Pareto frontier illustrate the dual-demand solution very intuitively.
Value: ⭐⭐⭐⭐⭐ Upgrades any frozen ViT with text-steerable capabilities using minimal parameter overhead and enables zero-shot domain transfer. Relying on a single model to replace multiple specialized fine-tuned models is both highly practical and inspiring.