Controlling Embedding Spaces with Text-Conditioned Transformations¶
Conference: ECCV 2026
Paper: ECCV Official Poster
Project Page: https://joefioresi718.github.io/ControlEmbed_webpage/
Area: Multimodal VLM
Keywords: Vision-Language Models, embedding space control, attribute disentanglement, attribute-aware retrieval, multi-clustering
TL;DR¶
By employing a lightweight three-layer MLP hypernetwork that predicts global affine transformations conditioned on natural language attribute descriptions, this approach achieves state-of-the-art attribute-aware retrieval over frozen galleries and unsupervised multi-clustering at near-zero inference cost.
Background & Motivation¶
Large-scale vision-language models like CLIP and SigLIP learn versatile multimodal embedding spaces from web-scale data, underpinning core capabilities such as zero-shot classification, cross-modal retrieval, and semantic similarity search. However, because these foundation models compress rich, multifaceted visual concepts into a single dense vector, high-level object-level semantics inherently dominate the latent space. Subtler yet critical visual propertiesโsuch as camera perspective, lighting, color tone, or artistic styleโare left entangled and suppressed within dominant semantics. When a user queries a gallery with a reference image, current systems offer no control over whether similarity ranks matches based on the foreground subject or on compositional nuances.
Prior methodologies addressing attribute disentanglement generally follow two extremes. One paradigm fine-tunes the base encoder or attaches distinct attribute-specific adapters; this demands expensive retraining, scales storage and memory linearly with the number of attributes, degrades the base model's generalizability, and necessitates complete re-indexing of large image galleries. The alternative paradigm cascades large vision-language models and LLMs to caption images and execute multi-turn heuristic clustering at inference time; although training-free, it incurs immense latency and prohibitive computational costs, rendering it impractical for real-time applications.
To resolve this dilemma, this work operates directly on the latent geometry of frozen foundation embeddings without altering base weights. Core idea: train a single text-conditioned hypernetwork to dynamically generate affine transformation parameters that reshape the latent embedding space, enabling precise attribute-aware retrieval over a static frozen gallery and unsupervised multi-clustering with negligible computational overhead.
Method¶
Overall Architecture¶
The proposed framework formulates attribute control as a learned transformation operator within the latent embedding space. Both the vision encoder and text encoder remain completely frozen. Given a natural language attribute category (such as "color" or "camera angle"), a lightweight MLP hypernetwork predicts the parameters of an affine transformation matrix \(W\) and bias vector \(b\). This transformation serves dual inference capabilities: in the query setting, it is applied solely to the query embedding to perform attribute-aware retrieval against an un-modified frozen gallery; in the global setting, the same transformation reshapes the entire collection of image embeddings, creating an attribute-disentangled metric space for downstream unsupervised clustering.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Attribute Description<br/>e.g., color / camera angle"] --> B["Frozen Text Encoder ft<br/>CLIP / SigLIP text branch"]
B --> C["Hypernetwork Transform Generation<br/>Predict affine parameters W and b"]
C --> D{"Inference Mode"}
D -->|Retrieval Mode| E["Query Transform & Scoring<br/>Transform query only, gallery frozen"]
D -->|Clustering Mode| F["Global Space Reshaping<br/>Transform full gallery for clustering"]
Key Designs¶
1. Hypernetwork Transform Generation: Unified Multi-Attribute Mapping Instead of dedicating independent adapters or separate heads to individual attributes, the framework deploys a single three-layer MLP hypernetwork \(H_\phi\). Taking the normalized text embedding of the queried attribute category \(t = f_t(T) \in \mathbb{R}^D\) as input, the hypernetwork generates the full parameter set of an affine operator: $\(W, b = H_\phi(t)\)$ where \(W \in \mathbb{R}^{D \times D}\) and \(b \in \mathbb{R}^D\). An image embedding \(v = f_v(I)\) is mapped to an attribute-specialized representation via: $\(v' = W^\top v + b\)$ Compared to pure translation vectors that merely shift coordinates, an affine operator can scale and rotate arbitrary sub-manifolds, amplifying directions aligned with the desired attribute while attenuating irrelevant dominant semantic axes. Training a unified hypernetwork across all attribute categories prevents parameter footprint expansion as new attributes are registered, while facilitating cross-attribute structural transfer.
2. Multi-Positive Contrastive Alignment: Preserving Frozen Gallery Compatibility To ensure that transformed query embeddings effectively retrieve target items from a completely frozen, pre-computed gallery, the training workflow employs a multi-positive contrastive loss based on StableRep. During training, an attribute category is sampled per batch, and class labels are determined based on the subclasses under that category. For any pair \((i, j)\) sharing the same subclass label, an indicator variable is set to \(\mathbb{1}_{i,j} = 1\). The objective pulls transformed query embeddings \(v'_i\) toward original frozen base embeddings \(v_j\) belonging to identical subclasses, while pushing away non-matching samples: $\(\mathcal{L}_{\text{SR}} = - \frac{1}{N} \sum_{i=1}^N \frac{1}{\sum_{k=1}^N \mathbb{1}_{i,k}} \sum_{j=1}^N \mathbb{1}_{i,j} \log \frac{\exp(v_i'^\top v_j / \tau)}{\sum_{k=1}^N \exp(v_i'^\top v_k / \tau)}\)$ Because the retrieval score factorizes as \(\langle W^\top v_i + b, v_j \rangle = v_i^\top W v_j + b^\top v_j\), the transformation acts as an asymmetric bilinear metric on the static gallery vectors \(v_j\). Consequently, large-scale gallery indexes need never be re-computed or updated.
3. Symmetric Multimodal Alignment: Grounding Semantics in Pretrained Structure
To anchor transformed visual features to their linguistic concepts and enhance cross-modal generalization, a bidirectional multimodal contrastive loss is incorporated. For each subclass label \(c\), an average text embedding \(\bar{t}_i\) is constructed across multiple prompt templates (e.g., "a photo of {c}" or "an image in the {c} style"). The visual-to-text loss is defined as:
$\(\mathcal{L}_{\text{i2t}} = - \frac{1}{N} \sum_{i=1}^N \log \frac{\exp(v_i'^\top \bar{t}_i / \tau)}{\sum_{j=1}^N \exp(v_i'^\top \bar{t}_j / \tau)}\)$
Coupled with the symmetric text-to-visual component \(\mathcal{L}_{\text{t2i}}\), the overall loss balances visual grouping and textual alignment:
$\(\mathcal{L} = \mathcal{L}_{\text{SR}} + \omega (\mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}})\)$
with weighting hyperparameter \(\omega = 0.1\). This explicit multimodal guidance significantly improves zero-shot text-classification performance on transformed embeddings and fosters generalization to unseen attribute subclasses.
4. Compositional Retrieval: Training-Free Linear Query Blending When a search requires conditioning on multiple attribute dimensions simultaneously (e.g., finding images matching both a specific "action" and a "location"), the model enables training-free composition. By computing separate transformation parameters \((W_a, b_a)\) and \((W_b, b_b)\), a composite query embedding is synthesized via a convex combination: $\(q_{\text{blend}} = \alpha (W_a^\top q + b_a) + (1 - \alpha) (W_b^\top q + b_b)\)$ At \(\alpha = 0.5\), this linear blend successfully isolates intersectional matches without destabilizing the metric space, confirming the orthogonal selectivity of the learned affine transformations.
Key Experimental Results¶
Main Results¶
The method was evaluated across synthetic and realistic benchmarks: Clevr-4 (geometric attributes: color, count, shape, texture), Stanford 40 Actions (action, location, mood), and ShotBench (cinematic photographic attributes: camera angle, shot size, framing, lighting, etc.).
In Table 1, attribute-aware retrieval is measured by mean Average Precision (mAP, %) over frozen galleries across dataset taxonomies. Baselines include frozen base features, training-free concept projections (LCR, CWP), fine-tuned adapters (ArcFace, SupCon, StableRep+), PEFT models (MaPLe, LoRA), and zero-shot composed retrieval (SEARLE-XL).
| Method | Multi-Attr. Single Model | Clevr-4 mAP (%) | Stanford 40 mAP (%) | ShotBench mAP (%) |
|---|---|---|---|---|
| Base Features (CLIP ViT-L/14) | โ | 33.8 | 43.7 | 25.2 |
| LCR (Linear Concept Reconstruction) | โ | 15.9 | 19.9 | 18.9 |
| CWP (Concept-Weighted Projection) | โ | 43.8 | 49.8 | 21.4 |
| ArcFaceโ | โ | 12.3 | 18.8 | 17.5 |
| SupConโ | โ | 12.7 | 18.8 | 17.7 |
| StableRep+โ | โ | 49.5 | 21.3 | 20.9 |
| MaPLe | โ | 33.5 | 47.8 | 23.7 |
| LoRA | โ | 42.9 | 49.3 | 22.0 |
| SEARLE-XL | โ | 24.6 | 33.9 | 20.3 |
| Ours-Query | โ | 77.0 | 86.3 | 41.0 |
In Table 2, unsupervised multi-clustering accuracy (Hungarian-matched c.ACC, %) and inference runtime (seconds for 5,000 images) are evaluated across 14 attribute categories.
| Method | Inference Time (s) | Clevr-4 c.ACC (%) | Stanford 40 c.ACC (%) | ShotBench c.ACC (%) |
|---|---|---|---|---|
| Base Features | 0.59 | 48.8 | 65.2 | 20.8 |
| LCR | 0.62 | 56.4 | 64.8 | 30.2 |
| CWP | 0.68 | 60.3 | 64.9 | 32.2 |
| Multi-MAP | 750 | 62.8 | 62.8 | 32.6 |
| Multi-Sub | 15,300 | 72.2 | 66.1 | 32.4 |
| IC | TC | 37,680 | 57.9 | 76.1 |
| SSD-LLM | 27,600 | 56.5 | 74.1 | 25.0 |
| X-Cluster | 104,760 | 64.9 | 68.3* | - |
| SEARLE-XL | 81 | 30.7 | 46.9 | 24.5 |
| Ours-Space | 0.64 | 73.0 | 80.0 | 51.6 |
Ablation Study¶
Table 3 investigates transformation types, demonstrating that affine operators decisively outperform pure translation vectors.
| Dataset | Variant | Cluster c.ACC (%) | Query mAP (%) | Global mAP (%) |
|---|---|---|---|---|
| Clevr-4 | Base | 32.2 | 33.8 | โ |
| Clevr-4 | Translation-only | 32.2 | 35.4 | 34.5 |
| Clevr-4 | Affine | 75.4 (+43.2) | 77.6 (+42.2) | 85.1 (+50.6) |
| Stanford 40 | Base | 46.4 | 43.6 | โ |
| Stanford 40 | Translation-only | 55.6 | 43.8 | 44.5 |
| Stanford 40 | Affine | 79.4 (+23.8) | 84.4 (+40.6) | 87.7 (+43.2) |
Table 8 isolates the effect of the multimodal alignment loss (\(\mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}}\)).
| Objective | Multi-Clustering (Clevr4/S40/Shot) | Attr.-Aware Retrieval (Clevr4/S40/Shot) | Text-based Classification (Clevr4/S40/Shot) |
|---|---|---|---|
| Full Model (Ours) | 73.0 / 80.0 / 51.6 | 77.1 / 91.2 / 41.1 | 88.7 / 83.2 / 51.6 |
| w/o Multimodal Guidance | 72.7 / 78.4 / 53.2 | 76.8 / 87.3 / 40.7 | 71.7 / 78.5 / 31.3 |
Key Findings¶
- Affine Transformation is Essential: Translation offsets yield minimal improvement on Clevr-4 retrieval (33.8% \(\to\) 35.4%), whereas affine transformation jumps to 77.6%. SVD analysis confirms that matrix \(W\) contracts the stable rank (e.g., Color stable rank drops from 11.3 to 5.0) and isolates a dominant singular subspace yielding 98.4% retrieval mAP.
- Extreme Data Efficiency: On Stanford 40, our global affine transformation nears peak clustering performance with as few as 6 training samples per class (>78% accuracy).
- Interference-Free Scaling: Scaling the hypernetwork from 1 attribute category to 19 categories with 187 subclasses results in a performance variation of less than 1.5%, demonstrating outstanding cross-attribute stability.
Highlights & Insights¶
- Asymmetric Scoring Compatibility: The inner product factorization \(\langle W^\top v_i + b, v_j \rangle = v_i^\top W v_j + b^\top v_j\) enables axis-aware retrieval while retaining 100% frozen, pre-indexed gallery embeddings.
- Emergent Global Restructuring from Query-Only Training: Although trained exclusively on query-to-gallery contrastive alignment, the identical learned transformation operates globally (\(V \to V'\)) during inference, establishing superior clustering geometry.
- Cross-Domain and Cross-Architecture Transfer: Transformations trained on synthetic primitives (Clevr-4) transfer seamlessly to complex real-world MS-COCO images. Furthermore, the approach delivers massive gains across vision-language encoders (CLIP, SigLIP, Qwen3-VL-Embed) as well as self-supervised visual encoders lacking language alignment (DINOv2).
Limitations & Future Work¶
- Generalization Bounded by Training Taxonomies: The hypernetwork effectively generalizes to unseen subclasses within trained categories, but struggles when queried with entirely novel attribute domains (e.g., physical surface hardness) absent from training data.
- Fine-Grained Color Disentanglement: Distinguishing adjacent hues (such as cyan versus blue) remains challenging due to substantial latent overlap in base embeddings.
- Future Directions: Scaling up the vocabulary of attribute categories during multi-task pretraining to discover zero-shot emergent transformation properties for unconstrained open-world language queries.
Related Work & Insights¶
- vs Parameter-Efficient Fine-Tuning (LoRA, MaPLe): PEFT methods require separate adapter weights per attribute category and force expensive re-encoding of gallery collections. This method maintains a single static hypernetwork and performs zero-re-encoding retrieval.
- vs Fine-Tuned Multiple Clustering (Multi-MAP, Multi-Sub): Prior clustering methods run optimization passes per user query upon receiving the dataset; our method executes in 0.64s via direct matrix multiplication, running 4 orders of magnitude faster.
- vs LLM/LMM-Assisted Clustering (IC|TC, SSD-LLM, X-Cluster): Multimodal LLM cascades consume tens of thousands of seconds captioning and reasoning over collections; our method accomplishes superior clustering accuracy directly in latent space in fractions of a second.
Rating¶
- Novelty: โญโญโญโญโญ Elegant shift from weight fine-tuning to latent-space operator generation via text-conditioned hypernetworks.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage across retrieval, clustering, SVD geometric analysis, few-shot scaling, and multiple backbones.
- Writing Quality: โญโญโญโญโญ Crisp exposition with clear mathematical formulation and well-structured empirical validation.
- Value: โญโญโญโญโญ Practical, zero-re-indexing solution highly suitable for enterprise-scale visual search engines and multi-faceted media organization.