What Images Cannot Say: Language-Guided Olfactory Representation Learning¶
Conference: ECCV 2026
Paper: ECCV Official
Project: Project Page
Area: Information Retrieval & RAG
Keywords: Olfactory Representation Learning, Cross-Modal Retrieval, Vision-Language Models, Electronic Nose, Odor Disentanglement
TL;DR¶
Addressing the fundamental challenge that camera images cannot observe out-of-frame diffusing odor sources and ambient contexts, SCENT leverages Vision-Language Models (VLMs) to generate structured language descriptors across objects, context, and inferred smells as a semantic bridge, achieving state-of-the-art cross-modal retrieval and interpretable odor mixture disentanglement.
Background & Motivation¶
Multimodal perception systems have traditionally focused on visual, auditory, and textual streams, yet human perceptual embodiment in physical spaces extends far beyond visible pixels. Olfaction, as a primary chemical sense conveying airborne environmental properties, is critical for food safety assessment, airborne hazard monitoring, and embodied environmental comprehension. Recent advances in portable electronic noses (e-noses) now permit the capture of high-dimensional multi-channel sensor measurements in unconstrained settings, prompting large-scale benchmarks such as New York Smells (NYS) to pair visual scenes with real-world olfactory data.
However, aligning raw multi-channel resistance signals from an e-nose array with semantic representations faces two deep physical and perceptual bottlenecks. First, smells diffuse freely throughout ambient environments, frequently originating from odor sources completely outside the camera's limited field of view—such as ventilation exhaust in a subway concourse or distant vehicle fumes on a street. Visual supervision alone suffers from severe partial observability, as pixel features cannot account for ambient airborne molecules recorded by the physical sensors. Second, natural olfactory recordings capture complex mixtures where target object emissions and surrounding ambient background odors are intrinsically blended, making it virtually impossible to isolate individual semantic factors without structured supervision.
Pretrained on web-scale multimodal corpora, modern Vision-Language Models (VLMs) exhibit vast commonsense world priors that extend well beyond surface pixel descriptions into physical reasoning. Given an image, a VLM can infer plausible ambient smells and environmental context suggested by the scene. This paper leverages language as a semantic bridge to translate unseen olfactory factors into explicit supervision. Core idea: propose SCENT, a framework that employs VLM-generated object, context, and ambient smell inferences as semantic supervision to align e-nose sensor signals with visual and textual representations, followed by a language-guided latent decomposition that disentangles object-specific odors from environmental contributions.
Method¶
Overall Architecture¶
The SCENT (Semantic Context-aware e-Nose Transformer) architecture operates across three sequential and complementary stages: VLM-based semantic augmentation, multimodal olfactory alignment, and latent smell disentanglement. The system takes paired scene images \(I\) and 32-channel e-nose time-series signals \(X = [B; S] \in \mathbb{R}^{C \times 2T}\) (concatenating an ambient baseline recording \(B\) and an object sample recording \(S\)), learning an expressive, decoupled embedding space aligned with visual and language modalities.
Initially, a pretrained VLM queries the image to produce three levels of structured textual descriptors \(T(I)\), which are converted into high-dimensional semantic anchors via a frozen text encoder. Concurrently, a Transformer-based smell encoder models temporal dynamics across the 32 sensor channels, and dual MLP heads project the olfactory embedding into visual and textual spaces to compute bidirectional InfoNCE contrastive losses against fine-tuned image features and frozen text features. Finally, dedicated latent projection heads and a signal decoder split the smell representation into object and context components, supervised by separate text targets and a signal reconstruction loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Scene Image I + E-Nose Signal X"] --> B["Stage 1: VLM Semantic Augmentation<br/>infer object/context/ambient smells"]
B --> C["Stage 2: Multimodal Representation Learning<br/>dual-head contrastive alignment"]
C --> D["Stage 3: Latent Smell Disentanglement<br/>decompose object & context with rec loss"]
D --> E["Cross-Modal Retrieval & Zero-Shot Synthesis"]
Key Designs¶
1. VLM Semantic Scene Augmentation: Grounding Out-of-Frame Smells with Commonsense World Knowledge
To overcome the partial observability inherent in camera images, the framework queries a strong VLM (Qwen3-VL-30B) using a structured prompt that elicits scene knowledge across three distinct semantic tiers: $\(T(I) = \{ t_{\text{obj}}, t_{\text{ctx}}, \{ t_{\text{smell}}^{(k)} \}_{k=1}^K \}\)$ Here, \(t_{\text{obj}}\) identifies the primary object, \(t_{\text{ctx}}\) captures the macroscopic environmental context, and \(\{ t_{\text{smell}}^{(k)} \}_{k=1}^K\) explicitly commands the model to infer plausible ambient olfactory elements suggested by the setting but not directly visible in the frame (e.g., printer ozone, damp wood, or air-conditioning airflow). These descriptors are packed and encoded using a frozen CLIP text encoder \(f_T\), yielding a stable, high-dimensional semantic anchor \(z^T = f_T(\text{prompt}(T))\). This formulation provides dense linguistic supervision that bridges the gap between raw chemical measurements and restricted visual observations.
2. Multimodal Olfactory Alignment: Dual-Head Co-Embedding of Physical Sensors, Vision, and Language
To map multi-channel sensor resistance profiles into semantic spaces, SCENT employs a 6-layer, 8-head Transformer encoder \(f_S\). The 32-channel concatenated array input \(X \in \mathbb{R}^{32 \times 2T}\) is linearly projected to 448 dimensions with sinusoidal positional encodings, yielding global representation \(z^S = f_S(X)\). Rather than forcing disparate modalities into a single rigid vector space, SCENT introduces two modality-specific MLP projection heads: $\(z^S_I = \phi_I(z^S) \in \mathbb{R}^{512}, \qquad z^S_T = \phi_T(z^S) \in \mathbb{R}^{512}\)$ During training, the text encoder remains frozen to preserve broad semantic manifolds, whereas the visual encoder \(f_I\) is fine-tuned to adapt towards olfactory-correlated visual cues. The entire encoder is optimized via symmetric bidirectional InfoNCE objectives: $\(\mathcal{L}_{\text{total}} = \lambda_{IS}\mathcal{L}_{IS} + \lambda_{ST}\mathcal{L}_{ST}\)$ where \(\mathcal{L}_{IS}\) aligns the visual-smell pair \((z^S_I, z^I)\) and \(\mathcal{L}_{ST}\) aligns the text-smell pair \((z^S_T, z^T)\). This co-training regularizes the olfactory latent space against both concrete visual details and abstract semantic priors.
3. Language-Guided Latent Disentanglement: Isolating Object and Context Olfactory Components
Natural smell recordings are chemical mixtures combining target volatile compounds and background ambient air. Taking advantage of the compositional structure of language, SCENT decomposes the learned text-projected smell latent \(z^S_T\) into two specialized sub-vectors via dedicated projection heads: $\(z^S_{\text{obj}} = \phi_{\text{obj}}(z^S_T), \qquad z^S_{\text{ctx}} = \phi_{\text{ctx}}(z^S_T)\)$ These heads are supervised by contrastive losses \(\mathcal{L}_{\text{obj}}\) and \(\mathcal{L}_{\text{ctx}}\) against the individual CLIP text embeddings \(z^T_{\text{obj}}\) and \(z^T_{\text{ctx}}\) derived in Stage 1. To prevent representation collapse and ensure that the decomposed latents preserve the physical characteristics of the underlying sensor signal, an auxiliary decoder \(d(\cdot)\) reconstructs the original sensor measurements from the concatenated latents: $\(\mathcal{L}_{\text{rec}} = \| d([z^S_{\text{obj}}; z^S_{\text{ctx}}]) - X \|^2\)$ The combined disentanglement objective is \(\mathcal{L}_{\text{dis}} = \mathcal{L}_{\text{obj}} + \mathcal{L}_{\text{ctx}} + \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}\). This ensures that the decomposed representations faithfully isolate target object odors from environmental context while remaining grounded in physical sensor readings.
Loss & Training¶
Optimization follows a two-stage regimen: - Stage 1 (Representation Learning): Trains smell encoder \(f_S\), dual projection heads \(\phi_I, \phi_T\), and fine-tunes visual encoder \(f_I\). The overall loss balances image-smell and text-smell contrastive terms equally (\(\lambda_{IS} = \lambda_{ST} = 1.0\)) with learnable temperature \(\tau\). - Stage 2 (Latent Disentanglement): Freezes the Stage 1 backbone and trains decomposition heads \(\phi_{\text{obj}}, \phi_{\text{ctx}}\) along with decoder \(d(\cdot)\), balancing decomposition contrastive objectives with the reconstruction regularization \(\lambda_{\text{rec}}\).
Key Experimental Results¶
Main Results¶
Evaluations are conducted on the New York Smells (NYS) benchmark comprising 7,000 paired image-smell recordings (5,996 training / 936 validation samples across 3,500 distinct object categories). Tasks include single-modality retrieval (Smell-to-Image S2I, Smell-to-Text S2T) and joint-modality retrieval (Smell-to-Image+Text S2IT). The baseline is the official NYS vision-only model, as well as an adapted NYS baseline using an Image Bridge (IB) protocol (retrieving the nearest image, which subsequently queries text via frozen CLIP).
| Method / Supervision | Text Guidance | S2I R@5 | S2I R@10 | S2I R@20 | S2T R@5 | S2T R@10 | S2T R@20 | S2IT R@5 | S2IT R@10 | S2IT R@20 |
|---|---|---|---|---|---|---|---|---|---|---|
| NYS Original [44] | None | 16.5 | 29.6 | 43.1 | — | — | — | — | — | — |
| NYS Reproduced | None | 20.0 | 29.9 | 42.0 | — | — | — | — | — | — |
| NYS Adapted (IB) | O | — | — | — | 6.2 | 10.4 | 14.4 | 15.6 | 25.8 | 39.1 |
| NYS Adapted (IB) | O + Ctx | — | — | — | 8.7 | 12.7 | 18.3 | 15.9 | 25.2 | 38.7 |
| NYS Adapted (IB) | O + Ctx + S | — | — | — | 8.1 | 12.9 | 19.2 | 18.5 | 27.5 | 40.7 |
| SCENT (Ours) | O | 22.1 | 32.7 | 42.4 | 8.0 | 13.8 | 21.5 | 22.0 | 32.4 | 42.6 |
| SCENT (Ours) | O + Ctx | 21.8 | 32.4 | 45.2 | 10.7 | 17.3 | 28.1 | 21.3 | 32.5 | 45.7 |
| SCENT (Ours) | O + Ctx + S | 23.0 | 33.5 | 43.6 | 11.9 | 19.8 | 29.2 | 23.3 | 32.7 | 42.5 |
Ablation Study¶
The authors perform extensive ablations on the architecture design, zero-shot recombination capabilities, and VLM backbone sensitivity.
Table 1: Three-Stream Architecture vs. Image Bridge Proxy Ablation
| Model Config | Image Bridge (IB) | S2T R@5 | S2T R@20 | S2IT R@5 | S2IT R@20 | Note |
|---|---|---|---|---|---|---|
| NYS Adapted Baseline | Yes (IB) | 8.1 | 19.2 | 18.5 | 40.7 | Routes text retrieval via retrieved top-1 image |
| SCENT Restricted | Yes (IB) | 9.1 | 18.8 | 21.4 | 43.8 | Disables native ST head; uses image proxy |
| SCENT Full (Ours) | No (Native 3-Stream) | 11.9 | 29.2 | 23.3 | 42.5 | Directly projects olfactory signals to text space |
Table 2: Zero-Shot Recombination Retrieval of Decomposed Latents
| Query Construction Protocol | R@1 | R@5 | R@10 | R@20 | Note |
|---|---|---|---|---|---|
| Raw Sensor Recombination | 3.9 | 5.9 | 7.8 | 9.8 | Concatenates raw sensor baseline and sample recordings |
| Decoded Synthesis (Ours) | 2.0 | 9.8 | 11.8 | 13.7 | Synthesizes fingerprints from recombined \(z^S_{\text{obj}}\) and \(z^S_{\text{ctx}}\) |
Table 3: Sensitivity to VLM Reasoning Capability
| Annotation VLM Backbone | MMLU Score | S2I R@5 | S2I R@20 | S2T R@5 | S2T R@20 | S2IT R@5 | S2IT R@20 |
|---|---|---|---|---|---|---|---|
| Gemma 4 E4B | 69.4 | 19.6 | 43.3 | 7.7 | 19.8 | 20.2 | 44.4 |
| Qwen2.5-VL-72B | 71.2 | 20.0 | 42.1 | 8.1 | 20.5 | 19.7 | 41.9 |
| Qwen3-VL-8B | 71.6 | 19.1 | 39.9 | 6.5 | 22.0 | 20.3 | 40.0 |
| Qwen3-VL-30B (Ours) | 77.8 | 23.0 | 43.6 | 11.9 | 29.2 | 23.3 | 42.5 |
Key Findings¶
- Crucial Role of Contextual and Inferred Semantics: Advancing from object labels (O) to contextual descriptions (Ctx) and VLM-inferred smells (S) consistently boosts retrieval performance across all metrics. On the S2T task, adding inferred ambient cues increases R@5 from 8.0 to 11.9 (+48.7% relative gain), confirming that VLM inferences provide essential out-of-frame olfactory supervision.
- Verification via Second-View Ground Truth: To prove that VLM inferences reflect genuine physical cues rather than hallucinations, a held-out second view (View 2) of the identical recording environment was inspected by an independent VLM judge. Over 98.6% of inferences were judged plausible, and 32.2% specifically predicted physical objects that were entirely hidden or out-of-frame in View 1 but clearly present in View 2.
- Compositional Generalization via Disentanglement: On the zero-shot recombination task, recombining object and context latents to synthesize new smell fingerprints achieves an R@20 of 13.7%, substantially outperforming raw sensor signal concatenation (9.8%), demonstrating effective semantic separation of complex odor mixtures.
Highlights & Insights¶
- Language as an Anchor for Weakly Observable Signals: Instead of treating VLMs merely as caption generators, SCENT utilizes them as commonsense physics reasoning engines to bridge the spatial mismatch between ambient odor diffusion and limited camera apertures.
- Native Tri-Modal Co-Training vs. Indirect Proxy Bridges: By pairing the smell encoder with dedicated projection heads for vision and text simultaneously, the model avoids error accumulation inherent in cascade/bridge architectures and achieves mutual semantic regularization.
- Discriminative Asymmetry Between Vision and Olfaction: Analysis reveals that while image cosine similarities remain largely flat across retrieval ranks, olfactory embeddings exhibit sharp decay, indicating that chemical sensors provide fine-grained discriminative fingerprints that visual features alone cannot capture.
Limitations & Future Work¶
- Sensor Drift and Environmental Sensitivities: Metal oxide semiconductor (MOX) e-nose arrays are susceptible to baseline drift over time and cross-sensitivity to fluctuations in humidity and ambient temperature, requiring frequent recalibration.
- Scarcity of Multimodal Olfactory Datasets: With benchmarks like NYS containing only ~7,000 paired in-the-wild samples, olfactory learning remains several orders of magnitude smaller than image-text foundation corpora, constraining the training of end-to-end foundation models.
- Future Directions: Future research could incorporate continuous fluid-dynamic diffusion models to trace 3D odor plumes, alongside integrating active mobile robotics for autonomous chemical source localization.
Related Work & Insights¶
- vs NYS Baseline [44]: NYS relies on direct vision-olfactory contrastive alignment, leaving the network vulnerable to out-of-frame odor confusion; SCENT introduces language guidance to supply contextual supervision and enables interpretable odor disentanglement.
- vs Universal Multimodal Frameworks (ImageBind [23], LanguageBind [73]): General star-topology alignment frameworks force all sensory modalities into a single rigid anchor; SCENT designs dual projection pathways and latent decomposition with physical reconstruction loss, tailored to the additive mixture nature of chemical signals.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering use of VLM commonsense reasoning as a bridge for olfactory representation learning and latent mixture disentanglement.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across S2I, S2T, and S2IT tasks, backed by held-out view confirmation and zero-shot recombination protocols.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly coherent narrative with clear physical insights into the nature of olfactory diffusion and sensor-vision alignment.
- Value: ⭐⭐⭐⭐⭐ Paves an important foundation for expanding embodied AI perception into the chemical and olfactory domain.