See & Sniff: Learning Visuo-Olfactory Representations¶

Conference: ECCV2026
arXiv: 2606.27307
Project Page: https://mm.kaist.ac.kr/projects/SeeandSniff
Area: Multimodal Perception
Keywords: Visuo-olfactory representations, multimodal learning, self-supervised learning, odor localization, synthetic pairing

TL;DR¶

Using the key insight that "odor identity remains invariant to visual transformations within semantic categories," this paper extends the smell-only SmellNet dataset into a paired visuo-olfactory dataset, SmellNet-V. It designs a self-supervised dual-stream framework named See & Sniff based on dense local contrastive alignment. The proposed method significantly outperforms smell-only baselines in odor classification, cross-modal retrieval, and a newly introduced pixel-level odor localization task.

Background & Motivation¶

Multimodal AI systems have successfully integrated vision with language, audio, and even touch. However, olfaction—a sensory modality playing a central role in human and animal perception—remains largely overlooked in AI research. A natural and tight coupling exists between human vision and olfaction: seeing coffee beans evokes the anticipation of a roasted aroma, while smelling coffee brings the image of dark brown beans to mind. This cross-modal association indicates a systematic, learnable intrinsic link between visual appearance and odor identity. However, building computational models to align visuo-olfactory representations faces a fundamental data bottleneck: large-scale paired visuo-olfactory datasets previously did not exist.

Existing odor datasets mainly rely on two technical routes: predicting odor perception based on molecular structures (such as DeepNose and GNN-based methods) or collecting real odor signals via portable electronic noses. The recently released SmellNet is the first large-scale real-world odor dataset, featuring around 180,000 temporal samples collected using multi-channel gas sensors across 50 food ingredients and natural objects. However, SmellNet only contains odor data without corresponding visual images. To learn joint visuo-olfactory representations, a method must be developed to associate these two modalities. Concurrent works have attempted to collect naturally paired visuo-olfactory data in the real world using specialized handheld sensing devices, but this approach is highly expensive, device-dependent, and difficult to scale.

The core insight of this paper is that odor identity remains highly invariant to visual transformations within a semantic category. A small red apple and a large red apple emit nearly identical volatile organic compounds, which humans perceive as the same odor category ("apple"). This implies that diverse visual forms of a food category (different angles, lighting conditions, sizes, and arrangements) can share a semantic space with the same odor embedding. Leveraging this invariance, the smell-only SmellNet can be synthetically paired with open-world web images, bypassing the bottleneck of paired data scarcity at scale. Core Idea: Leverage the invariance of odors to visual transformations within semantic categories to synthetically pair the smell-only SmellNet with semantically aligned web images. This constructs the first large-scale visuo-olfactory training dataset, SmellNet-V. A joint representation is then learned via dense local contrastive alignment, enabling the model to perform cross-modal retrieval, odor classification, and spontaneously generate odor saliency maps for spatial-level odor source localization.

Method¶

Overall Architecture¶

The overall training pipeline consists of two main stages. The first stage leverages the insight of "odor invariance to visual transformations within semantic categories" to segment the long-term odor sensor data of 50 ingredients in SmellNet into short "sniff units." It then uses an LLM to generate diverse scene query prompts for each food category to collect web images, which are filtered through a three-stage pipeline and randomly paired by category to construct the SmellNet-V dataset. The second stage adopts a dual-stream encoder architecture—the visual branch utilizes a frozen DINOv3-Small backbone with a lightweight aligner, while the olfaction branch employs a 4-layer Transformer (ScentFormer) with a lightweight aligner—projecting both modalities into a shared embedding space. During training, the dense similarity map between the olfactory features and spatial visual feature maps is computed, and its maximum value is used as the cross-modal similarity score to drive the symmetric InfoNCE contrastive loss. Once trained, the similarity map naturally acts as an odor saliency map, which can be directly applied to odor source localization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["SmellNet<br/>50 food categories, 6 channels<br/>~180k temporal samples"] --> B["Segment into sniff units<br/>W=50, stride 25<br/>Odor temporal difference"]
    C["LLM-generated diverse<br/>scene query prompts"] --> D["Web image collection"]
    D --> E["Three-stage filtering<br/>CLIP pos/neg prompt validation<br/>Watermark detection + manual verification"]
    E --> F["Random pairing by category<br/>Resampled every epoch"]
    B --> F
    F --> G["Olfactory encoder<br/>ScentFormer<br/>+ 2-layer MLP aligner"]
    F --> H["Visual encoder<br/>Frozen DINOv3<br/>+ LayerNorm<br/>+ 1x1 Conv aligner"]
    G --> I["Odor feature vector<br/>f_o ∈ ℝ^C<br/>Temporal average pooling"]
    H --> J["Visual feature map<br/>f_v ∈ ℝ^{C×H×W}<br/>Preserving spatial structure"]
    I --> K["Similarity map M[h,w]<br/>= f_o · f_v[h,w]<br/>→ max(M) as score"]
    J --> K
    K --> L["InfoNCE contrastive loss<br/>Symmetric bidirectional optimization"]
    K --> M["Odor saliency map<br/>→ Odor source localization"]
    L --> N["Joint representation<br/>Odor classification / Cross-modal retrieval"]

Key Designs¶

1. SmellNet-V: Semantic Invariance-Driven Synthetic Pairing Strategy

The fundamental challenge of constructing a utility-paired visuo-olfactory dataset lies in the need for specialized hardware to simultaneously capture odor signals and aligned visual frames, which is expensive and hard to scale. The key to bypassing this bottleneck is leveraging the natural invariance of odors to visual transformations within semantic categories. Different visual appearances of the same ingredient (e.g., apples of different sizes, angles, and lighting) emit almost identical volatile organic compounds. Thus, as long as the odor sample and the image belong to the same ingredient category, they form a valid cross-modal training pair.

The concrete implementation consists of three steps. First, the 6-channel gas sensor temporal data of each ingredient in SmellNet (originally recorded for 10 minutes per category across 6 different days) is segmented into short, fixed-length windows of \(W=50\), termed "sniff units." This simulates the transient sniffing perception in biological olfaction. The stride is set to 25, meaning adjacent sniff units overlap by \(50\%\) to increase the data volume. First-order temporal differencing is also applied to emphasize changes in the sensor signals. Second, an LLM is utilized to generate diverse, realistic scene query phrases for each ingredient (e.g., "a whole apple on a wooden table," "apples in a supermarket produce aisle," "apples hanging on a tree") to harvest web images. These images enter a three-stage filtering pipeline: CLIP validates category consistency, photo-realism, and ingredient freshness using positive and negative prompts; an off-the-shelf watermark detector removes heavily watermarked images; and a final rapid manual QC is performed. Third, each sniff unit is paired with a randomly selected image from the same category to form a training sample. To prevent the model from memorizing fixed pairs, images are randomly re-paired in each training epoch, exposing each sniff unit to different visual contexts.

2. Dense Local Alignment: From Global Semantics to Fine-grained Spatial Correspondence

Standard cross-modal contrastive learning typically extracts a single global vector (such as the CLS token or average-pooled features) from each modality and computes their similarity. While sufficient for retrieval, this paper aims to equip the model with fine-grained spatial understanding—not only identifying if an image and an odor match, but also locating which region of the image is emitting the odor. To achieve this, a local alignment method based on a dense similarity map is developed.

The visual branch encodes an image using a frozen DINOv3-Small backbone into a spatial feature map \(f_v \in \mathbb{R}^{C \times H \times W}\) to preserve spatial dimensions. The olfactory branch averages the temporal features of the sniff unit over the time dimension to obtain a normalized \(\bar{f}_o \in \mathbb{R}^{C}\), aligning with the visual channels. Then, the dense similarity map \(M \in \mathbb{R}^{H \times W}\) is computed, where the value at each spatial location \([h, w]\) is the dot product of the pooled olfactory feature vector and the visual feature vector at that location: \(M[h, w] = \bar{f}_o \cdot f_v[h, w]\). The final cross-modal similarity score is defined as the maximum value across \(M\), measuring how well the odor signal matches the most relevant region of the image. This max operator serves two key purposes: during training, it acts as the similarity score inside the contrastive loss, forcing the model to learn precise local matching to achieve high similarity; during inference, \(M\) itself can be directly used as an odor saliency map without any extra heads. The visual branch only appends a channel-wise LayerNorm and a \(1 \times 1\) convolution as a lightweight aligner to preserve the pre-trained representation quality, while the olfactory branch uses a two-layer MLP with residual connections.

3. Odor Localization: From Cross-Modal Matching to Spatial Odor Source Recognition

In addition to conventional odor classification and cross-modal retrieval, this paper extends joint visuo-olfactory representation learning to spatial localization: identifying the regions emitting the given odor in an image. This task does not require training an extra localization head—upscaling the similarity map \(M\) to the input image size naturally yields an odor saliency map, where high-response regions highlight the most plausible physical source of the odor. To benchmark this capability, the authors construct the SmellNet-V-Source dataset, using the SAM interactive annotation tool on top of the SmellNet-V test set to generate pixel-level segmentation masks for the ingredient regions in each paired image.

Experiments show that See & Sniff significantly outperforms the SmellNet baseline in odor-to-odor retrieval, raising R@1 from \(52.54\%\) to \(69.44\%\). This indicates that visual supervision not only teaches the model to align cross-modal data but also significantly enhances the intrinsic structure of the odor embedding space. Consequently, the embedding space becomes far more discriminative even when only odor signals are present at inference time. Visual bias analyses also validate the source of localization capability: while DINOv3 attention maps only capture general objectness cues (failing to target semantic regions corresponding to specific odors), the saliency maps of See & Sniff accurately pinpoint the exact ingredient regions matching the input odor. This proves that dense local alignment successfully learns cross-modal spatial correspondences.

Loss & Training¶

The training objective is a symmetric InfoNCE contrastive loss. For a batch of sniff units \(o_i\) and images \(v_i\), the similarity is computed as \(\max(M)\), with other images in the batch acting as negative samples. The loss is computed symmetrically in both directions (odor-to-image and image-to-odor). The visual backbone DINOv3-Small is frozen, with only the lightweight aligners and the olfactory encoder being trained. The default sniff unit window size is set to \(W=50\), with a first-order difference lag of \(p=25\) across the 6 sensor channels. The model is trained on an RTX A5000 with a batch size of 64.

Key Experimental Results¶

Main Results¶

Task	Metric	SmellNet Baseline	Ours (See & Sniff)	Gain
Odor Classification (\(W=50\))	Acc.	50.6	57.71	+7.1
Odor Classification (\(W=50\))	F1	49.5	56.68	+7.2
Odor Classification (\(W=100\))	Acc.	57.9	63.75	+5.9
Odor Classification (\(W=100\))	F1	56.0	62.66	+6.7
Odor-to-Visual Retrieval	R@1	-	56.14	-
Visual-to-Odor Retrieval	R@1	-	63.20	-
Odor-to-Odor Retrieval	R@1	52.54	69.44	+16.9

Ablation Study¶

Configuration	Odor Classification Acc. (\(W=50\))	Description
SmellNet (Transformer)	50.6	Odor-only baseline, no visual signals
Global-CLIP (CLIP-Large Vision + Global Contrast)	53.19	Global alignment, strong visual backbone
Ours-Global	53.74	Our architecture + global pooling alignment
Ours-Local	54.94	Dense local alignment, without lightweight aligner
See & Sniff (Ours)	57.71	Dense local alignment + lightweight aligner

Key Findings¶

Visual supervision significantly improves the quality of olfactory representations: Even when only odor signals are present during inference, See & Sniff outperforms the odor-only baseline in odor classification by over 7 percentage points. This indicates that cross-modal training shapes a more structured odor embedding space.
Dense local alignment consistently outperforms global alignment: It excels across all classification and retrieval metrics. Notably, the R@1 for odor-to-odor retrieval improves by 16.9 percentage points, indicating that local comparisons force the model to build finer cross-modal correspondences, which in turn enhances single-modality representations.
All ingredient families benefit consistently: Fruits, nuts, and vegetables show the most substantial improvements (due to distinct object-level geometric structures), while herbs receive the smallest gains (owing to fine leaf textures and high visual similarity). This suggests that the utility of visual signals is positively correlated with the visual distinctiveness of the ingredients.

Highlights & Insights¶

The synthetic pairing strategy fundamentally overcomes the hardware bottleneck in multimodal data collection: Rather than designing expensive recording devices, this method leverages odor invariance to pair existing odor data with massive web images. It is simple and highly scalable—any labeled odor dataset can generate visuo-olfactory pairs using this pipeline.
The dense similarity map accomplishes two tasks with a single design: During training, it acts as the fine-grained similarity score in contrastive learning to drive model optimization; during inference, it directly serves as the odor saliency map for spatial localization. It is elegant, requiring no extra parameters or computational overhead.
Visual supervision inversely reconstructs the olfactory feature space: The 16.9 percentage point R@1 boost in odor-to-odor retrieval shows that daily cross-modal training does more than align modalities—more importantly, it reconstructs the structure and discriminability of the single-modality (odor) feature space itself. This is an intriguing and counter-intuitive finding.

Limitations & Future Work¶

Synthetic pairing relies on the "category-level alignment" assumption, which fails to capture real-world cross-modal differences that occur within the same category due to variety, state, or freshness. Naturally paired, real-world captured data would theoretically be stronger, making the two approaches complementary.
Currently, the dataset only covers 50 ingredient categories, rendering the scenarios highly constrained. Scaling to more diverse scenes like perfumes, environmental odors, and chemical gases requires additional data engineering.
The segmentation masks for the odor localization benchmark are derived via ingredient category labels rather than being independently annotated by sensor signal encoders in a real odor mixture scene, leaving a gap between this benchmark and practical, real-world odor localization.

vs SmellNet (ScentFormer): SmellNet focuses purely on odor representation learning. This paper introduces visual supervision on top of it, proving that vision positively enhances olfactory representations.
vs Contemporary Work (Ozguroglu et al.): That work collects naturally paired visuo-olfactory data using specialized sensing devices with a global contrastive alignment architecture. This paper diverges by employing synthetic pairing and dense local alignment, presenting a complementary research paradigm.

Rating¶

Novelty: ⭐⭐⭐⭐ [Pioneers the research direction of visuo-olfactory representation learning; the synthetic pairing strategy is clever and scalable to other labeled sensor datasets]
Experimental Thoroughness: ⭐⭐⭐⭐ [Comprehensive evaluation across three tasks, along with thorough ablation study and qualitative analysis; however, the localization benchmark labeling carries synthetic artifacts]
Writing Quality: ⭐⭐⭐⭐⭐ [Progresses logically from motivation to methods and experiments; the philosophy of data construction is clearly articulated and highly convincing]
Value: ⭐⭐⭐⭐ [Fills a gap in multimodal perception by addressing olfaction; the framework is simple, efficient, and highly inspiring]