Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/Hunter-Will/VVM-Tuning
Area: Multimodal VLM
Keywords: Large Multimodal Models, Cross-modal Generalization, Fabricated Modality Synthesis, Instruction Tuning, Zero-shot Adaptation

TL;DR

To overcome the heavy reliance of Large Multimodal Models (LMMs) on in-modality data for non-RGB vision, this work introduces VVM-Tuning, a training framework that synthesizes appearance-varied images from RGB data to disentangle invariant semantics from photographic features, achieving zero-shot generalization to unseen visual modalities via textual modality contexts.

Background & Motivation

Large Multimodal Models (LMMs) have demonstrated remarkable visual and language reasoning capabilities on standard RGB inputs. However, their ability to generalize to non-RGB visual modalities—such as infrared/thermal, depth, and X-ray—remains severely bounded and largely unexplored. Current efforts to extend LMM capabilities beyond RGB typically rely on scaling up training corpora with domain-specific paired non-RGB data. This data-hungry paradigm introduces a profound generalization gap: real-world imaging mechanisms and physical sensor modalities are virtually limitless, and modern models collapse when confronted with novel, unseen modalities not encountered during training.

The core tension lies in the over-alignment of existing visual-language pre-training pipelines. Digital images from distinct visual modalities are essentially different physical signal samplings of the exact same real-world scene; hence, their underlying semantics remain invariant, while modality differences manifest primarily as permutations of basic visual elements (such as pseudo-color mappings and altered histogram profiles). In standard RGB training, models deeply entangle high-level semantic identity with specific RGB appearance distributions. Consequently, models cannot separate modality-invariant scene semantics from modality-variant photographic representations, necessitating expensive, specialized re-alignment whenever a new sensor modality is introduced.

Drawing inspiration from human visual cognition—where humans can effortlessly parse object contours and geometry in unfamiliar modality imagery without domain-specific training—this paper decomposes multimodal capability into modality-unaware perception and modality-aware understanding. Core idea: synthesize diverse fabricated modalities from RGB scenes via spline-based histogram randomized perturbations and pseudo-colormaps to train a disentangled perception baseline, and leverage in-context textual modality prompts to dynamically map appearance elements to physical properties, enabling zero-shot adaptation to unseen modalities without in-modality training.

Method

Overall Architecture

VVM-Tuning decouples pan-modal generalization into two complementary phases: modality-unaware perception and modality-aware understanding. In the perception phase, the model is trained on fabricated images to distill invariant semantics while maintaining strict fidelity to low-level photographic elements (colors, edges, coordinates). In the understanding phase, textual modality contexts are introduced into prompts to instruct the language decoder to associate specific visual appearances with physical attributes (e.g., mapping temperature or depth gradients). The framework is optimized end-to-end using a single-stage supervised fine-tuning (SFT) objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RGB Scene Images"] --> B["Fabricated Modality Synthesis<br/>Spline Histogram Perturbations + 22 Colormaps"]
    B --> C["Modality-unaware Perception<br/>Disentangled Semantic & Photographic VQA"]
    B --> D["Modality-aware Understanding<br/>Modality Context Prompts + Physical Mapping VQA"]
    C --> E["Single-Stage Instruction Tuning<br/>Cross-Entropy SFT on 50k Mixed Instruction Samples"]
    D --> E
    E --> F["Zero-shot Cross-Modal Inference<br/>Unseen Modality Images + Modality Context"]

Key Designs

1. Fabricated Modality Synthesis: Generating Diverse Non-RGB Substitutes from RGB Scenes To address the severe scarcity and narrow coverage of non-RGB paired annotations, this design develops an automated, lightweight data synthesis pipeline. On the visual front, standard RGB images are converted to grayscale and modulated via cubic spline interpolation to randomize and invert pixel histograms, simulating the dynamic contrast ranges of diverse imaging sensors. The grayscale representations are subsequently colorized using 22 scientific visualization colormaps and augmented with noise injection, blurring, and smoothing to mimic sensor degradations. On the language front, human-designed templates bind colormap intervals with semantic objects when segmentation masks are available; alternatively, leading LMMs generate contextual descriptions of color-to-object distributions via in-context prompting, creating rich textual modality contexts.

2. Modality-unaware Perception: Disentangling Invariant Semantics from Photographic Attributes To eliminate the spurious correlation between high-level semantics and specific RGB color palettes, this design enforces a dual-task training scheme. The first component comprises semantic perception tasks, which compel the model to identify identical object classes, positional relations, and topology across paired RGB and synthesized images, ignoring drastic chromatic shifts. The second component comprises photographic perception tasks, which instruct the model to faithfully report exact pixel colors, edges, and visual markers without semantic hallucination. Both components generate detailed image descriptions and diverse VQA question-answer pairs, establishing a robust, modality-invariant visual foundation.

3. Modality-aware Understanding: Zero-Shot Physical Property Remapping via In-Context Prompts To handle the unique physical interpretations of unfamiliar modalities during inference (e.g., red representing higher temperatures in thermography or cooler tones indicating proximity in depth maps), this design injects textual modality contexts directly into input prompts. During training, the model is exposed to varied fabricated physical rules, training the LLM backbone to dynamically bind low-level photographic tokens to textual physical explanations and conduct commonsense reasoning. At test time, without updating model weights, injecting a concise natural language description of an unseen sensor's physical mechanism achieves zero-shot understanding on real-world infrared, depth, or novel sensor data.

Loss & Training

The framework follows the standard visual instruction tuning paradigm, optimized with cross-entropy loss over target output tokens:

\[\mathcal{L}_{SFT} = -\sum_{i=1}^{n} \log P(y_i \mid X, y_1, y_2, \dots, y_{i-1})\]

where \(X\) represents the input visual tokens concatenated with the textual modality context prompt, and \(y_i\) denotes the target response sequence. Training is executed on a single server equipped with 4 NVIDIA A800 GPUs using the AdamW optimizer with a learning rate of \(1\times 10^{-5}\) and a global batch size of 32. The full training corpus comprises approximately 50k samples trained for 1 single epoch, with 12k original RGB visual instruction instances retained to prevent catastrophic forgetting of base RGB capabilities.

Key Experimental Results

Main Results

The authors evaluate performance on the newly assembled VVM-Bench, which spans 6 distinct imaging modalities (3 real modalities: Thermal, Depth, X-Ray; 3 synthetic modalities: Optical Flow, Synthesized 1, Synthesized 2) across 11 VQA tasks divided into Perception and Understanding subsets:

Model	Size	Perception Avg. Acc. (Base / Ours)	Perception Avg. \(\Delta\)	Understanding Avg. Acc. (Base / Ours)	Understanding Avg. \(\Delta\)
LLaVA-1.5	7B	60.4% / 74.1%	+13.6%	79.4% / 84.9%	+5.6%
Qwen-2.5-VL	3B	72.2% / 81.9%	+9.7%	86.7% / 90.6%	+3.8%
Qwen-2.5-VL	7B	78.1% / 84.1%	+6.0%	86.2% / 90.6%	+4.5%
Qwen-3-VL	4B	77.6% / 82.8%	+5.3%	87.9% / 89.9%	+2.0%
Qwen-3-VL	8B	77.6% / 84.1%	+6.5%	88.3% / 91.5%	+3.2%

Ablation Study

Ablation experiments conducted on Qwen-2.5-VL 7B demonstrate the individual and joint contributions of Modality-unaware Perception (P) and Modality-aware Understanding (U) tasks across all benchmark modalities:

Training Tasks (Perception / Understanding)	Eval Subset	Thermal	Depth	X-Ray	Optical Flow	Synthesized 1	Synthesized 2	Overall Average
Baseline (Untuned)	P	74.5%	73.3%	75.7%	77.0%	87.1%	81.1%	78.1%
Baseline (Untuned)	U	86.4%	83.4%	84.1%	84.6%	88.8%	89.8%	86.2%
Perception Only	P	75.3%	75.6%	79.6%	81.7%	90.7%	87.6%	81.7% (+3.6%)
Perception Only	U	86.7%	82.5%	86.3%	92.9%	90.3%	91.3%	88.3% (+2.1%)
Understanding Only	P	76.8%	74.1%	75.4%	74.1%	85.8%	81.7%	78.0% (-0.1%)
Understanding Only	U	87.6%	85.4%	85.1%	92.1%	88.8%	90.3%	88.2% (+2.0%)
Full Model (P + U)	P	79.8%	79.2%	79.6%	88.4%	91.2%	86.4%	84.1% (+6.0%)
Full Model (P + U)	U	88.5%	90.2%	85.9%	93.8%	91.9%	93.5%	90.6% (+4.5%)

Furthermore, removing Modality Context (M.C.) prompts during inference causes an average accuracy drop of 32.75% on Depth (e.g., LLaVA-1.5 drops from 80.3% to 40.7%) and 15.37% on Thermal, confirming that explicit text-guided remapping is essential for physical comprehension of novel sensor data.

Key Findings

• Synthetic Training Transfers to Physical Reality: Despite training exclusively on artificially synthesized colormaps, tested models demonstrate an average improvement of over 6.0% across real-world physical modalities (Thermal, Depth, X-Ray, Optical Flow).
• Perception as a Necessary Prerequisite: Training solely on understanding tasks fails to improve basic perception (78.1% vs 78.0%), whereas establishing a disentangled perception base provides substantial downstream gains for both perception and higher-level reasoning.
• Cross-Architecture Scalability: Improvements are observed across diverse model families and scales, ranging from early architectures (LLaVA-1.5 +13.6% on perception) to cutting-edge models (Qwen-3-VL 8B +6.5% perception and +3.2% understanding).

Highlights & Insights

• Conceptual Paradigm Shift: Instead of collecting costly non-RGB sensor datasets, the method treats all visual modalities as diverse samplings of identical physical entities, unlocking zero-shot pan-modal generalization via procedural appearance perturbation.
• Two-Track Disentangled Alignment: Separating invariant scene semantics from photographic primitives mirrors human visual reasoning, preventing LMMs from collapsing when encountering inverted or false-color distributions.
• Extremely Efficient Data Pipeline: The synthesis pipeline requires only standard open-source RGB images and basic scientific colormaps, generating 50k high-quality multimodal training samples on a commodity GPU in hours.

Limitations & Future Work

• Residual Synthetic Gap: The authors note that procedural colormap synthesis cannot fully replicate complex, non-linear sensor physical artifacts, such as radar clutter, backscattering, or multi-path interference in specialized hardware.
• Scope of Geometric Transformations: The current synthesis primarily focuses on photometric and histogram shifts; future extensions could incorporate physically based rendering (PBR) to simulate geometric distortions, perspective transformations, and tomography slice reconstructions.

Related Work & Insights

• vs ImageBind / PandaGPT: These approaches require paired cross-modal data to bind modalities into a shared latent space during pre-training, limiting applicability to pre-determined modalities; VVM-Tuning operates without any paired non-RGB training data and generalizes zero-shot to completely unseen sensors.
• vs Infrared-LLaVA / SpatialBot / EarthGPT: Prior domain-specific models require in-modality fine-tuning tailored to single sensors (e.g., infrared or depth), which sacrifices generalizability; VVM-Tuning trains a single unified model that adapts dynamically to arbitrary modalities via in-context prompts.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering concept of zero-shot generalization to unseen visual modalities using purely synthetic RGB perturbations and in-context physical mapping.
• Experimental Thoroughness: ⭐⭐⭐⭐☆ Rigorous evaluation across 5 foundational LMMs, 6 diverse modalities, and comprehensive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear theoretical rationale, compelling cognitive motivation, and elegant structural organization.
• Value: ⭐⭐⭐⭐⭐ Provides a highly cost-effective and scalable blueprint for extending general-purpose LMMs into specialized scientific and multi-spectral computer vision domains.