Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs¶

Conference: ECCV2026
arXiv: 2607.00302
Code: https://github.com/xxx
Area: Multimodal VLM / Tactile Perception
Keywords: Tactile Alignment, Small Multimodal Models, Catastrophic Forgetting, Parameter Sparseization, Mask Fine-tuning

TL;DR¶

This paper proposes Splash, a mask-isolated tactile alignment framework. By utilizing a Wanda-style importance scoring metric, the LLM parameters of a small MLLM are partitioned into "dormant" and "critical" subspaces. Only the dormant subspace is updated to encode tactile capabilities, while the frozen critical subspace serves as a stable anchor to preserve original vision-language reasoning.

Background & Motivation¶

Touch is a core channel for robots to perceive the physical world — visual observation alone struggles to determine whether a surface is smooth or rough, soft or hard, but a gentle press of a finger reveals these properties. Recently, researchers have attempted to endow multimodal large language models (MLLMs) with tactile perception capabilities, aligning tactile signals to the vision-language representation space via visuo-tactile-language (VTL) contrastive learning. While these efforts have shown encouraging progress in discriminative tasks like classification and retrieval, they are far from sufficient for complex operational decisions that require high-level semantic reasoning and instruction following. MLLMs, with world knowledge encoded in their LLM backbones, are naturally suited for open-set semantic reasoning of tactile signals.

However, a key contradiction lies in deployment: to run on resource-constrained edge robots, MLLMs must be sufficiently small (under 3B parameters), but small models have highly constrained parameter budgets. Shoving a completely new modality like touch into them inevitably triggers a "zero-sum competition" with existing vision-language knowledge — to learn tactile attribute descriptions, the model inadvertently overwrites features learned during visual pre-training. A typical example is Qwen2.5-VL-3B, which originally identifies objects accurately but frequently suffers from visual hallucinations after tactile fine-tuning, misclassifying metal as smooth or fabric as metal. This represents catastrophic forgetting. Existing VTL methods (such as TVL and UniTouch) adopt a two-stage pipeline (first aligning the tactile encoder, then fine-tuning the LLM using PEFT methods like LoRA). Although this eases the parameter burden to some extent, the low-rank delta (θ+Δθ) of LoRA still perturbs the entire parameter manifold, failing to eradicate the forgetting problem.

The Core Idea of this paper is that instead of forcing the model to grow new parameters for touch, it is better to wake up the already existing but underutilized "dormant" parameters inside the model, employing them for tactile encoding while keeping the critical vision-language parameters completely frozen and untouched. To this end, the authors propose Splash, a tactile alignment framework based on parameter importance masking. Using a calibration set of 128 samples to calculate the relative vision-language importance of each weight, the LLM parameter space is divided into a frozen critical subspace and a trainable dormant subspace. During training, the tactile front-end and dormant parameters are updated simultaneously, while critical parameters receive zero gradients. This approach requires no external adapter modules (like LoRA), retains the original model structure, and incurs zero extra overhead during inference.

Method¶

Overall Architecture¶

The overall pipeline of Splash consists of two steps: offline mask construction and single-stage training. First, using a small batch of vision-language calibration data (128 CC3M image-text pairs), the relative vision-language importance score of each weight in the LLM part of a pre-trained small MLLM (such as Qwen2.5-VL-3B) is calculated layer-by-layer. Weights below a threshold of s% are marked as "dormant" (trainable), and the rest as "critical" (frozen). Then, in single-stage training, both the tactile front-end (ViT-Tiny + 2-layer MLP projection) and the dormant parameters in the LLM are updated simultaneously. Critical parameters receive zero gradients, aligning tactile description generation through autoregressive language modeling loss. The entire training process is single-stage and does not require external modules like LoRA.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB Image +<br/>Tactile Contact Image"] --> B["Visual Front-end<br/>(Frozen)"]
    A --> C["Tactile Front-end<br/>ViT-Tiny + MLP<br/>(Full Parameter Update)"]
    B --> D["Feature Concatenation"]
    C --> D
    D --> E["LLM Backbone"]

    subgraph F["Offline Mask Construction"]
        G["VL Calibration Set<br/>128 Samples CC3M"] --> H["Wanda Importance Score<br/>S = |W| * ||x||₂"]
        H --> I["Generate Binary Mask M<br/>at sparsity=60%"]
    end

    I --> E

    subgraph J["In-Training: Mask Isolation"]
        K["Critical Parameters M=0"] --> L["Frozen<br/>(Zero Gradients)"]
        M["Dormant Parameters M=1"] --> N["Update<br/>Tactile Encoding"]
    end

    E --> O["Tactile Attribute Description"]

Key Designs¶

1. Dormant Subspace Localization: Differentiating "repurposable" parameters via relative vision importance

The core premise of Splash is that a pre-trained LLM contains a large amount of redundant parameters — as shown by the Lottery Ticket Hypothesis, only a small subset of parameters is essential for model functionality. The key question is how to define "important" in the context of tactile alignment: importance should not be relative to language modeling itself, but to the vision-language reasoning ability. The authors leverage the idea of Wanda (a simple and effective LLM pruning method), defining the importance of each weight W_i,j as the weight magnitude multiplied by the ℓ2-norm of its corresponding input activation: S_i,j = |W_i,j| · ||x_i||₂. This product jointly reflects the structural importance of the weight and the activation level of its location. By averaging activation values across samples on a 128-sample VL calibration set, the parameters are sorted by importance score within each linear layer. The bottom s% of weights are designated as dormant parameters (M=1), and the rest as critical parameters (M=0keys). Additionally, the first and last few layers (the first and final transformer blocks of the LLM) are forced to be critical (M=0), as incoming layers handle grounding and outgoing layers handle high-level semantic representation — both are highly sensitive to distribution shifts. This design ensures that the dormant subspace captures genuine redundancy in the model structure rather than overfitting to a specific calibration set. This robust behavior was validated by replacing calibration data with random noise images and nonsensical text, showing highly stable performance.

2. Mask-Isolated Training: Zero-gradient critical parameters as stable anchors

Once mask construction is complete, the training process is exceptionally clean: the tactile front-end ℱ_ψ (ViT-Tiny + MLP projector) undergoes full parameter updates, while updates to the LLM parameters are strictly governed by the binary mask M. Gradients for critical parameters (M=0) are zeroed out via Hadamard product, while dormant parameters (M=1) are updated normally. This differs fundamentally from LoRA: LoRA introduces Δθ in a sidepath, shifting the entire parameter distribution; Splash, on the other hand, performs direct sparse replacement where old values of dormant parameters are overwritten while critical parameters remain completely untouched. Experiments show that this "structural isolation" preserves VL capabilities much better than the "incremental adaptation" of LoRA. Since no external parameters or side pathways are added, the computational graph and parameter count during inference remain identical to the original model, incurring zero extra overhead (P99 latency of 134.55ms, consistent with the original Qwen2.5-VL-3B).

3. Unified Single-stage Training: Bypassing the "pre-alignment" phase of tactile alignment

Existing VTL methods (e.g., TVL, UniTouch) typically require a two-stage training workflow: first, aligning the output of the tactile encoder with the vision or language representation space via contrastive learning, and then freezing the tactile encoder to fine-tune the LLM with LoRA for multimodal reasoning. Benefiting from the parameter isolation design, Splash requires only a single stage: the tactile front-end and the LLM's dormant subspace are trained simultaneously under the supervision of the autoregressive language modeling loss (standard next-token prediction loss). This drastically simplifies the pipeline (converging in just 1 epoch) and eliminates the need for designing complex contrastive loss alignment targets. The tactile front-end can be trained from scratch (using an ImageNet pre-trained ViT-Tiny) rather than initializing from VTL pre-trained weights. Controlled experiments showed that even when the tactile front-end is initialized with the same pre-trained weights as TVL, Splash still significantly outperforms TVL, verifying that the gains are mainly derived from the mask-isolation strategy itself rather than the front-end initialization.

Loss & Training¶

The training objective is the standard autoregressive language modeling loss:

\[ \mathcal{L} = -\sum_{n=1}^{N} \log P(y_n | Y_{<n}, X_R, X_C, X_T) \]

where X_R is the RGB image, X_C is the tactile contact image, and X_T is the text prompt. The tactile front-end parameters ψ undergo full-parameter updating, while LLM parameters θ are updated only in the dormant subspace (M ⊙ ∇θℒ). Key hyperparameters: sparsity ratio s=60%, initial learning rate 2×10^-5 (Splash-3B) with cosine decay and a warmup of 0.05, AdamW optimizer with weight decay of 0.0, and training for 1 epoch (3B) or 2 epochs (1B).

Key Experimental Results¶

Main Results: Tactile Description Generation (LLM Judge Score, Max 10)¶

Dataset	Metric	Splash-3B	Prev. SOTA TVL (Qwen3B)	Gain
SSVTP	GPT-4o Score	5.48	4.98	+0.50
TVL	GPT-4o Score	4.39	4.29	+0.10
TacQuad	GPT-4o Score	4.86	4.22	+0.64
Average	-	4.91	4.50	+0.41

Splash also sweeps the board on objective metrics (F1 / Top-5 Accuracy). It is particularly noteworthy that Splash-1B (built on the InternVL 1B backbone) already outperforms UniTouch's 7B LLaMA-based baseline on average score (5.01 vs. 3.74), showing that the mask isolation strategy maximizes the utilization of small models' limited capacity.

VL Capability Preservation¶

Benchmark	Zero-shot Qwen3B	TVL (Qwen3B)	Splash-3B
MMMU_val	53.1	50.0	55.3
MathVista	62.3	52.8	65.3
MMBench-EN	79.1	76.8	78.0

Splash-3B even surpasses the original Zero-shot model on MMMU_val and MathVista, implying that the mask-isolated adaptation process introduces a pruning-like regularization effect that further stabilizes the pre-trained reasoning manifold. In contrast, TVL's LoRA fine-tuning exhibits significant performance degradation on both benchmarks (MMMU dropping from 53.1→50.0, MathVista dropping from 62.3→52.8).

Ablation Study¶

Configuration	VTL Avg.	MMMU_val	Description
Full-parameter FT (s=100%)	4.73	22.0	Catastrophic collapse of VL capabilities
s=80%	4.85	52.6	Significant degradation
s=60%	4.91	55.3	Best balance
s=40%	4.90	52.6	Slight VL decrease
s=30%	4.79	52.0	Limited tactile learning

Hyperparameter analysis reveals that s=60% is the optimal trade-off point. Leaving too many parameters dormant (s > 60%) leads to a catastrophic loss of VL capabilities (MMMU drops from 53 to 22 when s=100%), while dropping below 40% over-freezes the model, leading to insufficient tactile representation capacity. Furthermore, performance remains almost identical across 64, 128, and 256 calibration samples, demonstrating extreme robustness. Even when replacing calibration data with meaningless noise images and empty text ("calibration-free" setting), the average VTL score drops by a mere 0.03, proving that Wanda captures intrinsic redundant patterns of the architecture rather than overfitting to specific semantic distributions.

Highlights & Insights¶

The intuition of "waking up dormant parameters" is highly natural and elegant: It introduces no external modules, keeps the model structure intact, and adds zero inference latency. Simply using an offline mask, it embeds tactile capabilities into the LLM's internal parameter space. This is much "cleaner" than PEFT solutions like LoRA/Adapter because, even when folded back in at inference, LoRA still slightly perturbs the original weight distribution, whereas Splash's direct replacement offers strict structural isolation.
The modality-level isolation paradigm can be extended to modality-level multi-task learning: This work successfully scales the concept of "cross-task parameter isolation" in pruning from task-level to modality-level (vision → tactile). If this concept holds, one could theoretically allocate a specific dormant subspace for various sensory modalities (such as audio or temperature) within the same small MLLM, training individual front-end encoders to achieve "one-model-multi-modality" without interference.
Single-stage training significantly lowers deployment barriers: Instead of a two-stage pre-alignment and fine-tuning pipeline required by prior VTL methods, the tactile front-end can be directly initialized with generic pre-trained weights. Reaching SOTA performance in just 1 epoch makes it highly suitable for robotics teams with limited hardware resources.

Limitations & Future Work¶

The dormant subspace is statically determined offline. In robotic interaction, however, activation patterns fluctuate dynamically with operating states (e.g., grasp force, contact pose, task phase), making a static mask difficult to adapt to long-horizon tactile reasoning. The paper proposes dynamic adaptive masking as a future direction.
The current experiments are conducted solely on the DIGIT optical tactile sensor. Generalization across different sensor types (e.g., GelSight, TacTip) remains unverified. Fortunately, Splash's tactile front-end design is sensor-agnostic; adapting to a new sensor only requires swapping the ViT encoder.
The sensitivity of the sparsity ratio in 1B-scale models is higher than in 3B models (with 1B models exhibiting more noticeable VL performance drops at s=60%). Smaller models have lower redundancy and a narrower "safety margin", demanding a more sophisticated, non-uniform layer-wise allocation strategy.

vs. TVL / UniTouch: These baselines employ a 7B LLaMA backbone coupled with a two-stage alignment and LoRA fine-tuning. Splash utilizes a 3B sMLLM combined with single-stage training and parameter isolation. Splash achieves comparable or superior tactile reasoning performance with a significantly smaller model footprint without sacrificing visual-language capabilities on any benchmarks.
vs. PEFT like LoRA: LoRA is an incremental adaptation scheme (θ + Δθ), causing uncontrollable disturbances across the entire parameter manifold. Splash is a sparse replacement adaptation (M ⊙ θ), updating only dormant parameters while critical ones remain untouched. When the goal is to safeguard existing multimodal capabilities, isolation strategies clearly outperform incremental adaptations.
vs. Wanda (Pruning): Splash borrows the importance scoring formula from Wanda, but with the exact opposite objective. While Wanda uses it to prune away unimportant parameters, Splash uses it to locate "repurposable" dormant parameters to train. Beneath both pruning and specialization lies a shared hypothesis: models contain extensive redundancy.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Cleverly inverting parameter importance metrics from the pruning domain for the "guided activation" of a new modality. The mask-isolated training idea is simple yet elegant, showing pioneering value in tactile alignment.
Experimental Thoroughness: ⭐⭐⭐⭐ Rigorously evaluated across three VTL benchmarks and five general VL benchmarks, with dense ablations on sparsity, calibration data, and adaptation strategies. Qualitative and quantitative analyses are thoroughly presented, though it lacks cross-sensor generalization and in-the-wild closed-loop robotic deployment.
Writing Quality: ⭐⭐⭐⭐⭐ Highly clear motivation, intuitive methodologies (transitioning smoothly from the Lottery Ticket Hypothesis to dormant subspace localization and mask-isolated training), well-structured contributions, and top-tier table/figure presentation.
Value: ⭐⭐⭐⭐⭐ Holds direct practical value for deploying small MLLMs on embedded edge robotics. The parameter isolation concept could readily generalize to incremental learning for other modalities (such as audio, temperature, pressure), holding strong cross-domain transfer potential.