Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation¶

Conference: ECCV 2026
arXiv: 2606.29941
Code: https://shengqi77.github.io/Seeing-Touch-from-Motion/ (Project Page)
Area: Robotics / Embodied AI
Keywords: Visuo-tactile policy, optical tactile sensing, tactile motion correlation, contact state perception, Mixture-of-Transformers

TL;DR¶

Addressing the long-standing issue where optical tactile sensors struggle to distinguish fine-grained contact states, this paper reveals that the dot product of "transient motion" and "cumulative motion" can explicitly differentiate states such as "making contact / stable contact / releasing contact." Based on this, a Tactile Motion Correlation (TMC) representation is proposed. Using a Mixture-of-Transformers, TMC is integrated as an independent modality alongside vision, raw touch, and proprioception, achieving higher success rates across four real-world contact-rich manipulation tasks.

Background & Motivation¶

Contact-rich manipulation tasks (such as precision insertion, wiping, and polishing) are crucial for robots performing fine-grained work, and their success depends heavily on the accurate perception of subtle physical interactions. These tasks naturally rely on the complementarity of global vision and local touch: vision is responsible for planning coarse trajectories, while touch provides local feedback during contact. Optical tactile sensors are widely adopted due to their high resolution and ability to perceive textures and multi-dimensional forces—internally, they use a camera to monitor the deformation of an elastic gel surface to indirectly infer tactile cues, often with markers printed on the surface to amplify gel motion. However, the exact problem lies here: extracting "fine-grained contact states" from these visual measurements remains an unresolved challenge. Existing methods either directly use raw tactile images or cumulative motion fields (optical flow between the current frame and the initial contact-free frame). The former primarily captures appearance changes, making physically opposite states like "making contact" and "releasing contact" look highly similar in raw images. The latter only reflects the direction and magnitude of the overall gel deformation, which similarly exhibits almost identical patterns between "making contact / stable contact / releasing contact." Consequently, distinct contact states correspond to indistinguishable representations, leaving the robot unable to discern the current contact stage.

The core contradiction arising from this perceptual ambiguity is: contact-rich manipulation requires fine-grained temporal slicing of the contact process (e.g., when initial contact occurs, when deformation stabilizes, when rebound begins), yet both existing representations are merely "static deformation snapshots at a single moment," lacking the temporal dimension of how deformation evolves over time. Thus, they cannot separate states that are physically opposite but have similar snapshots. At the same time, even with proper tactile representations, how to effectively fuse local touch and global vision remains the other half of the puzzle. Existing fusion methods either simply concatenate or apply attention at the feature level (ignoring modality uniqueness and easily allowing one dominant modality to suppress another), or train separate policies at the decision level and combine their outputs (respecting modality differences but cutting off cross-modal interactions, failing to yield true complementarity).

This paper's point of entry is to tap into the overlooked dynamic priors in tactile motion. The authors observe that if one examines both "transient motion" (optical flow between adjacent frames) and "cumulative motion" (optical flow relative to the initial contact-free frame), their directional correlation can clearly separate states indistinguishable by static snapshots: they are co-directional during contact initiation, opposing during contact release, and the transient motion virtually disappears during stable contact. Core Idea: Use the pixel-wise dot product of transient and cumulative motion as a tactile representation (Tactile Motion Correlation, TMC). The sign of the dot product directly encodes "pressing in / rebounding / no deformation," while its magnitude is positively correlated with the contact force. TMC is treated as an independent modality and fused with vision, raw touch, and proprioception using a Mixture-of-Transformers for a "unified yet modality-aware" fusion, balancing cross-modal complementarity and individual modality distinctiveness.

Method¶

Overall Architecture¶

The method, named ViTacMotor, aims to solve "how to enable visuo-tactile policies to perceive fine-grained contact states and properly fuse vision and touch." The architecture is split into two major parts: the first half is the purely geometric tactile representation construction—at each step, transient and cumulative motions are computed using optical flow from the initial contact-free frame \(T_0\), current frame \(T_t\), and previous frame \(T_{t-1}\). Their pixel-wise dot product yields the Tactile Motion Correlation (TMC) map. This step is parameter-free and translates raw tactile videos into a motion correlation map that explicitly expresses contact states. The second half is the policy network—where four-channel observations (third-person + wrist RGB images, raw tactile maps, TMC correlation maps, and proprioception) are encoded into embeddings via pre-trained encoders and fed into a Mixture-of-Transformers (MoT) for fusion. Finally, a Transformer decoder regression-decodes an action chunk. The entire policy is trained from scratch using a \(\beta\)-VAE objective.

The key point is: TMC is treated as a distinct, independent modality from "raw tactile appearance" inside the MoT—since one represents appearance and the other dynamics, they are physically heterogeneous, and forcing them into the same modality leads to mutual interference. MoT provides each modality with its own dedicated QKV projection, output projection, FFN, and LayerNorm to preserve modality identity, while allowing all modalities to share a single global self-attention to achieve cross-modal interaction.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Tactile Video<br/>T0 / T(t-1) / Tt"] --> B["1. Tactile Motion Correlation (TMC)<br/>Transient × Cumulative Pixel-wise Dot Product"]
    V["Visual RGB<br/>Wrist + Third-person View"] --> ENC["Per-modality Pre-trained Encoder<br/>yielding 4 embedding streams"]
    A --> ENC
    B --> ENC
    P["Proprioception"] --> ENC
    ENC --> C["2. MoT Unified Modality-Aware Fusion<br/>Modality-specific QKV/FFN + Shared Self-Attention"]
    C --> D["Transformer Decoder<br/>Autoregressively predicts action chunk"]

Key Designs¶

1. Tactile Motion Correlation (TMC): Disentangling Indistinguishable Contact States via the Sign of the Transient × Cumulative Dot Product

The pain point is straightforward: raw images only capture appearance, and cumulative motion only measures "total deformation relative to the initial state." Both are static snapshots at a single moment, rendering opposing processes like "pressing in" and "rebounding" nearly identical. The breakthrough point of the authors is the introduction of transient motion—the optical flow between adjacent frames, which captures the instantaneous direction of gel movement. Computing the pixel-wise dot product of transient motion \(\mathbf{M}^{\text{tran}}_t=\text{Flow}(T_{t-1},T_t)\) and cumulative motion \(\mathbf{M}^{\text{cumu}}_t=\text{Flow}(T_0,T_t)\) yields the motion correlation:

\[Corr_t(x) = \mathbf{M}^{\text{tran}}_t(x)\cdot \mathbf{M}^{\text{cumu}}_t(x),\quad x\in\Omega\]

The physical meaning of this dot product is elegant and clear: during pressing in, the instantaneous deformation direction aligns with the cumulative deformation, yielding a positive dot product; during releasing, the gel rebounds, causing opposing directions and resulting in a negative dot product; during stable contact or no contact, transient motion approaches zero, leading to a near-zero dot product. Thus, the sign alone is sufficient to distinguish "making / stable / releasing / no contact"—which is impossible with raw appearance or cumulative motion alone. Even better, sliding contact exhibits a spatial split of "one side positive, one side negative": the negative region corresponds to gel rebounding and the positive region to gel pressing in, with the transition direction indicating the sliding direction. Furthermore, empirical tests show that the magnitude of the dot product is positively correlated with the applied contact force, effectively providing a free force perception signal. Since it only depends on optical flow and is decoupled from specific sensor architectures, TMC is fundamentally sensor-agnostic, which is validated using both sparse and dense marker sensors. Compared to other representation-learning methods that implicitly encode transient/cumulative changes into a latent space, TMC's dot product is an explicit, physically interpretable scalar field, offering superior interpretability and plug-and-play capability.

2. Integrating TMC as an Independent Modality in MoT Fusion: Disentangling Tactile Motion from Appearance and Enabling Complementarity Without Dominance

The pain point is the dilemma of existing fusion: feature-level concatenation erases modality-specific traits and lets dominant modalities suppress weaker ones; decision-level fusion trains separate networks and cuts off cross-modal interactions. The authors borrow the Mixture-of-Transformers (MoT) architecture for a "unified yet modality-aware" fusion. A critical decision is treating TMC and raw tactile appearance as two distinct modalities—due to their physically heterogeneous characteristics (one being motion dynamics, the other being surface appearance), merging them directly would dilute crucial information. Thus, the set of modalities is defined as four-fold: \(\{I(\text{vision}),\,T(\text{raw tactile}),\,Corr(\text{TMC}),\,p(\text{proprioception})\}\). Each modality \(m\) utilizes modality-specific projection matrices to calculate its own Q/K/V, ensuring that unique statistical characteristics are preserved. Subsequently, tokens from all modalities are concatenated and passed through a single shared global self-attention layer, enabling vision to query the current contact state and TMC to query the visual location of objects, achieving genuine cross-modal interaction:

\[\mathbf{A}_t=\text{softmax}\!\left(\frac{\mathbf{Q}_t\mathbf{K}_t^\top}{\sqrt{d}}\right)\mathbf{V}_t\]

After attention, each modality goes through its dedicated output projection \(W_O^{(m)}\), FFN, and LayerNorm (with residual connections). In other words, "shared attention for interaction, modality-specific pre- and post-processing to maintain identity." This design enjoys the best of both worlds: cross-modal complementarity (via shared self-attention) and modality specificity (via modality-specific parameters), truly binding global vision and local touch together. The authors claim this is the first work to utilize the MoT architecture for visuo-tactile fusion.

Loss & Training¶

The policy network is trained from scratch (individually per task) with a \(\beta\)-VAE objective to model the stochasticity in human demonstrations. The total loss consists of action prediction loss and KL regularization:

\[\mathcal{L}=\mathcal{L}_{\text{pred}}+\beta\,\mathcal{L}_{\text{KL}}\]

Where the prediction loss uses the L1 norm between the predicted action chunk \(\hat{a}_{t:t+k}\) and ground-truth sequence \(a_{t:t+k}\) (\(\mathcal{L}_{\text{pred}}=\lVert \hat{a}_{t:t+k}-a_{t:t+k}\rVert_1\)) to pursue precise actions; the KL term pulls the posterior \(q_\phi(z\mid a_{t:t+k},o_t)\) toward a standard normal prior, where \(\beta\) (set to 10) controls the information bottleneck intensity. Optical flow is computed using the efficient DIS algorithm (inverse search + multi-scale pyramid + variational refinement), achieving up to 600Hz on a single CPU core, sufficient for real-time operations. Each task is trained on a single RTX 3090 Ti for about 5 hours, with a batch size of 16, a learning rate of \(2\times10^{-4}\) with cosine decay, a chunk size of 100, a 4-layer encoder, and a 7-layer decoder.

Key Experimental Results¶

Main Results¶

Evaluated on physical robots (two 6-DoF AgileX manipulators + dense/sparse optical tactile sensors) across four contact-rich tasks, with average success rates reported over 15 runs per method per task. The tasks are divided into two categories: those requiring accurate contact state information (tube sorting, bulb insertion) and those requiring fine-grained force control (whiteboard erasing, pencil sharpening). The table below shows the complete task success rates (%):

Task	ACT	DP	ACT+T	Policy Consensus	TactileACT	ViTacMotor
Tube Sorting	53.3	40.0	60.0	53.3	60.0	73.3
Bulb Insertion	13.3	6.7	26.7	26.7	40.0	40.0
Whiteboard Erasing	46.7	40.0	60.0	66.7	73.3	86.7
Pencil Sharpening	33.3	33.3	40.0	33.3	46.7	60.0

Vision-only policies (ACT/DP) perform worst in scenarios with severe occlusions (e.g., gripper holding a test tube and blocking the third-person view) or when contact force perception is crucial (e.g., whiteboard erasing). Adding raw tactile maps (ACT+T) brings improvements but remains unstable, as fine-grained contact states are difficult to directly read from raw appearance. ViTacMotor shows particularly significant improvements back-to-back in the two tasks requiring force control (e.g., whiteboard erasing outperforms the second-best TactileACT by 13.4 percentage points), since the TMC magnitude implicitly encodes the contact force.

Ablation Study¶

Configuration	Whiteboard Erasing	Tube Sorting	Description
Base (Vision + Proprioception only)	46.7	53.3	Baseline
Base + TMC	73.3	66.7	Adds TMC representation
Base + MoT-Fusion	66.7	60.0	Adds MoT fusion
Full (TMC + MoT-Fusion)	86.7	73.3	Full model

Both components individually bring substantial improvements, with their combination yielding the best performance. Additionally, two targeted analyses were conducted: (1) Replacing raw appearance/cumulative motion representations with TMC in existing policies like ACT/DP: on whiteboard erasing, TMC consistently outperforms others (ACT: Raw 60.0 / Cumu 53.3 / TMC 73.3; DP: Raw 53.3 / Cumu 40.0 / TMC 66.7), proving TMC is plug-and-play and universally effective; (2) Replacing MoT fusion with common alternative fusion strategies: average success rates are Concatenation 70.0 / Cross Attention 66.7 / MoT 80.0.

Key Findings¶

TMC and MoT fusion contribute comparably; removing either results in a noticeable performance drop. The value of TMC lies in simultaneously providing both "fine-grained contact state discrimination" and "implicit force perception" signals.
TMC is sensor-agnostic: it demonstrates discriminative power for contact states on both sparse and dense marker sensors, primarily because it prioritizes modeling underlying gel deformation rather than raw appearance.
Interestingly, Cross-Attention fusion performs slightly worse than simple Concatenation (66.7 vs. 70.0), indicating that merely piling attention layers is inferior to structured designs like MoT, which features "shared attention + modality-specific parameters."
Robustness: The policy remains successful under time-varying spatially uniform/non-uniform illumination perturbations, as well as object variations like changing erasers or replacing test tubes with cylinders.

Highlights & Insights¶

Introducing the "temporal dimension" into the tactile representation is the most elegant move: Existing works are trapped in "static deformation snapshots." By simply adding transient motion and performing a dot product with cumulative motion, the authors cleanly separate opposing yet visually similar states using the sign—resolving with an almost zero-parameter, physically interpretable operator a perceptual ambiguity that learning-based methods have struggled to crack.
The positive correlation between the dot product magnitude and contact force serves as an unexpected bonus: Intended only for contact state classification, the same scalar field seamlessly encodes contact force. This directly benefits force control tasks (whiteboard erasing, pencil sharpening), achieving two goals with one stone.
The insight that "TMC and raw tactile appearance should be treated as distinct modalities" is crucial: Motion dynamics and surface appearance are physically heterogeneous. Feeding them separately as two distinct modalities into the MoT avoids mutual dilution—this suggests that in multimodal fusion, "how to slice modalities" is a critical design variable in itself, not a natural given.
TMC is plug-and-play and transferable: Being a purely geometric transformation of raw tactile videos, it can directly replace the tactile input channels within any visuo-tactile policy (as validated on ACT/DP in the paper), incurring minimal transfer overhead.

Limitations & Future Work¶

The authors acknowledge that generalization remains limited under extreme visual perturbations or large variations in object poses. Failure cases are detailed in the appendix, such as failing to insert the tube into the hole despite successful grasping under large pose shifts coupled with severe visual interference.
Self-identified limitations: All experiments involve "training from scratch individually per task" with no unified cross-task/cross-object policy. Generalization is mainly demonstrated through in-task lighting/object variations, possessing limited scale and diversity. The absolute success rates (e.g., 40% for bulb insertion, 60% for pencil sharpening) are still far from practical viability, highlighting the inherent difficulty of these tasks.
TMC relies on optical flow quality. When markers are sparse or gel deformation is extremely small, the transient motion signal is weak, potentially destabilizing the dot product. While the paper employs the fast DIS optical flow, it lacks direct analysis of the impact of optical flow noise on the representation.
Directions for improvement: Integrating TMC into larger VLA frameworks or World Models to achieve cross-embodied generalization, or explicitly utilizing the "contact state sign" as an intermediate supervision signal instead of merely feeding it implicitly to the policy.

vs. Raw Tactile Image Methods (e.g., TactileALOHA / FreeTacMan): These works directly feed GelSight-like raw images. In contrast, this paper shifts to motion correlation; unlike raw images that contain only appearance and fail to separate opposing contact states, TMC explicitly encodes them using the sign of the dot product, offering stronger discriminative capabilities.
vs. Cumulative Motion Methods (e.g., Bogert using Helmholtz-Hodge decomposition / Xue using PCA dimensionality reduction): These studies utilize only the direction and magnitude of cumulative deformation. This paper supplements them with transient motion, compensating for the temporal delay in cumulative motion and distinguishing stable/releasing contact states which cumulative motion alone blurs.
vs. Implicit Tactile Representation Learning (e.g., Sparsh / AnyTouch): These works implicitly encode transient and cumulative changes into a latent space. Conversely, TMC's dot product yields an explicit, physically interpretable scalar field, offering better interpretability and plug-and-play qualities, though learning-based approaches might harbor greater potential for general-purpose representations.
vs. Feature-level/Decision-level Fusion (e.g., TactileACT concatenation / Policy Consensus decision-level): The former ignores modality identity and the latter severs cross-modal interaction. This work employs MoT's "shared self-attention + modality-specific parameters" to simultaneously achieve complementarity and specificity, rendering it the first to introduce MoT to visuo-tactile fusion.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Utilizing the pixel-wise dot product of transient × cumulative motion to explicitly describe contact states is a concise, physically interpretable, and previously unexploited observation that hits the nail on the head.
Experimental Thoroughness: ⭐⭐⭐⭐ Comprises four real-world tasks, five baselines, and extensive ablations (including plug-and-play validation of TMC, fusion comparison, cross-sensor testing, and robustness analyses). However, it lacks large-scale simulation evaluations, and the absolute success rates remain somewhat low.
Writing Quality: ⭐⭐⭐⭐⭐ The motivation figures (Figs.1/3) exceptionally clarify the discriminative differences among the three representations. The methodology is naturally described, making it highly clear "why the dot product works."
Value: ⭐⭐⭐⭐ Being plug-and-play and sensor-agnostic, TMC holds direct reusable value for the wider visuo-tactile manipulation community. Constrained by individual training for each task, its downstream generalization in real-world deployment remains to be fully verified.