TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches¶

Conference: ECCV2026
Paper: ECCV Paper
Project: TouchAnything
Area: 3D Vision
Keywords: tactile sensing, 3D reconstruction, diffusion prior, signed distance field, robot contact

TL;DR¶

TouchAnything combines local geometry from sparse touches with a class-level shape prior from a frozen 2D diffusion model, reconstructing 3D objects through coarse-to-fine optimization without task-specific diffusion training and achieving lower EMD across six simulated categories with 20 touches.

Background & Motivation¶

Under severe occlusion or poor lighting, cameras may not reveal a reliable object outline, whereas contact directly indicates where a small surface region lies and how it is oriented. The camera inside GelSight observes the deformed elastomer rather than an external photograph of the object, so image-based touch does not eliminate the missing-global-geometry problem. A single touch covers only a small patch; even when these patches are accurately aligned in one coordinate frame, many different shapes may connect them. A neural signed distance field can represent a continuous surface, but it does not automatically know whether two contact regions should be connected by a bottle body, a camera body, or another structure. TouchSDF addresses this ambiguity with a dataset-specific shape prior, while Touch2Shape trains a touch-conditioned 3D diffusion model, both requiring specialized shape or tactile training resources.

This paper explores a different source of prior knowledge: can the shape information learned by large text-to-image models support tactile reconstruction? The challenge is not to generate an asset that resembles a camera, but to make generative preferences respect the particular camera that the robot has touched. Class-level text such as "a camera" or "a bottle" does not specify instance dimensions, lens geometry, or handle placement. Measurements should therefore constrain contacted surfaces, while the prior helps interpret untouched regions; local measurements cannot guarantee the correctness of all unobserved structures when these signals conflict. The paper assumes known contact poses, isolating sparse-observation shape completion from the additional problem of jointly estimating robot localization and object geometry.

The transfer becomes possible by converting touch into depth and normals, then using differentiable rendering to present the evolving 3D shape to a visual diffusion model. Stable Diffusion does not need to interpret raw tactile images, and a new diffusion network need not be trained for each category. However, no task-specific diffusion training does not mean no learning or instantaneous inference: the tactile geometry frontend is learned, and each target object still requires substantial geometry optimization. Core Idea: anchor real surfaces with tactile depth, normals, and contact-space constraints, guide unobserved geometry with a frozen 2D diffusion prior, and reconstruct the object through two-stage optimization from a neural SDF to an explicit tetrahedral grid.

Method¶

Overall Architecture¶

Inputs are GelSight images, contact poses supplied by robot kinematics, and weak class-level text; the output is a complete 3D object surface. The method proceeds through Tactile Geometry Bridging, Touch-Constrained Coarse Reconstruction, and High-Resolution Refinement, with the diffusion prior participating in the latter two without updating its weights. Intermediate representations are masked local depth and normal observations, an SDF represented by a multiresolution hash grid and MLP, and finally an explicit DMTet tetrahedral representation. Contacted regions are constrained by geometric residuals, while untouched regions primarily receive semantic and shape guidance through normal maps rendered from random viewpoints.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Tactile images and known poses"] --> Bridge["Tactile Geometry Bridging"]
    Bridge --> Coarse["Touch-Constrained<br/>Coarse Reconstruction"]
    Coarse --> Fine["High-Resolution Refinement"]
    Bridge -->|Depth and normal supervision| Fine
    Prior["Class-level text and frozen<br/>2D diffusion prior"] -->|SDS after warm-up| Coarse
    Prior -->|SDS guidance| Fine
    Fine --> Output["Complete 3D surface"]

Key Designs¶

1. Tactile Geometry Bridging: turn local contacts into renderable observations

Deformation of a real GelSight elastomer changes its internal illumination pattern, and an existing learning-based method predicts surface gradients from pixel RGB values and coordinates. This real-data frontend is a three-layer MLP; its predicted gradients are integrated into depth by a fast Poisson solver, and thresholding the depth produces a contact mask. The simulation frontend differs: a multi-head U-Net jointly predicts depth and contact masks, trained on 20k simulated GelSight images generated from 78 YCB objects. Real and simulated inputs therefore do not share a single untrained sensor-inversion procedure, although downstream reconstruction uses the same type of geometric interface. These frontends recover contact geometry from sensor signals rather than directly predicting complete category-level shapes from local touch.

For each contact, the system places a virtual camera 2.0 cm behind the contact patch and organizes the tactile estimates into depth, normal, and mask observations. This virtual camera is not an additional external RGB camera: it reformulates tactile measurements as ray observations compatible with existing visual reconstruction methods. Only pixels inside the contact mask create supervised rays, so untouched regions are not treated as observed surfaces. Known sensor poses align different patches in a common coordinate frame, allowing their local constraints to act on one global surface. The conversion reuses differentiable rendering machinery, but also passes tactile calibration, depth-estimation, and robot-kinematics errors into the reconstruction constraints.

2. Touch-Constrained Coarse Reconstruction: establish contact geometry before completing unseen shape

The coarse stage uses Neuralangelo-style multiresolution 3D hash features and an MLP to map spatial coordinates to truncated signed distances. The surface is the SDF zero level set; the following retains only this representation relation, which is explicit in the paper:

\[ \mathcal{S}=\{\mathbf{x}\in\mathbb{R}^{3}:f_{\theta}(\mathbf{x})=0\}. \]

Along each contact ray, volume rendering predicts surface depth and normals, which are compared against tactile observations. Depth constrains surface position, whereas normals constrain local orientation, providing complementary geometric information. The method also samples near the observed depth to supervise local SDF values directly and applies a free-space constraint between the virtual camera and the region before the surface. The near-surface supervision band and free-space sampling interval are defined below, where \(d(r)\) is the observed ray depth and \(\delta\) is the truncation distance:

\[ \mathcal{S}^{\mathrm{sdf}}_{r}=[d(r)-\delta,d(r)+\delta],\qquad \mathcal{S}^{\mathrm{fs}}_{r}=[0,d(r)-\delta]. \]

The free-space term suppresses spurious object geometry in the region in front of the contact that should be occupied by the sensor; the paper describes penalizing predicted SDF values smaller than \(\delta\) there. This is not a free-space observation of the entire scene and does not establish that every untouched location is empty. The four tactile supervision terms are combined and estimated using a batch of contact rays per iteration, while Eikonal regularization additionally constrains the distance field. These terms still leave large uncertain regions, so the method then introduces frozen Stable Diffusion instead of asking sparse touch alone to determine global completion.

Diffusion guidance uses random viewpoints on a sphere around the object to render normal maps from the current SDF, rather than supplying external RGB photographs of the target. The normal map is encoded into the Stable Diffusion VAE latent space, perturbed with noise, and processed by the noise predictor conditioned on class-level text. Score distillation sampling (SDS) uses the discrepancy between predicted and injected noise to propagate an update direction to geometry through the encoder and differentiable rendering path. Normal rendering directs guidance toward surface orientation and geometric detail, reducing the need to entangle texture optimization with shape reconstruction. Class-level text narrows the range of plausible shapes, while contact depth and normals constrain that category preference to the current instance. The diffusion model supplies a prior rather than additional measured data, so unseen regions can still acquire structures that match the text but not the real instance.

3. High-Resolution Refinement: change representation to reduce rendering cost

The coarse stage repeatedly queries a neural field along rays, making higher rendering resolutions expensive; consequently, SDS uses \(64\times64\) normal images. The second stage replaces this geometry with DMTet, storing SDF values at tetrahedral-grid vertices and learning a spatial offset for each vertex. Initialization queries the optimized first-stage SDF, transferring a coarse shape already constrained by contact rather than randomly generating a new object. Differentiable marching tetrahedra extracts the surface, and differentiable rasterization generates observations without per-ray MLP queries. This enables \(512\times512\) normal rendering, allowing diffusion guidance to inspect geometric details that are difficult to resolve in the coarse stage.

The fine stage retains tactile depth, tactile normals, and SDS supervision, now computed through mesh rasterization. The paper does not transfer every first-stage loss into the second stage, so near-surface SDF and free-space terms should not be assumed to remain explicit refinement objectives. An additional normal-consistency term penalizes angular deviations between adjacent vertex normals to encourage locally smooth surfaces. The purpose is not to increase tactile coverage but to optimize finer structure with a more economical representation under the same contact constraints and semantic prior. The authors also note that the two stages yield similar results for simulated objects with limited detail, so refinement benefits should not be assumed equally large for every object.

A Worked Example¶

Consider the paper's camera experiment: the robot acquires 20 contacts and uses "a camera" without knowing the exact instance geometry of the complete lens and body. Each contact becomes local depth, normals, and pose-aligned virtual camera rays, informing the optimizer which surfaces are supported by measurements. Warm-up establishes initial geometry using tactile supervision alone; after SDS is introduced, untouched regions receive the global shape preference associated with the camera category. The coarse SDF is then converted to DMTet and optimized with higher-resolution normal images, producing an extractable 3D surface. This explains the processing sequence rather than reporting additional per-step errors; the paper does not provide a complete numerical log for this instance trajectory. When text is changed to "an airplane," Figure 7 shows structures induced by incorrect semantics in unseen regions, demonstrating that touch directly corrects geometry only where it provides coverage.

Loss & Training¶

Within the coarse stage, steps 0–1000 use tactile supervision alone for warm-up, and steps 1000–7000 combine tactile supervision with SDS guidance. This is a schedule within the coarse stage and should not be confused with the subsequent switch to DMTet as the second geometry stage. Diffusion-guidance rendering uses batch size 8 at \(64\times64\) in the coarse stage and batch size 4 at \(512\times512\) in the fine stage. Stable Diffusion remains frozen; optimization first updates neural-field parameters and then DMTet vertex SDF values and offsets, rather than fine-tuning diffusion on touch. The paper reports approximately 1 hour for the coarse stage and 40 minutes for the fine stage per object on one NVIDIA A100, making this an offline reconstruction pipeline rather than online tactile control. Equations (1)–(5) in the cached text have visibly corrupted operators and subscripts, so this note does not guess their exact loss expressions, sign conventions, or weights. The zero level set and sampling intervals above come from readable textual definitions; other objectives are explained through the prose, while exact implementation requires checking the original typesetting or code.

Key Experimental Results¶

Main Results¶

Simulation uses 280 ShapeNetCore.V2 objects across bottles, bowls, cameras, guitars, jars, and mugs, with the same object subset provided by the TouchSDF authors. Each object receives 20 touches; Taxim produces \(320\times240\) GelSight images for a sensing area of \(2.0\ \mathrm{cm}\times1.5\ \mathrm{cm}\). Contact locations are uniformly sampled, contact orientation follows the local surface normal, and prompts are short category names. The following reproduces Table 1 on page 11: lower Earth Mover's Distance (EMD) is better, measuring geometric discrepancy between reconstructed and ground-truth surface samples. The main text does not fully specify EMD sampling, normalization, or units, so these values should not be interpreted as absolute errors in meters or millimeters.

Category	TouchSDF: EMD	Touch2Shape: EMD	TouchAnything: EMD
Bottle	0.047 ± 0.024	0.041	0.035 ± 0.020
Bowl	0.048 ± 0.017	0.049	0.039 ± 0.010
Camera	0.092 ± 0.043	0.056	0.047 ± 0.025
Guitar	0.155 ± 0.087	0.064	0.060 ± 0.025
Jar	0.071 ± 0.038	0.055	0.053 ± 0.023
Mug	0.066 ± 0.018	0.049	0.044 ± 0.008

The table preserves the source's "±" notation, but the readable main text does not define its statistical meaning; Touch2Shape lacks corresponding dispersion values, precluding significance claims from these entries alone. Real-robot experiments use a UR5e and GelSight Mini with 14 objects: 6 from YCB, 3 ShapeNet-derived 3D-printed objects, and 5 household objects. Objects are fixed to the table, and the main experiment uses 20 touches distributed uniformly over each surface; Figure 4 presents qualitative reconstructions rather than a unified real-world EMD results table.

Ablation Study¶

The following summarizes Figures 6–7 and their discussion on pages 13–15; it is an analysis table, not a second original numerical table, and the available full-text cache lacks the supplementary real-object EMD values. Figure 7 compares 20, 40, and all available touches; the full counts are 111 for the drill, 99 for the bottle, 121 for the camera, and 53 for the can.

Change or condition	Objects and contact conditions	Observation reported by the paper	Source
Remove the diffusion prior	Drill, bottle, camera, and can; tactile observations only	Reconstruction degrades substantially; local patches do not determine the complete shape	Figure 6, page 13
Use class-level text	Four real objects, 20 touches	Untouched regions can be plausibly completed	Figure 7, page 14 and discussion
Use detailed instance descriptions	Four real objects, 20, 40, and all touches	Not necessarily better than class-level text; precise geometry is difficult to specify in language	Figure 7, pages 14–15
Use empty text	Four real objects, 20, 40, and all touches	The diffusion prior can still connect aligned patches or complete partial cylindrical structures	Figure 7, pages 14–15
Use incorrect text "an airplane"	Four real objects, 20, 40, and all touches	Unobserved regions acquire structures matching incorrect semantics	Figure 7, pages 14–15

Key Findings¶

EMD is lower than both baselines in all six categories, but gains vary: cameras achieve 0.047 versus 0.056, while jars achieve 0.053 versus 0.055, so no fixed improvement ratio applies. A category name and 20 touches can yield plausible shapes, but this does not establish that 20 contacts uniquely identify arbitrary objects. Empty text is not equivalent to removing the prior: the former still runs diffusion, whereas the latter removes global generative guidance entirely, answering a different ablation question. The touch-count study provides visualizations and a supplementary reference, but the current cache does not support objectwise numerical trends or a claim that more touches must always improve results.

Highlights & Insights¶

Cross-modal transfer occurs through a geometric interface: touch becomes depth and normals, and rendering connects these constraints to a visual diffusion model, avoiding a new dedicated mapping from raw touch to full shape.
The division between prior and observation is explicit: physical measurements constrain contacted regions, and generative preferences guide untouched regions, also revealing where semantic hallucinations are most likely.
Representation switching serves supervision resolution: DMTet is not merely a mesh-output mechanism but makes high-resolution diffusion guidance computationally affordable.
A reader interpretation is that this design could transfer to other sparse geometric sensors, provided they reliably produce pose-associated depth or normal constraints; changing the sensor name alone is insufficient.

Limitations & Future Work¶

The authors propose reducing text dependence and replacing passive collection with informative active touch selection; the current method is not an implemented active exploration strategy.
Known contact positions, fixed objects, and efforts to cover surfaces uniformly simplify the experiments, preventing direct extrapolation to moving objects, in-hand manipulation, or substantial pose drift.
Nonuniform contact experiments and real-object quantitative results are referred to supplementary material, but only the main paper is available in the assigned cache, preventing verification of their full values and details.
Approximately 1 hour plus 40 minutes of optimization per object restricts online use; no diffusion training is not evidence of inexpensive inference.
Incorrect categories can induce unobserved structures, and the main paper does not provide reconstruction confidence or risk calibration; a reader suggestion is to use uncertain regions to choose subsequent active contacts.
Previously unseen instances do not establish absence from Stable Diffusion's pretraining distribution, and six simulated categories plus 14 real objects do not demonstrate reliable generalization to arbitrary categories.

TouchSDF uses a DeepSDF dataset prior and TacTip input, whereas this paper uses GelSight and a frozen visual diffusion prior; both sensor and prior differ, so the comparison is not a fully controlled single-module replacement.
Touch2Shape trains touch-conditioned diffusion and incorporates active exploration; this paper emphasizes prior transfer without task-specific diffusion training, while contacts remain passively collected.
DreamFusion / Fantasia3D / Magic3D contribute important ideas in SDS, normal-based geometry guidance, and coarse-to-fine optimization, respectively; this paper's key extension constrains those generative tools for real tactile-conditioned instance reconstruction.
Tactile DreamFusion primarily targets tactile-assisted asset generation, whereas this paper emphasizes consistency with known contact locations and local geometry; sharing touch as an input does not erase the distinction between generation and reconstruction.
The primary category remains 3D Vision because complete geometric reconstruction defines both the output and quantitative evaluation, while robotic touch supplies measurements; Robotics & Embodied AI is an appropriate cross-topic rather than a necessary category move.

Rating¶

Novelty: 4/5. Transfers a general 2D diffusion prior to sparse tactile-conditioned instance reconstruction, with innovation primarily in the problem formulation and constraint design.
Experimental Thoroughness: 3/5. Simulation comparisons and real demonstrations complement each other, but main-text real-world quantification, error robustness, and statistical reporting remain limited.
Writing Quality: 4/5. The geometric bridge and two-stage pipeline are clear, although some exact losses and supplementary experiments cannot be independently checked from the current text cache.
Value: 4/5. Demonstrates a route to reducing dedicated tactile-shape training through existing visual priors, with a substantial gap remaining to reliable real-time robotics.