Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-shot Real-World Deployment

Conference: ECCV 2026
Paper: ECCV 2026
Area: Robotics & Embodied AI
Keywords: visuotactile simulation, online reinforcement learning, sim-to-real transfer, incremental potential contact, multi-GPU parallelism

TL;DR

Tac2Real pairs a frictional PNCG-IPC soft-body solver (implemented in Taichi, pluggable into Isaac Lab / MuJoCo and other engines) that emits 7×9×2 marker displacement fields at 4,465 FPS over 4,096 environments on multiple nodes and GPUs with a four-stage calibration pipeline, TacAlign, that closes the sim-to-real gap; a policy trained entirely in simulation transfers zero-shot to a real Franka, succeeding in 55 of 60 blind peg-insertion trials (91.7%).

Background & Motivation

Vision-based tactile sensors (VBTSs) have become a mainstream modality for contact-rich manipulation. The GelSight family captures high-resolution deformation images or marker displacements of an elastomer, turning low-level contact physics into perceptual signals naturally compatible with vision and learning frameworks: whether a grasp is slipping, whether a peg is aligned with the hole wall, whether contact force is excessive — states that are nearly unobservable from vision alone are explicit in touch. The difficulty appears when such sensors are integrated into data-driven policy training, and especially into online reinforcement learning: the policy must run millions of steps in simulation, and every step must produce tactile readings close enough to the real sensor. Existing tactile simulators sit at two extremes. Penalty-based routes (TACTO, TacSL) approximate only the interpenetration region; they are fast and scale well, but cannot model the elastomer's soft deformation or multi-phase contact dynamics. Among physics-based routes, the Material Point Method family (Tacchi / Tacchi 2.0, Difftactile) reproduces deformation reasonably well yet suffers frequent numerical instability and particle splashing under shear and large deformation, while the Incremental Potential Contact family (TacIPC, Taccel) offers high fidelity without interpenetration or inversion but has essentially no architecture designed for multi-GPU parallelization.

The core tension is therefore explicit: online RL needs each simulation step to be cheap, whereas high-fidelity contact simulation is precisely what makes a solver expensive — and in existing implementations the two are directly opposed. Worse, even an accurate simulator must still cross two kinds of domain gap for zero-shot real-world deployment: structured discrepancies (mismatched robot dynamics, material parameters, contact models) and stochastic ones (unmodeled noise and environmental uncertainty). Most prior work simulates only at the sensor–object (S-O) level and does not place the full robot loop and gap mitigation inside one framework; the few that reach the whole-system level (TacSL, TacFlex, Taccel) are each limited by non-physical approximations, offline imitation learning only, or the absence of a reproducible calibration procedure.

This paper's angle is to acknowledge that opposition and press on it from both sides at once. On the simulation side it does not chase machine-precision convergence but picks a solver whose every iteration is extremely cheap and inherently GPU-friendly, trading accuracy for throughput, then stacks throughput with multiple nodes and GPUs until it reaches the scale online RL requires. On the transfer side it does not assume that "an accurate enough simulator will transfer by itself," but splits the gap explicitly into structured and stochastic parts and calibrates them level by level. Core idea: replace Newton's method with nonlinear conjugate gradient for frictional incremental potential contact (PNCG-IPC) and run it as Taichi GPU kernels across a Ray cluster to obtain fast, high-fidelity marker displacement field simulation; then use four-stage TacAlign calibration (robot control alignment → material parameter identification → task-level contact parameter fine-tuning → domain randomization) to suppress the structured and stochastic gaps separately, enabling zero-shot real-world deployment of tactile policies.

Method

Overall Architecture

Tac2Real consists of two parts: a visuotactile simulator and a sim-to-real calibration pipeline called TacAlign. The simulator runs as a plugin outside the physics engine — the engine advances the rigid-body dynamics of the robot and object as usual, while Tac2Real extracts the relative pose and relative velocity between the "sensor elastomer" and the "grasped object" from that step, feeds them to the PNCG-IPC soft-body solver as boundary conditions, advances the elastomer deformation, and finally interpolates a 7×9×2 marker displacement field from the deformed mesh, concatenating it with the engine's own observation before returning it to the RL agent. Because the interface consumes only relative quantities, it is engine-agnostic: Isaac Lab, Isaac Gym, PyBullet and MuJoCo can all host it. TacAlign, in turn, runs once before training and once before real-world deployment, aligning simulation and reality level by level, from controller gains through elastomer material parameters and task contact parameters to training-time randomization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Physics engine: relative pose and velocity"] --> B["PNCG-IPC frictional contact solve<br/>Taichi GPU kernels"]
    B --> C["Marker displacement field representation<br/>k-NN mapping + weighted interpolation"]
    C --> D["Pluggable interface and multi-node multi-GPU parallelism<br/>Ray cluster dispatches environments"]
    D --> E["Online RL training<br/>PPO / tactile observation"]
    E --> F["Four-stage TacAlign calibration<br/>control → material → task → randomization"]
    F --> G["Zero-shot real-world deployment"]
    F -.->|calibrates gains and sim parameters| B

Key Designs

1. PNCG-IPC: swapping Newton's method for conjugate gradient to make one contact simulation step affordable on GPU

IPC casts elastodynamic contact simulation as a variational minimization under implicit Euler integration; the position at each time step is obtained by minimizing

\[E(\mathbf{x}) = \tfrac{1}{2}(\mathbf{x}-\hat{\mathbf{x}})^\top \mathbf{M}(\mathbf{x}-\hat{\mathbf{x}}) + h^2\Psi(\mathbf{x}) + h^2 B(\mathbf{x}) + h^2 D(\mathbf{x})\]

whose four terms are the inertia potential, the hyperelastic energy, the log-barrier contact potential and the frictional potential. ⚠️ This equation is corrupted by OCR in the cached full text; it is reconstructed here in the standard IPC form consistent with the surrounding context — refer to the original paper and its supplementary material for exact symbols. The key modification is that the original PNCG-IPC handles only frictionless contact, whereas this work adds the frictional potential \(D(\mathbf{x})\) so that friction-transmitted contact states such as grasping and sliding are solved correctly.

What actually makes it runnable is the choice of solver. Standard IPC uses Newton's method with a continuous collision detection (CCD) line search: high per-iteration accuracy, but expensive and hard to parallelize on GPUs. PNCG-IPC instead applies nonlinear conjugate gradient to solve the nonlinear optimization directly, without assembling or factorizing the Hessian matrix; the algorithm needs only gradient evaluations, diagonal Hessian entries and vector–vector dot products — all highly GPU-parallelizable operations. The line search is also made CCD-free by deriving an analytical step-size upper bound, eliminating per-iteration collision detection. This is a deliberate "trade per-iteration accuracy for iteration throughput" design: a conjugate-gradient step is of course less precise than a Newton step, but its low per-iteration cost lets it converge to sufficient accuracy within only tens of iterations. For tactile simulation, visually plausible and physically consistent deformation matters far more than machine-precision convergence, so the trade is highly favorable. The whole solver is written in Taichi, where concise Python compiles to high-performance GPU kernels — the prerequisite for dropping it into arbitrary simulation pipelines as a plugin.

2. Marker displacement fields as the tactile representation: lower-dimensional but more sensitive, and directly attachable to the IPC mesh

The GelSight Mini can output either a 320×240 RGB tactile image or a 9×7 marker displacement field (depending on the gel type), and this paper chooses the latter. The choice rests on a direct collision test: with the sensor mounted on a Franka Panda gripper, the robot grasps a peg and performs four kinds of contact interaction with a socket — stationary, press-down, move-forward and move-backward — while both RGB output and the 2D marker field are recorded. Marker displacement fields vary substantially across the four contact states, whereas the RGB images differ only subtly: the marker field is more sensitive to which contact mode the robot is currently in. Combined with its far lower dimensionality, this means a more regular observation and more efficient learning at large-scale RL.

Matching the discrete real markers to the continuous simulation mesh uses a deliberately simple method: k-nearest neighbors build a mapping between the initial IPC mesh nodes and the markers, and each marker's displacement is then computed by weighted interpolation from the deformed mesh node positions. This brings a second benefit — it bypasses optical rendering entirely. An RGB route would require modeling illumination, gel surface texture and camera imaging, adding uncertainty while inflating observation dimensionality. The "markers + geometric interpolation" shortcut deletes that entire layer of rendering uncertainty.

3. Pluggable interface and multi-node, multi-GPU parallelism: tactile simulation lives outside the engine and speaks only in relative quantities

The obvious way to run online RL with a pair of GelSights in Isaac Lab is to bake tactile simulation into the engine, but that couples the solution to one engine and makes it hard to spread the load across GPUs. Tac2Real inverts this and keeps tactile simulation as an external interface: the RL agent emits an action → the physics engine takes one step → the framework extracts the relative position and rotation between the grasped object and the sensor during that step, along with the derived linear and angular velocities, and hands them to tactile simulation → the resulting marker displacement field, together with the engine's baseline observation, forms the combined observation returned to the agent. Because the interface consumes only relative physical quantities, it can be embedded directly in the environment file layered on top of the engine, for example _get_observations() in Isaac Lab or play_steps_rnn() in rl-games, so switching engines requires no change on the tactile side.

Parallelism comes from a Ray cluster. The framework builds a Ray cluster spanning multiple nodes, each with multiple GPUs, and instantiates one Ray-wrapped tactile simulation class per GPU, each managing tactile computation for a set of environments. During the online rollout this class's simulation function is invoked iteratively, taking relative pose quantities as input and returning marker displacement fields, after which Ray's cross-node distributed communication gathers results from all GPUs. Adding GPUs therefore translates into added throughput, whereas implementations such as Tacchi — which must construct large background grids for many environments on a single GPU — are held back precisely by that overhead.

4. TacAlign: splitting the structured and stochastic domain gaps into four calibration stages

The authors do not assume that an accurate simulator transfers by itself. The gap is explicitly divided into structured discrepancies (mismatched robot dynamics, material parameters, contact models) and stochastic ones (unmodeled noise, environmental uncertainty); the former is suppressed by deterministic calibration and the latter absorbed by randomization, over four stages.

(i) Robot control alignment. Both the simulated and the real Franka use Cartesian impedance control, with the target end-effector force \(F_{\text{targ}} = k_p \ast (p_{\text{targ}}(a) - p_{ee}) - k_d \ast v_{ee}\), where the damping gain is set to \(k_d = 2\sqrt{k_p}\) to ensure critical damping. A natural instinct is to make the simulated and real \(k_p\) as close as possible, but the authors sample 20 sim–real \(k_p\) pairs and plot the trajectory discrepancy, finding that it does not decrease monotonically as \(|k_p^{\text{sim}} - k_p^{\text{real}}|\) shrinks, and shows no significant linear correlation — unmodeled dynamics, actuator delays, friction and contact nonlinearities together make the relationship strongly nonlinear and non-monotonic. Parameter matching is therefore insufficient, and trajectory-level alignment is required: the discrepancy is minimized over six canonical end-effector motions (single-axis translation and rotation along the three axes),

\[\mathcal{D}(k_p^{\text{sim}}, k_p^{\text{real}}) = \frac{1}{T}\sum_{t=1}^{T}\left\| x_t^{\text{sim}} - x_t^{\text{real}} \right\|^2\]

Crucially, neither domain is treated as ground truth: alternating minimization updates both sides' gains, first optimizing the simulation gains with the real gains fixed, then optimizing the real gains with the new simulation gains fixed, iteratively wearing down the controller-induced structured mismatch.

(ii) Baseline IPC calibration. This stage identifies the elastomer material parameters \(\theta = [E, \nu, \rho, \mu]^\top\) (Young's modulus, Poisson's ratio, density, friction coefficient). The apparatus comprises a 6-DOF positioning stage, a GelSight Mini and several 3D-printed indenters; for each indenter, sequences of marker displacement fields are recorded under three deformation modes, and an identical calibration environment is built in PNCG-IPC to simulate the full deformation process, giving the mean squared error

\[\mathcal{L}(\theta) = \frac{1}{KN}\sum_{k=1}^{K}\sum_{i=1}^{N}\left\| u_{k,i}^{\text{sim}}(\theta) - u_{k,i}^{\text{real}} \right\|^2\]

where \(K\) is the number of frames and \(N\) the number of indenters. Since this objective is non-differentiable in the parameters, CMA-ES — a gradient-free evolution strategy — solves for \(\theta^*\). Four indenters (cube, cylinder, moon, triangle) each perform three deformation interactions with the elastomer — pressing 1 mm, sliding 1 mm and rotating 2° — with step increments of 0.1 mm, 0.1 mm and 0.5° respectively.

(iii) Task-based calibration. Baseline calibration covers only the elastomer's own mechanical response, while real contact-rich tasks span a wider interaction range, so the Isaac Lab contact parameters — friction coefficient \(\mu_{\text{isaac}}\) and contact stiffness \(s_{\text{isaac}}\) — are fine-tuned for the specific task. Four representative contact states are chosen (stationary grasping, press-down, forward collision, backward collision); marker displacement field sequences are recorded in both simulation and reality, with the real measurements as the ground-truth reference, and the contact parameters are tuned until the MSE between the closest-matching frames falls below a threshold. The experiments find that \(\mu_{\text{isaac}}\), together with the use of compliant contact in Isaac Lab, plays a critical role in reducing the discrepancy between IPC-simulated and real marker fields.

(iv) Randomization. The residual gap is modeled as parametric and observational uncertainty, and training randomizes physical parameters (controller gains, friction), geometric configurations (object and socket poses) and sensing channels (end-effector pose noise, IPC perturbations). This complements the three deterministic calibration stages: calibration shrinks the structured discrepancy but cannot drive it to zero, and randomization keeps the policy invariant to the remaining low-level dynamics mismatch and contact variation.

Note that the four stages operate on different time scales. The first two depend only on the robot and the sensor and need one-time calibration for identical hardware; the third essentially builds an alignment dataset for a specific task, so a new task requires re-collecting marker field sequences under static and random contact states — the four settings used in the paper are a representative choice.

A Worked Example

Consider one online RL training step: 512 environments spread over 4 nodes with 16 GPUs each, and the agent (rl-games PPO) first emits an incremental Cartesian action. After Isaac Lab advances the rigid-body dynamics by one step, the framework extracts the relative position and rotation between the peg and the fingertip elastomer, plus the linear and angular velocities derived from them; Ray dispatches these to the GPU responsible for that batch of environments, where the Ray-wrapped tactile simulation class invokes PNCG-IPC once: the solver advances the elastomer mesh by an implicit Euler step, converging after a few tens of conjugate-gradient iterations, and the 7×9×2 marker displacement field is read out by k-NN mapping and weighted interpolation. That field, together with the end-effector pose \(p_{ee} \in \mathbb{R}^7\) and the previous action \(a_{t-1}\), forms the observation (containing no object pose and no visual input), which returns to the agent for reward computation and policy update. Object pose remains invisible throughout the loop: the policy can only judge whether the peg has found the hole by the shape into which the marker field is being pressed.

Loss & Training

Policy training uses the off-the-shelf PPO in Isaac Lab (rl-games) with a shared actor–critic architecture, 512 environments distributed across 4 computing nodes with 16 GPUs each for the tactile simulation load, and results averaged over three random seeds. The observation contains only the end-effector pose, a single finger's marker displacement field and the previous action, explicitly excluding object pose and visual data to force the policy to rely on touch. Actions are incremental Cartesian updates executed through the impedance controller. The reward combines a task-progress term from keypoint alignment with sparse bonuses for engagement and insertion, and penalizes excessive contact forces to avoid damaging the sensor. The three calibration objectives (control trajectory discrepancy \(\mathcal{D}\), material parameter MSE \(\mathcal{L}(\theta)\), and task-level displacement field MSE) are optimized by alternating minimization and CMA-ES respectively; they only determine simulation parameters and do not participate in the policy gradient.

Key Experimental Results

Main Results

Policies are trained in simulation on two contact-rich tasks: random-orientation blind peg insertion (the peg's initial orientation is sampled from \([-35°, 35°]\), peg and hole diameters are both 8 mm, and the insertion — especially after the peg enters the socket — must be achieved entirely through active control rather than gravity) and random-orientation nut threading (the nut's initial orientation is likewise sampled from \([-35°, 35°]\), and success is defined as threading the nut 1.5 pitches deep). Simulation success rates are measured over 256 random initial configurations.

Task / peg diameter	Env.	TacAlign level	Tac2Real	TacSL	Tacchi	No Tactile
Peg Insertion (8 mm)	sim	—	0.776	0.789	0.173	0.168
Peg Insertion (12 mm)	sim	—	0.831	—	—	—
Peg Insertion (16 mm)	sim	—	0.857	—	—	—
Nut Threading	sim	—	0.702	0.708	0.152	0.313
Peg Insertion (8 mm)	real	1,2,3,4	0.917	0.150	0.083	0.067
Peg Insertion (12 mm)	real	1,2,3,4	0.933	—	—	—
Peg Insertion (16 mm)	real	1,2,3,4	0.933	—	—	—

The 8 mm real-world row comes from 60 trials: 20 trials each at peg orientations of 0° and ±15°, with 55 successful insertions, i.e. a zero-shot transfer success rate of about 91.7%. The nut threading task was also preliminarily deployed on the real robot with a 58.3% success rate (details in the supplementary material). For reference, the TacSL-based policy reaches only 15% on the real robot and Tacchi is worse still, while the policy without tactile observations manages only 6.7%.

The comparison with existing visuotactile simulators is as follows (following Table 1 of the original paper; the contact-fidelity ranking is with respect to suitability for online tactile RL, considering deformation realism, frictional contact handling and numerical robustness):

Method	Scene coverage	Sim. method / contact fidelity	Tactile representation	Policy learning	Sim-to-real
Tacchi	sensor–object	MPM / medium	RGB	✗	✗
Tacchi 2.0	sensor–object	MPM / medium	Markers	✗	✗
TacIPC	sensor–object	IPC / high	RGB	✗	✗
Difftactile	sensor–object	MPM+FEM / medium	Markers/RGB	✗	✓
Chen et al.	sensor–object	IPC / high	Markers	RL	✓
TacSL	sensor–object–robot	Non-physics penalty / low	Markers/RGB	RL/BC	✓
TacFlex	sensor–object–robot	FEM / high	Markers/RGB	BC	✓
Taccel	sensor–object–robot	IPC / high	Markers/RGB	✗	✓
Tac2Real	sensor–object–robot	IPC / high	Markers	RL	✓

On simulation speed, on a single node with 16 RTX 4090 GPUs at 4,096 environments, Tac2Real reaches 4,465 FPS (pseudo-structured mesh) and 1,665 FPS (unstructured mesh), outperforming Tacchi; TacSL achieves a higher raw FPS, but only because it performs simple SDF queries, at the cost of physical fidelity.

On simulation fidelity, the authors compare Tac2Real, Tacchi and TacSL on a cube-indenter rotation task: Tac2Real and Tacchi both produce deformations consistent with reality, whereas TacSL deviates noticeably because it is a non-physical penalty-based method that models only interpenetration regions. Under large rotational deformation, Tacchi exhibits particle splashing and numerical instability caused by adhesion deficiencies, while Tac2Real robustly handles large rotations and slips and the elastomer reliably recovers its original state after contact.

Ablation Study

The four TacAlign stages are labeled level 1 (control alignment), 2 (baseline IPC calibration), 3 (task-based calibration) and 4 (randomization), and are removed one at a time for real-world 8 mm peg insertion:

TacAlign levels	Tac2Real	TacSL	Tacchi	No Tactile	Note
1,2,3,4 (full)	0.917	0.150	0.083	0.067	all four stages
2,3,4 (w/o control alignment)	0.533	0.033	0.050	0.017	real-world rate halves; baselines drop too
1,2,4 (w/o task-based calibration)	0.250	0.150	0.016	0.067	largest drop
1,2,3 (w/o randomization)	0.767	0.100	0.100	0.017	about 15 points lower

The control alignment search itself can also be quantified (the last row is the gain set finally adopted and fixed for all subsequent calibration and RL training):

\(k_p^{\text{sim}}\)	\(k_p^{\text{real}}\)	Mean translation discrepancy \(\bar{D}_{\text{trans}}\) (mm)	Mean rotation discrepancy \(\bar{D}_{\text{rot}}\) (deg)
(100, 30)	(100, 30)	11.11	2.635
(300, 30)	(500, 40)	7.381	1.091
(600, 50)	(400, 20)	2.521	0.454

Key Findings

• Tied in simulation, far apart on hardware — the most informative pair of numbers in the paper. In simulated 8 mm peg insertion Tac2Real scores 0.776 against TacSL's 0.789 (nut threading: 0.702 vs 0.708), so TacSL is even slightly ahead; the same policies on the real robot become 0.917 against 0.150. Simulation success rate therefore barely predicts transferability — what decides real-world performance is whether the simulated tactile signal aligns with the real marker displacement field, not the task completion rate in simulation.
• Among the four TacAlign stages, task-based calibration (level 3) contributes the most. Removing it drops the real-world success rate from 0.917 to 0.250, the largest fall of any ablation; material-level baseline calibration is necessary but only covers the elastomer's own mechanical response, whereas task-side parameters such as \(\mu_{\text{isaac}}\) and the compliant contact setting decide whether the marker field aligns during contact.
• Control alignment matters at task scale, not at parameter scale. With the initial \(k_p^{\text{sim}} = k_p^{\text{real}} = (100, 30)\) the mean translation discrepancy is 11.11 mm, while the hole diameter is only about 8 mm — the discrepancy exceeds the hole, so the task is impossible. Alternating calibration reduces it to 2.521 mm (and rotation from 2.635° to 0.454°), after which the task becomes feasible. This also validates the paper's central observation that numerical closeness of \(k_p\) does not imply trajectory closeness.
• Touch is essential under blind operation. The observation contains no object pose and no visual information; removing tactile input leaves only a 6.7% real-world success rate, and 0.168 in simulation (nut threading is slightly higher at 0.313 because part of that task's progress is geometrically guided).
• Tacchi's failure is numerical, not methodological. Its real-world success rate of 0.083 is even below the no-tactile baseline, which the authors attribute to the large amount of useless tactile feedback produced by MPM numerical instability — feedback exists, but it points the wrong way, which is more harmful than none.
• Larger holes are easier. The 12 mm and 16 mm pegs beat the 8 mm one in both simulation and reality (0.933 on hardware for both), consistent with relaxed tolerance and confirming that task difficulty is set sensibly.
• Parallel throughput needs careful reading. The 4,465 FPS figure corresponds to a pseudo-structured mesh; the unstructured mesh that better matches a physical sensor reaches only 1,665 FPS, which is the more conservative reference for deployment.

Highlights & Insights

• The "trade accuracy for throughput" judgment is well aimed. Tactile RL does not need machine-precision convergence — it needs each iteration to be cheap — so replacing Newton's method with conjugate gradient and CCD line search with an analytical step-size bound buys convergence in tens of iterations and a GPU-friendly operator set. This reasoning transfers to any physics simulation embedded in an RL loop.
• Short-circuiting optical rendering with marker fields is a high-value trade. An RGB route must model illumination, gel texture and camera imaging, adding dimensionality and uncertainty; k-NN mapping plus weighted interpolation reads marker displacements straight off the mesh, deleting an entire layer of rendering uncertainty and keeping only the physics that actually drives the policy's decision.
• The empirical "parameter matching ≠ trajectory matching" result is well worth reusing. Sampling 20 \(k_p\) pairs and finding the trajectory discrepancy neither monotonic nor linearly correlated directly refutes the default assumption that calibration means making parameters identical, and yields bidirectional alternating minimization — treating the real side as adjustable too rather than as the sole ground truth. This is a transferable methodology for any sim-to-real calibration effort.
• Splitting the gap explicitly in two makes calibration reusable. Control alignment and material calibration depend only on the hardware and remain valid after one calibration; only task-based calibration needs redoing per task. This tiering lowers the marginal cost of the whole pipeline and is a large part of why it is reproducible.
• The protective mechanism that steps back when the consecutive marker field MSE exceeds a threshold is purely engineering but genuinely useful: it avoids crushing the gel while refraining from aborting an entire trial when the policy behaves abnormally.

Limitations & Future Work

• Task-based calibration is still task-specific, and the four contact states are an empirical choice. The authors themselves list this as future work, hoping to generalize TacAlign into a systematic sim-real pair sequence dataset covering robot control, IPC simulation and task-based calibration, rather than the per-task discussion given here.
• The real-world evaluation spans a far narrower orientation range than training. Training samples peg orientations from \([-35°, 35°]\), whereas the 60 real trials use only 0°, +15° and −15°. This is still zero-shot transfer, but the hardware evidence for "random orientation" is weaker than the claim suggests — the genuinely hard large-angle cases were never validated on hardware.
• Only one sensor is validated. The tactile representation is fixed at a 7×9 marker field; changing the gel type or the sensor requires redoing baseline calibration, and the effect of the marker count on policy learning is not analyzed.
• The RL training scale is conservative. Actual training uses only 512 environments (4 nodes × 16 GPUs), while the parallelism demonstration goes to 4,096; although the framework supports larger scales, the paper offers no evidence that scaling training further improves the policy.
• Robustness of the friction term and the CCD-free line search is not quantified separately. The paper emphasizes that the added frictional potential \(D(\mathbf{x})\) and the analytical step-size bound handle large rotations and slips, but does not report whether IPC's interpenetration-free / inversion-free guarantees still hold under extreme deformation (the evidence for the former is a qualitative deformation comparison).
• Real-world trials are limited. The 8 mm case has only 60 trials (20 per orientation), and the trial counts for the 12 mm / 16 mm pegs are not stated; nut threading at 58.3% is a single preliminary sentence with details deferred to the supplementary material.
• Threshold sensitivity of the protective mechanism is not analyzed. How the consecutive-marker-field MSE threshold should be set, and how much it affects the success rate, is not ablated.
• Promising directions: automating the choice of contact states for task-based calibration (for example, selecting calibration samples from the policy's own failure trajectories), and checking whether online training of the same scale still holds up on the unstructured mesh (1,665 FPS).

Related Work & Insights

• vs TacSL: TacSL generates marker/RGB tactile signals with a non-physical penalty-based method; it is fast and integrates into Isaac Gym for RL and imitation learning, making it the closest competitor. The difference is that it models only interpenetration regions without solving soft-body deformation or frictional contact, a cost invisible in simulation (0.789 vs 0.776 on 8 mm insertion, slightly in its favor) but exposed on hardware as 0.150 against 0.917. This paper's advantage is physical fidelity usable on real hardware at an affordable throughput; its disadvantage is lower raw FPS and much greater implementation complexity.
• vs Tacchi / Tacchi 2.0: both use MPM to model elastomer deformation and generate touch, evolving the representation from RGB to markers. This paper differs by switching to the IPC family and adding a friction term, buying numerical stability under large deformation — in the ablation Tacchi's real-world 0.083 falls below the no-tactile 0.067, exactly the consequence of unstable numerics producing wrong tactile feedback. Its disadvantage is that MPM may be more flexible for certain deformations, while IPC's barrier method requires extra handling of contact gaps.
• vs TacIPC / Taccel: both use IPC and share the "high" contact fidelity rating; TacIPC outputs RGB and simulates only at the sensor–object level, while Taccel reaches the whole-system level but demonstrates only sim-to-real data generation, with no policy learning. This paper fills the gap of "IPC-level fidelity + whole-system loop + online RL + multi-node multi-GPU" holding simultaneously.
• vs TacFlex: TacFlex uses FEM for high-fidelity multi-modal tactile imprint simulation and supports whole-system scenes, but its policy learning is offline behavior cloning, whereas this paper trains online with RL — a far more demanding setting for simulation throughput, and the reason the multi-node multi-GPU architecture is required here.
• vs TACTO (another representative of the penalty-based route): TACTO renders depth maps into high-resolution tactile images and is flexible and open-source, but it too belongs to the geometric-approximation route and cannot give correct responses for internal elastomer deformation or frictional shear.
• Transferable insight: making tactile simulation an external plugin that consumes only relative quantities means the same simulator can be reused with any engine; the authors suggest the architecture could in principle extend to simulating any physical sensor, including acoustic or even olfactory ones. Likewise, the two-tier calibration structure — one hardware calibration reused long-term, plus per-task recalibration — can be moved directly into sim-to-real pipelines for other sensing modalities.

Rating

• Novelty: ⭐⭐⭐⭐ The PNCG-IPC solver and the Taichi/Ray parallelism are adaptation and engineering, but this is the first work to bring frictional contact, IPC-level fidelity, whole-system online RL and multi-node multi-GPU together, with a reproducible four-stage calibration pipeline.
• Experimental Thoroughness: ⭐⭐⭐⭐ Both simulation and hardware, with ablations, peg-diameter generalization and a second task, plus quantified FPS and calibration process; points deducted because the hardware evaluation covers a narrow orientation range with few trials, and the choice of the four calibration contact states lacks justification.
• Writing Quality: ⭐⭐⭐ The main structure is clear and the tables are information-dense, but key equations are damaged in typesetting/caching (flagged above), and content carried by the supplementary material (randomization details, nut threading details) appears in the main text only as conclusions.
• Value: ⭐⭐⭐⭐⭐ A rare complete system that actually achieves zero-shot real-world deployment of a tactile policy; the 91.7% against TacSL's 15% on hardware shows that simulation fidelity is decisive for this task, and the alignment procedure and pluggable architecture are directly reusable for other sim-to-real work.

Related Papers

• [ECCV 2026] Sim, Yet Same: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds
• [ECCV 2026] ZAP: Zero-Shot Assembly Planning with Large Language Models
• [NeurIPS 2025] Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts
• [ECCV 2026] Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models
• [NeurIPS 2025] Real-World Reinforcement Learning of Active Perception Behaviors