RADmesh: Remesh-Aware Mesh Deformation

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://threedle.github.io/radmesh
Area: 3D Vision
Keywords: Explicit mesh deformation, Dynamic remeshing, Optimizer state interpolation, Score distillation sampling, Generative 3D editing

TL;DR

Addressing the severe triangle distortion and structural constraints in generative explicit mesh deformation under noisy visual supervision, RADmesh integrates a vertex-based 6D scale-rotation deformation quantity with coarse-to-fine periodic isotropic remeshing and barycentric optimizer state interpolation, enabling large-scale local growth and global detailing while strictly preserving isotropic element quality.

Background & Motivation

Recent years have seen remarkable progress in text-guided 3D mesh deformation supervised by 2D diffusion models via score distillation sampling (SDS/CSD). Pioneer methods such as TextDeformer, MeshUp, and Geometry in Style optimize deformation variables defined on a fixed mesh connectivity (e.g., face Jacobians or vertex rotations) and resolve global vertex displacements through differentiable solvers like dARAP or Poisson equations. However, because these methods strictly lock the original mesh topology, they can only reshape geometry within the representational bandwidth of the initial triangulation. When the desired edit demands large structural reconfiguration, drastic appendages, or thin volumes (such as growing massive wings, an elongated tail, or branching antlers on an animal), keeping connectivity fixed unavoidably stretches or crushes triangles. This induces severe geometric artifacts, self-intersections, and insufficient triangle resolution for fine features.

Deforming with remeshing in the loop is an intuitive solution, as triangle connectivity inherently reflects the underlying geometry and curvature distribution. Nevertheless, coupling remeshing with generative visual supervision poses formidable technical barriers. Remeshing is an inherently discrete and non-differentiable operation involving edge collapses, splits, and flips, whereas the gradient supervision provided by diffusion models is notoriously noisy. Existing on-the-fly remeshing techniques designed for inverse rendering (such as Palfinger or RoAR) tie remeshing heuristics directly to raw vertex coordinates and local gradient statistics, proving brittle and unstable when applied to generative synthesis. Meanwhile, differential quantities favored in generative deformation (like per-face Jacobians) are face-centered; when discrete topological edits alter the faces, mapping and interpolating these face-based attributes onto newly created mesh elements becomes lossy and ill-posed.

RADmesh addresses these challenges by moving away from face-based representations to a vertex-based differential parameterization that seamlessly survives discrete remeshing. Core idea: parameterize mesh deformation via a vertex-based 6D representation encoding both local rotation and anisotropic scaling, and periodically apply coarse-to-fine isotropic remeshing while interpolating the internal Adam optimizer momentum state onto new vertices via barycentric coordinates.

Method

Overall Architecture

Given an initial source mesh \(M_0 = (V_0, F_0)\) and a text prompt \(x\), RADmesh iteratively deforms and remeshes \(M_0\) to generate a target mesh \(M^* = (V^*, F^*)\). The framework natively supports both global deformation and local editing guided by a binary selection mask \(\text{sel}(k) \in \{0, 1\}\). The optimization proceeds in intervals of \(N=100\) epochs: between remeshing steps, the mesh connectivity \(F_{r_i}\) remains fixed, and gradients update only the vertex-based deformation quantity \(Q_i\). Every \(N\) epochs, the geometry optimization is paused, an isotropic remesher updates the base mesh to match the current deformed geometry, and the optimizer momentum state is interpolated onto the new discretization before resuming gradient descent.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source mesh M0 + selection mask sel"] --> B["Preprocessing & Initial Inflation<br/>Normalize to unit cube + patch volume dilation"]
    B --> C["Current base mesh M_ri + deformation variable Qi"]
    C --> D["Vertex-level Local Deformation & Global Solve<br/>6D parameters into rotation & scale + dARAP solve"]
    D --> E["Differentiable Rendering & Multi-view CSD Supervision<br/>nvdiffrast rendering + CSD gradient backprop"]
    E --> F{"Scheduled remesh epoch (N=100)?"}
    F -->|No: continue current base optimization| G["Adam update deformation quantity Qi+1"]
    G --> C
    F -->|Yes: perform discrete remeshing| H["Coarse-to-fine Isotropic Remeshing<br/>Edge collapse/split/flip/tangential smoothing/reprojection"]
    H --> I["Barycentric Optimizer State Interpolation<br/>Map Adam 1st & 2nd moments onto new vertices"]
    I --> J["Reset base mesh M_ri+1 & quantity Qi+1"]
    J --> C
    D --> K["Output target mesh M*<br/>High-detail geometry & isotropic elements"]

Key Designs

1. Vertex-based Scale-Rotation Deformation Quantity: Unlocking Large Growth & Lossless Transfer To ensure that deformation parameters and optimization history survive discrete remeshing without lossy projections, RADmesh departs from face Jacobians and defines the deformation quantity at each vertex as a 6D vector \(q_k = (q_k^{\text{dir}}, q_k^{\text{scale}}) \in \mathbb{R}^6\). The first three components \(q_k^{\text{dir}} \in \mathbb{R}^3\) specify the target normal direction, while the last three components \(q_k^{\text{scale}} \in \mathbb{R}^3\) dictate local scaling along the three world axes. In the local step, a Procrustes problem finds the optimal rotation matrix \(R_k \in \mathrm{SO}(3)\) aligning the vertex normal \(u_k\) and spokes-and-rims edge vectors \(N_k\) to the goal direction \(q_k^{\text{dir}}\): $\(R_k = \arg\min_{\tilde{R} \in \mathrm{SO}(3)} \sum_{(i,j) \in N_k} w_{ij} \|\tilde{R}(p_i - p_j) - (p_i - p_j)\|_2^2 + \lambda a_k \|\tilde{R} u_k - q_k^{\text{dir}}\|_2^2\)$ where \(w_{ij}\) represents the cotangent weights, \(a_k\) is the Voronoi area, and \(\lambda=8\) controls the alignment stiffness. Combining \(R_k\) with the diagonal scaling matrix \(S_k = \mathrm{diag}(q_k^{\text{scale}})\) forms the local transformation \(T_k = R_k S_k\). In the global step, new vertex coordinates \(V'\) are computed via the differentiable ARAP solve: $\(V' = \arg\min_{V} \sum_{(i,j) \in E} w_{ij} \| (p_i' - p_j') - \frac{1}{2}(T_i + T_j)(p_i - p_j) \|_2^2\)$ For localized edits, unselected vertices are assigned identity transformations \(T_k = I\) and substituted as fixed Dirichlet boundary conditions. The introduction of optimizable scaling \(q_k^{\text{scale}}\) is fundamental: it overcomes the distance-preserving stiffness of pure ARAP rotations to allow massive outward growth from small patches. Furthermore, for global deformations, applying a soft clamp (LeakyReLU with a floor of 0.98) on \(q_k^{\text{scale}}\) structurally counteracts the intrinsic shrinking bias of SDS visual losses.

2. Barycentric Optimizer State Interpolation: Preserving Optimization Momentum Across Topological Shifts Discrete remeshing completely restructures vertex indices and connectivity. Reinitializing the optimizer from scratch after each remesh erases accumulated gradient history, causing severe optimization instability, bilateral asymmetries, and runaway triangle proliferation. RADmesh leverages the final surface reprojection stage of the remesher, where each newly inserted vertex \(w\) is projected onto the pre-remesh surface and obtains barycentric coordinates \((\beta_1, \beta_2, \beta_3)\) relative to its parent triangle. Because the deformation representation is vertex-based, the Adam optimizer's internal state—specifically the first moment \(m\) and second moment \(v\) accumulators—can be directly interpolated onto new vertices: $\(\text{State}(w) = \sum_{j=1}^3 \beta_j \cdot \text{State}(v_j)\)$ Carrying forward the optimizer state ensures that newly refined vertices inherit the trajectory and momentum of the deformation. This momentum continuity stabilizes the optimization against noisy CSD loss signals, suppresses transient gradient shocks, ensures clean bilateral symmetry, and prevents unchecked element subdivision.

3. Coarse-to-Fine Periodic Isotropic Remeshing: Synchronizing Triangulation with Geometric Scale For remeshing, RADmesh incorporates the Botsch & Kobbelt isotropic remesher, executing 2 iterations every 100 epochs across five sequential operations: edge collapse, edge split, valence-improving edge flip, tangential smoothing, and surface reprojection. For local edits, a real-valued selection attribute \(\text{sel}_R(k)\) restricts topological modifications strictly to edges where both endpoints satisfy \(\text{sel}_R \ge 0.5\), leaving unselected regions, their UV mappings, and correspondences completely intact. Crucially, rather than relying on curvature-adaptive remeshing, RADmesh employs a coarse-to-fine target edge length schedule: - Local growth task: Target edge length begins at \(1.7\times\) the average edge length of the preprocessed mesh, linearly decays to \(1.3\times\) over 10 remeshes, further ramps to \(1.0\times\) across the next 5 remeshes, and remains at \(1.0\times\) thereafter. - Global deformation task: Target edge length starts at \(1.4\times\) the average edge length, decays linearly to \(1.0\times\) over 18 remeshes, and stays at \(1.0\times\). This gradual refinement prevents committing dense triangles too early during coarse exploration, ensuring stable evolution from high-level volume formation to delicate geometric detailing.

Loss & Training

• Preprocessing & Initial Inflation: Source meshes are normalized to the \([-1, 1]^3\) bounding cube. For local growths, to overcome weak initial SDS gradients on flat patches, an initial displacement \(\ell = a^{-b\sqrt[3]{V}}\) is added along the vertex normal, where \(V\) is the volume of the hole-closed selection patch, and hyperparameters are set to \(a=4.3, b=5.2\). An initial base remesh is executed before optimization starts.
• Differentiable Rendering & Supervision: Multi-view images are rendered via nvdiffrast using 8 randomly sampled camera views per epoch. Semantic supervision is driven by Cascaded Score Distillation (CSD) powered by the multi-stage diffusion backbone DeepFloyd IF.
• Computational Cost: On a single NVIDIA L40S GPU, local growth runs (up to 2,600 epochs) take 85–90 minutes, while global detailization runs (up to 2,200 epochs) take 70–75 minutes.

Key Experimental Results

Main Results

Quantitative comparisons were conducted on \(n = 64\) diverse shape-prompt pairs spanning local growths and global deformations. Baselines include intermediate proxy-based methods (MagicClay with implicit SDFs, Instant3dit with multi-view inpainting) and explicit mesh deformation methods (Geometry in Style, MeshUp). Metrics include CLIP score (\(\times 100\)), VQA score (\(\times 100\)), average triangle Face Quality (FQ, \(\times 100\), defined as \(4\sqrt{3}\frac{A}{\ell_1^2+\ell_2^2+\ell_3^2}\) where equilateral triangles equal 100), and average face count (#F).

Method	CLIP Score ↑	VQA Score ↑	Face Quality (FQ) ↑	Face Count (#F)	Notes
MagicClay [SIGGRAPH Asia 24]	29.821	57.150	86.784	17,203.55	Implicit SDF proxy; noticeable surface noise
Instant3dit [CVPR 25]	28.399	52.111	80.132	40,034.69	Multi-view inpainting; bloated faces & poor element quality
Geometry in Style [CVPR 25]	29.279	52.120	94.146	11,308.06*	Fixed topology; lacks remeshing (source mesh resolution)
MeshUp [3DV 25]	30.725	66.093	84.831	11,308.06*	Fixed topology; severe triangle stretching
MeshUp + our remesh schedule	31.148	69.360	96.310	29,461.34	Confirms general benefits of our remeshing schedule
RADmesh (Ours)	31.416	72.930	96.516	16,963.75	Highest CLIP/VQA scores, best face quality, triangle-efficient

*Note: Geometry in Style and MeshUp do not perform remeshing; their face counts reflect the input source meshes.

Ablation Study

The ablation studies systematically isolate the impact of the scale parameter \(q_k^{\text{scale}}\), optimizer state interpolation, and the coarse-to-fine remeshing schedule.

Configuration / Case	Key Metric & Visual Observation	Mechanism & Analysis
No scale parameter \(q_k^{\text{scale}}\)	Growths fail to extend outward; blunt extremities	Pure rotations cannot supply the local volume expansion required for massive appendages like branching antlers
w/o optimizer state interpolation ("old-timey video camera")	Face count explodes to 34,288	Loss of momentum causes violent gradient oscillations upon remeshing, generating asymmetric artifacts and unneeded subdivision
With optimizer state interpolation (Ours) ("old-timey video camera")	Face count maintained at 16,778	Preserved momentum ensures smooth convergence, bilateral symmetry, and triangle budget efficiency
w/o optimizer state interpolation ("tokyo tower")	Face count increases to 19,528	Optimization instability amplifies view-dependent distortion and irregular geometry
With optimizer state interpolation (Ours) ("tokyo tower")	Face count remains compact at 16,788	Crisp tiered geometry with neat isotropic elements
Single fine remesh once at start ("with two pegasus wings")	23,676 faces; malformed wing structures	Over-refining early traps the optimizer in complex local minima before high-level volume forms
Coarse-to-fine periodic remesh (Ours) ("with two pegasus wings")	22,698 faces; graceful, well-formed wings	Gradual resolution scaling allows coarse structural emergence followed by detailed feather synthesis
Curvature-adaptive remeshing	Large growths suppressed	Allocating dense triangles prematurely to dynamic, high-curvature growing fronts breaks coarse-to-fine exploration

Key Findings

• Remeshing unlocks the true expressive ceiling of explicit deformation: Incorporating the proposed remeshing schedule into the existing baseline MeshUp lifts its CLIP score from 30.725 to 31.148, VQA score from 66.093 to 69.360, and FQ from 84.831 to 96.310. This proves that the historical limitations of explicit mesh editing stem from topological rigidity rather than differential deformation mechanics.
• Optimizer momentum carry-over is indispensable for discrete geometry updates: Disabling optimizer state interpolation leads to uncontrolled face count escalation (e.g., a 104.4% increase in the video camera experiment) and geometric asymmetry. Barycentric interpolation bridges discrete topology changes with continuous momentum-based optimization.
• Direct mesh optimization surpasses proxy-based pipelines: In contrast to Instant3dit (which yields over 40,000 faces with a poor FQ of 80.132) and MagicClay (prone to implicit reconstruction noise), RADmesh delivers superior perceptual quality (VQA 72.930) with less than half the face count (16,963.75) and near-ideal element aspect ratios (FQ 96.516).

Highlights & Insights

• Bridging continuous optimization momentum and discrete mesh operations: The coupling of continuous deep learning optimizers with discrete topological surgery is a longstanding challenge in geometry processing. RADmesh's insight that barycentric coordinates from surface reprojection can serve as a natural transport operator for Adam's first and second moments provides an elegant, generalizable recipe for dynamic topology learning.
• Constructive suppression of generative shrinkage bias: Diffusion-based 3D generation is prone to volume shrinking. Instead of tuning fragile regularization weights, RADmesh enforces a structural LeakyReLU clamp (floor 0.98) directly on the local scaling dimension \(q_k^{\text{scale}}\), neutralizing the collapse tendency by construction.
• Seamless boundary preservation for asset workflows: By propagating vertex selection flags as continuous real attributes, all topological alterations remain strictly bounded within the target region. Non-selected surfaces preserve exact vertices, topology, and UV textures, enabling practical multi-stage iterative workflows (e.g., sequentially adding arms, head, and head accessories onto a torso).

Limitations & Future Work

• Supervision bottlenecks: Optimization speed is heavily constrained by multi-view 2D diffusion backpropagation (70–90 minutes per run on an L40S GPU). Future integration of fast feed-forward 3D reconstruction models or few-step distillation priors could dramatically accelerate turnaround.
• Strict genus and manifold constraints: The Poisson/dARAP solvers require 2-manifold meshes, and the current edge operations (collapse, split, flip) are strictly genus-preserving, precluding topological changes such as puncturing holes or splitting disconnected components.
• Non-manifold topological evolution: Extending the framework to non-manifold Laplacians and topological tearing/merging operators represents an exciting avenue for arbitrary topological synthesis.

Related Work & Insights

• vs Geometry in Style (Dinh et al., CVPR 2025): Authored by the same lab, Geometry in Style introduced normal-based dARAP stylization on fixed topologies. RADmesh expands this into a full 6D scale-rotation formulation with periodic remeshing and state interpolation, achieving a paradigm shift from shallow surface relief to drastic structural growth and detailing.
• vs MeshUp (Kim et al., 3DV 2025) & TextDeformer (Gao et al., SIGGRAPH 2023): Both rely on face-based Jacobian fields, which resist clean interpolation across changing mesh connectivity. RADmesh’s vertex-based parameterization pairs seamlessly with barycentric state transfer, yielding cleaner geometries and superior element quality.
• vs MagicClay (Barda et al., SIGGRAPH Asia 2024) & Instant3dit (Barda et al., CVPR 2025): These systems rely on implicit SDFs or multi-view inpainting proxies, suffering from conversion artifacts and excessive triangle counts. RADmesh operates natively and exclusively on triangle meshes, ensuring crisp details and finite-element-ready isotropic meshes.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the seamless integration of coarse-to-fine remeshing and optimizer momentum interpolation into generative explicit mesh deformation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive 64-pair benchmark, rigorous ablations with exact face counts, and compelling qualitative galleries across local and global tasks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, thorough exposition of discrete-continuous mechanics, and insightful ablation discussions]
• Value: ⭐⭐⭐⭐⭐ [Establishes a practical, robust paradigm for generative mesh editing with direct relevance to 3D asset production]