PointGT: Simultaneous Geometric and Textural Editing for Point-Based Representations¶

Conference: ECCV 2026
Paper: ECCV Official Page
Area: 3D Vision
Keywords: point-based neural representations, 3D shape deformation, neural UV mapping, texture editing, deformation-aware correspondence

TL;DR¶

PointGT combines proximity-attention point rendering (PAPR) with normal-free neural UV parameterization, utilizing geometric regularizers and a deformation-aware canonical correspondence mechanism to achieve simultaneous, decoupled editing of geometry and high-resolution appearance where textures remain firmly attached under non-rigid deformations.

Background & Motivation¶

Point-based radiance field representations, most prominently 3D Gaussian Splatting (3DGS), have emerged as a dominant paradigm for photorealistic novel view synthesis and high-fidelity 3D reconstruction. In 3DGS, scenes are modeled as dense collections of 3D Gaussian primitives, each endowed with opacity and a single view-dependent color. Consequently, capturing high-frequency appearance details inevitably forces the optimization of millions of Gaussian primitives. This intrinsic coupling of geometry and texture resolutions severely impedes editable asset manipulation: users cannot perform texture edits finer than the physical footprint of individual Gaussians, while the volumetric, rigid nature of Gaussians makes free-form, smooth non-rigid geometric manipulation notoriously difficult.

Recent efforts have attempted to decouple appearance and geometry in Gaussian splatting by introducing either global UV parameterizations (such as Texture-GS) or per-primitive texture maps (such as GSTex and Textured Gaussians). However, neither line of work can support simultaneous geometric deformation and texture editing. For per-primitive methods built on 2D surfels, Gaussians are rigid planar primitives; when undergoing non-rigid bending or extreme deformation, the surface fractures, requiring intractable primitive splitting and cloning. On the other hand, global UV mappings learned on static surfaces become invalid after deformation, as the perturbed ray–surface intersections no longer align with the pre-learned 2D texture charts. While traditional triangle meshes preserve texture attachment through explicit topological connectivity and barycentric UV interpolation, unstructured point clouds lack connectivity, causing texture lookups to pop and drift when points move non-rigidly.

To fundamentally circumvent these limitations, this paper turns to attention-based point renderers such as PAPR. Rather than modeling discrete volumetric primitives, PAPR represents a scene as an unstructured cloud of infinitesimal points with learned features, computing ray-surface intersections and radiance via cross-attention interpolation over spatial $K$-nearest neighbors. Core idea: exploit the inherent surface continuity of attention-based point rendering, establish a high-fidelity neural UV atlas through on-ray and on-surface geometric regularizations, and introduce a deformation-aware canonical correspondence mechanism that fuses neighbor displacements to reliably map deformed intersections back to canonical space, enabling simultaneous, persistent geometry deformation and fine-grained texture editing.

Method¶

Overall Architecture¶

The PointGT pipeline consists of a pre-training phase for regularized geometry and neural UV learning, followed by an interactive editing and rendering pipeline. During pre-training, PointGT optimizes the canonical point cloud and feature fields using PAPR enhanced with two novel geometric regularizers. Ray–surface intersections extracted from this regularized geometry are fed into a multi-chart neural UV coordinate network and a corresponding 2D texture atlas, supervised with a normal-free Jacobian distortion objective. At edit time, users can simultaneously apply non-rigid geometric transformations to the point cloud and paint arbitrary edits onto the 2D texture atlas. To render deformed assets, rays compute attention-weighted intersections in deformed space, which are projected onto the rays and shifted back to canonical space via attention-weighted displacement fusion before performing UV sampling and final rendering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multi-view images & Initial points"] --> B["Geometry-regularized PAPR pre-training<br/>close-to-ray loss + close-to-surface loss"]
    B --> C["Normal-free neural UV mapping<br/>Jacobian scaling & area distortion penalties"]
    C --> D["Decoupled 2D texture & geometry editing<br/>Texture atlas painting / 3D non-rigid deformation"]
    D --> E["Deformation-aware canonical correspondence<br/>Ray projection + Attention-weighted displacement fusion"]
    E --> F["Canonical space UV lookup & View rendering"]

Key Designs¶

1. Geometry-regularized PAPR pre-training: Enforcing intersection fidelity and eliminating off-surface collapse In standard PAPR, the surface intersection point $\mathbf{x}_i$ along ray $r_i$ is computed as the attention-weighted average of its $K$ nearest neighbors: $\mathbf{x}_i = \sum_{j=1}^K a_{ij} \mathbf{p}_{ij}$. However, the sparsity of learned softmax attention frequently causes $\mathbf{x}_i$ to collapse toward discrete supporting points, degrading geometric resolution to that of the point cloud. Furthermore, densifying from sparse initial points often spawns noisy off-surface floaters, causing the convex hull of selected neighbors to miss the actual surface.

PointGT resolves these geometric artifacts by introducing two complementary regularizers during pre-training: The on-ray regularizer (close-to-ray) strictly penalizes the deviation of predicted intersection $\mathbf{x}_i$ from its orthogonal projection $\mathbf{x}'_i$ onto ray $r_i$: $$\mathcal{L}_{\text{close2ray}} = \frac{1}{M} \sum_{i=1}^M \|\mathbf{x}_i - \mathbf{x}'_i\|_2$$ The on-surface neighborhood regularizer (close-to-surface) pulls all $K$ selected supporting points $\mathbf{p}_{ij}$ toward the predicted intersection $\mathbf{x}_i$, compacting the point cloud into a clean, tight surface manifold: $$\mathcal{L}_{\text{close2surface}} = \frac{1}{MK} \sum_{i=1}^M \sum_{j=1}^K \|\mathbf{sg}(\mathbf{x}_i) - \mathbf{p}_{ij}\|_2$$ where $\mathbf{sg}(\cdot)$ denotes the stop-gradient operator to prevent premature geometric collapse onto unstable floaters. This regularizer is activated mid-training once coarse geometry has stabilized.

2. Normal-free neural UV mapping: Parameterizing point clouds without surface normals After regularizing canonical geometry, PointGT learns a continuous mapping from 3D surface points to a multi-chart 2D UV atlas. Existing neural UV methods such as Nuvo compute conformal and stretch penalties by constructing tangent frames from ground-truth surface normals. However, point clouds inherently lack explicit faces and normals, and estimating numerical normals over raw, noisy point intersections introduces substantial instability.

To circumvent this hurdle, PointGT formulates a distortion loss operating directly on the singular values of the UV mapping Jacobians. For chart $k$, the Jacobian matrix $J_{ik} \in \mathbb{R}^{2 \times 3}$ of texture coordinates $\mathbf{u}_{ik} \in \mathbb{R}^2$ with respect to 3D surface point $\mathbf{x}_i \in \mathbb{R}^3$ yields two singular values, $\sigma_{ik}^1$ and $\sigma_{ik}^2$, representing principal local stretching factors. Anisotropic distortion is penalized by enforcing isotropic scaling ($\sigma^1 = \sigma^2$): $$\mathcal{L}_{\text{scaling}} = \frac{1}{Gn} \sum_{i=1}^G \sum_{k=1}^n (\sigma_{ik}^1 - \sigma_{ik}^2)^2$$ To prevent local area collapse toward zero or explosive stretching, an area distortion term is incorporated: $$\mathcal{L}_{\text{area}} = \frac{1}{Gn} \sum_{i=1}^G \sum_{k=1}^n \left(\log(\sigma_{ik}^1 \sigma_{ik}^2)\right)^2$$ The combined distortion loss $\mathcal{L}_{\text{distortion}} = \mathcal{L}_{\text{scaling}} + \mathcal{L}_{\text{area}}$ regularizes the UV atlas to be smooth, low-distortion, and area-preserving without ever querying surface normals.

3. Deformation-aware canonical correspondence: Preserving texture attachment under non-rigid edits When an object undergoes non-rigid geometric deformation, canonical points $\mathbf{p}_j^{\text{can}}$ are displaced to deformed positions $\mathbf{p}_j^{\text{def}}$. Rays in deformed space re-query $K$-nearest neighbors and compute deformed attention weights $a_{ij}^{\text{def}}$, yielding a preliminary intersection $\hat{\mathbf{x}}_i^{\text{def}} = \sum_{j=1}^K a_{ij}^{\text{def}} \mathbf{p}_{ij}^{\text{def}}$. Since the neural UV atlas resides in canonical space, one might naïvely reuse deformed attention weights to interpolate canonical positions ($\sum_{j=1}^K a_{ij}^{\text{def}} \mathbf{p}_{ij}^{\text{can}}$). However, due to attention sparsity, this naïve transfer collapses canonical queries onto discrete points, resulting in severe texture popping, seams, and drift across neighboring rays.

PointGT introduces a two-step edit-time correspondence transfer: First, orthogonal ray projection enforces the on-ray constraint in deformed space: $$\mathbf{x}_i^{\text{def}} = \Pi_{\mathbf{r}_i}(\hat{\mathbf{x}}_i^{\text{def}})$$ Second, attention-weighted displacement fusion exploits point identity preservation across deformation. The relative displacements of supporting points $(\mathbf{p}_{ij}^{\text{can}} - \mathbf{p}_{ij}^{\text{def}})$ are smoothly aggregated using the deformed attention weights to shift the projected intersection back into canonical space: $$\mathbf{t}_i = \sum_{j=1}^K a_{ij}^{\text{def}} (\mathbf{p}_{ij}^{\text{can}} - \mathbf{p}_{ij}^{\text{def}})$$ $$\mathbf{x}_i^{\text{can}} = \mathbf{x}_i^{\text{def}} + \mathbf{t}_i$$ The canonical intersection $\mathbf{x}_i^{\text{can}}$ is then passed to the UV atlas for texture retrieval. This ensures that texture edits remain pinned to the deforming surface geometry, and the order of applying geometric and textural edits is completely interchangeable.

Loss & Training¶

The overall objective for pre-training PAPR geometry is: $$\mathcal{L} = \mathcal{L}_{\text{rendering}} + \gamma \mathcal{L}_{\text{close2ray}} + \eta \mathcal{L}_{\text{close2surface}}$$ with hyper-parameters set to $\gamma = 0.002$ and $\eta = 0.01$. The geometric regularizers are introduced at step 25,000 (after 10% of total training). For the neural UV atlas and 2D texture map, optimization follows Nuvo with loss weights set to 0.4 for $\mathcal{L}_{\text{distortion}}$ and 0.04 for texture color supervision $\mathcal{L}_{\text{texture}}$. The number of charts $n$ is configured to 4 or 8 depending on geometric complexity, and per-chart texture resolution is fixed at $256\sqrt{2/n} \times 256\sqrt{2/n}$ to maintain predictable GPU memory usage.

Key Experimental Results¶

Main Results¶

PointGT is quantitatively benchmarked on novel view synthesis (NVS) across the DTU and NeRF Synthetic (Blender) datasets against representative textured Gaussian and point-based representations. Evaluations are conducted under two operational regimes: the authors' recommended primitive counts (unconstrained) and an identical budget of 5,000 primitives (5k capped).

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	# Primitives (# Pts)
DTU	Texture-GS	30.53	0.920	0.083	90k
	GSTex	32.87	0.956	0.038	186k
	Textured-Gaussian	33.61	0.970	0.056	240k
	NeST Splatting	33.65	0.964	0.042	80k
	PAPR	29.34	0.952	0.073	30k
	Ours (PointGT)	33.48	0.973	0.023	30k
DTU (5k Cap)	Texture-GS	26.81	0.833	0.206	5k
	GSTex	28.92	0.932	0.072	5k
	Textured-Gaussian	29.58	0.912	0.080	5k
	NeST Splatting	32.68	0.966	0.056	5k
	PAPR	24.87	0.846	0.040	5k
	Ours (PointGT)	33.25	0.970	0.024	5k
NeRF Synthetic	Texture-GS	28.97	0.938	0.055	90k
	GSTex	33.25	0.969	0.024	100k
	Textured-Gaussian	33.24	0.967	0.043	190k
	NeST Splatting	33.37	0.967	0.032	73k
	PAPR	32.07	0.971	0.038	30k
	Ours (PointGT)	33.57	0.982	0.021	30k
NeRF Synth (5k Cap)	Texture-GS	19.29	0.795	0.213	5k
	GSTex	30.20	0.897	0.149	5k
	Textured-Gaussian	26.21	0.919	0.086	5k
	NeST Splatting	30.48	0.958	0.057	5k
	PAPR	30.38	0.963	0.048	5k
	Ours (PointGT)	31.01	0.976	0.039	5k

Ablation Study¶

To evaluate texture persistence and rendering consistency under non-rigid motion, PointGT is assessed on five Objaverse assets (emperor-fish, blossom, dress, flag, rexy) animated with artist-created motion sequences. Over full 360-degree orbit video trajectories with simultaneous texture painting and deformation, quality is benchmarked using the VBench video generation evaluation suite:

Method Config	Subject Cons. ↑	Bkg Cons. ↑	Motion Smooth. ↑	Aesthetic Qual. ↑	Imaging Qual. ↑
GSTex (Baseline)	0.827	0.923	0.985	0.434	0.520
Ours (PointGT Full)	0.844	0.925	0.982	0.437	0.585
Relative Gain ($\Delta\%$)	+2.16%	+0.24%	-0.40%	+0.55%	+12.48%

Key Findings¶

Necessity of geometric regularization: Ablation results demonstrate that omitting the on-ray regularizer ($\lambda_{\text{on-ray}} = 0$) causes surface points to cluster onto discrete point supports, collapsing UV space coverage into sparse clumps and blurring rendered textures into blocky artifacts. Omitting the on-surface regularizer produces pervasive off-surface floaters and depth map holes.
Superiority in low-primitive regimes: Under an extreme budget of 5k primitives, 3DGS-based texturing techniques suffer catastrophic degradation (e.g., Texture-GS drops to 19.29 dB PSNR on Synthetic), whereas PointGT maintains high visual fidelity (31.01 dB on Synthetic, 33.25 dB on DTU), highlighting the efficiency of smooth attention interpolation.
Persistent texture attachment under deformation: Compared to the naïve canonical transfer baseline (Eq. 11), PointGT's deformation-aware correspondence (Eq. 13) eliminates discontinuous UV jumps, yielding a +12.48% boost in VBench imaging quality over GSTex.

Highlights & Insights¶

Representation paradigm pivot: Rather than attempting to patch the rigid geometric limitations of 3D Gaussian Splatting with complex cloning heuristics, PointGT demonstrates that attention-based point renderers (such as PAPR) provide the ideal smooth substrate for non-rigid deformation and continuous parameterization.
Normal-free Jacobian regularization: Formulating conformal and area penalties through the singular values of the 3D-to-2D mapping Jacobian sidesteps the prerequisite of smooth surface normals, making neural UV mapping seamlessly applicable to unstructured point clouds.
Residual displacement fusion: Preserving canonical point identity across deformations allows the method to transfer deformed ray intersections back to canonical space via attention-weighted displacement fusion, establishing an elegant, drift-free bridge between deformed physical space and canonical texture space.

Limitations & Future Work¶

Atlas seam artifacts on complex topologies: Inheriting continuous optimization from Nuvo, the UV parameterization struggles to automatically generate clean, distortion-free charts on objects with high genus or intricate topological branches. Integrating learning-based automated seam cutting represents a promising avenue for improvement.
Absence of dynamic relighting and shadow updates: The learned texture maps currently model static radiance and diffuse color; they do not simulate dynamic self-shadowing or view-dependent specular shifts when geometry is bent non-rigidly.
Non-uniform point density shifts under severe stretching: Extreme non-rigid stretching can alter local point densities, causing $K$-nearest neighbor radii to expand dramatically and potentially degrade attention balance. Adaptive neighborhood searching could resolve this challenge.

vs PAPR [30]: PAPR pioneered proximity attention for point clouds but suffers from discrete point collapse and cannot edit textures. PointGT introduces geometric regularizers and couples PAPR with a neural UV atlas and deformation-aware transfer for full geometry-texture editability.
vs Texture-GS [26] & GSTex [15]: Texture-GS yields blurry global textures and cannot handle deformation; GSTex binds textures to rigid 2D Gaussian surfels, which fracture and extrude under non-rigid bending. PointGT maintains smooth surface continuity and sharp texture edits under non-rigid motion.
vs Nuvo [19]: Nuvo maps neural implicit surface samples to multi-chart UV atlases using normal-based conformal losses. PointGT redesigns these objectives into a normal-free Jacobian singular value formulation, successfully transferring neural UV parameterization to point-based radiance fields.

Rating¶

Novelty: ⭐⭐⭐⭐☆ An elegant fusion of proximity-attention point rendering, normal-free neural UV parameterization, and deformation-aware canonical correspondence.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Thorough quantitative validation spanning novel view synthesis, low-primitive stress testing, VBench video consistency, and geometric ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough technical formulation, and insightful comparative analyses against 3DGS alternatives.
Value: ⭐⭐⭐⭐☆ Provides a compelling and robust foundation for 3D asset editing, game-ready character animation, and generative 3D workflows.