EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learning¶

Conference: ECCV2026
Paper: ECCV Paper
Project: https://editverse3d.github.io
Area: 3D Vision
Keywords: Local 3D editing, coarse bounding boxes, region-aware learning, flow matching, part data

TL;DR¶

EditVerse3D turns TRELLIS into a supervised editor conditioned on an original object, a coarse 3D bounding box, and a local reference image, combining coarse-input training with region-aware loss reweighting to improve whole-output PSNR from Repaint's 27.44 to 28.67 on addition while reducing the need for precise masks and fully edited views.

Background & Motivation¶

Local 3D editing must modify the target part without unintentionally changing the remaining geometry and texture. Rendering 2D views, editing them, and reconstructing a 3D object can accumulate errors, while edits across views may be inconsistent. SDS approaches use 2D priors to guide 3D optimization, but typically require expensive per-instance optimization with unstable quality and convergence. 3D generative models such as TRELLIS provide a more direct representation, yet transferring Repaint or FlowEdit inference strategies into 3D does not automatically produce reliable local editing. Users also prefer a rough region and a reference image of the target part over preparing a complete edited view or delineating an exact 3D boundary.

This creates two related training problems. First, an exact shape mask leaks target geometry: it can make training easy but leave the model unprepared for coarse boxes containing empty space and neighboring structures. Second, unedited regions are often larger and easier to predict, so a uniformly averaged generation loss can hide failures in a small target region. Even when whole-object error looks acceptable, the junction between a new part and the existing object may remain blurry, and existing textures may change unnecessarily. The paper therefore changes the input representation, supervised regions, and training distribution together, rather than merely adding control inputs.

Such supervision requires many corresponding before-and-after 3D editing pairs, whereas real human editing trajectories are difficult to collect at scale. The authors exploit complete part information to turn restoration of a removed part into an addition task, using the original object as unambiguous ground truth. Training can then directly compare edited 3D latents instead of relying on a 2D model to generate supervision on demand. Core Idea: construct supervision through part restoration, learn coarse-box editing in a shared spatial frame, and combine regional balancing with hard-location reweighting to learn both modification and preservation.

Method¶

Overall Architecture¶

The inputs are a 3D object to edit, a coarse 3D bounding box specifying the target region, and a 2D reference image describing the desired part appearance. The reference image may contain only local elements rather than the complete edited object; it remains a required condition, so this is not a text-only editor. The model first generates edited structure, then generates texture at the corresponding voxel coordinates, and finally produces a 3D result through TRELLIS decoders.

TRELLIS normalizes an object into \([-0.5,+0.5]\) and records active voxels on a \(64^3\) grid. Its structure encoder compresses this into a dense \(16\times16\times16\times C_s\) latent. The texture path renders 150 views, extracts image features with DINOv2, and projects active voxels into the views to aggregate corresponding features. The texture encoder produces a sparse \(N\times C_t\) latent associated with active voxels, so changes in geometry also require changes in texture-feature indexing. These 150 views encode the 3D asset and should not be confused with the 24 reference-image viewpoints used during training. The method reuses TRELLIS encoders and decoders and trains structure and texture rectified flow models for editing, rather than relearning the entire 3D representation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Parts["Assets with complete parts"] --> Data["Part-Restoration Supervision"]
    Data -.->|Training samples| Align["Joint Normalization<br/>and Coarse-Box Training"]
    User["Object + coarse box<br/>Local reference image"] --> Align
    Align --> Flow["Two-Stage Latent Editing<br/>Structure generation → texture generation"]
    Flow --> Result["Decoded 3D object"]
    Flow -.->|Predicted velocities| Loss["Region-Aware Loss Reweighting"]
    Data -.->|Target-latent supervision| Loss
    Loss -.->|Training updates only| Flow

Solid arrows represent asset preparation or inference data flow; dashed arrows represent training data and supervision. Joint normalization applies during both training and inference, whereas random box perturbations and loss reweighting belong only to training. Inference requires neither a ground-truth target object nor a new training optimization for each input object.

Key Designs¶

1. Part-Restoration Supervision: derive verifiable edit targets from complete assets

The authors construct training pairs from manually validated Partverse segmentation data and Objaverse assets with built-in part structure. An asset with a complete part removed is the input, the original asset is the target, and the removed part's box and renderings provide spatial and appearance conditions. Using complete parts rather than arbitrarily cutting meshes avoids missing cut-surface or fracture information that would make the supervision unsuitable. Cleaning removes low-quality objects such as distorted geometry or missing textures, yielding approximately 85k meshes and 500k editing pairs.

This construction mainly supplies addition operations, rather than a large collection of genuine replacement demonstrations. The authors regard addition as a special case of replacement where the target region initially contains no structure, and observe transfer to replacement experimentally. This relationship explains why training is feasible, but does not guarantee success on arbitrary replacements. Targets with very small occupied voxel volumes are also filtered out to reduce training examples with little practical editing significance. The main paper does not specify the filtering threshold, so the criterion is described without inventing a value.

2. Joint Normalization and Coarse-Box Training: preserve spatial relations and avoid exact-shape shortcuts

Normalizing the object and target mask separately into unit cubes changes their original relative positions. For example, a local part could be independently enlarged to the scale of the entire object, making the condition inconsistent with the intended editing location. The method computes one normalization factor for the input object and mask together and applies it consistently to the relevant content. The same principle is used at inference so that the user's box remains in the correct object region. The ablation without joint normalization failed to converge and was consequently omitted from Table 3.

Spatial alignment alone does not make an idealized mask appropriate. Training starts with the target part's minimum bounding box and perturbs its size and position to approximate the rough regions users supply. Not every voxel inside the box necessarily needs changing; the network must infer the actual shape from the original object and reference image instead of treating the mask as the answer. In Table 3, training with exact shape masks produces substantially worse test results than training with boxes, supporting the interpretation that matching the input distribution matters more than training-condition precision. Figure 8 additionally illustrates a box enlarged by 1.5 times, translation, and reference-view changes, but does not provide full quantitative curves over these perturbations.

3. Two-Stage Latent Editing: generate geometry before aligning texture support

The structure model concatenates the input structure latent, a Boolean mask, and the noisy state of the sample being generated along the channel dimension. The resulting input has shape \(16\times16\times16\times(2C_s+1)\), while the generated structure latent remains \(16\times16\times16\times C_s\). The reference image supplies the editing condition, and the structure decoder converts the generated latent into new active voxel coordinates. Original-object information conditions generation rather than being mixed back only at the end.

The texture-stage difficulty is that the new structure can have different active voxel counts and locations from the input. The method therefore aligns input texture latents with output geometry coordinates; newly occupied coordinates not covered by the input receive zero-padded features. These features are concatenated with the texture mask and noisy state to generate and decode texture latents at the new coordinates. Zero padding does not prescribe black texture for the new part: it indicates the absence of an input latent feature to inherit, which conditional generation must fill. This places preservation of existing texture and generation of texture for new geometry in one spatially explicit computation.

The texture paragraph on page 6 places the concatenated channel count \(2C_t+1\) after an expression carrying output notation, creating a notation ambiguity. Figure 2 and the encoder definitions reliably establish concatenation of input features, mask, and noise followed by texture-latent prediction; they do not justify claiming that the decoder receives \(2C_t+1\) channels. The structure and texture models are trained independently, but inference has an explicit sequential dependency rather than two entirely independent generation branches.

4. Region-Aware Loss Reweighting: address area imbalance before emphasizing hard locations

Standard generation training averages velocity-prediction error uniformly over all locations. In editing, easily preserved regions may occupy most locations, making a single average insufficient to reveal whether a small target region has been learned. The authors compute masked and unmasked mean errors separately, normalizing each by its own element count so that a larger region cannot dominate merely through size. This partition follows the supplied mask; it should not be interpreted as a semantic label that every location inside a coarse box truly changes.

The method then selects the highest \(\tau\%\) of per-location errors and computes an additional loss over these hard locations. Difficulty is determined by current error, not by a fixed boundary band or manually selected part categories. Supervision can therefore emphasize new-part details, structural junctions, or hard-to-preserve original textures without requiring annotations of error types. The overall objective also adaptively reweights terms by their numerical scales, preventing region and hard-location terms from becoming ineffective because of magnitude differences. This differs from multiplying the masked-region loss by a fixed constant: it addresses both region size and learning difficulty.

Equations (3) through (6) are visibly damaged in the text extraction, particularly the operators in the final weighting expression. The main text also does not establish the value of \(\tau\), gradient handling for weights, or numerical stabilization details, so guesses are not presented as author formulas or implementation facts. The prose clearly supports regional normalization and hard-location selection; the exact implementation still requires checking the original typesetting or supplementary material.

A Worked Example¶

Consider the sword reference part in Figure 7: a user supplies the object to edit, a coarse box where the sword should appear, and a local image of the sword. This example connects the processing stages without claiming any unreported geometric dimensions for the figure. Joint normalization preserves the relative position and scale of the sword region and object; the structure stage generates new voxel support from the original object and reference image. The texture stage inherits input features at existing coordinates and generates target texture after zero padding at newly introduced sword coordinates. During training, regional and hard-location losses encourage preservation of existing hat dots and arm patterns while improving the hilt-to-blade junction. These visual observations come from the authors' description of Figure 7 and are not a substitute for an experiment measuring boundary error separately.

Loss & Training¶

The clearly readable base flow-matching objective is Equation (2), where \(x_0\) is the target latent, \(\epsilon\) is noise, and \(x_t=(1-t)x_0+t\epsilon\).

\[ \mathcal{L}(\theta)=\mathbb{E}_{t,x_0,\epsilon}\left\|v_\theta(x_t,t)-(\epsilon-x_0)\right\|_2^2. \]

The network learns a velocity field rather than directly using final-mesh rendering error as this training objective; regional reweighting operates on velocity-prediction errors. For each target part, training renders 24 views from cameras uniformly sampled on a bounding sphere and randomly selects one condition image per iteration. This removes the need to strictly match the reference image to one fixed input-object view, but does not imply that all occlusion and pose ambiguities are resolved. The structure and texture models each train for 10k steps; the paper reports batch size 1, 96 V100 GPUs with 32GB memory, and approximately 10k GPU hours across the two stages. The main text does not clarify whether batch size is global or per GPU, so the reported wording is retained without inferring a total batch size. Inference encodes the original object, samples structure and texture latents from noise, and decodes the output without per-instance optimization using the regional loss.

Key Experimental Results¶

Main Results¶

The addition test set is derived from PartObjaverse-Tiny, with approximately 200 meshes and 1500 editing pairs; the VoxHammer replacement set contains 100 meshes and 300 editing pairs. Evaluation uses CD for geometry and renders 32 views for PSNR, SSIM, LPIPS, DINO feature similarity, and FID. Results are also averaged over 24 reference-image viewpoints, with varying test-box sizes and positions. Lower CD, LPIPS, and FID are better; higher values are better for the other metrics, and scores over different evaluation regions are not directly comparable.

The following selection from Table 1 on page 10 retains the principal geometry and image-quality metrics. Addition evaluates the whole output; replacement lacks target-region ground truth, so its numbers evaluate only preserved regions.

Task and evaluation scope	Method	CD ↓	PSNR ↑	LPIPS ↓	FID ↓
Addition, whole output	Instant3dit	0.110	8.075	0.501	81.78
Addition, whole output	Repaint	0.008	27.44	0.038	3.408
Addition, whole output	FlowEdit	0.017	19.46	0.102	6.129
Addition, whole output	VoxHammer	0.023	21.16	0.095	8.281
Addition, whole output	EditVerse3D	0.005	28.67	0.029	2.960
Replacement, preserved regions only	Repaint	0.007	35.96	0.010	29.32
Replacement, preserved regions only	EditVerse3D	0.005	36.32	0.008	28.34

The addition PSNR gain over Repaint is 1.23; replacement PSNR of 36.32 indicates preserved-region quality, not correctness of the replacement part. Table 2 on page 10 further reports target-region PSNR of 21.01 versus Repaint's 19.00, and unedited-region values of 35.45 versus 31.16. This regional protocol uses the target's minimum 2D bounding box for irregular image regions and treats the edited region as identical to ground truth when evaluating unedited regions. The authors acknowledge optimistic metric estimates, so these are not strict per-pixel regional measurements; varying target-box sizes also prevent reporting FID in that table.

Ablation Study¶

The following selection from Table 3 on page 12 compares training choices on PartObjaverse-Tiny. Exact denotes a precise shape mask, BBox a minimum bounding box, and BBox+ a box with size and position perturbations. High denotes manually annotated segmentation sources and Low built-in asset parts; these results should not be conflated with the full-training results in Table 1.

Config	Mask	Small-target filter	Loss	Segmentation	Condition views	CD × 10² ↓	PSNR ↑	FID ↓
#1	Exact	No	MSE	High	Random	2.502	21.03	9.329
#2	BBox	No	MSE	High	Random	0.701	24.65	3.709
#3	BBox+	No	MSE	High	Random	0.692	24.75	3.573
#4	BBox+	Yes	MSE	High	Random	0.658	24.82	3.542
#5	BBox+	Yes	Ours	High	Random	0.635	25.01	3.496
#6	BBox+	Yes	Ours	Low	Random	0.636	24.90	3.474
#7	BBox	No	MSE	High	Traverse	0.850	24.41	4.129

Key Findings¶

The largest visible single-step improvement comes from changing the training mask between #1 and #2: PSNR rises from 21.03 to 24.65, a gain of 3.62, exceeding the gain from the regional loss alone.
Moving from #4 to #5 isolates the loss change: PSNR rises from 24.82 to 25.01 and CD × 10² falls from 0.658 to 0.635, supporting the loss without attributing all overall gains to it.
Geometry scores for #5 and #6 are close, while #6 achieves slightly better FID, 3.474 versus 3.496; high-quality manual segmentation does not win on every metric.
Random reference-view selection in #2 outperforms traversal in #7, but this comparison does not establish a general theoretical advantage over systematic traversal.
Table 4 on page 14 reports 20 seconds for this method, 20 seconds for Repaint, and 120 seconds for VoxHammer; these timings must be distinguished from approximately 10k GPU hours of training.
The surrounding text does not specify timing hardware or complete preprocessing boundaries, so 20 seconds is not verified end-to-end user latency.

Highlights & Insights¶

Coarse inputs are part of the training distribution rather than an interface relaxed only at inference. The exact-mask ablation shows how additional geometric hints can create a deployment-time dependency.
Regional balancing and hard-location selection solve different problems: area dominance and local failures hidden by regional averages. Their combination is more informative than a generic emphasis on a local loss.
Texture alignment at new geometry coordinates is a useful representation choice. It explicitly separates inheritable features from features that must be generated, offering a direction for other sparse 3D editing tasks.
Part restoration turns missing editing trajectories into supervised recombinations of existing assets. Scalability comes from asset part structure rather than large amounts of manually authored before-and-after edits.

Limitations & Future Work¶

Training mainly covers addition; transfer to replacement is demonstrated, but missing replacement-region ground truth leaves quantitative evidence of editing correctness incomplete.
The method still requires an original 3D asset, a coarse box, and a reference image; reducing input burden does not eliminate manual target specification or reference collection.
Filtering low-volume targets may reduce coverage of tiny-accessory edits. This is a reader inference from the training choice, not a risk separately tested in the paper.
Training is expensive, and the paper does not provide a full data-scale versus quality curve or separately ablate regional balancing and the hard-location term.
Occlusion, viewpoint and appearance ambiguity in a single reference image, and error accumulation across successive edits deserve independent evaluation; the current results do not guarantee robustness to them.
Loss expressions in the available text are incompletely extracted, while details such as filtering thresholds and perturbation distributions are deferred to supplementary material; these implementation facts remain unverified here.

Compared with TRELLIS: the method reuses its structure and texture latent spaces and encoder-decoder capabilities, training the generative backbone into an editor explicitly conditioned on the input object and editing region.
Compared with Repaint and FlowEdit: the baselines enable training-free editing through inference-time mixing or velocity-field modification; this method learns coarse-region and local-image conditioning through supervision, at the cost of training data and compute.
Compared with VoxHammer: the latter uses inversion and intermediate features for consistency; this method reports better metrics and shorter runtime in the present setting, while the authors acknowledge that evaluation differences may affect baseline performance.
Compared with Instant3dit: editing is learned directly in 3D latent space rather than primarily editing four canonical views and reconstructing them; one evaluation does not establish that all multiview approaches are inferior.
Research direction: separately measure tiny targets, replacement semantics, locations inside the box that should remain unchanged, and drift from repeated edits to identify what regional reweighting improves.

Rating¶

Novelty: 4/5. Coarse interaction conditions, supervised data construction, and regional training objectives are combined on a mature 3D generative backbone.
Experimental Thoroughness: 4/5. Whole-output, regional, ablation, and efficiency comparisons are included, but replacement targets lack ground truth and loss components are not fully isolated.
Writing Quality: 4/5. The pipeline and motivation are clear, while some dimensional notation and implementation details need clarification; extraction damage is not itself treated as a writing defect.
Value: 4/5. The work informs practical 3D editing interfaces and dataset construction, although training cost limits lightweight reproduction.