Skip to content

GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoising

Conference: ECCV 2026
arXiv: 2606.30003
Code: https://github.com/Heey731/GeoEdit
Area: Diffusion Models / Image Editing / 3D Vision
Keywords: Geometry-Aware Editing, Dual-Branch Denoising, Variance-Homogeneous Injection, Training-Free, Single-Image Object Manipulation

TL;DR

GeoEdit proposes a training-free "lift-manipulate-render-denoise" pipeline. It first reconstructs a single-view image into 3D in a decoupled manner, renders a geometrically aligned coarse proxy image according to user-specified 3D transformations (translation, rotation, scaling), and then employs "dual-branch denoising" to inject the 3D constraints solely into the foreground in a variance-homogeneous manner within a denoising window. This allows the background to generate freely, simultaneously adhering to physical constraints like perspective and occlusion while balancing foreground rigidity and background realism.

Background & Motivation

Precise translation, rotation, and scaling of an object within a single photo—while keeping the output consistent with perspective, occlusion, and scene geometry—is a highly practical but unresolved task for diffusion-based image editors. Current mainstream approaches rely on mask-based 2D inpainting (e.g., RePaint), which operates entirely on the pixel plane. Consequently, they suffer from three systematic flaws: First, they lack 3D spatial awareness, and manipulating objects directly in image coordinates disrupts perspective and distorts geometry. Second, the strong generative prior of diffusion models often leaves a "ghosting" artifact at the object's original location. Third, directly inserting structural proxies (like hard masks) into the latent space causes a distribution mismatch. This abrupt intervention violates the variance homogeneity assumed by Diffusion Transformers, leading to self-attention leakage that blurs the background and collapses structure. Point-based manipulation models like DragGAN/DragDiffusion, as well as works injecting 3D conditions like GeoDiffuser, Diffusion Handles, and Object-3DIT, either suffer from in-plane confinement, rely on synthetic data causing a sim-to-real gap, or depend on costly per-scene SDS optimization with weak geometric accuracy, such as 3DitScene.

These failures point to an overlooked structural fact: the edited foreground and the unoccluded background impose asymmetrical requirements on the generation process. The relocated object must rigidly adhere to a predefined 3D geometry, whereas the exposed background (which contains both de-occlusion holes and the "footprints" of the original object) requires generative freedom to synthesize plausible content. Accommodating both demands through a unified denoising trajectory is fundamentally infeasible: granting enough freedom to the background leads to drift in the foreground skeleton, while restricting freedom to preserve geometry leaves the background plagued by artifacts of the coarse proxy.

The key insight of this work is that since a single trajectory cannot satisfy both, the foreground and background should receive different treatment within the same denoising process, where the key to joining them is to avoid disrupting the latent statistics of the pre-trained DiT. Core Idea: Lift the editing into 3D to obtain a geometrically aligned proxy, and then utilize "dual-branch denoising"—which leverages the temporal prior of a video diffusion backbone to preserve object identity without training, injecting 3D constraints solely into the foreground via "variance-homogeneous injection" within a narrow denoising window while letting the background denoise freely. Because the injected signal strictly matches the variance specified by the noise schedule at each timestep, the self-attention mechanism observes a spatially homogeneous distribution, suppressing leakage. Thus, foreground rigidity and background flexibility are achieved simultaneously without any training or architectural modifications.

Method

Overall Architecture

GeoEdit employs a training-free 2D → 3D → 2D pipeline, orchestrating off-the-shelf pre-trained models without fine-tuning any weights. The input is a source image \(I_{\text{src}}\) and user-specified 3D transformations \(\mathcal{T}=\{\mathbf{R},\mathbf{t},s\}\) (rotation, translation, scaling) on the target object; the output is a realistic synthesized image that strictly respects \(\mathcal{T}\) while keeping the unedited background consistent. The pipeline consists of two phases: the first phase "lifts, manipulates, and renders" the scene into a geometrically aligned proxy image \(I_{\text{proxy}}\), a structural depth map \(D_{\text{rep}}\), and a projected object mask \(M\). The second phase represents the core contribution of this work—"dual-branch denoising"—which refines this coarse proxy into a photorealistic composite.

The first phase combines existing techniques: first, a monocular depth/geometry model (VGGT) lifts the source image into an incomplete 3D scene, which is segmented into background point clouds and visible foreground point clouds using a mask. Then, a multi-view diffusion model (SV3D) is applied to the cropped foreground to hallucinate the occluded surfaces, yielding a geometrically complete object point cloud in an isolated canonical space. Since both the complete point cloud and the visible foreground point cloud originate from the same source image, the same pixel coordinates correspond to the same physical surface points. This allows the extraction of dense 3D-3D correspondences to solve a similarity transformation (rotation, translation, scale) that registers the object back to the global scene using least squares. The user applies the desired transformation \(\mathcal{T}\) in this unified coordinate system. The original visible foreground is then removed, and the resulting hole is coarsely filled using Telea fast marching inpainting to produce a structurally continuous background. Finally, the transformed 3D object is reprojected onto this background to obtain \(I_{\text{proxy}}\) and \(D_{\text{rep}}\). While the background texture is coarse at this stage, it serves as a structural baseline for the downstream denoising process to synthesize high-frequency details.

The dual-branch denoising in the second phase is crucial. It is built upon a depth-conditioned video diffusion backbone (Wan2.2-VACE): the proxy image is replicated into several identical frames to form a short sequence. The temporal self-attention of the backbone enforces cross-frame consistency, naturally preserving the object's identity without any ID-specific fine-tuning. The depth map \(D_{\text{rep}}\) is injected via a ControlNet-like control block, acting as a geometric scaffold to anchor the object's skeleton. On top of this, a warm-start is used to initiate denoising from an intermediate timestep, followed by variance-homogeneous injection within a carefully selected denoising window. This replaces only the foreground latent while letting the background evolve freely. Outside this window, the generation prior is fully unleashed to remove the old object and complete the background. The full pipeline is illustrated below.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Source Image + 3D Transform T"] --> B["Decoupled 3D Reconstruction<br/>and Canonical Alignment<br/>Independent scene/object lifting → point correspondence registration"]
    B --> C["Proxy Rendering<br/>Reprojection + Telea hole-filling<br/>→ Proxy Image + Depth Map + Mask"]
    C --> D["Video Diffusion Backbone<br/>Replicated frames + Temporal self-attention<br/>Preserve object identity"]
    subgraph DBD["Dual-Branch Denoising"]
    direction TB
    E["Warm-Start Initialization<br/>Start from intermediate step t_weak"]
    F["Variance-Homogeneous Injection<br/>Foreground = Proxy, Background is free in window"]
    end
    D --> E
    E --> F
    F -->|"After t_strong"| G["Global Harmonization<br/>Free denoising for high-frequency/boundary blending"]
    G --> H["Take last frame<br/>= Editing Result"]

Key Designs

1. Decoupled 3D Reconstruction and Correspondence-aware Alignment: Bypassing the occlusion ambiguity of monocular lifting to place the object precisely in scene coordinates

The biggest challenge in single-image 3D lifting is occlusion—since the back of the object is invisible, directly lifting the entire scene results in a clump of erroneous geometry. GeoEdit addresses this by reconstructing the scene and the object separately: first, VGGT lifts the entire image into an incomplete scene and segments the foreground (visible) and background. Then, multi-view diffusion (SV3D, generating 21 novel views) is run on the isolated foreground to hallucinate the occluded sides, obtaining a geometrically complete point cloud in canonical space. To move the completed object back into the scene, the authors leverage a key observation: both the completed point cloud \(\mathcal{P}_{fg}^{\text{comp}}\) and the visible foreground point cloud \(\mathcal{P}_{fg}^{\text{vis}}\) originate from the same source image, meaning the point at the same pixel coordinate \(\mathbf{u}\) corresponds to the exact same physical surface point. From this, they extract a dense correspondence set \(\mathcal{C}=\{(\mathcal{P}_{fg}^{\text{comp}}(\mathbf{u}),\,\mathcal{P}_{fg}^{\text{vis}}(\mathbf{u}))\mid M(\mathbf{u})=1\}\) and solve the similarity transformation to align the completed object into the global scene via least squares:

\[\mathop{\arg\min}_{\mathbf{R}_{\text{align}},\,\mathbf{t}_{\text{align}},\,s_{\text{align}}}\sum_{(\mathbf{p},\mathbf{q})\in\mathcal{C}}\left\|s_{\text{align}}\,\mathbf{R}_{\text{align}}\,\mathbf{p}+\mathbf{t}_{\text{align}}-\mathbf{q}\right\|_2^2\]

Instead of standard RANSAC, a deterministic least-squares formulation is used, and the top 95% of inliers are retained based on residuals to resist outliers. Once aligned, a unified coordinate system is established, allowing the user to precisely apply \(\mathcal{T}\). This step is highly effective because the "same pixel coordinate indicates the same physical point" prior bypasses arbitrary correspondence searching, aligning the completed object and the real scene seamlessly, providing a foundation for rendering physically plausible proxies.

2. Proxy Rendering: Providing a structurally continuous starting point with correct 3D structures for downstream denoising

After placing the object in 3D, it must be mapped back to 2D. Removing the original visible foreground leaves an exposed background hole in the source image. Rather than using black-box heuristics, the authors employ Telea inpainting based on fast marching (with the neighborhood radius strictly set to 3 pixels to achieve continuity without excessive blurring) to fill the hole, producing a structurally continuous background canvas. The aligned and transformed 3D object is then reprojected onto it, forming the geometrically aligned proxy \(I_{\text{proxy}}\) and the structural depth map \(D_{\text{rep}}\). The key trade-off here is "coarse but correct": the background texture is coarse, but it provides a structural baseline for downstream denoising to synthesize realistic high-frequency detail instead of guessing blindly from pure noise.

3. Dual-Branch Denoising + Variance-Homogeneous Injection: Enabling simultaneous foreground rigidity and background freedom within a single denoising process without disrupting latent statistics

This is the mathematical cornerstone of the paper, directly addressing the pain point that a single trajectory cannot satisfy asymmetric constraints. The authors point out that using a single-timestep initialization like SDEdit leads to a dilemma: setting a large initial noise level grants the prior the freedom to synthesize a realistic background but causes severe drift in the specified 3D skeleton; setting it small strictly preserves the proxy but leaves the background stuck with coarse rendering artifacts. Therefore, denoising is decoupled into asymmetric foreground and background branches. Starting from \(z_{t_{\text{weak}}}\) obtained via warm-start (Design 4), selective latent replacement is performed at each step within the window \(t_{\text{strong}}<t\le t_{\text{weak}}\): instead of using the backbone's prediction or generating raw noise, the foreground region is replaced by the proxy image forward-noised to the current timestep, while the background region retains the backbone's prediction:

\[\tilde{z}_{t-1}=M\odot\Big(\sqrt{\bar{\alpha}_{t-1}}\,\mathcal{E}(I_{\text{proxy}})+\sqrt{1-\bar{\alpha}_{t-1}}\,\epsilon'\Big)+(1-M)\odot z_{t-1}^{\text{pred}}\]

where \(z_{t-1}^{\text{pred}}\) is the video backbone's prediction and \(\epsilon'\) is a fixed Gaussian noise sampled once at the beginning and reused throughout. Outside the window (\(t\le t_{\text{strong}}\)), the constraints are released, allowing the background to evolve freely under the generative prior. The crucial trick in this formula is that "the foreground injection term uses the coefficients \(\sqrt{\bar\alpha_{t-1}}\) and \(\sqrt{1-\bar\alpha_{t-1}}\), which exactly match the variance specified by the noise schedule at timestep \(t-1\)." Because the variance of the injected signal is statistically identical to the state of a native denoising path at that timestep, it is indistinguishable from native trajectories. Consequently, the self-attention mechanism observes a spatially homogeneous distribution, suppressing leakage. Fixing \(\epsilon'\) ensures that the structure of the injected proxy remains consistent across steps. This marks the key difference between this method and Blended Diffusion: the latter performs masked blending without maintaining variance homogeneity, causing distribution mismatches and boundary artifacts. To quantify this, the authors define the Attention Leakage Ratio (ALR)—the proportion of attention weight that background queries erroneously assign to foreground proxy tokens: \(\mathrm{ALR}=\mathrm{mean}_{l,h}\, A^{l,h}_{\mathcal{B}\to\mathcal{F}}/A^{l,h}_{\mathcal{B}\to *}\). At the peak leakage step (\(t=46\)), variance matching reduces the ALR from 8.4% to 6.4%, empirically demonstrating its efficacy in suppressing leakage.

4. Warm-Start Initialization + Video Backbone Temporal Prior: Anchoring global color layout and object identity without fine-tuning

Two auxiliary but indispensable designs are presented together. First, rather than starting from pure noise (\(t=T\)) to avoid semantic ambiguity, the authors encode \(I_{\text{proxy}}\) and add noise to an intermediate timestep \(t_{\text{weak}}\) to serve as the start of the reverse process. This establishes the global color layout and structural baseline right from the beginning, preserving the overall scene identity (ablation shows that removing this drops PSNR from 23.499 to 21.320 and leads to unnatural background synthesis). Second, by using a video diffusion model as the backbone and replicating the proxy into a sequence of identical frames, the temporal self-attention forces cross-frame consistency. This cleverly transforms the identity preservation task (which typically requires dedicated fine-tuning or adapters) into a free-of-charge byproduct, while the depth map serves as a geometric scaffold to prevent skeleton drift during high-noise phases. The authors find that using the backbone's native 81-frame context is necessary to lock the 3D rigidity; cutting the frame count to 1, 21, or 41 leads to structural collapse and "double-image" ghosting. The final frame of the sequence is taken as the output.

Loss & Training

This is a training-free, zero-shot pipeline that does not fine-tune any weights; it only intervenes in the reverse sampling process of the pre-trained DiT. The backbone relies on the standard denoising objective of Latent Diffusion (built into the pre-trained model), and this work introduces no new loss. Key hyperparameters: diffusion schedule of \(T=50\) steps, warm-start threshold \(t_{\text{weak}}=47\), and injection ends at \(t_{\text{strong}}=40\) (heuristically optimal window); CFG scale of 5.0; resolution of \(720\times480\). On a single A800 GPU, the 3D extraction phase takes ~2 minutes with 19 GB VRAM, and the denoising phase takes ~16 minutes with 44 GB VRAM. Object segmentation uses rembg (U2-Net) for single objects and SAM2 for complex multi-object scenes.

Key Experimental Results

Main Results

Zero-shot evaluation was conducted on a fixed subset of 50 stratified pairs from the self-built GeoEditBench, covering reconstruction fidelity (PSNR), identity preservation (DINO/CLIP), perceptual difference (LPIPS/DreamSim), and two newly introduced geometric accuracy metrics: PoseMap IoU (comparing predicted pose maps with the targets) and Object IoU (evaluating silhouette alignment with the target mask). Additionally, AI (VLM) and human preference scores were recorded.

Method PSNR↑ DINO↑ CLIP↑ LPIPS↓ DreamSim↓ PoseMap IoU↑ Object IoU↑ Human Pref.↑
Qwen-Image 14.013 0.685 0.885 0.431 0.185 66.7% 8.8% 2.667
NanoBanana 19.164 0.948 0.976 0.118 0.031 75.0% 25.2% 2.286
3DiT 21.532 0.771 0.941 0.371 0.084 60.0% 33.5% 1.191
Flux-Kontext 20.250 0.841 0.927 0.261 0.112 66.7% 51.3% 2.714
Image Sculpting 22.101 0.802 0.939 0.147 0.104 80.0% 28.2% 2.619
GeoEdit (Ours) 23.499 0.961 0.952 0.114 0.027 94.9% 57.9% 4.810

GeoEdit outperforms existing methods across almost all metrics, achieving the highest PSNR (23.499) and DINO score (0.961), and the lowest LPIPS (0.114) and DreamSim score (0.027). Its advantage in geometric metrics is particularly prominent, with a PoseMap IoU of 94.9% and Object IoU of 57.9%, significantly exceeding the runner-up. While NanoBanana scores slightly higher on CLIP (0.976 vs 0.952), its geometric IoU is modest at 25.2%; GeoEdit delivers the best comprehensive trade-off between geometric correctness, identity preservation, perceptual quality, and background fidelity. It ranks first in both human and AI preference (human score of 4.810, vastly outperforming the second-best Flux-Kontext at 2.714; Krippendorff's \(\alpha=0.74\) indicates reliable inter-rater consistency). The authors of this work also verified generalization on a 30-case subset of the external 3DEdit-Bench: GeoEdit reaches an Object IoU of 0.460, substantially higher than the per-scene SDS-optimized 3DitScene (0.310) and NanoBanana (0.134), while requiring only a fraction of their compute.

Ablation Study

Two sets of ablations are conducted: the selection of the denoising window \((t_w, t_s)\), and the necessity of core components (leave-one-out under the unified 50-pair protocol).

Config PSNR↑ DINO↑ CLIP↑ DreamSim↓ Description
Pure Prior (50,50) 18.479 0.894 0.921 0.057 Pure prior; severe geometric drift
SDEdit Init (47,47) 20.724 0.932 0.921 0.044 Single-timestep initialization; weak geometry
Low Noise Init (40,40) 21.822 0.942 0.906 0.060 Low noise preserves geometry but limits semantics
Full Injection (50,0) 20.012 0.860 0.871 0.050 Continuous full-trajectory injection disrupts diffusion
Pure Proxy (1,1) 19.465 0.812 0.806 0.084 Excessive injection exposes proxy artifacts
Ours (47,40) 23.499 0.961 0.952 0.027 Calibrated window; optimal balance
Component PSNR↑ DINO↑ CLIP↑ LPIPS↓ DreamSim↓ Description
Naive Baseline 16.217 0.835 0.890 0.253 0.049 Distorted pose; blurred background
w/o Warm-Start 21.320 0.943 0.942 0.134 0.043 Unnatural background synthesis; altered lighting
w/o Variance Homogeneity 11.494 0.314 0.682 0.520 0.240 Catastrophic collapse
Ours (Full) 23.499 0.961 0.952 0.114 0.027 Full model

Key Findings

  • Variance-Homogeneous Injection is critical: Removing it (replacing synchronized forward-noised injection with uncalibrated constraints) leads to a catastrophic performance collapse—DINO plunges from 0.961 to 0.314, CLIP drops to 0.682, and DreamSim surges to 0.240, visually manifesting as extreme attention leakage and coarse rendering artifacts. This confirms that "ensuring the injected signal's variance aligns with the noise schedule" is a prerequisite for stable generation rather than an optional detail.
  • The denoising window is a geometry-semantics trade-off curve: A larger initial noise (47) grants sufficient freedom, yielding higher CLIP but lower PSNR and more geometric drift; shrinking it to 40 ensures spatial alignment (PSNR 21.822) but limits semantic diversity (CLIP 0.906). The calibrated \((47, 40)\) window achieves the optimal sweet spot, leading in PSNR, DINO, and DreamSim.
  • Warm-start's contribution is moderate yet essential: Removing it drops PSNR from 23.499 to 21.320 and increases LPIPS from 0.114 to 0.134. Its primary effect is on the naturalness of background synthesis and consistency of global illumination, indicating that it is responsible for anchoring low-frequency color layouts.
  • 81-frame temporal context acts as a rigid anchor: Reducing the frame count to 1, 21, or 41 causes severe geometric deviations and "double-image" ghosting. Only the native 81-frame context can lock the object's spatial rigidity during the injection phase, albeit at the expense of slower inference.

Highlights & Insights

  • Explicitly framing the "asymmetrical generation requirements" is the core insight of this work: Foreground demands rigidity and background demands freedom. Once this dichotomy is clarified, dual-branch denoising follows naturally. Many prior works struggled on a single trajectory precisely because they overlooked this contradiction.
  • Variance-homogeneous injection is a reusable, general trick: In any diffusion-based regional constraint or local editing task, "injecting a latent by forward-noising it to the current timestep to match the noise schedule variance" can effectively prevent self-attention leakage and background blurring. This is vastly more stable than the naive blending in Blended Diffusion.
  • Using temporal self-attention of video backbones to preserve identity for free: Replicating a static image into a frame sequence transforms the "ID preservation" task (which typically requires fine-tuning or adapters) into a built-in byproduct of temporal consistency. This elegant "framing a static problem as a sequence task" concept is highly transferable to other single-image editing paradigms that require consistency.
  • The ALR metric quantifies "leakage": By defining attention leakage as the proportion of attention weight that background queries assign to foreground tokens, the authors quantify "robustness," providing empirical support (8.4% -> 6.4%) for the method's superiority.

Limitations & Future Work

  • Cascaded error propagation: The entire pipeline is bottlenecked by the accuracy of the intermediate 3D point cloud reconstruction. Any errors in monocular depth estimation or multi-view synthesis propagate directly to the final composite; handling dense occlusions (e.g., an object behind thick foliage) remains challenging.
  • Failure under extreme spatial edits: Large translations expose un-occluded regions that exceed the background prior's capabilities, leading to inpainting failure. Large rotations require hallucinative synthesis of unseen backsides, which easily leads to blurriness or inconsistency.
  • High computational overhead and latency: To secure 3D rigidity, the video backbone's native 81-frame context must be used, resulting in a denoising phase of ~16 minutes and requiring 44 GB of VRAM. This is far slower than single-step 2D editors; exploring lightweight fine-tuning and more efficient architectures is a key future direction.
  • Reliance on learned priors without physical rendering: View-dependent lighting, shadows, and specular/glass reflections are synthesized via generative priors rather than explicit physical modeling, making strict simulation difficult. Future work could introduce PBR (Physically Based Rendering) guidance.
  • My Observation: The main evaluation table only uses a fixed 50-pair subset, which is relatively small. Also, PoseMap/Object IoU depend on external estimators, whose estimation errors might contaminate the geometric scores—a caveat for lateral comparisons. Even so, the highest Object IoU is only 57.9%, showcasing that "strict geometric correctness" remains largely unsolved in general.
  • vs 2D mask-based editing (RePaint / InstructPix2Pix / DragDiffusion): These operate entirely on the pixel plane and lack 3D spatial reasoning, making out-of-plane transformations prone to ghosting. This work lifts edits into 3D, keeping the foreground locked to rigid geometry while letting the background generate freely.
  • vs 3D-conditioned diffusion editing (GeoDiffuser / Diffusion Handles / Object-3DIT): Object-3DIT relies on synthetic data and suffers from sim-to-real gaps. This work utilizes monocular lifting to support comprehensive rigid transformations (rotation + scaling) without external 3D software or synthetic data.
  • vs 3DitScene (SDS + 3D Gaussian): While highly competitive in semantic alignment, SDS-based per-scene optimization is computationally heavy, has weak geometric accuracy, and suffers from ghosting. This work is training-free, secures a significantly higher Object IoU, and runs at a fraction of the compute cost.
  • vs BlenderFusion / ObjectMover: BlenderFusion requires external 3D software; ObjectMover uses video priors but is limited to translation. While also leveraging video priors, this work covers the full spectrum of translation, rotation, scaling, and camera movement.
  • vs Time-to-Move / Blended Diffusion / SDEdit: This work adopts the region-dependent scheduling ideology from Time-to-Move but implements it through "variance-homogeneous injection." Strictly maintaining latent statistics to suppress self-attention leakage represents a key improvement over the naive blending of Blended Diffusion and the single-timestep initialization of SDEdit.

Rating

  • Novelty: ⭐⭐⭐⭐ The insight on "asymmetrical constraints" is incredibly clear, and variance-homogeneous injection is an elegant mechanical innovation. However, the overall lifting-rendering pipeline mostly combines existing models, concentrating the core innovation in the denoising stage.
  • Experimental Thoroughness: ⭐⭐⭐⭐ The main table, two sets of ablations, external benchmarks, dual human/AI evaluation, and quantitative ALR analysis cover all bases. Points are deducted for the relatively small 50-pair main evaluation subset and the reliance on external estimators for geometric metrics.
  • Writing Quality: ⭐⭐⭐⭐⭐ The path from problem definition to mechanism to validation is perfectly cohesive. Algorithm 1 and the formulas clearly explain the mechanism, and the target-tradeoff visualizations are highly compelling.
  • Value: ⭐⭐⭐⭐ Enables physically-plausible 3D object editing zero-shot without training, proving highly practical for AR and content creation. The high transferability/reusability of the variance-homogeneous injection trick elevates its overall value ceiling, with latency being the primary drawback.