BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Image Generation / 3D Vision
Keywords: 3D-aware editing, generative compositing, dual-stream diffusion, source masking, object-camera disentanglement
TL;DR¶
BlenderFusion lifts image objects into editable geometry, expresses precise editing intent in Blender, and completes and fuses the scene with a dual-stream diffusion compositor, outperforming adapted Object 3DIT and Neural Assets on three video datasets while supporting fine-grained compositional edits beyond its training operations.
Background & Motivation¶
Generating an attractive picture is different from editing an existing one. Rotating a cup, shrinking a chair, and replacing the background while keeping the camera fixed and other objects unchanged requires control over individual geometry and identity. Text prompts or coarse 2D layouts rarely specify these constraints precisely. Object 3DIT primarily targets rigid transformations of a single object. Neural Assets supports multiple objects and camera control, but compressed object tokens can lose details or mix in background information. Image Sculpting already uses Blender, yet relies on per-scene optimization and primarily handles a single object from one image.
Graphics software alone does not solve the entire problem. Real photographs often provide only one viewpoint, so reconstructed meshes lack back surfaces and complete textures, producing holes or unnatural surfaces after rotation. Materials and lighting needed for full physical rendering may also be unavailable. Conversely, a diffusion model asked to infer geometry on its own may imitate object movement by moving the background. That can fit training videos while violating the instruction to move only one object. Objectron is a particularly relevant example: objects are mostly static and the camera moves, so ordinary cross-frame training contains this shortcut.
The paper therefore does not ask a generative model to learn every Blender operation. Blender supplies visible, explicit target geometry, and the model focuses on refinement and fusion. To preserve source identity without clinging to an obsolete layout, training must also teach when to discard source content and how to respond to object changes under a fixed camera. Core Idea: assign geometric control to explicit 3D editing and photorealistic compositing to dual-stream diffusion, using source masking and simulated object jittering to break source-copying and camera-motion shortcuts.
Method¶
Overall Architecture¶
The inputs are one or more images containing objects, together with object categories, 3D boxes, and camera information. The output is a 2D composite following the edited layout. Geometry Layering and Editing is followed by Source Masking to prepare valid context, after which Dual-stream Generative Compositing jointly uses the source scene and target render. Simulated Object Jittering constructs training conditions only; it is not a mandatory inference step for every edit.
The source stream retains the original image, source render, source camera, and object poses. The target stream contains the target render, target camera, target object poses, and the latent variable being denoised. Both renders originate from the same reconstructed objects. The model can therefore see both the original appearance and the explicitly requested geometric change, instead of guessing rotation directions or spatial relations from a description.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input images<br/>3D boxes and cameras"] --> B["Geometry Layering and Editing"]
B -->|Source image and render| C["Source Masking"]
C -.->|Training-only condition recombination| D["Simulated Object Jittering"]
B -->|Target render and poses| E["Dual-stream Generative Compositing"]
C -->|Valid source context at inference| E
D -.->|Fixed-camera training conditions| E
E --> F["Edited image"]
G["Target video frame"] -.->|Training-only supervision| E
Geometry here is a control interface, not a guarantee of a complete and correct real-world scene model. Default reconstruction covers surfaces visible from one image; the diffusion model adds missing textures and geometry and blends the background on top of this imperfect interface. Material or part-level edits can use more complete image-to-3D reconstructions without changing the subsequent compositor input format.
Key Designs¶
1. Geometry Layering and Editing: turn user operations into a visible target render
Known 3D boxes are first projected into 2D. Grounding DINO receives category prompts, and its prediction replaces the projected box only when their IoU exceeds 0.5. The refined box then prompts SAM2 to extract an object mask. Depth Pro predicts metric depth, the masks isolate instance-wise depth, and the corresponding 3D boxes calibrate scale. Back-projecting these depths into point clouds and connecting adjacent points produces a triangle mesh for each object. This is not a complete solution for automatically recovering every scene parameter from arbitrary photos: categories, boxes, and camera conditions are important parts of the experimental setup.
Once individual meshes enter Blender, translation, rotation, scaling, removal, and insertion become explicit operations. The camera object controls viewpoint changes, and a new background image replaces the source image for background replacement. RGB and Object Index Pass instance masks are rendered before and after editing to provide source and target conditions. Moving a cup is thus represented by its target silhouette, occlusion, and projected position, rather than only tokens that the model may misinterpret. When detailed deformation or material editing requires a more complete mesh, the progressive editing demonstrations use Hunyuan3D v2; the paper does not claim that default shallow meshes suffice for every operation.
2. Source Masking: preserve identity references while invalidating obsolete context
The source image is both a useful appearance reference and a potential obstacle to editing. After object removal, leaving the original image intact may cause the model to draw the object back. During background replacement, the old background should no longer be treated as reliable evidence. Training masks each object in the source image and source render with probability 0.5. Background regions are also randomly masked to avoid a fixed inpainting bias. The model must consequently read the target render instead of simply copying the source image or relying on relative camera pose.
At inference, masking depends on the editing task: invalid regions can be masked in both the source image and source render, or only in the source image. The distinction is whether only the old appearance context, or also the source geometric reference, should be disregarded. This does not mean erasing all source information or masking the target render; the target branch must still express the intended retained or added scene content. The main text does not enumerate every operation-specific masking rule, so no single mask pattern should be presented as a universal configuration.
3. Simulated Object Jittering: recombine conditions to teach object changes under a fixed camera
Ordinary training samples source and target video frames, transforms source reconstructions using object boxes and camera parameters, and supervises compositing with the target frame. The difficulty is that Objectron objects do not actually move, encouraging the model to attribute appearance changes to the camera. This fails when the user rotates an object while leaving the background fixed. The proposed jittering strategy does not fabricate a new dynamic video with real ground truth. Instead, it replaces the source-stream image and camera parameters with the target frame and target camera while retaining the geometric differences represented by the source render and the source and target object boxes.
The target-frame context and source render are then masked independently, and the compositor reconstructs the target image using the noisy target render. Image context and target now share the same camera, while the source render still supplies object references. The model must use object-pose conditions to recover the image rather than explain all change as camera movement. Masking also prevents simply copying the visible target frame. This simulates disentangled object changes through training conditions; it should not be confused with adding random jitter to the user's edit at test time.
4. Dual-stream Generative Compositing: share generation priors while exchanging source and target information explicitly
The compositor adapts a pretrained diffusion model, with Stable Diffusion v2.1 used in the experiments. Both streams share weights and process their own inputs, exchanging information through self-attention over features from both streams. The source image helps preserve identity and detail, while the coarse target render specifies the new layout. The model learns to fill regions exposed by transformations of incomplete meshes and fuse objects with the background. The two streams are not independently trained generators, nor does one model generate objects before a second model pastes them onto a background.
Each stream's input expands from 4 to 15 channels: 4 image or noise latent channels, 4 render latent channels, 1 instance-mask channel, and 6 Plücker camera-embedding channels, giving \(4+4+1+6=15\). Weights for the additional channels are zero-initialized. Categories are CLIP-encoded, while 3D boxes undergo positional encoding and an MLP; the resulting object tokens replace the text-conditioning sequence. The implementation uses the source camera as the coordinate reference and represents projected box corners with image coordinates and depth. Pixel-level renders, instance identity, camera rays, and box-level poses thus supply distinct forms of conditioning instead of forcing a single feature to encode every control signal.
A Worked Example¶
Consider the illustrative request to keep the camera fixed, rotate a cup on a table, and preserve the chair and background. Layering creates separate cup and chair meshes, and Blender rotates only the cup before rendering again. The target instance mask and RGB render express the cup's new silhouette, while chair and camera conditions remain unchanged. This example explains the pipeline; it is not an additional quantitative sample reported in the paper.
The compositor reads the cup's original color, texture, and surroundings from the source stream and its rotated position and shape from the target stream, completing surfaces hidden in the original view. If the next operation removes the cup, Source Masking can invalidate its old context so the compositor does not restore it. Editing the handle's shape may exceed what the default single-view mesh supports, requiring a more complete reconstruction. An operation being available in the interface does not imply that every input reconstruction can support it reliably.
Loss & Training¶
Real target video frames supervise conditional diffusion compositing. The main additions concern input structure and the two training-condition strategies. The available main text does not specify the full loss formula, all condition-drop probabilities, or the jitter sampling distribution. No dedicated geometric loss is therefore invented here, and a generic diffusion objective is not presented as the authors' explicitly reported exact formula.
The compositor trains on 8 NVIDIA A100 80GB GPUs with batch size 320 for 30,000 iterations. AdamW uses weight decay 1e-2 and a 500-step linear warmup. Learning rates are 5e-5 for the diffusion model and 1e-4 for the box-encoding MLP. Inference uses 50 DDPM sampling steps with classifier-free guidance scale 2.0.
Cars in WOD primarily translate, with insufficient rotational diversity. All three compared methods therefore initialize WOD training from MOVi-E pretraining and reduce the base learning rate to 1e-5. This shared initialization is a condition of the WOD results, not evidence that real driving videos alone provide sufficient disentangled rotation supervision.
Key Experimental Results¶
Main Results¶
Standard evaluation transforms a source video frame into a target frame given camera and object poses. MOVi-E comprises 10,000 videos generated with increased numbers of dynamic objects and a wider camera-motion range, at 512×512 resolution. Objectron contains 15,000 clips across 9 categories and uses 384×512 images. WOD contains 1,000 videos, uses the front-view camera, filters small cars, and uses 528×352 images. These figures describe generation scale or dataset totals, not test-set sizes; the main text does not provide the complete training/test split.
The table selects key columns from the paper's Table 2. PSNR measures pixel reconstruction quality in dB, with higher values better. LPIPS measures perceptual distance, with lower values better. FID measures feature-distribution differences between generated and real images, with lower values better. Object 3DIT is adapted to SD v2.1 with camera embeddings and multi-object conditions; it is not the unmodified original single-object model.
| Dataset | Method | Object PSNR ↑ | Object LPIPS ↓ | Frame PSNR ↑ | Frame FID ↓ |
|---|---|---|---|---|---|
| MOVi-E | Object 3DIT | 14.06 | 0.411 | 17.02 | 15.71 |
| MOVi-E | Neural Assets | 13.74 | 0.428 | 16.73 | 23.08 |
| MOVi-E | BlenderFusion | 18.90 | 0.227 | 21.32 | 9.11 |
| Objectron | Object 3DIT | 13.88 | 0.424 | 14.98 | 6.14 |
| Objectron | Neural Assets | 13.73 | 0.427 | 14.56 | 6.18 |
| Objectron | BlenderFusion | 16.06 | 0.291 | 16.54 | 3.25 |
| WOD | Object 3DIT | 18.90 | 0.255 | 23.21 | 11.92 |
| WOD | Neural Assets | 16.87 | 0.322 | 20.41 | 15.39 |
| WOD | BlenderFusion | 20.93 | 0.185 | 24.11 | 10.02 |
Standard cross-frame metrics do not directly measure moving only one object under a fixed camera. Table 3 separately compares object-identity features before and after editing across 60 disentangled and fine-grained control samples. On Objectron, DINO/CLIP cosine similarities increase from Neural Assets' 0.748/0.717 to 0.866/0.810. Higher similarity indicates closer features, not necessarily perfectly correct geometric editing.
The user study in Table 4 contains 54 examples: 18 video-frame transformations, 24 disentangled controls, and 12 fine-grained compositions. There are 1,294 choices from 24 participants. In pairwise comparisons against Neural Assets, BlenderFusion receives 87.04% of choices overall and 93.75% for fine-grained compositing. These are human preference rates, not task success rates.
Ablation Study¶
Figure 8 provides qualitative component ablations. Adding only dual streams with depth and segmentation conditions mainly improves translation. Adding Blender renders can still move the background together with the object. Source Masking improves object control but leaves camera-object coupling; adding Simulated Object Jittering further mitigates fixed-camera editing failures. The available main text does not provide individual numerical scores for these variants, so no exact PSNR gain can be attributed to a particular component.
The following table reproduces the upstream-error analysis from Table 5 under the same Objectron standard video evaluation, not substitute numerical results for the component ablations above. O-DINO is per-object DINO feature cosine similarity. Depth and mask perturbations are applied before mesh construction, with mask noise implemented through erosion/dilation.
| Config | Object PSNR ↑ | Object LPIPS ↓ | O-DINO ↑ | Frame PSNR ↑ | Frame LPIPS ↓ |
|---|---|---|---|---|---|
| Neural Assets | 13.73 | 0.427 | 0.921 | 14.56 | 0.427 |
| BlenderFusion default | 16.06 | 0.291 | 0.959 | 16.54 | 0.323 |
| Depth noise 15% | 15.69 | 0.307 | 0.957 | 16.24 | 0.334 |
| Depth noise 30% | 15.28 | 0.325 | 0.955 | 15.98 | 0.344 |
| Mask noise 10px | 15.58 | 0.309 | 0.957 | 16.12 | 0.337 |
| Mask noise 20px | 15.15 | 0.326 | 0.955 | 15.83 | 0.348 |
Key Findings¶
- MOVi-E frame PSNR reaches 21.32 dB, exceeding the adapted Object 3DIT's 17.02 dB by 4.30 dB. Complex multi-object changes make the value of explicit rendering conditions particularly visible.
- Depth noise of 30% reduces object PSNR from 16.06 to 15.28 dB, still above Neural Assets' 13.73 dB. This supports robustness to moderate upstream errors, not lossless recovery from severely missing surfaces or heavy occlusion.
- On Objectron, the full pipeline takes 13.08 seconds per image on A100 with bf16 at 384×512 and about 20GB peak GPU memory: 0.6 seconds for layering, 3.0 seconds for Blender scripting, and 9.48 seconds for compositing. The reported Neural Assets time of 5.94 seconds uses 256×256, so this is not an equal-resolution speed comparison.
Highlights & Insights¶
- A coarse render is a control signal, not the final product. It provides visible evidence of where and how geometry should change, leaving the generative prior to fill textures and fuse content instead of also guessing editing intent.
- More complete source context is not always better. Removing obsolete context separates identity reference from edit compliance, a useful direction for other reference-image-driven local generation tasks.
- Camera motion in a dataset is not equivalent to the object operations users request. Simulated Object Jittering matters because it changes statistical relationships among conditions, not merely because it adds random augmentation.
Limitations & Future Work¶
- The authors explicitly note that severe mask or depth errors cause missing geometry, making large rotations and heavy occlusions difficult. Insufficient pose diversity in real driving videos also makes synthetic pretraining important.
- The authors acknowledge imperfect physically accurate relighting and shadows. Stronger reconstruction, richer synthetic data, and explicit lighting supervision are proposed directions. Generalization to SUN-RGBD, ARKitScenes, and Hypersim is primarily qualitative, with degraded visual quality relative to in-domain results.
- This note's assessment: reliance on 3D boxes and camera conditions, plus more complete reconstruction for some demonstrations, means the system is not equivalent to zero-preparation editing of arbitrary photos. Roughly 13 seconds per image should not be described as real-time generation.
- This note's assessment: geometric error, occlusion consistency, and costs under different reconstruction models need further reporting to distinguish greater visual preference from strict compliance with physical editing constraints.
Related Work & Insights¶
- vs Object 3DIT: The baseline conditions editing on semantics and poses, whereas BlenderFusion directly supplies before-and-after renders. Even with multi-object and camera conditions added to the baseline, fixed-camera editing can expose motion coupling learned during training.
- vs Neural Assets: Neural Assets extracts object-appearance tokens using DINO/RoIAlign. BlenderFusion retains the source image and explicit renders, reducing appearance compression and foreground-background mixing at the cost of geometry processing and dual-stream computation.
- vs Image Sculpting: Both exploit Blender's editing capabilities. BlenderFusion extends control to multiple objects, cameras, and backgrounds and trains a general compositor. Image Sculpting is excluded from the main quantitative comparison because of per-scene optimization and a different task scope, so no matched-setting numerical advantage over it is established here.
Rating¶
- Novelty: 4/5, the contribution combines an explicit geometry interface, dual-stream conditioning, and disentangled training rather than introducing a new diffusion backbone.
- Experimental Thoroughness: 4/5, three datasets, a user study, and upstream-noise analysis provide breadth, but quantitative geometric validation of real fine-grained edits remains limited.
- Writing Quality: 4/5, the problem and failure modes connect clearly, while some reproduction details require supplementary material.
- Value: 4/5, a practical interface design for controllable image editing, with reconstruction quality and latency limiting direct deployment.