OREO: Fidelity Alignment in 3D Generation via On-the-fly Rendering-Editing Optimization

Conference: ECCV 2026
Paper: ECCV Official Link
Project Page: https://theericma.github.io/oreo/
Area: 3D Vision
Keywords: 3D Generation, Alignment, Image Diffusion Prior, Inversion-Free Editing, Contrastive Distillation

TL;DR

OREO establishes a closed-loop "Render-Edit-Optimize" framework that dynamically generates structure-preserving, high-fidelity 2D pseudo ground truths via reinforced editing and distills visual fidelity gains into 3D feed-forward generators via latent contrastive optimization without gradient attenuation through differentiable renderers.

Background & Motivation

Recent feed-forward 3D generative backbones, such as Trellis and the Hunyuan3D family, have revolutionized digital asset creation by enabling rapid, scalable synthesis of 3D shapes. However, while these models demonstrate impressive geometric competence, they frequently suffer from a noticeable lack of visual fidelity. The synthesized assets often display flat, cartoonish surface textures, lacking the micro-details, subtle wear, nuanced lighting reflections, and material richness found in real-world objects. This fundamental bottleneck originates from the acute scarcity of high-quality, photorealistic 3D assets: existing synthetic 3D repositories like Objaverse pale in comparison to billion-scale 2D image corpora, leaving 3D generators starved of realistic visual priors.

To bridge this dimensional divide, distilling the rich perceptual priors of pre-trained 2D diffusion models into 3D representations has emerged as a promising avenue. Nevertheless, existing paradigms face profound practical limitations. Optimization-based 3D editing approaches (e.g., Instruct-NeRF2NeRF, GaussianEditor) optimize individual assets per scene, rendering their learned fidelity non-amortizable into feed-forward generators. Conversely, score-distillation techniques (e.g., DreamFusion, ProlificDreamer, DMD) supervise 3D models via implicit score gradients evaluated across noise scales; these gradients exhibit high variance and directional conflict, frequently triggering color saturation collapse and severe geometric degradation during post-training. Furthermore, directly deploying pre-trained 2D image editors to generate reference targets often inadvertently mutates the camera viewpoint, foreground scale, or object skeletal pose, producing supervisory signals that conflict with the underlying 3D mesh.

This paper tackles the challenge by formulating a closed-loop, on-the-fly self-supervision paradigm: since 2D diffusion editors possess unmatched photorealistic generation capabilities, their trajectory can be constrained to create dynamic pseudo ground truths that faithfully preserve 3D geometry while injecting fine textures. Core idea: establish an on-the-fly "Render-Edit-Optimize" loop that produces structure-preserving pseudo ground truths via inversion-free Reinforced Editing with dynamic noise updates, and distills the fidelity gap into the 3D generator via latent-space contrastive supervision between edited positive and raw negative anchors.

Method

Overall Architecture

The OREO framework executes a closed-loop alignment cycle during each training iteration. Given an unposed reference image \(x^\text{ref}\), the 3D latent generator under training executes an on-policy ODE rollout to produce clean latent features, which are decoded and projected from a randomly sampled camera viewpoint \(\pi\) into a 2D source rendering \(x^\text{src}\). Subsequently, the Reinforced Editing module conditions a pre-trained 2D image editor on the reference view and prompt to inject realistic textures while strictly preserving the viewpoint and layout, yielding a pseudo ground truth \(x^\text{tgt}\). Finally, rather than propagating backpropagation gradients through a non-linear, lossy differentiable renderer, the original rendering \(x^\text{src}\) and the refined target \(x^\text{tgt}\) serve as negative and positive conditioning anchors inside the latent space of the frozen pre-trained 3D generator, optimizing the trainable generator parameters via a contrastive distillation objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Image Input<br/>x_ref"] --> B["On-Policy Rollout & Prior Regularization<br/>Generate 3D asset and render source view x_src"]
    B --> C["Reinforced Editing Pseudo-GT Construction<br/>Decouple unconditional branch and inject guided noise"]
    C --> D["Latent-Space Contrastive Distillation<br/>Construct z_0^+ and z_0^- and compute contrastive loss"]
    D -->|Update 3D Generator Weights| B

Key Designs

1. Reinforced Editing Pseudo-GT Construction: Decoupled unconditional branching with dynamic noise alignment

Standard image editing models tend to alter camera perspective or object posture when instructed to enhance appearance (e.g., snapping side-view characters back to frontal poses or altering limb stances). To ensure the supervisory edits remain strictly aligned with the underlying 3D geometry, Reinforced Editing builds upon the inversion-free FlowEdit mechanism. Since the rendered view \(x^\text{src}\) lacks an independent source text description, the method replaces the source branch velocity with an unconditional prediction, while guiding the target trajectory with the reference image and a camera rotation prompt \(c^\text{ref}\): $$ \tilde{v} = v_\phi^w(x_t^\text{tgt}; t, c^\text{ref}) - v_\phi(x_t^\text{src}; t, \emptyset) $$ Moreover, because the stochastic Gaussian noise \(\epsilon\) shared across branches can dilute the editing strength during intermediate sampling steps, the algorithm dynamically injects classifier-free guidance (CFG) gradients into the noise term at each step: $$ \epsilon \leftarrow \epsilon - \left(v_\phi(x_t^\text{tgt}; c^\text{ref}) - v_\phi(x_t^\text{tgt}; \emptyset)\right) \cdot (1 - t) $$ This directional noise update aligns perturbation vectors directly with the conditional guidance trajectory, delivering sharp, realistic textures while maintaining strict spatial and silhouette consistency with \(x^\text{src}\).

2. Latent-Space Contrastive Distillation: Renderer-free dual-anchor optimization

Directly applying pixel-space regression losses (e.g., \(\|x^\text{src} - x^\text{tgt}\|_2^2\)) requires computing gradients through both the differentiable renderer \(P\) and the non-linear 3D mesh decoder \(D\). In practice, the resulting gradient attenuation causes optimization to stagnate into blurry, color-shifted averages. OREO bypasses the renderer by elevating supervision into the latent space of the frozen generator \(v_{\theta_\text{pre}}\).

Using the current single-step clean prediction \(\hat{z}_0\) as the anchor, the model perturbs it with independent noise \(\epsilon'\) into \(\hat{z}_t\) and runs two single-step conditioning passes through the frozen generator using \(x^\text{tgt}\) and \(x^\text{src}\): $$ z_0^+ = \hat{z}t - t \cdot v}}(\hat{zt; t, x^\text{tgt}), \quad z_0^- = \hat{z}_t - t \cdot v}}(\hat{zt; t, x^\text{src}) $$ Detaching \(z_0^+\) and \(z_0^-\) defines a pair of positive and negative anchors. The latent contrastive loss pulls the generator prediction toward the texture-rich positive anchor while repelling it from the fidelity-deficient negative anchor: $$ \mathcal{L}\text{contrast} = |\hat{z}_0 - z_0^+|^2 - |\hat{z}_0 - z_0^-|^2 $$ This enables direct, stable parameter updates in the 3D latent space without propagating gradients through 2D pixel rendering.

3. On-Policy Rollout & Prior Regularization: Mitigating distribution shifts and geometric collapse

Operating under an off-policy regime—where renderings are generated by the frozen pre-trained generator rather than the evolving model—decouples the anchor from the active optimization path, empirically resulting in washed-out palettes and smoothed surface features. OREO enforces strictly on-policy ODE rollouts \(z_0 = G_\theta(x^\text{ref})\) to keep supervisory targets synchronized with the generator's current capability.

Furthermore, while the contrastive objective drives texture realism, unconstrained fine-tuning risks eroding the model's fundamental geometric stability. To prevent structural drifting, a velocity regularization loss penalizes deviations from the pre-trained flow field: $$ \mathcal{L}\text{reg} = |v\theta(z_t) - v_{\theta_\text{pre}}(z_t)|^2 $$ This prior regularization anchors the generator within its verified 3D manifold, ensuring that fidelity improvements do not compromise underlying geometry.

Loss & Training

The overall training objective combines the latent contrastive distillation loss with the velocity regularization penalty: $$ \mathcal{L}\text{total} = \mathcal{L}\text{contrast} + \lambda \mathcal{L}_\text{reg} $$ where \(\lambda\) controls the regularization weight. Hyper-parameter tuning sets the editing schedule ratio to \(N_e / N = 0.75\) with \(N_e = 9, N = 12\), striking an optimal trade-off between texture sharpness and silhouette fidelity. The model is fine-tuned end-to-end on 2,296 unposed reference images from the Conceptual Design Dataset without requiring any ground-truth 3D meshes or multi-view captures.

Key Experimental Results

Main Results

OREO is implemented on top of the Trellis 3D generator backbone, utilizing Qwen-Image-Edit as the 2D feedback editor. Quantitative evaluations are conducted on the curated Conceptual Design Dataset (featuring 2,396 intricate, out-of-distribution visual concepts) and the standard Google Scanned Objects (GSO) benchmark. Metrics report the mean CLIP and DINO embedding similarities to reference images across 5 rendered viewpoints:

Dataset	Metric	Trellis (Baseline)	Photo3D	OREO (Ours)	Relative Gain
Conceptual Design	CLIP Sim. ↑	0.7613	0.7380	0.7834	+0.0221
Conceptual Design	DINO Sim. ↑	0.7916	0.7837	0.8065	+0.0149
GSO	CLIP Sim. ↑	0.7722	0.7512	0.7764	+0.0042
GSO	DINO Sim. ↑	0.7022	0.7034	0.7069	+0.0047

In evaluating 2D supervisory feedback sources, Reinforced Editing demonstrates superior structural consistency (quantified by foreground Mask IoU) while simultaneously lifting semantic similarity toward the reference image:

2D Feedback Source	CLIP Sim. (to \(x^\text{ref}\)) ↑	DINO Sim. (to \(x^\text{ref}\)) ↑	CLIP Sim. (to \(x^\text{src}\)) ↑	DINO Sim. (to \(x^\text{src}\)) ↑	Mask IoU ↑
Unedited (Raw Rendering)	0.7613	0.7915	N/A	N/A	N/A
NanoBanana Pro	0.8803	0.9001	0.8199	0.8404	0.6830
Qwen-Image-Edit (Orig.)	0.7753	0.8269	0.7882	0.8084	0.6374
Reinforced Editing (Ours)	0.7992 (+0.0379)	0.8214 (+0.0299)	0.8833	0.9099	0.9520

Ablation Study

Ablations on the 3D generator training designs on the Conceptual Design Dataset confirm the necessity of each architectural component:

Config	CLIP Sim. ↑	DINO Sim. ↑	Note
Pretrained Trellis (Baseline)	0.7613	0.7916	Base pre-trained model
Full OREO (Full Model)	0.7834	0.8065	Complete on-policy + latent contrastive distillation
w/ Pixel-MSE Supervision	0.6617	0.6982	Gradient attenuation via renderer causes over-smoothing
w/ Off-policy Rollout	0.7279	0.7613	Static trajectory induces distribution drift and desaturation
w/ DMD (Score Distillation)	0.5393	0.5486	Single-step score gradient variance leads to geometric collapse

Ablations on key components of the Reinforced Editing pipeline:

Editor Variant	\(\Delta\)CLIP Sim. ↑	\(\Delta\)DINO Sim. ↑	Note
Full RE (Full Model)	+0.0379	+0.0299	Robust texture enhancement with structural alignment
w/o Source Branch	-0.0523	-0.0497	Trajectory drift causes severe over-saturation and distortion
w/o Noise Update	-0.0001	+0.0089	Random noise resampling dilutes editing guidance signals

Key Findings

• Failure of implicit score distillation: Replacing explicit pseudo-GT targets with single-step score distillation (w/ DMD) causes CLIP similarity to plunge to 0.5393, accompanied by severe geometric fragmentation and color collapse. This confirms that high-variance diffusion gradients cannot substitute for coherent multi-step editing trajectories.
• Differentiable rendering creates a gradient barrier: Supervising directly via pixel MSE (w/ Pixel-MSE) severely degrades performance below the baseline (CLIP 0.6617), verifying that lifting supervision into latent contrastive space is essential to bypass non-linear rendering attenuation.
• Editing step saturation threshold: Evaluating step configurations indicates that fidelity gains plateau around \(N_e = 9\); allocating additional steps (15 or 30) yields negligible perceptual improvements on facial and textural details while steadily degrading Mask IoU due to accumulated drift.

Highlights & Insights

• Bypassing the differentiable renderer via latent contrastive pairs: Conventional 3D optimization tightly couples supervision to differentiable rendering layers, often suffering from vanishing gradients and excessive VRAM usage. OREO's formulation converts 2D image edits into latent positive/negative anchors inside the pre-trained generator, establishing an elegant, renderer-free post-training paradigm.
• Condition-aligned dynamic noise steering: In flow-based editing, stochastic Gaussian noise frequently counteracts subtle conditional edits. OREO's noise update rule dynamically injects the CFG differential vector into the noise space, amplifying editing effectiveness without sacrificing structural layout.
• Targeted domain enhancement without catastrophic forgetting: While delivering large visual gains on challenging out-of-distribution concept art (+0.0221 CLIP), OREO preserves generation performance on the in-distribution GSO benchmark, verifying that flow field regularization effectively preserves general 3D priors.

Limitations & Future Work

• Computational overhead during training rollout: Executing on-policy ODE generation combined with multi-step FlowEdit iterations per training step incurs significant training time compared to standard offline supervised fine-tuning; exploring one-step distilled editing models is an important next step.
• View-independent synthesis limitations: The current framework edits single rendered viewpoints independently. Extending this to multi-view or panoramic editors would further guarantee multi-angle textural consistency in complex, large-scale scene generation.
• Absence of explicit PBR material decomposition: The fidelity enhancements currently target raw RGB radiance fields; explicitly decoupling physically based rendering (PBR) attributes—such as albedo, roughness, metallic, and normal maps—remains an open direction for future graphics pipeline integration.

Related Work & Insights

• vs Photo3D / 3D Generator Post-training: While Photo3D attempts post-training on Trellis, it struggles on complex conceptual designs (CLIP 0.7380); OREO achieves substantial gains (CLIP 0.7834) by pairing dynamic on-policy rollouts with explicit structure-preserving pseudo ground truths.
• vs DreamFusion / Score Distillation (DMD): Score distillation relies on accumulating noisy, local single-step gradients across diffusion time steps; OREO replaces stochastic score forces with a deterministic, multi-step edited target optimized via smooth latent contrastive bounds.
• vs Instruct-NeRF2NeRF / GaussianEditor: Instance-level 3D editing methods necessitate thousands of optimization steps per individual 3D scene; OREO amortizes the 2D visual prior into a feed-forward generator, enabling instant single-forward-pass inference on unseen reference inputs.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly unites inversion-free reinforced editing with latent contrastive distillation for closed-loop 3D generator post-training.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 2D feedback fidelity, 3D asset metrics, step sensitivity, and generator design ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear narrative structure, and insightful experimental analyses.
• Value: ⭐⭐⭐⭐☆ Provides a practical, highly generalizable methodology for infusing 2D foundation model priors into feed-forward 3D generation.

Related Papers

• [ECCV 2026] ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation
• [ECCV 2026] TanGO: Training-Free 3D Editing via Tangent-Space Guidance and Optimization
• [ECCV 2026] Edit3r: Instant 3D Scene Editing from Sparse Unposed Images
• [CVPR 2026] VIAFormer: Voxel-Image Alignment Transformer for High-Fidelity Voxel Refinement
• [CVPR 2026] LATTICE: Democratize High-Fidelity 3D Generation at Scale