GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

Conference: ECCV 2026
Paper: ECCV Paper
Project: https://igl-hkust.github.io/GO-Renderer
Area: Video Generation
Keywords: generative rendering, 3D geometric proxy, video diffusion model, multi-view consistency, object relighting

TL;DR

GO-Renderer proposes a generative object rendering framework that tightly couples feedforward coarse 3D geometric proxies with a video Diffusion Transformer (DiT), leveraging object-centric 3D coordinate maps and Negative RoPE Shifts to bypass brittle physical inverse rendering and synthesize high-fidelity, multi-view consistent object videos along arbitrary trajectories and novel lighting environments.

Background & Motivation

Synthesizing photorealistic objects in diverse environments with unconstrained viewpoint navigation is a cornerstone pursuit across film production, virtual advertising, and immersive spatial computing. Traditional "reconstruct-then-render" pipelines predominantly follow two directions: the first leverages neural radiance fields (NeRF) or 3D Gaussian Splatting (3DGS) for novel view synthesis, but these representations inherently bake captured environmental illumination into their parameters, demanding brittle and ill-posed inverse rendering optimizations to decouple materials from lighting; the second relies on feedforward 3D reconstruction networks to obtain geometric meshes, followed by multi-stage optimization of Physically Based Rendering (PBR) material textures, which easily suffers from compounding geometric errors and loss of fine-grained, non-Lambertian highlights.

Conversely, reference-based video diffusion foundation models exhibit remarkable generative priors for synthesizing realistic lighting, reflections, and rich surface appearances without explicitly resolving complex light transport equations. However, general-purpose video generation models inherently lack spatial determinism and strict camera pose controllability. When conditioned exclusively on a single reference image, unseen regions are subject to unconstrained model hallucination, causing severe geometric distortions, anatomical deformation, and flickering across frames whenever the camera moves along dynamic trajectories.

The fundamental tension stems from this dichotomy: explicit 3D representations guarantee strict multi-view spatial consistency but struggle with faithful physical material acquisition; 2D video diffusion foundation models provide photorealistic lighting and appearance synthesis but lack 3D spatial determinism. Core Idea: Use a fast feedforward coarse 3D reconstruction as a geometric structural proxy, project it into object-centric coordinate maps to establish dense feature-lookup channels within a video Diffusion Transformer, and decouple reference conditioning from video frames via Negative RoPE Shifts, thereby bypassing explicit inverse material rendering through generative priors.

Method

Overall Architecture

Given sparse multi-view reference images of an object \(\mathcal{I} = \{I_j\}_{j=1}^N\), a sequence of target camera poses \(\mathcal{P}^{\text{target}} = \{P_k^{\text{target}}\}_{k=1}^M\), descriptive text prompts \(c\), and optional appearance guidance videos \(A\), GO-Renderer renders a photorealistic, multi-view consistent video \(x\). The pipeline unfolds in two core stages: geometric proxy construction and joint reference-guided video diffusion generation. First, a rapid feedforward 3D reconstruction backbone (such as ReconViaGen or VGGT) estimates relative reference camera poses and extracts a coarse 3D mesh or point cloud \(\mathcal{O}_{\text{recon}}\). This coarse proxy is rasterized from both reference viewpoints and target camera paths to form object-centric 3D coordinate maps. Subsequently, in the video Diffusion Transformer (DiT), reference images are paired with reference coordinate maps along the channel dimension, while target coordinate maps are channel-wise concatenated with latent noise, guided by a Negative RoPE Shift to prevent temporal interpolation artifacts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Reference Images I + Target Trajectory P_target + Text Prompt c"] --> B["Feedforward Coarse 3D Reconstruction<br/>Fast pose estimation and coarse mesh/point cloud O_recon"]
    B --> C["Object-Centric 3D Coordinate Map Projection<br/>Rasterize normalized coordinate maps C_ref and C_target"]
    C --> D["Joint Reference Spatial Concatenation<br/>Pair I with C_ref / Pair target noise with C_target"]
    D --> E["Negative RoPE Shift Temporal Isolation<br/>Assign negative discrete temporal indices -N*g to reference latents"]
    E --> F["Video Diffusion Transformer Denoising<br/>Dense cross-attention and spatial appearance lookup"]
    F --> G["VAE Decoding into High-Fidelity Object Video"]

Key Designs

1. Object-Centric Coordinate Map Projection: Unambiguous Pixel-Level Spatial Bridges
Generic 2D video diffusion models struggle with precise viewpoint control due to the absence of direct geometric correspondences between 2D pixels and physical 3D space. To supply explicit, continuous geometric guidance, GO-Renderer rasterizes the coarse 3D proxy \(\mathcal{O}_{\text{recon}}\) into dense normalized coordinate maps. Within foreground pixels, the RGB channels encode the normalized \((x, y, z) \in [-1, 1]^3\) coordinates in the object's local canonical coordinate frame. Because local coordinates are invariant under global camera transformations, identical surface points on the object share identical numerical coordinates across both reference coordinate maps \(C_{\text{ref}}\) and target coordinate maps \(C_{\text{target}}\). This allows the DiT attention layers to operate like a memory lookup table, directly querying and transferring faithful appearance details from multi-view references based on shared 3D coordinates, eliminating single-view hallucination and viewpoint drift.

2. Joint Reference Spatial Concatenation: Seamless Fusion of Appearance and Geometry
Conventional controllable diffusion approaches often rely on external side-branch adapters like ControlNet, but separated feature pathways frequently encounter semantic attenuation or spatial misalignment in deeper layers. GO-Renderer employs an early channel-wise concatenation scheme: on the reference side, each of the \(N\) reference frames \(I_j\) is concatenated with its corresponding coordinate map \(C_{\text{ref}, j}\) along the channel dimension before entering the VAE encoder, forming unified geometry-appearance latent representations; on the generation side, the target coordinate maps \(C_{\text{target}}\) and optional environmental appearance videos \(A\) are directly concatenated with the noisy latent sequence along channels. This early coupling enforces dense correspondence across self-attention blocks, faithfully preserving fine-grained object textures while allowing the DiT to adaptively simulate dynamic environment-conditioned highlights and cast shadows according to prompt \(c\).

3. Negative RoPE Shift Temporal Isolation: Eliminating Frame-Transition Smearing
When static multi-view reference frames are concatenated as conditioning tokens alongside target video sequences, standard 3D Rotary Positional Embeddings (RoPE) enforce contiguous temporal indexing by default. If reference frames are assigned immediate predecessor indices (e.g., \(-N, \dots, -1, 0, \dots, M-1\)), the pre-trained video DiT falsely interprets reference views as immediate historical action frames, producing severe smearing and interpolation artifacts in the first generated frame. To eliminate this context pollution, GO-Renderer assigns discrete, negative temporal positions with a dedicated temporal buffer gap \(g\): $\(t_{\text{ref}, j} = - (N - j + 1) \cdot g, \quad j \in \{1, 2, \dots, N\}\)$ where \(g\) represents a predefined temporal gap (\(g=3\) achieves optimal results), while the target video sequence maintains canonical zero-based indices from \(0\) to \(M-1\). By shifting reference latents deep into the negative time axis with an explicit buffer, reference views are explicitly interpreted as global spatial appearance conditions rather than adjacent temporal motions.

Loss & Training

The framework is built upon the pre-trained Wan2.2Fun 5B Ref Control backbone. Training is conducted on 16 NVIDIA A800 GPUs for 20,000 steps with a batch size of 32 using the AdamW optimizer. Video sequences are formatted at a spatial resolution of \(480 \times 832\) with 81 frames. To overcome training data scarcity, the authors built a multi-source dataset comprising 57,000 video clips (combining Blender synthetic rendering with diverse HDRIs, real-world object extraction and tracking via SAM 3 and FoundationPose, and Wan2.2 AI-generated dynamic videos). Targeted augmentations include randomly sampling and shuffling \(3 \sim 8\) multi-view reference images per instance, along with injecting random spatial translations and scaling perturbations into the coordinate maps to prevent the model from overfitting to pristine geometric alignments.

Key Experimental Results

Main Results

Quantitative evaluations assess rendering quality, lighting consistency, and multi-view structural fidelity. Table 1 reports relighting and perceptual generation performance in novel environments against two-stage inverse rendering pipelines (UniLumos with Ground Truth geometry, ReconViaGen priors, and AnySplat 3DGS) and 2D reference-guided diffusion baselines (Phantom, Wan2.2Fun). Table 2 isolates object appearance under blank backgrounds to evaluate multi-view structural consistency.

Table 1: Quantitative comparison of Rendering Quality and Lighting Consistency | Method | PSNR ↑ | SSIM ↑ | Text Align ↑ | Motion Smoothness ↑ | Aesthetic Quality ↑ | Imaging Quality ↑ | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Phantom (Liu et al., 2025) | 11.400 | 0.514 | 24.620 | 0.984 | 0.568 | 0.709 | | UniLumos(GT) (Liu et al., 2025) | 12.060 | 0.585 | 25.630 | 0.978 | 0.569 | 0.669 | | UniLumos(ReconViaGen) | 11.010 | 0.338 | 25.870 | 0.975 | 0.571 | 0.666 | | UniLumos(AnySplat) | 11.160 | 0.362 | 25.930 | 0.975 | 0.557 | 0.624 | | Wan2.2Fun Baseline | 14.300 | 0.641 | 25.250 | 0.984 | 0.530 | 0.601 | | GO-Renderer (Ours) | 18.260 | 0.684 | 27.720 | 0.988 | 0.573 | 0.712 |

Table 2: Quantitative comparison of Multi-view Consistency and Novel View Synthesis | Method | CLIP Img/Avg. ↑ | CLIP Img/Max. ↑ | DINO Avg. ↑ | DINO Max. ↑ | | :--- | :---: | :---: | :---: | :---: | | AnySplat (Jiang et al., 2025) | 0.861 | 0.934 | 0.191 | 0.323 | | ReconViaGen (Chang et al., 2025) | 0.796 | 0.927 | 0.448 | 0.727 | | GO-Renderer (Ours) | 0.888 | 0.956 | 0.725 | 0.860 |

Ablation Study

The ablation investigates the sensitivity of the temporal offset gap \(g\) and the robustness against geometric proxy corruptions.

Table 3: Impact of Temporal Offset Gap \(g\) on Rendering Fidelity | Temporal Offset Configuration | PSNR ↑ | SSIM ↑ | Initial Frame Artifacts & Visual Quality | | :--- | :---: | :---: | :--- | | \(g = 0\) (Contiguous temporal indexing) | 15.63 | 0.6006 | Severe smearing on frame 0 caused by forced temporal interpolation | | \(g = 1\) | 17.89 | 0.6526 | Smearing noticeably mitigated as temporal buffer decouples conditions | | \(g = 3\) (Adopted Default) | 18.27 | 0.6770 | Optimal quantitative performance with clean spatial appearance injection | | \(g = 5\) | 18.09 | 0.6818 | Stable rendering fidelity with robust conditional isolation | | \(g = 10\) | 17.99 | 0.6773 | Minor metric fluctuation due to excessively distant positional offsets |

Key Findings

1. Outperforming Ground-Truth Geometry Pipelines: In novel lighting environments, GO-Renderer achieves 18.26 dB PSNR, outperforming UniLumos(GT) (12.06 dB) by 6.2 dB despite the latter using ground-truth 3D shapes. This demonstrates that internal video diffusion priors synthesize natural environmental reflections and shadows far more effectively than brittle multi-stage inverse rendering optimizations.
2. Graceful Degradation under Spatial Perturbations: Injecting random spatial coordinate noise during inference leads to a smooth, gradual degradation curve rather than a catastrophic collapse. As noise increases, performance stabilizes into a plateau, indicating that when the geometric proxy is partially corrupted, the DiT automatically falls back onto robust 2D generative visual priors to maintain visual plausibility.
3. Practical Consumer-Grade Latency: On a single 48GB RTX 4090 GPU, GO-Renderer generates an 81-frame video (\(480 \times 832\)) in approximately 58 seconds with 31.8 GB peak VRAM, demonstrating immediate viability for interactive rendering plug-ins in tools like Blender.

Highlights & Insights

• Bridging Explicit Spatial Determinism and Generative Expressiveness: 3D vision has historically struggled with PBR material recovery while generative video models lacked geometric anchoring. GO-Renderer positions the 3D proxy purely as a coordinate routing index, delegating lighting, shadowing, and shading entirely to the neural generative process.
• Coordinate Maps as Dimensionless Correspondence Hubs: Mapping local 3D coordinates into standard 2D image channels avoids heavy 3D graph convolutions or point cloud transformers, enabling seamless integration into conventional DiT pipelines without complex cross-domain operators.
• Minimalist Yet Effective Negative RoPE Scheduling: Introducing a discrete negative temporal gap cleanly decouples multi-view visual memories from sequential generation, resolving temporal transition artifacts with negligible architectural overhead.

Limitations & Future Work

• Structural Artifacts under Extreme Proxy Mismatch: While the model tolerates moderate geometric noise, presenting a severely conflicting geometric proxy (such as conditioning a cupcake appearance onto a bottle coordinate proxy) induces irreconcilable feature competition, degrading rendering quality and geometric fidelity.
• Extension to Articulated and 4D Non-Rigid Objects: The current system assumes static or quasi-rigid objects across the 81-frame window. Extending generative rendering to highly articulated or non-rigid dynamic objects will require 4D temporal geometric proxies.

Related Work & Insights

GO-Renderer synthesizes strengths from ViewCrafter, Diffusion as Shader, and Ominicontrol. It demonstrates that photorealistic object rendering does not require explicit, multi-bounce material parameter extraction. As long as reliable spatial coordinate anchors guide the generative foundation model, its internal physical world model can function as an expressive neural shader.

Rating

• Novelty: 4.5 / 5.0
• Experimental Thoroughness: 4.5 / 5.0
• Writing Quality: 4.5 / 5.0
• Value: 4.5 / 5.0