OmniRen: Neural Rendering wih Heterogeneous Scene Primitives

Conference: ECCV 2026
Paper: ECCV Official
Code: https://renderformer.github.io/v2
Area: 3D Vision
Keywords: Neural Rendering, Sparse Attention, Neural Material Embedding, Heterogeneous Scene Primitives, Global Illumination

TL;DR

RenderFormer-V2 (OmniRen) addresses the scalability and primitive diversity limitations of neural rendering by combining Hilbert space-filling curve sliding window attention with physics-informed attention sinks and a BRDF-agnostic neural material appearance space, enabling feed-forward global illumination rendering across heterogeneous primitives (>100k) without per-scene training.

Background & Motivation

Neural rendering aims to visualize complex virtual scenes without relying on manually crafted numerical integration routines or hand-written light transport code, instead learning relations among geometry, materials, and illumination directly from data. However, existing neural rendering paradigms face an uncomfortable dilemma: implicit radiance field (NeRF) and Gaussian Splatting approaches rely heavily on per-scene optimization and fail to generalize across unseen novel scenes, while screen-space G-buffer models only observe visible surface fragments and must hallucinate or discard indirect transport and caustics originating from occluded geometry or out-of-frustum light sources. RenderFormer took a bold step forward by formalizing global light transport simulation as an end-to-end regressive sequence-to-sequence (Seq2Seq) translation task. Yet, despite demonstrating cross-scene generalization, its first-generation architecture was strictly held back by severe structural assumptions.

The fundamental bottleneck in RenderFormer lies in its quadratic computational scaling and rigid representation space. In its view-independent intra-primitive transport stage, naive brute-force self-attention induces quadratic complexity \(O(N^2)\) in both compute and memory relative to the triangle count. When scaling past a modest 4k triangles, attention weights inevitably diffuse across thousands of distant tokens, causing severe loss of focus and manifest dark artifacts. Concurrently, the pipeline hard-coded an analytical microfacet GGX BRDF and allowed at most 8 triangular diffuse emitters, fundamentally failing to handle participating media volumetric scattering, high-dynamic-range environment maps, surface displacement textures, or arbitrary measured reflectance models.

Rather than resorting to traditional ray-tracing branches and specialized shading passes, this paper recognizes that multi-physical transport phenomena can be naturally unified as sequence transformations over heterogeneous tokens, provided that the long-sequence attention mechanism respects the underlying physics of light transport. Core idea: formulate global light transport as a sequence-to-sequence transformation over heterogeneous primitives, coupling Hilbert space-filling curve local sliding windows with rendering-informed attention sinks (light sources, global registers, and pooled geometry summaries) while decoupling material specifications into a continuous appearance latent manifold to achieve cross-scene feed-forward neural rendering beyond 100k primitives.

Method

Overall Architecture

RenderFormer-V2 retains a decoupled two-stage transformer-based rendering architecture. The first view-independent stage resolves indirect radiative transfer and energy exchange among all heterogeneous scene primitives in world coordinates. Subsequently, the second view-dependent stage receives camera ray bundles, queries the contextualized scene representation via cross-attention, and decodes the resulting features through shifted window attention into high-dynamic-range pixel patches.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Input: Triangles / Voxels / Emitters / Env Maps"] --> B["Unified Relative Positional Encoding & Primitive Embedding"]
    B --> C["Hilbert Curve Sorting & Local Sliding Window Attention"]
    B --> D["Physics-Informed Attention Sink<br/>Emitters + Global Registers + Geometry Summaries"]
    C --> E["View-Independent Light Transport Resolution"]
    D --> E
    E --> F["World-Space Ray-Bundle Cross-Attention & Swin View Decoding"]
    F --> G["High-Fidelity Full Global Illumination Render"]

The system maps triangle meshes, volumetric scattering voxels, triangular light sources, environment maps, and camera rays into a unified 768-dimensional token sequence. In the view-independent stage, space-filling curve serialization combined with local sliding window attention captures high-frequency local contact shadows and diffuse bounces, while rendering-informed attention sinks capture long-range illumination. In the view-dependent stage, \(8 \times 8\) ray bundles projected from the camera interact with the scene primitives through cross-attention and Swin shifted-window layers, followed by a dense vision transformer decoder to produce final image pixels.

Key Designs

1. Rendering-Informed Attention Sinks and Local Sparse Attention: Eliminating Quadratic Overhead and Attentional Focus Loss

Brute-force self-attention incurs prohibitive \(O(N^2)\) complexity and causes softmax normalization to dilute attention over vast distant surfaces, leading to severe under-illumination on large meshes. Observing that 3D geometric primitives are irregularly clustered across space, the model first serializes and sorts all primitive centroids along a 3D Hilbert space-filling curve, ensuring that elements adjacent along the 1D sequence remain close in physical 3D space. Local transport—such as contact occlusion and nearby diffuse bounces—is resolved using a sliding window attention taking 256 tokens before and after each target primitive. To capture long-range global transport without full pairwise interactions, the model adapts attention sinks into a rendering-informed tripartite architecture: 16 learnable global register tokens act as scratchpads for scene-wide energy bases; all scene light source tokens are persistently retained in the sink so that every primitive directly attends to every emitter regardless of distance (analogous to Next Event Estimation and light importance sampling in path tracing); and every 64 consecutive geometry tokens along the Hilbert curve are mean-pooled into a coarse summarization token added to the sink. This design guarantees near-linear complexity \(O(N \cdot W + N \cdot \frac{N}{64})\) while preserving both crisp contact features and macroscopic indirect illumination.

2. Latent Material Appearance Manifold and Texture Embedding: Decoupling Analytical BRDF Parameterizations

Conventional neural renderers hard-code reflectance parameters (such as roughness and index of refraction) into specific analytical BRDF equations, preventing them from supporting complex measured materials or layered coatings. RenderFormer-V2 observes that the material embedding does not need to be invertible to analytical parameters—it only needs to encode perceptual appearance, allowing the transformer to learn how to evaluate reflection given view and lighting. The authors pretrain a CNN-based autoencoder with a 9-dimensional latent bottleneck on images of a unit sphere rendered under the color-neutral, frequency-rich Uffizi Gallery light probe across diverse Disney Principled BRDF samples. By applying smoothness regularization and a hyperbolic tangent \(\tanh\) activation, the latent space is constrained to a coherent \([-1, +1]\) manifold. Analytical BRDF parameters can be mapped into this 9D space via a lightweight multi-layer perceptron (MLP), while arbitrary measured BRDFs (e.g., from the RGL dataset) are directly projected by rendering a probe sphere and passing it through the encoder. For spatially varying materials (SVBRDFs), per-triangle 9-channel latent material maps, 3-channel normal maps, and 1-channel height displacement maps are rasterized into \(32 \times 32\) patches, encoded by a pretrained VAE into a \(4 \times 4 \times 80\) feature grid, and compressed via a linear layer into a 768-dimensional token, allowing the network to natively learn surface micro-parallax and self-shadowing without bespoke displacement code.

3. Unified Heterogeneous Primitive Embedding: Native Support for Volumetric Media, Environment Lighting, and Ray Bundles

To accommodate diverse physical phenomena within a single architecture, the model establishes a standardized token encoding protocol across distinct primitive modalities. Volumetric scattering elements are discretized into voxels whose 3D rotation matrix and axis scales are encoded via 12-frequency NeRF positional encodings, concatenated with 7-dimensional optical parameters (RGB scattering coefficients, RGB absorption coefficients, and Henyey-Greenstein phase function anisotropy) across a \(4 \times 4 \times 4\) volumetric grid. High-dynamic-range environment maps (\(512 \times 256\)) are encoded via a pretrained VAE into \(8 \times 4\) patch tokens of size \(8 \times 8 \times 16\), augmented with explicit normalized world-coordinate direction vectors and log-maximum luminance scale factors. Spatial invariance across all localized primitives and camera ray bundles is enforced via a unified Rotary Position Embedding (RoPE) applied to primitive centroids with dimension 40 (20 frequency pairs), while global positionless entities (e.g., environment maps) are bound to the scene bounding box center.

4. World-Space View-Dependent Decoding and Swin Windowed Rays: Decoupling Resolution from Context Complexity

RenderFormer-V1 conducted its second stage entirely within camera coordinates, necessitating rigid-body coordinate re-projections and suffering from quadratic scaling when rendering dense pixel grids. RenderFormer-V2 formulates ray bundles directly in world coordinates, with ray origins handled through uniform centroid RoPE. The full self-attention in the view-dependent stage is replaced with Swin shifted-window attention with a window size of 8 and a shift of 4. This reduces the ray-processing computational cost from \(O(T \cdot R + R^2)\) to \(O(T \cdot R + W^2)\), where \(T\) is the number of scene tokens, \(R\) is the number of ray bundles, and \(W\) is the fixed window size. By decoupling the context window from image resolution, the architecture smoothly scales to ultra-high resolutions up to \(2048 \times 2048\) without explosive memory footprints.

Loss & Training

The network is optimized end-to-end using a composite log-space reconstruction and perceptual loss: $$ \mathcal{L} = \mathcal{L}1(\log I, \log \hat{I}) + 0.05 \cdot \mathcal{L}, \log(1))) $$ Applying a logarithmic transformation }}(\text{clamp}(\log I, \log(1)), \text{clamp}(\log \hat{I\(\log I\) compresses high-dynamic-range specular highlights and caustic spikes, preventing outlier gradients from dominating the \(L_1\) loss, while LPIPS preserves high-frequency visual textures.

To stabilize training across heterogeneous primitives, a progressive 5-stage curriculum is executed over 19 days on 32 NVIDIA A100 GPUs: 1. Stage 1 (4 days, 1k primitives, \(256 \times 256\)): Full self-attention to establish foundational transport mechanics, progressively introducing homogeneous materials, SVBRDFs, masked environment maps, and volumetric media. 2. Stage 2 (2 days, 4k primitives, \(512 \times 512\)): Increases primitive budget and resolution. 3. Stage 3 (3 days, 4k primitives, \(512 \times 512\)): Transitions from full attention to Hilbert sliding windows and rendering-informed attention sinks. 4. Stage 4 (7 days, 16k primitives, \(512 \times 512\)): Stabilizes long-range multi-occlusion transport under large primitive counts. 5. Stage 5 (3 days, 64k primitives, \(2048 \times 2048\)): Refines fine-scale geometric details and micro-surface shading.

Key Experimental Results

Main Results

RenderFormer-V2 was evaluated against RenderFormer-V1 and offline path tracing via Blender Cycles (4096 samples per pixel) across diverse unseen scenes featuring PolyHaven environment maps, MatSynth materials, and dense geometric meshes (e.g., Stanford Dragon and Lucy):

Model / Configuration	Primitives (#)	Resolution	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	Time (s/frame) ↓
RenderFormer (V1)	6.4k	\(512 \times 512\)	24.81	0.8120	0.2104	1.84
RenderFormer (V1)	83.2k	\(512 \times 512\)	18.23	0.6432	0.5420	5.86
RenderFormer-V2 (Ours)	6.4k	\(512 \times 512\)	28.14	0.8950	0.1031	0.38
RenderFormer-V2 (Ours)	83.2k	\(512 \times 512\)	27.95	0.8894	0.1082	1.12
RenderFormer-V2 (Ours)	64k	\(2048 \times 2048\)	27.84	0.8856	0.1105	12.4
Blender Cycles (Ground Truth)	64k	\(2048 \times 2048\)	\(\infty\)	1.0000	0.0000	~185.0

Note: RenderFormer-V1 experiences catastrophic quality degradation and severe darkening when scaled to 83.2k primitives due to attentional focus loss, whereas RenderFormer-V2 maintains high fidelity while running over two orders of magnitude faster than offline path tracing.

Ablation Study

A systematic ablation of the sparse attention components was conducted on stage-4 test scenes (16k primitives):

Model Variant	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	HDR-FLIP ↓	Note
RenderFormer-V2 (Full Model)	28.25	0.8982	0.0997	0.4200	Full SW + AS (L + S with 64:1 ratio)
w/o Attention Sink (w/o AS)	27.36	0.8841	0.1247	0.4359	Only local window; loses long-range indirect light
w/o Sliding Window (w/o SW)	26.92	0.8715	0.1289	0.4517	Loses high-frequency contact shadows and local bounces
AS w/o Light Tokens (AS w/o L)	27.40	0.8813	0.1081	0.4250	Inability to perform explicit emitter importance sampling
AS w/o Summarization (AS w/o S)	27.09	0.8754	0.1210	0.4485	Distant geometry loses macroscopic occlusion cues
AS w/o L, S (Registers Only)	26.13	0.8474	0.1514	0.4959	Severe performance drop; register tokens insufficient
More Summaries (#seq / 32)	27.78	0.8923	0.1072	0.4231	Redundant background tokens dilute local attention
Less Summaries (#seq / 128)	26.48	0.8690	0.1346	0.4655	Overly coarse representation misses macro structures
Full Attention Baseline	27.91	0.8953	0.1027	0.4287	Outperformed by sparse model due to attention diffusion

Key Findings

• Sparse Attention Outperforms Full Attention: The full RenderFormer-V2 architecture (28.25 dB PSNR) outperforms the brute-force full attention model (27.91 dB PSNR). In large 3D scenes, softmax over thousands of unpruned tokens dilutes gradients across irrelevant geometry, whereas structured local windows plus semantic sinks maintain sharp focus on primary energy pathways.
• Synergy of Light and Geometry Sinks: Omitting both light source and summarization tokens drops PSNR by 2.12 dB (to 26.13 dB) and inflates HDR-FLIP error to 0.4959. Light tokens supply critical direct boundary conditions, while a 64:1 summarization ratio strikes the sweet spot between structural occlusion fidelity and token compactness.
• Resolution-Driven Geometric Disambiguation: Increasing view resolution from 512 to 2048 without altering mesh topology enables the network to resolve subtle geometry that was blurry at lower resolutions (e.g., claw joints on the Stanford Dragon and facial features on Lucy), demonstrating effective cross-scale visual decoding.

Highlights & Insights

• Physical Radiative Principles Mirrored in LLM Attention Mechanics: The architecture elegantly maps core ray tracing principles into transformer operations: Next Event Estimation (NEE) is manifested as dedicated light source attention sinks, radiosity cluster hierarchy is captured by Hilbert-pooled summarization tokens, and near-field photon gathering maps to sliding-window attention.
• Perceptual Manifolds Over Inverse Material Parameter Fitting: Decoupling reflectance from analytical BRDF parameters circumvents ill-posed inverse estimation. Projecting materials onto a continuous neural perceptual manifold learned under a canonical probe enables arbitrary measured or procedural shaders to be evaluated natively.
• Unified Tokenization of Volumetric and Surface Light Transport: Overcomes the dichotomy between surface meshes and volumetric fields, enabling complex transmissive effects like caustics and smoke scattering to emerge naturally within a shared sequence without specialized integration passes.

Limitations & Future Work

• Emitter Scalability Bound: Inherited from its training distribution, the system currently handles at most 8 explicit triangular light sources, requiring dense illumination setups to be approximated via environment maps.
• Fixed Texture Tile Resolution: Allocating a fixed \(32 \times 32\) texture patch per triangle causes spatial under-sampling on coarse, large-area triangles, requiring pre-tessellation before rendering.
• Retraining Costs for Novel Primitives: Because heterogeneous interactions are seeded during Stage 1 of the curriculum, incorporating new geometric primitives (such as 3D Gaussians or neural SDFs) requires extensive re-training from scratch.

Related Work & Insights

• vs RenderFormer (V1): RenderFormer-V1 is restricted to <4k triangles, a hard-coded GGX model, and diffuse triangular lights. RenderFormer-V2 scales past 100k primitives via sparse Hilbert-sink attention and natively accommodates participating media, environment lighting, and measured BRDFs.
• vs DiffusionRenderer: DiffusionRenderer relies on video diffusion priors in screen space, which are vulnerable to temporal flickering and unphysical hallucinations in occluded regions; RenderFormer-V2 operates over complete 3D scene tokens, guaranteeing geometric consistency and physically grounded transport.
• vs Point Transformer v3: While both employ Hilbert serialization, PTv3 is tailored for point cloud segmentation, whereas RenderFormer-V2 adapts serialization for long-sequence radiative transfer via hybrid attention sinks.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering translation of light transport physics into rendering-informed sparse attention sinks and continuous neural material manifolds.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations covering complex caustics, volumetric smoke, measured BRDFs, and rigorous ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid architectural exposition with compelling mathematical framing and sound physical intuition.
• Value: ⭐⭐⭐⭐⭐ Establishes a foundational milestone toward universal, feed-forward, generalizable neural graphics shaders.