IRIS: Intersection-aware Ray-based Implicit Editable Scenes

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: neural anchors, analytical ray sampling, ray-coherent aggregation, scene editing, neural radiance fields

TL;DR

IRIS attaches neural features to movable Gaussians and replaces dense sampling plus per-query 3D neighbor searches with analytical ray sampling and local feature aggregation, achieving 32.65 dB PSNR at 6.038 FPS on NeRF Synthetic; its principal advantage is efficiency over earlier editable neural fields, not real-time rendering in every scene.

Background & Motivation

NeRF provides a continuous representation of appearance and view-dependent effects, but rendering requires repeated network queries along each ray. Explicit primitives in 3D Gaussian Splatting are easy to move and fast to render, but use a different appearance representation from continuous neural fields. Hybrid methods such as EKS therefore attach neural features to Gaussians to combine neural detail with editable geometry. However, explicit geometry does not remove the rendering bottleneck if every ray still requires dense samples and every sample performs a 3D KNN search.

Editing exposes another difficulty: after an object moves, fixed grids, stale occupancy information, or unsuitable neighborhoods can still affect sampling and interpolation. The system may waste network evaluations in empty space or blend the moved object with disconnected background features, producing trails or contaminated boundaries. IRIS is not designed to generate new objects; it keeps sampling locations, local appearance, and geometric deformation aligned within a reconstructed scene.

The method uses the existing Gaussian geometry to decide where to query, then approximates local neighborhoods using ordered candidates on each ray. Core Idea: attach neural features to movable anchors and make sampling and aggregation follow their geometry, reducing empty-space computation and incorrect blending after edits.

Method

Overall Architecture

The input consists of multiview images for scene reconstruction. Optimization produces neural anchors carrying geometry, opacity, and features, together with shallow decoding networks. For a novel view, the Ray Intersection Selector (RIS) locates samples associated with relevant anchors, Ray-Coherent Aggregation (RCA) combines their local features, and decoding followed by depth-ordered compositing produces pixels.

A multiresolution hash grid organizes anchor features during training; the features are baked into individual anchors before inference. Editing changes anchor geometry and corrects the query direction without retraining the entire neural field. An external physics engine supplies deformations, while IRIS maintains consistency between the deformed representation and its rendering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Multiview images"] --> Anchors["Movable Neural Anchors"]
    Anchors --> RIS["Ray Intersection Selector"]
    Camera["Target camera rays"] --> RIS
    RIS --> RCA["Ray-Coherent Aggregation"]
    RCA --> Decode["Geometry-Constrained Decoding<br/>and Editing Consistency"]
    Edit["External geometric deformation"] --> Decode
    Decode --> Output["Novel or edited view"]
    Edit --> Anchors

Key Designs

1. Movable Neural Anchors: bind appearance codes to geometric carriers

Each anchor contains a Gaussian center, covariance, opacity logit, and learnable feature vector. Covariance is parameterized through rotation and scale, with rotation stored as a quaternion. This represents anisotropic influence regions while supporting geometric manipulation. The feature vector is not an RGB value: it stores local texture and material-density information decoded by a shallow MLP.

During training, features are queried from a multiresolution hash grid at the anchor centers, providing spatially consistent initialization and feature organization. Before inference, optimized features are stored explicitly in their anchors, severing their dependence on global grid coordinates. A relocated object therefore carries its own appearance code instead of reading unrelated features at its new position. The absence of grid warping applies to baked inference and editing, not to training, which still uses a grid.

2. Ray Intersection Selector: replace blind volumetric sampling with geometric candidates

A conventional proposal sampler uses an auxiliary network to estimate density; an occupancy grid requires occupancy maintenance and voxel traversal; coarse-to-fine sampling allocates ray-query budgets across multiple passes. RIS instead exploits the fact that anchors already have explicit spatial extent. It reformulates sample placement as analytical ray interaction with oriented ellipsoids, producing neural queries at meaningful locations.

The implementation uses proxy geometry inspired by 3DGRT: affine-transformed icosahedrons conservatively bound Gaussians, and OptiX delegates intersection lookup to RT cores while identifying maximum-density locations along rays. Candidates are depth-sorted for subsequent aggregation, without first querying empty regions to locate an object. Because Gaussians have no natural finite support boundary, these are effective interactions within proxy bounds, not ordinary surface intersections with an infinite Gaussian distribution.

The maximum Intersections Per Ray (IPR) limits the candidate budget. Candidates follow the anchors, so objects moved outside the original training bounding box can still be sampled. However, an insufficient budget can miss necessary depth coverage: analytical sampling does not imply that arbitrarily few samples suffice.

3. Ray-Coherent Aggregation: approximate spatial neighborhoods on ordered candidates

Decoding individual anchors can expose discrete appearance, whereas per-query 3D KNN is expensive. RCA takes a symmetric sliding window over the ordered sample list and retrieves neighbor parameters together through vectorized unfolding. This is a 1D neighbor approximation along the ray, not a guarantee of retrieving exactly the same set as full 3D KNN.

Adjacent depth indices do not guarantee spatial proximity. RCA therefore requires candidates to belong to the same ray and the Euclidean distance from the sample to the candidate anchor center to remain below \(\tau_{\mathrm{dist}}\). Valid neighbors receive Mahalanobis-distance-based logits, while invalid candidates are masked to negative infinity before Softmax. The following restores the broken layout of Eq. (1) using its surrounding definitions, without introducing a different weighting rule:

\[ w_{kj}=\frac{\exp(l_{kj})}{\sum_{m\in\mathcal W_k}\exp(l_{km})},\qquad \hat{\boldsymbol f}(\boldsymbol x_k)=\sum_{j\in\mathcal W_k}w_{kj}\boldsymbol f_j,\qquad \hat\alpha(\boldsymbol x_k)=\sum_{j\in\mathcal W_k}w_{kj}\alpha^{\mathrm{raw}}_{kj}. \]

Here, \(\mathcal W_k\) is the window, \(l_{kj}\) is the validity-masked logit, and \(\alpha^{\mathrm{raw}}_{kj}\) multiplies Gaussian falloff by the sigmoid-activated anchor opacity. Aggregation produces both continuous features and a geometric weight. The former supplies local appearance; the latter indicates whether explicit geometry supports that location, rather than merely averaging opacity parameters without spatial conditioning.

4. Geometry-Constrained Decoding and Editing Consistency: let the network model material and anchors constrain space

A shallow MLP receives aggregated features and the viewing direction, producing view-dependent color and raw density. A truncated exponential with a negative bias activates the density, which is then multiplied by the aggregated geometric weight. A high neural density prediction cannot freely create visible content where anchor support is absent. The paper uses point-based volumetric compositing, with no additional ray-step length in its density-to-opacity expression; this note preserves that convention:

\[ \sigma_{\mathrm{eff}}(\boldsymbol x_k)=\operatorname{trunc\_exp}(\sigma_{\mathrm{mlp}}-1)\hat\alpha(\boldsymbol x_k),\qquad \alpha_k=1-\exp[-\sigma_{\mathrm{eff}}(\boldsymbol x_k)]. \]

Colors are then accumulated front to back in depth order, with later contributions multiplied by the remaining transmittance of preceding samples. The aggregated geometric weight \(\hat\alpha\) and final compositing opacity \(\alpha_k\) are different quantities: the latter also incorporates the network's material-density prediction.

Rigid or nonrigid edits update anchor centers and covariances while baked features remain attached to their original anchors. Moving geometry without correcting the network's viewing-direction input can make specular effects respond incorrectly to surface orientation. IRIS adopts Infinitesimal Surface Transformation (IST) from NeuralEditor, correcting the direction through the local deformation transform as \(\boldsymbol d'=R_{\mathrm{def}}^{\mathsf T}\boldsymbol d\) before querying the same appearance code. This enforces directional consistency; it does not recompute global illumination, shadows, or a complete physical material model.

A Worked Example

Consider bending a foreground object and moving it away from its original position. Anchors are first bound to low-resolution proxy geometry. External deformation updates their centers and shapes while their texture codes remain unchanged. The target camera casts new rays, and RIS generates candidates from the moved geometry rather than waiting for a stale occupancy grid to update.

If a sample is followed in the ordered list by a distant wall candidate, RCA's distance mask prevents wall appearance from leaking into the foreground; genuinely local anchors still contribute together. Decoding uses the IST-corrected direction and geometrically constrained density before compositing the bent object. This example explains the joint role of feature baking, geometric sampling, locality masking, and direction correction; it illustrates the mechanism rather than adding a quantitative experiment.

Loss & Training

The main text describes visibility-aware pruning. All anchors undergo continuous confidence decay, RIS-selected candidates receive positive updates, and persistently unvisited anchors below the pruning threshold are removed. This uses sampling participation to suppress redundant anchors and artifacts, but does not establish that every retained anchor has a substantial visual contribution.

The available cache contains the main paper and references, not the repeatedly cited appendix. Complete loss terms, optimizer, learning rate, training steps, distance threshold, default maximum IPR, hardware, and evaluation resolution cannot be confirmed from it. No conventional settings are substituted. Hash initialization, geometric density gating, feature baking, and visibility-aware pruning are the training and inference mechanisms that can be established.

Key Experimental Results

Main Results

Table 2 of the paper reports averages on NeRF Synthetic. PSNR is measured in dB, SSIM and LPIPS are dimensionless, and FPS denotes frames per second; training times retain the original units. Because hardware and image-resolution details are absent from the cache, these are comparisons under the paper's reported conditions, not absolute performance predictions for other devices.

Method	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	Training time ↓	FPS ↑
InstantNGP	29.31	0.939	0.053	7 min 27 s	1.894
Nerfacto	29.19	0.941	0.095	9 min 38 s	0.860
NeuralEditor	31.41	0.955	0.036	103 h 45 min	0.083
EKS	33.12	0.962	0.031	26 h 2 min	0.183
IRIS	32.65	0.957	0.034	51 min 18 s	6.038

Relative to EKS, IRIS loses 0.47 dB while achieving approximately 33.0 times its FPS and 30.4 times faster training; both ratios are calculated from the table. Training still takes longer than InstantNGP and Nerfacto, so acceleration over editable baselines should not be generalized to all methods.

In Table 3, mean indoor Mip-NeRF 360 PSNR is 30.85 dB for IRIS, 28.48 dB for EKS, and 29.29 dB for Radiance Meshes, versus 31.72 dB for static Mip-NeRF 360. Outdoors, IRIS reaches 24.11 dB, below Radiance Meshes at 24.73 dB but above EKS at 20.70 dB. Radiance Meshes was re-evaluated by the authors using its official implementation.

Ablation Study

The following reproduces the independent EKS KNN / RCA comparison zone in Table 5, trained on Bicycle, Garden, and Kitchen. It must not be conflated with the separate window and density ablations above it. PSNR is in dB, and memory retains the table's GB unit. Columns with partially merged extraction or unspecified time units are omitted.

Aggregation configuration	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	FPS ↑	Memory ↓ (GB)
EKS KNN	26.63	0.759	0.187	0.019	20.73
RCA	27.21	0.786	0.168	0.30	9.1

In this independent comparison, RCA improves PSNR by 0.58 dB while improving speed and memory use. In the separate upper zone of Table 5, window size 3 and the full configuration with window size 5 achieve 27.02 and 27.01 dB, respectively. Removing the validity mask (NDC) also gives 27.02 dB. These global metrics therefore do not support a strong claim that masking substantially improves mean PSNR; its role should be interpreted alongside boundary and deformation visualizations.

Figure 10 provides qualitative comparisons of sampling budgets and hash initialization: 16 IPR introduces geometric noise, 64 IPR improves structure, and the full budget better resolves high-frequency detail. The authors describe the quantitative difference without hash initialization as small. The cache does not supply a complete numerical comparison for these settings, so no ablation scores are inferred from the images.

Key Findings

• Table 4 averages the NeuralEditor deformation benchmark over NeRF Synthetic: IRIS achieves 25.70 dB, 0.932 SSIM, and 0.056 LPIPS, versus EKS at 25.92 dB, 0.934, and 0.055. Editability is demonstrated, but quality does not uniformly exceed EKS.
• Figure 6 measures sampler throughput, a different quantity from full-image FPS. RIS's batch-scaling advantage does not erase the 0.30 FPS in Table 5 or turn Table 2's 6.038 FPS into sustained high-frame-rate real-time rendering.
• Indoor versus outdoor performance is related to explicit-anchor coverage. Deep Blending or indoor results alone do not establish dominance across unbounded scenes.

Highlights & Insights

• The optimization target is where the neural field is queried, not merely MLP size. Existing explicit geometry can both locate samples and constrain output density, keeping the representation aligned with its rendering procedure.
• Training-time spatial priors are separated from inference-time mobility. Hash grids help learning, while baked features remove the global-coordinate dependency during editing; this distinction is more accurate than simply claiming to eliminate grids.
• Neighbor approximation needs geometric safeguards. Ordered lists supply efficient candidates and distance masking restores locality, but the engineering approximation is not a theoretical equivalence to full spatial search.

Limitations & Future Work

• The authors acknowledge that finite anchor coverage degrades far-field geometry in unbounded outdoor scenes and that RCA is sensitive to window size, potentially producing noise or floaters. Adaptive windows and improved far-field anchor allocation are plausible directions, not evaluated solutions in this paper.
• Speed gains are mainly relative to slow editable neural fields; the main results and large-scene analysis do not establish universal real-time performance. Dependence on OptiX and RT cores also requires separate evaluation when reproducing results across devices.
• Physics demonstrations depend on external engines and proxy geometry, rather than automatically inferring reliable physical parameters from photographs. IST maintains local viewing-direction consistency and does not establish physically correct relighting after edits.
• The missing appendix prevents verification of reproduction conditions and some ablations. Ordered windows and hard masks also mean that a continuous-neural-field description is not a strict smoothness guarantee across every neighborhood change; this is a reader's assessment of the mechanism's boundary.

Related Work & Insights

• vs EKS: Both attach neural features to Gaussians. IRIS primarily redesigns sampling and neighbor aggregation, improving speed substantially while remaining slightly worse in average NeRF Synthetic reconstruction and deformation quality.
• vs 3DGRT: IRIS borrows proxy geometry and hardware ray tracing to locate neural queries, then still performs feature aggregation and neural density decoding; it is not simply direct compositing of Gaussian colors.
• vs NeuralEditor / Radiance Meshes: The former supplies the deformation benchmark and IST, while the latter renders radiance fields through tetrahedral cells. They motivate evaluating geometric manipulation, view-dependent effects, and runtime together rather than ranking editability by static PSNR alone.

Rating

• Novelty: 4/5. Analytical sampling and ray-coherent aggregation form a clear system contribution, with some components inherited from prior work.
• Experimental Thoroughness: 4/5. Reconstruction, deformation, multiple datasets, and ablations are covered, but the cache lacks reproduction appendices and complete editing-efficiency measurements.
• Writing Quality: 3/5. The pipeline is clear, but real-time claims are strong and the cached equations and tables have extraction defects.
• Value: 4/5. Useful for improving editable hybrid neural fields, provided scene-scale and hardware constraints remain explicit.