LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: 3D Vision
Keywords: 3D reconstruction, Native Sparse Attention, inverse rendering, context window scaling, sequence parallelism

TL;DR

Addressing the persistent fidelity gap where feed-forward 3D reconstruction falls behind dense-view optimization in fine-grained appearance, LSRM scales object and image context windows by 20× and >2× using native sparse attention, geometric 3D-aware routing, and block-aware sequence parallelism to achieve state-of-the-art reconstruction and inverse rendering.

Background & Motivation

Feed-forward 3D foundation models built on large-scale transformers have drastically accelerated object-centric reconstruction and inverse rendering, cutting execution time from tens of minutes in optimization-based methods down to a single forward pass of a few seconds. Furthermore, feed-forward models frequently demonstrate superior geometric robustness on challenging specular surfaces where continuous optimization easily gets trapped in degenerate local minima. Nonetheless, existing feed-forward methods suffer from a severe limitation: reconstructed textures frequently lack fine-grained structural details, leading to smeared logos, blurry text, and distorted facial features. While recent approaches attempt to address this by adopting alternative 3D representations (such as 3D Gaussian Splatting) or applying post-hoc 2D texture refinement, a substantial fidelity gap compared to dense-view continuous optimization pipelines remains.

The root cause of this fidelity bottleneck lies in the tight constraint of the transformer context window. Recovering high-frequency micro-textures and precise geometric edges fundamentally requires dramatically increasing the number of active 3D object tokens and high-resolution 2D image tokens. However, standard dense self- and cross-attention scale quadratically—and in volumetric 3D representations, cubically—with token count, rapidly exceeding GPU memory and compute capacities. Furthermore, directly importing sparse attention algorithms from natural language processing leads to severe failures in early transformer layers, where attention weights based purely on semantic feature similarity cannot establish reliable 2D-3D geometric correspondences. Compounding this, the dynamic, spatially non-uniform distribution of active surface tokens induces severe GPU workload imbalance during multi-node training.

LSRM attacks these computational and architectural bottlenecks through a customized sparse attention framework designed specifically for 3D physical geometry. Core idea: adapt Native Sparse Attention (NSA) into a coarse-to-fine residual prediction pipeline with explicit 3D camera projection routing and block-aware sequence parallelism with an All-gather-KV protocol, scaling active object tokens by 20× and recovering pristine high-frequency geometry and material details.

Method

Overall Architecture

LSRM adopts an efficient two-stage coarse-to-fine pipeline. In Stage 1, a dense reconstruction transformer operates at lower spatial resolutions to capture global scene structure, outputting a coarse dense feature volume and pruning background image tokens. In Stage 2, the sparse reconstruction transformer—initialized with Stage 1 network weights and geometric priors—substantially expands input image and 3D volume resolutions to predict high-resolution sparse volume residuals over informative surface regions. During volume rendering, ray marching adaptively queries high-frequency features from the sparse volume on active surface voxels and macro-structural features from the dense volume in free space, decoding continuous SDF values and either RGB appearance or physically-based BRDF attributes (albedo \(a\), roughness \(r\), metallic \(m\)) via lightweight MLPs.

The complete training and inference pipeline coordinates multi-GPU workload balance, hierarchical residual prediction, and explicit geometric block routing as illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multi-View Posed Images<br/>(12-18 views) + Plücker Rays"] --> S1["Dense Reconstruction Initialization<br/>Low-res Dense Transformer predicts coarse Xd"]
    S1 --> S2A["Coarse-to-Fine Residual Prediction<br/>SDF thresholding selects surface voxels & predicts sparse residuals"]
    S2A --> S2B["3D-Aware Spatial Routing<br/>Explicit camera projection & Euclidean distance retrieve relevant token blocks"]
    S2B --> S2C["Block-Aware Sequence Parallelism<br/>Spatial block device localization + All-gather-KV load balancing"]
    S2C --> Out["Hybrid Decoded Rendering<br/>High-fidelity meshes, textures, SDF & BRDF maps"]

Key Designs

1. Coarse-to-Fine Sparse Residual Prediction: Anchoring High-Frequency Details on Geometric Priors Directly predicting an unconstrained high-resolution 3D volumetric token representation from scratch is computationally intractable and prone to convergence failure. LSRM addresses this through a progressive residual formulation. Stage 1 trains a dense transformer at moderate resolutions (\(S_\text{img}^\text{d}=32\), \(S_\text{vol}^\text{d}=16\)) to produce an initial dense feature volume \(X_\text{d}\) at spatial resolution \(S_\text{vol}^\text{df}=64\). When scaling to Stage 2 with \(S_\text{img}=96\) and \(S_\text{vol}=96\), background image tokens are pruned via foreground masks, while active 3D volume tokens are selectively instantiated only within voxels intersecting or immediately adjacent to the object surface. Specifically, within each voxel \((i, j, k)\), \(T=4^3=64\) uniform points are evaluated against the frozen Stage 1 feature volume \(X_\text{d}\) to obtain continuous signed distance values \(s_t\), activating a sparse token if:

\[M_{i,j,k}^\text{vol} = \mathbb{I}\left(\min_{p_t \in \mathcal{P}_{i,j,k}} |s_t| \le \tau \quad \text{or} \quad \left(\min_{p_t \in \mathcal{P}_{i,j,k}} s_t\right)\left(\max_{p_t \in \mathcal{P}_{i,j,k}} s_t\right) \le 0\right)\]

Rather than generating full volumetric features from scratch, the sparse transformer is formulated to predict sparse residuals added directly onto the upsampled Stage 1 features: \(x^{(m)} = \text{NSABlock}^{(m)}\left(x^{(m-1)} + \text{Linear}^{(m)}(x_\text{d}^\text{up}), y^{(m-1)} + \text{Linear}^{(m)}(y_\text{d}^\text{up})\right)\). The output projects to a high-resolution sparse feature volume \(X_\text{s}\) at resolution \(S_\text{vol}^\text{f}=384\). During volume ray marching, sampled points inside active voxels query fine-grained details from \(X_\text{s}\), empty space queries \(X_\text{d}\), and boundary points smoothly blend both via linear interpolation, dramatically reducing memory while preserving sharp boundaries.

2. 3D-Aware Spatial Routing: Resolving Modality Misalignment Beyond Attention Scores In vanilla Native Sparse Attention, key-value block selection relies entirely on attention similarity scores computed against compressed tokens (\(k^\text{cmp}\)). However, in multi-view 3D reconstruction, early transformer layers lack the global geometric context needed to match 3D voxel queries to their corresponding 2D visual patches purely through ungrounded dot products, causing noisy routing and blurred textures. LSRM introduces an explicit geometry-based routing mechanism utilizing the Stage 1 coarse shape and known camera parameters.

Each token is assigned an explicit 3D coordinate: volume tokens take their voxel center coordinates \(p_\text{vol}\), while 2D image tokens take the 3D position \(p_\text{img}\) of the maximum-opacity surface point rendered along their corresponding viewing ray from \(X_\text{d}\). When routing from a query point \(p \in \{p_\text{img}, p_\text{vol}\}\) to image blocks, \(p\) is projected onto each camera image plane using calibrated extrinsics and intrinsics. The algorithm selects the \(B_\text{i}\) candidate image blocks whose 2D centers are closest to the projection, and subsequently computes the minimal 3D Euclidean distance between \(p\) and the active token points within each candidate block, picking the top candidates with minimal 3D distance. Voxel-to-voxel block selection similarly sorts by Euclidean distance between block centers. This geometric prior ensures that transformer layers from bottom to top consistently route cross-attention to physically corresponding views, restoring fine text and delicate contours.

3. Block-Aware Sequence Parallelism: Workload Balancing with Preserved Block Locality Because active tokens cluster non-uniformly around object surfaces, token counts vary widely across instances, leading to severe GPU straggler bottlenecks where idle devices stall awaiting the slowest worker. Conventional reservoir sampling drops up to one third of active tokens, causing catastrophic optimization instability. LSRM designs a block-aware sequence parallelism scheme adapted from context parallelism.

Active image and volume tokens are grouped into \(8 \times 8\) and \(8 \times 8 \times 8\) spatial blocks. When sharding across GPUs, tokens are partitioned strictly at the block boundary—ensuring all tokens belonging to the same spatial block reside on the identical GPU. This preserves intra-block spatial locality, enabling block-level window attention and key-value compression (\(k^\text{cmp}=\text{AvgPool}(\text{ResBlock}(k_t))\)) to execute entirely on-device with zero cross-GPU communication. To execute global selected and compressed attention across the entire context window, LSRM exploits the extreme compression ratio of Grouped-Query Attention (32 query heads sharing only 2 KV heads) via an All-gather-KV protocol. Before each NSABlock, compact \((k^\text{cmp}, v^\text{cmp})\) and selected \((k, v)\) tensors are broadcast globally across GPUs, while queries (\(q\)) remain locally sharded. This maintains global receptive fields with minimal communication overhead, achieving near-linear multi-GPU scaling across 8 devices.

Loss & Training

The novel-view synthesis model is trained in three stages on 600K curated 3D models: 1. Stage 1 (5 days): Trains the dense reconstruction model (\(S_\text{vol}^\text{d}=16, S_\text{img}^\text{d}=32\)) with volumetric RGB rendering loss, foreground mask cross-entropy, and depth regularization. 2. Stage 2 (7 days): Freezes Stage 1 and trains the sparse transformer at intermediate resolution (\(S_\text{img}=64, S_\text{vol}=64\)) for rapid convergence with larger batch sizes. 3. Stage 3 (3 days): Scales to full resolution (\(S_\text{img}=96, S_\text{vol}=96\)) using block-aware sequence parallelism on 128 NVIDIA H100 GPUs, incorporating a numerical normal loss during the final 500–1000 iterations to refine surface geometry.

For inverse rendering, the model is initialized directly from the converged NVS checkpoint. Background images are encoded via linear projections to disentangle ambient illumination, and the RGB MLP head initializes the BRDF prediction head. Fine-tuning uses real HDR environment maps with extensive augmentations, requiring half the training iterations of NVS.

Key Experimental Results

Main Results

On the Google Scanned Objects (GSO) novel-view synthesis benchmark evaluated with 16 input views (\(768 \times 768\)) and 12 evaluation novel views (\(512 \times 512\)), LSRM demonstrates substantial improvements over leading feed-forward baselines:

Method	PSNR (↑)	SSIM (↑)	LPIPS (↓)
MeshLRM	28.13	0.923	0.093
GS-LRM	30.52	0.952	0.050
LIRM	30.65	0.949	0.054
LSRM (Ours)	33.08	0.971	0.028

On the StanfordORB, DigitalTwinCatalogue (DTC), and ObjectsWithLighting (OWL) inverse rendering benchmarks (PSNR-H and PSNR-L denote HDR and LDR evaluation; CD denotes Chamfer Distance \(\times 10^{-2}\)):

Dataset	Method	PSNR-H (↑)	PSNR-L (↑)	SSIM (↑)	LPIPS (↓)	CD (↓)
StanfordORB	NVDiffrecMc (Optimization)	24.43	31.60	0.972	0.036	0.51
	InvRender (Optimization)	23.76	30.83	0.970	0.046	0.44
	NeuralPBIR (Optimization)	26.01	33.26	0.979	0.023	0.43
	LIRM (Feed-forward)	25.09	32.45	0.972	0.025	0.38
	LSRM (Ours, Feed-forward)	25.47	32.85	0.977	0.022	0.29
DTC	NVDiffrecMc (Optimization)	27.78	34.55	0.952	0.042	-
	InvRender (Optimization)	29.52	35.98	0.961	0.037	-
	LIRM (Feed-forward)	27.65	34.84	0.960	0.031	-
	LSRM (Ours, Feed-forward)	29.67	35.66	0.964	0.025	-
OWL	NVDiffrecMc (Optimization)	19.82	-	0.730	0.389	-
	InvRender (Optimization)	23.77	-	0.780	0.369	-
	LIRM (Feed-forward)	23.27	-	0.770	0.322	-
	LSRM (Ours, Feed-forward)	24.88	-	0.790	0.269	-

In sparse-view generalization, despite being trained solely on 12–16 views, LSRM evaluated on GSO with only 8 input views achieves 32.46 dB PSNR and 0.031 LPIPS, outperforming 16-view LIRM (30.56 dB PSNR and 0.054 LPIPS). On StanfordORB with 6 views, LSRM achieves 0.023 LPIPS and 0.32 CD, outperforming 18-view LIRM (0.025 LPIPS and 0.38 CD).

Ablation Study

Ablation experiments on GSO validate the independent contributions of context window scaling and 3D-aware block routing:

Config	Token Resolutions	3D-Aware Routing	PSNR (↑)	SSIM (↑)	LPIPS (↓)	Note
Stage 1 Dense	\(S_\text{img}=32, S_\text{vol}=16\)	None	28.45	0.934	0.079	Coarse dense baseline
LSRM (Reduced Vol)	\(S_\text{img}=64, S_\text{vol}=48\)	w/ routing	31.68	0.958	0.044	Scaled image, reduced volume
LSRM (Reduced Img)	\(S_\text{img}=48, S_\text{vol}=64\)	w/ routing	31.82	0.959	0.043	Reduced image, scaled volume
LSRM (Medium Scale)	\(S_\text{img}=64, S_\text{vol}=64\)	w/o routing	31.34	0.956	0.045	Standard NSA score-based routing
LSRM (Medium Scale)	\(S_\text{img}=64, S_\text{vol}=64\)	w/ routing	31.93	0.962	0.040	Effect of geometric routing
LSRM (Full Scale)	\(S_\text{img}=96, S_\text{vol}=96\)	w/o routing	32.72	0.968	0.032	Large context without 3D routing
LSRM Full Model	\(S_\text{img}=96, S_\text{vol}=96\)	w/ routing	33.08	0.971	0.028	Best overall performance

Key Findings

• Context window scaling is the primary driver of fidelity: Scaling active token resolutions from \(64 \times 64\) to \(96 \times 96\) produces a 1.38 dB PSNR increase and cuts LPIPS from 0.045 to 0.032, proving that expanding the effective token context directly determines high-frequency detail recovery.
• Volume token capacity outweighs image token count: Reducing \(S_\text{vol}\) from 64 to 48 drops PSNR by 0.25 dB (31.93 \(\to\) 31.68), whereas reducing \(S_\text{img}\) from 64 to 48 drops PSNR by only 0.11 dB (31.93 \(\to\) 31.82), indicating that 3D spatial capacity is the more critical structural bottleneck.
• 3D-aware routing consistently sharpens micro-structures: Across both medium and full resolutions, adding explicit 3D routing provides a 0.36–0.59 dB PSNR boost and reduces LPIPS by 11%–12.5%, successfully recovering legible nutrition facts and uncorrupted facial features.

Highlights & Insights

• Geometric inductive bias for transformer attention: Substituting data-driven semantic attention scores with physical camera projections and Euclidean 3D proximity grounds cross-attention from the very first layer, eliminating early-stage geometric misallocations.
• Synergy of GQA compression with All-gather-KV: Leveraging an aggressive 16:1 query-to-KV head ratio allows global KV cache replication across GPUs with negligible communication overhead, resolving straggler delays while preserving global receptive fields.
• Superior few-view zero-shot generalization: Despite being trained exclusively on 12–16 views, LSRM's dense geometric representation allows it to reconstruct superior geometry and textures from 6–8 views compared to baselines explicitly trained on sparse view regimes.

Limitations & Future Work

• Global material parameter ambiguities: In inverse rendering, unresolvable ambiguities between illumination intensity, surface roughness, and diffuse albedo lead to slight global color or reflectivity shifts, limiting PSNR gains despite superior perceptual LPIPS.
• Ultra-fine micro-text illegibility: Highly minute text (such as tiny ingredient lists on beverage cans) remains challenging at \(96^3\) voxel grid resolutions; integrating sequence parallel mechanisms like Ulysses or Ring Attention could expand context sizes to tens of millions of tokens.
• Extension to large-scale unbounded scenes: Currently focused on object-centric domains, extending block-aware sparse attention to dynamic environments and unbounded scene-level reconstruction represents an important future direction.

Related Work & Insights

• vs LIRM / MeshLRM: Prior feed-forward models rely on monolithic self-attention or hexaplane representations restricted to tens of thousands of tokens, yielding smeared high-frequency textures. LSRM expands object tokens by 20× and image tokens by >2×, outperforming LIRM on GSO by >2.4 dB PSNR and >40% lower LPIPS.
• vs NeuralPBIR / NVDiffrecMc: Optimization-based methods require per-scene iterative tuning (often tens of minutes) and struggle with specular highlights. LSRM reconstructs objects in ~5 seconds per sample on a single H100 while matching or surpassing optimization methods in perceptual fidelity (0.022 vs 0.023 LPIPS on StanfordORB).
• vs Direct3D-S2: Direct3D-S2 explores sparse attention for single-image generative geometry synthesis without appearance modeling. LSRM adapts Native Sparse Attention to deterministic multi-view reconstruction and physical material decomposition, solving the fine-texture bottleneck.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Systematic integration of native sparse attention, explicit 3D geometric routing, and block-aware sequence parallelism for feed-forward 3D vision)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive validation across NVS, three standard inverse rendering benchmarks, fine-grained ablations, and sparse-view stress tests)
• Writing Quality: ⭐⭐⭐⭐⭐ (Exceptionally well-structured paper with clear motivation, rigorous mathematical formulation, and informative visual comparisons)
• Value: ⭐⭐⭐⭐⭐ (Bridges the long-standing fidelity gap between real-time feed-forward models and offline optimization, paving the way for scalable 3D digital twin pipelines)