RayRoPE: Projective Ray Positional Encoding for Multi-view Attention¶

Conference: ECCV 2026
Paper: ECCV Paper
Project: RayRoPE
Area: 3D Vision
Keywords: multi-view attention, rotary positional encoding, projective rays, depth uncertainty, SE(3) invariance

TL;DR¶

RayRoPE represents image patches as ray segments in the query camera frame and computes expected RoPE over uncertain predicted depths, improving multi-view geometry without additional supervision of internal depths and reducing small-scale LVSM's RE10K LPIPS from 0.112 with PRoPE to 0.085.

Background & Motivation¶

For a single image, two-dimensional patch coordinates tell attention where content is located; multi-view inputs have no shared two-dimensional grid. The same pixel index can belong to completely different cameras, while the same viewing ray can receive different indices after cropping. Concatenating Plucker raymaps with inputs supplies geometry, but the network must still handle changes caused by an arbitrary world coordinate frame. Applying ordinary RoPE directly to world-coordinate rays likewise fails to remove the effect of global rotations on the encoding.

CaPE, GTA, and PRoPE already use relative camera transformations to achieve SE(3) invariance: attention should not change when the entire camera-and-scene system undergoes a rigid coordinate transformation. However, these encodings mainly describe camera placement rather than using scene geometry to determine where two rays intersect. The additional two-dimensional patch RoPE in GTA and PRoPE supplies multi-frequency information, but assigns different position descriptions to the same physical observation after cropping and does not make camera relationships themselves multi-frequency.

The paper therefore seeks coordinate-frame invariance, consistent representation of the same physical ray, adaptability to scene depth, and multi-frequency similarity together. Here, uniqueness concerns the positional representation; it does not guarantee identical features for every token observing the same object. The central difficulty is that locating a point using predicted depth introduces depth errors into positional phases, with high-frequency channels particularly sensitive. Core Idea: let each layer predict a token's depth along its ray and an uncertainty interval, then compute expected multi-frequency positional encoding in the query camera frame so geometry informs attention while unreliable depth has less influence on high-frequency encoding.

Method¶

Overall Architecture¶

Inputs are multi-view images with camera intrinsics and extrinsics, together with their patch features. Outputs remain the host model's attention updates, followed by its original task head for novel views, depth, or 3D Gaussian Splatting parameters. RayRoPE does not first reconstruct a complete three-dimensional scene. Instead, attention progressively refines each token's position from a camera ray to a ray segment that may lie within the scene.

The sequence consists of depth-adaptive rays, query-camera projection, and uncertainty-aware expected encoding. The first two establish which positions are compared and in which coordinate frame; the third handles rotational phases when those positions are uncertain. Encoding also affects value and output features rather than only query-key scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view features<br/>Camera intrinsics and extrinsics"] --> B["Depth-adaptive rays"]
    B --> C["Query-camera projection"]
    C --> D["Uncertainty-aware expected encoding"]
    D --> E["Group by query view<br/>Update attention"]
    E --> F["Original task head<br/>Image or 3D output"]
    F -.-> G["Original task loss during training<br/>No extra internal depth supervision"]

Solid edges indicate the data flow shared by training and inference; the dashed edge indicates training supervision, not another inference module. In the ordinary RGB setting, internal depths are learned indirectly through the task loss. Experiments with known reference depths use a separate input condition and must not be interpreted as providing depth inputs in every experiment.

Key Designs¶

1. Depth-adaptive rays

Each patch initially corresponds to a ray from the camera center through the patch center. Representing only its origin and direction avoids binding the same physical ray to crop-dependent patch indices, but cannot distinguish a nearby tabletop from a distant wall along that ray. RayRoPE therefore represents position using the camera center and a point at a predicted depth. A new linear prediction head in each attention layer estimates depth from the current token features. Successive layers can revise their estimates instead of requiring shallow features to determine final geometry immediately.

The representation can be summarized in the following form, explicitly stated in the paper, with the camera center and ray point expressed in homogeneous coordinates:

\[ \mathbf{x}=(\mathbf{c},\mathbf{p}^{d}). \]

Depth determines where along the ray matching takes place, allowing cross-view relationships to depend on scene content. The camera center preserves the observation's origin, so the representation is not merely an encoding of a surface point. Features predict depth when it is unavailable; known depths can directly replace the internal estimates. The ablation without geometry adaptiveness uses a point at infinity, reverting to a direction-only ray rather than simply setting depth to zero.

2. Query-camera projection

Although rays have geometric meaning, applying multidimensional RoPE directly in world coordinates still depends on the orientation of the coordinate axes. The paper establishes a common comparison frame for each query camera: it transforms every token's camera center using the query extrinsics, then projects the ray endpoint using the query camera's full projection matrix. The center retains three spatial coordinates, while the endpoint contributes two projected coordinates and inverse depth, forming a six-dimensional position.

The prose clearly defines this vector as:

\[ \widetilde{\mathbf{x}}=[c_x,c_y,c_z,u,v,1/d']. \]

The first three components describe the camera center in the query camera frame, and the last three describe the projected endpoint. Here, \(d'\) is the endpoint's depth relative to the query camera and must not be confused with \(d\) predicted in its source camera. Applying multi-frequency RoPE to these components makes attention compare relative positions within the same query frame. Relative geometry cancels changes to the global coordinate system, while different frequency channels capture positional differences at different scales. Projection incorporates intrinsics, so this is more than rotary encoding of extrinsics alone.

3. Uncertainty-aware expected encoding

High-frequency rotations can turn small depth errors into large phase errors. To prevent attention from trusting unreliable early-layer depths too strongly, the linear head also predicts an uncertainty range. Subtracting and adding this range to depth yields two endpoints along the ray. After projection, the method assumes a uniform interval distribution for each coordinate component and analytically computes expected RoPE, avoiding repeated sampling along every ray. This is an assumption about projected coordinate components, not a proof that uniformly distributed depth remains uniform after perspective projection.

Intuitively, substantially different rotation phases within an interval cancel when averaged, suppressing high-frequency responses for uncertain positions. More precise positions approach ordinary RoPE, while deterministic components such as camera centers remain unsmoothed. An expectation of rotation matrices is generally no longer a pure rotation matrix. The cached equations involving products, transposes, and inverses are damaged, so this note does not reconstruct implementation-level identities from them or assume that the inverse of an average matrix equals the average of its inverses.

Encoding enters the query, key, value, and output paths, allowing geometry to affect both which tokens receive attention and how information is aggregated. Computation is grouped by query view: queries from the same camera share the projection reference and attend to all keys and values. This does not reduce the number of attention tokens and is not a sparse attention algorithm. Efficiency comes from organizing computation while retaining compatibility with implementations such as FlashAttention and KV caching. The reported modest overhead applies only to the corresponding measurement setup.

A Worked Example¶

Consider LVSM with two reference images and one target camera, whose three groups of view tokens undergo joint self-attention. A patch in a reference image observes a table edge. An early layer can only roughly determine where it lies along the ray and therefore predicts a broad uncertainty interval. This is an illustrative mechanism walkthrough, not an additional quantitative experiment from the paper.

When a target-view token issues a query, ray centers and candidate endpoints from both reference images are expressed relative to the target camera. If the other reference image observes the same edge, the predicted endpoint projections can provide correspondence cues across views, while the distinct camera centers remain represented. Unlike comparing two-dimensional patch indices alone, this relationship can reflect camera motion and object depth.

If endpoints remain unreliable, expected encoding suppresses high-frequency components that are sensitive to errors. Later layers use the fused features to predict depth and uncertainty again, potentially sharpening geometric positions. Attention remains a soft correspondence mechanism: it does not require rays to intersect exactly and introduces no separate explicit triangulation step. LVSM's original task head still generates the target image; internal depth maps are not required inputs to a renderer.

Loss & Training¶

RayRoPE's internal depth and uncertainty heads are trained jointly with the host network, without additional direct supervision of those internal variables. This does not mean that every downstream task avoids depth labels: UniMatch's final task is itself depth estimation, which is distinct from depth prediction inside attention.

The main experiments follow PRoPE's LVSM comparison protocol. Downsized models with approximately 47M parameters are trained from scratch using two posed reference images and a target camera, separately on CO3D, Objaverse, and RE10K. Objaverse uses rendered data from a high-quality 80K subset with diverse intrinsics. Both PRoPE and RayRoPE concatenate the CamRay intrinsics raymap at the input. Results are averaged over three random seeds except in the larger-scale experiments.

The larger experiments use approximately 150M parameters and an 8-fold batch size, so comparisons should remain within matching scales. In the known-depth experiments, all methods receive reference depth maps, while RayRoPE additionally replaces predicted reference depths with ground truth and sets their uncertainty to zero. Target-view depths and uncertainties remain predicted. The cached main paper does not include the referenced appendix, so this note does not supply unsupported learning rates, training steps, or exact split sizes.

Key Experimental Results¶

Main Results¶

The following results are selected from Table 1 for small-scale LVSM, trained and evaluated separately on each dataset rather than evaluated zero-shot across datasets. PSNR measures pixel reconstruction fidelity in dB and is higher-is-better; LPIPS measures perceptual discrepancy and is lower-is-better.

Dataset	Method	PSNR ↑	LPIPS ↓
CO3D	PRoPE	17.49	0.539
CO3D	RayRoPE	18.40	0.461
Objaverse	PRoPE	22.16	0.123
Objaverse	RayRoPE	22.42	0.110
RE10K	PRoPE	24.48	0.112
RE10K	RayRoPE	26.07	0.085

The relative RE10K LPIPS reduction is approximately 24.1%, using PRoPE's 0.112 as the denominator, not the original LVSM result. In the same table, RoPE-on-rays achieves 25.29 dB and 0.095 on RE10K, so the reduction relative to PRoPE must not be described as the reduction relative to the strongest baseline. With larger models, RayRoPE and PRoPE respectively achieve 28.31/27.77 dB and 0.055/0.059 LPIPS on RE10K.

Ablation Study¶

The following entries come from Table 2 under the same small-scale LVSM training conditions. CO3D and Objaverse LPIPS results distinguish the contributions of uncertainty and geometry adaptiveness.

Config	CO3D LPIPS ↓	Objaverse LPIPS ↓	Note
Full RayRoPE	0.461	0.110	Depth, uncertainty, multiple frequencies, and full feature encoding
Without uncertainty	0.594	0.175	Encode predicted points without interval expectations
Without geometry adaptiveness	0.553	0.111	Replace predicted depth points with points at infinity
Without multiple frequencies	0.550	0.223	Stop representing position at multiple frequencies
Without value / output encoding	0.510	0.127	Encode queries and keys only

Key Findings¶

Removing uncertainty raises CO3D LPIPS from 0.461 to 0.594. On Objaverse, the largest deterioration among the listed ablations comes from removing multiple frequencies, increasing LPIPS from 0.110 to 0.223. The dominant factors differ across datasets.
Removing geometry adaptiveness leaves Objaverse PSNR at 22.42 dB and changes LPIPS only from 0.110 to 0.111; depth prediction should not be characterized as a large contributor on every dataset. Conversely, adding ground-truth reference depths in Table 3 reduces RayRoPE's CO3D LPIPS from 0.461 to 0.284, but that is a separate experiment with additional input information.
In UniMatch, Table 4 reports that unseen ScanNet Abs Rel decreases from PRoPE's 0.101 to 0.095, and RMSE decreases from 0.285 to 0.276. These measure mean absolute relative depth error and root mean square depth error, respectively, and both are lower-is-better.
TokenGS in Table 5 is trained on DL3DV-10K with four views. At six-view evaluation, PSNR rises from PRoPE's 24.09 to 24.63 dB. Its 3DGS queries lack explicit positions and require special adaptation: only image features are encoded, and ray projection is skipped. The query-camera mechanism is therefore not applied unchanged to every cross-attention setting.
Figure 3 and Section 4.4 report 4% training and 13% inference overhead relative to PRoPE for LVSM with three total views. This is not a speedup or a fixed ratio across arbitrary hardware and view counts. Figure 5 shows a positive association between depth errors and predicted uncertainty in deeper layers but supplies no formal calibration-error metric.

Highlights & Insights¶

Position becomes a geometric variable updated across layers instead of a constant determined only by cameras and patch indices. Geometry can then influence attention and improve through the information attention aggregates.
Uncertainty directly changes the encoding's frequency response rather than serving only as an output confidence score. Other noisy-coordinate positional encodings could adopt this idea, but their distributional assumptions would require fresh validation.
The query camera frame combines global coordinate invariance with multi-frequency comparison. Its benefit comes from choosing the comparison space, not merely adding a larger depth network.

Limitations & Future Work¶

The authors explicitly identify camera-matrix uncertainty as unmodeled in Section 6. Unposed inputs and mixtures of posed and unposed images remain open problems; depth uncertainty does not automatically address camera-calibration errors.
This note's analysis: uniform distributions over projected coordinate components are an analytically convenient approximation, not a complete scene-depth posterior. Whether multimodal depths, occlusion boundaries, and projection singularities require richer distributions needs dedicated experiments.
This note's analysis: internal-depth visualizations and error correlations establish meaningful geometric behavior, but do not independently establish metric accuracy or calibrated uncertainty. The available main paper does not contain all reproduction details, and performance improvements cannot substitute for implementation-level verification.

vs CaPE / GTA / PRoPE: these methods mainly establish relative transformations through camera matrices. RayRoPE incorporates ray endpoints, projected positions, and multiple frequencies into a common geometric representation, with endpoints that can vary with the scene.
vs RoPE-on-rays: this baseline applies rotary encoding directly to world-coordinate rays, retaining multiple frequencies without global rotation invariance. The distinction shows that using rays alone does not make a method independent of coordinate choices.
vs LVSM / TokenGS / EscherNet: these are host task models, not complete architectures that RayRoPE replaces. Transfer results support the value of the positional encoding module, while TokenGS's special adaptation emphasizes checking whether queries genuinely have camera semantics.

Rating¶

Novelty: 4/5. Query-frame ray positions and uncertainty-aware expected encoding form a coherent multi-view RoPE design.
Experimental Thoroughness: 4/5. Multiple tasks, ablations, scales, and generalization settings are covered, but camera errors and uncertainty calibration still need dedicated evaluation.
Writing Quality: 4/5. Design goals align clearly with ablations; damaged equations in the available cache and unresolved figure references in the prose reduce reproducibility-oriented readability.
Value: 4/5. Useful for multi-view models with known cameras, strengthening geometric inductive bias at a limited additional computational cost.