URoPE: Universal Relative Position Embedding across Geometric Spaces¶

Conference: ECCV2026
arXiv: 2604.18747
Code: https://urope-pe.github.io/
Area: 3D Vision
Keywords: Position Embedding, Cross-view Attention, RoPE, Projective Geometry, Multi-view 3D

TL;DR¶

URoPE lifts key image patches into 3D points along camera rays at fixed depth anchors, then reprojects them onto the query view image plane. This enables standard 2D RoPE to directly encode relative positional relationships across views or geometric spaces. It serves as a parameter-free, intrinsic-aware, coordinate-system-agnostic, and FlashAttention-compatible universal position embedding.

Background & Motivation¶

Transformers have become the dominant architecture for multi-view perception and representation tasks. Challenges ranging from novel view synthesis and stereo matching to 2D/3D object detection require encoding spatial relationships across different views, coordinate systems, and even different geometric modalities (2D images and 3D point clouds). Relative position embeddings (especially RoPE) have become the mainstream choice due to their generalization capability and length extrapolation. However, standard RoPE can only operate within a single, flat coordinate space—assigning positions based on sequence indices or image grid locations. In cross-view scenarios, pixels that are close in 3D space across two cameras may be far apart in their respective 2D grids. This fundamental misalignment prevents standard 2D RoPE from encoding cross-view correspondences.

Recent works have begun integrating camera geometry into attention modules, but each has limitations. Plücker ray embeddings are absolute encodings and lack relative bias properties; GTA and P-RoPE decouple cross-view geometry and in-image positions into different dimension blocks, preventing interaction between them, and cross-view geometry is considered only at the camera level rather than the patch level; RayRoPE uses a learnable module per layer to predict depth, but unsupervised prediction cannot guarantee realistic scene depths. The fundamental difficulty of cross-view relative position encoding lies in depth ambiguity—it is uncertain which point on the query view epipolar line a source view pixel corresponds to.

The observation of URoPE is that the core issue of cross-view relative position is "where the 3D content of the key token appears in the query token's image." Core Idea: For each key image patch, sample 3D points at multiple fixed depth anchors along its camera ray, project them onto the query view image plane, and then use standard 2D RoPE to encode the relative relationship between the query position and the projected positions, with different attention heads assigned to different depth hypotheses.

Method¶

Overall Architecture¶

The process of URoPE starts from each pixel in the key view. First, it constructs a 3D ray for the pixel using camera intrinsic and extrinsic parameters, samples 3D points at multiple fixed depth anchors, and projects these 3D points onto the query view's image plane to obtain a set of depth-conditioned projected pixel coordinates. Finally, within the query image plane, standard 2D RoPE is applied to the query position and the projected position corresponding to each depth anchor, generating a geometry-aware relative position bias. Different attention heads are assigned to different fixed depth anchors, allowing multi-head attention to collectively cover all possible correspondences from near to far fields.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Key View Pixel<br/>(u,v)"] --> B["Construct 3D Ray<br/>r = R^T K^{-1}[u,v,1]^T"]
    B --> C["Depth Anchor Lifting<br/>p^h = o + d^h · r"]
    C --> D["Project to Query View<br/>(u^h,v^h) = proj(p^h)"]
    D --> E["Head h: 2D RoPE<br/>(u',v') ↔ (u^h,v^h)"]
    E --> F["K Heads Cover K Depths<br/>Near-to-Far Correspondence"]

Key Designs¶

1. Depth Anchor Lifting and Projection: Bringing Cross-View Correspondences onto the Same Image Plane Each pixel in the key view corresponds to a ray through the camera center in 3D space. URoPE uses a set of fixed depth anchors \(D = \{d^1, \dots, d^K\}\)—avoiding reliance on layer-by-layer learned depth predictions (where shallow features lack geometric cues and learnable module predictions can be unstable). For each pixel, 3D points \(p^h = o + d^h \cdot r(u,v)\) are sampled along the ray at depth \(d^h\), and then projected onto the query image plane using the query view's intrinsic and extrinsic parameters to obtain the projected coordinates \((u^h_{i \to j}, v^h_{i \to j})\). The projected point lies precisely on the epipolar line induced by the source pixel, providing an explicit, intrinsic-aware mapping from the source view to the query view. When the source view and query view are identical, the projection degenerates to an identity mapping, and URoPE naturally downscales to standard 2D RoPE.

2. Depth Anchor Multi-Head Attention: Encoding One Depth Hypothesis per Head Cross-view projection inherently suffers from depth ambiguity—a source pixel corresponds to an entire epipolar line in the query view. URoPE assigns different attention heads (or head groups) to handle different fixed depth anchors. Each head independently applies 2D RoPE (splitting channels equally between horizontal and vertical) to the query position and the projected key position within the query image plane. Head-wise assignment does not cause multi-head collapse: experiments show balanced attention distributions across heads, and the importance of depth anchors in intermediate layers highly correlates with actual scene depth, enabling the network to adaptively leverage anchors of different depths based on local information. Fixed anchors outperform learnable depth prediction (PSNR 25.57 \(\to\) 26.01), as shallow features are insufficient for accurate depth estimation.

3. Natural Extension to 2D-3D Cross-Dimensional Interactions For tasks like 3D object detection that require interaction between image features and 3D queries, URoPE skips the image plane projection step and directly calculates RoPE based on the relative positions between query locations in 3D space and the lifted 3D points. The same set of formulations unifies 2D-2D (cross-view attention), 2D-3D (3D detection and tracking), and temporal scenarios, demonstrating the simplicity and universality of the URoPE concept.

4. Efficient Parallel Implementation for Multi-View Sequences When queries come from \(N\) views of the same sequence, the view dimension is shifted from the sequence length to the batch dimension, copying keys/values \(N\) times along the batch dimension. Consequently, each batch sample corresponds to a single query view, and the attention computation becomes completely identical to standard RoPE. The computational complexity remains unchanged at \(\mathcal{O}(B \cdot H \cdot L^2 \cdot C)\). URoPE introduces no extra asymptotic computational overhead and is fully compatible with RoPE-optimized kernels such as FlashAttention.

A Complete Example¶

Taking novel view synthesis as an example: given an image patch \((u', v')\) in the target view (query) and pixels from two reference views. For a reference view pixel \((u, v)\), URoPE samples 3D points along its ray at 4 fixed depth anchors (e.g., [2m, 8m, 14m, 20m]). Assuming this pixel actually corresponds to an object at 10m in the scene, the anchor at depth 8m will project closest to the object's true position in the query view. The attention head corresponding to this anchor will assign a higher weight, while other heads will assign lower weights. The 4 heads are aggregated via multi-head attention, collectively covering all possible corresponding depths along the epipolar line.

Key Experimental Results¶

Main Results¶

Novel View Synthesis (Objaverse / RealEstate10k): URoPE outperforms all baselines across PSNR, SSIM, and LPIPS. It achieves state-of-the-art results on both Objaverse (synthetic data with randomly varying camera focal lengths) and RealEstate10k (real-world scenes).

Dataset	Metric	Plücker	6D RoPE	P-RoPE	RayRoPE	URoPE
Objaverse	PSNR↑	22.28	24.42	24.88	24.96	25.09
Objaverse	SSIM↑	0.856	0.891	0.896	0.897	0.900
RealEstate10k	PSNR↑	23.95	25.73	25.28	24.94	26.02
RealEstate10k	SSIM↑	0.764	0.819	0.806	0.799	0.827

3D Object Detection and Tracking (nuScenes): Consistently improves detection and tracking performance under both PETR (single-frame) and StreamPETR (multi-frame) frameworks.

Method	NDS↑	mAP↑	AMOTA↑
PETR	34.9	30.9	0.222
+ URoPE	37.3	32.2	0.255
StreamPETR	47.6	37.5	0.335
+ URoPE	50.6	41.1	0.380

Ablation Study¶

Configuration	PSNR	Description
Plücker(Abs)+2D RoPE	24.96	Absolute encoding + standard relative
Plücker+URoPE	25.89	Hybrid absolute + relative
Ours Only	25.85	No absolute encoding assistance needed
Depth Anchors = 1	25.37	Single anchor severely degrades
Depth Anchors = 4	26.01	4 anchors reach optimum
Channel-wise Splitting (4 anchors)	25.47	Far weaker than head-wise splitting
Learnable Depth Prediction Module	25.57	Inferior to fixed anchors

Key Findings¶

Around 4 depth anchors yield the best performance; more anchors (8, 16) bring no additional gains.
Head-wise depth assignment is far superior to channel-wise splitting—the head-wise approach can accommodate more varied frequency components, handling both short-range and long-range relationships simultaneously.
Fixed anchors outperform learnable depth prediction: shallow features lack geometric cues, making it difficult for learnable modules to accurately estimate depth.
URoPE is insensitive to the choice of depth range (only needing to cover the near range) and exhibits strong robustness to camera noise.
URoPE maintains significant gains (PSNR +0.58) even at a 50x computational scale, showing excellent scalability.

Highlights & Insights¶

Most Elegant Design: The combination of fixed depth anchors and multi-head attention elegantly resolves the depth ambiguity in cross-view projection without any learnable parameters, theoretically covering all possible correspondences along the epipolar line.
Unified Framework: The same set of formulations naturally covers 2D-2D (cross-view), 2D-3D (detection), and temporal scenarios, degenerating to standard RoPE in single-view scenarios to maintain backward compatibility.
Pure RoPE Approach: It avoids altering the Q/K/V multiplication format and introduces no additional matrix multiplications, being naturally compatible with FlashAttention—a key engineering advantage over GTA (matrix multiplication) and P-RoPE (RoPE + MatMul).
Highly Valuable Table 1: Systematically compares all cross-view position encoding methods across five dimensions: mechanism, geometric information, per-patch granularity, SE(3) invariance, and parameter-freeness.

Limitations & Future Work¶

Relies on known camera intrinsic and extrinsic parameters, making it inapplicable to uncalibrated scenarios (e.g., in 3D reconstruction frameworks like VGGT or DepthAnything).
Limited by computational resources, validation on large-scale models has not been performed (though a 50x scaling experiment was conducted to provide preliminary evidence of scalability).
Future directions include extending to uncalibrated setups by incorporating estimated camera parameters and integrating into larger generative/perceptual foundation models.

vs. P-RoPE: Both use projection matrices; however, P-RoPE decouples cross-view geometry from in-image RoPE into different head dimension blocks with no local interaction, whereas URoPE unifies them on a single image plane through explicit projection.
vs. RayRoPE: RayRoPE uses a learnable module per layer to predict depth, but unsupervised prediction cannot guarantee accuracy; URoPE's fixed depth anchors require no learning, are more stable, and yield superior results.
vs. Plücker Ray Embedding: Plücker is an absolute encoding (concatenating 6D ray vectors) and lacks relative bias; URoPE encodes relative positions via rotation matrices.

Rating¶

Novelty: ⭐⭐⭐⭐ Combines explicit projective geometry with RoPE to solve the cross-view position encoding problem, offering a simple and engineering-friendly approach.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three major tasks + comprehensive ablation studies + scaling experiments + noise robustness analysis + multi-head collapse validation.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, clean methodology derivations, and Table 1 provides a highly informative systematic comparison.
Value: ⭐⭐⭐⭐⭐ Parameter-free, plug-and-play, and compatible with efficient kernels; has the potential to become the standard position encoding scheme for cross-view Transformers.