RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation¶

Conference: ECCV 2026
arXiv: 2606.22749
Code: https://github.com/MAP-RaysUp/RaysUp
Area: Model Compression
Keywords: Feature Upsampling, Ultra-lightweight Network, RayPE, Geometry-Aware Attention, VFM-agnostic Upsampling

TL;DR¶

An ultra-lightweight, task-agnostic, and VFM-agnostic universal feature upsampling framework named RaysUp with only 0.14M parameters is proposed. Through three key designs—spatial decoupled guidance encoder, Ray Positioning Encoding (RayPE), and geometry-aware neighborhood cross-attention—it lifts feature reconstruction from the 2D pixel plane to the 3D ray domain. It achieves state-of-the-art (SOTA) or near-SOTA performance across various dense prediction tasks such as semantic segmentation, depth/normal estimation, video object segmentation, and open-vocabulary segmentation, while offering an inference speed approximately 7 times faster than AnyUp, the only prior VFM-agnostic method.

Background & Motivation¶

Visual Foundation Models (VFMs) such as DINOv2/v3, SigLIP, PE Spatial, and MAE have become the backbone of modern computer vision, providing strong semantic representation and cross-task generalization through large-scale pre-training. They are widely applied in downstream tasks like depth estimation, open-vocabulary segmentation, and 3D semantic field reconstruction. However, these models generally rely on patchification or pooling operations, leading to extremely low spatial resolutions for output feature maps (typically \(16 \times 16\) or \(32 \times 32\)), which must be upsampled for dense prediction tasks requiring fine pixel-level understanding. Although traditional interpolations (bilinear, nearest-neighbor) are fast, they lack content adaptability, causing semantic distortion. Learnable upsampling methods like LiFT, FeatUp, LoftUp, and JAFAR demonstrate task-agnostic potential, but most require re-training for different VFMs or image-by-image optimization during inference, incurring high deployment costs. The recent AnyUp achieved VFM-agnostic universal upsampling for the first time, but its network is heavy (0.87M parameters, 84 GFLOPs), remaining impractical for large-resolution or high-throughput scenarios.

A deeper common limitation is that all these methods operate on 2D image grids, implicitly assuming that Euclidean proximity between pixels is equivalent to geometric proximity in 3D space. However, this assumption often collapses under perspective projection—adjacent pixels at depth discontinuities might be far apart in 3D space, whereas distant pixels might belong to the same physical surface. Consequently, interpolation or attention mechanisms based on 2D neighborhoods cannot guarantee geometric consistency of reconstructed features in real 3D space, which manifests as blurry boundaries and structural drift. This contradiction is particularly prominent in geometrically sensitive tasks such as surface normal and depth estimation.

Starting from the classic Joint Bilateral Upsampling (JBU) paradigm, this paper clearly upgrades its three core components one by one: replacing the RGB difference range kernel with a lightweight directionally decoupled guidance encoder, injecting 3D geometric priors via RayPE, and replacing the fixed 2D local neighborhood with geometry-aware neighborhood cross-attention. Core Idea: To lift feature upsampling from the 2D pixel plane to the 3D ray domain, inject implicit geometric priors into the attention mechanism via 6D Plücker ray coordinates in the form of rotational modulation, and achieve high-quality, task-agnostic, VFM-agnostic, and arbitrary-resolution feature reconstruction within an ultra-lightweight framework of only 0.14M parameters.

Method¶

Overall Architecture¶

The input of RaysUp is an RGB image \(\mathcal{I}\) and a low-resolution feature map \(\mathcal{F}^{lr}\) extracted from an arbitrary VFM (typically with a resolution of only \(16 \times 16\) or \(32 \times 32\)). The goal is to restore \(\mathcal{F}^{lr}\) to any specified target resolution. The entire pipeline consists of four steps: ① A lightweight spatial decoupled guidance encoder extracts direction-aware guidance features \(\mathcal{F}_g\) from the RGB image; ② Two adaptive average poolings are applied to \(\mathcal{F}_g\) to generate the target-resolution query \(\mathcal{Q}_g\) and the VFM-resolution key \(\mathcal{K}_g\), respectively; ③ Both are encoded by RayPE to obtain the ray query \(\mathcal{Q}_{ray}\) and ray key \(\mathcal{K}_{ray}\) carrying 3D geometric priors; ④ Geometry-aware neighborhood cross-attention uses \(\mathcal{K}_{ray}\) and \(\mathcal{Q}_{ray}\) as the attention pair and the original low-resolution features \(\mathcal{F}^{lr}\) as the value \(\mathcal{V}\) to perform cross-resolution feature aggregation within local windows, outputting high-resolution features \(\mathcal{F}^{hr}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB Image I +<br/>Low-Res VFM Feature F_lr"] --> B["Spatial Decoupled Guidance Encoder<br/>4 Parallel Directional Branches"]
    B --> C["Guidance Feature F_g<br/>(Dg×H×W)"]
    C --> D1["Adaptive Pooling → Target Resolution<br/>Query Q_g"]
    C --> D2["Adaptive Pooling → VFM Resolution<br/>Key K_g"]
    D1 --> E["RayPE Ray Position Encoding<br/>(6D Plücker Coordinates<br/>→ Multi-Frequency Rotary Modulation)"]
    D2 --> E
    E --> F1["Q_ray (Geometry-Aware)"]
    E --> F2["K_ray (Geometry-Aware)"]
    F1 --> G["Geometry-Aware Neighborhood Cross-Attention<br/>Local k×k + Dilated Alignment"]
    F2 --> G
    A -->|"F_lr as Value V"| G
    G --> H["High-Resolution Feature Map F_hr"]

Key Designs¶

1. Spatial Decoupled Guidance Encoder: From Anisotropic Empirical Observations to Directional Decomposition

The guidance encoder is responsible for extracting structural priors from the RGB image. In analyzing the weight distribution of the \(3 \times 3\) convolutional kernel in JAFAR, the authors discovered a critical pattern: the kernel weights are not uniformly distributed—the four corners hold high weights (0.97–0.99), whereas the central cross area has significantly lower weights (only 0.78 at the center). This implies that standard square convolutions are anisotropic during spatial aggregation, and insufficient interaction in the center channel can lead to holes in the upsampled features. Inspired by this, instead of a single \(3 \times 3\) convolution, RaysUp explicitly decomposes the receptive field into four orthogonal directional components processed in parallel: a center branch with \(1 \times 1\) convolution for point-wise channel mixing, a horizontal branch with \(1 \times 3\) convolution to capture left-right textures, a vertical branch with \(3 \times 1\) convolution to capture top-bottom textures, and a diagonal branch with \(2 \times 2\) dilated convolution (dilation rate of 2) to capture diagonal structures. Each branch is only responsible for \(D_g/4\) channels, which are finally concatenated along the channel dimension. This decoupling restores the center weight to 1.00 while maintaining high weights of 0.89–0.99 for the peripheral branches, fixing the insufficient center channel mixing while preserving directional sensitivity. In terms of parameter efficiency, a standard \(3 \times 3\) convolution requires \(27D_g\) parameters, whereas the decoupled encoder only needs about \(8.25D_g\), saving nearly 70% of parameters while achieving stronger performance.

2. RayPE Ray Position Encoding: Replacing 2D Coordinate Encoding with 3D Rotary Modulation

Traditional positional encodings are defined on 2D pixel grids, assuming closer pixels are more relevant, which is erroneous at depth discontinuities. RaysUp associates each pixel with a 3D ray: given pixel coordinates \(\mathbf{x}\), the ray is uniquely determined by the camera center \(\mathbf{o}\) and the normalized direction \(\mathbf{d}(\mathbf{x})\), concatenated into a 6D descriptor \(\mathbf{r} = [\mathbf{o}, \mathbf{d}(\mathbf{x})] \in \mathbb{R}^6\). The ingenuity of RayPE lies in leveraging RoPE-like rotary modulation to inject geometric coordinates into attention calculations: multi-frequency harmonic encoding is applied to each component of the 6D ray descriptor to generate rotation phases \(\boldsymbol{\theta} \in \mathbb{R}^{6N}\), and the features of attention heads are split into halves, then element-wise rotated using a rotation matrix \(\begin{bmatrix} \cos\boldsymbol{\theta} & -\sin\boldsymbol{\theta} \\ \sin\boldsymbol{\theta} & \cos\boldsymbol{\theta} \end{bmatrix}\). Consequently, the attention is aligned based on the cosine similarity of the ray directions rather than the pixel-plane distance—ray features pointing to the same 3D point or similar directions naturally yield high responses. RayPE introduces no learnable parameters (the phase is analytically calculated from geometric coordinates). Ablations demonstrate that RayPE yields the largest performance boost (improving mean mIoU from 81.05% to 82.17%), making it the most cost-effective design in the paper.

3. Geometry-Aware Neighborhood Cross-Attention: Local Aggregation on the Ray Manifold

After obtaining geometry-aware queries and keys, they must be associated across different resolutions. Global attention requires massive computation (\(O(H_{any}W_{any} \cdot H_{lr}W_{lr})\)) and can cause interactions between geometrically unrelated rays, disrupting geometric consistency. For each high-resolution query position \((i,j)\), RaysUp locates the corresponding mapped position on the low-resolution keys and performs cross-attention within its \(k \times k\) neighborhood \(\mathcal{N}_{i,j}\). To bridge resolution discrepancies, adaptive dilation factors \(d_h = \max(1, \lfloor H_{any}/H_{lr} \rfloor), d_w = \max(1, \lfloor W_{any}/W_{lr} \rfloor)\) are introduced to expand the key sampling stride. The geometric meaning of this design is that the local neighborhood acts as a smooth sampling window on the ray manifold—as camera projection is approximately continuous in local regions, ray directions within the neighborhood are naturally similar, restricting attention only to geometrically similar rays. The computational complexity of \(\mathcal{F}^{hr}_{i,j} = \sum_{(u,v) \in \mathcal{N}_{i,j}} \text{Softmax}\left((\mathcal{Q}_{ray})_{i,j}^\top (\mathcal{K}_{ray})_{u,v} / \sqrt{d}\right) \mathcal{V}_{u,v}\) is reduced to \(O(H_{any}W_{any} \cdot k^2)\). Setting \(k=6\) in experiments enables highly efficient cross-resolution reconstruction from \(16 \times 16\) to any \(2\text{K} \times 2\text{K}\) target.

Loss & Training¶

The training adopts a reconstruction objective: a low-resolution image \(\mathcal{I}_{lr}\) is downsampled from a high-resolution image \(\mathcal{I}_{hr}\) by a random scaling factor \(s \in [2, 4]\). Both are fed into the same frozen VFM encoder to extract target features \(\mathcal{F}_{tgt}\) and source features \(\mathcal{F}^{lr}\), respectively. RaysUp reconstructs features based on \(\mathcal{F}^{lr}\) and \(\mathcal{I}_{hr}\) (guidance) and is supervised using a combination of cosine similarity loss and L2 loss: \(\mathcal{L} = \mathcal{L}_{cos}(\hat{\mathcal{F}}^{hr}, \mathcal{F}_{tgt}) + \mathcal{L}_{L2}(\hat{\mathcal{F}}^{hr}, \mathcal{F}_{tgt})\). Additionally, a crop training strategy is introduced: random local regions are cropped from the high-resolution images, forcing the model to align locally rather than globally, which helps it learn accurate local texture reconstruction capabilities. The model is trained on ImageNet using AdamW for 100k steps with a batch size of 4 and a learning rate of \(2 \times 10^{-4}\). It takes only about 1 hour on a single A100 (approximately 4 hours with crop training), signifying extremely low training costs.

Key Experimental Results¶

Main Results¶

RaysUp is compared systematically with bilinear interpolation, FeatUp, LoftUp, JAFAR, and AnyUp on five dense prediction tasks with a DINOv2-S backbone:

Task	Dataset	Metric	Bilinear	FeatUp	LoftUp	JAFAR	AnyUp	RaysUp
Semantic Segmentation	COCO-Stuff	mIoU	59.58	61.89	62.23	61.79	62.14	62.32
Semantic Segmentation	Pascal-VOC	mIoU	81.70	83.37	84.50	83.89	84.18	84.64
Semantic Segmentation	ADE20K	mIoU	40.47	42.33	42.17	42.16	42.15	42.34
Semantic Segmentation	Cityscapes	mIoU	59.72	60.18	62.09	61.40	60.62	61.88
Surface Normal	NYUv2	RMSE↓	28.23	28.94	28.45	27.80	27.83	27.69
Depth (Abs)	NYUv2	RMSE↓	0.4789	0.4781	0.4828	0.4693	0.4781	0.4658
Depth (Rel)	NYUv2	RMSE↓	0.3348	0.3393	0.3353	0.3255	0.3244	0.3195
Video Object Segmentation	DAVIS	J&F	64.87	68.74	70.92	70.90	70.98	71.47
Open-Vocabulary Segmentation	COCO-Stuff	mIoU	25.73	26.67	27.54	27.13	27.30	27.11

RaysUp consistently outperforms AnyUp in VFM-agnostic testing (DINOv2/v3, SigLIP2, PE Spatial, ViT-S/M/L). For instance, with DINOv2 ViT-L, RaysUp achieves 86.33 mIoU vs. AnyUp's 85.47, and a depth RMSE of 0.376 vs. 0.393. In terms of efficiency, RaysUp has only 0.14M parameters, 10.17 GFLOPs, and 1.26GB memory footprint, achieving 55 FPS at \(224 \times 224\) (where AnyUp only reaches 11 FPS). Even at extreme resolutions of \(2\text{K} \times 2\text{K}\), it can still run (1 FPS) while all other methods trigger out-of-memory (OOM) errors.

Ablation Study¶

Configuration	Avg mIoU (4 VFMs)	Params (M)	Description
Full model (Decoupled + RayPE)	82.17	0.14	Full model
w/o position encoding	81.05	0.14	Drops by 1.12% after removing RayPE
RoPE instead of RayPE	81.92	0.14	2D position encoding is inferior to 3D
SinRays instead of RayPE	81.59	0.47	More parameters but worse performance
Single-Branch guidance encoder	81.87	0.266	Single \(1 \times 1\) convolution
Multi-Branch (incl. \(3\times3\))	82.09	0.268	Four branches including \(3 \times 3\), large parameter size
Dual-Branch (\(1\times1\) + \(3\times3\))	82.03	0.66	Dual branches, largest parameter size

Key Findings¶

RayPE is the most cost-effective design point: it introduces zero parameters but contributes a 1.12% mIoU improvement, consistently effective across all four VFMs and all backbone scales.
The spatial decoupled encoder achieves higher performance (82.17% vs. 82.09%) despite having only half the parameters of the Multi-Branch counterpart (0.14M vs. 0.268M), demonstrating that directional decoupling is far more efficient than naive branching strategies.
RaysUp shows its most prominent advantage in geometrically sensitive tasks (depth/normal estimation) with a depth RMSE of 0.4658, substantially beating AnyUp's 0.4781. This validates the effectiveness of the injected geometric priors. For semantic segmentation, LoftUp performs slightly better on some datasets (e.g., Cityscapes) due to its use of SAM mask auxiliary supervision, but LoftUp cannot be generalized across different VFMs.
Training efficiency is outstanding: it takes only about 1 hour to complete training on a single A100 (compared to ~5 hours for AnyUp and tens of hours for LoftUp), significantly lowering the barrier to adoption.

Highlights & Insights¶

Kernel weight distribution-driven design: Instead of blindly stacking multi-branch layers based on intuition, the authors identified the central low-weight pattern through visualizing the \(3 \times 3\) kernel weights of existing methods, and designed the decoupled directional branches accordingly. This approach of "observing issues from data \(\rightarrow\) designing solutions" is more convincing than "intuitive branching" and is far more parameter-efficient.
Zero-parameter 3D geometric prior injection: RayPE calculates the rotation phase purely based on pixel coordinates and camera geometry, contributing the largest performance boost without introducing learnable parameters. Transferring the ray representation of NeRF into 2D attention via RoPE-style rotation modulation is an elegant cross-domain technology integration—retaining the original structure of attention computation while fundamentally altering the geometric semantics of the interaction.
Balance of ultra-lightweight and broad applicability: Achieving task-agnostic, VFM-agnostic, and arbitrary-resolution capabilities with only 0.14M parameters presents distinct advantages for deployment-heavy applications (e.g., edge-end real-time semantic segmentation, large-scale video understanding pipelines).
The entire framework is trained from scratch in just ~1 hour and is based entirely on public data (ImageNet), making replication and utilization highly accessible.

Limitations & Future Work¶

Currently, RayPE uses an identity matrix as the camera extrinsic pose (Identity pose), assuming a pinhole camera model with the camera fixed at the origin. Introducing more precise pose estimations (e.g., poses provided by DA3 depth estimators) could further enhance performance in geometrically sensitive scenes, but at the cost of additional computation.
Its performance on Cityscapes semantic segmentation is slightly inferior to LoftUp, which utilizes SAM auxiliary supervision. This suggests that the pure self-supervised paradigm still has room for improvement in specific densely annotated scenarios—incorporating weak supervision signals is a natural future direction.
The paper only validates the method on ViT-S/M/L backbones; its performance on ViT-H/G or larger scale backbones (e.g., EVA, InternViT) remains unexplored.
The guidance encoder requires full-resolution RGB images as inputs, which necessitates extra downsampling preprocessing on edge devices with constrained input resolution.

vs. AnyUp: Both support arbitrary VFMs and arbitrary target resolutions, but AnyUp uses local window attention and feature-agnostic convolutional layers to unify VFM dimension discrepancies, resulting in a heavier network (0.87M parameters). RaysUp compresses this to 0.14M parameters using the decoupled guidance encoder and RayPE, running ~7 times faster with superior overall performance.
vs. JAFAR: They share the basic paradigm of guidance \(\rightarrow\) cross-attention, but JAFAR utilizes Spatial Feature Transform to modulate keys, and uses standard \(3 \times 3\) convolutions in its guidance encoder. RaysUp's decoupled encoder and RayPE replace these two components, yielding better geometric consistency while being lighter (0.14M vs. 0.62M).
vs. LoftUp: LoftUp uses SAM masks as auxiliary supervision, which makes it stronger on some semantic segmentation tasks. However, this comes at the cost of relying on SAM and losing VFM-agnostic capabilities. RaysUp achieves comparable performance via pure self-supervision with broader applicability.
Design Migration: The concept of RayPE injecting 3D geometric priors into 2D attention via positional encodings can be naturally extended to tasks such as multi-view reconstruction and cross-frame feature matching in video tracking—essentially any scenario where camera geometric information is accessible.

Rating¶

Novelty: ⭐⭐⭐⭐ Incorporating ray representation into feature upsampling, combined with empirical analysis-driven decoupled encoder design regarding anisotropic kernels, is novel and well-grounded.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 5 downstream tasks \(\times\) 4 VFMs \(\times\) 2–3 backbone scales \(\times\) comprehensive ablation analysis, with efficiency and effectiveness backed by solid experimental data.
Writing Quality: ⭐⭐⭐⭐ Smooth narrative from motivation to method and experiments, with the core concept of "2D \(\rightarrow\) ray domain" consistently maintained. Figures and tables are clear and robust.
Value: ⭐⭐⭐⭐⭐ An ultra-lightweight, universal, and highly efficient upsampler provides high practical deployment value. The zero-parameter geometric prior injection of RayPE offers insightful inspiration for future works.