Foveated Instance Segmentation¶

Conference: CVPR 2025
arXiv: 2503.21854
Code: None
Area: Segmentation
Keywords: Foveated vision, Instance segmentation, Gaze guidance, Saliency sampling, Computational efficiency

TL;DR¶

FSNet proposes an instance segmentation framework simulating the human foveated vision mechanism. By guiding non-uniform downsampling via a learnable saliency map, it maintains high-resolution details in target gaze regions while lowering resolution in peripheral areas. This achieves plug-and-play efficiency gains across various pre-trained segmentation networks.

Background & Motivation¶

Background: Instance segmentation is a core task in computer vision, aiming to identify and segment each object in an image. Existing methods (DeepLab, PSPNet, HRNet, SegFormer, etc.) typically process images at full resolution or through uniform downsampling, allocating equal computational resources to all regions in the image.

Limitations of Prior Work: In many practical application scenarios (such as AR/VR headsets, autonomous driving, robot manipulation), the concentration of users or systems is focused on specific targets, yet traditional segmentation networks uniformly process the entire image, wasting significant computation. Under high-resolution inputs (e.g., 2K images in autonomous driving), uniform processing incurs massive computational overhead. Moreover, when users only care about specific objects, accurately segmenting the gazed object is more valuable than segmenting all objects.

Key Challenge: There is a contradiction between the high computational cost of uniform global processing at high resolutions and the lower demand of only needing to accurately segment the focal target in gaze-centric scenarios. The human visual system naturally addresses this issue via the foveated mechanism, where high-density photoreceptors in the center of the retina provide fine vision, and peripheral receptor density drops sharply.

Goal: Define and solve the "foveated instance segmentation" task—given a gaze position, segment only the gazed object while reducing computational overhead.

Key Insight: Drawing inspiration from the non-uniform sampling mechanism of human foveated vision, a learnable saliency DNN is utilized to generate a saliency map of the gaze region. This map guides non-uniform downsampling via a Gaussian kernel, substantially shrinking the overall input size while retaining high resolution in the gaze region.

Core Idea: Model foveated vision using non-uniform downsampling driven by a learnable saliency map, preserving high resolution in the target gaze area and significantly downsampling the periphery, achieving efficient and accurate segmentation of the target of interest.

Method¶

Overall Architecture¶

FSNet consists of three modules: (1) a Saliency DNN that generates a saliency map based on the input image and gaze position; (2) a Saliency-guided Sampler that performs non-uniform downsampling using a Gaussian kernel based on the saliency map, producing a warped, lower-resolution image (with the gaze region magnified and the periphery compressed); (3) a Segmentation DNN (which can be any pre-trained semantic segmentation network) that performs segmentation on the warped image, which is then mapped back to the original coordinates. The object at the gaze position is labeled as foreground (value of 1), and the rest as background (value of 0), forming a binary classification segmentation task.

Key Designs¶

Saliency DNN:
- Function: Generates a spatial saliency map based on the image and gaze position, indicating which regions need to maintain high resolution.
- Mechanism: Uses a lightweight 3-layer U-Net (with a base channel size of 16) that takes the original image as input and outputs a saliency weight for each pixel. The saliency map is combined with a Gaussian kernel to compute the mapping function for non-uniform sampling. The mapping function employs a weighted average: $$G^h(i,j,F) = \frac{\sum_{i',j'} D_\theta(i',j',F) k_\sigma((i,j),(i',j')) i'}{\sum_{i',j'} D_\theta(i',j',F) k_\sigma((i,j),(i',j'))}$$, where $D_\theta$ represents the saliency weights, and $k_\sigma$ is the Gaussian kernel.
- Design Motivation: The lightweight U-Net adds negligible extra computational overhead but adaptively allocates sampling density based on image content, providing more flexibility than a fixed foveated decay pattern.
Alternating Training:
- Function: Coordinates the optimization directions of the Saliency DNN and the Segmentation DNN.
- Mechanism: Stage 1 freezes the Segmentation DNN and trains the Saliency DNN to learn the optimal sampling strategy (500 iterations, NAdam optimizer); Stage 2 freezes the Saliency DNN and fine-tunes the Segmentation DNN to adapt to the warped image distribution (800 iterations, AdamW optimizer). These two stages alternate until convergence.
- Design Motivation: Jointly training both modules simultaneously would cause gradients from the Saliency DNN and Segmentation DNN to interfere with each other—changes in the saliency map would constantly alter the input distribution of the Segmentation DNN, leading to instability. Alternating training decouples the two optimization objectives.
Gaussian Kernel Sampler:
- Function: Performs saliency-based non-uniform spatial transformation.
- Mechanism: Employs a Gaussian kernel with a fixed standard deviation $\sigma$ (size $2\sigma+1$) to perform warping via saliency-weighted local coordinate mapping. A higher $\sigma$ value offers a larger receptive field but more uniform weights, whereas a lower $\sigma$ value concentrates on local regions. Experiments indicate that a kernel size of 33 ($\sigma=16$) achieves the optimal balance between performance and training efficiency.
- Design Motivation: The Gaussian kernel provides smooth spatial transformation, preventing severe aliasing artifacts after sampling.

Loss & Training¶

The segmentation task is formulated as a binary foreground/background segmentation problem. The object at the gaze position is labeled as foreground (mask value 1), and all other objects as background (mask value 0). A standard pixel-level cross-entropy loss is used. Gaze positions are sampled using gaze trajectories from the OpenEDS2020 dataset, ensuring a balanced distribution across categories.

Key Experimental Results¶

Main Results¶

Five different pre-trained segmentation networks were validated on three datasets:

Segmentation Network	Parameters	CityScapes IoU	ADE20K IoU	LVIS IoU
DeepLab (ResNet50)	42M	0.52	-	-
PSPNet (ResNet50)	24.3M	0.49	-	-
HRNet (W48)	67.12M	0.47	-	-
SegFormer-B4	64M	0.46	-	-
SegFormer-B5	84.6M	0.51	-	-

Ablation Study¶

Gaussian Kernel Size	DeepLab IoU	PSPNet IoU	FLOPs
17	0.48	0.45	2.38M
25	0.50	0.49	5.12M
33	0.52	0.49	8.92M
41	0.52	0.48	13.77M

Loss Component	SSIM	FVD
Base Loss Only	-	-
+ Adversarial Loss	Gain	-
+ Saliency Guidance	Further Gain	Optimal

Key Findings¶

As a plug-and-play framework, FSNet is effective across all five pre-trained segmentation networks, proving the generalizability of the method.
A Gaussian kernel size of 33 achieves the optimal IoU on most models; increasing the kernel size from 17 to 41 results in a 5.79x increase in FLOPs but yields limited IoU improvement.
The extra parameter count of the Saliency DNN is minimal (3-layer U-Net, base channel 16), making the computational overhead negligible.
Visualization results demonstrate that the warped images preserve clear details in the gaze region, while the peripheral regions, although compressed, retain sufficient context.

Highlights & Insights¶

Introducing the foveated vision mechanism to instance segmentation is a well-defined problem—in gaze-tracking scenarios such as AR/VR, users only care about the objects they look at, eliminating the need to segment the entire scene. This task definition has direct value on the application side.
The plug-and-play design idea is highly practical—FSNet does not modify the internal structure of any segmentation network but only applies space-variant transformation at the input level; any pre-trained model can be used directly.
The alternating training strategy is a classic approach to decoupling two interdependent modules, which is simple to execute yet produces robust results.

Limitations & Future Work¶

Foveated downsampling inevitably loses information in peripheral areas. If there is a strong dependency between the gazed target and the background (such as in occlusion scenarios), segmentation quality may degrade.
The paper only deals with binary segmentation (single gazed object vs. background) and does not explore multi-class scenarios that simultaneously segment the gazed object and its semantic labels.
The gaze position needs to be provided externally (e.g., by eye-tracking devices); the paper does not discuss robustness when gaze estimation is inaccurate.
The absolute IoU values in experiments are not high (best is 0.52 on CityScapes), and there is a lack of direct comparison with full-resolution segmentation performance to quantify the accuracy-efficiency trade-off.
End-to-end validation was not conducted on actual AR/VR devices or autonomous driving scenarios.

vs. Traditional Segmentation Methods: Traditional methods like DeepLab/PSPNet uniformly process the entire image, while FSNet concentrates computations on important regions via non-uniform sampling, representing an efficiency-oriented design.
vs. Foveated Rendering: Foveated rendering is already widely used in computer graphics (e.g., VR rendering); FSNet introduces a similar concept to the segmentation task, which can be viewed as a migration from "rendering" to "understanding".
vs. Attention Mechanisms: Standard attention mechanisms perform weighting at the feature layer, whereas FSNet performs spatial transformation at the input layer. The two can be used complementarily.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Introducing foveated vision into the task definition of instance segmentation is novel, though the core technique (saliency-guided sampling) has many precursors.
Experimental Thoroughness: ⭐⭐⭐⭐ Validation across multiple segmentation networks is sufficient, but analysis of the accuracy-efficiency trade-off compared to full-resolution, as well as real-world scenario validation, is lacking.
Writing Quality: ⭐⭐⭐⭐ The methodology description is clear, but the paper is overall relatively short, with an overly heavy reliance on supplementary materials.
Value: ⭐⭐⭐⭐ The problem definition is interesting, but under its current form, the application scenarios are limited and require further engineering and scenario validation.