Constrained Rotation Optimization: Revisiting Crop-Based Gaze Estimation¶

Conference: ECCV2026
Paper: Official paper page
PDF: Full paper
Authors: Riccardo Santambrogio, Jiawei Qin, Matteo Matteucci, Yusuke Sugano
Area: Human Understanding
Keywords: Gaze estimation, constrained rotation, face cropping, head pose, multi-task learning

TL;DR¶

CROp models the gaze-coordinate change associated with face cropping as a constrained approximate rotation and supplements it with joint head-pose supervision, reducing within-dataset angular error on ETH-XGaze from 6.45° to 5.55° with detector-based preprocessing, but it does not outperform normalization under every accurate-annotation or severe bounding-box-noise condition.

Background & Motivation¶

Appearance-based gaze estimation predicts a three-dimensional viewing direction from a face image, yet the same gaze can look different across cameras, head locations, and poses. Standard pipelines first locate facial landmarks and estimate head pose, then normalize the image to a virtual camera view. The network predicts from a more standardized appearance distribution, and the direction is rotated back into the original camera coordinates. This process transforms both the image and label coordinates rather than merely removing background, so landmark and pose errors directly affect the subsequent input.

Using face crops directly avoids landmark detection and image reprojection, but leaves an easily overlooked issue: the network sees a face referenced to the crop center, and its predicted angles cannot necessarily be interpreted directly in the original camera coordinates. Earlier methods such as Gaze360 use local reference frames, but controlled comparisons of the geometric differences and appropriate use cases of cropping versus normalization have been limited. Rather than proposing a larger network to remove every error, this paper first clarifies the reference frame of crop-based predictions and then tests whether reducing preprocessing is worthwhile.

There is another difference: normalization already requires a head-pose estimate, whereas cropping lacks this explicit input. Removing the entire procedure may also discard head–eye coupling information useful for gaze estimation. Core idea: construct an explicitly constrained crop-coordinate rotation from the bounding box and camera intrinsics, then jointly predict gaze and head pose with one network to retain interpretable camera-coordinate outputs at a lower preprocessing cost.

Method¶

Overall Architecture¶

Inputs are the original camera image, a face detection box, and camera intrinsics. Outputs are the gaze direction and head rotation in the original camera coordinates. A geometric branch computes the crop reference frame through “Constrained Rotation Mapping,” while the appearance branch directly feeds a face crop into “Joint Gaze and Head-Pose Prediction,” without first normalizing the face or resampling a rotated view. The same rotation converts training labels and predictions between crop and camera coordinates.

The crucial distinction is that the rotation defines how directions are converted; it does not generate an image captured by a physically rotated camera. The authors explicitly acknowledge that two-dimensional cropping is generally not a three-dimensional rigid rotation. CROp therefore finds a constrained best approximation rather than establishing exact equivalence between the two image views. The network still has to learn head-pose and gaze variations from unrectified face appearance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image, box, and intrinsics"] --> B["Constrained Rotation Mapping"]
    A --> C["Face crop"]
    C --> D["Joint Gaze and Head-Pose Prediction"]
    B -->|Convert training-label coordinates| D
    B --> E["Inverse rotation to camera coordinates"]
    D --> E
    E --> F["Gaze direction and head rotation"]

Key Designs¶

1. Constrained Rotation Mapping: align the crop center first, then fit the corners

Given a face box, the authors define two intrinsic matrices in the original image pixel coordinates. One retains the original camera principal point, while the other places it at the box center and keeps the original focal lengths. This crop reference frame captures how the face's image location affects the interpretation of directions, instead of treating a cut-out image as an independent input with no camera geometry. Geometric quantities are computed at the original resolution before the face crop is resized for the network, avoiding a mixture of resized pixels and original intrinsics.

The box center and its four corners are then back-projected into unit rays using each intrinsic matrix. The center rays specify the desired virtual optical axis, while the corner rays constrain directions around the field of view. Since a single rotation generally cannot align both ray sets exactly, the algorithm enforces center alignment and minimizes the remaining corner residuals. The following objective restates the prose description; the cached equation is corrupted, so the PDF remains authoritative for coefficients and original notation:

\[ R^* = \underset{R\in SO(3)}{\arg\min}\;\sum_{i=1}^{4}\left\|R\hat p_{\mathrm{cam},i}-\hat p_{\mathrm{crop},i}\right\|_2^2, \qquad R\hat c_{\mathrm{cam}}=\hat c_{\mathrm{crop}}. \]

Hatted vectors are unit rays, with \(\hat c\) denoting the center and \(\hat p_i\) the corners; \(SO(3)\) requires a valid three-dimensional rotation. The solver first finds a rotation aligning the centers, then optimizes the remaining rotation around the target center ray. The paper states that Kabsch alignment can solve this step. This is more general than simply requiring the face to be upright, while prioritizing the requirement to face the box center rather than allowing corner fitting to displace the network's reference axis.

During training, \(g_{\mathrm{crop}}=R^*g_{\mathrm{cam}}\) converts camera-coordinate gaze labels into crop coordinates. At test time, the network predicts pitch and yaw in crop coordinates; these are converted to a direction and mapped back with \(g_{\mathrm{cam}}=(R^*)^{-1}g_{\mathrm{crop}}\). Label conversion and output recovery must be paired. Cropping the face while interpreting predicted angles directly in the original camera frame would reintroduce the coordinate problem that the paper makes explicit.

2. Joint Gaze and Head-Pose Prediction: replace removed pose preprocessing with a learned auxiliary task

Cropping retains changes in head orientation instead of explicitly adjusting them through normalization. The authors therefore make the same face network predict both gaze and three-degree-of-freedom head rotation. Shared appearance features must explain the relationship between eye and head directions rather than regress an angle from local eye texture alone. This is not a separate head-pose detector whose output is passed into a gaze network, nor does it restore a landmark-detection chain: head pose is another output of the joint network.

Head rotation uses the continuous nine-dimensional SVDO+ representation, allowing network outputs to be converted into valid rotations through differentiable SVD orthogonalization. This avoids the discontinuities and gimbal-lock issues of directly regressing Euler angles. Head pose is also learned in crop coordinates and converted into camera coordinates by left-multiplying the inverse rotation: \(\hat H_{\mathrm{cam}}=(R^*)^{-1}\hat H_{\mathrm{crop}}\). Gaze and head pose thus share the same geometric convention rather than defining separate reference axes.

The benefit of joint supervision is an empirical consequence of related tasks, not a geometric guarantee. It recovers pose information missing from the crop pipeline and provides a pose output for downstream use. However, datasets differ in head-pose distributions, and the auxiliary task can change the trade-offs in shared features. The ablations contain both clear improvements and combinations with higher gaze error, so multi-task learning should not be described as unconditionally beneficial.

A Worked Example¶

Suppose a face appears on the right side of the original image. The detection box first determines which pixels enter the face crop, which goes directly to the network. Meanwhile, Constrained Rotation Mapping constructs rays through the box center and corners in the original image coordinates and obtains the camera-to-crop rotation. This step neither straightens the face nor requires prior localization of eye corners or the nose tip.

During training, gaze and head-pose labels in camera coordinates are transformed into the crop reference frame, where the network learns both tasks. At test time, even a gaze prediction close to the crop's central axis does not imply a direction along the original camera optical axis: inverse rotation is still necessary. If the detection box shifts, both the face appearance and the geometric reference axis change. Bounding-box noise therefore affects both appearance and coordinates, explaining why severe box perturbations can still harm CROp.

Loss & Training¶

Gaze training uses the angular difference between predicted and true directions, while head-pose training uses geodesic distance between predicted and true rotations. The joint objective combines these supervision terms. The first measures gaze-direction error; the second measures the angle of the relative three-dimensional rotation rather than simply adding errors in three Euler angles. Symbols are missing from the cached total-loss equation, so the exact weighting is not claimed as verified here.

The main backbone is ResNet18, with a standard input of \(256\times256\). Models train for 25 epochs using Adam at a learning rate of \(1\times10^{-4}\), multiplied by 0.1 every 10 epochs. Within-dataset MPIIFaceGaze evaluation instead uses \(448\times448\) inputs and 15-fold cross-validation. Within-dataset ETH-XGaze uses the first 60 subjects for training and the remaining 20 for testing. Within-dataset training includes scaling and translation augmentation up to 20% of the box size; this should not be confused with the additional severe-noise evaluation.

Key Experimental Results¶

Main Results¶

All metrics below are mean gaze angular errors in the original camera coordinates, measured in degrees; lower is better. GT uses dataset-provided landmarks or head-pose annotations for preprocessing. Det uses a common InsightFace pipeline: Norm additionally requires landmarks and PnP head-pose estimation, whereas CROp uses only the detected box. “Ours” denotes CROp with joint head-pose supervision, not rotation mapping alone. Norm is a controlled baseline, not a universal historical state of the art.

Evaluation setting	Preprocessing	Norm error ↓	Ours error ↓	Error reduction (degrees)
Cross-dataset MPII → XG	GT	31.23	29.37	1.86
Cross-dataset XG → MPII	GT	7.19	7.41	-0.22
Cross-dataset EVE → XG	Det	39.13	36.01	3.12
Cross-dataset EVE → MPII	Det	9.53	8.51	1.02
Within-dataset MPII	GT	4.93	5.02	-0.09
Within-dataset MPII	Det	5.32	4.98	0.34
Within-dataset XG	Det	6.45	5.55	0.90
Within-dataset EVE Test	GT	4.99	4.51	0.48

MPII denotes MPIIFaceGaze, and XG denotes ETH-XGaze. The final column is the baseline minus the proposed method in absolute degrees, not percentages or percentage points; negative values indicate worse performance. Cross-domain XG errors remain large despite improvements, so relative superiority alone does not establish deployment-grade accuracy.

On an RTX A6000, reported end-to-end latency decreases from 20.74 ms to 15.97 ms, approximately a 23% reduction. This mainly comes from removing 2.79 ms of landmark detection and replacing 2.67 ms of normalization with 0.69 ms of CROp. Face detection takes 10.88 ms and the model 4.40 ms in both pipelines. “Faster” here refers to pipeline latency in this hardware and software configuration, not an identical speedup on every platform.

Ablation Study¶

The following Det results are selected from clearly aligned entries in the paper's Table 4. Head-pose supervision is indicated as present or absent. Every value is gaze angular error, with lower being better.

Train → Test	Input and coordinate processing	Head-pose supervision	Error ↓
MPII → XG	Norm	Absent	30.55
MPII → XG	Norm	Present	29.56
MPII → XG	CROp	Absent	28.29
MPII → XG	CROp	Present	29.56
EVE → ED	Norm	Absent	20.47
EVE → ED	Norm	Present	20.76
EVE → ED	CROp	Absent	21.13
EVE → ED	CROp	Present	18.47

ED denotes EYEDIAP. Adding head-pose supervision to CROp improves EVE → ED by 2.66°, but increases MPII → XG error by 1.27°. CROp alone also does not always beat Norm: its 21.13° on EVE → ED exceeds 20.47°. The paper's broad statement of an advantage in “all scenarios” should therefore be read against the individual table entries.

Key Findings¶

Bounding-box robustness has a clear limit. On EVE → MPII, 20% box perturbation gives 8.92° for the proposed method versus 9.53° for Norm; at 30%, both reach 9.54°; at 50%, the proposed method reaches 10.83° versus 9.63° for Norm. Landmark refinement helps normalization correct an inaccurate initial box, so CROp is not more robust to arbitrary levels of box noise.
Occlusion behaves differently from box noise. Under the same transfer setting, random black-box occlusion covering 50% of the box area changes the proposed method's error from 8.51° to 9.90°, versus 9.53° to 11.17° for Norm. This supports reducing landmark dependence, but only within the tested synthetic-occlusion protocol.
Larger backbones do not remove every boundary. Additional ResNet50 and GazeTR-Hybrid experiments generally support cropping when trained on MPII or EVE, but some XG-trained combinations still favor Norm. The authors relate this to learning trade-offs introduced by the head-pose auxiliary task.

Highlights & Insights¶

Treat preprocessing as part of label geometry. CROp's main value is not merely removing normalization, but clarifying how predicted directions should be interpreted afterward. Cropping, network outputs, and original-camera evaluation consequently share a consistent conversion chain.
Replace an expensive prerequisite with an auxiliary output from shared features. Head pose no longer has to be solved before the gaze network can run; it is learned alongside gaze. Showing both gains and negative transfer makes the ablations more informative than treating auxiliary supervision as a free improvement.
Provide selection criteria rather than claim replacement. Detection noise, occlusion, large box shifts, and accurate annotations lead to different outcomes. Engineering choices should reflect input quality and latency budgets rather than a single average error.

Limitations & Future Work¶

Rotation only approximates cropping. A single rotation cannot remove all differences as off-axis angles and perspective distortion increase. The paper also approximates the gaze origin by the box center rather than the anatomical origin; more accurate origin modeling remains unresolved.
This is not calibration-free or supervision-free. Camera intrinsics, face boxes, and training head-pose labels are still required. The main datasets provide these signals, so equivalent effectiveness with unknown intrinsics or missing pose labels is not established.
Multi-task benefits are conditional. Head-pose supervision reduces gaze accuracy in some transfer combinations, and Norm can be preferable with accurate landmarks or severe box perturbations. Deployment decisions should retain controlled comparisons on the relevant target domain.
The cache limits exact verification. Equation extraction and parts of side-by-side tables are corrupted. This note uses only unambiguous experimental entries and labels the constraint objective as a prose-based restatement; exact equation formatting and unresolved loss weights should be checked in the PDF.

vs standard normalization: Normalization uses landmarks, head pose, and image transformation to reduce appearance variation. CROp keeps the direct crop appearance and only makes the directional reference frame explicit. The former retains advantages with high-quality geometry, while the latter avoids landmark costs and error propagation.
vs Gaze360: Both use a face-centered local gaze reference frame, but this paper emphasizes explicitly solving the crop-to-camera rotation without restricting it to an upright, zero-roll case. Its contribution centers on geometric formalization and systematic comparison rather than being the first to estimate gaze from face crops.
vs EFE and IntrApose: EFE directly regresses camera-coordinate gaze from a full frame, reducing preprocessing but introducing more appearance variation. IntrApose emphasizes intrinsics and coordinate conversion for head pose. CROp brings similar geometric considerations to gaze estimation and makes head pose an auxiliary output of the same network.

Rating¶

Novelty: 4/5 — Clear constrained geometric formalization of an existing crop-based paradigm, with relatively conventional network components.
Experimental Thoroughness: 4/5 — Covers cross-domain and within-domain evaluation, noise, occlusion, speed, and backbones, but joint supervision is not uniformly beneficial.
Writing Quality: 4/5 — The problem and selection criteria are clear, although some broad claims exceed the support of individual table entries.
Value: 4/5 — Provides actionable evidence for preprocessing choices in practical camera-based gaze pipelines.