WebEyeTrack: Scalable Eye-Tracking for the Browser via On-Device Few-Shot Personalization

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/RedForestAI/WebEyeTrack
Area: Human Understanding
Keywords: gaze estimation, metric head pose, few-shot learning, on-device personalization, browser deployment

TL;DR

WebEyeTrack combines monocular metric head pose, the lightweight BlazeGaze eye encoder, and on-device few-shot personalization for browser eye-tracking with an initial 9-point calibration, reaching 2.32 cm gaze error on GazeCapture while prioritizing deployment efficiency and drift resistance over leading accuracy on every dataset.

Background & Motivation

Eye-tracking estimates where a user looks on a screen from eye appearance, making it both a visual estimation problem and a human-computer interaction systems problem. Dedicated Tobii or EyeLink devices provide stable references, but constrain experiments to environments with suitable hardware and controlled viewing distances and poses. Ordinary webcams reduce access costs, yet the same eye appearance need not correspond to the same screen location after a user changes devices, moves their head, or changes posture. Existing appearance-based models often optimize offline error without jointly addressing model download size, low-power execution, browser compatibility, and personal calibration effort. Browser tools such as WebGazer use lightweight regression for accessibility, but lack explicit head-pose modeling, allowing posture changes during extended typing and reading to cause drift.

The challenge is not simply to compress a large network into a browser; the input representation must fit the browser's observation conditions. Monocular face meshes usually provide relative depth: they describe head rotation without reliably establishing the physical distance from the camera. Ignoring this scale ambiguity forces a regressor to infer eye motion, head motion, and camera distance simultaneously from limited eye imagery. Conversely, retraining an entire network for every user spends the convenience of webcam tracking on lengthy calibration and device computation. Cloud processing is not a cost-free alternative either, because raw faces, eye features, and gaze trajectories all carry sensitive information.

The paper first uses geometric priors to recover approximately centimeter-scaled head pose, then assigns eye appearance to a small network and restricts user adaptation to a tiny regression head. Here, scalability primarily means access across browsers and consumer devices with local execution, rather than increased server inference throughput. Few-shot training supplies an initialization that adapts readily; new interaction samples then refine the personal mapping instead of relearning general visual features. Core Idea: use explicit metric head pose to explain head motion, preserve eye information in a frozen lightweight representation, and update the gaze head with limited local supervision to combine geometric stability with individual adaptation.

Method

Overall Architecture

The system takes camera images and outputs a two-dimensional screen Point-of-Gaze (PoG), not a three-dimensional gaze direction. MediaPipe supplies facial landmarks; generic preprocessing extracts the eye region and uses the eye aspect ratio, EAR, to detect open or closed eyes and suppress blink artifacts. Metric Head Pose Recovery estimates rotation and centimeter-scaled translation, Lightweight Gaze Representation encodes eye images, and Few-Shot Personalization combines both inputs to predict gaze. Offline training first learns the encoder, reconstruction decoder, and regression head, then removes the decoder, freezes the encoder, and meta-trains the head. The deployed browser retains only the necessary encoding and regression components; initial calibration and subsequent click updates take place on-device.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Camera frames and landmarks"] --> Pose["Metric Head Pose Recovery"]
    Input --> Eye["Eye crop and blink filtering"]
    Eye --> Encoder["Lightweight Gaze Representation"]
    Pose -->|Rotation and metric translation| Adapt["Few-Shot Personalization"]
    Encoder -->|Frozen encoder features| Adapt
    Support["Initial calibration points<br/>Additional clicks in video"] -.->|Update head locally| Adapt
    Adapt --> Output["Two-dimensional screen gaze"]

The dashed edge denotes personalization supervision, not a label required for every prediction frame. Metric head pose and eye encoding are separate input branches; the pose is not fed through the image encoder first. The reconstruction decoder is used only for offline representation learning and is therefore not shown as a browser inference branch. Screen targets are normalized to \([-0.5,0.5]^2\) with the origin at screen center, reducing output-representation differences caused by screen size and resolution. This normalization does not eliminate cross-device domain differences; centimeter errors must still be interpreted using the physical display scale.

Key Designs

1. Metric Head Pose Recovery: turn a relative face mesh into physically scaled conditioning

MediaPipe's face mesh has a fixed 468-point topology, normalized image-plane coordinates, and depth from a weak-perspective representation. Its original face transformation contains rotation and translation, but the translation scale cannot be read directly as a true distance in centimeters. The paper reprojects landmarks into three dimensions, subtracts the nose position, and normalizes the distance between the face boundaries to a unit face width. It uses landmark ID 4 for the nose and IDs 356 and 127 for face width, separating shape, center, and absolute scale. For datasets with camera intrinsics, reprojection uses focal lengths and principal-point coordinates. Handling browser intrinsics requires separate attention, because the main text does not fully explain recovery when they are unknown.

Scale recovery uses an anatomical prior: approximate iris diameter as 1.2 cm and estimate physical face width from the image-space ratio of face width to iris diameter. The following is a summary of the scale relationship described in Section 3.1, not a character-by-character reconstruction of the damaged extraction of Equation (5):

\[ s_{\mathrm{face}} \approx \alpha\frac{d_{\mathrm{face,px}}}{d_{\mathrm{iris,px}}},\qquad \alpha=1.2\,\mathrm{cm}. \]

The ratio of the two pixel lengths is dimensionless; multiplying by iris diameter yields a centimeter scale used to resize the unit-width mesh. This avoids asking users to measure their face with a ruler, but a fixed iris diameter provides only an approximation, not an exact anatomical fact for every person. The authors argue that subsequent personalization can mitigate residual scale error and mention a supplementary sensitivity experiment; the supplied full text does not contain its numerical results.

Once scale is recovered, the system retains MediaPipe's rotation and mainly refines translation. It initializes depth at 60 cm and derives initial horizontal and vertical position from the image location of the nose. It compares radial deviations between the projected three-dimensional face and observed landmarks, using radial Procrustes alignment to determine the direction and magnitude of a depth update. Intuitively, an overly large or small projected face indicates whether distance needs adjustment, rather than requiring a new solution for the entire rigid rotation. After each depth update, horizontal and vertical translation are adjusted through similar triangles so that the projected nose remains aligned with its observation. The update coefficient is 0.1, the maximum depth update is 5 cm, the stopping threshold is 0.25 cm, and the iteration cap is 10. These bounds support real-time execution and stable updates; they do not prove global optimality or a bound on metric error. Because Equations (6) through (8) lose symbols in the text extraction, this note retains only the procedure supported by the prose and does not invent an exact optimization formula.

2. Lightweight Gaze Representation: learn eye appearance before narrowing online adaptation

The system constructs a homography from facial landmarks to warp the eye region into an approximately upright, centered appearance. This avoids an iterative PnP solver in that preprocessing step, but should not be confused with resolving the intrinsic-camera issue in the metric geometry branch. The encoder takes a \(128\times512\times3\) eye image, uses single and double BlazeBlocks, and produces a 512-dimensional embedding. BlazeBlocks originate from efficient mobile vision architectures; the paper organizes them into a lightweight gaze model rather than deploying a large full-face backbone. During training, a mirrored decoder with transposed convolutions reconstructs the eye image, requiring the embedding to preserve visible eye structure. Gaze supervision simultaneously requires that structure to explain screen targets rather than merely reproduce image texture.

Representation learning combines image reconstruction, weighted gaze regression, and embedding consistency objectives. The reconstruction term is pixel-wise mean squared error, encouraging compressed features to retain visible eye structure. The gaze term uses weighted L2 error, with weights derived from inverse frequencies in a precomputed \(30\times30\) screen grid for each dataset. This reduces domination by frequently observed gaze regions, preventing a concentration of central samples from encouraging indiscriminate regression toward the center. Embedding consistency compares pairwise feature distances with normalized gaze distances, making representations close when their gaze targets are close. It is not simply contrastive learning over same-user and different-user identities; continuous two-dimensional gaze labels shape the embedding geometry. The described gaze-distance normalization involves the maximum within-batch distance and a numerical stability term; damaged weighting and norm details are not reconstructed here. The three objective coefficients are selected empirically per dataset, and the main text provides no complete coefficient table for direct reproduction.

The gaze regressor combines the 512-dimensional features with head pose and outputs two coordinates through a three-layer MLP with widths 16, 16, and 2. The decoder is removed after training, so reconstruction improves the representation without imposing image-reconstruction costs on browser prediction. Freezing the encoder prevents online calibration from updating many visual parameters, directing scarce labels toward the personal feature-to-screen mapping. The lightweight BlazeGaze network must be distinguished from the complete WebEyeTrack system, which also includes landmark detection, geometric processing, and browser scheduling.

3. Few-Shot Personalization: learn an adaptable head instead of retraining a network per user

The paper treats gaze prediction for one user as a task, with separate support and query sets within that task. The support set supplies a small number of calibration samples, while the query set tests whether the updated model predicts other samples from that user. Each support sample includes a frozen eye embedding, head pose, two-dimensional target, and sample weight. The meta-training inner loop first updates the regression head on support data, then evaluates the adapted model on query data. The outer loop improves the shared initialization so that a few gradient steps suit a new user, rather than only fitting an average user before adaptation. The implementation uses first-order MAML and does not require the browser to perform full higher-order meta-gradient training. Meta-training and meta-testing users are disjoint in the method definition; a new user's browser only performs lightweight support-set adaptation.

The standard configuration uses 9 support samples and 100 query samples, with 5 inner-loop updates per task. Query samples belong to offline training or evaluation protocols; live browser prediction does not require obtaining 100 future gaze labels first. After initial calibration, inference reuses the personalized head instead of restarting training from the shared initialization on every frame. The paper also treats click coordinates as pseudo-labels, pairs them with simultaneous eye and head features, and appends them to the support set for further updates. This online click calibration is used only in the video evaluation and must not be described as extra supervision used in every static-image benchmark. Additional clicks enlarge the support set over a session, so 9 points describe the low-burden initial configuration, not the total labels used throughout a long session. Approximating gaze with a click coordinate is an interaction assumption: users need not keep looking at the click location, so these labels may be noisy. Instead of fitting an independent online regressor, the added samples update the same gaze head obtained from the MAML initialization. Keeping raw frames, embeddings, and gaze predictions on-device is the system design described by the paper, not a formal privacy guarantee or an independent security-audit finding.

A Worked Example

Consider a new user who completes an initial \(3\times3\) dot-grid calibration in a browser and then starts reading and typing. Each calibration observation produces an eye image and landmarks; the geometry branch estimates orientation and distance, while the encoder compresses the image into fixed features. The 9 support samples adapt the pretrained regression head to this person's eye appearance and screen relationship without training a new encoder. If the user subsequently leans backward, metric translation changes even with limited eye motion, allowing the regressor to consider both eye appearance and head position. Without clicks during reading, the system can still predict with the current personalized head, but receives no additional interaction supervision to correct residual bias. Only a click adds a corresponding pseudo-label and triggers further adaptation; geometric conditioning and online learning thus play distinct roles. This example explains system state changes and is not a reported trajectory or error measurement for a particular participant.

Loss & Training

Representation learning runs for 20 epochs with batch size 8 and Adam, an initial learning rate of \(10^{-3}\), and exponential decay factor 0.95. The best model is retained, the decoder removed, and the encoder frozen before 1000 meta-training steps. The MAML inner loop uses SGD with learning rate \(10^{-5}\); the outer loop uses Adam with learning rate \(10^{-3}\). Support and query losses use weighted gaze error, whereas reconstruction and embedding consistency are restricted to offline representation learning. Training uses TensorFlow 2 on an RTX 3080, followed by conversion through tensorflowjs_converter. The browser implementation uses TensorFlow.js LayerModels for on-device prediction, calibration, and model caching. The supplied main text does not fully specify every dataset split, all objective coefficients, or deployment with unknown intrinsics, so these hyperparameters alone do not establish complete reproducibility.

Key Experimental Results

Main Results

The following selection reproduces accuracy and efficiency measurements from Table 1 on page 11; the error columns are MPIIFaceGaze, GazeCapture, and EyeDiap, all in cm with lower values preferred. Model timing uses an Intel Core i7-11700F @ 2.50GHz, matched backends and warm-up settings, and 1000 frames, averaging frames 100 through 900.

Method	MPIIFaceGaze error	GazeCapture error	EyeDiap error	GFLOPs	Parameters (M)	Delay (ms)	FPS
Mnist	7.29	NA	9.06	0.10	1.82	3.13	319.0
iTracker	7.67	2.81	10.13	3.97	6.28	33.33	30.0
Gaze360	4.66	NA	6.37	3.65	11.94	24.39	41.0
AFF-Net	4.21	2.30	9.25	26.14	1.94	109.81	9.1
BlazeGaze	4.56	2.32	7.53	0.15	0.16	0.88	1137.0

BlazeGaze approaches AFF-Net's GazeCapture error with substantially lower model computation and runtime cost; this is not an across-the-board accuracy lead. The 0.88 ms and 1137.0 FPS measurements refer to model timing, not an end-to-end browser frame rate including the camera, MediaPipe, geometry estimation, and rendering. The abstract separately reports 2.4 ms inference on an iPhone 14, a different platform from the CPU table; the supplied main text lacks the complete corresponding mobile protocol.

Ablation Study

Figure 6(a) on page 14 presents the following tabular ablation; BG denotes BlazeGaze, R denotes rotation, and Metric RT denotes rotation plus metric translation, with PoG error in cm. The values are preserved exactly: notably, the full model's MPIIFaceGaze result remains 2.75 rather than being replaced with 4.56 from the main table; the text does not sufficiently explain the protocol difference.

Config	MPIIFaceGaze error	GazeCapture error
BG	9.06	4.22
BG + R	9.06	4.42
BG + Metric RT	3.78	4.16
MAML BG	8.99	4.01
MAML BG + R	9.00	3.84
MAML BG + Metric RT	2.75	2.32

Within this ablation, rotation alone does not improve MPIIFaceGaze, whereas adding metric translation reduces error from 9.06 to 3.78, indicating that rotation cannot substitute for distance and position. MAML alone yields a smaller improvement, but combining it with full metric head pose performs best; the contributions interact and should not be treated as simply additive. Figure 6(b) sweeps \(k\in\{0,1,2,\ldots,32\}\), and the authors describe relatively stable static-benchmark performance, but the supplied text lacks individual curve values, so no pointwise gains are invented.

Key Findings

The cross-dataset experiment transfers an MPIIFaceGaze-trained model to Eye of the Typer, calibrates on the initial 9-point Dot Test, and evaluates a final Dot Test after an approximately 20-minute session. The following values come from the discussion of Figure 5 in Section 5.3 on page 14; the metric is L1 PoG error in cm and should not be pooled indiscriminately with static-table results.

Method	Initial Dot Test	Final Dot Test	Error increase reported by the paper
WebGazer	7.79	11.62	49%
WebEyeTrack	7.24	8.72	20%

The authors report a Mann–Whitney U test with \(p<0.05\) for final error and analyze session error using 10 s windows. This supports greater drift resistance relative to WebGazer, but the absolute 8.72 cm error also indicates that the system is not a direct replacement for high-precision dedicated hardware. The video experiment permits additional clickstream calibration, and the methods start with different errors, so the final difference cannot be attributed entirely to one module.

Highlights & Insights

• Geometry assumes part of the explanatory burden for a small network. Explicit distance and position are more targeted than asking limited calibration data to rediscover every pose effect from eye pixels.
• The reconstruction decoder is a training-only auxiliary component. It constrains eye features without charging the browser for reconstruction on every frame.
• Few-shot learning matters beyond a fixed benchmark score. Reusing one head for initial adaptation and continued click updates lets user supervision accumulate gradually.
• Matched hardware protocols make model efficiency comparisons informative. Full-system responsiveness still requires separate measurement rather than reliance on network FPS.

Limitations & Future Work

• The fixed 1.2 cm iris diameter is a scale prior; occlusion, iris detection error, and individual variation can affect metric recovery, and the main text provides no verifiable sensitivity values.
• Section 3.1 uses camera intrinsics while Section 4.1 emphasizes their unavailability in browsers; eye-region homography correction does not fully explain how the entire metric branch resolves this issue.
• The 4.56 cm MPIIFaceGaze result in the main table cannot yet be reconciled with 2.75 cm in Figure 6(a) from the text; protocols and implementation need checking before exact reproduction.
• Local execution reduces data transmission but is not differential privacy or evidence of a browser security audit; click pseudo-labels can also contain interaction noise.
• The authors list fairness and energy use as future work; readers should additionally seek stratified evaluation across populations, glasses, lighting, and long-term model updates.
• Some equations are damaged in extraction, and supplementary material is absent from the supplied source; this note neither reconstructs missing formulas nor treats unread mobile or sensitivity experiments as verified results.

Related Work & Insights

• vs WebGazer: Both use interactions for browser calibration, but this paper adds metric head pose and a MAML regression head; the difference is not merely replacing a regressor with a CNN.
• vs FAZE: Both address few-shot personalization; this paper emphasizes a frozen lightweight encoder, a narrow adaptation parameter set, and TensorFlow.js deployment rather than moving a large training pipeline on-device.
• vs BlazeFace: The paper borrows efficient BlazeBlocks, while gaze representations, pose conditioning, and personalization training are additional designs specific to eye-tracking.
• vs three-dimensional gaze methods: The target here is two-dimensional screen gaze, so angular-error rankings cannot directly determine superiority; applications must first distinguish screen location from gaze direction.
• Possible extension: Estimate scale confidence without ruler-based calibration and use it to adjust geometric conditioning or click-update weights; this is a reader-proposed direction, not a validated mechanism in the paper.

Rating

• Novelty: 4/5. Combining metric head pose, lightweight representations, and continuing personalization addresses concrete deployment constraints, although meta-learning and efficient convolution are not themselves new.
• Experimental Thoroughness: 3/5. Static benchmarks, ablations, and temporal cross-dataset tests are included, but protocol differences, end-to-end performance, and population-level evidence remain insufficiently resolved.
• Writing Quality: 3/5. System goals and training stages are clear, but table-versus-ablation discrepancies and incomplete treatment of unknown intrinsics hinder reproduction; damaged equations are a separate limitation of the current text source.
• Value: 4/5. The work offers a concrete route to accessible browser eye-tracking, although absolute error limits fine-grained gaze localization.