Event-based Gaze Control Systems for Real-time Spin Estimation in Professional Ball Games¶

Conference: ECCV 2026
Paper: ECCV 2026
Area: Robotics / Embodied AI
Keywords: event-based camera / active gaze control / ball spin estimation / hybrid visual servoing / spherical contrast maximization

TL;DR¶

To overcome severe motion blur, narrow field of view, and translation-rotation conflation in high-speed sports, this paper presents an active gaze control system coupling an event camera, dual galvanometer mirrors, and an electrically tunable liquid telephoto lens, combined with spherical contrast maximization and an uncertainty-aware CNN to achieve 750 Hz throughput, 3 ms latency, and 1.2% magnitude / 1.5° axis error on unmodified spinning balls.

Background & Motivation¶

In professional ball sports such as table tennis, baseball, tennis, and golf, ball spin governs trajectory curvature via the Magnus effect as well as contact dynamics upon bouncing or racket impact. Table tennis rallies routinely feature spins up to 9,000 rpm, baseball curve balls reach 2,500 rpm, and golf wedge shots exceed 10,000 rpm. Athletes rely on spin to control ball trajectory, deceive opponents, and establish tactical advantages. Accurate, instantaneous estimation of 3D spin is therefore critical for tactical sports analytics, aerodynamics modeling, trajectory forecasting, and competitive robotic table tennis gameplay. However, measuring the spin of unmodified balls in free flight using conventional vision systems faces fundamental physical bottlenecks: high linear velocity causes severe motion blur even with microsecond shutter speeds, and fast spin surpasses the Nyquist limit of standard frame rates, causing acute temporal aliasing. A wide-angle camera leaves the ball spanning only a handful of pixels—rendering subtle surface texture such as printed brand logos unobservable; conversely, a static telephoto lens suffers from a tiny field of view (FoV) and a razor-thin depth of field, causing the ball to vanish from the frame or blur out of focus within milliseconds.

While event vision sensors (EVS) provide microsecond temporal resolution and wide dynamic range without motion blur, existing event-based spin estimation approaches rely almost exclusively on static wide-angle setups. Under free flight, the ball's rapid linear translation produces a massive deluge of events that pollutes and obscures the localized phase signals generated by pure rotation, fundamentally conflating translational and rotational motion. Furthermore, standard contrast maximization (CMax) frameworks evaluate sharpness on a planar image of warped events (IWE), which fails to capture the 3D spherical geometry: events warped across a curved surface project non-linearly onto a 2D plane, concentrating artificial contrast at the ball silhouette rather than dispersing it across the spherical surface, thereby trapping non-convex optimization in spurious local optima and heavily biasing the estimated rotation axis.

To dismantle these compounded barriers of spatial resolution, translational event contamination, and spherical projection distortion, this work introduces the Gaze Control System (GCS)—an active vision framework that mechanically and optically locks onto the ball. High-speed pan/tilt galvanometer mirrors continuously cancel the ball's translational motion across the image plane, while an electrically focus-tunable liquid telephoto lens keeps the ball sharply focused at high magnification, ensuring that captured event streams encode almost exclusively pure ball spin. Core idea: combine galvanometer mirrors and a tunable liquid lens into an ultra-fast active gaze control loop to physically nullify translational motion events, and formulate spherical contrast maximization (s-CMax) coupled with an uncertainty-aware CNN and GPU parallel refinement to achieve real-time, marker-free 3D spin estimation at 750 Hz and 3 ms latency.

Method¶

Overall Architecture¶

The system is structured into four cooperative tiers: active optomechanical gaze hardware, hybrid visual servoing tracking, high-accuracy offline pseudo-ground-truth generation via spherical contrast maximization (s-CMax), and a low-latency online estimation pipeline. On the hardware side, an external ball position measurement system (BPMS) continuously streams 3D ball coordinates at 200 Hz; dual-axis galvanometer mirrors with minimal rotational inertia (400 µs step response, sustained speed of 142.5 rad/s) steer the optical line of sight, while an Optotune EL-10-30 liquid lens (3 ms response) dynamically adjusts optical power to maintain focus for a 50 mm telephoto EVS (FoV ~1.8°). On the algorithmic side, the EVS asynchronous stream generates polarity-separated time surfaces from which a YOLO detector infers 2D ball offsets to close the visual tracking loop. In the offline tier, an optical flow initial guess seeds s-CMax optimization on the sphere to extract artifact-free pseudo-ground-truth spin labels. In the online tier, a lightweight ResNet-18 directly regresses 3D angular velocity alongside heteroscedastic log-variances, followed by asynchronous GPU batch magnitude refinement and inverse-uncertainty-weighted sliding window filtering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["BPMS 3D ball stream + EVS asynchronous events"] --> S1["Hybrid Visual Servoing Tracking<br/>BPMS feedforward prediction + EVS 2D closed-loop servoing"]
    S1 --> S2["Optical Flow Spin Initialization<br/>Plane-fitting tangential velocity + normal flow least squares"]
    S2 --> S3["Spherical Contrast Maximization s-CMax<br/>Orthographic unprojection + 3D rotation unwarping + spherical variance"]
    S3 --> S4["Uncertainty-Aware CNN<br/>15ms time surface input + 3D spin and diagonal log-variance regression"]
    S4 --> S5["GPU Batch Contrast Refinement<br/>Fixed predicted rotation axis + parallel 1D grid search over magnitudes"]
    S5 --> S6["Uncertainty-Weighted Sliding Filter<br/>MAD outlier rejection + inverse-variance weighting + contact reset"]
    S6 --> Out["Real-Time 3D Spin Output<br/>750 Hz throughput / 3 ms end-to-end latency"]

Key Designs¶

1. Hybrid Visual Servoing Tracking: Open-loop 3D dynamic extrapolation and closed-loop 2D image servoing

Given the telephoto lens's narrow FoV (~1.8°), an unconstrained table tennis ball at 3 m spans a spatial window of merely 9 cm, making the tracking loop exceptionally sensitive to sensing latency and sudden trajectory deflections. To maintain robust centering, the system employs a two-tier hybrid visual servoing scheme. The open-loop tier employs an improved Nakashima aerodynamics model accounting for gravity, air drag, and Magnus forces fed by previous spin estimates, predicting future positions across the measured hardware delay to compute mirror pan/tilt angles $(\phi_0, \phi_1)$ via gradient descent and adjust liquid lens focal power. Complementarily, the closed-loop tier constructs a 5 ms polarity-separated time surface from incoming events and runs a lightweight YOLO model to extract 2D pixel center offsets $(\Delta u, \Delta v)$ and radius $r$. The spatial offset is mapped back to 3D Cartesian coordinates: $$\Delta p_t = z_t R_{vc}^{-1} K_c^{-1} \begin{bmatrix} \Delta u \\ \Delta v \\ 1 \end{bmatrix}$$ This offset is added directly to the filter state, allowing the galvanometer mirrors to sustain angular velocities up to 142.5 rad/s. By keeping the ball locked to the image center, the system cancels translational optical flow at the physical source, ensuring captured events arise purely from surface spin.

2. Optical Flow Spin Initialization: Normal flow constraints and overdetermined linear solving

Contrast maximization is an inherently non-convex optimization landscape fraught with local optima caused by texture periodicity and noise. To guarantee reliable convergence, the offline pipeline extracts a coarse initial spin estimate from local event motion. Within the segmented ball event mask, 2D flow vectors are calculated via spatiotemporal plane fitting, normalized by the pixel radius, and lifted to 3D tangential velocities $v_k$ on the spherical surface. Because tangential velocity relates to angular velocity via $v_k = \omega \times x_k = -[x_k]_\times \omega$ (where $[x_k]_\times$ denotes the skew-symmetric matrix of $x_k$), and aperture effects restrict each event flow to a scalar normal constraint, an orthogonal projection operator $P_k = \frac{v_k v_k^\top}{v_k^\top v_k}$ projects the observation equations. Stacking $N$ constraints yields an overdetermined linear system $b = A\omega$, solved via singular value decomposition least squares, followed by a fast 1D grid search over magnitude to yield a high-quality initialization vector.

3. Spherical Contrast Maximization s-CMax: Unprojecting events to eliminate boundary clustering artifacts

Standard planar CMax warps events onto a 2D image plane, where the non-linear projection of a spinning 3D sphere compresses tangential velocities near the silhouette. This causes severe artificial event clustering along the ball's outer boundary, producing spurious variance peaks that derail rotation axis estimation. To resolve this geometric mismatch, s-CMax formulates contrast maximization natively on the unit sphere. Each event $(u_k, v_k)$ at timestamp $t_k$ is unprojected onto the visible hemisphere via orthographic unprojection: $$x_k = \frac{u_k - c_x}{r}, \quad y_k = \frac{v_k - c_y}{r}, \quad z_k = -\sqrt{1 - (x_k^2 + y_k^2)}$$ Using candidate spin $\omega$ and reference time $t_{\text{ref}}$, the 3D coordinate is rotated according to $x'_k = R(-\omega \Delta t_k) x_k$, where $R \in SO(3)$ is the rotation matrix. The unwarped 3D points are converted to spherical polar coordinates $s'_k = [\theta'_k, \phi'_k]$ ($\theta'_k = \arcsin(z'_k), \phi'_k = \text{arctan2}(y'_k, x'_k)$) and accumulated into an Image of Warped Events $I[s]$ via bilinear interpolation followed by a $5 \times 5$ Gaussian blur. The optimal spin vector $\omega$ is retrieved by maximizing the variance $\text{Var}(I(\omega))$ using the Nelder-Mead simplex algorithm. Modeling events directly on the sphere eliminates silhouette clustering and slashes axis estimation error down to 1.5°.

4. Uncertainty-Aware CNN: Heteroscedastic learning for self-supervised confidence estimation

While s-CMax delivers state-of-the-art accuracy, its iterative simplex optimization requires tens of milliseconds, exceeding the latency budget of real-time robot control. Furthermore, when the ball's logo faces away from the camera, lack of texture renders spin unobservable. To achieve sub-millisecond inference while identifying unobservable frames, a custom ResNet-18 model takes a 15 ms polarity-separated time surface as input and outputs a normalized 3D spin vector $\hat{\tilde{\omega}}$ alongside a diagonal log-variance vector $s = (s_x, s_y, s_z)$ with $s_i = \log \sigma_i^2$. The network is trained on 3,969 real shots labeled by offline s-CMax under a heteroscedastic Gaussian negative log-likelihood loss: $$\mathcal{L}(\hat{\tilde{\omega}}, s) = \frac{1}{2} \sum_{i \in \{x, y, z\}} \left[ \exp(-s_i) (\tilde{\omega}_i - \hat{\tilde{\omega}}_i)^2 + s_i \right]$$ This formulation penalizes overconfident incorrect predictions, forcing the network to output large variances when the brand logo is occluded or texture evidence is ambiguous, while generating tight, highly confident predictions when clear texture traces are present.

5. GPU Batch Contrast Refinement and Uncertainty Filtering: Parallel search and multi-view aggregation

At spin rates exceeding 6,000 rpm, CNN magnitude predictions tend to be conservative due to long-tail training distributions, whereas the predicted rotation axis remains remarkably accurate. To exploit this complementarity, the system deploys an asynchronous online s-CMax refinement module on the GPU. The time surface representation is decompressed into an event buffer, and thousands of CUDA threads evaluate discrete spin magnitudes along the CNN-predicted rotation axis in parallel against the spherical variance objective. If a searched candidate produces a sharper variance than the CNN prediction, the magnitude is updated while holding the directional axis fixed. In the temporal domain, a 45-frame sliding buffer performs outlier filtering via Median Absolute Deviation (MAD) and computes an inverse-variance-weighted average $\sum \sigma_i^{-2} \omega_i / \sum \sigma_i^{-2}$. In multi-GCS deployments, the viewpoint with the lowest predicted uncertainty is automatically selected, seamlessly handling the 2.2% of flight paths where the logo is temporarily hidden from a single camera.

Key Experimental Results¶

Main Results¶

The method was evaluated on a precision motor-driven ball spinner equipped with optical encoders and AprilTag orientation markers across multiple ball sports (table tennis, baseball, tennis, and golf at 1,000–9,000 rpm), followed by controlled free-flight tests comparing the active GCS against a static wide-angle EVS.

Table 1: Quantitative comparison of spin estimation methods on the motorized ball spinner benchmark

Method	Mag. Error [%] ↓	Axis Error [°] ↓	Runtime [ms] ↓	Operational Characteristics & Limitations
Gossard et al. (Flow) [16]	$35.2 \pm 19.5$	$37.1 \pm 24.2$	$7.7 \pm 14.1$	Constrained by event-rate peak finding; brittle under irregular texture
Flow (Ours, initialization)	$17.3 \pm 22.6$	$9.0 \pm 17.6$	$7.3 \pm 13.3$	Closed-form normal flow projection; halves magnitude error and cuts axis error by 4×
Nakabayashi et al. (CMax) [26]	$8.2 \pm 14.3$	$5.0 \pm 10.4$	$317.4 \pm 425.3$	Planar projection; suffers from silhouette event clustering and slow convergence
s-CMax (Ours, offline full)	$1.2 \pm 2.3$	$1.5 \pm 2.2$	$36.7 \pm 19.5$	Spherical formulation eliminates geometric bias; 9× faster than baseline with state-of-the-art accuracy

Table 2: Comparison between static wide-angle EVS and active GCS on free-flight dotted table tennis balls

Sensor Setup	Spin Estimation Method	Mag. Error [%] ↓	Axis Error [°] ↓	Failure Mode & Physical Analysis
Static Wide-Angle EVS [16, 26]	Gossard et al. (Flow) [16]	$18.8 \pm 12.9$	$88.7 \pm 8.1$	Catastrophic failure; flow aligns entirely with ball flight velocity vector
Static Wide-Angle EVS [16, 26]	Flow (Ours)	$14.1 \pm 8.7$	$59.9 \pm 5.0$	Severe translation-rotation conflation; optical flow corrupted by linear motion
Static Wide-Angle EVS [16, 26]	CMax [26]	N/A	N/A	Ball spans too few pixels; insufficient events to form a meaningful IWE
Static Wide-Angle EVS [16, 26]	s-CMax (Ours)	N/A	N/A	Severe event sparsity under low spatial resolution prevents optimization
Active GCS (Ours)	Gossard et al. (Flow) [16]	$14.7 \pm 12.6$	$75.6 \pm 18.5$	Frequency-domain heuristic fails on real ball marks
Active GCS (Ours)	Flow (Ours)	$18.0 \pm 5.4$	$11.8 \pm 29.4$	Galvo mirrors cancel translation, enabling flow to lock onto the true spin axis
Active GCS (Ours)	CMax [26]	$10.0 \pm 20.0$	$12.3 \pm 29.3$	Recovers valid estimates, but degraded by planar boundary artifacts
Active GCS (Ours)	s-CMax (Ours)	$2.1 \pm 2.5$	$5.4 \pm 3.7$	Active translation compensation combined with spherical CMax achieves benchmark-grade accuracy

Ablation Study¶

The uncertainty-aware online CNN and filtering pipeline were benchmarked on a held-out test split of 1,050 real-world shots (42,796 evaluated frames) from 7 elite table tennis matches across various uncertainty percentiles and spin magnitude brackets.

Table 3: Online CNN performance across spin magnitude ranges and predicted uncertainty percentiles

Spin Range (rpm)	Sample Count $N$	Top 10% Confident Error (Mag / Axis)	Median 50% Confident Error (Mag / Axis)	All 100% Samples Error (Mag / Axis)
0 – 2,000	16,820	4.2% / 3.1°	6.5% / 4.8°	12.8% / 8.2°
2,000 – 4,000	24,372	5.8% / 4.2°	7.9% / 5.7°	16.4% / 10.5°
4,000 – 6,000	540	7.1% / 5.0°	9.8% / 7.2°	21.5% / 13.6°
6,000 – 8,000	1,021	8.9% / 6.3°	12.4% / 8.9°	28.0% / 17.1°
8,000 – 10,000	43	11.2% / 7.8°	15.6% / 11.2°	34.2% / 22.4°
All Ranges	42,796	5.4% / 3.8°	8.8% / 6.4°	15.2% / 9.8°

Key Findings¶

Translational conflation fatally breaks static cameras: As demonstrated in Table 2, a static wide-angle EVS exhibits an apparent magnitude error of 14%–18%, but an axis error near 90°. The apparent magnitude is a coincidental numerical artifact caused by the ball thrower's linear launch velocity. Without physical gaze compensation, static event sensors cannot separate translation from spin.
Spherical formulation rectifies non-convex optimization: On the motorized spinner, s-CMax reduces magnitude error from 8.2% to 1.2% and axis error from 5.0° to 1.5° compared to planar CMax, while cutting optimization runtime from 317 ms to 36.7 ms. Unprojecting events directly onto the sphere prevents artificial clustering at the silhouette, smoothing the optimization landscape and avoiding false local optima.
Predicted uncertainty strictly correlates with physical observability: Filtering frames by predicted uncertainty monotonically drives down both magnitude and axis errors across all spin tiers. The learned log-variance reliably flags logo occlusion (observed in 2.2% of elite match trajectories), enabling multi-view fusion to seamlessly select the optimal camera view.

Highlights & Insights¶

Hardware-algorithm co-design at extreme velocities: Rather than relying on heavy robotic pan/tilt heads or brute-force high-speed framing cameras, the system pairs low-inertia galvanometer mirrors (142.5 rad/s) with a millisecond-response liquid lens and an event sensor. This physically magnifies the target by 10× and cancels image translation at the optical level, fundamentally purifying the input signal.
Closing the loop between physics-based optimization and deep learning: Offline s-CMax generates dense, high-accuracy pseudo-ground-truth without manual annotation or invasive physical markers. The lightweight CNN learns complex temporal patterns and uncertainty in real time, and the GPU parallel batch search provides instantaneous model-based correction for out-of-distribution high-spin cases.
Direct transferability to industrial and dynamic robotics tasks: The paradigm of "galvanometer gaze tracking + focus-tunable telephoto optics + event-based geometric optimization" extends naturally beyond sports analytics to high-speed defect inspection on manufacturing conveyor lines, agile drone collision avoidance, and orbital debris docking.

Limitations & Future Work¶

Dependence on natural surface texture: The system fundamentally relies on natural markings (such as ITTF manufacturer stamps, baseball seams, or golf dimples). A completely pristine, uniform white ball without any surface markings produces zero event contrast during uniform rotation, preventing spin measurement.
Tracking stability under extreme multi-ball clutter: While the hybrid tracker effectively tolerates brief dropouts and measurement noise via Nakashima dynamic modeling and EVS visual servoing, dense multi-ball scenarios or prolonged body occlusions during close-table rallies can occasionally challenge mirror re-acquisition.
Future directions: Integrating non-visible band illumination (e.g., short-wave infrared) to capture microscopic surface roughness on unprinted spheres, and compiling the neural-geometric pipeline onto neuromorphic or ultra-low-latency edge ASIC accelerators.

vs. Static Event-based Spin Estimation (Nakabayashi et al., Gossard et al.): Prior event setups rely on wide-angle stationary sensors where the ball covers only a few pixels and translation swamps rotation, yielding catastrophic axis errors (>60°) on in-flight balls. The GCS physically tracks the ball to eliminate translation and introduces spherical contrast maximization to eliminate silhouette projection distortion.
vs. Multi-Exposure and Patterned Marker Methods (Sueishi et al., SpinDOE): Existing optical systems require custom high-contrast dot patterns or specialized reflective paint, preventing their use in official competition. The GCS operates entirely on unmodified, commercially approved tournament balls.
vs. Trajectory-based Aerodynamic Inversion (Conti et al., Bi et al.): Trajectory-based methods infer spin indirectly from Magnus trajectory curvature, which requires observing a substantial portion of the flight path and remains sensitive to venue air currents. The GCS directly reads surface spin within milliseconds of racket release at 750 Hz, delivering immediate state estimates essential for robotic return stroke planning.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of galvanometer mirrors, focus-tunable liquid optics, and event cameras with spherical contrast maximization.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning multi-sport motor spinners, free-flight controlled comparisons, and 40,000+ frames from 7 elite professional table tennis matches.
Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, elegant hardware-software co-design narrative, and rigorous physical analysis of failure modes.
Value: ⭐⭐⭐⭐⭐ Directly deployed in the landmark Nature 2026 robotic table tennis system, setting a new benchmark for high-speed active vision and embodied sports intelligence.

Method	Mag. Error [%] ↓	Axis Error [°] ↓	Runtime [ms] ↓	Operational Characteristics & Limitations
Gossard et al. (Flow) [16]	\(35.2 \pm 19.5\)	\(37.1 \pm 24.2\)	\(7.7 \pm 14.1\)	Constrained by event-rate peak finding; brittle under irregular texture
Flow (Ours, initialization)	\(17.3 \pm 22.6\)	\(9.0 \pm 17.6\)	\(7.3 \pm 13.3\)	Closed-form normal flow projection; halves magnitude error and cuts axis error by 4×
Nakabayashi et al. (CMax) [26]	\(8.2 \pm 14.3\)	\(5.0 \pm 10.4\)	\(317.4 \pm 425.3\)	Planar projection; suffers from silhouette event clustering and slow convergence
s-CMax (Ours, offline full)	\(1.2 \pm 2.3\)	\(1.5 \pm 2.2\)	\(36.7 \pm 19.5\)	Spherical formulation eliminates geometric bias; 9× faster than baseline with state-of-the-art accuracy

Sensor Setup	Spin Estimation Method	Mag. Error [%] ↓	Axis Error [°] ↓	Failure Mode & Physical Analysis
Static Wide-Angle EVS [16, 26]	Gossard et al. (Flow) [16]	\(18.8 \pm 12.9\)	\(88.7 \pm 8.1\)	Catastrophic failure; flow aligns entirely with ball flight velocity vector
Static Wide-Angle EVS [16, 26]	Flow (Ours)	\(14.1 \pm 8.7\)	\(59.9 \pm 5.0\)	Severe translation-rotation conflation; optical flow corrupted by linear motion
Static Wide-Angle EVS [16, 26]	CMax [26]	N/A	N/A	Ball spans too few pixels; insufficient events to form a meaningful IWE
Static Wide-Angle EVS [16, 26]	s-CMax (Ours)	N/A	N/A	Severe event sparsity under low spatial resolution prevents optimization
Active GCS (Ours)	Gossard et al. (Flow) [16]	\(14.7 \pm 12.6\)	\(75.6 \pm 18.5\)	Frequency-domain heuristic fails on real ball marks
Active GCS (Ours)	Flow (Ours)	\(18.0 \pm 5.4\)	\(11.8 \pm 29.4\)	Galvo mirrors cancel translation, enabling flow to lock onto the true spin axis
Active GCS (Ours)	CMax [26]	\(10.0 \pm 20.0\)	\(12.3 \pm 29.3\)	Recovers valid estimates, but degraded by planar boundary artifacts
Active GCS (Ours)	s-CMax (Ours)	\(2.1 \pm 2.5\)	\(5.4 \pm 3.7\)	Active translation compensation combined with spherical CMax achieves benchmark-grade accuracy