Unwarping the Lens: A Physics-Grounded Approach to Video Glasses Removal¶

Conference: ECCV 2026
Paper: ECCV 2026 Poster 3821
Code: https://radimspetlik.github.io/unwarpingthelens/
Area: Image Generation
Keywords: Video Editing, Image Editing, Glasses Removal, Optical Simulation, Gaze Preservation

TL;DR¶

To tackle identity drift and temporal flickering in generative eyeglasses removal, this paper transfers large-scale generative priors to a lightweight deterministic network (JFSnet) via rigorous three-stage geometric filtering, physics-based lens refraction/HDR reflection augmentation, and translation equivariance constraints, achieving real-time (27.68 FPS) video restoration with high ocular fidelity.

Background & Motivation¶

Precise local facial attribute editing in high-resolution video remains a challenging problem in computer vision. Unlike generic opaque occluders, eyeglasses are complex optical components that introduce continuous depth-dependent refractive distortions (such as minification from myopic lenses) and view-dependent specular reflections. Standard generative inpainting and diffusion models struggle to disentangle these optical effects from underlying facial geometry, often hallucinating synthetic eye features that fail to preserve the subject's unique identity, gaze direction, and delicate expressions.

Current eyeglasses removal approaches predominantly rely on stochastic generative priors to synthesize occluded regions. However, the human visual system is exceptionally sensitive to ocular cues, where subtle pupil shifts or deformed contours trigger noticeable identity drift. While generative models can synthesize plausible static eyes in isolation, unconstrained latent hallucination causes severe temporal flickering and structural divergence when deployed frame-by-frame on video sequences. Meanwhile, generic video inpainting methods treat eyewear as opaque targets and rely on neighboring frame propagation, which fails because the underlying eye geometry is refracted and corrupted by specular reflections rather than entirely missing.

To reconcile the photo-realism of generative models with physical geometric consistency, this paper proposes an offline generative-to-deterministic transfer framework. Core Idea: Multi-view face sets synthesized by a commercial generative prior are curated via three-stage structural filtering and ARAP alignment, augmented on-the-fly with physics-based ray-tracing refraction and HDR reflection, and transferred into a deterministic pixel-space restoration network (JFSnet) regularized by translation equivariance to eliminate temporal flickering.

Method¶

Overall Architecture¶

The framework transfers rich generative knowledge into a feed-forward restoration network that inverts optical distortions without relying on stochastic latent sampling. The pipeline operates in three sequential phases: first, high-fidelity clean and glasses-wearing multi-pose face sets (13 poses/expressions) are generated using Nano Banana (Gemini 3 Pro Image), refined via ARAP deformation mesh alignment, and filtered through three-stage structural consistency checks; second, an online physics-grounded augmentation module simulates lens optical refraction via Snell's law and HDR specular reflection via inverse-view environment mapping; third, the synthesized degraded images train JFSnet—a hybrid architecture combining a DINOv2 semantic encoder, a multi-scale convolutional decoder, and direct pixel-level skip connections—optimized with reconstruction, localized ocular, adversarial, and translation equivariance losses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-View Generative Prior Extraction<br/>Nano Banana 13-pose sets + segmentations"] --> B["Three-Stage Structural Filtering & ARAP Alignment<br/>L1 background/ocular filtering + U-Net + ARAP warp"]
    B --> C["Physics-Based Lens Optics Augmentation<br/>Snell's law refraction + inverse-view HDR reflection"]
    C --> D["JFSnet Joint Feature-Spatial Restoration<br/>DINOv2 encoder + Conv decoder + input skip"]
    D --> E["Translation Equivariance & Ocular Objectives<br/>Spatial shift regularization + localized eye losses"]
    E --> F["High-Fidelity Eyeglass-Free Video Frames"]

Key Designs¶

1. Three-Stage Structural Filtering & ARAP Mesh Alignment: Eliminating Latent Drift Raw image pairs generated from large-scale foundation models suffer from subtle stochastic variations, head rotation discrepancies, and gaze drift due to spatial compression in latent tokenizers. To construct pristine ground-truth pairs, the framework implements a three-stage hard-sampling filter. In the first two stages, background $L_1$ pixel differences filter out identities with global head motion, followed by localized ocular $L_1$ and gaze-consistency filtering to eliminate expression shifts (with physics substitution down-weighted for extreme profiles beyond 3/4 view). In the third stage, a proxy U-Net reconstruction network is trained on candidate pairs to reject samples with high residual fitting errors. Surviving pairs undergo As-Rigid-As-Possible (ARAP) mesh deformation guided by facial landmarks and multi-scale photometric constraints, perfectly aligning clean faces to the corresponding glasses-wearing poses and yielding 1,860 clean synthetic identities (24,180 image pairs).

2. Physics-Grounded Refraction & HDR Reflection Simulation: Bridging the Optical Gap Generative models exhibit simplicity bias, failing to capture diverse physical lens coatings, strong chromatic reflections, and high diopter minification. To overcome this limitation, the authors replace the generated lens area with an on-the-fly physics-based optical simulation during training. A 3D parametric face model extracts head orientation $R$ and translation $t$, with adult metric head width anchoring depth scale. Spherical lens surfaces are situated 15mm along the face normal from pupils, parameterized by target diopter $D \in [-6.0, +1.0]$ via Vogel's Rule: $D_1 = D/2 + 6.0$ and $D_2 = D - D_1$, with refractive index $n \approx 1.5$. Pixel rays cast into the lens are refracted according to Snell's law, generating a dense spatial warp field $W$. For reflections, an inverse-view environment map synthesized by Nano Banana is sampled along reflection rays in High Dynamic Range (HDR) with randomized anti-reflective (AR) coatings. The composite training input is formed as: $$I = (1 - M_{\text{lens}}) \odot G_{\text{glasses}} + M_{\text{lens}} \odot (W(G_{\text{clean}}) + R)$$ compelling the network to explicitly invert the refractive warp $W$ and subtract reflection $R$.

3. JFSnet Joint Feature-Spatial Architecture: Preserving High-Frequency Detail Unlike latent diffusion models that squeeze image information into a lossy autoencoder manifold, JFSnet operates end-to-end in full pixel space. The encoder leverages pre-trained DINOv2 (ViT-L/14) to extract invariant semantic representations, fine-tuning only its final four layers to maintain generalizability while adapting to facial restoration. Multi-scale feature maps from the ViT are fused into a ResNet-based convolutional decoder. Crucially, a direct pixel-level skip connection bypasses the semantic bottleneck, funneling unaltered non-occluded facial regions directly to output layers and allowing the network capacity to focus entirely on reconstructing the unwarped ocular region.

4. Translation Equivariance Regularization: Zero-Overhead Video Stabilization To suppress high-frequency frame-to-frame flickering without the heavy computational burden of multi-frame cross-attention or dense optical flow estimation, the authors enforce translation equivariance. During training, a small random spatial 2D translation operator $T_{\Delta s}$ is applied to the input image $I$. The temporal stability loss penalizes non-commutation between the restoration network $f_\theta$ and the translation operator: $$\mathcal{L}_{\text{temp}} = \| f_\theta(T_{\Delta s}(I)) - T_{\Delta s}(f_\theta(I)) \|_1$$ This constraint forces the deterministic mapping to commute with small rigid motions, ensuring that reconstructed eye contours and fine textures translate coherently across frames without inducing temporal fluttering.

Loss & Training¶

JFSnet is optimized end-to-end with composite multi-scale objectives: $$\mathcal{L} = \lambda_{\text{pix}}\mathcal{L}_{\text{pix}} + \lambda_{\text{perc}}\mathcal{L}_{\text{perc}} + \lambda_{\text{eye-pix}}\mathcal{L}_{\text{eye-pix}} + \lambda_{\text{eye-perc}}\mathcal{L}_{\text{eye-perc}} + \lambda_{\text{adv}}\mathcal{L}_{\text{adv}} + \lambda_{\text{temp}}\mathcal{L}_{\text{temp}}$$ where $\mathcal{L}_{\text{pix}}$ is the global $L_1$ reconstruction loss, and $\mathcal{L}_{\text{perc}}$ represents VGG-19 Gram-matrix perceptual loss. Eye-specific objectives $\mathcal{L}_{\text{eye-pix}}$ and $\mathcal{L}_{\text{eye-perc}}$ are evaluated on $64 \times 64$ patches centered at the center of gravity of each lens mask to guarantee sharp pupil restoration. Adversarial loss $\mathcal{L}_{\text{adv}}$ is provided by a PatchGAN critic. The pipeline is implemented in JAX/Flax on TPUs/GPUs at $518 \times 518$ resolution for 400,000 steps using AdamW (base learning rate $3 \times 10^{-4}$ for JFSnet/critic, $1 \times 10^{-5}$ for the fine-tuned DINOv2 layers).

Key Experimental Results¶

Main Results¶

Evaluation is conducted on the curated FFHQ dataset (12,163 clear glasses images) evaluated against real unoccluded portraits, measuring distribution fidelity (FID), pupil displacement error via MediaPipe FaceMesh, facial landmark $L_2$ error, and runtime throughput.

Note: Data cited directly from Table 1 of the paper.

Method	FID ↓	Pupil Disp. (px) ↓	Landmark L2 (px) ↓	FPS ↑
Nano Banana (Gemini 3 Pro)	0.389	2.854 ± 2.387	0.699 ± 0.462	1.29
Inpaint Anything	0.391	2.259 ± 2.295	0.664 ± 0.902	1.84
InstructPix2Pix	1.089	5.541 ± 6.522	8.038 ± 18.293	0.05
LEDITS	1.953	3.905 ± 2.999	1.527 ± 2.437	0.08
Take-Off-Eyeglasses (TOE)	3.256	2.593 ± 1.696	0.877 ± 0.773	13.31
IP-FaceDiff	3.832	17.607 ± 8.718	21.257 ± 27.171	0.04
RAVE	11.559	3.577 ± 2.373	2.356 ± 2.773	0.16
TokenFlow	2.430	3.689 ± 2.812	1.905 ± 1.678	0.04
Flow-Guided Transformer (FGT)	0.569	3.078 ± 2.714	0.965 ± 0.952	1.28
ProPainter	0.408	2.684 ± 2.838	0.947 ± 1.095	8.77
Ours (JFSnet)	0.379	2.249 ± 1.957	0.632 ± 0.271	27.68

User Preference Study¶

A perceptual study involving 37 participants assessed 60 CelebV-Text video clips and 60 FFHQ portrait images (cited from Figure 6).

Baseline Comparison	Video Gaze (%)	Video Stability (%)	Video Quality (%)	Image Gaze (%)	Image Removal (%)	Image Identity (%)
vs. Nano Banana	97%	100%	100%	81%	42%	75%
vs. Runway Gen-4.5	74%	59%	75%	N/A	N/A	N/A
vs. ProPainter	76%	76%	96%	84%	93%	84%
vs. TOE	91%	97%	98%	85%	96%	89%
vs. V-LASIK	88%	88%	100%	N/A	N/A	N/A
vs. LEDITS	100%	100%	96%	90%	85%	98%

Key Findings¶

Superior Structural Fidelity and Distribution Realism: JFSnet achieves the lowest FID (0.379) on FFHQ, outperforming large foundation models (Nano Banana at 0.389) and video inpainting methods (ProPainter at 0.408). It also attains the lowest pupil displacement (2.249 px) and landmark error (0.632 px) with the tightest standard deviations.
Ablation Validations: Full ablation demonstrates that omitting refraction or reflection simulation severely impairs FID. Eliminating dataset filtering or ARAP mesh warping reintroduces misalignment artifacts. Unfreezing the last 4 layers of DINOv2 provides crucial domain adaptation.
High Computational Efficiency: Operating at 27.68 FPS in feed-forward pixel space, JFSnet is orders of magnitude faster than diffusion-based editing pipelines (0.04 - 0.16 FPS) while entirely bypassing recurrent multi-frame propagation.

Highlights & Insights¶

Inverting Physical Optics rather than Inpainting Blindly: Treating eyeglasses as refractive and reflective optical elements rather than solid masks turns an ambiguous hallucination task into a well-posed physical inverse problem.
Translation Equivariance as Free Temporal Stabilization: Enforcing simple spatial translation invariance during single-image training successfully eliminates frame-to-frame ocular fluttering in video playback without needing flow estimation or temporal attention.
Synthesizing at Scale with ARAP Guardrails: Combining massive foundation model data generation with rigid-as-possible mesh deformation and multi-stage hard rejection solves the fundamental identity drift problem inherent in synthetic pair generation.

Limitations & Future Work¶

Degradation at Extreme Profile Angles: Monocular depth estimation and 3D face fitting degrade when head poses approach full profile views (>3/4 rotation), forcing the system to fall back on raw generative outputs.
Opaque Sunglasses Inpainting: The physical refraction model assumes some light transmission through the lens; for completely opaque or mirror sunglasses with zero ocular visibility, the network must rely purely on generative hallucination.
Future Directions: Integrating explicit 3D Gaussian Splatting (3DGS) head avatars with physical ray tracing to provide unified view-consistent 3D eyewear removal.

vs. Foundation Video Generators (Runway Gen-4.5 / Nano Banana): While commercial models excel at style synthesis, they frequently distort gaze, blink states, and facial identities; JFSnet preserves identity and achieved a 100% win-rate against Nano Banana on video stability.
vs. Video Inpainting Methods (ProPainter / FGT / STTN): Video inpainting methods treat the ocular area as an empty hole, leaving ghosting shadows and blurry pupils; JFSnet explicitly reconstructs minified eye geometry.
vs. Dedicated Eyeglass Removal (TOE / V-LASIK): Prior dedicated architectures struggle with strong specular highlights or lack real-time throughput; JFSnet combines DINOv2 semantics with HDR simulation for sharper, faster restoration.

Rating¶

Novelty: ⭐⭐⭐⭐ [Solid fusion of generative foundation model distillation, physical ray optics, and translation equivariance]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation on 12k FFHQ images, CelebV-Text videos, detailed ablations, and 37-person perceptual study]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear formulation, well-structured pipeline, and mathematically grounded optical formulation]
Value: ⭐⭐⭐⭐⭐ [Solves a long-standing artifact in facial video editing while delivering real-time 27.68 FPS speed]