Large-Scale Light Field Synthesis from Videos Enables Geometrically Consistent Bokeh Editing

Conference: ECCV 2026
Paper: ECCV Official
Code: https://www.hm-cai.com/vid2bokeh/
Area: 3D Vision
Keywords: Bokeh Editing, Light Field Synthesis, Geometric Fidelity, Diffusion Models, Feed-Forward 3D Reconstruction

TL;DR

Addressing the failure of depth-based bokeh editing on complex geometries such as reflections and intricate occlusions, Vid2Bokeh presents a scalable data acquisition pipeline that reconstructs dense 25×25 light fields from casual videos via feed-forward 3D reconstruction to bypass depth estimation entirely, producing a 100K-scene dataset that empowers a diffusion model to perform robust, geometrically consistent bidirectional focus and depth-of-field editing.

Background & Motivation

Focus and depth of field (DoF) are the fundamental visual storytelling instruments that dictate where viewer attention lands and what range of a scene remains sharp. In computational post-capture bokeh editing, creators increasingly seek an unconstrained, full-parameter manipulation paradigm: simultaneously shifting the in-focus plane forward or backward and widening or narrowing the synthetic aperture radius, whether starting from an all-in-focus photograph or an image already exhibiting natural camera defocus. However, existing post-capture bokeh editing algorithms consistently break down on intricate geometries—producing distracting boundary halos, inaccurate depth bleed, and structural distortion across mirror reflections, water puddles, transparent containers, and porous barriers like wire fences and tennis nets.

This widespread vulnerability originates from an unavoidable trilemma in existing training data pipelines: they either sacrifice geometric fidelity for scalability, or sacrifice scalability for geometric fidelity. Physically captured datasets relying on multi-camera rigs or motorized focus stepping provide optically faithful defocus blur and natural circle-of-confusion (CoC) discs, yet their immense manual calibration overhead limits dataset size to a few hundred or thousand scenes. Conversely, synthetic data pipelines scale easily but uniformly rely on 2.5D monocular depth estimation models (such as Depth Anything). When depth estimators inevitably fail on non-Lambertian surfaces—such as assigning the distant reflection in a mirror to the near mirror glass, or merging thin bicycle spokes into the background—these fatal geometric hallucinations become baked into the training supervision, training downstream networks to produce distorted blur.

The key insight of this paper is that dense 4D light fields naturally represent multi-view parallax, specular disocclusions, and see-through structures without needing single-view depth proxies, and recent feed-forward 3D novel view synthesis models have reached the efficiency threshold required to turn casual monocular videos into dense multi-view light fields in seconds. Core idea: reconstruct dense 25×25 light fields from casual videos via feed-forward 3D novel view synthesis, bypassing depth estimation entirely to generate a 100K-scene geometrically faithful Vid2Bokeh dataset, and train a diffusion model on physical optical integration pairs to achieve geometrically consistent bidirectional focus and DoF editing.

Method

Overall Architecture

The Vid2Bokeh framework encompasses two stages: a scalable, depth-estimation-free data acquisition pipeline that processes in-the-wild video clips into 4D light field representations, and a latent diffusion model conditioned on relative optical offsets for bidirectional focus and aperture manipulation. Given monocular video sequences from the DL3DV-10K dataset with calibrated camera trajectories, keyframes are designated as center views around which a \(25 \times 25\) grid of virtual camera viewpoints with small baseline perturbations is established. An optical-flow-driven adaptive baseline calibration dynamically matches scene disparity scales, after which the feed-forward 3D model LaCT synthesizes the complete 625-view angular grid in under 30 seconds on an A100 GPU. During diffusion model training, supervision image pairs \((I_{\text{in}}, I_{\text{gt}})\) are rendered on the fly from the light field using standard physical refocusing integration under two randomly sampled focal plane and aperture radius settings \((\kappa_1, a_1)\) and \((\kappa_2, a_2)\), feeding the target offset \((\Delta\kappa, \Delta a)\) into an SDXL-based latent diffusion backbone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video Sequence + Calibrated Poses"] --> B["Adaptive Baseline Calibration<br/>Flow-guided viewpoint spacing adjustment"]
    B --> C["Feed-Forward 3D Light Field Synthesis<br/>LaCT generates dense 25x25 view grid"]
    C --> D["On-the-Fly Light Field Refocusing<br/>Physical optical integration over (κ, a) pairs"]
    D --> E["Bidirectional Focus-DoF Diffusion Model<br/>Spatial latent concatenation with optical offsets"]
    E --> F["Geometrically Consistent Edited Image"]

Key Designs

1. Adaptive Baseline Calibration: scene-aware disparity normalization across diverse depths

Directly applying a rigid physical baseline spacing across arbitrary real-world video scenes causes severe disparity mismatch: scenes with ultra-close foreground objects exhibit excessive angular disparity, inducing visible phase aliasing artifacts during aperture integration; conversely, expansive outdoor environments yield negligible disparity, severely attenuating the visible refocusing effect. To enforce consistent, physically plausible parallax across 100K diverse scenes, Vid2Bokeh incorporates a flow-guided adaptive calibration step. For any designated center view, the pipeline calculates pairwise optical flow between adjacent virtual viewpoints using a pretrained RAFT model. The physical spacing between adjacent virtual cameras is then dynamically scaled so that the mean optical flow magnitude strictly falls within an empirically optimal target range of one to three pixels, guaranteeing stable, artifact-free refocusing dynamics across arbitrary scene scales.

2. Feed-Forward 3D Light Field Synthesis: bypassing the geometric bottleneck of depth estimation

To eliminate the systemic failure modes of 2.5D monocular depth estimators, Vid2Bokeh adopts feed-forward novel view synthesis (LaCT) to directly hallucinate the angular ray bundle \(L(u, v, s, t, c)\) from sparse video context views, where \((u, v)\) represent angular aperture coordinates across the \(25 \times 25\) virtual lens array and \((s, t)\) denote spatial pixel coordinates. Because this feed-forward formulation requires no per-scene optimization and runs in under 30 seconds per scene on an A100 GPU, it scales effortlessly across large video corpora. More fundamentally, representing the visual volume as a multi-view light field inherently models multi-plane parallax and non-Lambertian transport: virtual reflections behind mirror glass and see-through background textures behind fine wire meshes naturally exhibit realistic angular shift, completely avoiding the depth edge bleed and flat-plane assumptions that cripple depth-based rendering pipelines.

3. On-the-Fly Light Field Refocusing: continuous coverage of bidirectional focus and DoF space

With the dense 4D light field constructed, training supervision pairs do not need to be pre-rendered and saved to disk. Instead, bokeh images are integrated dynamically during data loading via standard light field shear-and-sum refocusing:

\[I(s, t, c;\, \kappa, a) = \iint_{\mathcal{A}(a)} L(u, v,\, s + \kappa u,\, t + \kappa v,\, c) \, \mathrm{d}u \, \mathrm{d}v\]

where \(\mathcal{A}(a)\) denotes the circular virtual aperture domain of radius \(a\), and \(\kappa\) defines the focal plane shift parameter. As \(a \to 0\), the integration contracts to a pinhole all-in-focus view; as \(a\) expands, ray integration across diverse angular viewpoints synthesizes realistic circle-of-confusion blur. In each training iteration, two independent optical states \((\kappa_1, a_1)\) and \((\kappa_2, a_2) = (\kappa_1 + \Delta\kappa, a_1 + \Delta a)\) are sampled to generate the source image \(I_{\text{in}} = I(s,t,c;\kappa_1, a_1)\) and ground truth \(I_{\text{gt}} = I(s,t,c;\kappa_2, a_2)\). Because the relative offsets \((\Delta\kappa, \Delta a)\) freely assume positive or negative values, the model learns a unified bidirectional mapping capable of shifting focus near-to-far, far-to-near, creating shallower depth of field, or removing blur to restore all-in-focus imagery.

4. Bidirectional Focus-DoF Diffusion Model: latent-space conditional guidance and denoising

The downstream editing backbone leverages a fine-tuned SDXL latent diffusion model. The input image \(I_{\text{in}}\) is encoded through the frozen VAE encoder into source latent \(z_s\). Continuous optical offsets \((\Delta\kappa, \Delta a)\) are mapped through high-frequency Fourier feature embeddings and a lightweight MLP into a conditioning vector integrated into cross-attention blocks. Following the spatial concatenation paradigm of InstructMove, the U-Net takes the channel-wise concatenation of the noisy target latent and the source latent \(z_s\) as input. At test time, a user inputs a single photograph with arbitrary focus and specifies target optical shifts \((\Delta\kappa, \Delta a)\); the model iteratively denoises a pure Gaussian noise latent guided by \(z_s\), synthesizing the edited output without relying on intermediate depth maps.

A Worked Example

Consider an in-the-wild input photograph \(I_{\text{in}}\) focused tightly on the front wheel hub of a bicycle, where background spokes and distant buildings are heavily blurred by natural shallow depth of field. The user desires two adjustments: shifting the sharp focus back to the distant building facade while narrowing the aperture to increase overall sharpness. 1. Specifying Parameter Offsets: The user inputs \(\Delta\kappa = +1.5\) (moving focus plane toward far infinity) and \(\Delta a = -0.6\) (reducing blur disc radius). 2. Condition Injection: The pair \((\Delta\kappa, \Delta a)\) is projected via Fourier feature transformation and an MLP into cross-attention tokens that modulate the diffusion denoiser. 3. Latent Inversion & Denoising: The encoded source latent \(z_s\) is concatenated with Gaussian noise. Guided by the geometric priors learned from Vid2Bokeh's 100K light fields, the diffusion model recognizes that the narrow gaps between foreground spokes correspond to distant physical space. It reconstructs sharp architectural textures within those spoke gaps without mistaking the spokes for an opaque foreground surface, while smoothly tapering the front wheel hub into a mild, natural defocus state.

Key Experimental Results

Main Results

The authors conduct quantitative evaluations across Blender-rendered physical simulations (measuring data acquisition fidelity), the Vid2Bokeh test set for bidirectional refocusing, and the EBB! real bokeh dataset.

Table 1: Quantitative Validation of Data Acquisition Fidelity on Blender-Simulated Scenes Evaluated on 100 diverse camera viewpoints, comparing Vid2Bokeh and the depth-based baseline BokehDiff against native path tracing (Blender Direct Bokeh) and ideal light field integration (Blender LF-based Bokeh):

Method	Direct PSNR↑	Direct SSIM↑	Direct MS-SSIM↑	Direct LPIPS↓	LF PSNR↑	LF SSIM↑	LF MS-SSIM↑	LF LPIPS↓
Blender LF-based (Upper Bound)	31.03	0.956	0.978	0.069	\(\infty\)	1.000	1.000	0.000
BokehDiff (Depth-based Baseline)	28.70	0.900	0.948	0.141	30.72	0.889	0.954	0.216
Vid2Bokeh (Ours)	29.71	0.929	0.971	0.108	33.37	0.931	0.984	0.147

Table 2: Quantitative Comparison of Bidirectional Refocusing on Vid2Bokeh Test Set Evaluating refocusing transitions across both Near-to-Far and Far-to-Near directions:

Method	Near-to-Far PSNR↑	Near-to-Far SSIM↑	Near-to-Far LPIPS↓	Near-to-Far DISTS↓	Far-to-Near PSNR↑	Far-to-Near SSIM↑	Far-to-Near LPIPS↓	Far-to-Near DISTS↓
DRBNet + Dr.Bokeh	28.03	0.841	0.294	0.169	27.90	0.838	0.299	0.171
DRBNet + BokehMe	28.18	0.846	0.286	0.166	28.05	0.843	0.291	0.168
DRBNet + BokehDiff	28.82	0.834	0.232	0.132	28.67	0.861	0.238	0.145
DiffCamera	28.32	0.864	0.202	0.125	28.84	0.872	0.201	0.138
Ours	29.62	0.907	0.198	0.108	29.51	0.893	0.183	0.121

Ablation Study

Table 3: Quantitative Comparisons of Bokeh Rendering across Five Progressive Aperture Radii (Vid2Bokeh Test Set)

Config / Method	Vid2Bokeh PSNR↑	Vid2Bokeh SSIM↑	Vid2Bokeh LPIPS↓	Vid2Bokeh DISTS↓
BokehMe	26.91	0.833	0.261	0.194
Dr.Bokeh	27.10	0.838	0.269	0.180
Bokehlicious	27.42	0.842	0.198	0.152
BokehDiff	27.36	0.846	0.231	0.148
Ours (Full Model)	28.05	0.897	0.129	0.092

Key Findings

• Immunity to monocular depth collapse on specular surfaces: In mirror and puddle reflection scenes, monocular depth models consistently map the reflected virtual objects onto the planar reflective surface, forcing refocusing algorithms to blur reflections when focusing on distant structures. Models trained on Vid2Bokeh correctly preserve sharp far-field reflections.
• Inherent robustness via angular integration averaging: Stress-testing the pipeline by deliberately starving the feed-forward 3D reconstructor with sparse views produces noticeable local visual artifacts in individual light field sub-aperture views; however, the \(25 \times 25\) aperture integration acts as an effective spatial-angular low-pass filter, canceling out random independent view artifacts and yielding clean bokeh.
• Accurate handling of see-through partial occlusions: In challenging scenes featuring fine structural boundaries (e.g., tennis court nets, thin flagpoles, bare tree branches), cascaded deblur-and-refocus pipelines treat the foreground mesh as an opaque occluding layer, severely distorting or erasing background geometry. In contrast, Vid2Bokeh accurately resolves see-through blur.

Highlights & Insights

• Shifting the paradigm from depth maps to physical ray fields: While previous works attempted increasingly complex heuristic layers, Matting Laplacians, or neural depth refine networks, they remained bound to 2.5D geometric approximations. Vid2Bokeh demonstrates that feed-forward 3D novel view synthesis is now fast and robust enough to reconstruct dense light fields, solving the root cause of geometric distortion.
• Aperture integration as an error-canceling mechanism: A common concern in 3D-assisted data synthesis is that novel view synthesis flaws will corrupt training. The authors reveal that the optical refocusing integral over 625 sub-aperture views inherently washes out isolated per-view reconstruction glitches.
• Unlocking scalable acquisition from casual consumer video: Bypassing dedicated multi-camera rigs and cumbersome gantry systems allows the training data corpus to scale to over 100,000 real-world scenes directly from handheld mobile recordings.

Limitations & Future Work

• Circular aperture approximation & sRGB clipping: The current implementation restricts virtual apertures to ideal circular discs, omitting polygonal aperture blades or anamorphic characteristics. Furthermore, capturing true high-dynamic-range highlights (bokeh balls) requires processing in RAW linear sensor space, whereas existing feed-forward 3D reconstruction models operate predominantly on tone-mapped sRGB video frames.
• Static scene assumption: Because the feed-forward reconstructor relies on multi-view camera poses derived from monocular camera motion, the pipeline assumes largely static environments; fast independent object motions during recording can cause local reconstruction fluttering.
• Future Directions: Integrating RAW-capable feed-forward radiance fields (e.g., RawNeRF variants) to render true specular highlight bokeh discs, and scaling to 4K native output models.

Related Work & Insights

• vs BokehDiff / BokehMe / Dr.Bokeh: Prior methods rely on single-image depth estimation or multi-plane images (MPIs) followed by depth-aware blurring or diffusion-based one-step inpainting. Under reflections or fine occluders, erroneous depth causes irreversible haloing and blur bleed; Vid2Bokeh bypasses depth estimation completely, achieving geometrically truthful blur via physical ray accumulation.
• vs DPDD / RealBokeh / EBB!: Real-world camera datasets provide authentic optical defocus but are constrained by complex capture setups to limited scene counts (hundreds to low thousands). Vid2Bokeh scales to 100K diverse scenes while providing fully continuous, bidirectional focus and aperture control.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Replaces the conventional depth-dependent bokeh synthesis philosophy with feed-forward 3D light field reconstruction from casual videos.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning Blender ground-truth physics benchmarks, bidirectional refocusing, bokeh rendering, and defocus blur removal.
• Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative, rigorous technical explanations of physical optics, and clear conceptual alignment between framework architecture and key designs.
• Value: ⭐⭐⭐⭐⭐ Establishes a 100K-scene light field foundation dataset and proves that geometric fidelity in data acquisition is the primary driver of robust post-capture focus editing.