InFlux++: Real and Synthetic Data for Estimating Dynamic Camera Intrinsics

Conference: ECCV 2026
Paper: ECCV Official
Project: https://influx.cs.princeton.edu/
Area: 3D Vision
Keywords: dynamic camera intrinsics, real and synthetic datasets, lens distortion, focal length estimation, procedural generation

TL;DR

InFlux++ introduces a dual-track data suite comprising 1,841 procedurally generated synthetic videos (441K+ frames with thin lens breathing and realistic optical aberrations) and 334 high-resolution real-world videos (514K+ frames with rich translation and reverse large-FSF calibration), where finetuning SOTA dynamic calibration models on synthetic data significantly boosts focal length estimation accuracy.

Background & Motivation

Camera intrinsics define the fundamental perspective projection mapping 3D scene coordinates to the 2D image plane, serving as the essential geometric cornerstone for robotic visual navigation, Structure-from-Motion (SfM), Visual SLAM, 3D Gaussian Splatting (3DGS), and Neural Radiance Fields (NeRF). However, virtually all classical and learning-based 3D vision algorithms rely on the rigid assumption that camera intrinsics remain constant throughout the input video sequence. In in-the-wild video capture, zoom lenses frequently undergo optical configuration shifts due to mechanical adjustments of the zoom ring and focus ring. Violating the constant-intrinsics assumption introduces catastrophic calibration errors into downstream geometric pipelines, inducing severe scale drift, degraded bundle adjustment, and distorted 3D point cloud reconstructions. Accurately estimating per-frame dynamic camera intrinsics directly from monocular RGB video is therefore crucial for bringing 3D vision systems into open-world, time-varying optical scenarios.

Prior work, notably InFlux, established the first real-world benchmark featuring per-frame ground truth intrinsics to evaluate dynamic calibration algorithms. Despite this advance, state-of-the-art single-view and video self-calibration models struggle, with top-performing methods like AnyCalib achieving only a 34.1% endpoint error (EPE) recall at a 50-pixel threshold. This performance bottleneck stems from two major obstacles: first, training data for dynamic intrinsics estimation is exceedingly scarce and lacks parameter diversity. Real-world datasets feature static intrinsics; panorama-crop approaches lack temporal continuity and suffer from stitching seams and lens artifacts; and existing synthetic suites rely almost exclusively on ideal pinhole models that omit physical defocus blur, lens breathing, and chromatic aberrations. Second, existing benchmarks exhibit narrow scene and camera motion diversity. InFlux concentrates heavily on campus outdoor views and staged, static indoor workspaces, while camera trajectories are predominantly rotational, leaving algorithms untested on translational motion with rich motion parallax.

To bridge this dual gap in training supervision and evaluation coverage, the field demands a physically grounded, large-scale synthetic video dataset alongside an extended, motion-rich real-world benchmark. Core idea: build the comprehensive InFlux++ data suite, combining InFlux++ Synth (procedural generation with thin lens optical parameterization, bounded random-walk parameter variation, lens breathing, and on-the-fly lens distortion) and InFlux++ Real (reverse large-FSF static-target calibration, diverse domestic activities, and vehicular translation), establishing synthetic supervision as a powerful engine for dynamic camera intrinsics estimation.

Method

Overall Architecture

InFlux++ tackles dynamic intrinsics estimation through a coordinated two-pronged framework: physically realistic procedural synthetic generation and comprehensive real-world geometric benchmark collection. The pipeline spans procedural scene rendering, coupled optical parameterization, depth-guided temporal trajectory sampling, online distortion augmentation, and reverse-motion spatial calibration for real-world lenses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["InFlux++ Dynamic Intrinsics Suite"] --> B["InFlux++ Synth Procedural Dataset<br/>441K+ frames / 1,841 videos"]
    A --> C["InFlux++ Real Extended Benchmark<br/>514K+ frames / 334 videos"]

    B --> D["Thin Lens Optical Parameterization & Breathing<br/>Coupled LFL & LTO physics solving CFL"]
    D --> E["Temporally Smooth Bounded Random Walk<br/>Bézier interpolation & depth-adaptive focus"]
    E --> F["Online Distortion Injection & View Planning<br/>Brown-Conrady model & normal-depth pruning"]

    C --> G["Reverse Large-FSF Static Calibration<br/>Moving camera around fixed multi-story AprilTag target"]
    G --> H["Translational Parallax Real Video Capture<br/>Handheld/vehicular daily scenes with privacy blur"]

    F --> I["Synthetic Supervised Pretraining / Finetuning"]
    H --> I
    I --> J["Robust Dynamic Intrinsics Estimation"]

Key Designs

1. Thin Lens Optical Parameterization & Breathing Simulation: Grounding effective focal length drift in lens physics

Standard graphics renderers, including baseline Blender setups, treat camera focal length and object focus distance as independent numerical settings, ignoring the coupled physical optics of real lens assemblies. In real photographic zoom systems, the effective Camera Focal Length (CFL) depends not only on the Lens Focal Length (LFL, the focal length when focused at infinity, dictated by the zoom ring) but also on the Lens to Object Distance (LTO, dictated by the focus ring). To faithfully replicate real zoom dynamics, InFlux++ Synth parameterizes CFL via the Gaussian thin lens equation rather than directly assigning it:

\[ \frac{1}{\text{LFL}} = \frac{1}{\text{CFL}} + \frac{1}{\text{LTO}} \implies \text{CFL} = \frac{\text{LFL} \cdot \text{LTO}}{\text{LTO} - \text{LFL}} \]

This formulation is natively aligned with ray tracing in Blender's Cycles engine. Crucially, it naturally reproduces "lens breathing"—the subtle field-of-view (FOV) shift that occurs in physical lenses when adjusting focus distance (LTO) even while the zoom ring (LFL) remains static. Introducing lens breathing into synthetic renders supplies models with an authentic optical cue for estimating time-varying camera geometry.

2. Temporally Smooth Bounded Random Walk & Depth-Adaptive Focus Sampling: Producing continuous, in-focus video trajectories

Real zoom and focus adjustments are smooth, damped mechanical operations rather than discrete jumps; furthermore, unconstrained focus adjustments risk rendering entire scenes completely defocused. InFlux++ Synth introduces a depth-aware bounded random walk to generate natural parameter trajectories. For LFL, target keyframe values are sampled within physical bounds (\([8\,\text{mm}, 100\,\text{mm}]\)) with bounded step increments \(\delta_{\max}\), smoothly connected via cubic Bézier interpolation. For LTO, unconstrained random walks would frequently produce completely blurred, unusable frames as scene depth shifts. The system resolves this by computing near and far visible scene depths (\(d_{\text{near}}, d_{\text{far}}\)) at each keyframe via ray tracing, then parameterizing focus distance through a bounded fractional factor \(\alpha \in [0, 1]\):

\[ d_{\text{focus}} = d_{\text{near}} + (d_{\text{far}} - d_{\text{near}}) \cdot \alpha \]

Executing a bounded random walk on \(\alpha\) guarantees continuous optical transitions while ensuring that a substantive portion of the 3D scene remains sharply in focus throughout complex camera motions.

3. Reverse Large-FSF Static Calibration: Inverting motion to calibrate expansive fields of view

Acquiring ground truth intrinsics for real-world zoom lenses requires building a dense Look-Up Table (LUT) over the (LFL, Focus Distance) parameter grid. In large field-of-view spatial footprint (Large FSF) regimes—which occur when the camera is zoomed out with large focus distances—the calibration target must be positioned meters away to remain in focus, requiring an impractically massive physical board that cannot be manually articulated across full rotational axes. Prior work attempted cumbersome drone-mounted boards. InFlux++ Real inverts this convention by fixing the calibration board and moving the camera. A \(5.45\,\text{m} \times 3.06\,\text{m}\) rigid screen displaying an array of AprilTags was installed across a multi-story lecture hall. Operators moved and tilted the camera across four discrete vantage positions (ground floor left/right and balcony left/right), readily sweeping the full FSF and exciting pitch, yaw, and roll axes against a static, stable reference target.

4. Online Distortion Injection & Viewpoint Normal-Depth Planning: Balancing optical aberrations with rich 3D parallax

To bridge the synthetic-to-real optical gap, InFlux++ Synth incorporates an on-the-fly data loading pipeline applying Brown-Conrady radial (\(k_1, k_2\)) and tangential (\(p_1, p_2\)) lens distortion sampled at peripheral control points. Trajectory synthesis builds upon Infinigen's RRT planner but integrates normal and depth-based viewpoint filtering: trajectories dominated by planar, textureless surfaces (such as blank walls) are pruned, and segment-level replanning enables rapid recovery without discarding global paths. In InFlux++ Real, video capture explicitly integrates wearable rigs, moving vehicle mounts, household tasks (cooking, laundry), and dynamic sports (basketball), introducing rich translational motion and substantial parallax.

Loss & Training

Finetuning experiments are conducted using AnyCalib, a state-of-the-art manifold-learning architecture for single-view and dynamic calibration. AnyCalib predicts dense FOV vector fields and distortion parameters. The network was finetuned on a representative subset of InFlux++ Synth for 15 epochs using AdamW, with model selection based on the lowest CFL percent error on the InFlux++ Real validation split.

Evaluation incorporates two crucial metric refinements: first, monotonic branch cutoff in EPE computation, which truncates the Brown-Conrady polynomial at radius \(r^*\) to discard unphysical non-monotonic mappings that spuriously fold far-off-axis points back into the sensor area; second, LUT-reliable masking, which performs leave-one-out (LOO) validation across LUT grid vertices and restricts evaluation to frames whose bounding grid vertices achieve LOO EPE recall@T px \(\ge 0.95\).

Key Experimental Results

Main Results

The table below compares baseline intrinsics estimation methods evaluated on the combined test splits of InFlux and InFlux++ Real. Metrics report recall across percentage error thresholds for focal lengths (\(f_x, f_y\) @10%), principal points (\(c_x, c_y\) @1%), and LUT-reliable endpoint error (EPE @10px, @50px, @300px).

Method	\(f_x\) @10% (%)	\(f_y\) @10% (%)	\(c_x\) @1% (%)	\(c_y\) @1% (%)	EPE @10px (%)	EPE @50px (%)	EPE @300px (%)
AnyCalib (original)	25.2	25.2	76.5	37.4	2.70	25.1	87.3
AnyCalib (InFlux++ Synth finetuned)	31.2	31.3	67.2	33.4	1.93	18.7	78.6
GeoCalib	19.7	19.6	82.3	40.9	0.85	16.7	75.4
UniDepthV2	14.0	13.1	45.4	9.14	0.57	6.14	63.0
WildCamera	12.9	13.0	8.91	10.8	0.17	4.26	59.1
Perspective Fields	7.65	7.53	2.97	3.09	0.13	0.81	23.7
DroidCalib	5.99	5.95	6.07	4.45	0.33	1.86	16.3
COLMAP	6.34	6.36	7.16	1.96	0.23	4.78	11.9

Ablation Study

Evaluation of AnyCalib before and after 15 epochs of finetuning on InFlux++ Synth, broken down across InFlux, InFlux++ Real, and Combined test sets:

Evaluation Set	Config / Model	\(f_x\) @1% (%)	\(f_x\) @10% (%)	\(f_x\) @20% (%)	\(c_x\) @1% (%)	\(c_y\) @1% (%)	EPE @50px (%)
InFlux	AnyCalib original	1.30	11.5	20.8	90.0	76.1	27.9
InFlux	AnyCalib finetuned	2.21	17.5	34.1	94.4	51.3	34.1
InFlux++ Real	AnyCalib original	3.36	28.2	45.7	62.3	24.2	17.9
InFlux++ Real	AnyCalib finetuned	4.00	34.2	54.0	72.6	18.2	24.3
Combined	AnyCalib original	3.00	25.2	41.3	76.5	37.4	25.1
Combined	AnyCalib finetuned	3.68	31.2	50.4	67.2	33.4	18.7

Key Findings

• Synthetic supervision consistently boosts focal length accuracy: Finetuning AnyCalib on InFlux++ Synth yields substantial gains across all \(f_x\) and \(f_y\) recall thresholds. On the combined benchmark, \(f_x\) @10% recall increases from 25.2% to 31.2%, and \(f_x\) @20% recall jumps from 41.3% to 50.4%, demonstrating that thin lens parameterization and breathing dynamics provide high-fidelity supervisory signals.
• Tension between loss objective and metric alignment: While focal length estimation improves across all splits, \(c_x\), \(c_y\), and overall EPE recall degrade under certain settings. AnyCalib's training objective minimizes FOV-field discrepancies rather than full reprojection EPE; synthetic distortion parameters near image margins amplify 2D endpoint errors when distortion modeling is slightly mismatched.
• Persistent difficulty of dynamic camera self-calibration: Classical multi-view geometry (COLMAP) degrades severely under continuous zoom and rapid translational motion (\(f_x\) @10% of 6.34%). Even with synthetic supervision, SOTA models achieve less than 2% recall at strict 10px EPE thresholds, highlighting dynamic intrinsics estimation as an open frontier.

Highlights & Insights

• Thin lens physics replacing heuristic augmentations: Rather than generating artificial zoom via panorama crops or pinhole scale shifts, grounding camera focal length in coupled LFL-LTO thin lens equations organically introduces lens breathing, offering an authentic optical learning signal.
• Inverted calibration kinematics: Inverting the traditional paradigm—moving the camera around a massive fixed lecture-hall AprilTag screen rather than wrestling with moving boards or drones—provides an elegant, robust methodology for calibrating large spatial footprints.
• Rigorous metric formulation: Correcting EPE evaluation by pruning unphysical non-monotonic polynomial mapping branches and enforcing leave-one-out LUT reliability guarantees metric fidelity and prevents artificial evaluation artifacts.

Limitations & Future Work

• Limitations acknowledged by authors: InFlux++ Synth does not yet model complex off-axis lens decentering or higher-order chromatic aberrations, and finetuned models exhibit slight sensitivity on lenses with substantial principal point offsets.
• Future directions: Existing baselines operate predominantly on individual frames. Developing recurrent or transformer-based architectures that exploit continuous temporal optical flow alongside geometric consistency will be vital for unlocking true video-rate dynamic self-calibration.

Related Work & Insights

• vs InFlux (NeurIPS 2025): InFlux pioneered real dynamic intrinsics benchmarking but lacked synthetic training data and featured mostly rotational camera motions in academic settings. InFlux++ Real expands real coverage with 514K+ frames across heavy translational motions, while InFlux++ Synth supplies the missing large-scale synthetic training supervision.
• vs AnyCalib (ICCV 2025): AnyCalib demonstrated strong single-view calibration through manifold learning on panorama crops. Finetuning on InFlux++ Synth enables AnyCalib to generalize effectively to physical video sequences with coupled zoom and focus transitions.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Physical thin lens breathing simulation and inverted large-FSF calibration methodology provide foundational conceptual contributions.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ 7 baseline evaluations, rigorous metric reformulation (monotonic cutoff & LOO reliability), and multi-benchmark cross-validation.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition of optical mechanics, detailed dataset curation workflows, and candid experimental analysis.
• Value: ⭐⭐⭐⭐⭐ Essential data and benchmark infrastructure enabling modern 3D vision (SLAM, 3DGS, NeRF) to operate reliably on dynamic-intrinsics videos.