Skip to content

SynFlow: Scaling Up LiDAR Scene Flow Estimation with Synthetic Data

Conference: ECCV 2026
Paper: ECCV 2026 / project page
Code: https://huggingface.co/datasets/KTH/SynFlow
Area: Autonomous Driving
Keywords: LiDAR scene flow, synthetic data, sim-to-real transfer, zero-shot generalization, data engine

TL;DR

SynFlow builds a motion-oriented synthetic data engine on CARLA and uses it to produce SynFlow-4k, 4,000 sequences (~940k frames) with noise-free per-point ground truth; a model trained on synthetic supervision alone transfers zero-shot to nuScenes/TruckScenes/Aeva, beats the fully supervised ∆Flow baseline (0.402) on TruckScenes by 31.8% at 0.274, and exceeds a from-scratch baseline trained on 4x the label budget after fine-tuning on just 5% of real labels.

Background & Motivation

LiDAR scene flow estimates the 3D displacement of every point between two consecutive scans. Being category-free, it offers a more direct motion-centric representation for downstream planning and interaction than the detect-then-track pipeline. Its progress, however, has been throttled by supervision: densely and accurately annotating 3D motion in real driving logs is practically infeasible at scale, which pushed recent work toward self-supervision on large collections of unlabeled real sequences — SeFlow constructs pseudo-labels from rigidity clustering, TeFlow adds multi-frame supervision, VoteFlow emphasizes local rigidity, and other lines rely on cycle consistency or neural Eulerian flow fields. All of these proxy signals rest on geometric consistency assumptions, so they are inherently noisy and under-constrained under sensor sparsity, measurement noise, and non-rigid motion. More tellingly, empirical evidence shows that simply scaling up unlabeled real data yields diminishing returns, leaving a persistent gap to fully supervised methods.

The situation is awkward: dense 3D motion labels are too expensive to buy, while the proxy supervision that can be extracted from real data for free is structurally unable to match them. This paper's judgment is that instead of further patching proxy signals on real data, the source of supervision should be replaced altogether — a simulator already knows the exact rigid-body pose of every agent at both timesteps, and such supervision is noise-free and unaffected by occlusion. Yet existing synthetic LiDAR pipelines (SHIFT, CarlaScenes, ReSimAD, MegaSynth, among others) were designed for detection, segmentation, and domain transfer, optimizing for semantic realism or sensor-noise alignment, while image-based synthetic scene flow datasets such as FlyingThings3D pursue visual realism. In other words, no prior work has designed synthetic data for dense kinematic complexity itself.

This paper's angle follows from the observation that scene flow learning depends on physically consistent object kinematics rather than texture or semantics, so 3D motion tasks exhibit a different sim-to-real behavior from appearance-dominated tasks: a simulated environment with accurate rigid-body states can already provide transferable supervision. SynFlow therefore shifts generation from "does it look like a real sensor" to "is the motion rich enough," and uses three procedural policies to actively control road topology, speed regimes, and multi-agent interaction. Core idea: move the supervision source for LiDAR scene flow entirely into simulation and saturate kinematic complexity with motion-oriented procedural generation (4k sequences / 940k frames, a 34x scale-up over nuScenes' annotated volume), yielding a domain-invariant motion prior that transfers zero-shot across sensors and serves as a high-quality initialization for few-label fine-tuning.

Method

Overall Architecture

SynFlow has two halves, neither of which contains any domain adaptation module. The first is the data engine: given a CARLA town topology and a sensor configuration, three sampling policies answer where to drive, how fast, and how the surrounding agents behave; the engine then rolls out in deterministic synchronous mode at a fixed step size and writes each frame's LiDAR point cloud, ego pose, aligned scene flow ground truth, validity mask, and per-point instance metadata into HDF5. The second half is the training and evaluation protocol: a ∆Flow backbone is pre-trained with supervised learning on this synthetic supervision, after which it is either evaluated zero-shot on real benchmarks or fine-tuned on only 5%-20% of the real labels.

The output scale of this pipeline is what "scaling up" in the title means concretely: 4,000 sequences and 939,083 annotated frames, a 34x increase over nuScenes' 137,575 labeled training frames and 46x over TruckScenes' 101,902, at a generation cost of only 3-6 minutes per sequence (Intel i7-12700KF plus a single RTX 3090). The sensor setup deliberately mixes 32-beam and 64-beam configurations so that the learned motion prior carries no bias toward a particular beam layout.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["CARLA town topology<br/>+ sensor config"] --> B["Topological discretization<br/>greedy route bank"]
    B --> C["Speed-regime coverage<br/>road speed limits + highway towns"]
    C --> D["Multi-agent interaction<br/>density perturbation + deadlock release"]
    D --> E["Synchronous rollout<br/>Δt = 0.1 s, 32/64-beam"]
    E --> F["Noise-free label export<br/>instance voting + rigid-body poses"]
    F --> G["SynFlow-4k<br/>4k sequences / 940k frames"]
    G --> H["Supervised pre-training<br/>∆Flow backbone"]
    H -->|zero-shot| I["Real-world benchmark evaluation"]
    H -->|5-20% label fine-tune| I

Key Designs

1. Topological discretization policy: treating the route bank as a coverage problem

Random route sampling makes the model see the same few road types over and over, while long-tail geometries such as roundabouts and complex junctions are equally rare in real datasets — so the model never learns the motion contexts that go with them. SynFlow discretizes a town's drivable topology into fine-grained lane segments (road_id, section_id, lane_id), maintains a set \(C\) of already-covered segments, and expands a route bank by greedy search: a candidate route \(R\) is accepted only if it contributes enough previously unvisited segments, \(|R \setminus C| > \tau\).

Every accepted route must therefore "pay for itself" with new geometry, which forces the simulator onto structures such as Town04's loop highway and Town06's complex junctions instead of cycling through a few arterials. This policy delivers the largest individual gain in the ablation (1k sequences, Aeva zero-shot, Dynamic metric 0.346 → 0.330), showing that the generalization of a motion prior depends not only on the number of samples but on the geometry distribution actually sampled.

2. Speed-regime coverage policy: widening the displacement support via road-type speed limits

Real scenes span an enormous range of displacement magnitudes: on a highway a single point can move several meters between consecutive frames, while stop-and-go urban traffic produces near-zero motion. If synthetic data were almost entirely low-speed urban traffic, the model would fail systematically on the high-velocity tail — one of the reasons self-supervised methods perform so poorly on long-haul datasets such as TruckScenes. Rather than hand-tuning a target speed per scene (expensive and poorly covering), SynFlow lets the road-type-dependent speed limits and CARLA's Traffic Manager produce the distribution: scenes are initialized by sampling valid waypoints and letting vehicles drive at the limit of their current road segment, and towns containing highway-like stretches (Town04's loop highway, Town06's long multi-lane roads) are explicitly included in the generation pool.

High-displacement samples then appear naturally without any per-scene parameter search, complementing low-speed stop-and-go urban traffic so that both ends of the displacement distribution are covered. Adding the highway towns in the ablation further reduces the Dynamic metric from 0.330 to 0.319, with the gain concentrated in the foreground-dynamic (FD) term of Three-way EPE (50.21 → 48.17 cm) — exactly the large-displacement prediction for moving objects.

3. Multi-agent interaction policy: buying hard frames with interaction density and deadlock handling

The hard part of scene flow is relative motion, and the vast majority of real log frames show near-constant-velocity straight-line motion; what is genuinely difficult — merges, overtakes, emergency braking — is both rare and usually accompanied by occlusion. During rollouts, SynFlow perturbs local traffic density and NPC behavior policies so that such interaction-heavy situations occur far more often than they would naturally, enriching the supply of non-linear relative motion.

The pipeline also adds a lightweight deadlock-resolution rule: if the ego vehicle stays stationary too long because an intersection is blocked, the engine overrides the local traffic signal to release the blockage and resumes the rollout. The rule looks minor, but what it protects is the effective yield of supervision — stationary frames carry almost no scene flow signal, and a fully gridlocked sequence is wasted generation time. This policy therefore works by raising the share of hard frames rather than the raw frame count.

4. Noise-free label export: per-point ground truth from instance voting and rigid-body poses

Scene flow ground truth requires each point's position at time \(t+1\), but points in the scan at \(t+1\) do not correspond one-to-one to those at \(t\) (occlusion, sparsity, non-repetitive scanning), so the labels cannot be assembled by registration. The simulator does expose each agent's exact rigid-body pose \(\mathbf{T}_k^t \in SE(3)\), but its actor IDs and the per-point instance tags of the LiDAR points are two different numbering systems that cannot be matched directly. SynFlow solves this in two steps: at time \(t\), take the 3D bounding box of agent \(k\) and resolve its tag by majority voting over the per-point instance tags inside the box, which yields the set of points belonging to that agent; then move those points with the agent's two poses.

\[ \hat{u}_k^{t} = \arg\max_{u}\sum_{i \in \mathrm{bbox}(k,t)}\mathbf{1}[u_i = u],\qquad \mathcal{M}_k^{t}=\{\,\mathbf{p}_i \in \mathcal{P}_t \mid u_i = \hat{u}_k^{t}\,\} \]
\[ \mathbf{p}_i^{\star} = \mathbf{T}_k^{t+1}\,(\mathbf{T}_k^{t})^{-1}\,\mathbf{p}_i,\qquad \mathbf{f}_i = \mathbf{p}_i^{\star} - \mathbf{p}_i \]

The first step aligns the point-tag space with the actor space; the second uses \((\mathbf{T}_k^{t})^{-1}\) to map a point into the agent's local body frame and \(\mathbf{T}_k^{t+1}\) to place it back into world coordinates at \(t+1\), so the difference is the ground-truth flow (formulas reconstructed from the paper's textual description; ⚠️ refer to the original paper for exact notation). The value of this construction is that the labels come from the simulator's internal physical state rather than from estimation, hence noise-free, and that they naturally cover points occluded at \(t+1\) — precisely the cases that are most expensive to annotate in the real world and least reliable under self-supervised proxies.

A Worked Example

Take Town06 (dominated by complex junctions) and walk through one sequence. The greedy search selects a route that contributes enough unvisited lane segments; the ego vehicle is spawned at the route start with a 64-beam LiDAR, heterogeneous NPCs (vehicles and pedestrians) are spawned in its local neighborhood and controlled by the Traffic Manager, and simulation advances synchronously at \(\Delta t = 0.1\) s.

At frame \(t\), a laser point lands on a truck ahead that is decelerating: majority voting inside the box assigns it to that truck, so the truck's two poses are used for the rigid-body transform. If the truck is moving at roughly 12 m/s (about 43 km/h) at that moment, its displacement over 0.1 s is about 1.2 m, and that vector becomes the ground-truth flow for this point; a point on a pedestrian on the crosswalk in the same frame receives a much shorter vector. If the ego vehicle is blocked at a junction for longer than the threshold, the deadlock rule releases the signal so the rollout continues instead of degenerating into stationary frames for the rest of the sequence. Every frame is written to HDF5 together with a mask marking which points provide valid supervision in that frame.

Loss & Training

The default backbone is ∆Flow: each 5-frame sequence is voxelized at 0.15 m within a 38.4 m range, temporal context is aggregated, a sparse 3D convolutional network (MinkUNet-style) extracts motion-aware features, and the voxel features are interpolated back onto the points of frame \(t\) to decode the forward scene flow. The loss follows ∆Flow's supervised scene flow objective with its motion-awareness, category-balanced, and instance-consistency terms (⚠️ the paper does not modify these terms; refer to the original ∆Flow paper for their exact form). Because supervision comes directly from simulated ground truth, no geometric-consistency or cycle-constraint self-supervised term is needed.

Optimization uses 15 epochs of Adam with a learning rate of 0.002 and a total batch size of 20, with only standard augmentations (z-height perturbation, random x-y flips) and no domain adaptation whatsoever. There are two usage modes: evaluate the synthetic pre-trained weights zero-shot on real benchmarks, or pre-train on synthetic ground truth and continue supervised fine-tuning on 5%/10%/20% subsets of the target real benchmark.

Key Experimental Results

Main Results

The primary metric is Dynamic Bucket-Normalized EPE: points are first bucketed by motion speed, and errors are then normalized within each bucket by the error of an ego-motion-only baseline. The Ego Motion Flow row is therefore exactly 1.000, and 0.242 means the error drops to 24.2% of that baseline (⚠️ see the Khatri et al. reference cited in the paper for the bucketing and normalization details). The secondary metric is Three-way EPE, reported in centimeters for foreground dynamic (FD), foreground static (FS), and background static (BS).

Method Supervision nuScenes Mean ↓ nuScenes CAR TruckScenes Mean ↓ TruckScenes CAR
Ego Motion Flow – 1.000 1.000 1.000 1.000
SeFlow 100% unlabeled real 0.544 0.396 0.681 0.494
VoteFlow 100% unlabeled real 0.538 0.355 0.680 0.517
SeFlow++ 100% unlabeled real 0.509 0.327 0.653 0.519
TeFlow 100% unlabeled real 0.395 0.303 0.425 0.254
DeFlow 20% labeled real 0.314 0.163 0.570 0.180
Flow4D 20% labeled real 0.279 0.204 0.456 0.176
∆Flow 20% labeled real 0.216 0.138 0.402 0.196
SynFlow-4k (zero-shot) 0% real, synthetic only 0.242 0.177 0.274 0.109
SynFlow-4k + FT 20% labeled real 0.157 0.110 0.266 0.082

The zero-shot row never sees any real data: 0.242 on nuScenes, 38.7% better than the best self-supervised baseline TeFlow (0.395) and already approaching the supervised ∆Flow trained on 20% of the labels (0.216); on TruckScenes it reaches 0.274, 35.5% better than TeFlow (0.425) and 31.8% better than the supervised ∆Flow (0.402). Winning on TruckScenes — a dual 64-beam platform dominated by large commercial vehicles that was never seen during pre-training — indicates that what transfers is kinematics rather than sensor characteristics.

In terms of label efficiency, fine-tuning on 20% of real labels brings nuScenes down to 0.157 (27.3% better than ∆Flow's 0.216) and TruckScenes to 0.266 (33.8% better).

Training scheme Real labels nuScenes Mean ↓ TruckScenes Mean ↓
∆Flow (supervised from scratch) 20% 0.216 0.402
SynFlow-4k (zero-shot) 0% 0.242 0.274
SynFlow-4k + FT 5% 0.201 0.271
SynFlow-4k + FT 10% 0.175 0.261
SynFlow-4k + FT 20% 0.157 0.266

Ablation Study

The generation-policy ablation is run with 1k sequences and evaluated zero-shot on Aeva (P1 = topology coverage, P2 = speed regime / highway towns):

P1 Topology P2 Speed Dynamic Mean ↓ Three-way EPE Mean ↓ FD (cm) ↓
– – 0.346 18.96 52.34
– ✓ 0.331 18.45 51.25
✓ – 0.330 17.89 50.21
✓ ✓ 0.319 17.11 48.17

Backbone agnosticism (trained on SynFlow-4k, zero-shot on Aeva): the Ego Motion Flow baseline that applies odometry-only compensation gives a Dynamic metric of 1.000 (Three-way EPE 44.31 cm), while the three learned backbones reach DeFlow 0.426, Flow4D 0.308, and ∆Flow 0.288 — the synthetic supervision is not tuned to one architecture.

Synthetic-real complementarity (zero-shot on Aeva): UniFlow pre-trained on a massive union of real data (Argo-v2 + nuScenes + Waymo) gives 0.303 alone, SynFlow-4k alone gives 0.288, and combining both further reduces the error to 0.263. The most striking case is the pedestrian category, which improves from UniFlow's 0.398 to 0.251 when combined (about 37%).

Key Findings

  • Topology coverage is the single most important design: on a 1k-sequence budget it contributes 0.346 → 0.330 by itself, more than adding highway towns alone (0.331), and the two stack to 0.319 — the quality of a motion prior is highly sensitive to which geometries were sampled.
  • The speed-regime gain lands specifically on large dynamic displacements: adding highway towns moves the Dynamic metric by only 0.011, yet the FD term of Three-way EPE drops from 50.21 to 48.17 cm, exactly the fast-moving objects — which also explains why synthetic pre-training pays off most on the long-haul highway benchmark TruckScenes.
  • Synthetic scale yields the steepest gains up to 2k and then saturates: the 1k → 2k step helps most, after which accuracy stabilizes, leading the authors to conclude that the core distribution of rigid-body kinematics is essentially covered by 4k. By category, vehicles benefit most (TruckScenes zero-shot CAR 0.109, better than its own supervised baseline mean).
  • Synthetic and real data are orthogonal rather than substitutable: UniFlow supplies real sensor characteristics (beam layout, range-dependent sparsity, measurement noise) while SynFlow-4k supplies dense kinematics for rare interactions; the gain from combining them shows that even mega-scale real data cannot make up for the motion density of small dynamic agents such as pedestrians.
  • Minority categories and Three-way EPE expose the real gap: on TruckScenes the zero-shot Three-way EPE mean is 25.25 cm versus 6.75 cm after fine-tuning and 7.28 cm for supervised ∆Flow — the 31.8% lead holds under the Dynamic Bucket-Normalized metric, while in centimeter-level EPE the zero-shot model is still far behind supervised baselines. PED/VRU categories (0.467/0.300) are also clearly weaker than CAR (0.109): pedestrian kinematics is the hardest part of the sim-to-real transfer.
  • Fine-tuning is slightly non-monotonic: on nuScenes, 5% of labels (0.201) already beats the from-scratch ∆Flow trained on 20% (0.216), and 10% → 20% continues down to 0.175 → 0.157; on TruckScenes, however, 20% (0.266) is marginally worse than 10% (0.261), suggesting that at a 20% budget TruckScenes fine-tuning has saturated or begun to overfit.

Highlights & Insights

  • Replacing "does it look like a real sensor" with "is the motion rich enough": this is the most valuable judgment in the paper. The authors argue that 3D motion tasks have a different sim-to-real behavior from appearance-dominated tasks — scene flow depends on physically consistent rigid-body kinematics rather than texture — which is why it transfers across sensors with no domain adaptation at all, a claim the zero-shot results substantiate. The same reasoning should transfer to any task whose supervision is a geometric or physical quantity rather than an appearance (optical flow, depth, odometry).
  • Ground truth comes from simulator states rather than registration, and no domain adaptation is used: most sim-to-real work spends much of its effort on appearance alignment or adversarial adaptation, whereas SynFlow has no such component — when the supervision itself is clean enough, the domain gap can be absorbed by the nature of the task instead of by an alignment module.
  • Majority voting inside the bounding box to align instance tags: a small but very practical trick — the simulator's actor IDs and the point cloud's instance tags are two different numbering systems that poses alone cannot connect; voting inside the 3D box bridges them cheaply and robustly, and any pipeline generating instance-labeled data with CARLA or Isaac Sim can reuse it directly.
  • The deadlock-release rule protects the effective yield of supervision: without it, gridlocked sequences degrade into long stretches of stationary frames — the data volume looks large while its supervisory value approaches zero. It is a reminder that the metric for a generative dataset should be the number of effective dynamic frames, not the total frame count.
  • Treating synthetic data as an orthogonal prior rather than a cheap substitute: combining with UniFlow improves the pedestrian category by about 37%, which is a much stronger statement than "synthetic data is usable too" — it shows that what real data lacks is the motion density of long-tail interactions, exactly what simulation is best at supplying.

Limitations & Future Work

  • Generation and training form an open loop: the authors acknowledge a generate-then-train paradigm in which the data distribution is fixed by predefined policies and never adapts to the model's learning state. A closed-loop or cascade framework is the natural fix: failure cases (specific occlusion patterns, rare speeds) would trigger targeted re-simulation of hard examples adversarial to the current model, improving data efficiency and enabling continuous improvement.
  • Non-rigid agents such as pedestrians remain the weak spot: zero-shot PED/VRU on TruckScenes (0.467/0.300) is far worse than CAR (0.109), and the Aeva complementarity experiment shows pedestrians are the one category that benefits most from synthetic data — which conversely means the simulated pedestrian motion model deviates most from reality. Adding articulated pedestrian motion or upweighting VRU sampling would be a low-cost next step.
  • Zero-shot does not lead in centimeter-level EPE: on TruckScenes the zero-shot Three-way EPE is 25.25 cm against 7.28 cm for the supervised baseline, roughly a 3.5x gap. The headline Dynamic Bucket-Normalized metric normalizes within speed buckets, weighting slow and static targets heavily, which masks the disadvantage in absolute error; readers should not read "31.8% over SOTA on TruckScenes" as an overall lead in absolute accuracy.
  • Noise-free simulated point clouds may make the prior optimistic: real LiDAR has range noise, weather attenuation, and point distortion from ego motion during the sweep, whereas both the points and the labels here come from clean simulation. If a downstream task needs uncertainty estimates or high-precision registration, such an overly clean prior may underestimate real noise; adding a sensor-noise and downsampling simulation layer would make a valuable control experiment.
  • The 34x scale advantage needs an apples-to-apples reading: 34x and 46x are frame-count ratios, but the synthetic sequences come from a limited set of CARLA towns (with Town12 deliberately subsampled), so the diversity of underlying scene layouts does not grow as linearly as the frame count suggests. The decomposition of "scale" versus "diversity" is not fully separated here.
  • Validated on a single simulator: the authors propose extending the approach to Isaac Sim for embodied manipulation and similar domains, but there is currently no cross-simulator evidence, so how much the motion prior depends on a particular physics engine remains unknown.
  • vs SeFlow / TeFlow / VoteFlow (self-supervision on unlabeled real data): they invest in squeezing proxy supervision out of geometric consistency, at the cost of noisy, under-constrained signals, and the paper's own evidence shows diminishing returns from adding more unlabeled data. SynFlow replaces the supervision source outright, substituting simulator ground truth for proxy signals; its zero-shot result on TruckScenes (0.274) already beats the best of those methods, TeFlow (0.425) — the difference lies in supervision quality, not in the model.
  • vs UniFlow (pre-training on a massive union of real data): UniFlow trades real-data scale for generalization while SynFlow trades simulated motion density for it. Their complementarity (0.303 and 0.288 → 0.263 combined) shows these are two orthogonal axes, suggesting future foundation models need both real sensor density and simulated motion density rather than one or the other.
  • vs ReSimAD / SHIFT / CarlaScenes (synthetic driving data): they target semantic realism, environment diversity, or sensor domain alignment for detection, segmentation, and domain adaptation; ReSimAD follows a reconstruct-real-then-simulate-the-target-domain route, whereas SynFlow generates purely procedurally without reconstructing any real scene, and optimizes for dynamics rather than appearance.
  • vs FlyingThings3D (synthetic scene flow for images): it likewise renders geometry to obtain dense motion ground truth, but targets stereo images and emphasizes visual realism; SynFlow moves the idea to LiDAR and additionally solves two problems — sensor-beam diversity and the alignment of simulator actor states into per-point labels.
  • vs 3DSFLabelling (pseudo auto-labelling): that line produces pseudo-labels on real point clouds with a model, so errors propagate with the model; SynFlow's labels come from the physics engine and depend on no model, at the cost of only covering what simulation can generate.

Rating

  • Novelty: ⭐⭐⭐⭐ The first synthesis pipeline designed specifically for LiDAR scene flow; the "motion-oriented rather than appearance-realistic" framing and the domain-adaptation-free zero-shot transfer are convincing, though each individual generation policy is a fairly standard procedural sampling device.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Three real benchmarks (nuScenes / TruckScenes / Aeva) across three supervision regimes (self-supervised / supervised / synthetic zero-shot), plus a scaling curve, policy ablation, backbone-agnosticism study, and a complementarity experiment against real-data pre-training.
  • Writing Quality: ⭐⭐⭐⭐ The motivation chain is clear and the claims line up with the results; it loses a point because the "31.8% over SOTA" framing on TruckScenes can obscure that zero-shot still trails badly in centimeter-level EPE, and because the 34x scale-up does not separate frame count from scene diversity.
  • Value: ⭐⭐⭐⭐⭐ The pipeline and dataset are open-sourced, offering a scalable alternative supervision source for the long-standing bottleneck of unobtainable dense 3D motion annotations, and the complementarity with real-data pre-training has direct engineering implications for autonomous driving foundation models.