WAFT-Stereo: Warping-Alone Field Transforms for Stereo Matching¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/princeton-vl/WAFT-Stereo
Area: 3D Vision
Keywords: stereo matching, feature warping, disparity classification, recurrent refinement, zero-shot generalization

TL;DR¶

WAFT-Stereo replaces cost volumes with a single disparity-classification initialization followed by recurrent regression driven by high-resolution feature warping, achieving 0.89% ETH3D BP-0.5-noc with synthetic-only training and 106 ms latency for 540p inputs on an L40 with DAv2-L and 5 iterations.

Background & Motivation¶

Stereo matching estimates horizontal disparity from rectified left and right images, allowing camera calibration to convert disparity into depth. Methods such as RAFT-Stereo and FoundationStereo usually construct a cost volume from both feature maps, query candidate matches, and recurrently update disparity. Storing multiple candidate locations makes this representation grow with the disparity range or local lookup window. Processing therefore commonly happens at 1/4 resolution, where small structures may already be lost.

The optical-flow model WAFT showed that a network can instead warp target features into reference coordinates using the current displacement and consume concatenated features without explicitly computing candidate matching costs. However, replacing its two-dimensional flow output with one-dimensional disparity is insufficient. Adjacent video frames typically have smaller motion, whereas high-resolution stereo pairs often contain displacements of hundreds of pixels. Regressing residuals from zero can trap large-disparity pixels in incorrect estimates that further iterations struggle to repair.

The paper retains classification-based initialization but separates it from cost volumes: first predict discrete disparity probabilities to establish coarse correspondence, then use warped features for continuous residual correction. Core Idea: classification handles large-range localization and regression handles precise correction; both can use standard networks and feature warping without a cost volume.

Method¶

Overall Architecture¶

The input is a rectified stereo pair, and the output is a continuous disparity field for the left image. Figure 3 first applies LoRA Feature Encoding, then Classification Initialization, and finally repeated Warping-Based Recurrent Refinement. The main model's 5 iterations comprise 1 classification step and 4 regression steps, not 5 regression steps plus classification.

The classifier and recurrent updater use the same architectural form, ViT-S + DPT, but the paper does not claim that their weights are shared. The classifier supplies a coarse estimate across the preset range. The regression stage maintains a hidden state and realigns right-image features using the latest disparity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Rectified stereo pair"] --> Encoder["LoRA Feature Encoding"]
    Encoder --> Classifier["Classification Initialization"]
    Encoder --> Refiner["Warping-Based<br/>Recurrent Refinement"]
    Classifier --> Refiner
    Refiner -->|Update disparity and hidden state| Refiner
    Refiner --> Output["Full-resolution disparity"]

Key Designs¶

1. LoRA Feature Encoding: adapt pretrained representations without a side U-Net

WAFT uses a small U-Net to adapt its pretrained encoder. WAFT-Stereo instead freezes the backbone, adapts it through rank-8 LoRA, and trains a DPT head to upsample low-resolution features. Removing the side U-Net reduces latency while retaining pretrained visual representations. Benchmark submissions use DepthAnythingV2-L; faster variants use DepthAnythingV2-S or B.

Features are warped at 1/2 of the input resolution rather than queried through a 1/4-resolution cost volume. High resolution here refers to where matching information is accessed, not to every Transformer layer operating directly over half-resolution pixels. The updater still uses patchification to control computation.

2. Classification Initialization: cover large disparities before regression

The classification head predicts probabilities over preset disparity locations. The default is \(B=40\) locations with maximum disparity \(D_{\max}=800\), uniformly spanning zero to that maximum. The target is not a one-hot label for the nearest candidate: distances between ground truth and candidate disparities define a soft distribution, supervised with soft cross-entropy. This preserves neighborhood relationships along the disparity axis instead of imposing abrupt discrete labels.

Soft-argmax converts the predicted probabilities into a continuous initialization, which then drives the first feature warp in the regression stage. The following expression reconstructs the summation in Section 3.3 from its accompanying definitions, because the cached equation has extraction damage:

\[ d_0=\sum_{i=0}^{B-1}\left(i\frac{D_{\max}}{B-1}\right)P_i. \]

The initialization need not provide immediate subpixel precision; it must avoid severe mistakes at large disparities. In Section 3.2, the classification first step obtains 0.89 px EPE on ETH3D training pixels above 40 px disparity, versus 1.11 px for the regression first step. Above 50 px, the values are 1.80 and 2.88 px. The authors also note that regression can have a better initial average error while leaving a subset of large-displacement pixels persistently wrong.

Classification is not another name for a cost volume. A standard ViT-S + DPT module directly predicts probabilities without first enumerating all left-right feature pairs into a correlation tensor. Replacing this initialization with cost-volume classification does not improve Table 5 results, supporting the value of the classification formulation itself.

3. Warping-Based Recurrent Refinement: read the current correspondence while preserving detail

At each iteration, the current disparity backward-warps right-image features through bilinear sampling. These features, the left-image features, and the hidden state enter the updater together. Rather than explicitly producing matching costs, the network interprets the relationship between aligned and misaligned features. Updated disparity determines the next sampling locations. Figure 4 contrasts reading features at the current correspondence with retaining a set of candidate matching costs.

The warping operator's computation and storage scale linearly with spatial resolution without a multiplicative disparity-search dimension. This property concerns the warping operator, not a claim that the entire network or classification head is independent of candidate count. Reliable large-disparity localization still depends on initialization, the training distribution, and the updater.

The recurrent updater uses ViT-S and DPT, with an \(8\times8\) patchifier before the ViT. To counteract detail loss in low-resolution processing, the high-resolution hidden-state skip connection in WAFT is replaced with 4 ResNet blocks. An MLP predicts the Mixture-of-Laplace parameters from the hidden state. Classification probabilities and regression-related output parameters are brought to input resolution through convex upsampling.

The improvements are not explained only by a larger backbone. Table 5 separately tests classification initialization, high-resolution blocks, and the regression loss using the same DAv2-S backbone. Warping accesses correspondence information, high-resolution blocks preserve detail, and the regression objective trains continuous updates; these serve distinct purposes.

Loss & Training¶

Initialization uses soft cross-entropy, while recurrent regression uses the Mixture-of-Laplace (MoL) loss. Instead of directly applying L1 to a point estimate, supervision incorporates a predicted distribution. The cache does not specify the full MoL parameterization or a numerical discount factor, so those settings are not supplied here. The total objective adds classification loss to discounted regression losses; notation below is reconstructed from the text and summation terms in Section 3.3:

\[ L_{\mathrm{total}}=L_{\mathrm{cls}}+\sum_{i=1}^{T-1}\gamma^{T-i}L_{\mathrm{MoL}}^{i}. \]

Here \(T\) includes the initial classification step, and \(\gamma\) is the discount factor. Main training uses approximately 7.7 million synthetic stereo pairs in SynLarge, drawn from SceneFlow, FallingThings, FSD, TartanAir, TartanGround, Spring, CREStereo, Sintel, Virtual KITTI 2, UnrealStereo4K, WMGStereo, and HR-VS. Training uses 480p random crops, batch size 32, learning rate \(5\times10^{-4}\), 400k steps, AdamW, and a OneCycle scheduler.

ETH3D submissions directly use the synthetic-only checkpoint. KITTI fine-tuning uses \(372\times1240\) crops, batch size 16, learning rate \(10^{-4}\), and 3k steps. Middlebury fine-tuning mixes real and synthetic data, using \(540\times960\) crops, batch size 16, learning rate \(2\times10^{-4}\), and 50k steps; its reported results must not be described as zero-shot.

Key Experimental Results¶

Main Results¶

BP-X is the percentage of pixels with disparity error above X pixels. D1 requires both absolute error above 3 px and relative error above 5% of ground-truth disparity. RMSE is root mean square error. The noc suffix covers non-occluded pixels and all covers all evaluated pixels. Lower is better throughout.

Dataset and source table	Training protocol and metric	WAFT-Stereo DAv2-L, 5	Comparison method	Comparison value
ETH3D test, Table 1	Zero-shot, BP-0.5-noc (%)	0.89	FoundationStereo, zero-shot	2.31
ETH3D test, Table 1	Zero-shot, BP-1-noc (%)	0.28	FoundationStereo, zero-shot	1.52
KITTI-2012 test, Table 2	Fine-tuned, BP-2-all (%)	1.47	MonSter++, fine-tuned	1.70
KITTI-2015 test, Table 2	Fine-tuned, D1-all (%)	1.28	MonSter++, fine-tuned	1.37
KITTI-2015 test, Table 2	Zero-shot, D1-all (%)	2.88	FoundationStereo, zero-shot	3.20
Middlebury test, Table 3	Fine-tuned, RMSE-all (px)	7.02	S2M2-XL	7.39
Middlebury test, Table 3	Fine-tuned, BP-2-all (%)	4.39	S2M2-XL	2.94

On ETH3D, 2.31 to 0.89 is approximately a 61% relative error reduction; 1.52 to 0.28 is the approximately 81% reduction. The abstract's 81% must not be attributed to BP-0.5. KITTI zero-shot D1-all decreases from 3.20 to 2.88, a 10% reduction calculated from the table, although the paper also uses a 9% description. Raw values are retained here to avoid conflating reporting conventions.

Ablation Study¶

Table 5 uses a synthetic subset of approximately 1.6 million pairs, 100k training steps, and a reduced maximum disparity of \(D_{\max}=320\). Other default training settings are 480p crops, batch size 32, and learning rate \(5\times10^{-4}\). Evaluation is zero-shot on ETH3D and Middlebury-Q training splits, not directly comparable to the test-set results above.

Configuration (original Table 5)	ETH3D BP-1-all (%)	Middlebury-Q BP-2-all (%)	MACs
DAv2-S, 1 classification + 3 regression steps, 4 high-resolution blocks, 40 bins, MoL	1.48	4.62	560G
0 classification + 4 regression steps	2.59	6.45	561G
4 classification + 0 regression steps	66.8	26.1	560G
Cost-volume classification initialization + 3 regression steps	1.53	4.74	561G
Remove high-resolution blocks	1.99	6.20	494G
5 bins	2.46	6.45	560G
Replace MoL with L1	2.44	6.39	560G
Switch backbone to DAv2-L, still 4 total iterations	0.80	2.63	2172G

Key Findings¶

At nearly identical MACs, classification plus regression clearly outperforms regression alone. Classification alone is much worse than the hybrid, showing the need for both coarse localization and continuous refinement.
More classification candidates are not universally better. With 80 bins, Table 5 reports 1.27% on ETH3D but 5.39% on Middlebury-Q, worse than the default 40-bin result of 4.62% there.
Table 6 profiles 540p inputs with BF16 on an NVIDIA L40: DAv2-S/4 takes 47 ms and DAv2-L/5 takes 106 ms, versus 708 ms for FoundationStereo, 195 ms for S2M2-XL, and 51 ms for BridgeDepth. The large model is not the fastest method; approximately 21 FPS refers to the small model.

Highlights & Insights¶

Classification gains can be tested separately from cost-volume gains. The nearly compute-matched cost-volume initialization does not outperform direct classification, helping distinguish a useful formulation from a customary implementation.
Half-resolution feature access is compatible with a standard Transformer operating after patchification. High-resolution ResNet blocks restore local processing, separating geometric information access from broader contextual computation.
Architecture should be evaluated together with data coverage. Strong generalization on diverse synthetic data and severe failure on a narrow synthetic dataset provide a more informative conclusion than claiming that removing cost volumes is always better.

Limitations & Future Work¶

The authors identify substantial left-right illumination differences in Middlebury's Classroom2E scene, which strongly affect aggregate BP-2. Despite the best RMSE-all, BP-2-all is 4.39%, behind S2M2-XL's 2.94%; not every error metric is state of the art.
In Table 4, DAv2-S/4 trained only on approximately 35k SceneFlow pairs reaches 71.0% ETH3D train BP-1-all, versus 0.86% with SynLarge. This exposes sensitivity to dataset bias. Comparisons involving different backbones also cannot isolate architecture quality.
Figure 6 shows roughly linear latency growth with iteration count because recurrent dependencies limit parallelism. Reducing the number of updates remains useful, and the large model's 106 ms result does not itself establish high-speed real-time deployment.
The setup implies dependence on rectified stereo inputs and a preset classification range. More illumination-robust features, broader disparity distributions, and adaptive stopping are plausible directions, but the paper does not evaluate them.

WAFT supplies the pure-warping optical-flow framework. This paper adds classification initialization for large stereo displacement and changes encoding and updating through LoRA and high-resolution ResNet blocks, rather than merely changing the output dimension.
RAFT-Stereo / FoundationStereo drive recurrent refinement through cost-volume queries. WAFT-Stereo preserves iteration but changes how matching information is obtained; it does not reject recurrent refinement itself.
BridgeDepth / S2M2 initialize correspondence through cost-volume classification or optimal transport. Direct classification provides a useful alternative here, although BridgeDepth remains faster than the large WAFT-Stereo model in Table 6.
SEA-RAFT provides relevant foundations for MoL regression supervision. WAFT-Stereo transfers this objective to disparity refinement and supports it through an L1 comparison.

Rating¶

Novelty: 4/5. The contribution combines classification initialization with a cost-volume-free warping framework; most individual building blocks are inherited.
Experimental Thoroughness: 4/5. Multiple benchmarks, architecture and data ablations, and profiling are provided, but illumination changes and small-data failures need deeper evaluation.
Writing Quality: 4/5. The architecture and ablations are clear, although some percentage claims require careful metric-level interpretation.
Value: 5/5. The method provides a competitive counterexample to the necessity of cost volumes in stereo matching, with public code and model access.