Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching¶

Conference: ECCV 2026
Paper: ECCV
Code: https://github.com/qrzyang/Pseudo-Stereo
Area: 3D Vision
Keywords: self-supervised stereo matching, occlusion handling, pseudo-stereo inputs, photometric consistency, disparity estimation

TL;DR¶

The paper replaces one view of the input pair with a pseudo-view rendered by the model currently being trained, decoupling the network's inputs from the photometric feedback so that occlusion falls on either side of an occluder with equal probability; combined with a rendering-derived occlusion mask and a fully pseudo-stereo input strategy, it lowers D1-all on the KITTI 2015 test set from the previous best of 6.11% to 4.39% and cuts the D1 error inside occluded regions from 20.82% to 9.05%.

Background & Motivation¶

Self-supervised stereo matching substitutes photometric consistency between views for expensive disparity ground truth. It should have advanced alongside supervised methods, yet it hit a bottleneck early on occlusions: the photometric-consistency assumption simply does not hold in occluded regions, so reprojecting one view by the disparity necessarily produces large color discrepancies and back-propagates pure noise. This erroneous feedback is large-scale, persists throughout training, and does not disappear with changes in network architecture — it is the dominant reason self-supervised stereo matching has long been unable to match supervised accuracy. The authors' key observation is disarmingly simple: for any occluder, a reliable feedback signal can only come from the side that is not occluded. The problem is that in the standard paradigm the network can never access that side.

Prior solutions fall roughly into two camps. One identifies occluded regions and removes them from the photometric loss (the various occlusion-mask methods), at the cost of leaving an information void that must then be filled by edge-aware smoothness or by regularity between the occluded region and its neighborhood's disparity — in essence all of them assume the occluded area is strongly correlated with its immediate surroundings, an assumption that frequently fails in real scenes. The other camp tries to source information elsewhere: flip augmentation attempts to borrow from symmetric regions, but the input images and the feedback images are the same pair and the occlusion direction is geometrically locked, so after flipping the occlusion still lands on the same side of the occluder. The paper's ablation (Tab. 4) shows naive flip and the baseline are almost identical (D1-all 5.61 vs 5.67), confirming that this route is a dead end. Methods that introduce a third viewpoint (a trinocular camera, or a novel view rendered with NeRF) are conceptually correct — take feedback from an auxiliary viewpoint where the occluded region is visible — but they either require dedicated symmetric hardware or need an additional scene reconstruction, and therefore cannot be applied to conventional binocular datasets. That contradicts the very point of self-supervision, which is to train directly on any target scene.

If valid feedback exists only on one side of an occluder and that side is geometrically fixed, then patching the fixed side over and over is not the answer; instead, make the occlusion itself no longer fixed. Core idea: replace one view of the input pair with a pseudo third-view image rendered by the network itself, thereby fully decoupling the network inputs from the photometric feedback — inputs are (real image, pseudo image) while feedback uses only the original images — turning the static defect of "occlusion is always pinned to one side of the occluder" into a dynamic process in which occlusion appears on either side with equal probability, so that both sides of an occluder can supply correct supervision.

Method¶

Overall Architecture¶

The problem this paper attacks is that occluded regions never receive a correct supervision signal during self-supervised training. Its approach splits each training iteration into two independent paths: use a pseudo image as the input, and use real images to compute the feedback. Given a real stereo pair, the flow is as follows. First, a gradient-free forward pass of the model currently being trained yields the disparity aligned with the right view; that disparity renders the right view into a pseudo image. The network's actual input thus becomes the asymmetric combination (right view, pseudo image), and it outputs a disparity aligned with the right view. Meanwhile the feedback path uses only original images: the real left view is reprojected into the right view's frame using the just-predicted disparity and compared photometrically against the real right view. Pixels deemed invalid during rendering — those falling outside the image boundary or occluded by other pixels — serve as the occlusion mask, and gradients at those locations are detached. An edge-aware smoothness term is finally added to form the total loss. The whole chain is self-bootstrapping: the more accurate the model, the closer the rendered pseudo image is to a real image, and the better the binocular information the network can exploit; the left input and the feedback images are always real and reliable, providing a stable anchor for that bootstrapping. The scheme is active only during training — no pseudo image is generated at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real stereo pair I^L, I^R"] --> B["Pseudo-stereo inputs<br/>real image + pseudo image"]
    B -->|Fully pseudo-stereo inputs| C["Pseudo-image generation<br/>render → pad → wider crop"]
    C --> D["Stereo matching network<br/>disparity in the current view"]
    D --> E["Occlusion estimation & gradient detaching<br/>render-invalid pixels don't back-propagate"]
    D --> F["Photometric feedback<br/>reproject the original opposite view"]
    E --> G["Loss L_p + λ L_s"]
    F --> G
    G -.->|gradient-free forward, re-render next round| C

Key Designs¶

1. Pseudo-stereo inputs: fully decoupling the network inputs from the photometric feedback

In the original paradigm the input pair and the feedback pair are the same two images: the network takes \((I^L, I^R)\) and outputs the disparity \(D^L\) aligned with the left view, and \(D^L\) then reprojects \(I^R\) back for a photometric comparison against \(I^L\). Inputs and feedback are locked into one geometric relation, so occlusion can only arise on one fixed side and the network permanently lacks information from the other side. This paper separates the two. The currently training model first estimates the disparity \(D^R\) aligned with the right view, from which a pseudo image \(\widetilde{I}^R\) is rendered. The network input becomes \((I^R, \widetilde{I}^R)\) and it outputs the disparity aligned with the right view. Feedback, by contrast, never touches the pseudo image and uses only original images: the real \(I^L\) is reprojected into the right view's frame by \(D^R\) to give \(I^L_{warp}\), which is compared photometrically against the real \(I^R\).

The crux is the direction reversal induced by swapping viewpoints: when the network's left input is \(I^L\), the occluded area appears to the left of the occluder; once the left input becomes \(I^R\), the occluded area moves to the right. Because the two configurations alternate, both sides of an occluder get a chance to be the side that is unoccluded and can supply correct feedback. It matters that the decoupling is one-directional in a specific sense — the feedback end always uses original images only, so the quality of the pseudo image never contaminates the trustworthiness of the supervision signal. That is precisely what makes the self-bootstrapping process stable.

2. Pseudo-image generation: a single gradient-free simplified rendering that makes the pseudo image realistic enough

Where the pseudo image comes from is the first hurdle. Mature novel-view synthesis methods (NeRF, 3DGS and the like) need additional data to train the synthesis model itself, importing external priors that conflict with the self-supervised premise; and the warp operation already available in a stereo pipeline is unusable here, because warping requires a disparity map aligned with the target view, whereas no such disparity map exists before the pseudo image is generated. The authors' substitute is a simplified rendering: each pixel of the right view is relocated backwards by its own disparity, and if several pixels land on the same coordinate only the one with the largest disparity is kept, exactly matching the occlusion behaviour (z-buffering) of rendering.

The result is an image with missing content: small-scale holes plus a large missing band along the right edge. To make the pseudo image as close to a real image as possible, the small holes are filled with the mean of adjacent non-hole pixels, and the right edge is handled by a "wider generation" strategy — render a wider image than the input and crop it back to size, which removes the large missing region at the border. This step is not cosmetic: the more realistic the pseudo image, the harder it is for the network to classify inputs by recognizing it as fake, and that recognition is exactly the root of the overfitting problem addressed below. Another easily overlooked detail is that pseudo-image generation costs only a single gradient-free forward pass of the current model: no extra GPU memory, and no gradient flows back along the rendering path.

3. Occlusion estimation and gradient detaching: using the rendering process itself as the occlusion mask

The pseudo-input strategy alone already beats the standard paradigm clearly, but it carries an internal contradiction: the strategy guarantees a 50% probability of correct information on the occluded side, which also means a 50% probability of incorrect information, so the network simultaneously minimizes two mutually conflicting losses and ends up at a compromise disparity in occluded regions. The magnitude problem makes this worse — the gradient of the photometric loss is negatively correlated with image similarity, so as training progresses and overall similarity rises the global average gradient approaches zero, while occluded regions still produce large color discrepancies even with the correct disparity because no corresponding pixel exists. Those unremovable erroneous gradients therefore take up an ever larger share of the total gradient.

The fix lets the rendering process hand over the answer for free: pixels judged invalid during rendering — those falling outside the image boundary or occluded by other pixels — are the occluded pixels, and setting their mask \(\widetilde{O}_{ij}\) to 0 detaches their gradients. Compared with the customary left-right consistency check, this requires no additional inference of a disparity map for the opposite view (halving that cost) and avoids the classic failure mode in which an incorrect disparity happens to satisfy the consistency check, which is most prevalent at object boundaries — because it uses information produced by the current view's own rendering rather than indirect evidence inferred from another viewpoint. In the ablation, adding this component brings a clear gain (Tab. 3: row (d) reaches 4.08% versus 4.99% for the pseudo-input-only configuration).

4. Fully pseudo-stereo inputs: making the network unable to tell real from fake, killing overfitting at the root

With the occlusion mask added, accuracy improves further, but long training runs expose a hazard: accuracy peaks and then slowly declines. Self-supervised training has no labels and therefore no validation set, so early stopping is unavailable and stable convergence must be guaranteed instead. The authors trace the root cause to the fact that in the "one real view, one pseudo view" configuration, which images are used directly determines which side the occlusion appears on. The network therefore only needs to learn to tell whether an input is a pseudo image, and it can converge to different solutions for the two input types — the design intent of "capturing information from both sides of occluders with equal probability" is completely bypassed, and the network memorizes the data instead of learning geometry.

If the network can tell them apart, remove the tell: the fully pseudo-stereo input strategy switches the inputs to the two fully pseudo combinations \((I^L, \widetilde{I}^L)\) and \((I^R, \widetilde{I}^R)\), each being "a real view as left input plus its corresponding pseudo image as right input," so the network loses any distinguishing feature to latch onto. The photometric loss then becomes two symmetric terms weighted by \(p = 0.5\):

\[L_p=\frac{1}{N}\sum_{ij}\Big(p\cdot pe\big(I^{R}_{ij},\ I^{L}_{warp,ij}\big)+(1-p)\cdot pe\big(I^{L}_{ij},\ I^{R}_{warp,ij}\big)\Big)\widetilde{O}_{ij},\qquad p=0.5\]

⚠️ Equations (1) and (3) of the original paper are badly corrupted in the PDF text extraction; the line above restates their structure from the paper's prose, namely two directional terms that each reproject the original opposite view and compare photometrically with the current view, with \(\widetilde{O}_{ij}\) the mask that equals 1 for non-occluded and 0 for occluded pixels. Refer to the original paper for the exact notation. This is also where the importance of design 2 becomes visible: when the generated pseudo image carries pronounced artifacts — that is, without the refined generation — forcing the fully pseudo strategy actually hurts (Tab. 3: 4.31% for (f) versus 4.06% for (g)), because the network can then find an identifiable shortcut in the artifacts themselves. Only once the pseudo image is realistic enough does the fully pseudo strategy actually deliver stable convergence.

Loss & Training¶

Beyond the photometric term, the paper adopts the edge-aware smoothness of Monodepth2 together with mean-normalized disparity \(D^{L*}=D^L/\overline{D^L}\):

\[L_s=\big|\partial_x D^{L*}\big|e^{-|\partial_x I^L|}+\big|\partial_y D^{L*}\big|e^{-|\partial_y I^L|},\qquad L=L_p+\lambda L_s\]

The photometric error combines L1 and SSIM with weight \(\alpha=0.85\), again following Monodepth2: \(pe=\frac{\alpha}{2}(1-\mathrm{SSIM})+(1-\alpha)\lVert\cdot\rVert_1\). The authors observe that an overly large smoothness loss in the early stages hinders the model's exploration capability and slightly hurts final performance, so \(\lambda\) is made a gradual parameter that starts at 0.001 and rises linearly to 0.5 over 10k iterations.

All datasets share one set of hyper-parameters: two RTX 4090 GPUs, PyTorch, Adam (\(\beta_1=0.9\), \(\beta_2=0.999\)) with cosine annealing, images cropped to \(512\times256\), batch size 8. Data augmentation is applied to the input images only and never to the feedback images, comprising random cropping, brightness enhancement and contrast adjustment. Everything trains from scratch, with no pre-training, no ground-truth labels and no post-processing. As for overhead, pseudo-image generation needs a single gradient-free forward pass and adds no GPU memory consumption, while total training time grows by about 25%; at inference the pseudo-image generation is not used at all, so the computational cost is identical to the baseline network.

Key Experimental Results¶

Main Results¶

KITTI 2012 / 2015 online benchmarks (test set, all numbers are the authors' official submissions). Two backbones are reported: PSMNet to show the gains come from the method rather than from architectural progress, and MonSter to show the method also applies to recent architectures.

Method	2012 3-noc (%)	2012 3-all (%)	2012 EPE-all	2015 D1-bg (%)	2015 D1-fg (%)	2015 D1-all (%)	2015 D1-noc (%)
OASM	6.39	8.60	2.0	6.89	19.42	8.98	7.39
PASMnet	-	-	-	5.41	16.36	7.23	6.69
OASM-DDS	4.81	5.69	1.2	4.46	15.76	6.51	6.02
UHP	6.05	7.09	1.3	5.00	13.70	6.45	5.93
Flow2Stereo	4.58	5.11	1.1	5.01	14.62	6.61	6.29
CRD-Fusion	4.38	5.40	1.1	4.59	13.68	6.11	5.69
Pseudo-Stereo-PSMNet	3.46	4.08	0.9	3.11	12.52	4.68	4.40
Pseudo-Stereo-MonSter	3.08	3.62	0.8	2.93	11.67	4.39	4.22

Generalization across backbones and datasets (EPE and the 3-px error, lower is better). Note that Spring is evaluated at half resolution owing to the authors' compute limits.

Backbone / Dataset	Metric	w/o Pseudo-Stereo	w/ Pseudo-Stereo
PSMNet @ SceneFlow	EPE (px) / 3-px (%)	5.41 / 16.85	3.10 / 9.80
GMStereo @ SceneFlow	EPE (px) / 3-px (%)	5.09 / 15.01	3.11 / 8.49
MonSter @ SceneFlow	EPE (px) / 3-px (%)	5.28 / 16.15	2.32 / 7.86
PSMNet @ Spring	EPE (px) / 3-px (%)	2.41 / 6.58	2.30 / 5.12
GMStereo @ Spring	EPE (px) / 3-px (%)	2.33 / 6.31	2.18 / 4.78
MonSter @ Spring	EPE (px) / 3-px (%)	2.37 / 6.56	1.77 / 4.65

Ablation Study¶

Component ablation on the KITTI 2015 training set. PS = pseudo-stereo inputs strategy, Occ = occlusion estimation and gradient detaching, RG = refined pseudo-image generation (padding + wider cropping), FPS = fully pseudo-stereo inputs strategy.

Config	PS	Occ	RG	FPS	D1 (%) ↓	EPE ↓	Original note
(a) baseline					5.67	1.17	/
(b)	✓				5.73	1.24	over-smoothing in occlusions
(c)	✓				4.99	1.07	overfitting
(d)	✓	✓			4.08	0.99	overfitting
(e)			✓		5.17	1.11	/
(f)	✓	✓		✓	4.31	1.04	pseudo-image artifacts hurt
(g) full	✓	✓	✓	✓	4.06	1.01	/

⚠️ The check-mark columns of Tab. 3 in the original paper are not reliably aligned after text extraction; the labels here follow the paper's own grouping in prose: rows (a)–(d) do not use the fully pseudo-stereo input strategy and exhibit overfitting, while rows (e)–(g) do and exhibit none. The prose explicitly pins down (f)=PS+Occ+FPS and (g)=PS+Occ+RG+FPS; check the original table for the exact checkbox combinations of the remaining rows.

Occluded regions themselves, plus a comparison against the naive flip baseline (KITTI 2015; occlusion regions identified by the rendering-based method of Sec. 3.2):

Config	D1-all	EPE-all	D1-occ	EPE-occ	D1-noc	EPE-noc
base	5.67	1.17	20.82	3.24	4.37	1.01
naive flip	5.61	1.14	20.77	3.20	4.34	1.00
ours	4.06	1.01	9.05	1.36	3.68	0.97

Key Findings¶

The gains come almost entirely from occluded regions. This is the paper's most convincing evidence: D1-occ drops from 20.82% to 9.05% and EPE-occ from 3.24 px to 1.36 px (a reduction of roughly 58%), while D1-noc in non-occluded regions only moves gently from 4.37% to 3.68%. The method acts on precisely the bottleneck it claims, rather than gaining points incidentally through global regularization.
Naive flipping is quantitatively refuted. naive flip matches the base configuration almost item by item (D1-all 5.61 vs 5.67, D1-occ 20.77 vs 20.82), confirming the authors' argument that flipping cannot change the fixed directionality imposed by the coupling of inputs and feedback — the occlusion remains on the same side.
The value of the fully pseudo strategy lies in stability, not in the single-point optimum. This deserves to be stated honestly: row (d) reaches D1 4.08 versus 4.06 for the full model (g), and on EPE (d)'s 0.99 is even marginally better than (g)'s 1.01. What the fully pseudo strategy really buys is that long training runs do not degrade (the training curves in the original Fig. 8), not a better peak in any single evaluation — which also means this component's necessity rests mainly on the convergence argument, and its contribution is easy to underestimate from the table peak alone.
Over-smoothing and overfitting are two distinct failure modes. The notes on (b) and on (c)–(d) show that pseudo inputs without occlusion handling fall into over-smoothing inside occlusions, while adding occlusion handling falls into overfitting; the two components must be used together. The (f)→(g) comparison further shows that refined pseudo-image generation is a precondition for the whole chain — with pronounced pseudo-image artifacts, forcing the fully pseudo strategy loses accuracy instead.
Generalization is backed by real evidence. Improvements hold across three very different backbones (PSMNet / GMStereo / MonSter) and two stylistically distinct datasets (SceneFlow / Spring); on SceneFlow the 3-px error is nearly halved (PSMNet 16.85% → 9.80%). This supports the claim that occlusion is a shared bottleneck independent of architecture and dataset.

Highlights & Insights¶

Reformulating "the supervision signal is broken" as "the input can be changed." Earlier work on occlusion all followed "the signal is unreliable → fix the signal" (masking, extra regularizers, swapping in neighbors). This paper turns the other way: make the geometric relation generating the signal itself symmetric. That reframing is the paper's "aha" moment — no extra constraint and no new loss term, just a change in how the input is constructed sidesteps the fundamental contradiction.
Using the network currently being trained as the renderer. Pseudo-image quality improves in lockstep with network accuracy, forming a virtuous cycle of "more accurate → better pseudo image → better supervision → more accurate," while pinning the left input and the feedback images to real images provides an anchor that does not drift with pseudo-image quality. The self-imposed constraint of importing no external model or data ends up being a design advantage, and fits the self-supervised premise better than NeRF or trinocular hardware would.
Reusing a by-product of rendering as the occlusion mask. Rendering already has to work out which pixels are hidden and which fall out of bounds; the authors simply treat those invalid pixels as occlusion labels, saving an extra disparity inference and avoiding the boundary misjudgments of a consistency check. This is a near zero-cost design that transfers to any self-supervised pipeline containing a synthesis or reprojection step.
The diagnosis "overfitting comes from inputs being distinguishable" is transferable. Whenever training data mixes samples of different provenance (real/fake, simulated/real, original/augmented), models tend to first learn to tell the sources apart and then fit each separately, bypassing the designer's intent. This paper's remedy is to remove the distinguishability (make every input contain a pseudo image), an idea that applies equally to sim-to-real training and generative data augmentation.
Train-time constraints, zero inference cost. Pseudo-image generation appears only during training, so inference costs exactly the same as the baseline, while training time grows only about 25% with no extra GPU memory. A change that alters only the training signal and leaves the inference architecture untouched is very friendly to deployment.

Limitations & Future Work¶

The authors acknowledge that they do not explicitly handle textureless or reflective regions, and note that providing accurate feedback for these ill-posed areas is inherently intractable with binocular cues alone. They point to the multi-frame self-supervised paradigm (e.g. UnOS) — temporal consistency across frames resolves texture ambiguity while single-frame disparity accuracy resolves depth ambiguity of moving objects — as a complementary direction.
Sources of error other than occlusion are untouched. The method targets erroneous feedback caused by occlusion, but photometric consistency is also broken by exposure changes, textureless surfaces and moving objects, for which no mechanism is offered.
The strength of the ablation evidence is uneven. The necessity of the fully pseudo strategy rests mainly on training curves not degrading, with no multi-seed variance or quantitative long-run curve provided, and rows (d) and (g) are nearly identical on single-point metrics, so a reader cannot independently judge how much the component is worth from the table.
The coupling between pseudo-image quality and network accuracy is never quantified. That early, severely unrealistic pseudo images still allow convergence is explained only qualitatively ("force the network to exploit monocular cues"), with no curve relating pseudo-image quality to final accuracy.
The evaluation scenarios are narrow. KITTI is a driving dataset with relatively uniform occlusion types (mostly the lateral edges of cars and pole-like objects), while Spring is evaluated at half resolution for compute reasons and is therefore not fully comparable with full-resolution methods. Behaviour on indoor, densely occluded or non-rigid scenes remains unknown.
There is a natural ceiling: because pseudo inputs are not used at inference, the method improves the quality of the training signal but does not change the network's intrinsic bias when facing occlusion at test time. If the observable information in an occluded region is simply insufficient at test time, the accuracy ceiling is still set by the architecture and its receptive field.
Possible improvements: pseudo-image generation is currently a minimal rendering ("relocate by disparity plus z-buffer"). Trying multi-scale splatting from the network's own features, or a lightweight amortized synthesis, could raise pseudo-image realism and let the fully pseudo strategy work on harder scenes (weak texture, reflections). One could also extend the single pseudo-input forward pass into a multi-scale iterative refinement so that pseudo-image quality and disparity accuracy reinforce each other inside one loop.

vs OASM / PASMNet / OASM-DDS and other masking methods: they identify occluded regions and remove them from the photometric loss, leaving an information void that is then compensated by smoothness or neighborhood regularity. This paper introduces no extra constraint or loss term at all; instead, the input construction itself lets both sides of the occluder receive correct feedback. The difference is "cut losses and extrapolate" versus "remove the information gap itself."
vs ES3Net / Godard-style flip and left-right consistency: they treat left-right consistency directly as a loss term or borrow from symmetric regions by flipping. But an incorrect disparity can satisfy consistency just as well, and flipping cannot change the fixed occlusion direction. This paper quantitatively shows naive flip yields essentially no gain over base (5.61 vs 5.67).
vs NeRF-Supervised Deep Stereo / multi-view self-supervision: conceptually the closest work — both obtain unoccluded information from an auxiliary viewpoint. But the former needs ground truth or extra data to train a NeRF for novel-view synthesis, and the latter needs dedicated symmetric multi-view hardware; here the pseudo viewpoint is rendered by the network being trained itself, with zero external dependency, so it runs on any ordinary binocular dataset.
vs occlusion handling in unsupervised optical flow (UnFlow / Janai et al.): the same root problem — photometric consistency fails at occlusions. Those methods are predominantly mask-based; this paper's "decouple inputs from feedback" idea transfers directly: render a pseudo frame from the current model as the input, compute photometry against the original frames, and relieve the occlusion and motion-boundary errors of optical flow.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Reframing occlusion from "fix the signal" to "change the input," using a self-rendered pseudo view to decouple inputs from feedback, is new in self-supervised stereo matching and depends on no external model or hardware.
Experimental Thoroughness: ⭐⭐⭐⭐ Three backbones × three datasets plus a full component ablation and dedicated occlusion-region metrics make a solid evidence chain; but the checkbox combinations in the original ablation table are hard to read, and the convergence argument for the fully pseudo strategy lacks multi-seed and quantitative curve support.
Writing Quality: ⭐⭐⭐⭐ The motivation is clearly derived and the figures (Fig. 1's flip-fails contrast, Fig. 5's pseudo-image generation) are persuasive; weaker points are that the formulas in Sec. 3 are not easy to read as typeset, and the mapping between component names and ablation columns takes effort to infer.
Value: ⭐⭐⭐⭐⭐ It alters only the training signal at zero inference cost and works across backbones and datasets, making it a directly applicable improvement for any self-supervised stereo or optical-flow pipeline built on photometric consistency.