Gaussian Belief Propagation Network for Depth Completion

Conference: ECCV 2026
arXiv: 2601.21291
Code: https://github.com/kakaxi314/GBPN
Area: 3D Vision
Keywords: Depth Completion, Belief Propagation, Markov Random Field, Probabilistic Graphical Model, Sparse Depth

TL;DR

GBPN reformulates depth completion as "performing Gaussian belief propagation on a Markov Random Field (MRF) dynamically constructed by a network." Instead of directly regressing depth, the network learns the MRF's potential functions, edge weights, and non-local edge structures, and then uses differentiable Gaussian BP to iteratively infer the dense depth distribution (treating the mean as depth and precision as confidence). It achieves SOTA performance on NYUv2 and KITTI, exhibiting robustness under extremely sparse inputs that far surpasses pure regression networks.

Background & Motivation

Depth completion aims to recover a dense depth map from a color image accompanied by sparse, irregular depth measurements (from LiDAR, SfM, or even user clicks). Sparse points provide absolute scale constraints, while the color image offers structural and semantic cues. Traditional methods rely on handcrafted pipelines or fixed graphical models (such as MRFs with simple smoothness potentials), which possess overly rigid priors and fail to capture complex geometry and fine details. In recent years, the mainstream trend has shifted toward deep networks directly regressing depth, significantly improving accuracy. However, a frequently criticized issue remains: standard convolutional architectures are inherently unsuited for handling sparse and irregular inputs. Sparse points manifest as scattered valid pixels on feature maps, and the convolutional kernels see a specific distribution (e.g., "500 points") during training. Once the number of points increases or decreases during testing, the input distribution shifts, causing network performance to degrade drastically. The authors' experiments show that for approaches like GuideNet or CFormer, when the number of points exceeds roughly 5,000 (trained on 500), the REL metric actually increases instead of decreasing.

Recently, a hybrid approach has gained popularity: combining learned features with traditional structured inference. For example, the CSPN series utilizes a network to predict the parameters of an anisotropic diffusion process to refine regression results, and BP-Net learns a bilateral-filtering-like propagation. However, the propagation range of these methods remains limited (local/neighborhood diffusion), leading to severe decay as information diffuses from sparse points to distant unmeasured pixels, still producing blurry results in extremely sparse scenarios. The Key Challenge lies in the fact that information from sparse points needs to be propagated reliably across the entire image, whereas prior methods either use specialized sparse layers to forcefully handle scattered points or restrict propagation to small neighborhoods, failing to address both sides of the issue.

The Key Insight of this paper is to completely decouple the problem of "how sparse points are processed" from the network and hand it over to a globally consistent probabilistic framework. Core Idea: Instead of letting the network regress depth, the network is used to dynamically build a scene-specific MRF (learning potential functions and non-local edge structures), and then differentiable Gaussian belief propagation is executed on this graph. Sparse points naturally enter the global optimization as MRF data terms, allowing depth to propagate along the entire image, while the output inherently includes confidence estimates.

Method

Overall Architecture

GBPN is a hybrid framework of "deep network + probabilistic graphical model." The inputs are a color image \(I\) and a sparse depth map \(S\) projected onto the image plane (where valid pixels are irregularly distributed, and their quantity and locations can vary significantly). The output is a Gaussian distribution of the dense depth, where each pixel provides a mean \(\mu_i\) (representing predicted depth) and precision \(\Lambda_i\) (representing confidence).

The entire pipeline consists of two main components: a Graphical Model Construction Network (GMCN) that takes the color image (and optionally the depth distribution from the previous round) to predict all "components" of this MRF—parameters of unary and pairwise potentials (edge weights \(w\), expected depth difference \(r\)), damping factor \(\beta\), and offsets \(o\) for constructing non-local edges. Then, a Gaussian Belief Propagation (GBP) module performs iterative message passing and belief updates on this learned graph to propagate the constraints of sparse points across the entire image, outputting the marginal distribution of each pixel upon convergence. The entire framework is trained end-to-end using a probabilistically grounded loss that simultaneously supervises depth and precision.

The paper provides two configurations: GBPN-1 constructs the MRF solely using the color image (single U-Net); GBPN-2 adds a multi-modal fusion U-Net that takes the depth distribution \((\mu_1,\Lambda_1)\) from GBPN-1 along with the color image, fusing them using cross-attention for a second-stage refinement. The final submitted model, referred to as GBPN, denotes GBPN-2.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Color Image I + Sparse Depth S"] --> B["Dynamically Construct Scene-Specific MRF<br/>GMCN learns potential functions, edge weights, and non-local edges"]
    B --> C["Gaussian Belief Propagation (GBP)<br/>Message Passing → Marginalization to obtain depth distribution"]
    C --> D["Serial & Parallel Message Passing<br/>4-directional local sweep + Non-local parallel"]
    D -->|GBPN-2 refinement| B
    D --> E["Dense Depth Distribution<br/>μ as depth · Λ as confidence"]

Key Designs

1. Dynamically Constructing Scene-Specific MRFs: Turning Sparse Points into Graphical Data Terms

When traditional MRFs are used for depth completion, edge weights \(w\) are calculated based on handcrafted features like color difference, and expected depth differences \(r\) are directly set to 0 (enforcing smoothness). Such priors are too coarse to capture complex geometry. GBPN uses GMCN to learn these parameters entirely end-to-end and dynamically update them along with the iterations: where measurements are reliable, data constraints are strengthened, while smoothness constraints adaptively vary according to image content. A key benefit of this is that sparse inputs are inherently absorbed—the joint distribution of the MRF is defined as:

\[p(X\mid I,S)\propto\prod_{i\in\mathcal{V}_v}\phi_i\prod_{(i,j)\in\mathcal{E}}\psi_{ij}\]

where unary potentials \(\phi_i=\exp(-\tfrac{w_i(x_i-s_i)^2}{2})\) only apply to measured pixels \(i\in\mathcal{V}_v\), using a learnable confidence weight \(w_i\) to pull the estimated depth towards the measured value \(s_i\). The pairwise potentials \(\psi_{ij}=\exp(-\tfrac{w_{ij}(x_i-x_j-r_{ij})^2}{2})\) encourage the depth difference of adjacent pixels to be close to the expected difference \(r_{ij}\). Sparse points thus enter the global optimization as data terms, completely eliminating the need to design specialized network layers for handling scattered points—this is the root cause of its robustness to sparsity and noise: whether there are 10 points or 20,000 points, it merely changes the number of valid data terms without altering the inference mechanism.

2. Dynamic Non-Local Edges: Empowering the MRF to Capture Long-Range Dependencies

With only fixed local edges like 8-neighborhoods, information in the graph moves "step-by-step," requiring many steps for distant pixels to receive constraints from sparse points. In addition to fixed local edges, GMCN predicts several non-local edges for each pixel: leveraging ideas from deformable convolution, the network regresses floating-point offsets \(o\) and uses bilinear interpolation to sample Gaussian parameters at non-integer positions as message sources for non-local neighbors. Bilinear interpolation ensures gradients can flow back, enabling end-to-end learning of non-local edges. Consequently, the state of each variable is simultaneously constrained by local neighbors and distant pixels that are "semantically related but not necessarily spatially adjacent." This extends the modeling capability of the MRF from fixed grids to adaptive long-range structures, propagating sparse point information to semantically related distant regions in a single step.

3. Gaussian Belief Propagation: Transforming Graphical Inference into Analytical Mean-Precision Algebraic Updates

Calculating exact posteriors on high-resolution graphs is intractable. The idea of BP is to iteratively update beliefs and pass messages: the belief of pixel \(i\) is proportional to its unary potential multiplied by all incoming messages from its neighbors, \(b_i\propto\phi_i\prod_{j}m_{j\to i}\), and the message \(m_{j\to i}\) is computed by marginalizing out the sending node \(x_j\). While this integration is generally complex in standard BP, the MRF in this work is a Gaussian Graphical Model—the crucial element that makes the entire method differentiable and analytical: assuming all beliefs and messages are Gaussian, the complex integration and products in Eq.(5) degenerate into algebraic updates on Gaussian parameters (information vector \(\eta\) and precision \(\Lambda\), associated via \(\eta=\mu\Lambda\)). The belief update simply accumulates neighbor messages weighted by precision:

\[\eta_i=w_is_i+\sum_{(i,j)\in\mathcal{E}}\hat\eta_{j\to i},\qquad \Lambda_i=w_i+\sum_{(i,j)\in\mathcal{E}}\hat\Lambda_{j\to i}\]

The message passing simplifies to algebraic operations involving adding expected differences to the mean and harmonic operations on precision (\(\mu_{j\to i}=\mu_{j\setminus i}+r_{ij}\), \(\Lambda_{j\to i}^{-1}=\Lambda_{j\setminus i}^{-1}+w_{ij}^{-1}\), where \(j\setminus i\) denotes all incoming messages to \(j\) except the one from \(i\)). This complete set of updates is purely algebraic and differentiable, allowing the "BP layer" to be embedded into the network for end-to-end training. Since the graph is loopy (loopy BP), a damping technique is applied to stabilize convergence (taking a weighted average of the previous and current messages using \(\beta_i\)), where \(\beta\) is also predicted by the network. Upon convergence, each pixel's depth is retrieved as \(\mu_i=\eta_i/\Lambda_i\), and its confidence as \(\Lambda_i\). The authors also noted that while GBP's mean is accurate, the precision might not be, so they let the network regress an additional residual term added to the GBP's \(\Lambda\), which is then passed through a sigmoid function to obtain the corrected precision.

4. Hybrid Serial-Parallel Message Passing: Ensuring Global Propagation and High Speed

The message propagation scheme directly dictates how far and how fast information runs. Serial propagation allows local evidence to influence distant nodes and spread throughout the entire graph (which is vital for depth completion to ensure every variable receives sufficient messages for valid beliefs), but it is slow. Parallel propagation only propagates locally but can fully exploit GPU parallelism, making it fast. GBPN fuses both approaches, while decomposing the loopy graph into acyclic subgraphs to stabilize convergence: it divides the MRF edges into four local directional sets—left-to-right (LR), top-to-bottom (TB), right-to-left (RL), and bottom-to-top (BT)—plus a non-local edge set \(\mathcal{E}_{NL}\). Local edges follow serial propagation: a sweep is performed for each direction (e.g., during the LR sweep, the update of the \(n\)-th column must wait until the \((n-1)\)-th column is computed, enabling a single sweep to push information from one side of the image to the other). Non-local edges follow parallel propagation: messages for all pixels across \(\mathcal{E}_{NL}\) are updated simultaneously. Each round sequentially executes four serial sweeps (LR/TB/RL/BT), followed by \(T_n\) steps of non-local parallel propagation, repeating for \(T\) rounds. This decomposition ensures that "a single serial sweep propagates information across the entire image." Visualizations in the paper demonstrate that in the extreme case of having only a single depth point, one round of serial propagation can spread information across the entire image to converge to a meaningful depth.

Loss & Training

The depth loss combines L1 and L2 losses, normalized by the maximum L1 error across the entire image to stabilize convergence: \(L_i^X=\frac{\|\mu_i-x_i^g\|_2^2+\alpha\|\mu_i-x_i^g\|_1}{\max(\|\mu-x^g\|_1)}\). Since the output is a Gaussian distribution, the final probabilistic loss is used to supervise the precision as well:

\[L=\frac{1}{|\mathcal{V}_g|}\sum_{i\in\mathcal{V}_g}\Lambda_i L_i^X-\log(\Lambda_i)\]

where \(\mathcal{V}_g\) is the set of pixels with ground truth depth. The elegance of this lies in the fact that although precision \(\Lambda\) has no ground truth, this loss forces the model to automatically learn to lower \(\Lambda\) (increase uncertainty) in high-error regions to reduce the penalty, thereby supervising the confidence directly without needing precision labels. Training is conducted on 4×RTX 4090 GPUs using AdamW (weight decay 0.05, L2 gradient clipping 0.1) and a OneCycle learning rate schedule for approximately 300,000 steps, with final weights evaluated via EMA.

Key Experimental Results

Main Results

Main results on KITTI (outdoor, evaluated on the test server) and NYUv2 (indoor), along with zero-shot generalization on VOID. GBPN refers to GBPN-2.

Dataset	Metric	GBPN	Prev. SOTA	Description
KITTI	iRMSE (1/km) ↓	1.79	1.82 (BP-Net/TPVD, etc.)	Rank 1 on the leaderboard at the time of submission
KITTI	RMSE (mm) ↓	681.61	678.12 (DMD3C)	Rank 2 on the leaderboard; however, DMD3C leverages extra supervision from foundation models, while GBPN is trained from scratch solely on the standard training set
KITTI	RMSE vs BP-Net	681.61	684.90 (BP-Net)	Surpasses the counterpart BP-Net across all metrics
NYUv2	RMSE (mm) ↓	0.085	0.085 (DMD3C)	Tied for the best
NYUv2	δ1.02 (%) ↑	89.3	88.3 (OGNI-DC)	\(\delta_{1.25}\) is saturated (\(\ge\)99.6%), hence the stricter \(\delta_{1.02}\)/\(\delta_{1.05}\) are used to differentiate performance

VOID generalization (trained on NYUv2 and directly evaluated in a zero-shot manner; across different sparsities, sparsity patterns, and scene domains):

Setting	Metric	GBPN	BP-Net	NLSPN
VOID 1500	RMSE (m) ↓	0.649	0.672	0.687
VOID 500	RMSE (m) ↓	0.692	0.721	0.758
VOID 150	RMSE (m) ↓	0.830	0.847	0.932

GBPN consistently leads in RMSE/MAE across all sparsity levels, validating the superior generalization brought by "treating sparse points as MRF observation terms without specialized sparse layers."

Ablation Study

Gradual addition of components on NYUv2 from the simplest optimization baseline \(V_1\) to \(V_9\) (GBPN-1), using a half training cycle.

Configuration	Increment	RMSE (mm) ↓	δ1.02 (%) ↑
\(V_1\)	Convolutional U-Net + affine fitting baseline (L2 loss only, no pairwise terms)	342.70	21.58
\(V_2\)	Switched to the proposed probabilistic loss	340.84	25.23
\(V_3\)	Solving a fixed MRF (fixed local edges, convolutional features)	108.29	83.71
\(V_5\)	Global-local cells (convolution + attention) as the backbone	107.01	84.01
\(V_6\)	+ Dynamic parameters	103.92	84.69
\(V_7\)	+ Dynamic non-local edges	101.42	85.19
\(V_8\)	Local edges 4 \(\rightarrow\) 8	100.95	85.20
\(V_9\)	GBP iterations 3 \(\rightarrow\) 5 (selected as GBPN-1)	100.69	85.27

Key Findings

• Problem formulation itself contributes the most: The transition from $V_2 \to \(V_3\) introducing the "solving MRF" step slashes the RMSE from 340.84 to 108.29 mm. Replacing direct regression with structured inference on an MRF is the most significant leap, validating the value of the core idea.
• Serial propagation is key to global information flow: In the appendix, comparing "introducing serial propagation" drops the RMSE from 340.84 to 108.29 mm. The dynamic non-local parallel edges (\(V_6 \to V_7\)) yield another reduction of 103.92 \(\to\) 101.42 mm, indicating that long-range edges provide stable gains, though smaller than the serial framework itself.
• Crushing advantage under extreme sparsity: With only a single point, GBPN can cover the entire image in a single serial sweep. When simulating 8-beam LiDAR (KITTI validation set), the RMSE of 2750.4 mm achieves an absolute improvement of 1791.5 mm over BP-Net's 4541.9 mm. Conversely, GuideNet/CFormer suffer from an in-rebound REL when the testing point count exceeds the training sparsity (>5,000) due to input distribution shifts caused by feeding sparse maps directly into convolutions.
• Pragmatic and reliable confidence estimation: Filtering out the 1% of pixels with the lowest confidence reduces NYUv2 RMSE from 85.51 mm to 26.89 mm, indicating that the predicted precision \(\Lambda\) correlates tightly with actual errors (with lower precision in ambiguous regions like boundaries and sky).
• Insensitive to iteration count, allowing flexible trade-offs: Increasing iterations from 5 to 13 only yields a minor RMSE change of 85.95 \(\to\) 85.24 mm, with linear runtime growth and constant GPU memory usage (0.18 GB).
• Efficiency bottlenecked by the custom GBP implementation: The GMCN requires 62.58 GFLOPs of computation but takes only 11.43 ms, while GBP takes 23.35 ms despite requiring only 0.26 GFLOPs—taking twice as long for 200 times fewer computations. This is because the GBP is processed via custom-written CUDA kernels without full optimization, which the authors point out has massive room for acceleration. The parameter count is 34.78M, larger than BP-Net (19.87M), but still relatively lightweight since the MRF is dynamically built by a small network.

Highlights & Insights

• Decoupling the "handling of sparsity" bottleneck from neural networks: Instead of obsessing over sparse convolutions/masks, the authors rebrand sparse coordinates as MRF data terms, delegating them to global optimization. This shift in perspective is the most "aha!" moment of the paper, explaining its innate robustness to sparsity, noise, and cross-domain transfer.
• The Gaussian assumption is pivotal for making BP differentiable and trainable: Because the potential functions are quadratic and the graph is Gaussian, the integration and multiplication in BP degenerate into algebraic updates of \(\eta / \Lambda\), allowing it to serve as a differentiable layer. This strategy of "using parametric distribution assumptions to turn intractable inference into closed-form algebra" can easily be transferred to other structured prediction tasks.
• Serial-parallel graph decomposition: Factoring the loopy graph into four directional acyclic subgraphs stabilizes convergence, enables "a single sweep to cross the entire image," and processes non-local edges in parallel to max-out hardware utilization. This represents a pragmatic harmonization of "propagating far" and "running fast," highly reusable for any spatial grid propagation task.
• Unsupervised uncertainty estimation without labels: The probabilistic loss term \(\Lambda_i L_i^X - \log\Lambda_i\) allows the model to naturally output lower precision in difficult areas to penalize less, bypassing the need for confidence ground truths. This is highly useful for downstream risk-aware planning.

Limitations & Future Work

• The authors acknowledge that several hyperparameters, such as the number of neighbors and propagation steps, are empirically determined and have yet to be systematically searched for optimal accuracy-efficiency trade-offs or adaptively computed.
• The model is trained only on limited, curated datasets. Although robust to sparsity, extending it to larger, more diverse datasets or distilling knowledge from foundation models (similar to how DMD3C leverages external supervision to achieve lower RMSE) remains a viable and highly promising direction.
• GBP is custom-written. Although GPU-parallelized, it lack PyTorch-level deep optimization, which results in a low computation but high runtime profile. The tested runtime of 114.82 ms (on KITTI) is still somewhat heavy for real-time deployment, making engineering-level acceleration a prerequisite for real-world deployment.
• The visualization and quantitative gains of non-local edges (\(V_6 \to V_7\) drops by only 2.5 mm) are relatively modest compared to the serial framework itself. Whether the actual contributions of long-range edges are partially superseded by the 4-directional serial sweeps warrants further investigation.

Related Work & Insights

• vs BP-Net: Both fall under the hybrid "model + learning" category and attempt to propagate sparse point information. BP-Net learns local bilateral-filtering-style propagation, which has a limited range; whereas GBPN performs GBP on a global MRF, allowing propagation across the entire image. GBPN also beats BP-Net across all KITTI metrics with fewer parameters.
• vs CSPN/NLSPN/DySPN series: These networks predict anisotropic diffusion parameters to refine regressed depths, which essentially remains local diffusion refinement. GBPN does not refine regression results; instead, it treats the task from scratch as MRF inference, where sparse points act as data terms rather than initializations, showing clear advantages under extreme sparsity.
• vs OGNI-DC: Similarly frames completion as a learned optimization problem (network predicts local depth differences and minimizes energy iteratively via conjugate gradient). GBPN utilizes GBP instead of conjugate gradient, while also learning non-local edge structures and confidence maps, resulting in better generalization (VOID) and robustness to extreme sparsity.
• vs Traditional MRF Depth Completion (Diebel & Thrun, etc.): Traditional methods set edge weights and expected differences manually (\(r\) is typically set to 0), yielding restrictive priors. GBPN allows the network to dynamically build the MRF parameters and structures end-to-end, harvesting both the expressiveness of learning and the robustness of structured inference.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Completely reformulates depth completion into "network-constructed MRF + differentiable Gaussian BP." The perspective is novel and self-consistent, rather than a mere stacking of modules.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluation across three benchmarks (KITTI/NYUv2/VOID) plus multi-dimensional ablations on sparsity, noise, LiDAR lines, and efficiency. The step-by-step \(V_1\)-\(V_9\) progression is highly transparent.
• Writing Quality: ⭐⭐⭐⭐ Consequential logic with clear motivation; the GBP derivations are comprehensive. However, the manuscript is formula-dense, and some components (residual precision correction, 3D positional encoding) are deferred to the appendix, which makes the main text feel slightly cramped.
• Value: ⭐⭐⭐⭐ Robust to extreme sparsity and domain shifts while generating usable confidence maps, making it highly practical for robotics/autonomous driving. The speed of the custom-written GBP remains the only bottleneck before real-world deployment.

Related Papers

• [CVPR 2026] Zero-Shot Depth Completion with Vision-Language Model
• [ECCV 2024] Hyperion: A Fast, Versatile Symbolic Gaussian Belief Propagation Framework for Continuous-Time SLAM
• [ICLR 2026] ORCaS: Unsupervised Depth Completion via Occluded Region Completion as Supervision
• [ECCV 2026] Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation
• [ICLR 2026] Large Depth Completion Model from Sparse Observations