CoDePose: Multi-View 3D Human Pose Estimation via Coupled 2D-3D Denoising Diffusion¶

Conference: ECCV2026
Paper: Official ECCV page
PDF: Full paper
Authors: Yanlu Cai, Yuxuan Liu, Weizhong Zhang, Yuan Wu, Cheng Jin
Area: Human Understanding
Keywords: Multi-view pose estimation, coupled diffusion, asymmetric initialization, body-coordinate consistency, joint confidence

TL;DR¶

Rather than treating 2D detections as immutable observations, CoDePose jointly denoises 2D and 3D poses while retaining detections as conditioning, combining cross-view fusion with stabilizing supervision to reduce MPJPE from FusionFormer's 15.1 mm to 12.1 mm on Human3.6M with ViTPose, four views, and 27 frames.

Background & Motivation¶

Multi-view human pose estimation can first detect 2D joints in each image and then reconstruct a 3D skeleton from the joint sequences of multiple cameras. These lifting-based methods process compact coordinates rather than video-wide image features, making temporal context relatively inexpensive. Their inputs, however, are not geometric ground truth: occlusion, motion blur, and left–right limb confusion displace detected joints from their true projections. Even a plausible 3D skeleton may not project onto the detected locations. Independently detected joints across cameras can also contradict one another, leaving no single 3D skeleton that explains all observations.

Existing multi-view fusion methods typically reweight observations or reduce the influence of outliers without directly correcting the observations. Earlier pose diffusion methods repeatedly update 3D predictions, but feed the same fixed 2D detections into every step, potentially reintroducing their errors. This paper starts from the observation that most CPN and ViTPose detection errors on Human3.6M approximately follow a zero-mean Gaussian distribution. Detected coordinates can therefore be viewed as noisy samples near ground truth rather than unmodifiable constraints. This Gaussianity is an empirical approximation, not a description of every severe occlusion or outlier.

Core Idea: include 2D observations in an updatable diffusion state, jointly correcting multi-view 2D joints and 3D skeletons within one denoiser; initialize inference from detected 2D poses and use joint confidence and body-coordinate consistency to suppress error propagation introduced by coupling.

Method¶

Overall Architecture¶

The input is a synchronized multi-view sequence of 2D joints, with four views, 27 frames, and 17 joints in the standard experiments. The output is the center-frame 3D pose in each camera's coordinate system, not a skeleton directly expressed in a calibrated world coordinate system. Without requiring camera extrinsics, the method constructs a coupled state containing 2D and 3D coordinates, predicts clean coordinates through cross-view joint denoising, and trains the predictor with stabilizing supervision.

During training, ground-truth 2D projections and 3D poses are separately corrupted, while original 2D detections supply an additional condition. At inference time, the 2D state starts from detections and only the 3D state starts from Gaussian noise. Each cross-view joint denoising pass predicts a 3D pose, refined 2D coordinates, and joint confidence; DDIM updates the coupled state. Confidence supervision and body-coordinate consistency are training constraints, not an extra geometric solver applied after inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view 2D joint sequences"] --> B["Coupled State and<br/>Asymmetric Initialization"]
    B --> C["Cross-view Joint Denoising"]
    C --> D["2D and 3D predictions<br/>and joint confidence"]
    D -->|DDIM state update| C
    D -.->|Training supervision| E["Confidence and<br/>Body-coordinate Stabilization"]
    D -->|Final sampling step| F["Center-frame 3D poses per view"]

Key Designs¶

1. Coupled State and Asymmetric Initialization: make 2D detections correctable

For each frame, view, and joint, the model concatenates the 3D coordinate and 2D projection into a five-dimensional state. The 2D component is not a visual feature re-extracted from an image; it is a coordinate variable describing the same joint as the 3D skeleton. The denoising objective thus shifts from merely predicting 3D points given 2D points to learning their joint distribution, allowing 2D correction and 3D reconstruction to inform each other.

\[ x_{\mathrm{joint}}^{\mathrm{gt}}=[x_{\mathrm{3D}}^{\mathrm{gt}};x_{\mathrm{2D}}^{\mathrm{gt}}] \in\mathbb{R}^{T_f\times V\times J\times 5}. \]

Here \(T_f\), \(V\), and \(J\) denote the numbers of frames, views, and joints. Both components share a diffusion timestep and cosine noise schedule during training, but their noise samples are independent. The 2D noise magnitude can be scaled separately, with \(\sigma_{\rm 2D}=1.0\) in the main experiments. The denoiser predicts clean coordinates directly rather than only regressing noise. Original detections \(c=x_{\mathrm{2D}}^{\mathrm{det}}\) remain an uncorrupted condition. Correcting the 2D state therefore does not discard the original 2D evidence: an updatable state and a fixed condition coexist, which is the crucial distinction from the 3D-only diffusion baseline with fixed 2D conditioning.

At inference time, pure Gaussian noise initializes only the 3D component, while detector outputs initialize the 2D component. The model need not create a 2D skeleton from pure noise and then align it with detections that already exist; it starts by correcting a broadly plausible but imperfect skeleton. The paper explains this as exploiting a sample near ground truth on the diffusion trajectory, and the low-step experiments support its efficiency. However, the two coordinate components have different initialization noise levels at the same diffusion timestep. Approximately Gaussian detection errors do not strictly prove identical training and inference distributions, so this is best understood as an empirically supported initialization strategy.

2. Cross-view Joint Denoising: use other cameras and neighboring frames to correct observations

The denoising network first encodes each view independently using a PoseFormer-style spatial–temporal Transformer. Joints interact within each frame to capture kinematic relationships, then aggregate into pose-level tokens so temporal attention can exploit neighboring frames. Spatial positional embeddings distinguish joint identities, while a timestep embedding identifies the current noise stage. The purpose is not simply greater network depth: it is to build view-specific representations containing body structure and motion context before cross-camera interaction.

The encoder then concatenates frame sequences from all views, enabling global self-attention across views and frames. The decoder uses each view's own sequence as queries and retrieves complementary evidence from shared memory through cross-attention. A joint obscured in one camera can consequently draw on a more reliable pose in another camera and the motion trajectory of neighboring frames. Fused features return to a per-view layout, and temporal convolutions with linear prediction heads produce center-frame 3D coordinates, 2D coordinates, and joint confidence. DDIM uses these predictions to update both coordinate components rather than only the 3D branch.

The network does not enforce triangulation with known camera matrices. Shared representations and joint supervision learn compatibility between 2D and 3D variables, so the paper's projection consistency largely arises from joint modeling rather than exact reprojection constraints at every step. The per-view 3D predictions still occupy different camera coordinate systems, motivating the body-coordinate supervision below.

3. Confidence and Body-coordinate Stabilization: prevent erroneous 2D states from contaminating 3D poses

Joint updates are not automatically beneficial: an outlier 2D joint can transmit its error to the 3D branch through coupling. The paper predicts a Sigmoid-bounded value for each joint. Although called an uncertainty head, its output is closer to confidence: values near 1 indicate reliable 2D estimates. Its supervision target depends on the current 2D reconstruction error:

\[ \sigma^*=\exp\!\left(-\|\hat{x}_{\rm 2D}-x_{\rm 2D}^{\rm gt}\|^2\right). \]

Accurate 2D predictions yield targets near 1, while larger errors push them toward 0. The authors describe this head as a stabilizer that reduces unreliable joints' influence, but the main text does not specify exactly how confidence multiplies features or regression terms. A concrete weighted-loss implementation should not be invented from that description. What can be verified is that the head learns 2D reconstruction reliability, and removing it makes the coupled model worse than the 3D-only diffusion baseline. Handling outliers is therefore material to whether joint modeling helps.

Body-coordinate stabilization addresses a different issue: different 3D coordinates in two cameras do not necessarily imply different physical poses, making direct pointwise consistency penalties inappropriate. The method places the origin at the hip root, defines an axis using the left-to-right hip direction, orthogonalizes the hip–spine direction against it, and completes the orthogonal basis with a cross product. Each view's predicted skeleton is compared with the others only after root subtraction and transformation into its own body frame. This is an extrinsics-free soft regularizer. If hip or spine predictions become degenerate, the frame can become unstable; the paper clamps the consistency loss to prevent harmful gradients from dominating rather than claiming that the coordinate transformation is reliable for every pose.

A Worked Example¶

Consider four cameras observing a seated person over a 27-frame input window. One camera mislocalizes a wrist hidden by the torso. Initialization retains all detections as conditioning and places them in the updatable 2D state, while the per-view 3D states start from random noise. The first denoising pass combines neighboring-frame motion with clearer wrist evidence from other cameras, adjusting both the 2D wrist location and the 3D arm structure instead of continually forcing the arm to explain an erroneous 2D point.

If sampling continues, the next pass receives an updated coupled state while retaining the original detection condition. The default configuration outputs center-frame 3D skeletons per view after 5 steps. During training, confidence supervision identifies joints whose reconstruction remains unreliable, while body-coordinate consistency encourages agreement in intrinsic skeletal shape across camera predictions. Ground truth is not required at inference time. This example illustrates the mechanism, not a quantitatively measured recovery trace for an individual paper example.

Loss & Training¶

The overall objective combines 3D coordinate mean squared error, 2D coordinate mean squared error, confidence supervision, and body-coordinate consistency, then applies a diffusion-timestep-dependent weight to the sum. The 2D term uses \(\lambda_{\rm 2D}=0.1\) to prevent it from dominating optimization; confidence and consistency use \(\lambda_{\rm unc}=1\) and \(\lambda_{\rm cons}=0.1\). Low-noise timesteps receive greater weight, capped at 3. Several full equations are damaged in the text cache, so only clearly verifiable mechanisms and coefficients are retained here rather than reconstructing uncertain formula details.

Human3.6M subjects S1, S5, S6, S7, and S8 are used for training, with S9 and S11 for testing. Training runs for 200 epochs using AdamW, an initial learning rate of \(4\times10^{-4}\), weight decay 0.1, learning-rate multiplication by 0.99 per epoch, and a total batch size of 600 across 12 RTX 4090 GPUs. The per-view spatial and temporal modules each have 2 layers, with joint embedding dimension 32. Diffusion training uses 1000 timesteps; the default deterministic DDIM configuration uses \(\eta=0\), 5 sampling steps, and 1 hypothesis.

Key Experimental Results¶

Main Results¶

MPJPE is the average Euclidean distance between predicted and ground-truth joints in mm; lower is better. P-MPJPE measures that error after Procrustes alignment and is also lower-is-better. Human3.6M rows below come from Table 1 and all use four views and 27 frames. Cross-dataset rows come from Table 2: both listed methods use 3-frame inputs, pretrain on Human3.6M, and fine-tune on the target dataset for 10 epochs. These are not zero-shot transfer results.

Dataset and protocol	Method	2D input / frames	MPJPE ↓ (mm)	P-MPJPE ↓ (mm)
Human3.6M	FusionFormer	CPN / 27	25.4	Not listed
Human3.6M	PoseIRM	CPN / 27	25.1	Not listed
Human3.6M	CoDePose	CPN / 27	24.8	Not listed
Human3.6M	FusionFormer	ViTPose / 27	15.1	Not listed
Human3.6M	Three-dimensional-only diffusion baseline	ViTPose / 27	13.5	Not listed
Human3.6M	CoDePose	ViTPose / 27	12.1	Not listed
HumanEva, target-domain fine-tuning	FusionFormer	Detector not listed in Table 2 / 3	15.4	12.7
HumanEva, target-domain fine-tuning	CoDePose	Detector not listed in Table 2 / 3	14.4	11.6
MPI-INF-3DHP, target-domain fine-tuning	FusionFormer	Detector not listed in Table 2 / 3	5.4	3.2
MPI-INF-3DHP, target-domain fine-tuning	CoDePose	Detector not listed in Table 2 / 3	4.2	2.6

With the same ViTPose input, error decreases by 3.0 mm relative to FusionFormer, a 19.9% relative reduction. The reduction against the 3D-only diffusion baseline is 1.4 mm, or 10.4%. Under CPN, the improvement over PoseIRM is only 0.3 mm, indicating dependence on detection quality: gains from switching detectors should not all be credited to coupling. HumanEva and MPI-INF-3DHP errors fall by 6.5% and 22.2% relative to FusionFormer, respectively, only under the stated fine-tuning protocol.

Ablation Study¶

The following results are from Table 3, all on Human3.6M with ViTPose. Changing the 2D weight to 1.0 removes the reduced weighting, not the 2D supervision itself.

Config	MPJPE ↓ (mm)	Increase over full model (mm)
Full model	12.1	0.0
No diffusion, single-pass deterministic regression	16.7	4.6
Three-dimensional-only diffusion baseline	13.5	1.4
2D weight changed to 1.0	12.4	0.3
No body-coordinate consistency	12.7	0.6
No confidence head	14.1	2.0

Table 4 shows how initialization affects low-step sampling. Two decimal places are retained here; the main result of 12.1 mm is a lower-precision presentation and does not conflict with the default 5-step result of 12.07 mm.

Initialization	1-step MPJPE ↓ (mm)	5-step MPJPE ↓ (mm)	20-step MPJPE ↓ (mm)
Symmetric: both coordinate components start from noise	22.15	15.31	12.10
Asymmetric: 2D starts from detections	12.15	12.07	12.06

Key Findings¶

Removing diffusion altogether causes the largest absolute degradation. Among the stabilizing components of the coupled design, removing the confidence head hurts most: its 14.1 mm error is worse than the 3D-only baseline's 13.5 mm, directly contradicting the interpretation that adding 2D to the state must help by itself.
Asymmetric initialization mainly saves sampling steps rather than changing the best high-step error. The initializations are close at 20 steps, but differ by 10.00 mm at 1 step.
Camera count remains important: Table 7 reports 12.1 mm with four views, 17.2 mm with three, and 21.4 mm with two. Learned priors do not eliminate the lack of observations in sparse-view settings.

Highlights & Insights¶

A 2D detection can be both a fixed condition and an updatable state without contradiction. The former anchors image evidence, while the latter lets denoising move beyond rigid obedience to erroneous observations.
Body-coordinate consistency compares intrinsic body structure rather than camera-coordinate values. It avoids requiring extrinsics while acknowledging unstable skeletal reference axes, making it a soft regularizer rather than a hard geometric guarantee.
Efficient low-step inference comes from useful initialization, not from making diffusion training unnecessary. Deterministic-regression and single-step DDIM results show why multi-noise-level training and inference iteration counts must be discussed separately.

Limitations & Future Work¶

Detection-distribution sensitivity remains. In Table 8, training under the CPN setting and testing with Student-T noise increases error from 24.8 mm to 32.7 mm; matching Student-T training and testing yields 25.7 mm. This supports distribution adaptation, not inherent robustness to arbitrary non-Gaussian noise.
Evidence in unconstrained settings is limited. Cross-dataset experiments include target-domain fine-tuning, while dynamic cameras and challenging real-world environments remain future directions identified by the authors. Jointly learned video representations and a separate 2D noise schedule are also directions proposed in the paper.
Reproduction details are incomplete. Some cached equations are damaged, the main text does not spell out the confidence-weighting connection, and the hip–spine axis direction is unclear between the prose and extracted equation. This note should not be treated as a line-by-line implementation specification.
Efficiency must be evaluated across the whole pipeline. Table 9 reports pose-lifting latencies of 12.97 ms for one step and 65.82 ms for five steps, but 2D detection may be the bottleneck. The comparison method's name in that table is questionable relative to the method associated with the same error in the main table, so no strict speed advantage over a named baseline is claimed here.

vs FusionFormer: Both fuse compact multi-view 2D sequences. CoDePose's central distinction is including 2D coordinates in a joint generative state, not simply replacing a fusion network; same-detector comparisons are the informative ones.
vs D3DP, DiffPose, and DDHPose: These diffusion pose methods retain 2D detections as external conditions and primarily update 3D hypotheses. CoDePose also updates the 2D state, but multi-view and monocular observation difficulties differ, preventing direct attribution of their absolute benchmark errors to this design alone.
vs triangulation-based multi-view methods: Explicit geometric methods use camera relationships to fuse observations. CoDePose's cross-view Transformer and body-coordinate regularizer do not require extrinsics, but their consistency relies more on learning than on exact projection constraints.

Rating¶

Novelty: 4/5. Promoting 2D detections from conditioning alone to a coupled state, together with low-step initialization, addresses a clearly defined problem.
Experimental Thoroughness: 4/5. Same-detector comparisons, component and initialization ablations, camera-count analysis, and heavy-tailed noise tests are substantial, though cross-domain results include fine-tuning.
Writing Quality: 3/5. The overall argument is clear, but the confidence connection and some table and equation details require clarification.
Value: 4/5. A useful approach to detection-error propagation in multi-view pose estimation, with benefits and scope tied to observation quality and deployment conditions.