XPos3R: Cross-Modal Transformer for Intraoperative 2D/3D Registration¶

Conference: ECCV 2026
Paper: ECCV 2026 poster page
Area: Medical Imaging / Intraoperative 2D/3D registration
Keywords: intraoperative 2D/3D registration, cross-modal transformer, DUSt3R, DRR synthesis, test-time optimization

TL;DR¶

XPos3R extends the DUSt3R-style feedforward geometry transformer to heterogeneous "2D X-ray + 3D CT" inputs: a pretrained RayDINO encodes the image, a 3D CNN trained from scratch encodes the volume, and a unidirectional image-to-volume cross-attention decoder directly regresses the intraoperative camera pose. A single pelvic pretraining works across patients, and paired with seconds of test-time optimization it reduces 3D error to about 3.3 mm and reprojection error to 0.45 mm, eliminating hours of per-patient preoperative retraining.

Background & Motivation¶

High-risk interventions such as pelvic trauma surgery rely on live intraoperative X-ray imaging to visualize anatomy and instruments, yet 2D projection inherently carries depth ambiguity that limits spatial reasoning; CT holds the full 3D anatomy but can hardly be acquired on demand during surgery because of hardware and safety constraints. Aligning the live X-ray with the preoperative volume — i.e., estimating the camera pose between them — is intraoperative 2D/3D registration, a foundational step of image-guided interventions. Classical intensity-based optimization aligns digitally reconstructed radiographs (DRRs) rendered from the CT with the real X-ray and has a high accuracy ceiling, but it is extremely sensitive to pose initialization; landmark-based methods solve the pose with PnP solvers but demand extensive expert annotation.

Regression methods attempt to predict the initial pose directly from the X-ray, yet to reach sufficient accuracy almost all of them take the per-patient training route: given a patient's CT, sample many poses and render DRRs to train a 2D CNN. DiffPose takes roughly 12 hours to train from scratch; XVR finetunes from a patient-agnostic pretrained model and cuts preparation to 5 minutes, but its input is still 2D only, never explicitly touching the patient's 3D anatomy — the cross-subject error distributions in this paper show such 2D-only models have large variance and many outliers, behaving more like they memorize 2D appearance than understand 3D anatomy. To date no regression model can be applied across patients without any preoperative preparation, which has long left emergency settings unsolved.

Geometry foundation models offer a fresh reference: thanks to transformers and large-scale training, DUSt3R-style models generalize across scenes on reconstruction, tracking, and pose estimation. Porting this directly hits two obstacles: first, geometry foundation models take homogeneous inputs (image–image pairs), while here we have heterogeneous paired inputs of 2D X-rays and 3D volumes, calling for new architectures and feature-alignment strategies; second, medical data are far scarcer than natural images and million-scale training sets are unavailable. Since medical practice spans more than 77,000 procedure codes, a unified all-anatomy model is both data-starved and hard to reconcile with per-procedure regulation, so the authors take a pragmatic setting: fix one anatomical region (pelvis) and train a model that generalizes across patients within it. Core idea: recast intraoperative registration from "single-view pose regression" to "image–volume pairwise inference" — a pretrained X-ray ViT supplies semantically rich image tokens that query a volume memory encoded by a 3D CNN trained from scratch, while hundreds of CTs are turned into 2.3 million training triplets through pose sampling and DRR rendering, achieving DUSt3R-style feedforward registration that crosses patients with zero preoperative preparation.

Method¶

Overall Architecture¶

Problem formulation: given a preoperative CT volume \(V\) and a 2D X-ray \(I\) acquired at an unknown pose \(T=[R|t]\in SE(3)\), X-ray image formation is described by a projection operator \(P(T): V \to I\) under a pinhole model with known intrinsics, and registration amounts to estimating \(T\) from \((I, V)\). XPos3R trains a feedforward model \(F\) that outputs \(\hat{T}=F(I,V)\) directly, optionally refined by test-time optimization (TTO) within seconds.

The network follows DUSt3R's encoder–decoder–head skeleton but reworks both ends for cross-modality: on the image side, RayDINO pretrained on X-rays encodes a 224×224 X-ray into 196 image tokens; on the volume side, a four-stage 3D CNN encodes a 96³ CT into 1,728 volume tokens plus a globally pooled patient-level embedding; the decoder applies self-attention only to the image tokens, then lets them cross-attend to the volume tokens as keys and values, producing image-aligned geometry tokens; the regression head concatenates the geometry tokens with the patient embedding and uses two MLPs to predict axis-angle rotation and translation, mapped back to SE(3). At TTO, the differentiable renderer DiffDRR iteratively optimizes NCC similarity to align the DRR with the real X-ray.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    I["2D X-ray"] --> E1["Asymmetric dual-modal encoding<br/>RayDINO image encoder"]
    V["3D CT volume"] --> E2["Asymmetric dual-modal encoding<br/>3D CNN volume encoder"]
    E2 --> G["Globally pooled patient embedding"]
    E1 --> D["Unidirectional image-to-volume cross-attention<br/>image tokens query volume tokens"]
    E2 --> D
    G --> H["Pose regression head<br/>two MLPs output rotation and translation"]
    D --> H
    H -->|Feedforward pose, optional| T["Test-time optimization<br/>DiffDRR intensity optimization, seconds"]
    T --> O["Refined pose<br/>TRE about 3.3 mm"]

Key Designs¶

1. Asymmetric dual-modal encoding: a pretrained image ViT and a 3D CNN trained from scratch

DUSt3R-style models process homogeneous inputs with a shared-weight Siamese encoder, whereas X-ray and CT are heterogeneous modalities that force a dual-encoder split — which makes "what backbone each side uses" a choice that directly determines cross-modal alignment quality. On the image side the paper picks RayDINO, a ViT self-supervisedly pretrained on large-scale X-ray data that yields dense, semantically rich representations; a 224×224 input produces 196 image tokens. On the volume side it does not reuse a pretrained 3D segmentation backbone but trains a compact four-stage Conv3D hierarchy from scratch (kernel 3, strides 1/2/2/2), extracting multi-scale feature grids that are linearly projected and flattened into 1,728 volume tokens; global average pooling over the feature grid additionally yields a one-dimensional patient-level embedding that serves as complementary context at decoding time.

Training the volume encoder from scratch is the paper's most counter-intuitive decision, and the ablation supports it most directly: pretrained 3D segmentation encoders (e.g., SuPreM) are optimized to detect specific organs and learn sparse, localized representations that misalign with the dense semantic features on the image side, dragging down the subsequent cross-attention decoding; swapping in a compact 3D CNN trained from scratch lets volume features align toward the high-quality image features instead. The numbers are stark — SuPreM's TRE is 74.03 mm, three times the from-scratch 3D CNN's 24.16 mm. The image side tells the mirror story: X-ray-domain-pretrained RayDINO beats both generic DINOv2 (TRE 29.77 mm) and training from scratch (36.72 mm). The value of pretraining lies in domain-adapted semantic density, not in "apply pretraining everywhere".

2. Unidirectional image-to-volume cross-attention: the volume as memory, the image as query

Volume tokens (1,728) vastly outnumber image tokens (196); copying DUSt3R's self-attention on both branches plus reciprocal cross-attention would concentrate compute on the volume side with questionable benefit. XPos3R's decoder is therefore unidirectional: 4 layers of self-attention on image tokens only, followed by 2 layers of cross-attention where image tokens query volume tokens:

\[ G=\text{CrossAttn}\big(\text{SelfAttn}(F_{\text{xray}}),\,F_{\text{vol}},\,F_{\text{vol}}\big), \]

with the query from image tokens and keys/values from volume tokens, outputting geometry tokens aligned with the X-ray position by position. Three reasons back the design: volume-side self-attention and the reverse volume-to-image attention are both costly; the 3D CNN's multi-scale hierarchy already encodes local-global structure, making extra self-attention of limited use; and since the task is to estimate the X-ray's pose, treating the volume as a key–value memory that image tokens actively probe is semantically the natural direction. The regression head concatenates the geometry tokens with the patient embedding and feeds two separate 3-layer 1,024-channel MLPs that output axis-angle rotation \(r\in\mathbb{R}^3\) and translation \(t\in\mathbb{R}^3\); the Lie-algebra representation \(\xi=(r,t)\in se(3)\) is then mapped back to the Lie group \(T\in SE(3)\).

3. Anatomy-specific million-scale synthetic data: scalability from pose diversity

DUSt3R's generalization comes from million-scale calibrated image pairs that the medical domain cannot provide, and more than 77,000 procedure codes also make "one model for all anatomies" unrealistic. The paper therefore converts the training-scale problem into an in-region data synthesis problem: collect pelvic CTs from the public CTPelvic1K, filter down to 461 scans (441 training / 20 validation), rigidly preregister each to a canonical orientation with ITK-SNAP using a bone segmentation template; then, for each volume, sample 5,000 random C-arm poses within a plausible intraoperative acquisition range, render the corresponding DRRs with DiffDRR, and apply random contrast augmentation to shrink the synthetic-to-real gap — 2.3 million volume–DRR–pose triplets in total, with camera intrinsics matching the evaluation benchmark.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Public pelvic CTs<br/>461 scans after filtering CTPelvic1K"] --> B["3D/3D preregistration<br/>unify to canonical bone-template orientation"]
    B --> C["Pose sampling and DRR rendering<br/>5,000 C-arm poses per volume"]
    C --> D["2.3M volume-DRR-pose triplets"]

The pivotal judgment here is that within one anatomical region, X-ray appearance varies far more with viewpoint than with inter-patient anatomical differences, so spending the synthesis budget on pose diversity (5,000 renders per volume) beats chasing more patients — that is, \(M_{\text{xray}} \gg M_{\text{vol}}\). Multiplying 441 training volumes by 5,000 gives about 2.2 million, while the paper reports 2.3 million; we suspect the 20 validation volumes are included in the rendering statistics, and we defer to the paper's figure. Preregistration to the canonical orientation compresses the pose search space so that the sampling distribution covers the views actually seen intraoperatively.

Loss & Training¶

Training uses the standard pose regression objective: a Huber loss on translation and a geodesic loss on rotation, \(\mathcal{L}_{\text{rot}}\propto\arccos\big(\frac{\operatorname{trace}(R_{\text{pred}}R_{\text{gt}}^{\top})-1}{2}\big)\), with the angular error scaled by the focal length to match the translation error's magnitude (exact prefactor ⚠️ refer to the original paper). The overall loss is

\[ \mathcal{L}_{\text{pretrain}}=\lambda_{\text{rot}}\mathcal{L}_{\text{rot}}+\lambda_{\text{trans}}\mathcal{L}_{\text{trans}},\qquad \lambda_{\text{rot}}=0.1,\ \lambda_{\text{trans}}=100. \]

The formula typesetting in our source is corrupted, so we keep only the structure and weight values the main text confirms. Training uses AdamW with a cosine schedule (peak 1×10⁻⁴, 5-epoch warm-up), 100 epochs, a global batch of 70, pretrained on 8 H100 GPUs for about 5 days.

Test-time optimization follows XVR's default settings: starting from XPos3R's feedforward prediction, DiffDRR iteratively updates the pose with multiscale NCC and gradient NCC as energy terms — NCC being the mean per-pixel product of two images each normalized as \(Z=(I-\mu(I))/\sigma(I)\). Because computing the NCC loss requires costly image rendering, it is deliberately excluded from pretraining, leaving the rendering budget for the seconds-long intraoperative refinement.

Key Experimental Results¶

Main Results¶

Training uses 461 filtered pelvic CTs from CTPelvic1K (441 training / 20 validation); evaluation runs on DeepFluoro — a pelvic 2D/3D registration benchmark of 6 cadaver subjects, each with 1 CT and 24–111 X-rays acquired by a Siemens CIOS Fusion mobile C-arm (30×30 cm² detector, 1,020 mm focal length), providing manually annotated 3D fiducial landmarks and ground-truth extrinsics for every X-ray; the evaluation CTs are likewise preregistered to the canonical bone template. Metrics cover pose-based rotation error (RE) and translation error (TE), fiducial-based target registration error (TRE) and projection error (PE), plus efficiency: preoperative preparation time \(T_{\text{preo}}\) (per-patient training or annotation) and per-image intraoperative time \(T_{\text{intra}}\). All accuracy metrics are reported per X-ray as mean ± standard deviation.

The table below is excerpted from Table 1 of the paper, grouped by paradigm; XPos3R's TTO uses DiffDRR as the optimization backbone.

Paradigm	Method	Preoperative prep	\(T_{\text{intra}}\) (s)	RE (°) ↓	TE (mm) ↓	TRE (mm) ↓	PE (mm) ↓
Feedforward regression	DiffPose	12 hours	0.15	3.347 ± 2.19	43.93 ± 27.79	43.88 ± 27.65	7.714 ± 5.10
Feedforward regression	XVR (pretrained)	–	0.16	6.742 ± 2.10	90.65 ± 26.82	90.61 ± 26.97	10.60 ± 3.52
Feedforward regression	XVR (finetuned)	5 minutes	0.15	2.587 ± 1.58	34.14 ± 21.13	33.90 ± 20.74	3.860 ± 2.08
Feedforward regression	XPos3R (pretrained)	–	1.52	1.352 ± 0.56	24.06 ± 6.19	24.16 ± 6.24	7.412 ± 3.11
Optimization-only	DiffDRR	–	4.22	18.52 ± 25.35	198.6 ± 227.46	200.2 ± 229.58	29.87 ± 50.52
Optimization-only	Regi2D3D-1	Annotation	9.03	3.594 ± 18.65	37.38 ± 177.60	37.21 ± 176.72	6.081 ± 25.35
Optimization-only	Regi2D3D-2	Annotation	18.57	6.387 ± 16.31	77.26 ± 183.46	76.42 ± 181.20	16.60 ± 43.70
Optimization-only	RayEmb	–	12.19	0.818 ± 1.53	10.84 ± 20.48	10.75 ± 20.19	2.174 ± 1.48
Feedforward + TTO	DiffPose + TTO	12 hours	1.55	0.277 ± 1.17	3.854 ± 15.37	3.810 ± 14.95	0.473 ± 1.20
Feedforward + TTO	XVR (pretrained) + TTO	–	1.56	0.407 ± 1.70	5.594 ± 22.54	5.553 ± 22.29	0.561 ± 1.45
Feedforward + TTO	XVR (finetuned) + TTO	5 minutes	1.39	0.277 ± 1.17	3.856 ± 15.36	3.811 ± 14.94	0.475 ± 1.20
Feedforward + TTO	XPos3R (pretrained) + TTO	–	2.93	0.236 ± 0.65	3.313 ± 8.45	3.288 ± 8.31	0.448 ± 0.94

In the feedforward setting, a single pretrained XPos3R is the best overall regression method: it cuts RE and TE by 5.39° and 66.59 mm against XVR (pretrained), and against the patient-specific DiffPose and XVR (finetuned) it delivers lower pose errors across the board and the best TRE, with only the 2D fiducial metric PE (7.412 mm) still clearly behind XVR (finetuned)'s 3.860 mm — the sole metric where per-patient methods keep their feedforward edge. More telling is the dispersion: XPos3R's RE standard deviation of 0.56 is under one third of the other regression methods' (1.58–2.19), and it posts the lowest TRE median and fewest outliers across all six cadavers, whereas DiffDRR without a learned initialization reaches a TRE of about 200 mm, underscoring the value of initialization.

With TTO enabled, XPos3R is best on all four accuracy metrics: RE drops to 0.236° (sub-degree), TE/TRE to about 3.3 mm, and PE to 0.448 mm (sub-millimeter) — matching the abstract's "<4 mm 3D error, <1 mm reprojection error". Patient-specific methods reach comparable means after finetuning (DiffPose + TTO's TRE is 3.810 mm) but with far larger variance (TE standard deviation 15.37 vs. 8.45), meaning their initial predictions are unstable and optimization converges less reliably. On efficiency, XPos3R needs no preoperative preparation at all (DiffPose needs 12 hours), its TTO finishes one X-ray in 2.93 seconds — faster than Regi2D3D's 9–19 seconds and RayEmb's 12.19 seconds — and its 1.52-second feedforward inference is a price worth paying next to the 0.15 seconds of 2D CNNs.

Ablation Study¶

Table 2 of the paper ablates the encoders and the patient embedding (DeepFluoro, feedforward), and Table 3 compares the pretrained model against patient-specific finetuning.

Ablation axis	Config	RE (°)	TE (mm)	TRE (mm)	PE (mm)
2D encoder	DINOv2 (generic pretraining)	2.041	29.95	29.77	11.54
2D encoder	Train from scratch	2.480	37.09	36.72	11.38
2D encoder	RayDINO (ours)	1.352	24.06	24.16	7.412
3D encoder	SuPreM (3D segmentation transformer)	2.607	73.82	74.03	13.43
3D encoder	3D CNN (ours)	1.352	24.06	24.16	7.412
Patient embedding	Without global embedding	2.012	32.33	32.58	9.709
Patient embedding	With global embedding (ours)	1.352	24.06	24.16	7.412

Training regime	Stage	RE (°)	TE (mm)	TRE (mm)	PE (mm)
Pretrained (cross-patient)	Feedforward	1.352	24.06	24.16	7.412
Patient-specific finetuned	Feedforward	0.980	10.61	10.56	2.968
Pretrained (cross-patient)	After TTO	0.236	3.313	3.288	0.448
Patient-specific finetuned	After TTO	0.234	3.302	3.275	0.449

Swapping the volume encoder is the costliest change: replacing the 3D CNN with SuPreM degrades TRE from 24.16 to 74.03 mm, confirming the misalignment of pretrained segmentation representations with image features; on the image side, replacing RayDINO with generic DINOv2 or a from-scratch encoder degrades TRE to 29.77 and 36.72 mm respectively, showing that domain-adapted pretraining matters more than the binary "pretrained vs. scratch" choice. Removing the global patient embedding pushes TRE to 32.58 mm, so this nearly free piece of context is not redundant. Patient-specific finetuning lifts feedforward TRE from 24.16 to 10.56 mm, yet after TTO the two nearly coincide (3.288 vs. 3.275 mm) — the bottleneck is the optimization algorithm itself (limited by image resolution and similar factors), not the initial pose quality; finetuning pays off only when higher feedforward accuracy is clinically required.

Key Findings¶

3D volume information is the source of cross-patient generalization. 2D-only methods show large variance and many outliers across six cadavers, the signature of memorizing 2D appearance; attention visualization shows XPos3R's strongest cross-modal attention concentrates on the 3D regions geometrically covered by the X-ray, indicating the model learns 2D–3D geometric relations rather than appearance memory.
Pretrained representations need domain adaptation, not universality. The X-ray-pretrained RayDINO beats generic DINOv2, while the 3D side is exactly reversed — a pretrained segmentation encoder loses to a compact CNN trained from scratch; alignment quality across modalities matters more than the strength of either side alone.
Per-patient finetuning's gains are absorbed by TTO. Before TTO, finetuning helps substantially (TRE 24.16 → 10.56 mm); after TTO the gap shrinks to 0.013 mm. If the clinical workflow already includes seconds-long intensity optimization, cross-patient pretraining suffices.
The efficiency triangle can be had together. No preoperative preparation (versus 12 hours), 1.52-second feedforward, 2.93-second TTO; the only accuracy concession is PE behind finetuned methods in the feedforward stage, and after TTO XPos3R leads every competitor.

Highlights & Insights¶

"Which side queries, which side is memory" should follow task semantics, not symmetry inertia. The task estimates the X-ray's pose, so image tokens query the volume memory — this saves the self-attention over 1,728 volume tokens and makes the output naturally aligned with the X-ray position by position, the most consequential decision packed into the word "asymmetric".
"Pretrain vs. from scratch" should be decided per modality. The same model uses domain-adapted pretraining on the image side and from-scratch training on the volume side, with two opposite choices both backed by ablations; treating "pretrained representations are more general" as the default assumption backfires in cross-modal settings.
The axis of data scaling can be swapped. Medical data lack patients, but DRR rendering makes "pose sampling" nearly free — moving scalability from patient diversity (441 cases) to pose diversity (5,000 per case) is a general recipe for DUSt3R-scale pretraining in medicine, transferable to any anatomy whose DRRs can be rendered.
Regression for robustness, optimization for accuracy. The regressor supplies a low-variance initialization that lets intensity optimization converge reliably, and optimization supplies the millimeter accuracy regression lacks — a two-tier division of labor that fits the real-time and safety constraints of the operating room better than one end-to-end stage.

Limitations & Future Work¶

Limitations the authors admit: validated on the pelvis only; regions such as cerebral vessels cannot be trained yet due to scarce public vessel data; CT is chosen because CT and X-ray share imaging physics and allow large-scale DRR synthesis, whereas extending to MRI or ultrasound would need modality translation first (e.g., pseudo-CT generation).
Evaluation scale and sample granularity. DeepFluoro has only 6 cadavers (24–111 X-rays each), and accuracy is aggregated per X-ray rather than per patient, leaving patient-level confidence unknown; camera intrinsics are fixed and consistent with training, and robustness to intrinsics mismatch or template preregistration mismatch is not reported.
Hidden dependence on the synthetic-to-real gap. The training distribution is random poses within "canonical template orientation + plausible acquisition range"; behavior outside that range, or under extreme pathology and metal implants, is untested. The 2.3 million triplets are rendered from 441 training volumes, so anatomical diversity remains the bottleneck (the paper reports 2.3M while 441×5,000 is about 2.2M; we suspect the 20 validation volumes are included in the count).
Inference cost and engineering prerequisites. Feedforward inference takes 1.52 seconds, an order of magnitude slower than 2D CNNs, and TTO requires differentiable rendering and a GPU; the 3D/3D preregistration depends on the quality of the bone segmentation template, and the impact of template mismatch is not quantified.
Improvement directions. Extend to other anatomies and non-CT modalities; compress or distill the volume encoder to cut feedforward latency; distill NCC-style intensity consistency into feedforward training to reduce reliance on TTO.

vs. DiffPose / XVR: both regress the pose directly from the X-ray with 2D CNNs and require per-patient training (12 hours / 5 minutes); XPos3R takes joint 2D+3D input, runs a single model across patients with no preparation, at the cost of a larger model and slower feedforward, with PE still behind XVR's finetuned version in the feedforward stage.
vs. DiffDRR / Regi2D3D / RayEmb: intensity-based optimization is hostage to initialization (DiffDRR without one reaches a TRE of about 200 mm), and landmark pipelines need preoperative expert annotation at 9–12 seconds per X-ray; XPos3R supplies a strong initialization that lets the same optimizer converge within 3 seconds to the best overall accuracy.
vs. DUSt3R / MASt3R / VGGT geometry foundation models: they process homogeneous image pairs and output point maps or tracks; XPos3R is the first to extend the paradigm to heterogeneous image–volume pairs with direct pose regression, and its asymmetric encoder–decoder can serve as a template for image–point-cloud and image–volume cross-modal geometry tasks.
vs. Pos3R (object 6D pose): both descend from DUSt3R, but Pos3R targets 6D poses of unseen objects from images; XPos3R targets cross-modal medical registration, additionally solving modality mismatch and medical data scarcity, two problems geometry foundation models rarely face.
Transferable insight: a medical "geometry foundation model" need not be one model for all anatomies — training per anatomical region and generalizing across patients within the region is a feasible compromise under data and regulatory realities; this setting choice itself is worth borrowing for similar tasks.

Rating¶

Novelty: 4/5. First to extend the DUSt3R-style feedforward geometry paradigm to 2D/3D heterogeneous modalities, with an asymmetric encoder–decoder and a pose-diversity data strategy that clearly target the problem — though the components are all combinations of mature techniques.
Experimental Thoroughness: 4/5. Compares against 12 methods on a real cadaveric benchmark with complete ablations over architecture, encoders, patient embedding, and finetuning, plus attention visualization and efficiency analysis; limited to one anatomical region and 6 subjects.
Writing Quality: 4/5. The motivation–design–ablation chain is clear and the numbers self-consistent; several formulas in our source are corrupted, and a few details (the geodesic loss prefactor) cannot be verified.
Value: 4/5. A ready-to-use registration without per-patient preparation directly matters for emergency and routine surgical navigation, and the data synthesis recipe plus the asymmetric cross-modal architecture transfer to broader vision geometry tasks.