UniStitch: Unifying Semantic and Geometric Features for Image Stitching

Conference: ECCV2026
Paper: ECCV Paper
Project: UniStitch
Area: Multimodal VLM (current filing); the research task is image stitching
Keywords: image stitching, geometric features, semantic features, mixture of experts, free-form deformation

TL;DR

UniStitch encodes sparse keypoints into grids aligned with image features and uses adaptive expert fusion to drive global-to-local stitching warps, reaching 25.07 dB / 0.813 on UDIS-D while reducing high-resolution warping memory through low-resolution TPS and B-spline reconstruction.

Background & Motivation

Image stitching places overlapping views on a shared canvas while avoiding ghosting and structural distortion. Traditional methods detect and match keypoints such as SIFT, then estimate a homography or elastic warp from the correspondences; their advantage is explicit geometric evidence without having to learn scene understanding from training data first. However, reliable matches can be scarce under weak texture, poor illumination, or repetitive structures, and incorrect correspondences can directly corrupt the estimated warp. Learning-based methods instead extract dense features from image pairs and learn global homographies plus local thin-plate spline transformations, often performing better within a training distribution such as UDIS-D. The difficulty is that exploiting image context does not guarantee stable geometric constraints across scenes, so deep methods need not outperform traditional ones on classical stitching datasets.

UniStitch addresses the failure of these two sources of evidence to cooperate in a shared representation, rather than simply seeking a larger semantic backbone. Treating keypoints as another pixel channel is not straightforward: point sets are unordered, sparse, and variable in size, whereas convolutional features occupy fixed two-dimensional grids. Even after resolving this shape mismatch, simple addition can let an unreliable modality interfere with the other branch, especially when both branches are consistently useful during training. The paper therefore first establishes how coordinates and descriptors enter a grid, then learns how to choose between unimodal independence and cross-modal complementarity. Here, "multimodal" refers to geometric and semantic representations of visual inputs, not image-language inputs or a language model.

A further practical obstacle arises during the final transformation: a high-resolution canvas creates large intermediate matrices for standard TPS, so resampling can run out of memory even when the feature network fits. The authors address representation fusion and warp computation separately, improving both the evidence used to estimate deformation and the cost of applying it. Core Idea: convert keypoints into spatially aligned geometric feature maps, fuse independent and joint experts according to reliability, and perform high-resolution stitching with low-resolution TPS followed by local B-spline reconstruction.

Method

Overall Architecture

The inputs are a reference image, a target image, and their keypoint coordinates and descriptors; the output is a stitched image on a shared canvas. The semantic branch uses ResNet-18, while the geometric branch uses neural point transformation to produce spatially compatible maps at \(1/8\) and \(1/16\) of the original resolution. Adaptive experts then fuse semantic and geometric information for each image at each scale, supplying features for subsequent correlation computation and warp regression. The lower-resolution fused features support global homography estimation, followed by finer-scale information for local control-point residuals. The global transformation projects both images onto a virtual middle plane, and local warping corrects residual parallax rather than simply forcing the target image onto the reference view. Training additionally uses latent-space modality robustification; inference requires no random dropout or noise, while cross-domain iterative adaptation is optional post-processing reported separately.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image pair and matched keypoints"] --> Alignment["Point-to-grid alignment<br/>NPT + ResNet-18"]
    Alignment --> Fusion["Reliability-adaptive fusion<br/>AMoE"]
    Robustifier["Training only: latent-space<br/>modality robustifier MR"] -.-> Fusion
    Fusion --> Warp["Memory-efficient global-to-local warping<br/>Homography + FFD-TPS"]
    Warp --> Output["Shared canvas and stitched result"]

Key Designs

1. Point-to-grid alignment: preserve keypoint evidence while connecting it to dense regression

The Neural Point Transformer (NPT) follows a transformation-then-projection strategy, rather than merely drawing a binary map of sparse coordinates. Each input point includes a two-dimensional position and a descriptor; PointNeXt first encodes and downsamples the set into high-dimensional point features with local context. The geometric branch therefore conveys more than the presence of a corner: it represents distinguishable nearby structure, with descriptors providing appearance evidence that coordinates alone cannot express. The semantic branch independently processes the complete image, so dense image information remains available where geometric points provide no coverage. Fusion has not yet occurred at this stage, allowing each branch to retain an encoder suited to its input.

NPT then projects the encoded features into a zero-initialized grid according to point positions, matching the resolution of the corresponding semantic feature map. At scale \(1/8\), a grid cell covers an \(8\times8\) pixel region in the original image; at \(1/16\), it covers a \(16\times16\) pixel region. Multiplying coordinates by the scale and taking the floor assigns cells, and channel-wise max pooling aggregates multiple points within one cell. Cells receiving no projected points retain their initial zero values, so "dense" describes the regular tensor layout rather than recovered geometric observations at every location. This conversion lets convolution and fusion operate at corresponding spatial positions without treating input point order as meaningful evidence. It also helps explain why shuffling point order barely changes the results in Table 5, although that experiment does not establish robustness to every possible perturbation.

Geometric input quality still matters: the paper compares raw points with matched points and finds the latter more useful, rather than claiming that more points are always better. SIFT, SURF, and ORB use KNN matching, while SuperPoint uses SuperGlue or LightGlue; the selected combination is SuperPoint with LightGlue. Matching first identifies more credible cross-view correspondences, and the network then learns how to use them, so the model changes how matching evidence enters warp regression rather than eliminating matching. In Table 8, SuperPoint + LightGlue with descriptors achieves 24.91 / 0.808, compared with 24.87 / 0.807 using coordinates alone, demonstrating a descriptor benefit in that ablation setting. Those values belong to the geometric-feature comparison and should not be substituted for the final full-model result of 25.07 / 0.813.

2. Reliability-adaptive fusion: allow independent judgments while learning cross-modal complementarity

The Adaptive Mixture of Experts (AMoE) contains three residual-based experts: semantic, geometric, and heterogeneous. The first two process their own modalities independently, while the heterogeneous expert processes concatenated features to learn relationships requiring both sources of evidence. A linear gated router reads modality information and uses Softmax to produce three normalized weights, which aggregate the expert outputs. Unlike forcing every feature through one convolutional fusion block, this design retains a path for an independent representation when the other branch is unreliable. The following is an equivalent shorthand based on Eqs. (4)-(6) and the accompanying text on page 7, not a literal transcription of the damaged equation layout:

\[ F_{\mathrm{fused}}=\sum_{\mathrm{expert}\in\{s,g,h\}}w_{\mathrm{expert}}H_{\mathrm{expert}},\qquad w_{\mathrm{expert}}\geq0,\quad\sum_{\mathrm{expert}\in\{s,g,h\}}w_{\mathrm{expert}}=1. \]

Here, \(s,g,h\) denote the semantic, geometric, and heterogeneous experts, and \(H\) denotes expert outputs rather than raw input images. Fusion operates at the corresponding scales; the paper describes a three-dimensional weight vector, which should not be expanded into an unspecified per-pixel confidence map. All experts participate in weighted aggregation, and the paper does not describe this module as sparse large-model routing that activates only Top-k experts.

Gating alone is insufficient because consistently complete, paired training features can encourage dependence on a fixed pattern of modality co-occurrence. The Modality Robustifier (MR) freezes the feature-alignment branches during the second training stage and applies random dropout or Gaussian noise to expert-branch features in latent space. The dropout and noise probabilities are both 0.25, with perturbations spanning multi-scale features in semantic, geometric, and heterogeneous branches. The authors place these perturbations in latent space to avoid disrupting the established spatial correspondence between original pixels and keypoints. MR is therefore a training component of the fusion design, not a separate repair network required at inference time. Its effect can be checked in Table 3: adding MR raises Classic performance from 18.56 / 0.581 to 18.80 / 0.596, with simultaneous in-domain gains.

3. Memory-efficient global-to-local warping: separate expensive TPS evaluation from output pixel count

The fused features first predict a global homography, which is decomposed into separate reference-image and target-image transformations following StabStitch++. Both images map to a virtual middle plane; the global transformations initialize local TPS control-point offsets, and subsequent regression estimates the remaining offsets. This global-to-local process first removes large viewpoint differences and then handles local parallax that a single homography cannot explain. For comparison, the authors adapt the spatial warp of StabStitch++ to image stitching and use the same number of control points as UniStitch, reducing differences in deformation capacity as a confounder.

Standard Thin-Plate Spline (TPS) warping evaluates many output coordinates, producing intermediate caches that grow with resolution and become a high-resolution memory bottleneck. The proposed approach compresses the output coordinate grid, evaluates TPS at low resolution, and uses Free-Form Deformation (FFD) to reconstruct the full-resolution displacement field. The low-resolution grid dimensions are empirically set to twice the TPS control-point resolution; this is not simply a blurred enlargement of the final output image. Reconstruction uses the local support of cubic B-splines, so each output location depends only on a \(4\times4\) neighborhood of the low-resolution displacement field. The expensive TPS computation consequently handles fewer coordinates, while high-resolution processing primarily performs local interpolation and image resampling. This is approximate evaluation, not mathematical identity with full-resolution TPS; Table 7 does show small metric changes. For example, at \(1329\times2000\), mPSNR changes from 17.14 to 17.12 while memory falls from 13.99 GB to 8.57 GB. The appropriate conclusion is lower cost for a small alignment difference, not strictly lossless behavior for every resolution and image.

Loss & Training

The objective includes content alignment, shape preservation, and expert regularization; the first two follow StabStitch++ to constrain overlapping content and structural plausibility of the deformation. Expert regularization constrains routing weights only at the \(1/16\) scale; the text describes a variance term for balanced utilization and an entropy-related term associated with expert selection. One reproducibility issue remains: the cached Eq. (10) has damaged operators, and the text describes negative entropy as promoting selectivity, whose actual optimization direction requires the exact signs. This note therefore does not reconstruct that regularizer; the page 8 hyperparameter list also uses \(w_b\) where the preceding objective uses \(w_r\). The reported values associated with the shape term, regularization term, and entropy coefficient are 10, 0.01, and 0.1, but the regularization-weight name needs verification against the original PDF or implementation. The cache contains no appendix, so complete definitions of the alignment and shape-preservation losses cannot be recovered from this main text alone.

Training uses the UDIS-D training split; stage one trains from scratch for 100 epochs without MR. Stage two runs for another 100 epochs with multimodal feature-alignment branches frozen and MR enabled, allowing the downstream network to adapt to partial feature failure. Experiments use one NVIDIA RTX 5090, batch size 8, an initial learning rate of \(1\times10^{-4}\), and an exponential decay scheduler. Ordinary inference retains keypoint extraction and matching, dual-branch encoding, AMoE, and warping, without training perturbations. Starred Classic results additionally use iterative adaptation from UDIS++ as post-processing and therefore do not have the same computational budget as unstarred forward predictions.

Key Experimental Results

Main Results

The table retains average metrics from Tables 1 and 2 on page 10; mPSNR is measured in dB and mSSIM is dimensionless, with higher values better for both. The UDIS-D test set contains 1,106 image pairs, and Classic comprises 147 cross-domain samples collected in RopStitch; the metrics measure average alignment quality in overlapping regions. The authors use average fusion consistently for visual comparisons so that ghosting and blur expose geometric misalignment rather than being hidden by stronger blending post-processing.

Method and setting	UDIS-D mPSNR	UDIS-D mSSIM	Classic mPSNR	Classic mSSIM
APAP	22.39	0.726	18.92	0.628
UDIS++	23.41	0.755	17.36	0.500
StabStitch++, adapted spatial warp	24.63	0.797	17.91	0.534
RopStitch, no iterative adaptation	24.70	0.800	18.44	0.568
UniStitch, no iterative adaptation	25.07	0.813	18.80	0.596
RopStitch*, iterative adaptation	Not listed	Not listed	19.74	0.645
UniStitch*, iterative adaptation	Not listed	Not listed	20.68	0.692

The gain over RopStitch on UDIS-D is 0.37 dB / 0.013; with adaptation applied to both methods, the Classic gain is 0.94 dB / 0.047. However, unadapted UniStitch remains below APAP's 18.92 / 0.628 on Classic, so the cross-domain advantage should not be summarized as outperforming every traditional method without adaptation.

Ablation Study

The following table comes from Table 3 on page 12; both result columns report mPSNR / mSSIM, and Classic has no iterative-adaptation star.

Config	UDIS-D	Classic
Geometric point branch only	24.22 / 0.784	18.02 / 0.553
Image branch only	24.63 / 0.797	17.91 / 0.534
Dual branches + AMoE	24.94 / 0.808	18.56 / 0.581
Dual branches + AMoE + MR	25.07 / 0.813	18.80 / 0.596

The geometric branch alone is weaker in-domain but slightly stronger cross-domain, directly supporting different operating strengths rather than complete replacement of one feature type by the other. MR adds 0.13 dB / 0.005 on UDIS-D and 0.24 dB / 0.015 on Classic over AMoE alone. Table 6 on page 13 also reports only 17.22 / 0.510 for simple addition on Classic, below both single branches, showing that more inputs do not automatically yield complementarity.

The next table comes from Table 7 on page 14, with time averaged over 10 rounds and peak GPU memory recorded; quality scores across resolutions should not be ranked as equally difficult samples.

Resolution	Warp	Time (s)	Peak GPU memory (GB)	mPSNR	mSSIM
566 × 800	Vanilla TPS	0.141	4.81	16.85	0.776
566 × 800	FFD-TPS	0.132	2.76	16.86	0.776
1329 × 2000	Vanilla TPS	0.202	13.99	17.14	0.475
1329 × 2000	FFD-TPS	0.177	8.57	17.12	0.474
2448 × 3264	Vanilla TPS	Not completed	OOM	Not measured	Not measured
2448 × 3264	FFD-TPS	0.386	22.62	22.71	0.627

Key Findings

• At \(566\times800\), FFD-TPS reduces peak memory from 4.81 GB to 2.76 GB; this validates compression in warping, not an identical proportional reduction in every system cost.
• Reliable dual-branch fusion and MR drive the full model's accuracy gains, while FFD-TPS separately enables high-resolution execution; these contributions should be evaluated independently.
• Better matching and descriptors remain useful, indicating that learned fusion does not eliminate dependence on geometric front-end quality.

Highlights & Insights

• Encoding point sets before projection preserves processing suited to sparse data while connecting it to established dense stitching networks; spatial correspondence matters more than adding an input channel.
• Keeping unimodal and joint experts allows fusion to reject harmful interactions, while MR makes the model exercise that fallback during training.
• FFD-TPS separates deformation-representation complexity from output pixel density, suggesting a transferable computation strategy for other control-point-driven image transformations, although such transfer is not experimentally established.

Limitations & Future Work

• The authors provide no dedicated limitations section; the main text acknowledges standard TPS's memory bottleneck, yet the proposed method still needs 22.62 GB at the highest resolution and is not thereby established as suitable for low-memory devices.
• Table 2 shows that cross-domain rankings depend on iterative adaptation; deployment comparisons should include adaptation iterations, additional latency, and failure cases rather than final quality alone.
• Testing on 147 Classic samples provides useful OOD evidence but cannot establish robustness to every extreme-parallax, occlusion, weak-texture, or dynamic scene, and the main text lacks complete stratified statistics for these factors.
• Damaged equation extraction, the missing appendix, and inconsistent regularization-weight names limit reproducibility; this note retains mechanisms supported by the text without presenting reconstructed formulas as author-defined objectives.

Related Work & Insights

• Versus APAP, SPW, and LPC: traditional methods optimize warps directly from point or line constraints, whereas UniStitch encodes geometric evidence for learned regression; APAP's cross-domain results remain a reminder that traditional geometry is a strong baseline.
• Versus UDIS++ and StabStitch++: UniStitch inherits global-to-local elastic transformations and related objectives, with new contributions concentrated in geometric representation fusion, robust training, and TPS evaluation efficiency rather than an entirely new stitching formulation.
• Versus RopStitch: RopStitch strengthens semantic priors, whereas UniStitch introduces explicit keypoint evidence; comparisons under matched adaptation settings are more interpretable than mixing starred and unstarred results.
• Possible extension (reader interpretation): matching confidence could be studied as a direct calibration signal for expert weights, with routing evaluated by modality-failure type; the paper does not provide this causal-level analysis of routing.

Rating

• Novelty: 4/5. Representation-level fusion of sparse geometry and dense semantics is task-specific, although point encoders and mixture-of-experts components build on existing techniques.
• Experimental Thoroughness: 4/5. Component, fusion, point-configuration, and memory ablations are covered, while cross-domain scale and adaptation-cost reporting remain limited.
• Writing Quality: 3/5. The overall pipeline is clear, but generalization claims require attention to table conditions, and regularization descriptions and cached equations need verification.
• Value: 4/5. The work offers practical insights into complementary stitching representations and high-resolution warp implementation.