SGMatch: Semantic-Guided Non-Rigid Shape Matching with Flow Regularization¶

Conference: ECCV 2026
Paper: ECCV
Area: 3D Vision
Keywords: non-rigid shape matching, functional maps, conditional flow matching, semantic priors, dense correspondence

TL;DR¶

SGMatch fuses semantic features obtained by rendering shapes from multiple views, encoding them with DINOv2, and back-projecting them onto the surface into geometric descriptors via channel-wise gating plus mesh-neighborhood-restricted cross-attention, and adapts conditional flow matching (CFM) into a time-conditioned regularizer on vertex-level feature transport that smooths dense correspondence recovery, reaching the best results on strongly non-isometric (SMAL geodesic error 2.5) and topology-noisy (TOPKIDS 2.9) benchmarks.

Background & Motivation¶

Establishing point-to-point correspondences between non-rigid 3D shapes is a prerequisite for texture transfer, pose transfer, and statistical shape analysis. The most successful line of work over the past decade is the functional maps framework: instead of searching for discrete point pairings, it represents the correspondence as a linear operator \(C\) between two function spaces, using the Laplace–Beltrami eigenbasis to compress a dense correspondence into a small k×k matrix. This yields a built-in low-frequency structural prior that keeps the problem both stable and easy to optimize under near-isometric deformation, and most subsequent progress — spectral refinement, structured regularization, and learned descriptors from FMNet and GeomFMaps to unsupervised methods such as ULRSSM and HybridFMap — has unfolded within this framework.

Two structural failure modes have nevertheless persisted. The first is descriptor ambiguity: intrinsic geometric descriptors such as HKS and WKS only see local geometric quantities, so they respond almost identically on bilaterally symmetric parts and on parts that are geometrically similar across categories (a quadruped's front and hind legs, for instance). No matter how accurate the functional map is, it can only pick one of the two. The second is spatial inconsistency caused by spectral truncation: in theory \(C = \Phi_Y^\dagger \Pi \Phi_X\), but recovery only uses the first k eigenfunctions, so even when \(C\) looks globally well aligned, the reconstructed dense map still jumps and scatters locally. This is unrelated to descriptor quality — it is information lost by truncation itself.

Semantic cues look like a ready-made remedy for ambiguity, because a "correct correspondence" is inherently semantic: a human shape's left hand should match a left hand, not the perfectly symmetric right one. Work such as Diff3F has shown that self-supervised features from 2D vision foundation models (DINOv2 and friends) can be lifted onto 3D surfaces to give robust zero-shot descriptors. Plugging them into a functional map pipeline, however, is not straightforward: global semantic signals are strong, and once they are naively concatenated or attended to globally they tend to override local geometric structure, making correspondences worse rather than better — what is needed is semantics that disambiguates without breaking manifold locality. The second failure mode is not something a better descriptor can fix; it calls for an additional spatial smoothness constraint at the dense recovery stage. Core idea: treat semantics as structure-aware anchors constrained by manifold locality — channel-wise gating modulates geometric features and cross-attention is confined to mesh neighborhoods, which settles "who should match whom" — and then port conditional flow matching to shape matching so that a continuous feature-transport trajectory supervises vertex-level transport under soft correspondences, which settles "how coherent the map is".

Method¶

Overall Architecture¶

The input is a pair of triangle meshes \(X\) (\(n_X\) vertices) and \(Y\) (\(n_Y\) vertices). The method has three parallel branches. On the geometry side, DiffusionNet produces a per-vertex descriptor robust to mesh resolution and sampling density. On the semantics side, following Diff3F, each shape is rendered from several viewpoints into depth- and normal-guided images, passed through a frozen DINOv2 encoder, back-projected onto the surface using the known camera parameters, and averaged across viewpoints to yield per-vertex semantic features. The two are fused by the SGLCA module. The fused features \(F^{fuse}\) are then used twice: once by the functional map solver to estimate the bidirectional maps \(C_{XY}\), \(C_{YX}\) and recover the soft correspondence matrix \(\Pi\), and once — after spectral heat diffusion — as the "smoothed" endpoint of a flow matching trajectory, paired with the target features transported through \(\Pi\). The whole pipeline is learning-based: feature extraction, fusion, functional map estimation, and CFM are trained jointly under a single differentiable objective. Inference requires no ODE integration — a single forward pass over the fused features gives \(\Pi\) — because CFM's smoothness preference is amortized into the shared fused-feature backbone rather than applied as a separate post-processing step at inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Shape pair X, Y"] --> B["DiffusionNet geometric features<br/>DINOv2 multi-view semantic features"]
    B --> C["Semantic-guided local cross-attention<br/>gating modulation + local attention"]
    C --> D["Functional map estimation and dense recovery<br/>C_XY, C_YX → Π"]
    D --> E["Spectral heat diffusion<br/>smooth flow endpoints z0, z1"]
    E --> F["Conditional flow matching regularization<br/>time-conditioned velocity field"]
    F --> G["L_spectral + λ_cfm·L_cfm"]

Key Designs¶

1. Semantic-Guided Local Cross-Attention (SGLCA): let semantics modulate geometry instead of overriding it

The observation that "injecting semantics blindly makes things worse" has two concrete sources. First, semantic and geometric features differ in dimensionality while semantics is globally highly consistent, so naive concatenation makes the network lean on the semantic half alone. Second, if every vertex attends to every other vertex, semantics produces long-range, unstructured responses across the whole mesh and blurs the local geometric structure that was previously crisp. SGLCA handles these with two cooperating mechanisms. It first performs semantic-guided gating along the channel dimension: the semantic features are linearly projected to the geometric dimensionality, a lightweight MLP turns them into per-channel gate weights \(G=\sigma(\mathrm{MLP}_{gate}(\tilde F^{sem}))\), and these modulate the geometric features as

\[\tilde F^{geo}=F^{geo}\odot(1+\alpha G),\]

where \(\alpha\) is a learnable scalar. Note the modulation is \((1+\alpha G)\) rather than \(G\) itself: geometric channels stay untouched where semantics gives no clear signal, so semantics only amplifies or attenuates and geometry remains the structural backbone.

Cross-modal aggregation is then done with neighborhood-restricted cross-attention: queries come from the modulated geometric features \(Q_i=W_Q\tilde F^{geo}_i\) while keys and values come from the semantic features \(K_j=W_K\tilde F^{sem}_j\), \(V_j=W_V\tilde F^{sem}_j\), but the softmax is normalized only over the mesh neighborhood \(\mathcal N(i)\) of vertex \(i\). The fused feature adds the neighborhood semantic weighted sum to \(\tilde F^{geo}_i\) and applies layer normalization. Each geometric feature therefore draws only on the semantics surrounding it. The ablation gives a clean causal picture: replacing the neighborhood with global attention leaves SMAL geodesic error at 2.6, no better than local attention alone and worse than the full model with gating plus local attention (2.5), confirming that unrestricted semantic interaction really does introduce irrelevant long-range responses; gating without attention (3.6) is clearly worse than the full model, showing the gain comes mainly from the locality side — the value of semantics is not "global context" but reweighting the channels that geometry cannot separate locally.

2. Spectral heat diffusion: smooth both ends of the flow before talking about trajectories

Porting CFM to shape matching runs into an immediate problem: flow matching is originally defined between distributions, whereas here both endpoints are single-vertex features, so if the soft correspondence in \(\Pi\) is itself wrong, the velocity field is asked to fit a wrong target displacement. Before building any trajectory, the method therefore applies spectral heat diffusion to the fused features. With the Laplace–Beltrami eigenbasis \(\Phi\), eigenvalues \(\Lambda\), and mass matrix \(M\), the diffused features are

\[Z=\Phi\exp(-\tau\Lambda)\Phi^\top M F^{fuse},\]

where \(\tau>0\) controls the diffusion scale. This is a purely intrinsic geometric operation: it preserves the shape's own spectral geometry and only suppresses high-frequency noise, amounting to heat-kernel smoothing in the feature domain. It helps on two levels — it de-noises local mesh artifacts (scan noise, uneven sampling) to stabilize the endpoints, and it removes part of the local high-frequency inconsistency up front so the displacement targets the velocity field must fit are more globally consistent. In the ablation, removing heat diffusion alone (everything else unchanged) degrades SMAL geodesic error from 2.5 to 3.0, a drop comparable in magnitude to removing CFM (2.7), showing that endpoint stability and trajectory supervision are complementary rather than redundant designs.

3. Conditional flow matching regularization: trajectory-level transport instead of endpoint matching or Laplacian smoothing

This is the most interesting step of the paper. Concretely, take the heat-diffused source features \(z_0=Z_X\) and transport the target features through the soft correspondence to get \(z_1=\Pi_{XY}Z_Y\). Then follow the linear interpolation path standard in flow matching, \(z_t=(1-t)z_0+tz_1,\ t\sim U(0,1)\), whose conditional velocity is the constant displacement \(z_1-z_0\). An MLP-parameterized time-conditioned velocity field \(v_\theta(z_t,t)\) is trained to fit that displacement: the scalar time \(t\) is first lifted by sinusoidal positional encoding and then injected into the MLP through FiLM so the network can model non-linear temporal dependencies along the trajectory. The objective uses the Charbonnier form

\[L_{cfm}=\mathbb E_{t,\,i\in S}\sqrt{\lVert v_\theta(z_{t,i},t)-v_{target,i}\rVert^2+\varepsilon^2},\]

together with similarity-based importance sampling: each vertex gets a confidence weight \(w_i=\exp(\alpha\cos(z_{0,i},z_{1,i}))\) and a subset \(S\) is sampled according to those weights. Vertices whose source and transported features agree in direction — i.e. whose soft correspondence is more reliable — are sampled more often, so clearly wrong soft correspondences early in training are not treated as hard targets.

The key question is why this improves spatial coherence; the paper addresses it explicitly because the objective is easy to misread as a fancy Laplacian smoother. The CFM loss is indeed evaluated pointwise on sampled vertices, and it is not an explicit pairwise smoothness term. Its spatial bias comes from two places: both endpoints have been heat-diffused on their respective manifolds and are thus already spatially smooth, and — the crucial part — all vertices share one time-conditioned velocity network. Locally inconsistent correspondences create high-frequency, mutually incompatible displacement targets (neighboring vertices demand conflicting directions), and a shared network with finite capacity can hardly fit those contradictions at once, so some optimization pressure is converted into a preference for spatially tidy displacement fields. The difference from adding Dirichlet energy or Laplacian smoothing is that the latter is an explicit low-pass filter on the map, which overly smooths and can erase genuinely folded regions, whereas CFM makes a finite function approximator "refuse" high-frequency targets, so smoothness emerges as a by-product of capacity rather than a hard constraint. Section 5.5 confirms this: under the same backbone, swapping CFM for endpoint-level feature matching, cycle consistency, or synchronous diffusion gives CFM the best SMAL geodesic error (2.5) and conformal distortion (1.96); on TOPKIDS, cycle consistency pushes Dirichlet energy lower than CFM (42.42 vs 48.33) but simultaneously worsens geodesic error, conformal distortion, and coverage — that is over-smoothing rather than a better correspondence, and it illustrates that map quality cannot be judged by Dirichlet energy alone.

Loss & Training¶

The total objective combines a spectral loss with the CFM regularizer. The spectral side inherits structured regularization: bidirectional cycle consistency \(L_{bij}=\lVert C_{XY}C_{YX}-I\rVert_F^2+\lVert C_{YX}C_{XY}-I\rVert_F^2\) encourages bijectivity, an orthogonality term \(L_{orth}\) promotes local area preservation, and the two are weighted into \(L_{struct}=\lambda_{bij}L_{bij}+\lambda_{orth}L_{orth}\). What actually makes the features learnable is the coupling term: it requires the spectrally solved \(C\) to agree with the map back-projected from the dense soft correspondence \(\Phi_Y^\dagger\Pi_{YX}\Phi_X\) (and symmetrically for the reverse direction), tying spectral alignment to pointwise alignment. Hence

\[L_{total}=\underbrace{L_{struct}+\lambda_{couple}L_{couple}}_{L_{spectral}}+\lambda_{cfm}L_{cfm}.\]

The dense soft correspondence itself comes from a temperature-scaled softmax over the dot products of fused features, where \(\tau_T\) controls the sharpness of the distribution. The paper does not report the specific values of the \(\lambda\)'s, \(\tau\), \(\tau_T\), or the network channel widths within the available text (⚠️ refer to the original paper / appendix).

Key Experimental Results¶

Main Results¶

The metric is mean geodesic error (Geo.Err, ×100, as defined by Kim et al.'s blended intrinsic maps; lower is better). Near-isometric evaluation uses FAUST, SCAPE, and SHREC'19 (all remeshed versions); non-isometric evaluation uses SMAL and DT4D-H, the latter split into intra-class and inter-class columns.

Benchmark	Setting	Ours	Prev. best	Note
FAUST	near-isometric	1.4	HybridFMap / DeepFAFM 1.5	essentially on par
SCAPE	near-isometric	1.8	HybridFMap 1.8	tied
SHREC'19	cross-dataset generalization	3.3	HybridFMap / DeepFAFM 3.6	largest benefit from semantic priors out of the training distribution
SMAL	non-isometric (cross-species)	2.5	HybridFMap 3.3	≈24% improvement over the previous SOTA
DT4D-H intra	non-isometric, same category	1.0	ULRSSM / DeepFAFM 0.9	tied with HybridFMap's 1.0; the table lists DeepFAFM and ULRSSM at 0.9, numerically lower
DT4D-H inter	non-isometric, cross-category	3.4	HybridFMap 3.5	cross-category semantic consistency is harder; semantics help here
TOPKIDS	topological noise	2.9	HybridFMap 5.0	≈42% improvement over the previous SOTA

The PCK-curve AUCs agree: 0.88 on SMAL (ULRSSM 0.82 / HybridFMap 0.85), 0.84 on DT4D-H inter (0.76 / 0.83), and 0.89 on TOPKIDS (0.77 / 0.82).

Beyond accuracy, the paper separately assesses map smoothness with conformal distortion, surface coverage, Dirichlet energy, and bijectivity against HybridFMap (↓ means lower is better):

Metric	Method	FAUST	SCAPE	SHREC'19	SMAL	DT4D-inter	TOPKIDS
Conf. distortion ↓	HybridFMap	0.654	0.779	1.030	2.473	1.866	3.201
Conf. distortion ↓	Ours	0.639	0.764	0.787	1.956	1.563	2.517
Coverage (%) ↑	HybridFMap	83.3	82.5	75.9	65.0	67.1	63.3
Coverage (%) ↑	Ours	83.5	81.9	76.1	71.2	66.3	63.9
Dirichlet energy ↓	HybridFMap	2.96	3.11	12.30	19.29	8.44	98.30
Dirichlet energy ↓	Ours	2.95	3.27	6.30	10.20	8.70	48.33
Bijectivity ↓	HybridFMap	0.0049	0.0060	–	0.0285	0.0130	–
Bijectivity ↓	Ours	0.0052	0.0064	–	0.0175	0.0166	–

The gains concentrate in the strongly deformed and topology-noisy settings: SMAL's Dirichlet energy drops from 19.29 to 10.20 and TOPKIDS from 98.30 to 48.33, while on SCAPE and DT4D-inter some metrics are merely comparable rather than better (SCAPE Dirichlet energy 3.27 vs 3.11 and coverage 81.9 vs 82.5; DT4D-inter Dirichlet energy 8.70 vs 8.44 and bijectivity 0.0166 vs 0.0130), matching the paper's own assessment.

Ablation Study¶

The first ablation, under the same SGMatch backbone, replaces CFM with other smoothness regularizers to test whether trajectory-level transport supervision is redundant with endpoint matching or diffusion-style smoothing:

Variant	SMAL Geo.Err ↓	SMAL Conf. ↓	SMAL Dir.E ↓	SMAL Bij. ↓	SMAL Cov. ↑	TOPKIDS Geo.Err ↓
w/o CFM (drop the CFM branch)	2.7	2.32	10.55	0.0210	66.8	3.4
Endpoint-only (match heat-diffused endpoints)	2.7	2.46	14.37	0.0225	65.0	3.4
Cycle-consistency	2.8	2.29	10.53	0.0201	67.2	3.4
Sync. Diff.	2.6	2.45	13.48	0.0221	64.7	3.5
Ours	2.5	1.96	10.20	0.0175	71.2	2.9

The second ablation on SMAL disentangles the SGLCA components together with heat diffusion and CFM:

Variant	Geo.	Sem.	Gating	Attention	Heat Diff.	CFM	Geo.Err ↓
Geo. only	✓	–	–	–	✓	✓	3.2
Sem. only	–	✓	–	–	✓	✓	21.2
Gating only	✓	✓	✓	none	✓	✓	3.6
Local attention only	✓	✓	–	local	✓	✓	2.6
Gating + global attention	✓	✓	✓	global	✓	✓	2.6
w/o heat diffusion	✓	✓	✓	local	–	✓	3.0
Ours	✓	✓	✓	local	✓	✓	2.5

Key Findings¶

Both modalities matter, but geometry is the backbone: semantics alone (21.2) is nearly 7× worse than geometry alone (3.2) and cannot support dense correspondence on its own, while the semantic benefit shows up in generalization and non-isometric settings — SHREC'19 cross-dataset generalization (3.3) and DT4D-H inter-class (3.4) are both best, precisely where geometric descriptors are weakest.
Locality matters more than gating: local attention alone (2.6) already approaches the full model, whereas gating alone (3.6) is clearly worse; conversely, opening attention to global (2.6) not only fails to help but cancels the gating gain — unrestricted long-range semantic interaction introduces irrelevant responses.
Trajectory supervision cannot be reduced to endpoint matching: replacing CFM with endpoint feature matching blows SMAL's Dirichlet energy up from 10.20 to 14.37, worse even than dropping CFM entirely (10.55), directly showing the gain comes from supervising along the trajectory rather than from the endpoint features themselves.
Low Dirichlet energy does not mean a good map: on TOPKIDS, cycle consistency pushes Dirichlet energy to 42.42 (better than Ours' 48.33) while geodesic error stays at 3.4 and conformal distortion and coverage are worse — classic over-smoothing, and a reminder that map regularity needs multiple metrics.
Heat diffusion and CFM are complementary, not redundant: removing either costs accuracy (2.5 → 3.0 / 2.7), since endpoint stabilization and trajectory regularization address different stages.

Highlights & Insights¶

Treating CFM as an implicit smoothness prior rather than a generative model: instead of generating samples, the velocity field is trained so that one shared, capacity-limited network must fit all vertices' displacement targets, and the inability to fit high-frequency contradictory targets is what produces spatial coherence. This transfers to any task with per-point prediction that needs neighborhood consistency — for instance wrapping optical flow or point-cloud registration displacement fields in a shared-velocity trajectory objective in place of an explicit smoothness loss.
Writing the gate as \((1+\alpha G)\) instead of \(G\): that single symbol lets semantics only amplify or attenuate the geometric basis, so geometry passes through untouched when semantics is silent, avoiding the most common multimodal fusion failure mode of "semantics overriding geometry". The design is minimal but the ablation gives clean causal evidence.
Reusing soft-correspondence similarity as the sampling weight: \(w_i=\exp(\alpha\cos(z_0,z_1))\) turns "how trustworthy is this soft match" into a sampling weight, and pairing it with a Charbonnier loss automatically masks unreliable pseudo-labels early in training without a separate confidence-estimation network.
Evaluating map regularity separately from pointwise accuracy: reporting conformal distortion, coverage, Dirichlet energy, and bijectivity alongside geodesic error, and explicitly noting that some metrics contradict each other (over-smoothing) in certain settings, is more informative than squeezing out one more accuracy digit.

Limitations & Future Work¶

The authors acknowledge two limitations. The framework is designed for complete shapes and does not explicitly handle partial matching, an important and actively studied problem in non-rigid shape analysis. Also, its semantic features come from pretrained 2D visual models, so performance depends on the quality and domain generalization of those external representations; domains where such visual priors are weak (e.g. medical statistical shape modeling) may need domain-specific encoders or task-specific semantic pretraining.
From the experimental design, several gaps remain: TOPKIDS has only 26 training shapes, so that setting is compared only against axiomatic and unsupervised methods with no strong supervised baselines, which limits the strength of the topological-noise conclusion; the ablations are all run on SMAL, so the independent contribution of each component under cross-category (DT4D-H inter) or topological noise is not separated out; and hyper-parameters and network details are absent from the main text, making reproduction harder.
Promising directions: replace the deterministic softmax construction of \(\Pi\) with a learnable stochastic transport (e.g. entropy-regularized optimal transport) so that the CFM endpoints are themselves samples from a distribution, which would naturally support unbalanced and partial matching; extend flow matching from the vertex-feature space to the spectral-coefficient space, where a trajectory regularizer on the eigenbasis may target the root cause — spectral truncation — more directly than a vertex-domain regularizer; and give semantic features spatially varying trust, using spectral-geometric confidence to decide which regions should listen to geometry and which to semantics.

vs HybridFMap (CVPR 2024): HybridFMap also targets crease-aware non-isometric matching but improves the geometric side (hybrid functional maps / crease-aware descriptors) and uses no semantic priors; it is the most direct comparison. It falls clearly behind under large deformation and topological noise (SMAL 3.3 → 2.5, TOPKIDS 5.0 → 2.9) while near-isometric results (FAUST/SCAPE) are essentially tied — semantics compensates for geometry's weak spots rather than improving everything.
vs ULRSSM / DeepFAFM: these improve the learning objective for robust spectral matching and frequency-aware functional maps respectively, representing the pure geometry/spectral line. The gap is largest in the cross-category setting (DT4D-H inter 3.4 vs 4.1 / 3.9), supporting the paper's claim that the bottleneck for cross-category correspondence is the semantic gap, not descriptor discriminability.
vs Synchronous Diffusion (ECCV 2024): it enforces consistency on the point map through a diffusion process to obtain smooth correspondences, the closest alternative and the one directly compared in the ablation. The measured distinction: synchronous diffusion reaches 2.6 / conformal distortion 2.45 on SMAL versus CFM's 2.5 / 1.96 — as two ways of "making the map smooth", trajectory-level transport supervision damages accuracy less than a diffusion-style smoothness constraint.
vs Diff3F / EchoMatch: the former supplies the extraction recipe for the semantic features used here (multi-view rendering, foundation-model encoding, back-projected averaging), while the latter applies semantic cues to partial-to-partial matching. The difference is that SGMatch does not use semantics as a descriptor but as a modulation signal constrained by locality — which explains why "semantics only" (ablation: 21.2) fails catastrophically.

Rating¶

Novelty: ⭐⭐⭐⭐ Porting conditional flow matching from generative modeling to trajectory-level regularization for dense correspondence, with a "shared velocity field rejects high-frequency targets" account of where its smoothness comes from, is a fresh angle; the semantic fusion part (gating + local attention) is more conventional.
Experimental Thoroughness: ⭐⭐⭐⭐ Covers near-isometric, non-isometric, cross-category, and topology-noisy settings across six benchmarks, plus map-regularity metrics and comparisons against alternative regularizers; however ablations are confined to SMAL and hyper-parameters and implementation details are under-reported.
Writing Quality: ⭐⭐⭐⭐ The method is clearly explained and the paper explicitly clarifies that CFM is not Laplacian smoothing, a point easily misread; the main weakness is that quantitative results are scattered across figures and tables, and a few in-text claims do not match the table values (e.g. calling DT4D-H intra the "best" result).
Value: ⭐⭐⭐⭐ The combination of semantic priors and trajectory regularization yields substantial, consistent gains under non-isometric deformation and topological noise, and both tricks (\((1+\alpha G)\) gating and attention localization) are lightweight and easy to transfer.