[ECCV 2026][Segmentation][Semantic Correspondence] XSemanticFlow predicts cross-instance soft correspondences on PartField feature fields with hierarchical self-/cross-attention that sees semantics only, never coordinates, trained in three stages: SE(3) backbone fine-tuning, self-supervised cross-pose alignment on Objaverse, and supervised fine-tuning on PartNet-Coseg; it improves self-segmentation by +19.1 mIoU / +20.6 accuracy and cross-instance cosegmentation by +10.9 mIoU / +12.5 accuracy over the strongest baseline, and its single-reference zero-shot remapping reaches 80% on Mug Hanging and 70% on Block Pouring, well above video-generation baselines.

Tags: ECCV 2026 · Segmentation · Semantic Correspondence · Zero-Shot Manipulation · Cross-Instance Alignment · Three-Stage Training

XSemanticFlow: Cross Object Semantic Alignment for Zero-shot Manipulation¶

Conference: ECCV 2026
Paper: ECCV page
Area: Segmentation
Keywords: Semantic Correspondence, 3D Cosegmentation, Zero-Shot Manipulation, Self-Supervised Cross-Pose Alignment, Feature Field Alignment

TL;DR¶

XSemanticFlow predicts cross-instance soft correspondences on PartField feature fields with hierarchical self-/cross-attention that sees semantics only, never coordinates, trained in three stages — SE(3) backbone fine-tuning, self-supervised cross-pose alignment on Objaverse, and supervised fine-tuning on PartNet-Coseg; it improves self-segmentation by +19.1 mIoU / +20.6 accuracy and cross-instance cosegmentation by +10.9 mIoU / +12.5 accuracy over the strongest baseline, and its single-reference zero-shot remapping reaches 80% on Mug Hanging and 70% on Block Pouring, well above video-generation baselines.

Background & Motivation¶

Establishing reliable semantic correspondences across object instances that differ in shape, scale, and topology is a foundational capability shared by 3D understanding and robotic manipulation: once every surface point on one object can be mapped to its functional counterpart on another, part labels transfer without per-instance annotation, and an entire manipulation demonstration can be remapped onto a novel object without retraining a policy. Correspondence prediction therefore acts as a general-purpose primitive — with an accurate aligned feature field, many category-level perception and control problems reduce to a simple lookup. 2D foundation models supply powerful open-vocabulary semantic descriptors that have been widely used to bootstrap correspondences, and on the 3D side methods like PartField learn part-aware embeddings; but these embeddings are never explicitly forced to align across instances, so the same functional part can land in far-apart regions of the feature space on different objects, and the fields are especially fragile under non-canonicalized poses, making direct feature-matching transfer unreliable.

A deeper obstacle lies on the data side: fine-grained 3D part annotations are severely scarce, while between structurally distinct instances there is no exact point-to-point ground truth to begin with — cross-instance alignment is inherently ill-posed. The authors' judgment is that the general mechanism of robust correspondence can be learned entirely through self-supervision: applying random SE(3) transforms to unlabeled shapes yields unlimited training pairs with perfect spatial ground truth; but for that self-supervision not to degenerate into memorizing geometric shortcuts, the architecture must be hardened first.

Core idea: restrict the cross-attention's access strictly to semantic tokens, with coordinates used only to define local neighborhood support within an object, forcing correspondences to follow functional semantics rather than spatial patterns; then spend the scarce labels where they matter through a three-stage recipe — SE(3) backbone fine-tuning, large-scale self-supervised cross-pose alignment on Objaverse, and supervised cross-instance fine-tuning on PartNet-Coseg.

Method¶

Overall Architecture¶

The input is two 3D geometries from the same semantic category (a source and a query object), and the output is a cross-instance aligned dense semantic feature field together with a soft correspondence map between the two objects' meta-points. XSemanticFlow builds on the PartField backbone: its triplane feature field supports querying a semantic feature at any coordinate via bilinear interpolation. On top of that, the network first samples a set of representative surface meta-points and performs local semantic aggregation, then runs intra-object self-attention and inter-object cross-attention for cross-object propagation, and finally writes the updated meta-point semantics back to the dense vertices, producing an aligned feature space. Geometric coordinates participate in only two scaffolding roles throughout — nearest-neighbor sampling within an object and neighbor lookup during dense write-back — and never enter the inter-object cross-attention. On the training side, a three-stage recipe supplies the supervision; downstream, the field directly serves cosegmentation label transfer and zero-shot demonstration remapping.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    I["Two 3D geometries of one category<br/>source MA and query MB"] --> F["PartField feature field<br/>semantics queryable at any coordinate"]
    F --> S["Semantic-only cross-attention<br/>meta-point aggregation → self-attn → cross-attn"]
    S --> D["Sparse correspondence and dense propagation<br/>soft assignment matrix → residual MLP write-back"]
    D --> O["Cross-instance aligned dense feature field"]
    O --> A["Downstream: cosegmentation label transfer"]
    O --> B["Downstream: zero-shot demonstration remapping"]
    T["Three-stage training recipe<br/>SE3 fine-tune → self-supervised → mixed fine-tune"] -.->|training-only supervision| S

Key Designs¶

1. Semantic-only cross-attention: keeping geometric coordinates out of correspondence prediction

The pain point is that off-the-shelf 3D feature matching is pose-sensitive and inconsistent across geometries — the same functional part occupies entirely different regions of the feature space on different objects. XSemanticFlow's mechanism is to sample a set of representative meta-points on the surface first, with each meta-point aggregating via local attention over its Euclidean K nearest neighbors — but the attention weights are computed from semantic features only, since the Euclidean neighbors merely define the local support within one object. The aggregated meta-point tokens then undergo standard multi-head self-attention across all meta-points of the same object, with no positional encoding at all, forcing the network to construct the object's internal layout purely through semantic relationships. Inter-object cross-attention matches one side's self-attended semantic queries against the other side's keys, making the attention mechanism completely blind to absolute coordinates. The payoff materializes during training: under random SE(3) transforms, coordinate information is itself a source of noise, and since cross-attention cannot touch coordinates, the self-supervised stage cannot degrade into memorizing spatial patterns — correspondences must rest on functional semantics, which is what lets them generalize across topologies.

2. Sparse correspondence and dense propagation: the attention weights are the soft assignment

The inter-object cross-attention block produces, for free while propagating semantics, a meta-to-meta soft assignment matrix \(W_{AB}\) (and the reverse \(W_{BA}\)), which is used directly as the target of the spatial correspondence loss during training; \(L_{\mathrm{corr}}\) requires each query point to recover its spatial target while the triplet loss preserves pairwise feature distances, jointly discouraging degenerate many-to-one assignments. On the inference side, the cross-updated meta-point semantics are written back to the dense mesh vertices: for each dense point, find its Euclidean K nearest meta-points, run one localized semantic-only attention step with the dense point's semantic feature as query and the neighboring meta-points as keys and values, then decode through a residual MLP that refines the backbone features into an aligned dense feature space. Because the aligned field shares a consistent semantic space, full-resolution point-to-point correspondences reduce to nearest-neighbor feature matching. The cosegmentation downstream builds directly on this: agglomerative clustering on the target object's features with a face-adjacency constraint (so clusters never cross the mesh's physical manifold), followed by the Hungarian algorithm matching the source object's labeled parts to the target's clusters for zero-annotation label transfer.

3. Three-stage training recipe: unlimited cross-pose self-supervision as leverage on scarce labels

The first stage fine-tunes the PartField backbone for foundational SE(3) invariance, ensuring the semantic features entering the cross-attention layers are already robust to arbitrary rigid transforms. The second stage freezes the backbone and trains the correspondence modules via self-supervision on Objaverse-v1 — each unannotated mesh generates two views under two independent random SE(3) transforms, and the transform of the queried point set in the second view serves as perfect spatial ground truth, with training driven by the spatial correspondence loss, a feature loss, and the triplet loss. The third stage fine-tunes on a mixture of Objaverse self-supervised pairs and supervised cross-object pairs from PartNet-Coseg. The trick in stage three is the composability of losses: cross-object pairs have no per-point geometric ground truth, so the absolute feature loss is strictly masked out, leaving only the canonical-space \(L_{\mathrm{corr}}\) (with ground truth obtained by transforming query points into PartNet's shared canonical space) and the distribution-level triplet loss — driving cross-instance semantic alignment without forcing absolute feature values to match.

A Worked Example: Zero-Shot Demonstration Remapping¶

Take "hanging a mug on a rack" and walk the five-step pipeline from a single reference demonstration to execution on a new instance. Step one, scene object reconstruction: identify and reconstruct both the manipulated object (the mug) and the stationary anchor object (the mug stand) in the demonstration and target scenes, using open-vocabulary segmentation, mesh reconstruction, and a pose estimator for 6-DoF mesh registration. Step two, anchor object alignment: query XSemanticFlow for the semantic correspondence between the two mug stands, derive a relative SE(3) transform, and shift and orient the entire base demonstration trajectory into the target scene's coordinate frame. Step three, manipulated object rotational alignment: query the correspondence field between the source mug at its goal state and the target mug, and derive the semantic rotation \(R_{\mathrm{semantic}}\) that aligns the target object's functional parts with the source's goal orientation. Step four, contact detection and translation refinement: track the source object through the demonstration to find the earliest contact frame where it approaches the stand within a strict distance threshold, then map the source contact vertices to their semantic equivalents on the target mesh; apply \(R_{\mathrm{semantic}}\) to the target object first, then compute the translation offset \(t_{\mathrm{delta}}\) so the rotationally-aligned target contact region's centroid coincides exactly with the source's contact region in the physical scene. Step five, grasp and trajectory warping: split the trajectory at the contact alignment point — the closest point in the demonstration trajectory to the target object's starting position — into two phases. In the approach phase the trajectory smoothly interpolates to correct the combined SE(3) offset of \(R_{\mathrm{semantic}}\) and \(t_{\mathrm{delta}}\); in the replay phase the robot follows the anchor-shifted, contact-aligned demonstration trajectory to complete the interaction, and the initial grasp is remapped to the semantically equivalent target vertices. The whole process generates no demonstration video per target and trains no policy.

Loss & Training¶

Running through both the self-supervised and supervised stages is the triplet relative consistency loss: rather than constraining absolute feature values, it requires that the intra-object feature distribution of the same object, before and after feature propagation, stay consistent across an SE(3) transform. For a sampled triplet of points \((p_1,p_2,p_3)\) on an object, compute pairwise cosine distances in the raw feature space to form the distance vector \(d_F\) (features are \(\ell_2\)-normalized first, so similarities are bounded in \([-1,1]\)), then convert the distance vector into a distribution \(P_F=\mathrm{softmax}(d_F/\tau)\) with temperature \(\tau=0.01\); compute \(P_{\hat F}\) identically from the propagated features \(\hat F\) on the transformed object, penalize the symmetric KL divergence between the two relational distributions, and average over randomly sampled triplets:

\[ \mathcal{L}_{\mathrm{triplet}}=\frac{1}{|T|}\sum_{t\in T}\big[\mathrm{KL}(P_F^{t}\,\Vert\,P_{\hat F}^{t})+\mathrm{KL}(P_{\hat F}^{t}\,\Vert\,P_F^{t})\big], \]

(the paper's equation is typeset corrupted in the cached text; this form is reconstructed from the prose "symmetric KL averaged over randomly sampled triplets" — ⚠️ refer to the original paper). It keeps an object's relative semantic distances proportionally consistent before and after transformation.

The three stages concretely: stage one fine-tunes the PartField backbone for 50 epochs to learn SE(3) invariance; stage two freezes it and trains self-supervised on Objaverse-v1 for 150 epochs, with the spatial correspondence loss \(L_{\mathrm{corr}}\) (each query point recovers its true position in the other view; the paper's equation is corrupted, so it is not reconstructed by guesswork), the feature loss \(L_{\mathrm{feat}}\) (minimizing the distance between propagated features \(\hat F\) and the ground-truth backbone features \(F\); only applicable to self-supervised pairs where the two views share identical geometry), and \(L_{\mathrm{triplet}}\); stage three fine-tunes for 50 epochs on mixed batches of Objaverse-v1 and the PartNet-Coseg training split, with \(L_{\mathrm{feat}}\) strictly masked out for cross-object pairs. All models are optimized with AdamW. The paper does not give the number of meta-points, the neighbor K, or the loss weights in the main text.

Key Experimental Results¶

Main Results¶

Evaluation is built on PartNet-Coseg: instance pairs from the same semantic category sharing a matching set of part labels are sampled from PartNet, with the train split for supervision and the validation and test splits for evaluation. Two protocols: PartNet-SE3 measures same-object self-segmentation consistency under SE(3) augmentations, and PartNet-Coseg measures cross-instance label transfer. Both protocols take a source point cloud with ground-truth part labels plus a target point cloud, and output the predicted part label assignment on the target; metrics are per-point accuracy (Acc, the percentage of correctly classified points) and mIoU (the average overlap between predicted and ground-truth regions across all present part categories). Since the baselines do not emit dense cross-instance correspondences, every method including this paper follows the same shared Hungarian protocol: take each method's 3D part segments, average its per-point/vertex descriptors within each segment, match parts across instances with the Hungarian algorithm on cosine distance, and transfer source part labels to the matched target parts; a variant propagating labels through XSemanticFlow's own dense correspondences is reported in the supplementary material.

Method	PartNet-SE3 Acc↑	PartNet-SE3 mIoU↑	PartNet-Coseg Acc↑	PartNet-Coseg mIoU↑
PartField	55.55	43.41	61.23	52.43
SAMesh	46.17	41.55	56.11	45.52
SAMPart3D	50.15	40.80	58.92	48.40
XSemanticFlow (ours)	76.18	62.47	73.75	63.32

On self-segmentation, mIoU rises from the strongest baseline PartField's 43.41 to 62.47 (+19.06) and Acc from 55.55 to 76.18 (+20.63) — raw PartField embeddings are highly sensitive to rigid transforms, whereas semantic-only matching with spatial anchoring learns substantially transform-consistent representations. On cross-instance evaluation, CoSeg-mIoU rises from 52.43 to 63.32 (+10.89) and CoSeg-Acc from 61.23 to 73.75 (+12.52), showing the aligned field really does align across different geometries.

The manipulation side compares two video-generation zero-shot baselines, NovaFlow and NovaPlan, on three tasks of increasing physical strictness; numbers come from the text and Figure 4:

Task	NovaFlow	NovaPlan	XSemanticFlow (ours)
Mug Hanging	10%	20%	80%
Water Plant	60%	90%	90%
Block Pouring	40%	40%	70%

The three tasks use different success criteria. Mug Hanging requires threading the mug handle precisely onto a rack, and the evaluation protocol is stricter than NovaFlow's original setting — a single reference demonstration generated with both start and goal images is provided, but for all subsequent trials on novel instances neither this method nor the baselines get goal images. Water Plant only requires correctly positioning and tilting the mug over the plant (no real liquid, for hardware safety), a lenient criterion. Block Pouring is its physical-verification variant: the vessel is filled with granular blocks and success requires at least one block landing in the pot. Under the lenient metric all three perform well; once contact precision decides the outcome, video-generation quality becomes the two baselines' bottleneck.

Ablation Study¶

Config	PartNet-SE3 Acc↑	PartNet-SE3 mIoU↑	PartNet-Coseg Acc↑	PartNet-Coseg mIoU↑
Full model	76.18	62.47	73.75	63.32
w/o rotational-invariant pre-training	72.51	61.62	68.86	58.74
w/o PartNet-Coseg fine-tuning	72.27	61.03	71.59	61.81
+ geometric features in cross-attention	75.10	61.86	71.69	62.21

Removing rotational-invariant pre-training: more rotationally robust PartField features help both self-segmentation and cross-instance alignment, with the cross-instance end dropping more (Coseg Acc −4.89, mIoU −4.58; SE3 Acc −3.67, mIoU −0.85).
Removing PartNet-Coseg fine-tuning: supervised cross-object alignment beyond self-supervision contributes concentrated on the cross-instance end, with CoSeg-mIoU falling from 63.32 to 61.81; self-supervision alone cannot fully substitute for a small set of labeled pairs.
Adding geometric features to cross-attention: injecting positional and geometric signals into the attention block consistently degrades correspondence on both protocols, though modestly rather than catastrophically — the model still receives strong semantic features. This directly supports the core claim: geometric signals are an overfitting source in cross-attention, not useful information.

Key Findings¶

The architectural constraint and the training data enable each other. Semantic-only cross-attention is what makes the "random SE(3) transforms generate unlimited perfect ground-truth pairs" self-supervision actually work; the ablation that adds geometry back and drops points across the board is the most convincing counter-evidence.
Labels act as leverage, not bulk. Stage three uses only PartNet-Coseg's labeled pairs for 50 epochs of mixed fine-tuning yet buys about 1.5 points of cross-instance mIoU without sacrificing the self-segmentation end.
Failures cluster on label-set mismatch, geometric symmetry, and physical constraints. Forcing one-to-one matching when same-category instances have different label sets causes misassignments or collapsed clusters; symmetric structures like a shelf's vertical and horizontal panels become indistinguishable after an SE(3) transform; on manipulation, remapped trajectories carry no obstacle-avoidance optimization — the paper shows a pouring failure from colliding with the camera pillar (the figure caption calls it a joint limit).

Highlights & Insights¶

"Coordinate blindness" is an architectural constraint, not a regularizer. Stripping positional encoding and coordinates entirely out of cross-attention turns preventing geometric overfitting from a nudge into an impossibility — the paper's central "aha," and the reason cross-pose self-supervision can scale.
The attention weights are the correspondence. The cross-attention block yields the soft assignment matrix for free, with no separate correspondence head; the training signal, the sparse correspondence, and the dense propagation all share one mechanism, leaving no redundant module in the structure.
Align relations, not values. The triplet relative consistency loss applies symmetric KL at the distribution level, allowing absolute feature drift as long as the relative semantic structure survives the transform — an exact fit for cross-instance settings without point-to-point ground truth.
A transferable design pattern. Any "feature field + attention" cross-domain matching task (point cloud registration, cross-object part transfer, category-level pose estimation) can borrow the same split — coordinates define neighborhood support, semantics do the attending.

Limitations & Future Work¶

Rigid objects only. The pipeline collapses the dense correspondences into a single rigid SE(3) transform; non-rigid manipulation is an explicitly stated future direction.
No obstacle awareness. Remapped trajectories are not optimized against obstacles — a pouring failure from colliding with the camera pillar prevents completing the planned trajectory; the authors leave this to future work.
Label-set and symmetry failures. Inconsistent label sets within a category induce ambiguous correspondences, and geometrically symmetric objects become indistinguishable after transforms; cosegmentation fails systematically in both regimes.
Limited manipulation evaluation scale. Three tasks with success rates reported without trial counts or confidence intervals; the dense-correspondence propagation variant for cosegmentation appears only in the supplement, while the main-text protocol routes everything through Hungarian matching, which may understate this method's true label-transfer ceiling.
Incomplete implementation details. Meta-point counts, neighbor K, loss weights, and training resources are absent from the main text; reproduction depends on the supplement.

vs PartField: PartField provides a continuous, queryable 3D semantic feature field but explicitly does not enforce alignment across instances or non-canonicalized poses; this paper adds a semantic-only alignment layer on top — a reinforcement, not a replacement of the backbone.
vs SAMesh / SAMPart3D: same 3D part-segmentation line, emitting discrete labels without a direct cross-instance correspondence field; this paper pulls them under one evaluation protocol via the shared Hungarian matching.
vs NovaFlow / NovaPlan: both extract executable actions from generated videos, requiring per-target demonstration generation whose quality is unstable; this paper transfers contact regions and trajectories directly from a single reference demonstration, avoiding video generation failures — far ahead on contact-critical tasks and no worse on lenient ones.
vs VLA / VLM planning: VLAs are limited by the cost of high-quality physical interaction data, while VLM planners struggle with fine-grained 3D localization; this paper reduces category-level control problems to correspondence lookups, sidestepping both routes' bottlenecks.

Rating¶

Novelty: 4/5. Most components are known (PartField, attention, triplet losses), but the "coordinate-blind cross-attention + three-stage recipe" combination is clear and ablation-backed.
Experimental Thoroughness: 4/5. Covers both 3D understanding and real-robot execution with three baselines, three ablations, and failure analysis; the manipulation side, however, has few tasks and no statistical intervals.
Writing Quality: 4/5. Architecture and training flow are narrated clearly, though some equations are typeset corrupted and details lean on the supplement.
Value: 4/5. Treating the correspondence field as a general-purpose primitive for category-level perception and manipulation is directly reusable for label transfer and grasp transfer; the rigid-object assumption limits the application surface.