E-M3RF: An Equivariant Multimodal 3D Re-assembly Framework

Conference: ECCV2026
Paper: ECCV Paper
Project: https://em3rf.github.io/
Area: 3D Vision
Keywords: fracture reassembly, rotation equivariance, multimodal fusion, flow matching, non-overlap constraints

TL;DR

E-M3RF fuses rotation-consistent geometric features with pointwise color, predicts fragment poses through SE(3) flow matching, and penalizes interpenetration during training, reducing RePAIR rotation RMSE from GARF's 49.31 degrees to 37.91 degrees without leading on every metric of every generalization dataset.

Background & Motivation

Three-dimensional fracture reassembly does not complete an image; it finds rotations and translations that restore physical fragments to a shared object configuration. Classical methods rely on fracture contours, local geometric descriptors and registration, whereas learning-based methods extract point-cloud features before regressing or progressively generating poses. Clean, distinctive fracture surfaces provide strong constraints, but small fragments, eroded boundaries and repetitive surfaces can make several placements geometrically plausible. Frescoes and archaeological objects may retain paint, texture or tool marks, so geometrically similar pieces need not have compatible appearance. An encoder that only observes coordinates cannot supply these disambiguating cues to the pose predictor.

Adding color is not sufficient because coordinates, normals and RGB obey different transformation rules. Coordinates and normals should rotate with a fragment, while RGB values should not be rotated as spatial vectors. Reassembly also requires the orientation of a potential contact surface: fully rotation-invariant descriptors may help retrieval while discarding information needed for alignment. Even small pose errors can conceal interpenetration, producing assemblies with nearby centroids but implausible surface relationships. The paper addresses these issues within one pipeline instead of merely enlarging a geometric encoder.

Its approach learns geometry and color separately, fuses them per point, and forms relative poses through a continuous flow. Fracture-boundary supervision directs the representation toward assembly-relevant regions, while a training-only non-overlap penalty supplies a physical requirement not explicitly expressed by pose error. Core Idea: preserve the transformation behavior of directional geometry, use rotation-invariant appearance as complementary evidence, and jointly learn fragment placement through flow matching and non-overlap supervision.

Method

Overall Architecture

The input is a collection of fragments belonging to one object, each represented by a point cloud with coordinates, surface normals and RGB when available. Each fragment is translated independently so that its centroid lies at the origin, removing the original scan translation from local encoding. The output is a rigid transformation \((R_i,\beta_i)\) for each fragment, which places its local coordinates into the predicted assembly. This is pose estimation for a supplied fragment collection, not fragment segmentation from images or generation of missing material.

The pipeline comprises equivariant geometry and color fusion, fracture-boundary pretraining, anchor-referenced flow matching, and a training-only non-overlap constraint. The pretrained pointwise representation conditions the flow model; inference does not require ground-truth fracture masks or assembled poses. Datasets without color use the no-color variant, so those experiments do not directly validate color fusion. Dashed arrows below indicate training supervision; the non-overlap constraint is not a collision-repair step after inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Centered fragment clouds<br/>coordinates, normals, available RGB"] --> Encode["Equivariant Geometry<br/>and Color Fusion"]
    Encode --> Pretrain["Fracture-Boundary Pretraining"]
    Masks["Ground-truth boundary masks<br/>pretraining only"] -.-> Pretrain
    Pretrain -->|learned pointwise representation| Flow["Anchor-Referenced<br/>Flow Matching"]
    Flow --> Output["Per-fragment rigid transforms<br/>assembled point cloud"]
    Flow -.->|predicted assembly during training| Overlap["Training-Only<br/>Non-Overlap Constraint"]
    Overlap -.->|penalize interpenetration| Flow

Key Designs

1. Equivariant Geometry and Color Fusion: preserve orientation without treating RGB as spatial vectors

The geometric branch treats each point's coordinates and normal as two three-dimensional vector channels and processes them with a Vector Neurons-based VN-Transformer. Its outputs remain vector channels, so rotating an input should rotate the outputs consistently instead of producing unrelated descriptors. Point ordering must not change the object's meaning either: permuting input points should correspondingly permute pointwise outputs. The paper expresses these requirements as joint equivariance to point permutations and rotations. The following equivalent shorthand isolates the rotation component, with rotation acting on coordinates, normals and output vector channels:

\[ \Phi_{\mathrm{geo}}(R\cdot F_{\mathrm{geo}})=R\cdot\Phi_{\mathrm{geo}}(F_{\mathrm{geo}}). \]

The color branch uses a standard Transformer over per-point RGB; it respects point permutations, but spatial rotations act as the identity on its color channels. The branches are then concatenated along the feature-channel axis at each point, rather than appending raw RGB to coordinates and feeding everything into one geometric network. The fused representation contains a rotating geometric block and a nonrotating appearance block, so its entire last dimension must not be interpreted as spatial direction. This distinguishes it from the simple color-concatenation version of GARF in Table 1. For repetitive fracture surfaces, geometry supplies contact orientation and shape, while color helps identify which surface patterns plausibly belong together. The paper constructs equivariance for the encoder; this does not automatically establish an equally strict end-to-end guarantee for the downstream pose network after Fourier features are added.

2. Fracture-Boundary Pretraining: teach the representation to attend to assembly interfaces

Contact interfaces largely determine reassembly, but many input points lie on the object's original exterior surface. Direct pose training could favor overall shape or color statistics without teaching the encoder to distinguish fracture regions. The paper attaches a two-layer MLP segmentation head to the fused encoder and predicts a fracture-boundary probability for every point. Binary ground-truth masks supervise the encoder and segmentation head jointly. This embeds information about potential contact regions in the pointwise representation before assembly learning.

It is not a hard pipeline that crops fracture surfaces and matches only the remaining points. The downstream model retains learned pointwise features, allowing appearance outside the boundary to inform discrimination. The segmentation head provides an auxiliary learning task rather than correct assembly answers at test time. The main text specifies binary supervision and the two-layer MLP but does not establish the complete segmentation-loss implementation. It would therefore be unjustified to insert assumed cross-entropy weights, class balancing or an encoder-freezing schedule.

3. Anchor-Referenced Flow Matching: remove global-frame ambiguity before continuously generating relative poses

The fused representation is concatenated with multi-frequency sinusoidal positional encodings of coordinates, normals and fragment scale, then processed by a shape embedding function. These Fourier features provide high-frequency geometric conditioning, not an additional source of color. A common rigid transformation leaves the assembled object's identity unchanged, so shape recovery alone leaves a shared rotation and translation undetermined. A learned anchor token designates a reference fragment and defines the other fragment poses around its frame. Training chooses the anchor randomly; inference uses the largest fragment, supporting robustness to reference choice and stable placement respectively.

The pose generator is a conditional SE(3) Riemannian flow-matching network with a Transformer-based architecture that predicts time-dependent rotational and translational velocities. It conditions on fragment representations and receives the current pose and time instead of directly predicting all final coordinates in one step. Training paths interpolate rotations along geodesics and translations linearly. Target velocities use the rotational logarithmic residual and translational residual to the target, scaled by the remaining time. The model thus learns the remaining motion, without arbitrarily interpolating rotation-matrix entries as ordinary Euclidean coordinates.

Inference uses Euler integration to update poses along the learned vector field, then applies the resulting rigid transforms to the centered fragments. The main text describes a training initial distribution of uniform rotations and Gaussian translations, but its inference equation shows identity rotation and zero translation; Figure 2 also uses a time direction inconsistent with the prose. This note therefore confirms continuous pose flow and Euler sampling without presenting one initialization or time convention as verified implementation behavior. Equation (7) is badly corrupted in the cache, so the readable mechanism is explained without reconstructing the authors' exact flow loss. The main-text cache also does not establish the sampling step count, optimizer or learning rate.

4. Training-Only Non-Overlap Constraint: avoid rewarding interpenetrating placements

Moving fragments near their ground-truth locations can still produce local interpenetration, particularly when fracture geometry is ambiguous. The paper represents placed fragments as soft occupancy masks on a regular volumetric grid, with values between 0 and 1. For each pair, the soft intersection sums per-cell minima, while the soft union sums per-cell maxima with a stabilizing constant added to the denominator. Their ratio defines soft IoU, and averaging it over the considered fragment pairs gives the non-overlap loss. Higher overlap incurs a larger penalty, constraining volume relationships rather than merely fragment centroids.

The total objective adds the weighted non-overlap term to flow matching; the relation below follows the prose accompanying Equation (10):

\[ \mathcal{L}=\mathcal{L}_{\mathrm{flow}}+\alpha\mathcal{L}_{\mathrm{no\text{-}overlap}}. \]

This term is used only during training, with no hard constraint solver guaranteeing collision-free inference. It is also not a contact constraint: moving fragments far apart can reduce overlap, so flow matching remains necessary to maintain correct relative placement. The main text does not establish the grid resolution, soft-occupancy construction or \(\alpha\) value available for verification here, preventing a complete reproduction of the collision module from this account alone. Removing this loss actually improves translation RMSE in Table 1, demonstrating that closer positions and more plausible surface relationships are different objectives.

A Worked Example

Consider fresco fragments with similar fracture surfaces but surviving painted lines; this illustrates the mechanism and is not an additional experiment reported by the paper. After centering each piece, the geometric branch represents fracture shape and normal orientation, while the color branch represents the pigment distribution. Rotating an input fragment should rotate its geometric representation without changing the color descriptor of the same physical point merely because of that rotation. Boundary pretraining makes the representation sensitive to possible contacts while preserving surrounding painted appearance. At inference, the largest fragment supplies the reference, and the flow model progressively adjusts the rotations and translations of the other pieces using both geometric and color evidence. The training loss has penalized interpenetrating solutions, discouraging that shortcut; ground-truth boundaries are not used again to correct test-time results.

Loss & Training

Training has a fracture-boundary segmentation pretraining stage and a conditional pose-flow stage, with non-overlap regularization in the latter. Evaluation samples 5k points per object while maintaining uniform per-fragment density; this is not 5k points for every fragment. RePAIR supplies the colored-model source data, and Presious is used only for cross-dataset testing without target-dataset training or fine-tuning. Breaking Bad supplies geometric training data, with Fantastic Breaks and FRACTURA evaluating generalization without color. The generalization paragraph uses “fine-tune from” ambiguously; this note follows the table headings and explicit statement that Presious is not used for fine-tuning.

Key Experimental Results

Main Results

Rotation RMSE is measured in degrees and translation RMSE in millimeters, both lower being better; CD is Chamfer Distance between the assembled and ground-truth object clouds, also lower being better. PA is the percentage of fragments whose individual Chamfer Distance falls below a threshold, with higher values being better. Page 11 writes this threshold as “0.01%” without clearly explaining its normalization; the CD table heading also omits a scale factor, so reported values are retained without inferring physical distance.

The following RePAIR test results are selected from main-text Table 1, page 12.

Method	Rotation RMSE (degrees)	Translation RMSE (mm)	PA (%)	CD
DiffAssemble	69.54	67.96	17.92	4.18
GARF	49.31	32.19	31.14	2.66
GARF + simple color concatenation	46.56	31.82	31.92	2.25
E-M3RF	37.91	27.93	35.91	2.17

Relative to GARF, the full model reduces rotation error by approximately 23.1%, translation error by 13.2%, and CD by 18.4%. PA rises from 31.14% to 35.91%, a gain of 4.77 percentage points or approximately 15.3% relative, not 15.3 percentage points. Simple color concatenation also improves GARF, so the full model should be compared with this stronger colored baseline rather than attributing every gain to merely using color.

Ablation Study

The following RePAIR ablations come from the same Table 1, page 12; all values are taken directly from the table.

Config	Rotation RMSE (degrees)	Translation RMSE (mm)	PA (%)	CD
Full model	37.91	27.93	35.91	2.17
Without color	40.17	29.01	33.71	2.98
Without non-overlap loss	45.81	25.01	32.91	3.83
Rotation-invariant representation	43.27	41.60	32.50	3.71
Without rotation equivariance	43.22	29.28	32.26	3.07

Removing color lowers PA by 2.20 percentage points and raises CD from 2.17 to 2.98, supporting complementary appearance evidence. Removing non-overlap loss raises CD from 2.17 to 3.83 and lowers PA by 3.00 percentage points, but reduces translation RMSE from 27.93 to 25.01 mm. The authors attribute this reversal to interpenetration bringing centroids closer while worsening surface placement; the metrics support that trade-off, but no separate collision rate quantifies it. The rotation-invariant variant achieves 32.50% PA, 3.41 percentage points below the full model; page 13's “9.5 pp” conflicts with the table, whereas approximately 9.5% is the relative decrease.

Key Findings

The cross-dataset results below come from Table 1, page 12, and Table 2, page 13; Presious transfers from RePAIR, while the other datasets transfer from Breaking Bad.

Test dataset	Method	Rotation RMSE (degrees)	Translation RMSE (mm)	PA (%)	CD
Presious	GARF	43.33	14.24	50.27	18.01
Presious	GARF + simple color concatenation	42.05	14.11	51.95	17.05
Presious	E-M3RF	30.26	12.85	57.49	20.95
Fantastic Breaks	GARF	10.62	2.10	91.00	2.12
Fantastic Breaks	E-M3RF, no color	7.96	2.02	92.01	2.19
FRACTURA Real	GARF	45.03	1.18	46.36	1.14
FRACTURA Real	E-M3RF, no color	37.36	0.15	52.90	0.91

On Presious, the full model achieves higher PA and lower rotation error, but its CD of 20.95 is worse than both GARF versions, precluding an all-metrics superiority claim. Fantastic Breaks also shows slightly worse CD; translation improves from 2.10 to 2.02 mm, not by approximately 25% as stated in the prose. The gains on FRACTURA Real are more consistent, with all four metrics better than GARF, but this does not establish reliable reconstruction of all real artifacts. Breaking Bad Artifact CD in Table 2 falls from GARF's 0.42 to 0.20, approximately 52.4% by the table rather than the 35.5% stated on page 14. Table 3, page 15, reports RePAIR efficiency: with 40 fragments, E-M3RF uses 223.0 ms and 2.27 GB, compared with GARF's 298.0 ms and 5.38 GB. The same page's 1.9M versus 12.7M parameter counts refer to trainable parameters during pretraining, not verified total parameter counts for the complete systems.

Highlights & Insights

The most transferable design is to separate modalities by their transformation rules rather than concatenate every numeric input indiscriminately. Coordinates, normals and color can all be written as triplets, yet they should not share the same spatial rotation action. Rotation invariance and equivariance are not interchangeable robustness techniques for reassembly: equivariance retains directional information that the output must recover. Auxiliary boundary supervision makes assembly-relevant representation learning an explicit pretraining objective instead of relying entirely on final pose error. The non-overlap ablation shows how translation RMSE alone can reward incorrect spatial relationships, motivating joint inspection of surface distance, part accuracy and collisions.

Limitations & Future Work

The authors note that increasing point density increases computational cost and propose linearized attention and more efficient architectures. They also propose moving from point clouds to triangle meshes and textures to exploit connectivity and detailed appearance; this is not an implemented component of the present method. Presious contains only six sets, making its transfer results informative but insufficient to cover all variations in material, erosion, imaging conditions and fragment count. The supplied cache contains only the main paper, not its cited supplementary material, so point-density curves, further parameter ablations and detailed computational settings cannot be confirmed here. Physical plausibility is mainly supported by soft-overlap penalties and qualitative figures, without a separate collision rate, contact-quality metric or strict zero-interpenetration guarantee. Missing color, illumination changes and incorrect appearance correspondences warrant further study; the reported experiments do not establish robustness to these conditions. Numerical prose, initialization notation, some tensor dimensions and corrupted extracted equations introduce uncertainty, so reproduction should consult the formatted paper and actual implementation rather than copy these passages.

Related Work & Insights

Compared with GARF: GARF is the main geometric reassembly baseline; E-M3RF retains learned assembly while strengthening equivariant representations, color conditioning and physical supervision. The simple color-concatenation baseline in Table 1 is an important control, but the full-model difference cannot be assigned entirely to one component. Compared with VN-Transformer: The paper reuses vector features and equivariant attention within fracture reassembly and appearance fusion rather than inventing equivariant attention anew. Compared with DiffAssemble and PuzzleFusion++: These methods also use progressive generation or denoising, whereas E-M3RF follows a continuous velocity field on pose space through SE(3) flow matching. Its distinction is not merely generative modeling, but orientation-consistent multimodal conditioning together with an explicit training cost for interpenetration. A useful extension would jointly measure boundary contact, texture continuity and volume penetration to determine whether the model restores genuine contacts or merely achieves favorable average distances; this is a reader suggestion, not a reported result.

Rating

The following are subjective reader assessments, not scores supplied by the paper. - Novelty: 4/5. Transformation-aware appearance and geometry are integrated purposefully into flow-based reassembly, although the underlying components have established predecessors. - Experimental Thoroughness: 4/5. Five datasets, a colored baseline and core ablations provide breadth, with limits in small-scale transfer, collision quantification and supplementary evidence available here. - Writing Quality: 3/5. The pipeline is clear, but several percentage, percentage-point and all-metrics claims disagree with tables, while corrupted cached equations complicate reproduction. - Value: 4/5. The work offers practical ideas for cultural-heritage reassembly, especially joint modeling of directional information and physical constraints.