2D Features Are All You Need for 3D Shape Understanding¶

Conference: ECCV 2026
Paper: Official paper page · PDF
Project: MeshFM
Authors: Jinfan Zhou, Richard Liu, Itai Lang, and Rana Hanocka
Area: 3D Vision
Keywords: 2D feature distillation, continuous feature fields, SAM boundary correction, triplane representation, rotation robustness

The manuscript title includes the method-name prefix MeshFM. The project address appears in the cached abstract; project availability and code release status were not checked online for this note.

TL;DR¶

MeshFM fits corrected 2D foundation-model features into continuous 3D teacher fields, then trains a rotation-augmented feedforward predictor to reproduce them without manual 3D annotations, supporting segmentation, correspondence, and deformation with the same features and achieving 0.549/0.539 semantic mIoU on original/rotated PartObjaverse-Tiny shapes.

Background & Motivation¶

2D foundation models already capture rich semantics and part relationships, but transferring that knowledge to a 3D surface is not simply a matter of projecting image features onto a mesh. Methods such as Diff3F and DFD require shape-specific rendering, feature extraction, and distillation processing, making inference relatively expensive. PartField and DenseMatcher instead incorporate 2D priors into 3D network training, but their objectives emphasize segmentation and dense correspondence, respectively, and may not preserve all the information needed across tasks.

MeshFM traces part of the problem to supervision quality. Coarse mesh vertices cannot fully exploit per-pixel evidence, while ViT patchification mixes features across object-background and part boundaries. A powerful 3D model can still end up faithfully fitting contaminated targets. The authors therefore hypothesize that some apparent need for task-specific 3D representations reflects imperfect transfer from 2D rather than an intrinsic lack of useful information in 2D descriptors. This is a hypothesis tested on selected tasks, not a universal result about all 3D understanding.

To obtain both clean features and efficient inference, the method reserves expensive per-shape distillation for training preparation and learns a predictor that generalizes to new shapes. Core idea: use SAM masks to repair boundary contamination in 2D features, then distill continuous 3D teacher fields into a rotation-augmented feedforward network so that shared features, rather than task-specific prediction heads, carry most of the shape understanding.

Method¶

Overall Architecture¶

Training uses 3D shapes and their multi-view renderings. DINOv2 supplies the main feature supervision, while SAM supplies boundaries for correcting it. Stage 1 independently optimizes a continuous 3D teacher field for each training shape; Stage 2 trains a shared network to predict triplanes from surface-sampled point clouds and regress teacher features at queried positions.

For a new shape, there is no teacher-field optimization at test time. A forward pass produces triplanes from the point cloud, after which features can be queried where needed. Segmentation uses clustering, correspondence uses regularized functional maps, and deformation uses the DFD handle-based framework. These downstream operations still have to run: a single forward pass describes feature prediction, not an entire task without postprocessing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training mesh<br/>Multi-view 2D features"] --> B["SAM Boundary Correction"]
    B --> C["Continuous Teacher-Field<br/>Distillation"]
    C -->|Frozen field supervision| D["Feedforward Triplane<br/>Prediction"]
    P["Surface-sampled<br/>point cloud"] --> D
    D --> E["Progressive Rotation<br/>Augmentation"]
    E --> F["Trained predictor<br/>Queries on new shapes"]
    F --> G["Clustering / Functional maps<br/>Handle-based deformation"]

Progressive rotation augmentation denotes the later part of Stage 2 training, not an extra inference module. The teacher, predictor, and downstream algorithms have distinct roles: the teacher cleans and consolidates supervision, the predictor generalizes across shapes, and downstream algorithms turn features into task outputs.

Key Designs¶

1. SAM Boundary Correction: replace only within-region feature outliers

When an image patch covers both a thin part and its background, interpolated descriptors can lose the true boundary before any 3D processing occurs. Optimizing a 3D field does not tell the model which mixed descriptors are wrong. MeshFM generates disjoint SAM masks for each rendered view, assumes each mask mainly covers a semantically consistent part, and computes the channel-wise median of its feature vectors. Rather than averaging an entire region indiscriminately, it replaces only descriptors that deviate substantially from this representative, retaining normal variation within the region.

The cached equations have extraction defects. The following equivalent expression is reconstructed from the adjacent prose: replace a pixel feature with its segment median when their Euclidean distance exceeds the threshold, and otherwise leave it unchanged.

\[ \widetilde{\mathbf f}_{u}= \begin{cases} \boldsymbol\mu_i, & \|\mathbf f_u-\boldsymbol\mu_i\|_2>\tau,\\ \mathbf f_u, & \text{otherwise}, \end{cases} \qquad \boldsymbol\mu_i=\operatorname{median}_{u\in M_i}\mathbf f_u,\quad \tau=1. \]

The median is computed independently per channel. The threshold of 1 is used with the paper's unit-normalized features and should not be transferred blindly to arbitrary feature scales. SAM primarily provides reliable partitions here; it does not replace DINOv2 as the semantic feature source. This correction also depends on the mask actually describing a coherent region: if it merges different parts, valid fine detail can be mistaken for an outlier.

2. Continuous Teacher-Field Distillation: supervise surface positions beyond coarse vertices

After correction, the method adopts DFD's barycentric feature distillation approach to associate rendered pixels with positions on mesh triangles and aggregate supervision across views. A neural field learns to map 3D coordinates to features instead of merely storing a vector at each existing vertex. The important distinction is that supervision density is no longer determined by mesh triangulation: many pixels can supervise locations inside one large triangle.

Each training shape has its own fitted teacher field, but all fields regress the feature space of the same 2D foundation model. Their outputs therefore share a semantic coordinate system, giving the second stage a common target across shapes. Being continuous means the field can be queried at coordinates; it does not establish reliable semantic supervision on never-observed surface regions or arbitrary off-surface points. Field fitting also cannot recover distinctions that the original 2D model never encoded.

3. Feedforward Triplane Prediction: turn a point cloud into queryable surface features

Stage 2 normalizes the shape to the unit cube and samples its surface to obtain a point cloud. A PVCNN encoder extracts per-point geometric features, which are orthogonally projected by mean reduction onto the XY, XZ, and YZ planes. These planes are a compact, interpolatable representation of a 3D field, not three new images sent through a semantic recognition model.

A 2D CNN downsamples the initial planes, a Transformer processes their flattened representation to capture global context, and a transposed 2D CNN upsamples the result into the final triplanes. To query a surface point, the method projects its coordinates onto each plane, bilinearly interpolates the three feature vectors, and sums them. The shared network can thus consume a sampled point cloud and answer queries on different vertex or surface-point sets without training a separate feature prediction head for each downstream task.

4. Progressive Rotation Augmentation: reuse 3D targets instead of regenerating 2D supervision

Axis-aligned triplanes and PVCNN are not intrinsically rotation-invariant. MeshFM first trains without rotation until convergence, then adds 365k iterations with random rotation angles along all three axes. During the first 125k of those iterations, the maximum rotation angle increases linearly to \(2\pi\). The predictor gradually encounters arbitrary orientations instead of relying exclusively on fixed coordinates as part cues.

The schedule benefits from having converted multi-view image supervision into queryable 3D teacher fields: Stage 2 does not need to regenerate SAM partitions for every augmented example. An implementation must preserve geometric consistency between rotated inputs, query locations, and corresponding teacher targets. The cached text does not spell out the coordinate-transform code or establish that angle sampling is uniform over SO(3), so those details should not be assumed. The evidence supports learned rotation robustness, not a strict group-equivariance guarantee.

A Worked Example¶

Consider a chair with thin legs and a backrest as an illustrative input. During training, multi-view renderings provide DINOv2 features, and SAM divides visible regions into parts. If some leg-boundary pixels mix in background descriptors, the correction replaces those outliers with the median of their region. Projecting the cleaned signals onto the surface provides supervision for this chair's teacher field.

The shared predictor then generates triplanes from surface points alone. Training queries those planes at sampled positions and aligns their features with the frozen teacher. Subsequent rotation augmentation encourages usable features for corresponding regions under different orientations. On a new chair, the trained predictor needs no new teacher optimization: clustering can produce parts, or functional maps can match it to another shape. Symmetric left and right legs can still be confused. This example explains the pipeline and is not a separately measured experimental sample.

Loss & Training¶

Stage 2 uniformly samples query points over the shape surface and regresses the frozen teacher-field outputs with an L2 feature loss. Equation 3 is damaged in the cached text, so this note retains the unambiguous training objective without inventing missing squaring or normalization notation. No additional part-label loss, correspondence-label loss, or task-specific feature fine-tuning is introduced.

The paper reports 4 L40S GPUs, 4 objects per GPU, Adam with learning rate \(10^{-4}\), and approximately 126M parameters. The main text does not specify a complete training-shape inventory, the number of iterations before initial convergence, total teacher-generation time, or inference latency. A feedforward predictor alone does not establish a particular real-time frame rate. Likewise, no manual 3D annotations does not mean no 3D data: training still uses shape geometry and 3D teacher targets generated from 2D features.

Key Experimental Results¶

Main Results¶

Segmentation is evaluated on 200 PartObjaverse-Tiny shapes and the 1,906-shape PartNetE test set. All compared features use agglomerative clustering, with the cluster count set to the number of ground-truth labels, followed by Hungarian label matching and mIoU evaluation. The rotated evaluation samples 5 random SO(3) rotations per shape. The table preserves the paper's decimals and error values rather than mixing them with percentages.

Task / Dataset	Setting and metric	Diff3F	PartField	DenseMatcher	MeshFM
Semantic segmentation / PartObjaverse-Tiny	Original mIoU, higher better	0.515	0.542	Not reported	0.549
Semantic segmentation / PartObjaverse-Tiny	Rotated mIoU, higher better	0.488	0.393	Not reported	0.539
Semantic segmentation / PartNetE	Original mIoU, higher better	0.496	0.517	Not reported	0.520
Semantic segmentation / PartNetE	Rotated mIoU, higher better	0.501	0.499	Not reported	0.510
Dense correspondence / DenseCorr3D, all classes	Original AUC / Err	0.46 / 11.8	0.35 / 17.1	0.50 / 10.2	0.52 / 9.6
Dense correspondence / DenseCorr3D, all classes	Rotated AUC / Err	0.42 / 13.8	0.24 / 21.0	0.46 / 11.8	0.50 / 10.5
Dense correspondence / TOSCA	Original AUC / Err	0.48 / 17.1	0.25 / 19.3	0.11 / 31.5	0.34 / 18.8
Dense correspondence / TOSCA	Rotated AUC / Err	0.28 / 24.8	0.05 / 39.5	0.02 / 41.7	0.33 / 19.3

These values come from Tables 1-4. DenseMatcher fails on non-manifold inputs in the segmentation datasets, so its missing entries must not be represented as zero. Correspondence uses the same regularized functional-map procedure for all methods. Err is the reported normalized geodesic error, lower is better; AUC uses a 10% threshold, higher is better. TOSCA provides 420 shape pairs, and DenseCorr3D has 306 test pairs spanning 24 classes, including 3 held-out classes.

On the original held-out DenseCorr3D classes, MeshFM has AUC/Err of 0.51/7.2 versus DenseMatcher's 0.52/6.8. Under rotation, these become 0.51/7.0 and 0.50/7.8, respectively. Strong all-class averages therefore do not imply superiority over specialized methods on every subset.

Classification mean-pools features and trains a three-layer MLP, so it is not a zero-shot result without task training. Table 5 reports original/rotated Manifold40 accuracy of 0.92/0.85 for MeshFM, 0.88/0.33 for Diff3F, 0.85/0.41 for PartField, and 0.82/0.28 for DenseMatcher. Mesh deformation is demonstrated qualitatively through control handles and DFD, without a separate quantitative error table.

Ablation Study¶

Teacher-field instance segmentation on PartObjaverse-Tiny	Without SAM correction, mIoU	With SAM correction, mIoU	Absolute change
Average	0.630	0.661	+0.031
Food	0.620	0.778	+0.158
Plants	0.592	0.633	+0.041
Human-Shape	0.680	0.714	+0.034
Daily-Use	0.653	0.647	-0.006

This is the teacher-field ablation in Table 6, not the final MeshFM semantic-segmentation result. Comparing 0.661 directly with the main table's 0.549 does not measure a student distillation gap. Changes are calculated by subtracting the two source columns; the average gain of 0.031 equals 3.1 percentage points, not a relative gain of 3.1%. Daily-Use declines, so the results do not support the broad wording that every category improves.

Teacher-field semantic segmentation on PartNetE, Table 7	Average mIoU
DINOv2	0.54
RADIO	0.54
SAM2	0.51
DINOv3	0.54

This ablation changes the 2D semantic feature source, rather than toggling the SAM mask correction used in the previous table. Values retain the original two-decimal precision. The authors attribute SAM2's lower performance to cross-view inconsistency, but do not provide a further quantitative experiment that isolates that explanation.

Key Findings¶

MeshFM's average semantic mIoU on PartObjaverse-Tiny moves from 0.549 to 0.539, a drop of 0.010; PartField moves from 0.542 to 0.393, a drop of 0.149. Robustness is the more pronounced advantage, but asymmetric rotation training affects causal interpretation.
Diff3F reaches 0.48 AUC on original TOSCA versus MeshFM's 0.34. MeshFM's strength is retaining 0.33 under rotation, not winning every original correspondence benchmark.
SAM correction helps on average rather than in every category. Similar teacher-field results across several foundation models establish some substitution flexibility only within the evaluated datasets and metrics.

Highlights & Insights¶

Supervision quality and inference efficiency are addressed separately. Expensive continuous teacher construction is amortized through a shared predictor, instead of asking one training objective to handle both noisy 2D transfer and cross-shape generalization.
SAM supplies boundaries without reducing all features to segmentation labels. Local median correction preserves the semantic feature space, allowing the resulting representation to remain useful for correspondence and deformation.
Shared downstream processing makes the comparison more informative about representation quality. PartField's stronger segmentation than correspondence performance illustrates that clusterable parts and cross-shape matchability are different properties.

Limitations & Future Work¶

The authors explicitly identify DINOv2's difficulty distinguishing symmetric left and right parts, causing mismatches between chair legs or animal limbs. Distillation does not automatically supply missing relative-position information.
The authors acknowledge that PartField did not receive the same rotation augmentation. The comparisons show that MeshFM's complete training recipe is more robust, but do not isolate teacher distillation or architecture as the cause of all gains.
Segmentation evaluation knows the true part count and uses optimal label matching. This measures feature clustering quality, not autonomous part-count selection or open-vocabulary labeling at deployment.
Teacher optimization remains expensive, and the main text omits timing, memory, and a complete training-data inventory, as well as a matched-augmentation predictor ablation. Strong claims about real-time performance or absence of data leakage cannot be established from the available evidence.
The assumption of semantic consistency within SAM regions can fail. Threshold sensitivity, loss of fine detail, and mask quality are not thoroughly disentangled. The referenced supplementary material is absent from this cache, so missing instance-segmentation tables and implementation hyperparameters are not reconstructed.

Compared with DFD: MeshFM reuses barycentric feature distillation and the deformation framework, but additionally learns a shared feedforward predictor from optimized teacher fields. Its central benefit is separating training-time optimization from feature queries on new shapes.
Compared with Diff3F: Both exploit 2D semantic priors; MeshFM emphasizes corrected teacher supervision and rotation augmentation. Diff3F remains stronger on original TOSCA, an important counterexample to blanket superiority claims.
Compared with PartField: Both use PVCNN and triplane-related structure, but MeshFM directly regresses teacher features while PartField emphasizes part clustering. Rotation augmentation is not controlled equally, so the observed difference cannot be assigned entirely to the training objective.
Compared with DenseMatcher: DenseMatcher is correspondence-oriented, whereas MeshFM seeks a single feature space usable across tasks. The specialized method's slight advantage on original held-out DenseCorr3D illustrates that generality and best performance on a particular subset are not equivalent.

Rating¶

These are the note author's subjective ratings out of 5, not conference review scores.

Novelty: 4/5. Boundary correction and two-stage general-purpose representation learning address a specific problem, although the backbone and continuous distillation reuse substantial prior machinery.
Experimental Thoroughness: 3/5. Segmentation, correspondence, classification, and deformation are covered, but matched rotation-training controls, timing measurements, and fuller component ablations are missing.
Writing Quality: 4/5. The two-stage argument is clear; some broad claims about improvements across categories are stronger than the actual tables support.
Value: 4/5. A practical route to cross-task shape features without manual 3D annotations, but not evidence that 2D descriptors have solved all of 3D understanding.