Beyond Categorical Matching: Intra-Class Graded Relevance Estimation for Cross-Modal 3D Retrieval¶

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://github.com/Listeningx/objaverse-benchmark-generation
Area: Multimodal VLM / 3D Vision
Keywords: 3D asset retrieval, intra-class graded relevance, multimodal fusion, mixture of experts, soft labels

TL;DR¶

ReMU3D reparameterizes the joint image-text-point-cloud space with SCI and then aggregates arbitrary query-modality combinations through Q-MoE, raising NDCG@10 from the best baseline's 0.76 to 0.93 on the new INGRE graded-relevance benchmark.

Background & Motivation¶

Cross-modal 3D retrieval is usually treated as category matching: a query and candidate are relevant if they share a class and irrelevant otherwise. Methods such as ULIP, OpenShape, and Uni3D can therefore separate a chair from a table, yet compress differences in armrest geometry, material, color, and usage context within the chair class. A real user rarely wants an arbitrary chair; they want the asset closest to a reference in style and structure. Binary labels cannot express that ranking preference, and conventional Recall or mAP can hide intra-class errors.

Once relevance becomes continuous, the model faces two coupled problems. Point clouds convey structure but omit many appearance and semantic cues, while directly aligning them to a pretrained CLIP space can inherit its intra-class semantic collapse. Meanwhile, a query may contain text, an image, a point cloud, or any combination, so fixed averaging cannot react to modality availability. Evaluation has the same gap: no scalable 3D benchmark supplies continuous relevance judgments aligned with human preference.

Core idea: reformulate cross-modal 3D retrieval from category-level binary matching into intra-class graded ranking, preserve fine-grained variation through SCI, adapt to arbitrary modality combinations through Q-MoE, and evaluate the ranking with MLLM-generated soft relevance labels in INGRE.

Method¶

Overall Architecture¶

Built on Uni3D, ReMU3D accepts any subset of image, text, and point-cloud inputs. SCI first places the available modalities in one Transformer so appearance and language conditions can reorganize the point representation; Q-MoE then sends learned query tokens through experts routed according to the input modalities, pooling and normalizing them into the final retrieval vector. Training still uses image-text-3D contrastive learning rather than fitting INGRE labels, while INGRE forms a separate MLLM annotation and ranking-evaluation pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image, text, point cloud<br/>any modality subset"] --> B["SCI Semantic-Conditioned Interaction<br/>reparameterize joint space"]
    B --> C["Q-MoE Query-Guided Aggregation<br/>route among experts"]
    C --> D["Normalized retrieval vector<br/>cosine-similarity ranking"]
    E["INGRE Soft-Label Construction<br/>MLLM graded relevance"] --> F["NDCG and rank correlation<br/>evaluate intra-class ranking"]
    D --> F

Key Designs¶

1. SCI Semantic-Conditioned Interaction: expand intra-class variation during fusion instead of forcing 3D back into CLIP category clusters

SCI extracts image, text, and point-cloud tokens, concatenates whichever are available, and processes them with a four-layer, 16-head Transformer. Each layer first applies global multi-head self-attention over the whole sequence, allowing every token to read every modality. It then splits tokens by modality, applies modality-specific feed-forward networks with residual normalization, and concatenates them again for the next layer. Image tokens contribute material, color, and style; text tokens contribute function and context. Rather than being averaged with the point vector at the end, they alter which geometric regions receive attention at every layer.

This design delays alignment until after interaction in the unified space. Samples packed into the same pretrained CLIP category cluster can spread out according to 3D structure and cross-modal conditions, preserving category separation while reserving capacity for intra-class ranking. If a modality is absent, SCI simply operates on the remaining tokens without an architectural substitution.

2. Q-MoE Query-Guided Aggregation: change how one model summarizes evidence for the modalities actually present

After SCI, the model appends 30 learnable query tokens that extract retrieval-relevant information from the fused sequence. Q-MoE has three experts. Its router reads global context, infers the active modality set, and produces expert weights summing to one; the weighted output is

\[ \mathbf{H}=\sum_{k=1}^{K}w_k\mathcal{E}_k(\mathbf{F}_{\mathrm{MoE}}), \qquad \sum_{k=1}^{K}w_k=1. \]

The model mean-pools the updated query tokens, applies \(\ell_2\) normalization, and retrieves by cosine similarity. Unlike fixed averaging, routing can favor geometry-oriented experts for point-only queries and appearance or semantic experts for image-text combinations. Training also adjusts modality-combination sampling ratios from validation performance and assigns higher probability to combinations containing point clouds, making missing-modality robustness a learned behavior rather than test-time zero filling.

3. INGRE Soft-Label Construction: unfold “same class” into a query-dependent continuous ordering

INGRE combines ESB, NTU, ModelNet40, GSO, and Objaverse and spans more than 70 categories. For geometry-dominant data with little appearance information, the pipeline builds category-wise query cases, scores candidates along fixed dimensions such as color, function, and texture, and inserts cross-class distractors. For large, heterogeneous Objaverse data, the authors first curate 10,742 high-quality assets. An MLLM selects the five most important dimensions for each query, assigns weights, writes multidimensional candidate descriptions, computes per-dimension and aggregate scores, and finally checks ranking consistency.

Evaluation substitutes continuous soft relevance for binary relevance in NDCG. For the first \(K\) results,

\[ \mathrm{DCG}@K=\sum_{i=1}^{K}\frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)}, \qquad \mathrm{NDCG}@K=\frac{\mathrm{DCG}@K}{\mathrm{IDCG}@K}. \]

The paper emphasizes NDCG@5/10 and uses Kendall's \(\tau\) and Spearman's \(\rho\) to measure relative ordering within a class. Candidate lists also contain zero-relevance objects from other classes, so the protocol does not reward intra-class ranking while ignoring class separation. On 100 manually reviewed query-centered lists, MLLM-to-expert agreement is \(\tau=0.68\), close to the 0.76 inter-expert agreement.

Loss & Training¶

The model does not regress INGRE scores. It uses bidirectional InfoNCE over matched 3D-text and 3D-image pairs, averaging the four directions. If \(\mathbf e^A,\mathbf e^I,\mathbf e^T\) denote normalized 3D, image, and text embeddings, each direction raises the similarity of the same-index positive relative to other batch samples. The temperature starts at \(\ln(1/0.07)\approx2.66\).

Optimization uses AdamW, a \(10^{-4}\) base learning rate, 0.1 weight decay, 500 warmup steps, and cosine decay; the point encoder uses 0.95 layer-wise learning-rate decay. Training on eight RTX H20 GPUs uses batch size 36 per GPU and eight-step accumulation for an effective batch of 2,304, with ZeRO-2 and checkpointing for 589M trainable parameters. With frozen features precomputed, online encoding costs 167--210 GFLOPs and about 0.03 seconds per query.

Key Experimental Results¶

Main Results¶

Method	NDCG@5	NDCG@10	NDCG@20	NDCG@50	Kendall's \(\tau\)	Spearman's \(\rho\)
ULIP	0.69	0.71	0.72	0.88	0.22	0.30
OpenShape	0.76	0.76	0.79	0.88	0.30	0.41
Uni3D	0.71	0.73	0.75	0.88	0.32	0.43
ReMU3D	0.94	0.93	0.92	0.91	0.39	0.52

ReMU3D leads the best baseline by 0.18 and 0.17 on the top-heavy NDCG@5/10 metrics. For global ordering, Kendall's \(\tau\) and Spearman's \(\rho\) exceed Uni3D by 0.07 and 0.09. The smaller NDCG@50 margin suggests that most gains occur among the candidates users see first.

Ablation Study¶

Configuration	NDCG@5	NDCG@10	Kendall's \(\tau\)	Spearman's \(\rho\)
Full ReMU3D	0.94	0.93	0.41	0.54
Without SCI	0.57	0.61	0.15	0.22
Without Q-MoE	0.66	0.70	0.24	0.35
Two-stage training	0.41	0.47	0.04	0.07
Fixed modality ratio	0.68	0.70	0.18	0.26

Removing SCI causes the largest module-level loss, dropping NDCG@10 by 0.32, so unfolding intra-class representations matters even more than dynamic aggregation. Training SCI and Q-MoE in separate stages falls to 0.47, supporting the need for both to shape one continuous embedding space jointly; fixed modality sampling is also clearly weaker than adaptive ratios.

Key Findings¶

ReMU3D obtains Top-1 accuracy of 53.2, 89.3, and 68.1 on Objaverse-LVIS, ModelNet40, and ScanObjectNN zero-shot classification, respectively, each above the corresponding best baseline in the table. Intra-class expansion therefore does not destroy category recognition.
Adding a soft-label MSE objective to InfoNCE produces 0.93/0.93 NDCG@5/10, while adding KL gives 0.92/0.91; neither beats the original 0.94/0.93. The gain comes from discriminative architecture rather than memorizing MLLM scores.
Cross-MLLM annotation agreement is \(\tau=0.58\), below MLLM-human agreement of 0.68 and human-human agreement of 0.76. The labels are useful but remain sensitive to the annotation model.

Highlights & Insights¶

The paper first repairs the task definition instead of only optimizing an old binary category metric. Continuous relevance distinguishes “retrieved the right class” from “ranked the closest asset first,” making progress more faithful to an asset-search experience.
SCI is more than another fusion Transformer: delaying alignment lets semantic conditions reorganize point tokens before projection. The same idea may transfer to medical-image or product retrieval settings where inter-class recognition is easy but intra-class ordering is hard.
Using soft labels for evaluation but not the main training objective is a valuable separation. It reduces the risk of optimizing for an automatic annotator, and the soft-label-loss ablation further argues against label leakage.

Limitations & Future Work¶

INGRE labels come from an MLLM. Although humans audit them, the study covers only 100 candidate lists, and cross-MLLM agreement of 0.58 shows that dimension choice and weighting can change with the annotator. Larger human audits and confidence intervals stratified by category, source, and attribute would strengthen the benchmark.
Curating 10,742 high-quality Objaverse assets improves reliability but may underrepresent broken meshes, missing renders, and long-tail noise in production repositories. An uncurated split and cross-domain stress tests would clarify robustness.
ReMU3D has 589M trainable parameters and is trained on eight H20 GPUs. Queries are fast after precomputation, but training and indexing remain expensive; Q-MoE distillation, sparse expert activation, and a smaller point backbone are natural next steps.
The main INGRE table aggregates several sources, making it hard to tell whether gains concentrate in particular categories or modalities. Per-dataset, missing-modality, and fine-grained-attribute breakdowns would better define the operating envelope.

vs Uni3D / ULIP: These methods align images, text, and point clouds for open-vocabulary category transfer. ReMU3D retains contrastive training but reparameterizes intra-class structure with SCI and Q-MoE, gaining fine-grained ranking and modality-combination robustness at the cost of a heavier model.
vs OpenShape: OpenShape obtains strong category-level features through large-scale 3D pretraining and is the best INGRE baseline at NDCG@5/10. ReMU3D raises NDCG@10 from 0.76 to 0.93, showing that scaling alignment alone does not automatically resolve intra-class semantic collapse.
vs conventional 3D retrieval benchmarks: ModelNet, ShapeNet, and ScanObjectNN mainly provide category labels, while PartNet and 3D-Future provide part or furniture attributes. INGRE adds query-conditioned continuous relevance, but its automatic labels still require ongoing human calibration.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The task formulation, model, and benchmark form a coherent loop; graded relevance is more consequential than an isolated architectural tweak.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Main results, module and training ablations, label reliability, and zero-shot generalization are covered, though source-wise and missing-modality breakdowns remain limited.
Writing Quality: ⭐⭐⭐⭐☆ The motivation and module-level causal story are clear, but some equations are poorly typeset and expert-count or pooling trends appear only in plots.
Value: ⭐⭐⭐⭐⭐ The work targets real 3D asset-search ranking needs, and the open benchmark-generation framework can improve comparability for future research.