MemPose: Category-level Object Pose Estimation with Memory¶

Conference: ECCV 2026
Paper: ECCV Official
Code: None
Area: 3D Vision
Keywords: Category-level Pose Estimation, Geometric Memory Buffer, 3D Vision, Dynamic Update, Gated Fusion

TL;DR¶

Addressing the limitation of static shape priors and fixed network parameters in adapting to diverse object instances, MemPose incorporates an external, non-parametric category-level geometric memory buffer into the pose estimation pipeline, achieving state-of-the-art 9-DoF accuracy with near-zero learnable parameter overhead.

Background & Motivation¶

In downstream robotics and human-robot interaction, Category-level Object Pose Estimation (COPE) aims to estimate the 9-DoF pose (3D rotation, 3D translation, and 3D metric size) of arbitrary unseen instances within predefined categories from RGB-D inputs without requiring instance-specific CAD models. However, this setup is inherently challenging due to substantial intra-category variations in geometric structures, aspect ratios, and surface textures. When encountering new objects, human perception does not operate in isolation; humans naturally retrieve accumulated experiences from memory to conduct analogical reasoning across instances and continuously update their memory as new observations arrive.

In contrast, existing category-level pose estimation methods predominantly adhere to static paradigms. Prior-based frameworks (e.g., SPD, SGPA, DPDN) reconstruct instances by deforming fixed categorical shape prototypes, which are inflexible once constructed and require expensive offline pre-processing pipelines. On the other hand, parametric deep learning frameworks (e.g., HS-Pose, AG-Pose, SpherePose) extract representations via point cloud architectures or vision backbones like DINOv2. However, these methods implicitly store category-level structural patterns entirely within static network weights, lacking any dynamic mechanism during inference to accumulate, retrieve, and reuse category-level structural experience.

Both families of approaches share the same fundamental bottleneck: they lack a flexible, dynamic mechanism to accumulate and leverage category-level geometric memory. Given that intra-category instances share recurring local part topologies, integrating parametric perception with a non-parametric dynamic memory repository provides a principled solution. Core idea: build an external category-specific geometric memory buffer that stores pooled local keypoint features, continuously updates them via similarity-based merging with stochastic perturbation, and enriches current features through attention-based retrieval and adaptive gated fusion for robust pose prediction.

Method¶

Overall Architecture¶

The MemPose pipeline comprises three main stages: partial feature extraction, memory buffer interaction, and decoupled pose estimation. Given cropped RGB-D inputs, frozen DINOv2 and PointNet++ backbones extract dense semantic and geometric features, which are cross-attended with learnable category embeddings to detect $M$ representative local 3D keypoints and their local feature representations. Next, pooled local features are projected into memory entries and pushed into category-specific memory buffers. Once the buffer capacity is reached during training, a similarity-based merging mechanism with stochastic perturbation dynamically refines stored representations before retrieval. Finally, global keypoint features query the memory buffer via scaled dot-product attention, and the retrieved structural features are merged with the current features through an adaptive gate before regressing NOCS coordinates and 9-DoF pose parameters.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB-D Input & Backbones<br/>DINOv2 + PointNet++"] --> B["Adaptive Keypoint Detection<br/>Local Geometric Representation"]
    B --> C["Local Geometric Memory Construction<br/>AvgPool & Linear Projection"]
    C --> D["Similarity-based Update Mechanism<br/>Gumbel-perturbed Cosine Merging"]
    D --> E["Attention Retrieval & Gated Fusion<br/>Global Query & Dynamic Merging"]
    E --> F["Decoupled Pose & Size Regressor<br/>NOCS Prediction & 9-DoF Regression"]

Key Designs¶

1. Local Geometric Memory Construction: preventing temporal recency bias via pooled local features

Conventional memory-augmented frameworks often store final network output embeddings into an external repository. However, in fine-grained 3D geometry tasks, final output features are strongly coupled with the current training iteration's network parameters. Consequently, attention retrieval tends to favor recently inserted entries simply because their feature distributions align closely with the current network state, rather than because they share true geometric similarity. To address this temporal recency bias, MemPose maintains category-specific buffers $\mathcal{M}^{(c)} \in \mathbb{R}^{L \times C}$ (length $L=96$). It performs average pooling over the $M$ local keypoint features $\mathcal{F}_{local}$ followed by MLP projection and normalization: $$\mathcal{F}_{mem} = \text{Norm}(\text{MLP}(\text{AvgPool}(\mathcal{F}_{local})))$$ Pooling local features isolates intrinsic, invariant structural part topologies rather than global instance styles. Furthermore, a warm-up strategy keeps retrieval and update mechanisms dormant until the buffer reaches its capacity $L$, preventing unstable, noisy features from early training epochs from corrupting the memory buffer.

2. Similarity-based Update Mechanism: mitigating memory bloat and preventing high-frequency entry starvation

Unconstrained accumulation of memory entries causes substantial GPU memory growth and computational drag during attention retrieval. MemPose executes a pre-retrieval memory update to guarantee that the buffer retains the most up-to-date geometric patterns. When updating, greedy selection of the entry with the highest cosine similarity with incoming feature $\mathcal{F}_{mem}$ risks creating a winner-take-all effect, where a few prototypical entries are updated repeatedly while rare but critical edge-case structures are neglected. To encourage balanced exploration, MemPose injects Gumbel noise $g = -\log(-\log(u)), u \sim \mathcal{U}(0, 1)$ into the similarity calculation: $$\cos(\mathcal{F}_{mem}, \mathcal{M}^{(c)}_i) \leftarrow \cos(\mathcal{F}_{mem}, \mathcal{M}^{(c)}_i) + \lambda g$$ The target entry $i^*$ with the highest perturbed similarity is then updated via a moving average: $$\mathcal{M}^{(c)}_{i^*} \leftarrow \frac{1}{2}\left(\mathcal{M}^{(c)}_{i^*} + \mathcal{F}_{mem}\right)$$ This stochastic perturbation preserves the compact clustering benefits of cosine merging while continuously maintaining structural diversity across all buffer entries.

3. Attention Retrieval and Adaptive Gated Fusion: dynamically injecting category-level geometric context

During retrieval, the goal is to extract category-level structural context that matches the query instance's pose and geometry without erasing instance-specific fine details. MemPose enriches keypoints via graph-based geometric aggregation to form global representations $\mathcal{F}_{global} \in \mathbb{R}^{M \times C}$, which serve as queries $q = \mathcal{F}_{global} W_q$. The stacked category buffer entries form keys $k = \mathcal{F}'_{mem} W_k$ and values $v = \mathcal{F}'_{mem} W_v$. Scaled dot-product attention reads out the retrieved memory features: $$\mathcal{F}_R = \text{Softmax}\left(\frac{1}{\sqrt{D_k}} q k^T\right) v$$ To integrate $\mathcal{F}_R$ without corrupting fine-grained observations, an adaptive gating weight vector $w_a = \sigma(\mathcal{F}_R W_R + \mathcal{F}_{global} W_f)$ balances the contribution of memory and current perception: $$\mathcal{F}_{aug} = w_a \odot \mathcal{F}_R + (1 - w_a) \odot \mathcal{F}_{global}$$ When high-confidence analogous memories exist, $w_a$ increases to resolve severe occlusions or ambiguous symmetries; for rare or extreme geometries, the gate preserves current perception features, providing adaptive flexibility.

Loss & Training¶

The network is optimized end-to-end. The memory buffer is registered via PyTorch's register_buffer, remaining non-trainable by gradient backpropagation and updating purely through online merging. The total training loss $\mathcal{L}_{all}$ integrates five terms: $$\mathcal{L}_{all} = \alpha_1 \mathcal{L}_{pose} + \alpha_2 \mathcal{L}_{div} + \alpha_3 \mathcal{L}_{ocd} + \alpha_4 \mathcal{L}_{rec} + \alpha_5 \mathcal{L}_{nocs}$$ Here, $\mathcal{L}_{pose} = \|R_{gt} - R\| + \|t_{gt} - t\| + \|s_{gt} - s\|$ supervises 9-DoF parameters; $\mathcal{L}_{div}$ enforces spatial dispersion among keypoints beyond distance threshold $th_1 = 0.01$; $\mathcal{L}_{ocd}$ is an object-aware Chamfer distance constraining keypoints to lie on the outlier-filtered object surface; $\mathcal{L}_{rec}$ evaluates 3D shape reconstruction Chamfer loss; and $\mathcal{L}_{nocs}$ applies Smooth $L_1$ loss between predicted and ground-truth projected NOCS coordinates. Balancing weights are set to $\alpha_1=0.3, \alpha_2=10.0, \alpha_3=2.0, \alpha_4=15.0, \alpha_5=2.0$. Optimization uses Adam on a single NVIDIA L40 GPU for 120k iterations with a triangular2 cyclical learning rate schedule between $2\times 10^{-5}$ and $5\times 10^{-4}$.

Key Experimental Results¶

Main Results¶

MemPose is extensively evaluated on REAL275 and Housecat6D against leading prior-based and prior-free baselines:

Dataset	Method	Shape Prior	IoU50	IoU75	5°2cm	5°5cm	10°2cm	10°5cm
REAL275	DPDN (ECCV'22)	✓	-	76.0	46.0	50.7	70.4	-
	GCE-Pose (CVPR'25)	✓	-	79.8	57.0	65.1	75.6	-
	AG-Pose (CVPR'24)	✗	-	80.1	57.0	64.6	75.1	-
	KeyPose (AAAI'25)	✗	-	80.8	57.7	66.0	78.8	-
	SpherePose (ICLR'25)	✗	-	79.0	58.2	67.4	76.2	-
	MemPose (Ours)	✗	-	81.0	59.9	67.7	79.0	-
Housecat6D	GCE-Pose (CVPR'25)	✓	76.1	55.6	22.2	-	49.3	53.5
	FS-Net (CVPR'21)	✗	48.0	14.8	3.3	-	17.1	21.6
	VI-Net (ICCV'23)	✗	56.4	20.4	8.4	-	20.5	29.1
	SecondPose (CVPR'24)	✗	66.1	24.9	11.0	-	25.3	35.7
	AG-Pose (CVPR'24)	✗	76.9	53.0	21.3	-	51.3	54.3
	SpherePose (ICLR'25)	✗	72.2	-	19.3	-	40.9	55.3
	MemPose (Ours)	✗	81.5	56.4	23.1	-	52.6	57.3

On the Wild6D benchmark evaluated across 486 unseen video sequences with 162 objects, MemPose achieves 46.2% on IoU75 (surpassing previous SOTA MH6D at 41.9% by 4.3%) and improves 5°2cm from 27.0% to 29.7%.

Ablation Study¶

Ablations on REAL275 evaluate memory entry representations, update dynamics, fusion operators, and buffer length:

Study Aspect	Setting	IoU75 (%)	5°2cm (%)	5°5cm (%)	Note
Stored Feature Type	Global Feature	80.5	58.0	66.5	Sensitive to overall scene context
	Fused Feature	80.4	58.1	66.8	Degraded generalization
	Local Feature (Default)	81.0	59.9	67.7	Encodes invariant part geometry; best
Construction & Update	Random Initialization	80.4	58.0	65.0	Early noise corrupts buffer stability
	w/o Warm-up	80.8	59.0	66.0	Premature updates degrade representations
	Standard FIFO	80.7	59.1	66.0	Overwrites representative early prototypes
	Post-retrieval Update	80.7	59.0	65.2	Retrieval misses current observation info
	Top-k Sampling Merge	81.0	59.2	65.3	Balanced update but coarser matching
	Merge + Gumbel Noise (Default)	81.0	59.9	67.7	Balances prototype clustering and diversity
Memory Fusion	Direct Addition (Add)	80.4	58.5	65.3	Modality interference
	Concatenation (Concat)	80.5	58.0	65.0	Lacks adaptive feature selection
	Gated Fusion (Default)	81.0	59.9	67.7	Dynamically balances memory and observation
Buffer Length	$L=0$ (Baseline AG-Pose)	79.5	57.0	64.6	Lacks historical geometric experience
	$L=16$	79.6	57.2	64.9	Limited capacity to capture variation
	$L=48$	80.0	58.5	66.5	Steady gains
	$L=96$ (Default)	81.0	59.9	67.7	Optimal accuracy-efficiency trade-off
	$L=256$	80.8	59.1	67.9	Diminishing returns with higher memory cost

Key Findings¶

Local geometric pooling is essential: Switching from final output features to pooled local keypoint features boosts 5°2cm from 58.1% to 59.9%, confirming that local geometric representations prevent retrieval from succumbing to temporal recency bias.
Stochastic perturbation maintains memory expressiveness: Injecting Gumbel noise into cosine similarity merges increases 5°2cm by 0.7% over greedy merging, demonstrating the value of mitigating entry starvation.
Negligible computational overhead: MemPose introduces only 2M parameters over AG-Pose (225M vs. 223M, due to lightweight projection/gating heads; memory buffer is non-parametric), while maintaining virtually identical training speed (47.8 vs. 47.5 min/epoch) and inference throughput (33 vs. 35 FPS). Adding matching MLP parameters to AG-Pose yields only 57.8% on 5°2cm, confirming that improvements stem from the memory mechanism.
Online inference adaptation capability: Updating the memory buffer during test-time inference pushes 5°2cm on REAL275 from 59.9% to 63.6% (paper Tab. 8), indicating powerful test-time adaptation capability.

Highlights & Insights¶

Decoupling categorical priors into non-parametric dynamic structures: Eliminates the fragility of CAD deformation and the rigidity of static network weights via an explicit, lightweight experience buffer.
Stochastic cosine merging for memory management: Simultaneously compresses redundant prototypes and prevents entry starvation with Gumbel noise injection.
High transferability: The plug-and-play memory module operates independently of specific pose regression heads and can be readily applied to 3D object detection, few-shot shape completion, or robotic manipulation.

Limitations & Future Work¶

Reliance on predefined category indices: Memory buffers are partitioned by discrete category IDs; handling open-vocabulary or open-set categories without predefined classes remains unaddressed.
Absence of explicit temporal decay: Low-quality or degraded point cloud representations can linger in the buffer through moving-average merging.
Future Directions: Exploring semantic-conditioned continuous memory buffers integrated with open-vocabulary vision-language models, and validating online memory evolution on physical robotic manipulators during continuous grasping streams.

vs SPD / SGPA / DPDN: Prior-based methods rely on rigid CAD prototypes and complex deformation networks; MemPose constructs and refines compact geometric memories dynamically without offline CAD preprocessing.
vs AG-Pose / SpherePose: Parametric prior-free methods memorize geometry implicitly in static weights; MemPose enables dynamic cross-instance memory retrieval with negligible parameter overhead, achieving clear SOTA margins on REAL275 and Housecat6D.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Pioneers dynamic non-parametric geometric memory for category-level 9-DoF pose estimation with a clear, principled design]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 4 diverse benchmarks, with rigorous ablations on buffer mechanisms, complexity analyses, and case study visualizations]
Writing Quality: ⭐⭐⭐⭐⭐ [Cohesive narrative, clear motivation, and detailed mathematical explanations]
Value: ⭐⭐⭐⭐⭐ [Provides a practical and effective memory-augmented blueprint for robust 3D perception and embodied robotics]