Skip to content

title: >- [Paper Note] StructPolicy: Structure-Guided Imitation Learning Robust to Visual Domain Shifts description: >- [ECCV 2026][robotics][Imitation Learning] Introduces StructPolicy, a plug-and-play imitation learning framework leveraging domain-invariant Structure Maps composed of geometric primitives and affordance meshes, significantly boosting robot manipulation robustness across lighting, background, viewpoint, and noise shifts. tags: - ECCV 2026 - robotics - Imitation Learning - Robot Manipulation - Domain Invariance date: 2026-09-19 content_hash: 285a3be84a8934d8

StructPolicy: Structure-Guided Imitation Learning Robust to Visual Domain Shifts

Conference: ECCV 2026
Paper: ECCV Official Link
Homepage: StructPolicy Homepage
Area: Robotics & Embodied AI
Keywords: Imitation Learning, Robot Manipulation, Visual Domain Shifts, Structure Map, Affordance

TL;DR

StructPolicy introduces a domain-invariant Structure Map composed of geometric primitives under physical constraints, utilizing hierarchical structural encoding and affordance-guided pose generation to enable robust and accurate robot manipulation under severe visual domain shifts including lighting, background color, camera viewpoint, and sensor noise.

Background & Motivation

Vision-based Imitation Learning (IL) has emerged as a cornerstone paradigm in robotic manipulation, offering a straightforward approach to map raw perceptual observations into continuous motor actions without requiring handcrafted reward functions. Through behavior cloning and modern generative policies, IL agents have demonstrated remarkable dexterity in complex, long-horizon manipulation benchmarks. However, conventional IL policies remain excessively vulnerable to shifts in visual distribution between training demonstrations and deployment environments. Substantial alterations in ambient lighting, shifts in background patterns, camera viewpoint misalignments, or visual rendering noise frequently induce catastrophic policy collapse, rendering even basic manipulation tasks, such as drawer pulling or object pushing, completely unexecutable.

To alleviate visual brittleness, prior studies explored object-centric representations (OCRs) designed to decompose scenes into distinct bounding boxes, keypoints, or self-supervised entity embeddings to filter out extraneous background clutter. Nonetheless, standard OCRs treat target items as monolithic instances or rely on visual appearance embeddings that remain sensitive to lighting fluctuations and perspective transformations. Crucially, they neglect the internal geometric topology, kinematic articulations, and functional contact surfaces of objects. Drawing upon Gibson’s ecological theory of perception, physical affordance is fundamentally grounded in invariant structural geometry: manipulation feasibility is governed by how components are spatially composed, connected, and constrained rather than their surface color or texture.

The central insight of this work is to bypass the fragile dependency on raw, mutable RGB pixel features by explicitly constructing a domain-invariant geometric intermediary. Core idea: construct an automated Structure Map that abstracts task-relevant objects into physically constrained geometric primitives and actionable affordance meshes, providing downstream imitation policies with domain-invariant structural and pose features to guarantee robust manipulation under severe visual domain shifts.

Method

Overall Architecture

StructPolicy operates as an agile, plug-and-play module designed to enhance standard imitation learning policies (such as Diffusion Policy, 3D Diffusion Policy, Lift3D, and MDT-V) without modifying their foundational action loss objectives. The pipeline comprises two core modules: StructCon, an automated module for constructing the Structure Map, and StructEncoder, a dual-stream hierarchical network dedicated to structural and affordance feature extraction.

During an execution step, the environmental RGB observation and task language instruction are ingested by StructCon. StructCon utilizes a Vision-Language Model (VLM) paired with dense retrieval over the ConceptFactory database to retrieve relevant object categories and their associated geometric primitive templates. A lightweight parameter estimator predicts 3D geometric dimensions and spatial poses to assemble the Structure Map. Subsequently, StructEncoder processes this representation via parallel affordance and structure encoding streams, producing compact map features that are concatenated with standard visual policy embeddings to drive the final action prediction head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inputs: RGB Observation + Task Description + Proprioception"] --> B["StructCon: RAG Retrieval & Primitive Assembly<br/>VLM Category Summary + BGE-M3 Dense Retrieval from ConceptFactory"]
    B --> C["StructCon: Parameter Estimation & Instantiation<br/>Physical Constraint Composition + 3D Scale & Pose Regression"]
    C --> D["Structure Map<br/>Primitive Topology Points S_o + Affordance Triangular Meshes A_o"]
    D --> E1["Affordance Encoding Stream<br/>Surface Normal Inversion Poses + MLP Feature Extraction f_p"]
    D --> E2["Structure Encoding Stream<br/>Primitive Attention Pooling + Point Transformer Topology f_s"]
    E1 --> F["Feature Fusion & Policy Head Execution<br/>MLP Map Fusion f_m + Visual Feature f_v Concatenation to Actions"]
    E2 --> F

Key Designs

1. Hierarchical Structure Map Formulation: Primitive Sampling & Affordance Grounding

The Structure Map is organized through a three-level structural hierarchy: atomic structure primitives, individual object topological structures, and the aggregated multi-object scene representation. Polyhedral geometric primitives (such as cuboids) are sampled via uniform surface triangulation taking mesh vertices, while curved primitives (such as cylinders) are discretized across planar arc cross-sections with fixed radial resolution. The complete object structure \(\mathcal{S}_{\mathbf{o}}\) is formalized as the spatial union of labeled primitive point sets: $\(\mathcal{S}_{\mathbf{o}} = \bigcup_{m=1}^{N_{\mathbf{o}}} \{(\mathbf{p}_i, \ell_i)\}_{i=1}^{N_m}\)$ where \(\mathbf{p}_i \in \mathbb{R}^3\) denotes the 3D coordinates of a sampled point and \(\ell_i\) signifies the primitive categorical type, subjected to rigid physical rules governing legal relational configurations. Simultaneously, functional actionable surfaces are captured as sets of triangular surface patches \(\mathcal{A}_{\mathbf{o}} = \bigcup_{i=1}^{K_{\mathbf{o}}} \mathcal{F}_i\), where each patch \(\mathcal{F}_i = \{\mathbf{v}_{i,1}, \mathbf{v}_{i,2}, \mathbf{v}_{i,3}\}\) denotes an explicit manipulation interface (such as a handle grip region). Aggregating over all \(K\) task-relevant entities yields the unified scene Structure Map \(\mathcal{M} = \{(\mathcal{S}_{\mathbf{o}_k}, \mathcal{A}_{\mathbf{o}_k}, k)\}_{k=1}^K\).

2. StructCon Automated Construction: RAG Grounding & Constrained Parameter Estimation

To ensure reliable structural instantiation despite visual corruption in the raw image, StructCon adopts a retrieval-augmented generation (RAG) framework. First, a VLM interprets the task instruction and visual scene to output a concise textual summary of manipulated entities. This textual description is projected into an embedding space via BGE-M3 to perform dense vector similarity search against ConceptFactory—a structured knowledge base spanning 39 object categories and 4,380 geometric compositions. Once the canonical template and physical composition rules are retrieved, the backbone policy visual features are fed into a 3-layer MLP parameter estimator. This estimator regresses the continuous 3D bounding dimensions, spatial positions, and orientations required to instantiate the primitives in world space. Because the structural connectivity and physical regularities are strictly enforced by the retrieved templates, the constructed Structure Map retains high geometric fidelity even under extreme visual shifts.

3. StructEncoder Dual-Stream Architecture: Affordance Poses & Hierarchical Topology

To transform the geometric Structure Map into action-oriented representations, StructEncoder deploys two specialized processing streams. The affordance encoding stream translates actionable surface patches into coarse gripper poses \(\mathcal{P} = \{(x_i, y_i, z_i, d_i^x, d_i^y, d_i^z)\}\). For each affordance patch, its geometric centroid initializes the position \((x, y, z)\), while the inverted surface normal establishes the unit approach direction \((d^x, d^y, d^z)\). An MLP regresses fine position offsets \(\Delta(x, y, z)\) to yield task-optimized candidate grasp positions, which are encoded into an affordance feature vector \(\mathbf{f}_p\). Concurrently, the structure encoding stream groups points by primitive type, applies lightweight MLPs with attention pooling to extract intra-primitive geometric tokens, and routes them through a Transformer structure encoder to capture global inter-primitive spatial layouts and topological dependencies, outputting a global structure feature \(\mathbf{f}_s\). Finally, \(\mathbf{f}_p\) and \(\mathbf{f}_s\) are concatenated and projected by an MLP into a unified map feature \(\mathbf{f}_m\), which concatenates with standard visual feature \(\mathbf{f}_v\) before being fed into the downstream policy action head.

Loss & Training

StructPolicy is trained jointly with the baseline imitation learning architecture end-to-end, adhering directly to the supervision paradigm of the underlying IL model (such as the standard \(\ell_2\) denoising score matching loss in Diffusion Policy or standard action chunk regression). No auxiliary loss functions or multi-stage curriculum objectives are introduced. All MLP blocks utilize three linear layers with ReLU non-linearities, and the Transformer components follow the standard Point Transformer architecture.

Key Experimental Results

Main Results

The performance of StructPolicy was systematically evaluated across simulation benchmarks—CALVIN (long-horizon sequential manipulation) and MetaWorld (15 diverse robotic tasks across varied difficulty levels)—as well as real-world robotic environments.

Main results from Table 1 of the paper (CALVIN average completed instruction chain length out of 5; MetaWorld mean success rate %):

Benchmark & Backbone Policy Baseline + StructPolicy (Ours) Improvement (\(\Delta\)) Metric & Setup
CALVIN D \(\to\) D: TaKSIE 3.18 3.34 +0.16 Avg. Instruction Chain Length
CALVIN D \(\to\) D: HULC++ 3.30 3.45 +0.15 Avg. Instruction Chain Length
CALVIN D \(\to\) D: MDT 3.59 3.70 +0.11 Avg. Instruction Chain Length
CALVIN D \(\to\) D: MDT-V 3.72 3.86 +0.14 Avg. Instruction Chain Length
CALVIN D \(\to\) D: FLOWER 4.35 4.42 +0.07 Avg. Instruction Chain Length
CALVIN ABCD \(\to\) D: DeeR 4.13 4.22 +0.09 Avg. Instruction Chain Length
CALVIN ABCD \(\to\) D: MDT-V 4.51 4.60 +0.09 Avg. Instruction Chain Length
CALVIN ABCD \(\to\) D: FLOWER 4.67 4.70 +0.03 Avg. Instruction Chain Length
MetaWorld: Diffusion Policy (DP) 65.2% 77.5% +12.3% 15 Tasks Mean Success Rate %
MetaWorld: 3D Diffusion Policy (DP3) 66.7% 79.2% +12.5% 15 Tasks Mean Success Rate %
MetaWorld: Lift3D 83.9% 90.5% +6.6% 15 Tasks Mean Success Rate %

Real-world robot manipulation results (Tables 8 & 9): On a physical Flexiv Rizon 4 arm across 8 manipulation tasks, StructPolicy boosted the baseline Lift3D success rate from 67.5% to 83.3% (+15.8%) under standard clean conditions. Under severe visual perturbations, the robustness gap became dramatic: under variable lighting, success rose from 30.0% to 65.8% (+35.8%); under altered background colors, from 28.3% to 69.1% (+40.8%); and under camera perspective shifts, from 24.2% to 60.0% (+35.8%).

Ablation Study

Table 6 contrasts the Structure Map against alternative object-centric representations on MetaWorld (15-task average success rate %):

Policy Backbone Baseline + Mask + Keypoints + 6D Pose + 3D BBox + Structure Map (Ours)
Diffusion Policy (DP) 65.2% 69.6% (+4.4%) 71.1% (+5.9%) 67.8% (+2.6%) 70.2% (+5.0%) 77.5% (+12.3%)
3D Diffusion Policy (DP3) 66.7% 70.2% (+3.5%) 70.8% (+4.1%) 73.6% (+6.9%) 69.4% (+2.7%) 79.2% (+12.5%)

Ablation investigations into sampling paradigms and StructEncoder module variants (Tables 4 & 5): - Structured Sampling vs. Uniform Sampling: Paired with StructEncoder, structured geometric sampling reached 90.5% on MetaWorld compared to 80.5% using uniform surface point sampling (a 10.0% margin), validating that point sampling concentrated along geometric boundaries preserves vital topological constraints. - Encoder Branch Isolation: Utilizing affordance pose features alone (Ex1) achieved 74.6% success; using structure features alone (Ex2) reached 85.2%; combining both streams via MLP fusion (Ex8) attained the peak performance of 90.5%.

Key Findings

  • System Bottleneck Identification (Table 3): Supplying oracle ground-truth structure templates yielded only a minor success gain from 90.5% to 91.0% (+0.55%), demonstrating the high reliability of the VLM-RAG retrieval stage. Conversely, providing ground-truth primitive parameters led to a leap from 91.0% to 98.4% (+8.73%), highlighting 3D parameter regression as the principal performance bottleneck.
  • Geometric Invariance Under Shift (Table 2): Across four categories of visual corruption in CALVIN (lighting variations, chromatic background changes, viewpoint shifts, and ray tracing noise), the F-Score between constructed Structure Maps and ground-truth point clouds remained remarkably stable at 50.2%–53.7% (relative to 61.6% without perturbations), confirming robust shape recovery.
  • Negligible Latency Overhead (Table 7): Average per-step inference latency increased by only 2.99 ms (from 67.21 ms to 70.20 ms over Lift3D), readily satisfying the requirements of high-frequency 30 Hz real-time robot control.

Highlights & Insights

  • Invariance Through Structural Decoupling: Rather than hoping visual representation models passively memorize environmental nuisance variables, the method anchors action learning directly to domain-invariant physical geometry and topological connections.
  • Affordance as Geometric Normal Inversion: Repurposing inverted surface normals of actionable triangular facets into directional approach vectors provides downstream policies with grounded physical priors for grasp synthesis.
  • Lightweight Plug-and-Play Integration: By outsourcing object topology rules to an offline knowledge database (ConceptFactory) and utilizing lightweight MLPs for parameter estimation, the approach incurs minimal computational latency (<3 ms) while consistently augmenting diverse IL backbones.

Limitations & Future Work

  • Dependency on Geometric Primitive Priors: StructPolicy relies on predefined primitive categories stored within ConceptFactory. Novel objects with highly irregular geometries or deformable structures (such as clothing or fluids) fall outside the current template space.
  • Single-View Parameter Regression: Regressing spatial parameters from limited camera viewpoints can incur positional drift under severe physical occlusions.
  • Future Directions: Integrating online tactile feedback or multi-view implicit neural fields to dynamically correct estimated primitive parameters, alongside extending primitive definitions to continuous deformable continuum models.
  • vs. Lift3D / Perceiver-Actor (3D Imitation Learning): While 3D point-cloud and 2D-lifted foundation models improve spatial reasoning, raw sensory point clouds still degrade under heavy visual noise and viewpoint transformations. StructPolicy abstracts noisy observations into clean geometric primitives, providing superior domain invariance.
  • vs. VIOLA / SORNet (Object-Centric Manipulation): Conventional OCRs isolate entire objects as bounding boxes or single-point centroids, missing fine-grained articulated part structures and contact affordance planes. StructPolicy models intra-object topological kinematics and surface affordance explicitly.

Rating

  • Novelty: ⭐⭐⭐⭐☆ (Presents a refreshing perspective by grounding imitation learning in domain-invariant geometric primitives and affordance maps.)
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive validation across 49 simulation tasks, 8 real-world physical setups, and 4 systematic visual shift modalities with insightful error breakdowns.)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Well-structured paper with clear algorithmic explanations, rigorous mathematical definitions, and detailed ablation studies.)
  • Value: ⭐⭐⭐⭐☆ (Highly practical plug-and-play architecture with negligible computational overhead, offering strong practical utility for real-world robotic deployments.)