Grasp-Oriented Non-Prehensile Manipulation via Learning a Graspability Field¶

Conference: ECCV2026
Authors: Licheng Zhong, Gim Hee Lee
Paper: ECCV official page
PDF: Full paper
Project: https://zlicheng.com/gomp_page/
Area: Robotics & Embodied AI
Keywords: Non-prehensile manipulation, graspability field, grasp preparation, reinforcement learning, closed-loop control

TL;DR¶

GOMP reformulates preparatory pushing, sliding, and reorientation from reaching a specified object pose to reducing distance to a set of graspable configurations, then uses predicted distance to trigger grasping; it completes 31 of 50 end-to-end real-robot trials from challenging initial states, rather than achieving 97% overall success.

Background & Motivation¶

Grasping does not fail only because the gripper chooses poor contact points. An object's current placement can leave suitable contacts blocked by the table, or cause it to slide during approach. Non-prehensile manipulation first changes the object state through pushing, sliding, or reorientation, then attempts a grasp. CORN and DyWA already learn such contact-rich behavior, but typically formulate it as goal-conditioned control: specify a target object pose and reward the robot for reducing the discrepancy to that pose. This formulation suits repositioning, but does not directly answer whether the object is already sufficiently graspable.

An object may have several graspable configurations. Some are near its initial state; others are geometrically graspable yet difficult to reach stably under table-contact dynamics. Enforcing one target can pull the robot away from an already graspable region or encourage unnecessary corrections near the target, producing oscillation. Conversely, object configurations derived from synthesized grasps need not be stable resting poses. Passing them directly to a pose-tracking policy conflates grasp preparation with precise pose regulation.

The paper retains training-time geometry but changes its role. Rather than supplying a pose that the controller must reach, it constructs a set distance so the policy can learn from point clouds which states better support subsequent grasping. Core Idea: use a graspability field for both manipulation rewards and termination, allowing the robot to seek reachable graspable regions instead of tracking a manually specified single endpoint.

Method¶

Overall Architecture¶

GOMP takes an object point cloud and robot proprioceptive state and outputs a non-prehensile action, a graspability distance, and a terminal grasp pose. During training, Reference Set Construction supplies graspable configurations and Smooth Anchored Distance defines the graspability field. Field-Driven Policy Learning turns that field into rewards and distance supervision; Scale-Normalized Transition then connects ongoing manipulation to grasp execution.

The diagram distinguishes training information from the execution loop. Reference configurations construct training signals rather than serving as a target list queried repeatedly at deployment. Only the student policy is deployed; full geometry, the teacher, and reference configurations are not supplied as external inputs.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Full object geometry<br/>Training only"] --> B["Reference Set Construction"]
    B --> C["Smooth Anchored Distance"]
    C -->|Reward and distance supervision| D["Field-Driven Policy Learning"]
    P["Partial point cloud<br/>Robot state"] --> D
    D --> E["Scale-Normalized Transition"]
    E -->|Not sustained: manipulate and observe again| P
    E -->|Sustained criterion| F["Execute predicted grasp pose"]

Key Designs¶

1. Reference Set Construction: convert grasp candidates into graspable object configurations

An analytical graspable set cannot readily be specified for arbitrary objects. The paper therefore uses the existing grasp synthesis model SE(3)-DiffusionFields to produce multiple gripper-relative-to-object grasp candidates from object geometry. It fixes a vertically downward gripper orientation and inverts each relative grasp transform to determine how the object would need to be placed for that grasp. The object is then translated vertically onto the table, yielding several reference object configurations. These references are not trajectories for the gripper to track; they are training samples that approximate the set of graspable object states.

This conversion moves the center of the problem from the end effector to the object: how far is the current object arrangement from a graspable one, rather than how far is the gripper from a grasp action? Multiple references permit different effective configurations instead of excluding readily reachable regions at task-definition time. However, grasp synthesis provides evidence of geometric feasibility, not a guarantee that each reference is a stable resting configuration. The paper consequently also considers object-velocity constraints during training and evaluation.

2. Smooth Anchored Distance: allow multiple graspable regions without repeatedly switching direction

The configuration discrepancy is first defined geometrically. Fixed keypoints sampled from the object bounding box are transformed by the current and reference configurations, and their corresponding Euclidean distances are averaged. Thus the distance measures spatial disagreement between object geometries rather than directly combining translation and rotation parameters, and it does not use the gripper pose. Taking the minimum over references measures proximity to the nearest graspable arrangement, but switching nearest references introduces nondifferentiable boundaries that can destabilize learning.

The paper replaces the minimum with a soft-min aggregation and adds an anchor term relative to the previously selected reference. Combining these two steps gives the graspability distance:

\[ \tilde d(s_o^t)=-\tau\log\sum_{i=1}^{G}\exp\!\left(-\frac{\mathcal D(s_o^t,\hat s_o^i)}{\tau}\right)+\lambda\mathcal D(s_o^t,\hat s_o^{i^*}). \]

Here, \(\mathcal D\) is the mean keypoint distance described above, \(G\) is the number of references, \(\tau\) controls soft aggregation, \(\lambda\) sets the anchor strength, and \(i^*\) denotes the previously selected reference. Soft aggregation lets several nearby references contribute guidance, while anchoring adds temporal persistence to the direction of progress, discouraging repeated switching between two promising configurations. References still shape the reward landscape, but are not explicit targets in the policy observation. No target-pose input does not mean reference-free training geometry. Lower distance indicates greater graspability; the value is not a calibrated probability of grasp success.

3. Field-Driven Policy Learning: learn to improve states and estimate that improvement from partial observations

Rewarding only a successful final lift would leave many pushing actions without timely feedback. GOMP converts distance into a potential that increases as distance decreases, then forms a dense reward from the potential difference between successive states:

\[ \Phi(s_o)=k_1\beta^{k_2\tilde d(s_o)},\qquad r_t=\gamma\Phi(s_o^{t+1})-\Phi(s_o^t). \]

Here, \(\beta\in(0,1)\), \(k_1\) and \(k_2\) control the potential's scale and sharpness, and \(\gamma\) is the discount factor. Intuitively, the policy learns which contact actions bring the object closer to graspable regions without physically attempting a grasp at every training step. Strictly, the discount factor means that not every arbitrarily small distance reduction necessarily produces positive reward. The important function is to provide a signal of incremental progress.

The implementation uses CORN's object-centric encoder, freezing earlier layers and fine-tuning only the final attention layer. Robot-state-conditioned cross-attention aggregates object and robot tokens. The teacher observes full point clouds and learns manipulation through reinforcement learning. The student observes partial point clouds and uses a GRU to aggregate information from recent observations and actions, reducing single-frame ambiguity from occlusion and contact-induced motion. Online policy distillation transfers the teacher's behavior to the student. Prediction heads output the action, distance, and terminal grasp pose, with distance supervised by the training-time graspability field. Distillation here transfers the policy to partial observations; the paper's main contribution is the grasp-preparation objective and termination signal, not a new model-compression algorithm.

4. Scale-Normalized Transition: do not close the gripper after one apparently graspable frame

The same geometric error means different things for small and large objects. The paper normalizes predicted distance by the object bounding-box diagonal length \(L\), producing \(\bar d_t=\hat d_t/L\), and checks whether the maximum normalized distance over a short temporal window is below a threshold \(\epsilon\). Using the maximum requires the criterion to remain satisfied throughout the window instead of triggering on a single low prediction. At execution time, this can be understood as checking an already observed short window rather than requiring knowledge of the future.

Once the criterion is met, the policy executes its predicted grasp pose; otherwise, it continues non-prehensile manipulation and obtains another observation. This gives improvement and termination a shared distance concept, but does not eliminate engineering parameters: a threshold, temporal window, and safety force limit remain. The cached main text does not specify numerical transition-threshold or window-length settings, and does not fully explain supervision of the grasp-pose head. Consequently, no external planner should not be read as no manually specified settings whatsoever.

A Worked Example¶

Consider a box whose initial placement leaves unsuitable contact positions for a top-down gripper approach. This is an explanatory example, not an additional measured trajectory. During training, synthesized grasps first map to several box configurations; geometric distance indicates that a push-and-reorient action moves the current arrangement closer to one graspable region. Anchoring discourages switching to another reference as the robot approaches that region.

At deployment there is no list of reference arrangements. The student acts incrementally from partial point clouds, proprioception, and the history stored by its GRU, while predicting the current distance. Even a low prediction in one frame is insufficient: normalized distance must remain low throughout the short window before the system switches to the predicted grasp pose. Preparation therefore does not mean exactly reproducing a target box pose; it means satisfying the system's graspability criterion.

Loss & Training¶

Training uses IsaacSim and IsaacLab with a simulated Franka Emika Panda and parallel-jaw gripper. The teacher initially trains for 15k iterations, reported to take approximately 10 hours on one NVIDIA RTX A6000; object-velocity constraints are introduced after 5k iterations. Before distillation, the teacher trains for another 10k iterations with an action-scale curriculum that reduces the effective action range. The total training cost therefore cannot be summarized as only 10 hours.

Object mass, scale, friction, and initial poses are randomized throughout training. The student distance head regresses graspability distance, while online policy distillation learns behavior under partial observations. The cached main text does not completely specify loss weights or all control hyperparameters, so no particular optimizer or grasp-head loss formula is inferred here.

Key Experimental Results¶

Main Results¶

Simulation uses task-suitable objects manually selected from DexGraspNet, GraspFactory, and YCB. Success in the following table means successful grasp preparation, not a successful physical lift. Geometric tolerances are position error below 0.05 m and orientation error below 15°; GOMP additionally requires linear velocity below 0.01 m/s and angular velocity below 0.1 rad/s. The table preserves the stability-protocol differences in Table 1 and reports average time as given there.

Method	Velocity stability check	Seen success ↑	Seen average time ↓	Unseen success ↑	Unseen average time ↓
Retrained target-pose baseline	No	36.1%	5.3 s	28.8%	6.1 s
Nearest-reference Oracle	No	52.0%	8.1 s	50.2%	8.8 s
Multi-reference Oracle	No	42.2%	11.3 s	39.4%	9.9 s
GOMP Teacher	Yes	84.0%	4.6 s	77.9%	4.8 s
GOMP Student	Yes	75.5%	6.1 s	68.1%	6.2 s

The nearest-reference Oracle supplies the retrained baseline with the currently closest reference at every step. The multi-reference Oracle evaluates candidate references separately and reports the best outcome. Default evaluation uses 10 references. GOMP Teacher exceeds the nearest-reference Oracle's unseen success rate by 27.7 percentage points, but success definitions, target conditioning, and stability requirements are not an identical protocol. This is not a matched-condition gain on the original CORN/DyWA public benchmark. The paper explicitly notes that passing potentially dynamically unstable grasp-synthesis references to a pose tracker changes its original task conditions.

Real-world evaluation uses a Franka Research 3 and fused point clouds from three Intel RealSense D435IF cameras, without ground-truth poses, markers, or external tracking. Each of 10 objects is tested 5 times from challenging initial configurations; direct grasping with a fixed approach succeeds in 8% of trials. The three success rates in Table 2 have different meanings:

Real-robot metric	Successes / denominator	Success rate ↑	Denominator meaning
Successful non-prehensile reconfiguration	32/50	64%	All attempts
Conditional grasp success after successful preparation	31/32	97%	Successful reconfigurations only; percentage rounded in the paper
End-to-end success	31/50	62%	All attempts

Ablation Study¶

The following selection comes from simulation Table 3. Higher success rates and lower times are better. The first three rows test reference-set size; the last two test soft aggregation and anchoring respectively. The insufficiently specified final-distance column is not used to rank different configurations.

Config	Seen success ↑	Unseen success ↑	Unseen average time ↓
1 reference	82.2%	72.8%	4.8 s
10 references, full model	84.0%	77.9%	4.8 s
20 references	76.0%	68.1%	6.0 s
Use hard minimum	81.1%	74.3%	5.2 s
Remove anchor term	78.1%	74.1%	5.1 s

Key Findings¶

More references are not always better. Using 10 rather than 1 improves unseen success by 5.1 percentage points, but increasing to 20 reduces it to 68.1%. The authors attribute the degradation with excessive references to competing directions; the gain is not monotonic.
Both soft aggregation and anchoring contribute. Relative to the full model, hard minimum and removal of anchoring reduce unseen success by 3.6 and 3.8 percentage points respectively; removing anchoring lowers seen success by 5.9 percentage points.
Reconfiguration is the main real-world bottleneck: only 32/50 attempts reach successful preparation, after which 31/32 grasps succeed. The high conditional success rate supports the transition signal, but does not establish reliable handling of every initial state.

Highlights & Insights¶

The objective changes from reproducing a pose to improving the object state, with one scalar reused for reward and transition. The key is not another grasp-planning module, but linking termination to the state the task actually needs.
Full geometry supplies training supervision while partial point clouds drive execution. This uses privileged simulation information for dense signals without requiring a ground-truth pose tracker on the real robot.
Reporting reconfiguration, conditional grasp, and end-to-end success separately locates the bottleneck in preparation. Quoting only 97% would obscure overall performance.

Limitations & Future Work¶

The authors report consistent failure on the soft, slippery Snack object; rigid objects can also fail through sliding, timeout, or reaching the safety force limit. A distance defined using rigid reference geometry cannot ensure reachability under complicated contact dynamics.
Objects and challenging initial states are manually selected, and real-world evaluation contains 50 trials. The 8% direct-grasp result uses a fixed approach, not the upper bound of a strong general-purpose grasping system, and cannot be extrapolated directly to arbitrary clutter.
The softly aggregated scalar is a geometric proxy, not complete physical feasibility or a success probability. Figure 6 separates predicted distances for graspable and non-graspable configurations, but the main text reports no probability-calibration metric.
Reproduction still requires checking the threshold, temporal window, anchor-reference update rule, and grasp-pose-head training details. The main text provides limited detail on these points, and unspecified parameters are not treated here as established implementation facts.

vs CORN / DyWA: These methods provide goal-conditioned non-prehensile control foundations. GOMP reuses perception and a teacher–student approach but changes the learning objective to distance from a graspable set. Its comparison primarily exposes objective mismatch, not failure of those methods on their original pose-reaching tasks.
vs SE(3)-DiffusionFields: Grasp synthesis supplies training-time reference configurations rather than an external module that replans continuously at deployment. GOMP learns to make the object graspable first, rather than merely generating gripper candidates.
vs grasp-affordance-guided manipulation: Many methods improve reachability for a selected grasp candidate; this paper first optimizes a property of the object configuration. The useful distinction is between making one grasp executable and bringing an object into a graspable state, while recognizing the limits of the proxy metric.

Rating¶

Novelty: 4/5. The set-distance formulation clearly unifies reward and transition, while the network and potential construction build on existing tools.
Experimental Thoroughness: 3/5. Simulation Oracles, component ablations, and stage-wise real-robot statistics are provided, but the real-world sample is small and comparison protocols require care.
Writing Quality: 3/5. The central reasoning is accessible, but transition parameters and some training details remain underspecified, and summary claims simplify actual engineering conditions.
Value: 4/5. The method provides an interpretable control objective for grasp preparation and informative real-system results, provided the 62% end-to-end success rate is not overlooked.