Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents

Conference: ECCV2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: 6D pose rearrangement, closed-loop agents, multi-view reasoning, coordinate axis visualization, single-axis rotation

TL;DR

A single pretrained vision-language model (VLM) alternates between evaluating a scene and proposing incremental object pose updates, corrects errors through re-rendered feedback, and combines three geometric prompting techniques to improve overall Open6DOR V2 6-DoF goal-pose prediction success from SoFar's 26.3% to 46.4% without fine-tuning.

Background & Motivation

When a robot receives an instruction to put a marker inside a mug with its cap facing upward, recognizing the marker and mug is only the beginning. It must determine the required translation, the appropriate rotation, and whether the final pose satisfies both containment and orientation constraints. A single image can project different depths into similar image regions, while occlusion makes contact relationships difficult to assess. A VLM that describes scenes fluently therefore does not necessarily predict accurate goal 6D poses directly. Methods such as SoFar organize scenes into structured information that helps express positional relationships but can still miss global visual cues and target orientation.

The problem here is not estimating an object's current pose, but inferring a goal state that has not yet appeared from language. Predicting a complete rotation in one shot requires resolving observation ambiguity, coordinate directions, and 3D rotation composition simultaneously, without an opportunity to inspect the result. The authors instead use a renderable 3D scene: update the object first, then return the resulting visual evidence to the model for inspection. This loop operates primarily during goal-pose inference; it does not mean that every physical robot step invokes a VLM. The downstream arm experiments subsequently pass the predicted goal to grasping and motion planning to test whether it supports manipulation.

This approach must still address the case where a model notices an error but cannot specify an accurate correction. Multiple views expose depth and occlusion errors, explicit axes establish direction and scale, and single-axis rotation reduces the difficulty of each action. Core Idea: replace one-shot goal-pose prediction with incremental correction supported by visual feedback and context memory, allowing the model to approach the language-specified state through an interpretable geometric interface.

Method

Overall Architecture

Inputs are an RGB-D image with camera intrinsics or a composition of object meshes, together with a text instruction describing the desired state. The output is the target object's final position and orientation; related objects and other scene objects remain stationary during inference. For RGB-D input, preprocessing lifts the observation into a point cloud, obtains object masks using off-the-shelf segmentation models, and removes outlier points from each object. This provides independently transformable object geometry rather than learning an additional goal-pose prediction network.

Before the loop, the system places cameras around the center of the full scene's axis-aligned bounding box (AABB), rendering multiple views and object masks. The VLM first chooses a view where objects are clear and distinguishable, then identifies the target and instruction-related objects using boxes and labels overlaid on that view. For instance, the mug is a related object in an insertion instruction, the marker is the target to update, and other tabletop objects do not move with it. Subsequent iterations use closed-loop evaluation and memory, multi-view supporting view, object-centered coordinate axes, and single-axis rotation update before re-rendering. The evaluator and proposer are two invocation roles of the same VLM, not two separately trained models.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["3D scene and instruction<br/>Identify target and related objects"] --> Eval["Closed-loop evaluation<br/>and memory"]
    Eval -->|Instruction not yet satisfied| Views["Multi-view supporting view"]
    Views --> Axes["Object-centered<br/>coordinate axes"]
    Axes --> Rotate["Single-axis rotation update"]
    Rotate -->|Apply increment and re-render| Eval
    Eval -->|Satisfied or iteration limit reached| Output["Final goal pose<br/>Optional grasping and motion planning"]

Key Designs

1. Closed-loop evaluation and memory: turn errors into evidence for the next step

The evaluator inspects current renderings for target and related-object visibility, satisfaction of the instructed spatial relationships, and physically implausible interpenetration. It returns a judgment of instruction satisfaction, the index of a supporting view, and a rationale. If the scene already satisfies the instruction, inference ends; otherwise, the proposer uses visual evidence to output translation, a dominant rotation axis, a rotation angle, and its rationale. The renderer turns this proposal into a geometric change, so the next model invocation observes transformed images rather than simply trusting its previous description. It can discover that a supposedly sufficient displacement still leaves an object outside the mug, or that an intended correction tilted it in the opposite direction.

Context memory accumulates earlier evaluations, pose updates, and rationales, and is supplied to both roles in later iterations. This connects the current observation to previous attempts and helps reduce oscillation between opposing corrections. Memory is neither training-data replay nor a weight update; it is an expanding inference context. The loop ends when the evaluator accepts the scene or the maximum iteration count is reached; reaching the limit ends computation but does not automatically imply success. The main paper places iteration-budget analysis in the supplementary material, so this note does not invent a specific limit or per-iteration invocation cost.

2. Multi-view supporting view: inspect across views, then ground one correction

Each iteration recomputes the joint AABB of the target and related objects and places equally spaced cameras on a circular trajectory. Framing is centered on this object group and keeps its members in view, avoiding an outdated observation position after the target moves. The evaluator checks spatial relationships across views, especially depth differences and occlusions that a single projection cannot resolve. For example, an object that appears to have entered a bookshelf may still be in front of it or blocked by a side panel when viewed from another angle. These are views rendered from the available 3D representation, not newly acquired physical observations supplementing the initial RGB-D image.

When correction is needed, the evaluator selects the view that best supports its judgment, and the proposer predicts the next action around that image. Evaluation thus benefits from multi-view coverage, while action generation receives a concrete visual reference instead of coordinating every camera projection simultaneously. The supporting view can change between iterations, preventing occlusion of a fixed view from removing the loop's visual basis. However, rendering a new viewpoint cannot recover surfaces absent from the original point cloud; sensitivity to missing geometry still depends on the input representation. This last observation is a reader interpretation of the input mechanism, not an additional experiment reported by the authors.

3. Object-centered coordinate axes: retain world directions and express translation through visible scale

The coordinate origin is placed at the center of the target object's current AABB, while axis directions remain aligned with world coordinates. Object-centered means that the origin follows the object, not that the axes rotate with its orientation; confusing these conventions changes the action's meaning. Axes extend outward from the centers of the corresponding bounding-box faces, providing visible direction and length references. The proposer always receives this overlay; for instructions with ambiguous front/back or left/right relationships, the evaluator can also receive axis-augmented multi-view images. This resolves cases where the same world direction appears leftward in a front view but rightward in a rear view.

The physical axis length is determined from the target object's AABB dimensions in canonical coordinates and stays fixed across iterations. The VLM is instructed to treat the visible axis length as one unit and output normalized translation rather than guessing distances directly in meters. If the physical axis length is \(L\) and the model predicts \(\hat{\mathbf{t}}_i\), the translation applied in world coordinates is:

\[ \mathbf{t}_i=L\hat{\mathbf{t}}_i. \]

This relation is stated in the prose on page 7 and shows that visualization supplies both directional grounding and a scale reference, not merely decoration. The extracted piecewise conditions in Eq. (5) are corrupted, so the axis-length selection rule cannot be reliably recovered and is not reconstructed here.

4. Single-axis rotation update: decompose complete orientation into observable decisions

At each iteration, the proposer selects one dominant world-coordinate axis and an angle about that axis, following the right-hand rule. Translation may still contain components along all three directions; the restriction concerns rotation parameterization, not one-dimensional movement of the entire object. Different iterations can select different axes, so their composition can express complex orientations without predicting three coupled Euler angles in one shot. Writing the incremental rotation as \(\mathbf{R}_i\), target orientation is updated by left multiplication under the world-axis convention:

\[ \mathbf{R}_{i+1}^{\mathrm{tgt}}=\mathbf{R}_i\mathbf{R}_i^{\mathrm{tgt}},\qquad \mathbf{R}_i=R_{\omega_i}(\theta_i),\quad \omega_i\in\{x,y,z\}. \]

The geometric update rotates around the center of the target's current AABB and incorporates the predicted translation, rather than swinging the object around the world origin. After rotation, the next bounding-box center and rendered observations are recomputed from the current geometry. The complete translation expansion in cached Eq. (6) has missing characters; the equation above retains only the legible orientation update rather than presenting a guessed expansion as the original. Compared with predicting full SO(3) at once, a single-axis action makes its direction and progress toward the goal easier to inspect visually.

A Worked Example

Consider the instruction in Figure 1 to put a marker inside a mug with its cap facing upward; this walkthrough explains the mechanism rather than reporting an additional experimental trajectory. Initial selection identifies the marker as the target and the mug as related, and the loop changes only the marker's geometric pose. The evaluator checks multiple views for insertion into the mug opening and upward cap orientation; image-space overlap alone does not establish successful insertion. If a side view reveals misalignment between the opening and marker, it becomes the supporting view, and the proposer uses visible axes to estimate translation and the necessary single-axis rotation. After applying the update and re-rendering, the next iteration can detect whether the marker remains outside, points the wrong way, or interpenetrates another object. Earlier actions and rationales remain available, allowing the current correction to address a previous deviation rather than starting with an unrelated guess. A cap-down instruction changes the orientation criterion, so the same final position cannot reuse the cap-up success judgment. Only after goal-pose inference does the robot attempt grasping and execution toward that pose; a visually correct goal still does not guarantee that the arm can reach it successfully.

Loss & Training

The method introduces no new training loss, additional fine-tuning stage, or specialized learnable pose module; its changes concern the inference interface and feedback process. Experiments use GPT-5.2, Gemini-2.5-Flash Lite, and InternVL3-14B, and backbone capability affects final success rates. Training-free means that the framework performs no task-specific training, not that its backbone, segmentation, or grasping models have never been trained. Robot experiments use GraspNet for grasp prediction and OMPL for motion planning; the VLM does not directly emit high-frequency joint commands. In LIBERO, the system aligns the gripper center rather than the wrist with the grasp point and uses a lift, planar translation with pose adjustment, and vertical lowering trajectory. Both execution-side changes are applied to SoFar and the proposed method, preventing an exclusive planning improvement from being mistaken for a pose-inference advantage.

Key Experimental Results

Main Results

Open6DOR V2 originally contains 4,389 tasks; evaluation uses 4,370 after excluding scenes with preprocessing failures. The following values are selected from Table 1, page 8. All values are success rates (%); position and rotation tracks use benchmark tolerances, while the 6-DoF track requires both predictions to succeed. SoFar and Open6DOR-GPT both use GPT-5.2; the main text does not specify numerical tolerances here, so none are supplied in this note.

Method	Position track overall	Rotation track overall	6-DoF track position	6-DoF track rotation	6-DoF track overall
Open6DOR-GPT	61.0	Not reported	Not reported	Not reported	Not reported
SoFar	93.7	31.3	94.7	28.0	26.3
Ours, GPT-5.2	94.4	56.6	91.0	51.1	46.4
Ours, Gemini-2.5-Flash Lite	92.3	32.3	83.7	32.4	26.7
Ours, InternVL3-14B	81.0	24.4	80.2	28.2	22.8

The GPT-5.2 version improves overall 6-DoF success by 20.1 percentage points, but position success within that track decreases from 94.7% to 91.0%. The main gain should therefore be attributed to rotation and joint-goal evaluation, not uniform superiority across every metric. InternVL3-14B scores below SoFar on overall 6-DoF success, also showing that applicability to multiple backbones does not mean every backbone exceeds the strong baseline.

Ablation Study

Table 4, page 13: 180 subtasks are randomly sampled from each of the position, rotation, and 6-DoF tracks, totaling 540; values are success rates (%). This subset differs from the main evaluation, so its full-configuration result of 40.6% cannot be compared with 46.4% on the full task set as an otherwise controlled difference.

Config	Position track overall	Rotation track overall	6-DoF track overall
Without context memory	77.8	52.8	28.9
Without single-axis rotation prediction	80.6	57.6	36.7
Without coordinate axis visualization	71.7	56.9	31.1
Without multi-view reasoning	81.7	60.4	38.9
Full configuration	85.6	61.1	40.6

Removing memory reduces overall 6-DoF success by 11.7 percentage points, while removing coordinate axes reduces it by 9.5 percentage points. Multi-view reasoning contributes only 1.7 percentage points on this subset; the authors attribute this to tabletop scenes often being clear from the input view, not to multi-view reasoning being unnecessary under severe occlusion. Without the single-axis restriction, Euler rotations are applied in x, y, z order; the authors observe that the model often predicts a nonzero angle for just one axis, weakening this ablation's contrast.

Key Findings

Robot execution must be distinguished from geometric goal prediction: Table 2, page 11 evaluates 528 random tasks under the LIBERO/Franka protocol and reports overall 6-DoF success of 14.1% for this method and 12.9% for SoFar. This is an improvement of only 1.2 percentage points, much smaller than the goal-prediction gap; 46.4% must not be presented as end-to-end arm success. A failed grasp with an unchanged scene counts as failure, preventing false positives when the original scene happens to satisfy the goal. SIMPLER uses the WidowX + Bridge setup; the following selection from Table 3, page 11 retains grasp and final success to distinguish bottlenecks.

Task	SoFar grasp (%)	SoFar final (%)	Ours grasp (%)	Ours final (%)
Put Spoon on Towel	62.5	58.3	62.5	58.3
Put Carrot on Plate	79.2	70.8	83.3	45.8
Stack Green Block on Yellow Block	50.0	33.3	50.0	37.5
Put Eggplant in Yellow Basket	62.5	8.3	70.8	66.7

The carrot task has higher grasp success but lower final success, illustrating that acquiring an object and completing placement are different outcomes. The spoon task ties SoFar, so the authors' use of best performance should not be read as a strict win on every task.

Highlights & Insights

• Coordinate axes provide both a visual anchor for directional language and a translation scale. One prompt addresses viewpoint-dependent direction confusion and distance estimation together.
• Evaluation and proposal share a backbone but receive different visual contexts. Multiple views find errors, while the supporting view grounds the correction, making feedback more specific.
• Context memory has the largest ablation effect, suggesting that the bottleneck extends beyond single-frame recognition. Consistency between successive attempts is also important for spatial reasoning.
• The transferable idea is to express a complex goal through executable increments whose consequences can be observed again. This is closer to a test-time search interface than a new robot-policy training method.

Limitations & Future Work

• The authors explicitly identify repeated VLM invocation and rendering as sources of compute cost and latency, making direct low-latency control unsuitable; the main paper provides no numerical timing result to cite.
• The evaluator remains limited by backbone perception, and visual acceptance is not equivalent to rigorous collision checking or dynamics validation. The paper provides no formal guarantee of geometric feasibility.
• Robot experiments are simulated and do not establish real-robot deployment; closed-loop pose inference also does not automatically handle slipping or grasp errors during execution.
• Additional VLA comparisons, failure cases, and iteration-budget analysis are deferred to supplementary material absent from the supplied cache, so this note does not extend those claims.
• Extracted Eqs. (5) and (6) are corrupted; page 9 describes difficulty levels as level 1/2, while the position columns of Table 1 use Level 0/1. This note compares aggregate metrics without silently reconciling the numbering.
• A reader-proposed direction is to add geometric checks and return execution failures to goal inference while reporting success and latency under equal invocation budgets; these are future directions, not implemented features.

Related Work & Insights

• Versus SoFar (paper reference [31]): SoFar uses language-grounded orientation and structured scene representations, while this method emphasizes direct visual feedback after re-rendering; its primary advantage concerns orientation-sensitive goal prediction.
• Versus Open6DOR-GPT ([8]): its released code supports only the position track, explaining absent rotation and joint results in Table 1; missing results must not be treated as zero success.
• Versus Dream2Real ([17]): both use rendering and VLMs, but Dream2Real emphasizes candidate scoring, whereas this method asks the model for incremental pose corrections and then observes their consequences.
• Versus OpenVLA and SpatialVLA ([18], [33]): these are action policies, while this method combines goal-pose inference with separate grasping and planning; the evaluated VLAs use OXE pretrained checkpoints rather than task-specific fine-tuned variants.
• Research direction: test whether an independent geometric verifier correcting VLM stopping decisions is more effective than simply adding reasoning iterations. Premature acceptance, infeasible goals, and actual control failures should be measured separately to locate the source of improvement.

Rating

• Novelty: 4/5. Closed-loop reasoning has precedents, but the visual coordinate interface and incremental actions are specifically designed for 6D goal poses.
• Experimental Thoroughness: 4/5. Multiple backbones, two simulation benchmarks, and component ablations are included, but physical execution, cost, and statistical uncertainty need further evidence.
• Writing Quality: 4/5. The pipeline and ablation explanations are clear, although broad claims need to be read against nonuniform table results, and corrupted cached equations impede precise reproduction.
• Value: 4/5. The work offers a practical route to better orientation inference without task-specific fine-tuning while preserving the distinction between goal prediction and robot execution.