RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks

Conference: ECCV2026
Paper: ECCV Official Page
Code: https://github.com/RoboClaw-Robotics/RoboClaw
Area: Robotics & Embodied AI
Keywords: long-horizon manipulation, self-resetting data collection, entangled action pairs, process supervision, vision-language-action models

TL;DR

RoboClaw uses the same vision-language model (VLM) agent for robot data collection and execution, pairing forward actions with inverse reset behaviors and supervising multiple vision-language-action (VLA) policies online; real-robot experiments show reduced human collection time and improvements across four manipulation policies as closed-loop data accumulate.

Background & Motivation

VLA models can map images, language instructions, and robot states to actions, but placing one object correctly is different from reliably tidying an entire table. A tidying task combines grasping, transport, insertion, drawer closing, and wiping; the state left by one operation may not satisfy the next policy's preconditions. When independently trained policies are simply executed in a predefined sequence, an empty grasp can be mistaken for completion, allowing later actions to proceed until the whole task fails.

Training these skills also involves more than collecting demonstrations. After each trial, a person may need to reposition objects, judge success, handle failures, and start another attempt. If collectors, trainers, and deployment operators use different success criteria or subtask boundaries, a state labeled as complete in the data may not satisfy execution requirements. The problem therefore concerns both physical state distributions and task semantics: deployment encounters displaced states rarely seen in demonstrations, while disconnected stages may interpret those states differently.

RoboClaw places these stages under one high-level controller and teaches the robot how to return to a state from which it can practice again. Recovery need not be as difficult as the forward operation: removing a lipstick from a narrow slot is generally easier than inserting it precisely, making that asymmetry useful for repeated collection. Core Idea: maintain consistent task semantics across collection, training, and execution through shared memory and tool interfaces, generate data with learned forward-inverse action pairs, and improve long-horizon execution through process supervision and experience feedback.

Method

Overall Architecture

Inputs are a user task, visual observations, and robot states; outputs can be a completed long-horizon task or newly collected training trajectories. An off-the-shelf VLM interprets the scene, progress, and next step, while low-level π0.5 policies produce short-horizon action chunks. High-level language reasoning is not used directly as high-frequency motor control.

The framework distinguishes three levels. A Policy is a VLA model that produces robot actions; a Tool is an MCP interface for starting, stopping, or switching policies and querying the environment; a Skill is a reusable procedure that orchestrates tool calls. A Skill is not another neural network with the same name: a long-horizon execution skill, for example, can query the environment and then start the appropriate manipulation policy.

A shared context drives two runtime branches: collection uses EAP forward-inverse action pairs, while deployment uses process supervision and skill scheduling. Both feed experience into the same learning loop. Solid arrows below denote runtime control or experience recording; dashed arrows denote policy training and updates, not back-propagation at every robot control step.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Task, images, and robot states"] --> B["Shared Context and Tool Hierarchy"]
    B -->|Collection mode| C["EAP Forward-Inverse Action Pairs"]
    B -->|Deployment mode| D["Process Supervision and Skill Scheduling"]
    C --> E["Experience Feedback and Policy Expansion"]
    D --> E
    E -.->|Training data| F["π0.5 policy pool fine-tuning"]
    F -.->|Update callable policies| B
    D -->|Observation feedback| B

Key Designs

1. Shared Context and Tool Hierarchy

The high-level controller does not reinterpret the task from scratch whenever an image arrives. It maintains three types of structured memory: role identity specifies collection or execution mode and available tools; task-level memory stores the global goal, decomposed subtasks, and completion states; working memory retains the active skill and recent tool calls. The VLM uses observations and these memories for in-context learning and reasoning, identifying relevant scene elements, determining the current objective and success criteria, and deciding whether to continue, switch, or correct execution. The extracted memory equation in Section 3.2 is inconsistent with nearby prose, so this note follows the clear three-part definition in Section 3.1 rather than reproducing the damaged expression.

MCP tools connect decisions to execution: starting and terminating policies, retrieving environment summaries and robot states, changing policies, and requesting human help. When invoking a policy, the agent dynamically generates a subtask instruction, which conditions π0.5 together with visual inputs and joint states. Thus, the meaning of placing an item in a drawer and closing it can govern both collection-time completion checks and deployment-time progression. Tool results update memory. Shared decision logic reduces semantic fragmentation, but does not guarantee correct visual judgments.

2. EAP Forward-Inverse Action Pairs

Entangled Action Pairs (EAP) couple a forward manipulation policy with an inverse reset policy. The forward policy moves the object to its target; after confirming completion, the controller invokes the inverse policy to return the scene to a precondition region suitable for another attempt, then starts the next cycle. The aim is to restore usable preconditions, not reproduce every pixel or mechanically replay actions backward. The inverse behavior is itself learned. Both trajectories are recorded for subsequent policy learning.

This mechanism exploits asymmetric task difficulty. Inserting a lipstick into a narrow slot requires precise positional and rotational alignment, whereas removing it does not require solving that insertion problem again. A relatively reliable inverse policy can therefore support repeated practice of a harder forward policy. Resetting is not free: the four inverse policies achieve only 36/50 to 43/50 successful trials, rather than guaranteed success, and abnormal states can still require human assistance. Here, entanglement mainly means that paired behaviors provide the conditions needed for subsequent collection cycles; it should not be interpreted as a new joint forward-inverse loss.

3. Process Supervision and Skill Scheduling

At deployment, the controller selects a forward policy for the current subtask and periodically reads environment summaries and robot states during execution. It updates task memory and advances only when the success condition is satisfied; otherwise, it can retry the current policy or use Change Policy to select another available policy. Issuing a command is therefore distinct from completing a subtask, preventing later skills from blindly proceeding after a failed action. This is a high-level supervisory loop; the paper does not provide latency measurements establishing a real-time control frequency.

Failure handling also depends on whether the scene has deteriorated. An empty grasp that leaves the bottle upright is a non-degrading failure and can usually be retried. A grasp that tips the bottle over or moves it outside the normal starting region is a degrading failure requiring recovery or replanning. Deployment primarily composes forward skills and subsequently added recovery skills; it does not invoke the collection-time inverse reset policy after every successful subtask, which would undo task progress. Repeated failures, unsuccessful recovery, or safety triggers lead to a Call Human request, so human oversight remains part of the system.

4. Experience Feedback and Policy Expansion

Deployment trajectories are not merely evaluation logs: they enter subsequent training data, exposing policies to states produced by real sequential execution. The training pipeline includes human demonstrations, autonomously collected trajectories, and human interventions after failed autonomous rollouts. Shared task interpretation and tool interfaces allow experience to accumulate under relatively consistent semantics. The objective is to reduce the state-coverage gap left by isolated demonstrations, not to claim that all distribution shift has been eliminated.

Experience can also expand the policy library. Early degrading failures, such as a fallen bottle, may require manual scene recovery. After relevant experience is collected, recovery behaviors can be trained into dedicated policies and invoked in later runs. Improvement therefore includes both stronger existing policies and a wider range of recoverable states. However, Section 4.4 provides a mechanism description and failure examples, not a separate quantitative curve for incremental recovery-skill acquisition. It should not be presented as a fully evaluated autonomous skill-discovery algorithm.

A Worked Example

Consider the paper's vanity-table scenario, which combines primer placement in a drawer, lipstick insertion, lotion placement, and tissue wiping. The following walkthrough illustrates the described method; it is not a step-by-step transcription of a recorded trial.

During collection, the system practices placing the primer in its target region and closing the drawer. The controller inspects the scene, generates the subtask instruction, and starts the corresponding π0.5 policy. After confirming success, it invokes the paired reset behavior to restore conditions for another attempt. Forward and inverse trajectories are recorded continuously, allowing subsequent policy updates to use real execution data without requiring a person to reposition the scene after every trial.

During deployment, the controller again interprets the scene, but completing the primer task leads to other unfinished subtasks rather than removing the primer for another collection cycle. If the lotion grasp fails, it checks the bottle: an unchanged bottle permits a retry; a fallen bottle calls for a recovery skill; unsuccessful recovery leads to human assistance. Task memory is updated only after observation confirms completion.

The example separates the roles of EAP and process supervision. EAP supports repeated training-data acquisition, while supervision makes an individual task tolerant of local failures. Displaced states and recovery experiences recorded during deployment feed later fine-tuning; they do not guarantee that the policy immediately learns every new state during the current execution.

Loss & Training

The low-level π0.5 model predicts action chunks conditioned on images, agent-generated language instructions, and joint states. Conditional flow matching trains a velocity field that transports Gaussian noise toward the action distribution. The cached Equation (5) contains damaged norm notation, subtraction, and arguments, so this note explains the objective without reconstructing an exact author equation. The main contribution is the collection and orchestration pipeline, not a new VLA objective. The high-level VLM uses in-context learning; the paper does not describe joint gradient training of that controller.

Table 1 specifies π0.5 fine-tuning with bfloat16, batch size 16, 10k training steps, 100 warmup steps, a learning rate of 2.5 × 10⁻⁵, and gradient checkpointing. LoRA rank and alpha are both 16, dropout is 0.1, target modules are all-linear, and inference uses 3 steps. The main text does not identify the VLM model or training GPU configuration, so these hyperparameters alone do not establish complete resource requirements for reproduction.

The subtask-policy experiment fixes the human demonstration budget for each forward policy and compares 1 to 5 collection iterations, adding 50 data samples per iteration. The main text does not state the absolute initial demonstration count per policy or fully explain how every failed trajectory is filtered or labeled. Results should therefore be interpreted as the effect of adding closed-loop data under a fixed human demonstration budget, not as a collection-mechanism ablation with total data volume strictly controlled.

Key Experimental Results

Main Results

All experiments use the Agibot G01 dual-arm mobile manipulation platform, with 20 degrees of freedom excluding end-effectors and an AGIBOT OmniPicker gripper with one active degree of freedom on each arm. Collection-efficiency experiments cover a bedroom vanity table, kitchen shelf, study desk, and convenience-store shelf. Detailed subtask and long-horizon results focus on the vanity-table task; this is not a full generalization evaluation across four independent public test sets.

Both metrics below are normalized to RoboClaw = 1 and compare against manual collection for the same amount of data. Human time and rollout intervention are different quantities: the 8.04 ratio is not robot throughput or a time speedup.

Metric and Condition	Manual Baseline	RoboClaw	Interpretation and Source
Relative human time for the same trajectory count	2.16	1.00	Figure 4(a); equivalent to approximately 53.7% less human time
Relative human intervention during rollouts	8.04	1.00	Figure 4(b); the exact intervention accounting is not fully specified

For long-horizon evaluation, Baseline 1 is π0.5 trained on the same dataset without the RoboClaw framework. Baseline 2 multiplies the success rates of four subtask policies; it is an estimate under an independence assumption, not a second executed system. Figure 4(c) reports results over 20 trials, and the abstract claims a 25% success-rate improvement. However, the main text does not clarify whether that means relative percent or percentage points, and the cached text extraction does not preserve exact curve values. This note therefore retains the qualitative advantage without converting 25% into an independently verified absolute gain.

Ablation Study

The paper does not provide conventional ablations separately removing memory, EAP, supervision, or recovery. The following table summarizes the data-iteration analysis from Table 3 alongside inverse-policy evaluation from Table 2. The inverse column is not a matched-condition ablation of the fifth-iteration forward policy. All success counts have a denominator of 50.

Subtask	Forward Success, Iteration 1	Forward Success, Iteration 3	Forward Success, Iteration 5	Gain, Iteration 1→5	Inverse Reset Success
Body Lotion	21/50	32/50	43/50	44 percentage points	36/50
Primer	23/50	31/50	40/50	34 percentage points	38/50
Lipstick	2/50	11/50	23/50	42 percentage points	43/50
Tissue Wipe	11/50	14/50	26/50	30 percentage points	39/50

Key Findings

• Closed-loop collection improves all four operations, but difficult tasks remain unsolved: fifth-iteration lipstick insertion reaches only 46%, and tissue wiping reaches 52%. Recovery and retry mechanisms remain important for long-horizon execution.
• Lipstick reset succeeds in 43/50 trials, whereas first-iteration insertion succeeds in only 2/50, directly illustrating the forward-inverse asymmetry motivating EAP. This does not establish that resetting is easy for every physical task.
• The long-horizon advantage belongs to the complete monitored, retry-capable framework. Without matched retry counts, execution durations, and VLM-call budgets, the entire improvement cannot be attributed to one module.

Highlights & Insights

• Learning reset behaviors turns a human-labor problem into a policy-design problem. When removal is easier than placement, the inverse skill can provide infrastructure for repeatedly practicing the forward skill.
• Having the same controller define success during collection and execution addresses a practical pipeline issue beyond sharing model weights. It helps align training-data task boundaries with deployment scheduling boundaries.
• Distinguishing non-degrading from degrading failures gives recovery libraries a concrete organizing principle. The former mainly need retries; the latter require restoring preconditions rather than repeatedly invoking the same policy.

Limitations & Future Work

• The authors acknowledge latency from cloud-based large models and the assumption that practical inverse reset behaviors exist. Environment-changing operations such as wiping or consuming materials particularly need a precise definition of a reusable reset state.
• This note's assessment: the main text omits the VLM identity, complete data scale, training resources, failure-filtering details, and confidence intervals. Four scenarios should not be interpreted as broad cross-robot or cross-task generalization evidence.
• This note's assessment: the meaning of the 25% gain and the contributions of high-level modules remain unclear. Future evaluation should report complete long-horizon success rates, task duration, retry and human-intervention budgets, and ablations matched for total data volume and runtime budget.

Related Work & Insights

• vs π0.5: π0.5 provides language-conditioned action generation, while RoboClaw adds a unified collection, supervision, and experience-feedback layer around it. It is an orchestration framework that can augment the underlying VLA, not a replacement architecture.
• vs Genie Centurion / RoboCopilot: These systems also address failure correction and human participation. RoboClaw emphasizes learned inverse resets and deployment scheduling under one lifecycle controller, while retaining human fallback.
• vs SayCan / HiRobot / Inner Monologue: These methods offer related ideas in skill selection, hierarchical control, and feedback-driven replanning. RoboClaw emphasizes shared decision context between collection and deployment and returning deployment experience to policy training; the paper does not directly compare these systems quantitatively.

Rating

• Novelty: 4/5. Forward-inverse pairing and lifecycle integration offer system-level value, although several underlying components have established precedents.
• Experimental Thoroughness: 3/5. Real-robot results cover collection costs and policy iteration, but module ablations and resource specifications remain incomplete.
• Writing Quality: 3/5. The main narrative is clear, but headline gain definitions and implementation details need clarification; cached equations also suffer extraction damage.
• Value: 4/5. The approach provides useful engineering guidance for robots with expensive repeated collection and feasible learned reset skills.