Skip to content

Controlling the World by Sleight of Hand

Conference: ECCV 2024
arXiv: 2408.07147
Code: Project Page
Area: Image Generation
Keywords: Action-Conditioned Generation, Hand Interaction, Diffusion Models, World Models, Robot Generalization

TL;DR

Proposes CosHand, which uses binary hand masks as action conditions and fine-tunes on pretrained Stable Diffusion to predict future images after hand-object interaction, showing zero-shot generalization capabilities to robotic end-effectors.

Background & Motivation

Humans possess an innate mental simulation capability—visualizing an object allows one to imagine the physical changes that would occur after interaction. Existing generative models primarily rely on text or unconditional settings for image generation/editing; however, text cannot precisely describe the spatial location, orientation, and force of interactions. For instance, for an instruction like "squeeze the pillow to deform it horizontally", text struggles to encode the exact direction and extent of the deformation.

Core Problem: How can machines be endowed with action-conditioned interaction imagination? Specifically, given an image of the current scene and a desired hand interaction location/shape, how can the future image following the interaction be generated?

Key Insight: There are massive amounts of unlabeled video data demonstrating hand-object interactions on the internet (e.g., 180k+ videos in SomethingSomethingv2). These videos naturally provide "before-after" interaction pairs, enabling highly efficient, large-scale training of action-conditioned generative models. Furthermore, leveraging binary hand masks as conditioning signals instead of specific hand appearances endows the model with the inherent potential to generalize across different embodiments.

Method

Overall Architecture

CosHand is based on the Latent Diffusion Model (LDM) architecture. The key mechanism is to encode hand interaction information as conditioning signals injected into the diffusion model. The system takes three inputs:

  1. Current image \(x_t \in \mathbb{R}^{H \times W \times 3}\): The scene prior to interaction.
  2. Current hand mask \(h_t \in \mathbb{R}^{H \times W}\): Denoting the current hand position in the image.
  3. Target hand mask \(h_{t+1} \in \mathbb{R}^{H \times W}\): Denoting the expected hand position/shape after interaction.

The model learns the function \(f(x_t, h_t, h_{t+1}) = \hat{x}_{t+1}\), which outputs the future image post-interaction.

Key Designs

1. Data Acquisition Pipeline

Training data is automatically extracted from the SomethingSomethingv2 dataset: - Videos are decoded at 12 FPS, sampling "before-after" image pairs with a step of 3 frames. - Segment Anything (SAM) combined with the provided bounding boxes is used to obtain binary hand masks. - This pipeline is fully automated and requires no manual annotations, facilitating scale-up to larger datasets.

2. Dual-Path Conditioning Mechanism

CosHand utilizes two complementary conditioning pathways:

  • Channel Concatenation: The current image \(x_t\), the current hand mask \(h_t\), and the target hand mask \(h_{t+1}\) are individually encoded into latent representations via a VAE encoder. These are concatenated along the channel dimension to yield a context latent vector \(c_i \in \mathbb{R}^{h \times w \times 3c}\), which is then concatenated with the noisy latent vector \(z_i \in \mathbb{R}^{h \times w \times c}\) along the channel dimension to serve as the input for the U-Net. This pathway provides spatially aligned, fine-grained control signals.

  • Cross-Attention: A frozen CLIP image encoder is employed to extract the semantic embedding of the input image \(\tau(x_t)\), injecting global semantic information into the U-Net via cross-attention layers. This ensures that the generated image remains semantically consistent with the input (preserving object identity, background, etc.).

3. Leveraging Pretrained Priors

  • The U-Net and VAE encoder/decoder are initialized from pretrained Stable Diffusion.
  • Pretrained models have seen billions of hand-object interaction image pairs during pretraining, accumulating rich prior knowledge of physical interactions.
  • The fine-tuning strategy leverages these priors, enabling the model to generalize to novel objects and scenarios outside the training distribution.

4. Agent-Agnostic Design

Relying on binary masks instead of RGB hand images as conditions decouples the control signals from the specific embodiment's appearance. Consequently, during inference, the hand mask can be replaced directly with the mask of a robotic end-effector, achieving zero-shot cross-embodiment migration without extra fine-tuning.

Loss & Training

Training Objective: Standard LDM noise prediction loss

\[\min_\theta \mathbb{E}_{z, c \sim \mathcal{E}(x), i, \epsilon \sim \mathcal{N}(0,1)} \| \epsilon - \epsilon_\theta(z_i, c_i, \tau(x_t), i) \|_2^2\]

Training Details: - Hardware: 8×A100-80GB, trained for 7 days. - Optimizer: AdamW, learning rate \(10^{-4}\). - Image Resolution: 256×256 (latent space 32×32) to support a large batch size of 192. - Classifier-free guidance: Conditioning signals \(c_i\) and \(\tau(x_t)\) are randomly dropped out with a 5% probability during training. - During inference, the CFG scale is set to 2.5 (optimal value determined via ablation studies).

Key Experimental Results

Main Results

Datasets: SomethingSomethingv2 (SSv2) test set + self-collected In-the-wild dataset (45 videos).

Method SSv2 PSNR↑ SSv2 SSIM↑ SSv2 LPIPS↓ In-the-wild PSNR↑ In-the-wild SSIM↑ In-the-wild LPIPS↓
MCVD Lowest Lowest Highest Lowest Lowest Highest
UCG (Unconditional) Medium Medium Medium Medium Medium Medium
InstructPix2Pix Lower Lower Higher Lower Lower Higher
TCG (Text-conditioned) Lower Lower Higher Lower Lower Higher
CosHand Highest Highest Lowest Highest Highest Lowest

CosHand consistently outperforms all baseline models in PSNR, SSIM, and LPIPS across both datasets.

Ablation Study

Ablation Variant Performance Change Analysis
No SD pretraining (trained from scratch) Performance drops significantly Lacks prior knowledge of hand-object interactions
No CLIP conditioning Performance drops across all three metrics Loss of global semantic information makes reconstructing details difficult
10% training data Drop in both performance and generalization Dataset scale is positively correlated with model capability
Multi-frame context (4 frames) Performance improvement Introduces temporal understanding, but multi-frame scenarios are hard to obtain in practice
CFG scale analysis scale=2.5 remains optimal Scale > 2.5 results in overly conservative generations; scale < 2.0 ignores the input image

Key Findings

  1. Zero-Shot Cross-Embodiment Generalization: Taught strictly on human hand data, CosHand generalizes directly to robotic end-effectors (BridgeDataV2), successfully predicting results of simple interactions such as pushing towels or picking up cups.
  2. Multi-Future Prediction: Given the same input but different hand masks, the model can predict diverse, diverging futures. Sampling multiple times under the same condition models the inherent uncertainty of interaction/environmental forces.
  3. Robustness to Hand Mask Quality: Generates plausible results even with rough hand-drawn masks, while fine-grained masks (e.g., generated by SAM) produce more accurate outputs.
  4. Generalization Over Interaction Types: Best performance is observed in translation, stretching, and squeezing; more complex interactions like rotation and folding are also reasonably represented.

Highlights & Insights

  1. Elegant Problem Formulation: Using hand masks instead of text as interaction conditions is intuitive yet precise, perfectly binding spatial information and action semantics.
  2. Graceful Data Pipeline: Automated training data extraction from unlabeled videos bypasses manual annotations, streamlining scale-up to web-scale datasets.
  3. Agent-Agnostic Paradigm: Relying on binary masks allows training on human data and inference on robotic execution, showcasing a highly valuable transfer paradigm.
  4. Probabilistic Modeling of Interaction Uncertainty: The inherent stochasticity of diffusion models naturally models the uncertainty in the direction/magnitude of interaction forces; multiple sampling runs yield a variety of physically plausible futures.
  5. Creative Image Editing Applications: Hand masks can be superimposed on any image to perform physically realistic editing, such as moving the Golden Snitch in Harry Potter.

Limitations & Future Work

  1. Failure in Unrealistic Scenarios: The model fails when confronted with highly implausible physical interactions (e.g., manually pushing a building or altering the shape of a cloud).
  2. Segmentation Ambiguity: When physical objects are close together, interacting with one object sometimes undesirably affects surrounding items.
  3. Resolution Limits: Restricted to 256×256 resolution. While higher resolutions would retain more spatial details, they demand more computational resources.
  4. Single-Frame Context Limitation: Only uses a single frame of context by default. Although multi-frame models improve results, multi-frame scenarios are harder to acquire in real-world pipelines.
  5. Challenges with Complex Interactions: Demanding movements such as large-angle rotations or intricate folding still require improvement.
  6. Limited Robot Generalization: Zero-shot cross-embodiment transfers succeed only on simple actions; intricate robotic manipulations remain to be exhaustively validated.
  • InstructPix2Pix: A representative for text-conditioned editing; however, text struggles to locate spatial interactions. CosHand bridges this gap using hand masks.
  • Zero-1-to-3: Camera-pose-conditioned generation, which similarly leverages Stable Diffusion priors to handle distinct conditioning controls.
  • World Models (Ha & Schmidhuber): CosHand is essentially a visual world model, predicting state changes induced by actions.
  • BridgeData V2: A robotic manipulation dataset. CosHand's zero-shot generalization to this dataset exhibits great potential for human-to-robot knowledge transfer.
  • ControlNet / LoRA: Similarly injects conditioning control via extra weights, but the hand mask conditioning in CosHand is more intuitive and physically precise.

Insight: This work highlights a promising direction—learning physical interaction dynamics from web-scale human interaction videos and zero-shot transferring them to robotic agents. Future iterations could benefit from scaling up training data, leveraging higher resolutions, and modeling temporal contexts.

Rating

  • Novelty: ⭐⭐⭐⭐ — Utilizing hand masks as an interaction control is an innovative and highly intuitive design.
  • Experimental Thoroughness: ⭐⭐⭐⭐ — Comprehensive ablations and convincing cross-domain generalization experiments.
  • Writing Quality: ⭐⭐⭐⭐ — Well-structured and delivers a strong, logical motivation.
  • Value: ⭐⭐⭐⭐ — Holds significant reference value for robot planning and visual world models.