Skip to content

Source paper

SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg, C. Karen Liu
arXiv 2602.16863 · 202612 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents SimToolReal, a framework for achieving zero-shot dexterous tool manipulation in the real world using a single Reinforcement Learning policy trained entirely in simulation. By training on procedurally generated "primitive" objects (simple combinations of handles and heads) to track random goal poses, the policy generalizes to diverse real-world tools and tasks (like hammering or sweeping) by conditioning on a coarse geometric abstraction rather than visual appearance.

How it works (AI summary)

The authors employ model-free Reinforcement Learning (using the SAPG algorithm) within a massively parallel physics simulator (IsaacGym). The core technique involves an object-centric abstraction where the policy input is the object's 6D pose (tracked via FoundationPose) and a grasp bounding box (from SAM 3D), decoupling control from raw visual perception. At inference time, goal trajectories are extracted from human demonstration videos, and the policy operates as a general-purpose pose tracker.