Robust Visual-Tactile State Estimation for Occlusion Handling
Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.
Source paper
Generated by AI. The paper’s authors did not write these.
Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.
Augment robotic policy observation spaces with wrist force-torque sensor readings and modify reward functions to support contact-rich tasks.
Extend procedural tool generation pipelines to include articulated primitives and train policies to track internal joint states alongside 6D poses.
This paper presents SimToolReal, a framework for achieving zero-shot dexterous tool manipulation in the real world using a single Reinforcement Learning policy trained entirely in simulation. By training on procedurally generated "primitive" objects (simple combinations of handles and heads) to track random goal poses, the policy generalizes to diverse real-world tools and tasks (like hammering or sweeping) by conditioning on a coarse geometric abstraction rather than visual appearance.
The authors employ model-free Reinforcement Learning (using the SAPG algorithm) within a massively parallel physics simulator (IsaacGym). The core technique involves an object-centric abstraction where the policy input is the object's 6D pose (tracked via FoundationPose) and a grasp bounding box (from SAM 3D), decoupling control from raw visual perception. At inference time, goal trajectories are extracted from human demonstration videos, and the policy operates as a general-purpose pose tracker.