Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationNo close match found

Robust Visual-Tactile State Estimation for Occlusion Handling

Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.

Time

3-5 days

Compute level

Multiple GPUs

Resources

A multi-GPU training workstation with several high-end GPUs (e.g., 4x RTX 4090 or A100) for running parallel IsaacGym simulation environments and training recurrent filters.

What the paper used

IsaacGym with 24,576 parallel environments on GPUs using SAPG and an LSTM+MLP actor-critic architecture.

Prerequisites

computer vision, state estimation, sensor fusion

Scope

modify existing code

The Gap

The paper reports that 43.7% of real-world failures are caused by pose tracking loss using FoundationPose, specifically due to heavy occlusion by the robot hand or visual ambiguity of symmetric objects like markers.

“Conditioning on object pose goals alone is environment-blind, which can lead to collisions in cluttered scenes. We also currently assume tools are rigid. Pose alone can be insufficient to describe the state of non-rigid tools (e.g., scissors).”
From the paper, Section V, page 9. Matched against the PDF text.

Limitation

The system relies exclusively on third-person vision for state estimation. When the hand occludes the tool (common in dexterous manipulation), the visual tracker fails, causing the policy to lose track of the object state entirely.

Proposed Solution

Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history. Implement a recurrent filter (e.g., Kalman Filter or LSTM) that predicts the object's pose update based on hand joint velocities and contact feedback when visual confidence scores from FoundationPose drop below a threshold.

Why This Matters

Solving occlusion is critical for deployment; a policy that goes blind during the most critical part of manipulation (grasping and using) is unreliable for production environments.

How to Get Started

How to test it: Hypothesis: Fusing visual input with tactile and proprioceptive history via an LSTM filter will prevent tracking loss during hand occlusions. Baseline: Vision-only FoundationPose tracking. Metric: Task progress percentage and object tracking loss frequency. Stopping criteria: Failure rate reduction of at least 30% under heavy occlusion.

  1. Collect demonstration data featuring heavy hand-object occlusions using a robotic arm equipped with a wrist camera and joint sensors.
  2. Implement an LSTM-based recurrent filter or Extended Kalman Filter that takes hand joint velocities and contact states as inputs.
  3. Integrate the filter with FoundationPose to predict object poses when visual confidence scores fall below a predefined threshold.
  4. Benchmark pose tracking accuracy and success rates against the vanilla visual-only pipeline on cluttered manipulation tasks.

Code linked to this paper on Hugging Face: github.com/tylerlum/simtoolreal

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

The listed results discuss world models, surveys on robotic perception, deformable objects, and physical AI, but none of them cover the specific proposal of fusing visual input with tactile history using a recurrent filter when FoundationPose confidence drops. Therefore, this idea remains novel regarding these specific papers.

We searched OpenAlex for “Develop a multi-modal state estimator that fuses visual input with proprioceptive and tactile history using a recurrent filter to handle severe occlusions.” and “Robust Visual-Tactile State Estimation for Occlusion Handling” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg +1 more · 2026

This paper presents SimToolReal, a framework for achieving zero-shot dexterous tool manipulation in the real world using a single Reinforcement Learning policy trained entirely in simulation. By training on procedurally generated "primitive" objects (simple combinations of handles and heads) to track random goal poses, the policy generalizes to diverse real-world tools and tasks (like hammering or sweeping) by conditioning on a coarse geometric abstraction rather than visual appearance.

Related Ideas

Is this a good idea?

Share onXLinkedIn