Fully Automated Metric Depth Estimation for Dynamic Small Objects
Develop a physics-informed tracking module combining small-object detection with dynamics constraints to automate 3D depth estimation for fast-moving items.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The current data engine relies on a semi-automatic process where human annotators must manually click ground projections for balls and correct the lowest mesh vertex for player depth. Monocular models like DepthAnythingV3 fail to accurately estimate the metric depth of small, fast-moving objects like sports balls.
“Ball positions in the images are manually annotated. We observe that existing monocular depth estimation methods fail to produce reliable metric reconstruction. Instead, we manually estimate object heights relative to the court plane to enable accurate camera-to-world transformation.”
Limitation
The reliance on human-in-the-loop annotations for ball and player depth bottlenecks the scalability of the dataset to millions of uncurated sports videos.
Proposed Solution
Develop a fully automated, physics-informed 3D tracking module. Integrate a specialized small-object detection model with a physics engine that enforces trajectory constraints (gravity, air resistance, spin) to automatically optimize the 3D position of the ball and feet contact points without human intervention.
Why This Matters
Eliminating the human annotation bottleneck would allow the curation of web-scale, multi-sport spatial intelligence datasets, drastically increasing training volume for VLMs.
How to Get Started
How to test it: Hypothesis: A physics-informed 3D tracking module improves ball and player depth estimation without human intervention. Baseline: Manual annotation and PromptHMR. Metric: 3D localization error in centimeters. Stop if the error does not fall below 15cm.
- Benchmark existing monocular depth models like DepthAnythingV3 on dynamic small-object datasets.
- Integrate a specialized small-object detector to localize fast-moving items across video frames.
- Incorporate a physics engine enforcing gravity, air resistance, and spin to correct trajectory hypotheses.
- Automate the extraction of 3D positions and contact points without relying on human manual clicks.
Code linked to this paper on Hugging Face: github.com/Visionary-Laboratory/CourtSI
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the provided results cover a physics-informed 3D tracking module using small-object detection paired with trajectory constraints and foot contact points. Papers like UniDepthV2 and Lifelong-MonoDepth discuss general monocular metric depth estimation, but do not combine physics engines with small-object dynamics.
- UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler · 2025 · 274 citations
- Lifelong-MonoDepth: Lifelong Learning for Multidomain Monocular Metric Depth Estimation · 2023 · 13 citations
We searched OpenAlex and arXiv for “Develop a physics-informed tracking module combining small-object detection with dynamics constraints to automate 3D depth estimation for fast-moving items.” and “Fully Automated Metric Depth Estimation for Dynamic Small Objects” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in SportsYuchen Yang, Yuqing Shao, Duxiu Huang +11 more · 2026
This paper introduces CourtSI, the first large-scale spatial intelligence dataset and benchmark tailored for sports scenarios to evaluate Vision-Language Models (VLMs). By leveraging well-defined court geometries, the authors reconstruct 3D sports scenes to generate over 1 million metric-aware QA pairs, revealing significant gaps in current VLMs' spatial reasoning capabilities.
Related Ideas
- Unsupervised Adaptation to Human Multiplayer GameplaySolaris: Building a Multiplayer Video World Model in Minecraft🟡Fairly easy
- Physics-Constrained Reward Modeling for RLHFHunyuanVideo 1.5 Technical Report🟠Medium
- Adaptive Early-Stopping and Feed-Forward Alternatives for 3DGS OptimizationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🟠Medium
- Extending Spatial Curation to Dynamic Scenes via 4D RepresentationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🔴Hard
- Adaptive Chunking Mechanisms for Event-Driven Test-Time Training in Spatial MLLMsSpatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training🟡Fairly easy