Skip to content

Source paper

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports

Yuchen Yang, Yuqing Shao, Duxiu Huang and 11 more

Yuchen Yang, Yuqing Shao, Duxiu Huang, Linfeng Dong, Yifei Liu, Suixin Tang, Xiang Zhou, Yuanyuan Gao, Wei Wang, Yue Zhou, Xue Yang, Yanfeng Wang, Xiao Sun, Zhihang Zhong

arXiv 2603.09896 · 202622 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

Fully Automated Metric Depth Estimation for Dynamic Small Objects

Develop a physics-informed tracking module combining small-object detection with dynamics constraints to automate 3D depth estimation for fast-moving items.

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#Vision#Data & evaluation

Overcoming Perspective Ambiguity in 2D-to-3D VLM Reasoning

Inject camera intrinsics and extrinsics directly into vision-language model encoders using geometric adapters to resolve perspective ambiguity in spatial reasoning.

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
🟠Medium💡Moderate InnovationMultiple GPUsRelated work found#Multimodal#Reasoning

Domain-Agnostic Pre-training for Zero-Shot Cross-Sport Spatial Generalization

Implement a domain-agnostic spatial pre-training objective using abstract geometric graphs to improve zero-shot cross-sport spatial generalization.

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
🟡Fairly easy💡Moderate InnovationLarge clusterNo close match found#Multimodal#Vision

Instruction-Controllable VLMs for Strict Formatting via Preference Optimization

Fine-tune open-source vision-language models using Direct Preference Optimization on a format-compliance dataset to strictly eliminate verbose explanations and enforce exact prompt output formats.

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports
🟢Easy🔧Minor TweakMultiple GPUsRelated work found#LLMs#Multimodal

AI summary

This paper introduces CourtSI, the first large-scale spatial intelligence dataset and benchmark tailored for sports scenarios to evaluate Vision-Language Models (VLMs). By leveraging well-defined court geometries, the authors reconstruct 3D sports scenes to generate over 1 million metric-aware QA pairs, revealing significant gaps in current VLMs' spatial reasoning capabilities.

How it works (AI summary)

The methodology uses a semi-automatic data engine leveraging fixed court dimensions for metric-aware camera calibration via a Perspective-n-Point (PnP) solver. This is combined with PromptHMR for 3D human mesh recovery and human-in-the-loop annotations to establish a unified world coordinate system, enabling the programmatic generation of spatial QA pairs.