Domain-Agnostic Pre-training for Zero-Shot Cross-Sport Spatial Generalization
Implement a domain-agnostic spatial pre-training objective using abstract geometric graphs to improve zero-shot cross-sport spatial generalization.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The Qwen3-VL-8B model fine-tuned on CourtSI showed only limited improvements when evaluated on CourtSI-Ext (pickleball), specifically struggling with localization in the unseen sport.
“Furthermore, models trained on existing spatial intelligence benchmarks generalize poorly to CourtSI-Bench, suggesting that current datasets fail to sufficiently capture the challenges posed by dynamic sports scenarios.”
Limitation
The spatial reasoning capabilities learned by the VLM overfit to the specific court semantics and geometries of the training sports, failing to generalize to unseen, structurally similar domains.
Proposed Solution
Implement a domain-agnostic spatial pre-training objective. Instead of QA pairs tied to specific sports terms, fine-tune the model using abstract geometric representations (e.g., replacing court textures with grid lines via segmentation masks) or use contrastive learning to align abstract spatial graphs with textual descriptions.
Why This Matters
This yields a universally applicable spatial VLM capable of out-of-the-box 3D reasoning for any new environment without requiring costly domain-specific 3D data collection.
How to Get Started
How to test it: Hypothesis: Abstract geometric pre-training improves cross-sport spatial generalization. Baseline: Base Qwen3-VL-8B fine-tuned on CourtSI. Metric: Accuracy on CourtSI-Ext (pickleball). Stop if accuracy on unseen sport drops below baseline levels.
- Prepare the Qwen3-VL-8B model baseline and evaluate zero-shot performance on the CourtSI-Ext pickleball benchmark.
- Create an abstract spatial dataset by replacing specific court textures with standardized grid lines and segmentation masks.
- Design a contrastive pre-training objective aligning the abstract spatial layouts with descriptive spatial text.
- Fine-tune the model on the abstracted spatial pre-training task before testing zero-shot transfer to unseen sports.
Code linked to this paper on Hugging Face: github.com/Visionary-Laboratory/CourtSI
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature contains general sports deep learning surveys like A Survey of Deep Learning in Sports Applications and transfer learning works like OmniTraj, but none propose domain-agnostic pre-training using abstract geometric representations and grid masks for zero-shot cross-sport generalization.
- A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision · 2025 · 75 citations
- OmniTraj: Pre-training on heterogeneous data for adaptive and zero-shot human trajectory prediction · 2026 · 6 citations
We searched OpenAlex and arXiv for “Implement a domain-agnostic spatial pre-training objective using abstract geometric graphs to improve zero-shot cross-sport spatial generalization.” and “Domain-Agnostic Pre-training for Zero-Shot Cross-Sport Spatial Generalization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in SportsYuchen Yang, Yuqing Shao, Duxiu Huang +11 more · 2026
This paper introduces CourtSI, the first large-scale spatial intelligence dataset and benchmark tailored for sports scenarios to evaluate Vision-Language Models (VLMs). By leveraging well-defined court geometries, the authors reconstruct 3D sports scenes to generate over 1 million metric-aware QA pairs, revealing significant gaps in current VLMs' spatial reasoning capabilities.
Related Ideas
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation⛔Very hard
- Cross-Domain Zero-Shot Adaptation for Niche Visual ModalitiesSAM 3: Segment Anything with Concepts🔴Hard
- Scaling Unified Multimodal Models via Large Pre-trained Vision-Language InitializationsCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation🔴Hard
- Physics-Constrained Reward Modeling for RLHFHunyuanVideo 1.5 Technical Report🟠Medium
- Decoupled 4D Perception-to-Code Pipeline Using Geometric and Body Reconstruction PriorsVibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender🔴Hard