Extending Spatial Curation to Dynamic Scenes via 4D Representation
Incorporate 4D Gaussian Splatting and dynamic deformation fields to separate static backgrounds from moving objects in dynamic indoor environments.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The current 2D-to-3D lifting and scene-level refinement stages assume a perfectly static indoor environment, relying heavily on standard Structure-from-Motion (SfM) and static 3DGS.
“The pipeline relies on multiple upstream components and per-scene optimization, which can be computationally expensive and may degrade under challenging videos (e.g., limited viewpoints, motion blur, heavy occlusion, or dynamic objects).”
Limitation
The pipeline degrades significantly or fails entirely when applied to videos containing dynamic objects, heavy occlusions, or severe motion blur, limiting its use to static environment scans.
Proposed Solution
Incorporate 4D Gaussian Splatting or dynamic deformation fields into the Geometric Optimization phase. Modify the scene-level tracking to separate static background Gaussians from dynamic foreground Gaussians, enabling the extraction of temporally-aware 4D bounding boxes.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This allows the framework to process in-the-wild dynamic videos (such as YouTube vlogs or egocentric human-object interaction videos), massively expanding the domain diversity of the generated dataset.
How to Get Started
How to test it: We hypothesize that incorporating dynamic deformation fields will improve reconstruction fidelity on videos with moving objects. We will compare against static 3DGS baselines on dynamic subset sequences using 3D metric accuracy and rendering PSNR. The experiment will stop if dynamic tracking fails to exceed static baseline metrics on non-static scenes.
- Review existing dynamic 3D scene datasets containing moving foreground objects and complex backgrounds.
- Modify the Geometric Optimization phase to integrate 4D Gaussian Splatting and deformation field modules.
- Develop scene-level tracking algorithms to explicitly segregate static background Gaussians from dynamic foreground elements.
- Evaluate 4D bounding box extraction and tracking performance on synthetic and real-world dynamic sequences.
Code linked to this paper on Hugging Face: github.com/Visionary-Laboratory/holi-spatial
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives and SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction explicitly address separating static and dynamic components using 4D representations for scene reconstruction. They cover the core concepts of dynamic deformation fields and decoupled foreground-background modeling proposed in the idea.
- 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives · 2024 · 25 citations
- CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving · 2025 · 37 citations
- SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction · 2025 · 6 citations
We searched OpenAlex and arXiv for “Incorporate 4D Gaussian Splatting and dynamic deformation fields to separate static backgrounds from moving objects in dynamic indoor environments.” and “Extending Spatial Curation to Dynamic Scenes via 4D Representation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial IntelligenceYuanyuan Gao, Hao Li, Yifei Liu +14 more · 2026
This paper introduces Holi-Spatial, a fully automated pipeline designed to convert raw video streams into high-fidelity 3D spatial annotations without human intervention. The primary contribution is the creation of Holi-Spatial-4M, a massive, spatially-aware multimodal dataset that significantly enhances the spatial reasoning and 3D grounding capabilities of Vision-Language Models.
Related Ideas
- 4D Spatiotemporal Pretraining for Dynamic ScenesUtonia: Toward One Encoder for All Point Cloudsđź”´Hard
- Decoupled 4D Perception-to-Code Pipeline Using Geometric and Body Reconstruction PriorsVibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blenderđź”´Hard
- Unsupervised Motion Disambiguation via Semantic Scene Priorsπ^4: A Foundation Model for 4D Scene Geometry and Motion🟠Medium
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generationâ›”Very hard
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representationâ›”Very hard