High-Fidelity Initialization via Large Reconstruction Models
Replace monocular depth map reconstruction with a large reconstruction model outputting 3D Gaussian Splats, providing accurate geometric initialization for physics simulations.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.1 describes using monocular depth estimation (MoGE-2) and inpainting to create a point cloud representation. The paper's Limitations section explicitly admits this leads to inaccurate geometry and depth errors, which propagate into the physics simulation causing unnatural collisions or object distortions.
“Reconstructing 3D scenes can be inaccurate due to errors in depth estimation, leading to suboptimal simulation and video results.”
Limitation
The physics simulation is limited by the quality of the initial 3D reconstruction, which currently relies on prone-to-error depth maps and heuristics.
Proposed Solution
Replace the depth-map-based reconstruction with a Large Reconstruction Model (LRM) that outputs 3D Gaussian Splats. Integrate these splats directly into the physics solver (leveraging techniques like PhysGaussian), providing a more accurate, fully textured 3D initialization that reduces geometric artifacts and improves simulation fidelity.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Improving the geometric prior is the most direct way to increase the physical realism of the simulation, ensuring collisions and interactions behave predictably.
How to Get Started
How to test it: We hypothesize that replacing heuristic depth reconstruction with Large Reconstruction Models producing 3D Gaussian Splats will improve geometric fidelity and simulation stability compared to the baseline. If visual artifacts are not reduced in user studies, we will halt.
- Acquire public weights and inference code for a Large Reconstruction Model capable of outputting 3D Gaussian Splats.
- Integrate the Gaussian Splat representation into the PhysGaussian or equivalent physics solver pipeline.
- Benchmark the geometric accuracy and collision realism of simulated objects against the original depth-map method.
- Evaluate end-to-end video generation quality and simulation fidelity on standard object-interaction datasets.
Code linked to this paper on Hugging Face: github.com/liuwei283/RealWonder
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed surveys and works such as Avat3r: Large Animatable Gaussian Reconstruction Model for High-Fidelity 3D Head Avatars discuss Gaussian splatting and reconstruction models broadly. However, none of the listed papers combine Large Reconstruction Models with physics solvers like PhysGaussian for 3D initialization in the manner proposed.
- Avat3r: Large Animatable Gaussian Reconstruction Model for High-Fidelity 3D Head Avatars · 2025 · 67 citations
We searched OpenAlex and arXiv for “Replace monocular depth map reconstruction with a large reconstruction model outputting 3D Gaussian Splats, providing accurate geometric initialization for physics simulations.” and “High-Fidelity Initialization via Large Reconstruction Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
RealWonder: Real-Time Physical Action-Conditioned Video GenerationWei Liu, Ziyu Chen, Zizhang Li +3 more · 2026
RealWonder proposes a real-time system for generating videos conditioned on 3D physical actions (forces, gravity, collisions) from a single image. The method bridges the gap between abstract physical actions and visual generation by using a physics simulator to produce intermediate optical flow and coarse RGB previews, which then condition a distilled, fast-inference video diffusion model.
Related Ideas
- Scaling Physics Data Synthesis via Rasterized 3D Simulation EnginesInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🔴Hard
- Physics-Informed Diffusion Guidance for Data GenerationGigaBrain-0: A World Model-Powered Vision-Language-Action Model🔴Hard
- Depth-Confidence Gating for Non-Geometric DomainsNeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos🟢Easy
- Hybrid Implicit-Explicit Residual DecodingtttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction🟠Medium
- Adaptive Early-Stopping and Feed-Forward Alternatives for 3DGS OptimizationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🟠Medium