Scaling Physics Data Synthesis via Rasterized 3D Simulation Engines
Generate synthetic physics image editing pairs using scriptable 3D simulation engines and fine-tune an image-to-image translation adapter to bridge the sim-to-real gap.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
For physics-centric image editing data, the paper describes an automated pipeline on Page 18 that relies on converting textbook diagrams into SVG codes using Gemini. This allows for high-quality, resolution-independent generation of vector graphics by manipulating code.
Limitation
While SVG-based synthesis is highly cost-effective for idealized, textbook-style vector diagrams, it entirely fails to capture the noisy, continuous, and complex nature of real-world photographic physics scenarios (e.g., fluid dynamics, optical refraction in real environments, mechanical wear).
Proposed Solution
Develop a hybrid data engine utilizing a scriptable 3D physics simulation engine (such as Blender with fluid/rigid-body add-ons) to render photorealistic 'before and after' physics editing pairs. Augment this pipeline with a diffusion-based image-to-image translation adapter fine-tuned on real-world noisy physics experiments to bridge the sim-to-real domain gap.
Why This Matters
This expands the unified model's scientific reasoning and editing capabilities from clean textbook diagrams to messy, real-world scientific observations, significantly broadening its utility for experimental data analysis.
How to Get Started
How to test it: We hypothesize that a hybrid Blender-based physics data engine will improve physical realism in generated science images. We will evaluate on GenExam physics benchmarks and stop if the sim-to-real domain gap fails to close by at least 5% on accuracy metrics.
- Set up a Blender simulation pipeline using rigid-body and fluid add-ons to script automated 'before and after' physics renders.
- Export the rendered image pairs alongside ground-truth bounding box masks or difference maps.
- Fine-tune a baseline diffusion-based image-to-image adapter on the synthetic renderings and real-world noisy physics images.
- Evaluate the generalization performance of the resulting adapter on real-world physics editing benchmarks.
Code linked to this paper on Hugging Face: github.com/OpenGVLab/InternVL-U
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like RE 3 SIM and PhysAlign explore the use of simulation and 3D representations for generating synthetic data or bridging the sim-to-real gap. However, the specific combination of a scriptable physics simulation engine for physics editing pairs augmented with a fine-grained image-to-image diffusion adapter remains open.
- PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment · 2026 · 9 citations
- RE 3 SIM: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation · 2026 · 86 citations
We searched OpenAlex and arXiv for “Generate synthetic physics image editing pairs using scriptable 3D simulation engines and fine-tune an image-to-image translation adapter to bridge the sim-to-real gap.” and “Scaling Physics Data Synthesis via Rasterized 3D Simulation Engines” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and EditingChangyao Tian, Danni Yang, Guanzhou Chen +26 more · 2026
This paper introduces InternVL-U, a lightweight 4B-parameter unified multimodal model designed to integrate visual understanding, reasoning, generation, and editing. By combining an open-source MLLM backbone with a custom continuous visual generation head and leveraging a reasoning-centric Chain-of-Thought (CoT) data synthesis pipeline, the model bridges the gap between aesthetic image generation and high-level semantic intelligence.
Related Ideas
- Physics-Informed Diffusion Guidance for Data GenerationGigaBrain-0: A World Model-Powered Vision-Language-Action Model🔴Hard
- Democratizing Scientific Illustration: Fine-Tuning Open-Weight Models for Spatial Reasoning and SVG GenerationAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- High-Fidelity Initialization via Large Reconstruction ModelsRealWonder: Real-Time Physical Action-Conditioned Video Generation🟠Medium
- Hybrid Neuro-Symbolic Rendering for Statistical PlotsPaperBanana: Automating Academic Illustration for AI Scientists🟡Fairly easy
- End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic WorkflowsSage S2.1 Flash Technical Report🟠Medium