Action-Conditioned Counterfactual Synthesis for Embodied Physical Planning
Synthesize action-conditioned counterfactual video outcomes in robotic physics environments to train video LLMs on physical causality and trajectory feasibility.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section B (Counterfactual data) and Section 4, counterfactual generation relies on deterministic pixel transformations (e.g., reversal, speed changes, or heuristic synthetic simulator edits in CLEVRER) without modeling interactive, action-conditioned motor dynamics or physical feedback.
Limitation
The offline pixel-level counterfactual generator cannot produce action-conditioned physical counterfactuals needed for embodied robot manipulation and causal action planning.
Proposed Solution
Develop an extensible action-conditioned counterfactual generator using a physics simulator (e.g., RoboSuite or ManiSkill). For a given robot intervention, execute paired action sequences—such as apply target force vs. opposite force, or execute trajectory A vs. perturbed trajectory B—and render paired dynamic videos with auto-generated counterfactual QA labels regarding trajectory success, contact dynamics, and obstacle collision.
Why This Matters
Extends non-annotated counterfactual supervision from simple video frame transformations to complex physical causation and robotics planning tasks.
How to Get Started
How to test it: Evaluate whether action-conditioned counterfactual training improves embodied planning accuracy on standard manipulation benchmarks over baseline V-JEPA-LM. Baseline: Standard V-JEPA-LM 2B. Metric: Pair accuracy on physical action outcome predictions. Stopping result: Stop if synthesized physical counterfactuals fail to outperform standard counterfactual video training by at least 3 points on action QA.
- Script a simulator pipeline in ManiSkill/RoboSuite to render paired robot manipulation clips with perturbed control actions.
- Format the generated paired trajectories into counterfactual multiple-choice QA items balancing option lengths.
- Train V-JEPA-LM on the synthetic physical action dataset mixture.
- Evaluate model performance on physical dynamics and robot manipulation benchmarks (e.g., RoboVQA or EgoDex).
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like 'DreamTrue' and 'Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis' investigate action-conditioned counterfactual generation and failure synthesis for robotics. However, they leave open the specific formulation of rendering paired simulator trajectories with counterfactual QA labels for contact dynamics and force perturbations.
- DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training · 2026 · 1 citations
- Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis · 2026 · 15 citations
We searched OpenAlex and arXiv for “action conditioned counterfactual video generation robotic planning VLM” and “Action-Conditioned Counterfactual Synthesis for Embodied Physical Planning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned RepresentationsKeuntae Kim, Dongjun Kim, Yong Suk Choi · 2026
This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.
Related Ideas
- Force-Aware Policy for Contact-Rich InteractionsSimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation🟡Fairly easy
- Affordance-Grounded Execution Filters to Bridge the Text-to-Action Refusal GapEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- Physics-Constrained Reward Modeling for RLHFHunyuanVideo 1.5 Technical Report🟠Medium
- Optical Flow-Guided Physics RegularizationYume-1.5: A Text-Controlled Interactive World Generation Model🟡Fairly easy
- Test-Time Planning via Mental SimulationGigaBrain-0: A World Model-Powered Vision-Language-Action Model🟠Medium