Video2World: Transforming Embodied Video into Executable Physics Simulators via Agentic Code Synthesis

The central bottleneck of physical AI is no longer compute allocation; it is environment diversity. While foundation models for natural language scale on internet-scraped text tokens, foundation policies for robotics require physically grounded, interactive environments where agents can fail millions of times without breaking real-world hardware.

Until now, the industry has faced a severe dichotomy. On one side lies manual digital twin engineering: teams of technical artists and robotics engineers writing Unified Robot Description Formats (URDFs), tuning joint damping, and calibrating contact friction in engines like MuJoCo or NVIDIA Isaac Sim. This process costs between $4,500 and $12,000 per complex interaction scene and requires 30 to 60 engineering hours. On the other side sits neural world modeling—generative video models that predict future visual frames. While visually rich, neural video generation suffers from physics hallucination, non-deterministic object permanence, and inference latency exceeding 220 milliseconds per frame, rendering it mathematically incompatible with high-frequency (500 Hz) reinforcement learning (RL) control loops.

The arrival of Video2World (arXiv:2610.04432) fundamentally disrupts this trade-off. Rather than treating world modeling as an end-to-end pixel generation problem, Video2World reframes it as an autonomous software engineering challenge: deploying multimodal coding agents to observe real-world manipulation videos and iteratively write, debug, and calibrate executable simulation software.

The Video-to-Code Mechanism

The operational premise of Video2World rests on programmatic abstraction. Real-world dynamics are governed by immutable mathematical laws: kinematics, friction cones, mass distributions, and restitution coefficients. When an agent expresses an environment as Python or XML code targeting a deterministic physics engine, it imports the entire rigor of classical mechanics for free.

The Video2World framework decomposes the generation pipeline into four tightly coupled autonomous agent phases:

1. Spatial-Temporal Primitive Decomposition

The agent ingests raw RGB-D or monocular video feeds from an embodied run (e.g., an operator opening an industrial valve or an end-effector assembling a PCB). Rather than predicting subsequent frames, a multimodal perception agent extracts:

• Object segmentation masks and 3D bounding primitives (bounding boxes, capsules, and meshes).
• Dynamic interaction graphs, identifying kinematic parent-child hierarchies (e.g., drawer sliding within a cabinet housing).
• Observed trajectory splines across discrete time horizons.

2. Symbolic Code Generation

The system queries a frontier reasoning model tasked with synthesizing native simulation code (such as PyBullet scripts or MuJoCo MJCF XML schemas). The agent instantiates 3D geometries, defines joint axes (prismatic vs. revolute), sets joint limits, and assigns initial physical parameter priors (mass, lateral friction, rolling friction, and damping).

3. Execution-Based Visual-Physics Feedback Loops

No zero-shot LLM output can accurately deduce unobservable physical parameters, such as a hinge’s internal spring constant, from pure visual inspection. Video2World addresses this through iterative execution feedback. The generated simulation is executed headlessly, applying the estimated actuation forces to mirror the physical demonstration. The simulator logs synthetic trajectory rollouts (T_sim).

4. Residual Optimization and Self-Debugging

The agent calculates trajectory residual errors across the discrete observation horizon:

L_trajectory = Σ ||x_observed(t) - x_sim(t)||²
(Mean squared Euclidean deviation between observed physical video keypoints and simulated rollouts)

When visual drift occurs or simulation execution raises parsing exceptions (e.g., unconstrained mesh penetration or kinematic singularity), the compiler returns the stack trace and state discrepancy to the coding agent. Through targeted code modification, the agent adjusts parameters such as contact stiffness and mass offsets until the trajectory error falls below an empirical convergence threshold.

Empirical Benchmarks and Unit Economics

The paper’s benchmark results present clear commercial and architectural trade-offs:

1. Parameter Calibration Accuracy: Baseline zero-shot frontier LLMs generate non-functional or dynamically invalid code on 68.4% of complex embodied trials. With Video2World’s closed-loop execution and visual feedback, the success rate for syntax and kinematic validity rises to 91.2%, while kinematic parameter alignment error drops from 34.2% to just 4.8% within five feedback iterations.
2. Computational Throughput: Neural video diffusion architectures require roughly 240 ms per frame on an NVIDIA H100 SXM5, topping out at approximately 4.2 frames per second. Once Video2World outputs deterministic C++/CUDA simulation code, the resulting environment executes at over 45,000 simulation steps per second on standard CPU/GPU topologies. This represents a 10,700× throughput advantage for training downstream robotic policies.
3. Unit Cost Per Environment: Traditional manual scene modeling costs approximately $8,400 across mechanical design, sensor calibration, and asset tuning. Video2World generates an interactive, fully calibrated environment at an average consumption of 1.4 million input tokens and 85,000 output tokens. At prevailing enterprise frontier API pricing, this equates to roughly $14.20 per verified environment—a greater than 99.8