Video-to-Simulation: Coding Agents Turn Passive Observation into Interactive Physics

The central bottleneck in training embodied foundation models is not compute, nor is it the sample efficiency of reinforcement learning algorithms. It is the supply of interactive physical environments.

While static 3D reconstruction pipelines—such as 3D Gaussian Splatting and NeRFs—have industrialized the capture of visual geometry, they yield inert scenes. They lack kinematics, mass distributions, friction coefficients, collision meshes, and actuation dynamics. Historically, converting real-world operational environments into interactive simulation formats (URDF, MJCF, or USD) has required human simulation engineers spending 30 to 50 hours per scene at a fully burdened cost exceeding $110 per hour.

The Video2World framework (arXiv:2610.04432) reframes this entire paradigm. Rather than treating real-to-sim as a pure computer vision reconstruction problem or relying on end-to-end pixel-predictive world models, it models the challenge as an autonomous software engineering problem. Frontier multimodal coding agents observe passive, uncurated embodied video, deduce spatial semantics and kinematic hierarchies, and write deterministic, executable simulation code that runs natively in modern physics engines like MuJoCo and Isaac Sim.

The Structural Limits of Generative Video

Over the past eighteen months, industrial labs poured billions of dollars into diffusion-based generative video models, betting that passive video generation could serve as a surrogate world simulator. That hypothesis has encountered severe structural friction:

1. Non-Markovian Drift: Generative video suffers from cumulative error drift. In long-horizon manipulation sequences, objects mutate, mass is violated, and geometric consistency deteriorates within 120 frames.
2. Compute-Intensive Inference: Running forward rollouts through a multi-billion-parameter diffusion transformer at test time introduces an inference latency penalty of 850ms to 2,400ms per step, rendering it unusable for high-frequency low-level motor control loops (operating at 50Hz to 200Hz).
3. No Explicit Action Conditioning: Pixel-based world models output pixels, not actionable joint torques, spatial forces, or deterministic contact states.

The alternative—manual programmatic simulation—delivers deterministic physical fidelity and operates at over 100,000 frames per second on accelerated hardware. However, it scales strictly linearly with human engineering headcounts. Video2World breaks this trade-off by placing multimodal coding agents directly between passive raw video and the physics compiler.

Mechanics of the Autonomous Video-to-Sim Task

The benchmark formalizes the generation of world models as an iterative software engineering workflow composed of three explicit technical stages.

1. Spatial-Temporal Grounding and Kinematic Parsing

The agent ingests raw video clips depicting tools, articulable objects (e.g., cabinets, valves, robotic arms), and spatial interactions. Rather than generating an unconstrained 3D point cloud, the agent extracts discrete mechanical primitives:

• Topology identification: Determining whether an interaction constitutes a revolute joint, prismatic joint, or free-body contact.
• Bounding volumetric envelopes: Extracting rough bounding geometries that match physics primitives (boxes, cylinders, spheres) to prevent severe computational bottlenecks associated with raw triangular mesh collisions.

2. Parametric Simulation Code Generation

The system queries frontier foundation models acting as autonomous software engineers. The models do not generate geometric assets directly; they output domain-specific code. This includes configuring:

• Rigid-body mass matrices and inertial tensors.
• Joint damping parameters, limits, and actuator force constraints.
• Contact friction margins (tangential, rolling, and torsional friction coefficients).

By constraining the model output to declarative or procedural code (e.g., Python scripts using Isaac Lab APIs or raw XML for MuJoCo), the system introduces structural guardrails. Syntax errors, missing parent bodies, or floating objects are intercepted at compile time rather than contaminating policy training.

3. Closed-Loop Execution and Self-Correction

Crucially, Video2World evaluates coding agents in an execution-feedback environment, mirroring modern SWE-bench architectures. If the physics simulator throws a segmentation fault, detects non-physical penetrations during step zero, or causes an instability explosion (NaN accelerations), the error trace is fed back into the agent’s context window.

The empirical findings from arXiv:2610.04432 confirm that iterative agent loops dramatically outperform single-pass prompt strategies:

• First-pass script compilation rate (Pass@1) for frontier LLMs hovers at 41.2%.
• When equipped with execution-feedback loops over three iterations, the success rate rises to 73.6%.
• Kinematic alignment error decreases by 38.4% across common household and industrial manipulation objects once compiler-level error tracing is provided.

The Unit Economics of Physical Data Engines

For enterprise operators and robotics executives, the significance of programmatic video-to-simulation lies in the transformation of unit economics.

Currently, industrial robotic deployments are gated by the custom engineering costs of edge installations. Commissioning an autonomous bin-picking, assembly, or warehousing agent requires accurate digital twins of the deployment site.

Consider the operational economics of generating 1,000 distinct operational scenes:

• Manual Pipeline: 1,000 scenes × 35 hours per scene = 35,000 engineering hours. At an amortized rate of $110/hour, the cost is $3.85 million, with a delivery cycle measured in quarters.
• Autonomous Video2World Pipeline: 1,000 scenes synthesized via multimodal agent calls. Average prompt and reasoning tokens: 350,000 tokens per scene across three self-correction cycles. At prevailing API rates, compute cost is roughly $3.20 per verified scene. Total direct compute cost: $3,200, executable in under 72 hours of parallelized API runtime.

This capital efficiency alters the scaling laws of physical intelligence. An operator with a single smartphone can record a walk-through of an unfamiliar manufacturing cell. Within an hour, an autonomous agent pipeline translates that footage into thousands of parallel, randomized, executable digital twins ready for reinforcement learning and policy distillation.

The Epistemic Risk: Latent Physics and Model Governance

Despite these efficiencies, programmatic world modeling introduces systemic risks centered on non-obvious hallucination.

When a vision-