Without explicit planning, generated motion can be visually plausible but unsafe, jerky, or inefficient. OptiWorld first plans an optimized trajectory, then uses that trajectory as generation guidance.
Video generation models are becoming a scalable form of world models, but they mainly generate plausible motion rather than proactively control or optimize the underlying dynamics. As a result, an object in the generated video may follow trajectories that are unsafe, not smooth, inefficient, or physically inconsistent. In this work, we propose OptiWorld, a framework that brings classical optimal control into video generation at inference time. OptiWorld first extracts a compact, task-relevant world state, then plans an optimal trajectory under physical constraints, and finally renders the video conditioned on this trajectory. We formulate planning as a geometric problem on a continuous manifold, which converts 3D geometry and task-dependent physical constraints into a unified planning geometry. By adding this optimal-control layer, OptiWorld generates videos with preferable dynamics across goal-conditioned image-to-video generation, video dynamics editing, and counterfactual generation.
Understanding
Builds a compact 3D world state from geometry, segmentation, VLM reasoning, and 3D tracks when editing a source video.
Planning
Converts 3D geometry and task constraints into a continuous planning geometry where hazards, goals, smoothness, and efficiency can be optimized together.
Generation
Renders video from the first frame, prompt, and optimized 3D trajectory using a controllable video generator.
Open-world video generation mixes geometry, intent, and physical constraints. A unified manifold gives these signals a common space, so planning can reason over them as one geometry rather than many disconnected rules.
A shared space
Different constraints become comparable only after they are placed on the same planning surface.
Geometry as judgment
The manifold expresses what is costly, reachable, or preferred through distance and direction.
One path, many needs
A single trajectory can balance task intent, physical structure, and generative guidance before rendering.
Click a case to compare OptiWorld with image-to-video and physics-aware baselines.
| Method | Goal err. ↓ | Viol.@succ. ↓ | Accel. ↓ | Jerk ↓ | Energy ↓ | Mot. smooth ↑ | BG cons. ↑ | Flicker ↑ |
|---|---|---|---|---|---|---|---|---|
| HunyuanVideo-1.5 | 0.548 | 0.523 | 0.0118 | 0.0197 | 0.0576 | 0.996 | 0.961 | 0.995 |
| Wan2.2 | 0.549 | 0.523 | 0.0172 | 0.0304 | 0.0860 | 0.993 | 0.958 | 0.990 |
| Cosmos-Predict2.5 | 0.733 | 0.286 | 0.0104 | 0.0186 | 0.0398 | 0.994 | 0.936 | 0.985 |
| VLIPP | 0.331 | 0.511 | 0.0099 | 0.0176 | 0.0419 | 0.994 | 0.964 | 0.992 |
| OptiWorld | 0.310 | 0.260 | 0.0070 | 0.0116 | 0.0126 | 0.997 | 0.982 | 0.996 |
Each case shows the planned trajectory and the generated video rendered from it.
OptiWorld refines source 3D tracks before rendering, producing shorter, smoother, and more physically reasonable motion.
The track visualizations compare the source motion with OptiWorld's optimized motion. Optimized Path Comparison overlays one tracked point before and after optimization.
Overlay of a single point trajectory before and after optimization.
| Method | Track dev. ↓ | Accel. ↓ | Jerk ↓ | Path len. ↓ | Energy ↓ | Mot. smooth ↑ | BG cons. ↑ | Flicker ↑ |
|---|---|---|---|---|---|---|---|---|
| Source video | - | 0.0124 | 0.0205 | 0.647 | 0.0269 | 0.993 | 0.969 | 0.988 |
| Wan2.1 FLF2V | 0.123 | 0.0157 | 0.0266 | 0.733 | 0.0387 | 0.995 | 0.957 | 0.993 |
| OptiWorld | 0.113 | 0.0107 | 0.0181 | 0.524 | 0.0236 | 0.996 | 0.967 | 0.992 |
OptiWorld can change goals or constraints before rendering, producing controlled physical variations from the same scene.
For the same scene, changing the target goal produces a different optimized trajectory while keeping the scene understanding and renderer fixed.
Plan A
Plan B
Plan A
Plan B
Plan A
Plan B
@article{Yuan_2026_OptiWorld,
title={{OptiWorld}: Optimal Control for Video World Generation under Physical Constraints},
author={Yuan, Yu and Yuan, Jianhao and Wang, Xijun and Li, Daiqing and He, Liu and Ling, Lu and Chan, Stanley H.},
journal={arXiv preprint arXiv:2606.00499},
year={2026}
}