'Multiverse simulation engine' predicts every possible future to train humanoid robots and self-driving cars

LAS VEGAS — Researchers have built a new "multiverse simulation" platform that can generate massive amounts of data to train advanced self-learning robots powered by artificial intelligence (AI).

The suite of tools, dubbed "Cosmos," lets researchers create "world foundation models" — neural networks that simulate real-world environments and the laws of physics to predict realistic outcomes, according to Nvidia, which engineered the platform. These generative AI models can create synthetic data to train embodied or physical AI systems such as autonomous vehicles (AVs) or humanoid robots.

Training AI systems require massive amounts of data, but scientists estimate that we may run out of publicly available data by 2026. AI systems that interface with the real world, like robots, usually need real-world footage that is incredibly difficult to produce and costly to acquire.

But creating genuinely useful synthetic data is also challenging, and one study previously warned that using poorly construed synthetic data could break down into unintelligible nonsense. Cosmos is designed to resolve these issues by letting scientists rapidly generate a monumental amount of artificial video footage based on real-world physics.

Related: Robotics breakthroughs in 2024: Test your knowledge of the most exciting advancements in the field

"Today's humanoid developers have hundreds of human operators performing thousands of repetitive demonstrations just to teach a couple of skills," Rev Lebaredian, vice president of Omniverse and simulation technology at Nvidia, said at a virtual news conference Monday (Jan. 6) at CES 2025 in Las Vegas. "Today's AV developers need to drive millions of miles; even more resources-intensive is processing, filtering and labeling the thousands of petabytes of data captured, and physical testing is dangerous. Humanoid developers have a lot to lose when one robot prototype can cost hundreds of thousands of dollars."

Simulating the multiverse

A key component of this new platform is multiverse simulation, in which Cosmos combines with Nvidia's Omniverse software system to generate every possible future outcome in a specific scenario. This would then be fed into the training of a robot or self-driving car.

It uses diffusion models used in image generation — machine learning algorithms that generate data by adding "noise" (grainy specs) to a dataset and then learning to remove the noise — as well as autoregressive models, which are statistical models used to predict the next step in a process. Together, the platform can take in text, images or videos and then generate footage to predict what comes next in a particular scenario in real time.

"The ChatGPT moment for robotics is coming. Like large language models, world foundation models are fundamental to advancing robot and AV development, yet not all developers have the expertise and resources to train their own," Jensen Huang, founder and CEO of Nvidia, said in a statement. "We created Cosmos to democratize physical AI and put general robotics in reach of every developer."

The world foundation models created using Cosmos are also available under open-source licensing terms.