Generating Synthetic Data with a World Foundation Model

Whether for reinforcement learning or perception, robot AI is starved for data. Collecting diverse real-world scenarios with an actual robot is slow, expensive, and hazardous. The emerging solution is the World Foundation Model (WFM) — a physics-aware video generation model that manufactures synthetic data at scale.

The Data Flywheel

The core idea is a virtuous cycle where each iteration enriches the data supply.

Rendering diagram…

Simulation provides structural ground truth — depth maps, segmentation masks, LiDAR sweeps, poses, trajectories — for free. The problem is that simulation imagery looks slightly artificial: the domain gap. WFMs close that gap.

NVIDIA Cosmos — Video Generation That Understands Physics

Cosmos, released by NVIDIA in 2025, is a leading WFM (open license, commercially usable). It comprises three components:

Cosmos-Transfer is the flywheel's engine — it takes the structural GT from simulation and converts it into photorealistic imagery. The same scene can be varied across thousands of lighting conditions, weather states, and material properties, all with free labels inherited from the simulation.

💡 "Free labels" is the key insight — Annotating real footage by hand is grueling. Because simulated scenes carry per-pixel depth, class, and coordinates by construction, photorealistic transformations via WFM preserve those labels intact. WFM supplies realism; simulation supplies labels — a clean division of labor.

Our Place — Two Paths to Realism

There are two routes to photorealistic data. One is reconstructing a real scene with Gaussian Splatting; the other is generation via WFM. Reconstruction gives you "exactly what's there"; generation gives you "variations that don't exist yet." Using both together combines the fidelity of reconstruction with the diversity of generation.

⚠️ Always validate generated data — Even if WFM-generated imagery looks physically plausible, it can be subtly wrong (impossible shadows, misaligned geometry). Models trained on synthetic data must be cross-validated against real data or physics-simulation ground truth. Generation is powerful — blind trust is not.

In a nutshell

📌 World Foundation Models (NVIDIA Cosmos: Predict / Transfer / Reason) convert simulation's structural GT into photorealistic video, mass-producing synthetic data. The data flywheel — WFM for realism, simulation for labels — addresses the data-starvation problem. Generated data must still be cross-validated against real measurements.