World models for robots — Gokul Puthumanaillam

Start here: a robot that imagines before it moves

I’ll start with a distinction: a network can produce an action directly from what the robot sees. That is a policy: a reflex, however sophisticated. A world model is a different kind of object. It answers “if I do this, what happens?” Given the current situation and a candidate action, it predicts the next situation. Ask it again with the predicted situation and the next action and you have imagined two steps ahead. Keep going and you have imagined a whole future, without moving a motor.

Why would a robot want that? Because real experience is the most expensive thing in robotics. A real trial takes real seconds, can break real objects, and gives one data point. Imagined trials cost milliseconds, break nothing, and can be run by the thousand. If the model is good, the robot can rehearse a plan before committing to it, practice a skill in its head, generate training data it never collected, or check a new policy for accidents before anyone stands near the arm. If the model is bad, all of that is confidently wrong. I want to keep both possibilities in view: what the model makes possible, and what happens when it is wrong.

Plan in imagination, then act in the world

Start at the bottom of the table, with the goal at the top, the cup in between, and a muddy patch on the left where the robot bogs down and crawls. The robot is a point with momentum: each step its velocity decays a little and the push you command adds to it. Nobody typed those rules into the world model. A small network was trained offline on 20,000 transitions of the robot wandering at random, and its job is to predict the next state from the current one and a push. The planner has only that network and a cost that says “be near the goal, stay away from the cup”. Press Plan to run the cross-entropy method inside the model: 200 candidate futures are imagined, the best 20 are kept and the search narrows, six times over. Then press Execute as planned to run the winning plan blindly in the real world, or Replan every step to re-run the planner after each real step.

planning horizon H (steps)

best imagined cost, per CEM round–

imagined final distance to goal–

real final distance to goal–

real run touched the cup–

planning time–

Here are three things I want to draw attention to. First, the planner found a route around the cup without anyone programming a route: it imagined hundreds of futures and kept the ones that scored well. Second, everything the planner knows about the robot’s momentum, its drag and the mud came from random bumping around; nobody wrote down a single equation of motion for it (the mud it learned only roughly, as chapter 6 will measure). Third, the blind execution often ends somewhere else than the imagined path did, and sometimes touches the cup the plan carefully skirted. The model is wrong by a few percent per step, and the planner, being an optimizer, picks precisely the plan that lives at the edge of what the model says is safe. Replanning after every real step fixes it: reality corrects the model’s drift before it accumulates. I’ll return to that pattern throughout the discussion.

Next: what world models are for

1. What a world model is for, and the three families

I find it useful to strip away the architectures first. Every world model is the same object: a learned function from a state and an action to the next state, often with a prediction of reward or of whether the episode continues. Written p(s_t+1 | s_t, a_t), it is a probability rather than a point because the world is not fully predictable from what a camera shows: the box may or may not tip, the pedestrian may or may not step out. What differs between families is what the “state” s is made of, and that single choice decides almost everything else.

The loop every world model lives in, and the three families. They differ in what they predict: a compact latent state trained with actions and rewards; the next frame, learned from raw video; or an abstract representation, with no decoder at all.

Four jobs

I’ll work through each in chapter 5 with math and a demonstration. For now, here is the map.

Planning. Imagine many candidate action sequences, score them, execute the best first action, repeat. This is model-predictive control, and it is what the opening demo did. PlaNet, TD-MPC2 and V-JEPA 2-AC plan this way.
Learning in imagination. Instead of planning at run time, train a policy on imagined rollouts and deploy the fast policy. Dreamer does this; DayDreamer did it on physical robots; Dreamer 4 did it in Minecraft from a fixed dataset, without touching the game.
Synthetic data. A generative video model that follows instructions can film the robot doing tasks it never did, in rooms it never saw. Label those videos with inferred actions and you have demonstrations for free. That is DreamGen’s recipe and much of what Cosmos is sold for.
Offline evaluation. Run a new policy inside the model and count successes and accidents before touching hardware. WorldEval, WorldGym and DeepMind’s Veo-based evaluation of Gemini Robotics policies all report strong correlation with real success rates.

Two words that will keep coming back

A model is action-conditioned if the action is one of its inputs. Only an action-conditioned model can answer “what if I did something else”, which is what planning needs. Much of the video on the internet, and much of VaViM’s training data, comes without actions; such models are action-free: they predict what will probably happen, not what would happen if. Genie’s trick, and one of Dreamer 4’s and V-JEPA 2’s, is to learn most of the model from action-free video and add the action input from a small amount of labelled data. Chapter 3 shows how actions can be recovered from video that never recorded them.

Why not predict raw observations directly?

You could define the state as the camera image itself and learn p(image_t+1 | image_t, a_t). The generative video family does essentially that, through a tokenizer. The cost is that a 512 × 288 frame is 442,000 numbers and a robot arm’s relevant state is perhaps twenty. Predicting every pixel means predicting the flicker of a reflection and the exact texture of a carpet, which is expensive and mostly irrelevant to grasping. The latent-state and representation families answer by predicting something smaller: a vector that the model itself decides to keep. The question of which vector, and what forces it to keep the right information, is the disagreement between chapters 2 and 4, and the subject of the debate in chapter 4.

Stochastic or deterministic?

The world is only partly predictable from images. A deterministic model forced to predict an ambiguous future produces the average of the possibilities: a blurred frame, or a latent state halfway between “the cup tips” and “the cup stays”. Every serious world model therefore has a stochastic part: sampled latent variables in the RSSM, diffusion or flow sampling in the video models, and a latent variable z in the JEPA formulation. The same multimodality problem arises in action generation; here, it appears in the dynamics.

Model-based versus model-free, in one sentence

Model-free reinforcement learning learns a policy or a value function from experience directly, and each real step is used once. Model-based learning spends the same experience training a model, then extracts many imagined steps from it. The exchange rate, how many imagined steps are worth one real step, is the model’s accuracy, and it decays with the length of the imagined rollout. Chapter 6 quantifies that decay.

Next: Dreamer and the RSSM

2. Dreamer: a latent state, and a policy learned inside it

The Dreamer line (PlaNet 2018, Dreamer 2019, DreamerV2 2020, DreamerV3 2023, Dreamer 4 2025, all led by Danijar Hafner) is the most complete answer to “learn a model, then learn to act inside it”. Its model keeps a small internal state and predicts how that state moves. Pixels are decoded from the state only to train it; once trained, the agent imagines in the compact state, thousands of futures at a time, and trains a policy on them. DayDreamer put the same recipe on physical robots in 2022: a quadruped learned to roll over, stand and walk in about an hour of real experience, because most of its practice happened in imagination.

The recurrent state-space model

To understand how Dreamer works, I’ll focus on the RSSM. Its state at time t has two parts. A deterministic part h_t, produced by a recurrent network from the previous state and action, is the model’s memory. A stochastic part z_t carries what the memory cannot determine: which way the coin fell, whether the cup tipped. Two networks propose z_t. The prior guesses it from h_t alone: what do I expect to see? The posterior also looks at the actual observation x_t: what did I see? When the robot is acting in the world, it uses the posterior. When it imagines, there is no observation, so it uses the prior. Training pulls the two together, and that is what makes imagination trustworthy.

One step of the RSSM. Memory h_t is deterministic; z_t is sampled, from the prior when imagining and from the posterior when an observation is available. The KL term trains the prior to predict what the posterior will conclude.

Training is one loss with four terms, summed over the time steps of recorded sequences:

The first term is reconstruction: the state must carry enough to redraw the frame. This is the term chapter 4 will argue about.
The reward and continue terms make the state carry what matters for the task: they are small, but they are the reason the imagined futures can be scored.
The KL term is the one to understand. With z_t drawn from the posterior during training, the prior is being taught to predict, from memory alone, the same distribution the posterior reaches after seeing the frame. That is exactly the skill imagination needs.
This is the evidence lower bound (ELBO), unrolled in time: the posterior q is the encoder, the prior is now learned and conditioned on the past, and the negative of this loss is a lower bound on the log-likelihood of the whole sequence.

An RSSM you can compute exactly

Take the simplest possible world: a cart on a rail, pushed by a fixed pattern of forces, with drag. Its state is position and velocity; the camera measures position only, with noise; velocity is hidden. For this linear-Gaussian world, the RSSM’s two distributions have closed forms and are called the Kalman filter: the prior is the predicted belief before the next measurement, the posterior is the belief after it, and their KL divergence is a formula. Observations arrive up to the step you choose; after that the filter can only imagine, chaining priors. Watch the band widen. Then give the prior the wrong drag and watch the KL, the quantity the RSSM’s loss minimizes, refuse to go to zero.

observe until step 20

measurement noise σ

prior believes the wrong drag (0.8 instead of 0.9)

KL(posterior ‖ prior), mean over observed steps–

posterior std of position at the last observation–

imagined std of position 10 steps later–

true velocity was never observedhidden state

Everything the RSSM does, this filter does with a formula: the memory carries the belief forward, a measurement sharpens it, and without measurements the belief widens step by step. The network version learns the drag, the noise levels and the observation model from data, in a nonlinear, high-dimensional space; the KL term is the signal that teaches it, and it gets large exactly when the model’s expectation was wrong.

Learning in imagination

With the model trained, Dreamer never plans at run time. It trains two more networks entirely on imagined data: an actor that maps a latent state to an action, and a critic that estimates how much reward lies ahead of a state. The procedure: take latent states from real, recorded experience; from each, roll the model forward for H steps (fifteen, in every Dreamer) with actions from the actor and states from the prior; collect the predicted rewards; and improve the actor so that those imagined returns go up. In the original Dreamer the improvement is a gradient computed straight through the model’s dynamics, from the imagined rewards back to the actor’s parameters; the model is differentiable, so imagination is not only a simulator but a teacher that says in which direction to change the action.

Learning in imagination. From real latent states, short imagined rollouts are scored with predicted rewards and the critic’s value at the horizon; the actor is updated to raise that score. No real steps are spent.

Train a policy without touching the robot

The tabletop again, with the same frozen dynamics network as the opening demo. A small policy network receives the robot’s state and outputs a push. Each update imagines 32 rollouts of H steps through the model, sums the known cost along them (distance to the goal, plus the cup penalty), and updates the policy by the exact gradient of that imagined cost through the model, Dreamer-style. Where should the imagined rollouts start? Dreamer starts them from states in its replay buffer, spread over everything the robot has experienced. Try both choices, then test the policy on the real dynamics: twenty runs from the bottom of the table.

imagination starts from

imagination horizon H

updates so far0

imagined cost per step–

real successes (reached, no contact)–

real runs that touched the cup–

real steps used for training0

Two lessons hide in that demo. Starting imagination only at the task’s first frame gives a policy that pushes straight at the goal, gets stuck against the cup’s cost wall and never learns to go around: the gradient at the wall points sideways, but the states to the side were never imagined from, so the policy has nothing to say there. Starting from states all over the table gives a policy that knows what to do everywhere, and the route around the cup assembles itself from those pieces. That is why Dreamer starts imagination from replay-buffer states, and it is a small instance of a big fact: a policy learned in imagination is only as good as the coverage of the states it imagined from. The second lesson is the counter at the bottom of the stats: zero real steps were spent on learning. The twenty test runs are the only real ones, and they are only there so that you can check.

The RSSM in equations

h_t = f_θ(h_t−1, z_t−1, a_t−1)deterministic memory, a GRU

ẑ_t ~ p_θ(z_t | h_t)prior: used in imagination

z_t ~ q_φ(z_t | h_t, x_t)posterior: used with real observations

x̂_t ~ p_θ(x_t | h_t, z_t), r̂_t ~ p_θ(r_t | h_t, z_t), ĉ_t ~ p_θ(c_t | h_t, z_t)decoders: frame, reward, continue flag

PlaNet and Dreamer used a diagonal Gaussian for z. DreamerV2 switched to 32 categorical variables with 32 classes each, sampled with straight-through gradients, and found it both more stable and better at representing sudden changes; DreamerV3 kept that choice. Every distribution is a small network head on h_t (and the encoded frame, for the posterior).

Why the loss is a sequential ELBO

Write the joint model of a sequence as p(x_1:T, z_1:T | a) = Π_t p(x_t | h_t, z_t) p(z_t | h_t) and the approximate posterior as Π_t q(z_t | h_t, x_t). Jensen’s inequality gives log p(x_1:T | a) ≥ Σ_t E_q[log p(x_t | h_t, z_t)] − KL(q(z_t | h_t, x_t) ‖ p(z_t | h_t)). The difference from the VAE is that the prior is no longer a fixed N(0, I) but a learned, history-dependent distribution. The KL is therefore not a pressure toward zero, as it was for the VAE; it is a pressure toward agreement between expectation and observation, and it can be driven small only by the prior learning to predict.

Imagination, step by step

Dreamer’s outer loop

Collect a little real experience with the current actor (plus exploration noise); store it.
Train the world model on sampled sequences with the four-term loss.
Encode those sequences with the posterior to get starting latent states.
From each start, imagine H = 15 steps: a from the actor, h from the GRU, z from the prior, r̂ and ĉ from the heads.
Compute λ-returns along each imagined rollout using r̂, ĉ and the critic’s value at the horizon.
Update the critic toward those returns; update the actor to increase them.
Repeat. Real steps are spent only in step 1, and each is reused for many imagined steps.

The λ-return blends multi-step imagined rewards with the critic: V^λ_t = r̂_t + γ ĉ_t [(1−λ) v(s_t+1) + λ V^λ_t+1], ending with V^λ_H = v(s_H). It lets a fifteen-step imagination stand in for an unbounded future, and it is why the horizon can stay short: beyond H, the critic takes over, and short horizons are where the model is still accurate.

DreamerV3 and DayDreamer

DreamerV3 (2023; Nature, 2025) is notable for a negative claim: one set of hyperparameters across more than 150 tasks with images, proprioception, discrete and continuous actions, and it was the first agent to collect diamonds in Minecraft from scratch. The tricks that made a single configuration possible are robustness devices rather than new ideas: a symlog transform of inputs and targets, a two-hot encoding for the critic, return normalization by percentiles, KL balancing, and free bits. DayDreamer (2022) ran the earlier recipe on hardware: a Unitree A1 quadruped learned to walk from scratch in about an hour of real interaction, and arms learned pick-and-place from images without simulators, because the ratio of imagined to real steps was in the hundreds.

Dreamer 4: the RSSM becomes a transformer

Dreamer 4 (Hafner, Yan and Lillicrap, 2025) keeps the philosophy and replaces the machinery. A tokenizer compresses frames; a transformer dynamics model predicts the next latent frame, action-conditioned; and it is trained with shortcut forcing, a flow-matching-style objective arranged so that a frame can be generated in very few steps. The result is a world model of Minecraft, learned from 2,541 hours of gameplay video, that runs in real time on one GPU and predicts object interactions well enough to be played by humans. Its agent trains in imagination inside that model and was the first to obtain diamonds purely from a fixed offline dataset, without ever interacting with the game during training. Two details matter for robotics: the same imagination-training recipe transferred from a recurrent latent to a large video model, and the model learned most of what it knows from unlabelled video, needing only a small labelled portion to become action-conditioned.

KL balancing and free bits

The KL term has two learners: the posterior, which could cheat by ignoring the observation so as to match the prior, and the prior, which should do the work of predicting. DreamerV3 splits the term into two copies with stopped gradients: a dynamics loss KL(sg(q) ‖ p) with weight 0.5 that trains only the prior, and a representation loss KL(q ‖ sg(p)) with weight 0.1 that trains only the posterior. The asymmetry keeps the prior chasing the posterior rather than the reverse. Both are clipped from below at one nat, the free bits trick, so that the model stops squeezing the latent once agreement is good enough and cannot collapse it.

Actor gradients: through the model or not

Dreamer and DreamerV2 trained continuous-control actors with gradients through the model’s dynamics, the route the demo above takes. It is efficient but biased when the model is wrong and unusable for discrete actions. DreamerV3 uses the REINFORCE estimator with the imagined returns as the signal and the critic as a baseline, plus an entropy bonus, which works for every action type and is what lets one configuration serve all tasks. The returns are normalized by an exponential moving average of their 5th-to-95th percentile range, so that sparse-reward and dense-reward tasks see gradients of the same size.

The Kalman filter as the linear special case

For the cart, write z_t = Az_t−1 + Ba_t−1 + w with w ~ N(0, Q) and a measurement x_t = Cz_t + v with v ~ N(0, σ²). If the belief at t−1 is N(μ, Σ), the prior at t is N(Aμ + Ba, AΣA^T + Q) and the posterior is the prior corrected by the innovation x_t − C(Aμ + Ba) with the Kalman gain K = Σ⁻C^T(CΣ⁻C^T + σ²)⁻¹. The KL between two Gaussians is ½[tr(Σ₀⁻¹Σ₁) + (μ₁−μ₀)^TΣ₀⁻¹(μ₁−μ₀) − 2 + ln(det Σ₀ / det Σ₁)]. The demo evaluates exactly these; an RSSM learns A, B, Q and the observation model as networks, and its GRU is the nonlinear stand-in for Aμ + Ba.

Next: generative video world models

3. Generative video world models: predict the next frame

Next, I’ll turn to models that take the state to be the frame itself, or a compressed version of it, and learn to generate what the camera will see next. The appeal is the data. Every hour of driving footage, every robot video, every game recording on the internet is a training example for “what happens next”, with no actions, rewards or labels required. The recipe is: a tokenizer turns each frame into a grid of discrete tokens or a compact continuous latent; a sequence model, autoregressive or diffusion-based, predicts the next frame’s tokens from the past frames (and, when available, from actions or text); a decoder turns predicted tokens back into pixels. The output is a video you can watch, which is both the family’s greatest strength and, as chapter 4 argues, its most contested feature.

The generative video recipe. Tokenize, predict the next frame’s tokens given the past and whatever conditioning exists, decode. The two branches differ in the sequence model: token-by-token autoregression or diffusion in a latent space.

How actions get in

A frame predictor becomes a world model the moment the robot’s action is one of its inputs. There are three ways to get there, and the families split along them.

Recorded actions. GAIA-1 (Wayve, 2023) trained on 4,700 hours of its own driving with the steering and speed logged, so its 9-billion-parameter model takes video, text and action tokens and predicts the next frame’s tokens; a separate video-diffusion decoder renders them. GAIA-2 (2025) moved to a latent diffusion model in a continuous space and conditions on ego speed and curvature, the 3D boxes of other agents, weather, time of day and road layout, across multiple cameras, so that a rare scenario can be requested rather than waited for.
Inferred latent actions. Genie (DeepMind, 2024) trained on 200,000 hours of internet gameplay video with no controller logs at all. A latent action model looks at consecutive frames and assigns a code from a tiny vocabulary (eight, in Genie) to the change between them; the dynamics model learns to predict the next frame from past frames and that code; and after training, the codes are a controller. The demo below builds one from scratch.
No actions, then a separate action module. VaViM predicts driving video with no actions at all. It cannot be planned with, so VaVAM bolts an action expert onto its representation and imitates human drivers instead. The paper’s conclusion names what would be needed to close the loop: a reward model over its latent states.

Discover actions from video that never recorded them

A six-by-six grid with one ball, filmed for 600 short clips while someone pressed random keys: up, down, left, right, or nothing. The key presses were not recorded, only the frames. Genie’s idea in miniature: look at every pair of consecutive frames, describe the change between them by a vector (here, exactly, the ball’s displacement), and quantize those vectors with a codebook of K entries using a VQ step. Each code is a latent action. Then fit a next-frame table: from (frame token, latent action) to the next frame token, by counting. Press the discovered codes to play the world. With K = 5 the codes should sort themselves into the five keys; with K = 3 they cannot, and with K = 8 some are wasted. Also watch the counter of “unseen” situations: when a (frame, code) pair never occurred in the clips, the table has no entry and the world invents one.

codebook size K

Discovered latent actions (press one to step the imagined world; the arrow is the code’s mean displacement):

frame pairs used–

codes that are used (share of pairs)–

imagined steps taken0

steps where the table had no entry (invented)0

Real latent action models are learned networks rather than a displacement vector and k-means, and the frames are token grids rather than one ball, but the logic is the same: the change between frames is compressed into a small discrete vocabulary, and because the same code appears whenever the same kind of change happens, the codes end up meaning “left”, “jump” or “brake” without anyone saying so. Genie’s follow-ups added text as a second control channel; Genie 3’s “promptable world events” (“make it rain”) are conditioning tokens of exactly this kind.

The systems, briefly

Genie 1, 2, 3. Genie 1 (February 2024) was an 11-billion-parameter model of 2D platformer video, with a spatiotemporal tokenizer, the latent action model above and a MaskGIT-style dynamics model; a single image could be turned into a playable level. Genie 2 (December 2024) generated 3D worlds from a single image, playable for tens of seconds. Genie 3 (August 2025) generates worlds from a text prompt, renders them at 720p and 24 frames per second in real time, keeps them consistent for several minutes, remembers what was off-screen for about a minute, and accepts promptable world events. DeepMind uses it to train and test its SIMA agents; its stated limits are a narrow action space (the agent moves, the text does the rest), weak modeling of other agents, and minutes rather than hours of coherence. No architecture or size has been published.

GAIA-1 and GAIA-2. Wayve’s models are world models in the strict sense: action-conditioned, with the ego vehicle’s controls as inputs, built to generate the scenarios a fleet rarely meets. GAIA-1 tokenized 288 × 512 frames into 18 × 32 grids, the same layout VaViM later used, and predicted tokens autoregressively; GAIA-2 replaced the token transformer with latent diffusion trained by flow matching, added multi-camera consistency and structured controls (agents, weather, road semantics), and is used both to make training data and to stress-test the driving model in loops it never faced on the road.

Cosmos. NVIDIA’s Cosmos (January 2025) is a platform rather than a model: video tokenizers, a family of diffusion and autoregressive predictors trained on 20 million hours of robotics and driving video, a “Transfer” model that repaints simulator renders (depth, segmentation) into photoreal video for sim-to-real, and a “Reason” vision-language model for annotation and judging. Cosmos-Predict 2.5 (late 2025) unified text-, image- and video-to-world generation in one flow-based model at 2 and 14 billion parameters. Cosmos 3 (June 2026) merged the pieces into a single mixture-of-transformers model: an autoregressive reasoner interprets the scene, a diffusion generator conditioned on it produces video and, for the first time natively, actions, in three modes: forward dynamics (actions in, video out), inverse dynamics (video in, actions out) and joint video-and-action prediction, which is a policy. Cosmos is what most groups mean when they say they use a world model to make synthetic robot data.

The game engines. DIAMOND (2024) modeled Atari with a diffusion world model and showed that visual detail matters for the policy trained inside it; GameNGen ran Doom in real time from a diffusion model; Oasis did Minecraft. Dreamer 4, from chapter 2, is the state of the art of this line for control, because its world model is also the place its agent learns.

VaViM and VaVAM, read as a world model

I’ll look at VaViM, an autoregressive token model, through the question its paper asks. Large video generators produce plausible futures, so they must have learned something about the world. But what, and is it useful for driving? The authors trained an open, reproducible video model on 1,800 hours of driving, most of it unlabelled dash-cam video from YouTube, then tested what its representation knows and whether it transfers to control.

Design. A frozen LlamaGen VQGAN with a 16,384-entry codebook turns each 512 × 288 frame into 576 tokens; eight frames at 2 Hz make a 4,608-token context; a GPT-2-style causal transformer with 24 layers, at 185 million, 318 million and 1.2 billion parameters, predicts the next token with cross-entropy, using shared spatial position embeddings and per-frame temporal ones, following GAIA-1. Maximal-update parameterization let the hyperparameters found on a 60-million-parameter model transfer to the large one, and a fitted scaling law predicted the large model’s loss to within 0.06%, while saying the model was under-trained: more data would help more than more parameters. The largest run took about 4,800 H100-hours.
What the representation knows. Zero-shot segmentation probes and PCA visualizations show the features group pedestrians, cars, road and markings, on cities the model never saw, although below dedicated self-supervised encoders like DINOv2. Two findings bear directly on chapter 4: the largest model generated the best video, as measured by FID, and yet did worse on segmentation than the smaller ones; and the fine-tuned models kept the pre-trained representation’s quality. Better generation did not mean better understanding.
What transfers to driving. VaVAM adds the π0-style flow-matching action expert and reaches state of the art on the frontal scenarios of the NeuroNCAP closed-loop benchmark with a single front camera and no annotations, cutting collisions by 27% relative to a fully annotated end-to-end planner at comparable progress. But scaling the model or the data, which improved video quality and open-loop trajectory error monotonically, did not improve closed-loop safety monotonically; the large models followed the guiding path more tightly and collided more, a conflict the authors attribute to imitation learning: the same expert trajectories are training data and guide, so the model never sees the expert deviate to avoid a hazard.
What is missing. The conclusion is explicit. VaViM can generate plausible futures, but VaVAM does not use them: it imitates. To become a planning world model in the sense I’m using here it would need an action input and a reward or value model over its latent states, so that imagined futures can be compared. The two additions I’ve emphasized for a video model, action conditioning (Genie’s latent actions, GAIA’s logged controls, Dreamer 4’s small labelled set) and a way to score futures (Dreamer’s reward head, V-JEPA 2’s goal energy), are exactly the two the paper lists as future work.

Autoregressive tokens or latent diffusion?

Both branches predict the next frame in a compressed space; they differ in how they sample it. The autoregressive branch (GAIA-1, VaViM, IRIS) draws the frame’s tokens one at a time from softmax tables, which gives an exact likelihood, easy scaling laws and a simple training loop, but 576 sequential decodes per frame and a resolution ceiling set by the tokenizer. The diffusion branch (GAIA-2, Cosmos, DIAMOND, Dreamer 4) treats the frame’s latent as one continuous vector and denoises it in a few steps, conditioned on the past; it parallelizes across the frame, degrades gracefully, and can be steered by classifier-free guidance, at the cost of the iterative sampling loop. Dreamer 4’s shortcut forcing exists to make that loop short enough for real time. Masked models (Genie 1’s MaskGIT dynamics, MAGVIT) sit between: discrete tokens, but many predicted in parallel per pass.

Why the tokenizer is a ceiling

Whatever the sequence model learns, it cannot see what the tokenizer threw away. VaViM’s authors list, as a limitation and a direction, a tokenizer that keeps text on signs, road markings and traffic lights, which the general-purpose LlamaGen codebook blurs. For a world model that is also a safety question: a predictor that cannot represent a red light cannot predict what a car at a red light will do. Cosmos ships its own tokenizers for the same reason.

Evaluating a video world model

FID and FVD measure whether generated frames look like real ones in the feature space of a classifier. They say nothing about whether a rolling ball stops where it should. Physics-IQ (2025) and similar benchmarks test exactly that, and generative video models that score well on appearance often fail them. VaViM’s paper makes a second, sharper point about evaluation: open-loop trajectory error correlates loosely with closed-loop driving, and a closed-loop safety score can be gamed, as their accidental model that drove off the road and never met a hazard demonstrated; hence their added deviation and progress metrics. Chapter 5 returns to evaluation when world models are the evaluators rather than the evaluated.

Next: JEPA, and the argument about pixels

4. JEPA: predict the representation, not the pixels

To introduce the third family, I’ll start with its objection to predicting pixels. Yann LeCun’s 2022 position paper, A Path Towards Autonomous Machine Intelligence, argues that a model of the world should not be asked to generate the world. Most of what a frame contains is unpredictable in detail (the exact shape of a wave, the individual leaves in wind, the sensor noise) and irrelevant to acting. A model trained to reproduce pixels must spend its capacity on precisely those details, because they dominate the pixel loss, and it is punished for every one it gets wrong even when it got the situation right. His proposal, the joint-embedding predictive architecture, encodes both the present and the future, and trains a predictor to map one representation to the other. Nothing is drawn. The encoder is free to discard whatever cannot be predicted, and prediction error in this space is what the paper calls energy: low for futures that fit, high for futures that do not. Planning becomes energy minimization over actions, which is exactly what V-JEPA 2-AC does on a real arm.

Generative prediction versus JEPA. Left: the loss lives in pixel space and every unpredictable detail counts against the model. Right: both frames are encoded and the loss lives in representation space; a regularizer or a slowly moving target encoder prevents the trivial solution of encoding everything to the same vector.

The obvious danger is that the trivial solution wins: if every frame is encoded to the same vector, prediction is perfect and useless. That is collapse, and all the engineering in the JEPA line is about preventing it without contrastive negatives: an explicit regularizer that demands variance in every dimension and decorrelation between them (VICReg), or, in I-JEPA and V-JEPA, a target encoder that is an exponential moving average of the trained one, with no gradient flowing into it, plus masking so that the predictor has to fill in what it cannot see.

From I-JEPA to V-JEPA 2-AC

I-JEPA (2023) masks blocks of an image and predicts the target encoder’s representations of the masked blocks from the visible context, with no pixel loss and no hand-designed augmentations. V-JEPA (2024) did the same over space-time blocks of video and showed that frozen features from feature prediction beat frozen features from pixel reconstruction on motion-centric tasks.
V-JEPA 2 (June 2025) scaled the recipe to a billion-parameter ViT-g encoder trained on over a million hours of internet video and a million images, with an L1 “mask denoising” loss against the EMA target and 3D rotary position embeddings. Its features set the state of the art on motion understanding (Something-Something v2) and action anticipation (Epic-Kitchens-100), and a language model aligned to them matched larger systems on video question answering. None of that involved generating a frame.
V-JEPA 2-AC is the world model. With the encoder frozen, a 300-million-parameter action-conditioned predictor was post-trained on under 62 hours of unlabelled robot video from the Droid dataset (the arm’s own motions serve as the action), and then deployed, without any data from the new labs, on Franka arms to reach, grasp and pick-and-place objects it had never seen. The goal is an image. Planning minimizes the energy: the L1 distance between the predicted representation after a candidate action sequence and the goal image’s representation, optimized with the cross-entropy method, one executed action at a time. Against an action-conditioned Cosmos model planning in the same loop on the same GPU, V-JEPA 2-AC took 16 seconds per action with 800 samples where Cosmos took four minutes with 80, and it succeeded more often, at 100% on reaching and 65 to 80% on pick-and-place. The paper is also plain about its limits: sensitivity to the camera’s position (the model infers the action axes visually), horizons kept to a single step because error accumulates, and goals that must be images.

Predict pixels or predict the representation? A toy where you can see what each keeps

A synthetic “video”: 28 pixels per frame. Hidden inside is a two-dimensional state that rotates deterministically, the analogue of an object’s position: perfectly predictable, but small (variance 0.3 per dimension). Also mixed into every pixel is a 24-dimensional “texture” that fades and renews itself at random, like leaves in wind: only 64% of its variance is predictable, but it is big (variance 1 per dimension). A fixed random rotation mixes both into the pixels, so nothing is readable off a single coordinate. Two models train side by side, each with a two-dimensional bottleneck h. The pixel model predicts the next frame’s 28 pixels by squared error. The JEPA encodes both frames and predicts the next representation from the current one, with a variance-and-covariance regularizer against collapse. Afterwards a linear probe measures how much of the hidden state, and how much of the texture, each h retains. Then try the two switches.

JEPA regularizer on (uncheck to watch collapse) add a “clock”: a perfectly predictable but useless rotating pair of pixels

steps0

pixel model: state kept (probe R²)–

JEPA: state kept–

pixel model: texture kept–

JEPA: texture kept–

clock kept (pixel / JEPA)–

JEPA representation std (collapse → 0)–

With the regularizer on and no clock, the outcome is LeCun’s argument in miniature. The pixel model’s two-dimensional bottleneck fills up with texture, because texture is where the pixel variance is; the hidden state, small and quiet, is thrown away, and the probe finds almost none of it. The JEPA keeps the state almost perfectly and ignores the texture, because in representation space the only thing that lowers the loss is predictability, and the state is the most predictable thing in the frame. Switch the regularizer off and the JEPA collapses within a few hundred steps: its representation’s standard deviation heads to zero, the prediction loss to zero, and the probe to nothing. Now add the clock. Both models take it. The JEPA drops the state for a rotating pair of numbers that predicts itself perfectly and means nothing, because its objective rewards predictability and only predictability; the pixel model takes it because it is large and predictable. Neither knows what matters. Something outside the frames has to say.

Both sides of the argument

This is a live disagreement among serious people, and the honest summary is that both sides have evidence.

	The case for predicting representations	The case for generating
Capacity	Pixel losses are dominated by unpredictable detail; the toy above shows a small state lost under big texture. Frozen V-JEPA features beat frozen reconstruction features on motion tasks; VaViM’s largest model made better video and worse segmentation.	Capacity turned out to be affordable. Genie 3 keeps a world consistent for minutes; Dreamer 4 predicts Minecraft mechanics in real time; Cosmos and GAIA-2 ship as products. DreamerV3 reconstructs pixels and still trained 150 tasks with one configuration.
What gets learned	The encoder drops the unpredictable and keeps the predictable, an abstraction chosen by the objective rather than by hand. Planning in that space is fast: 16 seconds per action against four minutes.	Predictable is not relevant. The clock in the toy shows a JEPA taking a useless signal over the useful one; static backgrounds and camera artifacts are real-world clocks. Something task-related (actions, rewards, goals) has to shape the representation, and once it does, the argument is about targets, not about pixels.
Verifiability	Energy, a scalar, says how surprising a future is; goals can be specified as images without hand-written rewards; V-JEPA 2-AC ran zero-shot in new labs.	A generated video can be watched, judged by a human or a vision-language model, used for training data and for policy evaluation. A representation can only be probed. When V-JEPA 2-AC drifts, you learn it from the robot.
Training	No decoder, no discriminator, no diffusion loop; one encoder and one predictor, trained on a million hours of unlabelled video.	Collapse must be prevented by regularizers or EMA targets, and these are delicate; a generative loss cannot collapse. Autoregressive pretraining from video also gives competitive representations when scaled.
Where they meet	Almost nobody predicts raw pixels any more. GAIA-2, Cosmos and Dreamer 4 predict in a tokenizer’s latent; the disagreement is whether the target of prediction should be defined by a decoder that redraws the frame or by an encoder that only has to be predictable. TD-MPC2 predicts a latent whose only targets are reward, value and the next latent, a task-shaped JEPA; DreamerV3 keeps the decoder and calls it a regularizer; V-JEPA 2-AC drops it and pays with image goals and short horizons.

The JEPA objective, with its regularizer

L = E[ ‖ P_θ(E_φ(x_t), a_t, z) − sg(E_φ̄(x_t+1)) ‖ ] + λ_v Σ_j max(0, 1 − std_j(E_φ(x))) + λ_c Σ_i≠j Cov_ij(E_φ(x))²

The first term is the prediction error in representation space; V-JEPA uses an L1 norm. E_φ̄ is the target encoder: in the paper’s framework a copy of E_φ, in I-JEPA and V-JEPA an exponential moving average of it, with a stop-gradient (sg) so that the target does not move to meet the prediction.
z is a latent variable that carries whatever the future contains that the present cannot predict; it must be regularized (kept low-information) or it will explain everything. The demo omits it because its world is deterministic.
The variance term pushes every dimension of the representation to have standard deviation at least one; the covariance term pushes dimensions to be decorrelated. Together they make the constant solution expensive. This is VICReg (Bardes, Ponce and LeCun, 2022), and it is the regularizer the demo uses. The EMA target achieves the same end implicitly.

Why the pixel loss chases variance

A squared error over pixels is a sum over directions of the frame; a direction with variance v and predictability ρ² contributes ρ²v of reducible loss. A bottleneck has room for a few directions, so it takes those with the largest ρ²v: big, mostly predictable things, whether or not they matter. In the demo each texture direction offers 0.64 × 1 = 0.64, each state direction offers 1 × 0.3 = 0.3, and the bottleneck fills with texture. In representation space the variance is normalized away by the regularizer, so only ρ² counts, and the state, with ρ² = 1, wins. That is the precise content of “generative models waste capacity on unpredictable detail”, and its precise limit: the JEPA ranks by predictability alone, so a perfectly predictable nuisance ranks equal to a perfectly predictable state.

How V-JEPA 2-AC plans

Goal-image planning in representation space

Encode the current frame and the goal image with the frozen encoder: s₀ and s_g.
Sample N candidate action sequences a_1:T from a Gaussian (800 samples in the paper).
For each, roll the action-conditioned predictor forward from s₀ to get ŝ_T; the energy is ‖ŝ_T − s_g‖₁.
Keep the lowest-energy sequences, refit the Gaussian, repeat (the cross-entropy method of chapter 5).
Execute the first action only; observe; return to step 1.

The energy is defined in a space where “the block is at the target” is a small distance and “the block is elsewhere” is a large one, without any reward being written. That is the payoff of the whole architecture. The cost is that the space only knows what it was trained to predict: the paper reports success falling with camera displacement, since the predictor’s idea of “left” was learned from a particular viewpoint.

DINO-WM, and the frozen-encoder shortcut

A cheaper way to get a JEPA-style world model is to freeze a strong pretrained encoder and learn only the predictor on top of its features. DINO-WM (2024) did this with DINOv2 patch features and planned by model-predictive control from goal images across several manipulation and navigation tasks, with no reconstruction and no reward. V-JEPA 2-AC uses the same shortcut: the encoder comes from the million-hour pretraining and never changes; the 62 hours of robot video only teach the predictor what actions do. The shortcut is why so little robot data suffices, and also why the camera matters so much.

Energy-based models and the “regularized” family

The 2022 paper frames all of this as energy-based learning: an energy function F(x, y) should be low for compatible pairs and high elsewhere. Contrastive methods raise the energy of sampled negatives, which in high dimensions means an unmanageable number of them. Regularized methods instead limit the volume of low-energy space, by constraining the information content of the representation and the latent z; VICReg, Barlow Twins and the EMA-target methods are of this kind. In a JEPA, F is the prediction error minimized over z, and planning, as in V-JEPA 2-AC, is F minimized over actions. Hierarchical JEPAs, with predictors at several time scales, are the paper’s proposal for long-horizon planning; nothing at scale has yet demonstrated them.

A fair statement of the strongest objection

The strongest objection to the pure representation view is not that it fails, since V-JEPA 2-AC works, but that its successes lean on things outside the argument: a goal image (which is pixels), single-step horizons (which sidestep the predictor’s drift), and a frozen encoder whose quality came from a masking objective rather than from world modeling. Meanwhile the generative side has produced the things robots are actually trained on today: Cosmos and DreamGen data, WorldEval-style evaluation, Dreamer 4’s agent. The strongest objection to the generative view is symmetric: none of its impressive videos has been shown to contain a better decision than a smaller representation would, VaViM’s own results being a case in point, and the physics benchmarks say that looking right is not the same as being right. Both sides agree on more than the debate suggests: predict in a learned latent, condition on actions, and let something task-related shape what the latent keeps.

Next: what a robot does with a world model

5. What a robot does with a world model

I’ll now work through the four jobs, each with its own math, its own champion systems and its own way of going wrong. The demos use the tabletop world and its frozen dynamics network throughout, so the same imperfect model is doing all four.

Planning: model-predictive control

Given a model and a cost, planning is optimization: find the action sequence whose imagined future costs least, execute its first step, observe, and do it again. The re-planning is not decoration. It is what turns an open-loop plan, which drifts with the model’s error, into a feedback controller that corrects itself against reality every step. The optimizer can be anything that works on sequences; robotics mostly uses sampling methods, because they need no gradients and parallelize across a GPU.

Model-predictive control with a sampling planner. Steps 1 to 4 happen inside the model; only step 5 touches the world, and it commits to a single action before everything is recomputed.

The cross-entropy method (CEM)

Start with a Gaussian over action sequences: mean μ_1:H (zeros, or last step’s plan shifted by one), standard deviation σ.
Sample N sequences; roll each through the model from the current state; sum the cost c(s_t) along the way, weighting the last state more if you want to arrive.
Select the K lowest-cost sequences (the elites); set μ and σ to their per-step mean and standard deviation.
Repeat steps 2 and 3 for a handful of rounds; return the best sequence seen.

MPPI, the softer cousin

Same samples, but weight every sequence by w_n ∝ exp(−J_n / λ) instead of keeping only elites, and set μ to the weighted mean. λ trades exploration for greed. TD-MPC2 plans this way, with a learned value function scoring the end of each rollout so that a short horizon suffices.

The opening demo is this loop, with N = 200, K = 20 and six rounds. PlaNet planned in the RSSM’s latent with CEM; TD-MPC2 plans in a latent trained only for reward and value, with MPPI and a terminal value; V-JEPA 2-AC and DINO-WM plan in a frozen encoder’s space with an image goal. What they share is the structure above, and what they share as a weakness is the cost of step 2: N × H model calls per real step. That is why V-JEPA 2-AC needs 16 seconds per action and Cosmos four minutes, and why a 50 Hz arm is still driven by a policy, not a planner.

Policy learning in imagination

Chapter 2 did this in full; the summary is the trade. Planning spends compute at run time and can adapt to any new cost immediately. A policy trained in imagination spends the compute once and runs in a single forward pass, but knows only the task it was trained for. Dreamer, DayDreamer and Dreamer 4 take the second route; TD-MPC2 does both, training a policy in imagination and using it to propose the samples for its planner. In all of them the imagined rollouts are short (fifteen steps or fewer), start from real states, and hand the rest of the future to a critic.

Synthetic data

A generative world model that follows instructions is a film studio. DreamGen (NVIDIA, 2025) fine-tunes an image-to-video model (Cosmos or a similar backbone) on a robot’s own teleoperation videos, then prompts it with a first frame and a new instruction, in a new room, to produce videos of the robot doing things it never did. The videos contain no actions, so a second model supplies them: an inverse dynamics model, or a latent action model as in Genie, labels each frame pair with a plausible action. The result, which the authors call neural trajectories, trains an ordinary policy. From teleoperation data of a single pick-and-place task in one room, a humanoid learned 22 new behaviors in seen and unseen environments, and average success on the real robots rose, for example, from 37% to 46% on the GR1 humanoid tasks and from 21% to 46% on a small arm, with only ten to thirteen real trajectories per task. GAIA-2 plays the same role for driving, generating the near-misses a fleet meets too rarely, and Cosmos Transfer’s repainting of simulator output is synthetic data of a third kind: real physics from the simulator, real appearance from the model.

Neural trajectories. The world model supplies appearance and variety; the inverse dynamics model supplies actions; the policy never knows the difference. Neither does it know when the generated physics was wrong.

Offline policy evaluation

The quietest use may be the most valuable. Evaluating a manipulation policy for real means hundreds of trials, resets, broken objects and a person watching; comparing five checkpoints means five times that. A world model can run the trials. WorldEval (2025) conditions a video model on the policy’s own latent action representation, generates the rollout, asks a vision-language model whether the task succeeded, and reports success rates that correlate strongly with real ones across policies and across checkpoints of one policy, and that also flag dangerous actions before deployment. WorldGym (2025) does the same with an autoregressive action-conditioned video model from a single starting frame, preserving the relative ranking of RT-1-X, Octo and OpenVLA with mean success rates within a few percent of reality. DeepMind evaluated Gemini Robotics policies in a Veo-based simulator; DreamDojo (2026) reports a rank correlation near one. The standard metrics are the Pearson correlation between simulated and real success and the mean maximum rank violation, borrowed from SIMPLER: how badly the proxy misorders policies. The demo below runs this evaluation in the tabletop world, where the real answer is known, so that you can see when the proxy lies.

Grade five policies without running the robot

Five hand-written controllers for the tabletop: drive straight at the goal; detour to the right; detour to the left through the mud; hug the cup closely on the right; and a slow, cautious right detour that never arrives in time. Each is run twenty times from the bottom of the table, first inside a world model (imagined) and then in the real dynamics (truth). A run counts as a success if it ends near the goal without touching the cup; inside the model, “touching” means the imagined path entered the cup, which the checker sees even though the model does not. Choose which model does the grading.

world model used as the evaluator

policy	imag. success	real success	imag. contact	real contact
press Evaluate

rank agreement (Kendall τ between imagined and real success)–

largest error in a success rate–

With the well-trained model the ranking comes out right and the numbers are close, including for the left detour through the mud, which the model learned, roughly, from data. Switch to the model that only ever saw the left half of the table and the right-hand policies are condemned: in that model the right half is a hallucinated wall, so every right detour stalls or “collides”, and the best policy in reality is ranked at the bottom. The scarce-data model is wrong about everything at once. The lesson generalizes: a world model grades reliably only inside the distribution it was trained on, and a policy that does something the model never saw will be graded by fiction.

Planning as optimization, written out

a*_1:H = argmin E_{p_θ} [ Σ_t=1^H c(ŝ_t, a_t) + V(ŝ_H) ], ŝ_t+1 ~ p_θ(· | ŝ_t, a_t)

The terminal value V is optional; without it the horizon must be long enough to reach the goal, and every extra step costs model accuracy. TD-MPC2 learns V and keeps H at three to five; PlaNet used no value and horizons of twelve; the opening demo uses fifteen and a terminal weight. In a goal-image formulation the cost is the energy ‖ŝ_H − s_g‖ and the value is absent.

Why sampling planners and not gradients

The model is differentiable, so one could descend the cost with respect to the actions directly. In practice sampling wins for two reasons: contact-rich costs are full of plateaus and cliffs where gradients are useless, and gradients through a learned model point into its errors, the exploitation problem of chapter 6. Sampling only ever evaluates the model at actions that were actually proposed, which keeps the search in the model’s comfortable range if σ is not too large. The gradient route survives in policy learning, where the many short rollouts average the errors out.

Pseudo-actions and their risk

The inverse dynamics model in the synthetic-data pipeline is trained on real (frame, frame, action) triples, so its labels are trustworthy only where generated videos resemble real ones. When the video model makes a physically wrong frame, say a gripper passing through a cup, the labeler still emits an action, and the policy learns to attempt the impossible. DreamGen mitigates by filtering with success and consistency checks and by mixing real demonstrations back in. This is also why Cosmos 3 folds inverse dynamics into the world model itself, so that video and action are predicted jointly.

Model-based evaluation, the classical version

Before video models, offline policy evaluation used learned dynamics models in the reinforcement-learning sense, and its failure was the same: the policy under test drives the model into states it never saw. MOPO and MOReL (2020) added a pessimism penalty proportional to model uncertainty, estimated by ensemble disagreement, so that the evaluation trusts the model only where it is confident. Chapter 6 shows ensemble disagreement in the tabletop world; the video-model evaluators have not yet adopted it, and their reported correlations come from policies and scenes close to the training distribution.

Next: how world models fail

6. How world models fail

I’ll focus on three failure modes that recur across all three families, and the discipline each calls for. Error compounds along a rollout, so keep rollouts short and replan. Models hallucinate where data is thin or physics is abrupt, so know where the model is confident and distrust it elsewhere. And imagination costs compute, sometimes more than the real thing, so pay for it once, where it is worth paying.

Compounding error

A one-step model with a small error becomes a many-step model with a large one, because each step starts from the previous step’s mistake. Suppose the model’s one-step prediction is off by at most ε, and the dynamics amplify a small difference in state by at most a factor L per step. Then the error after k steps obeys

e_k ≤ L e_k−1 + ε ⇒ e_k ≤ ε (1 + L + L² + … + L^k−1) = ε (L^k − 1) / (L − 1)

If L < 1, the dynamics forget (drag, friction, damping): errors stay bounded by ε / (1 − L) and grow roughly linearly at first. The tabletop robot is like this, which is why fifteen-step plans work at all.
If L > 1, the dynamics amplify (a balancing pole, a rolling ball at a ridge, anything unstable): errors grow exponentially and no accuracy is enough for long horizons. Contact is the everyday case: a millimeter decides whether the cup is hit, and after the hit nothing is as it was.
For learning, the damage is measured in return, not in state: the classic bound says the value of a policy evaluated in the model can differ from its true value by an amount that grows quadratically in the horizon and linearly in the one-step model error. Short horizons are not a heuristic; they are the bound talking.

Watch error compound, and watch an ensemble notice

Two hundred random action sequences are run through the real dynamics and through the learned models from the same start states; the curves show the mean position error of the imagined path after k steps. Three models: the one the whole page uses (20,000 transitions), one trained on 120 transitions, and one trained only on the left half of the table. The dashed curve is ensemble disagreement: the distance between the predictions of two well-trained models with different random seeds. Where the models have data, they agree and are right; where they lack it, they disagree, which is a usable warning. Choose where the rollouts start.

rollouts start

error after 1 step (main model)–

after 5 steps–

after 15 steps–

ensemble disagreement after 15 steps–

From random starts the main model is off by about a hundredth of the table after one step and about a fifth of it after fifteen: roughly linear growth, the signature of forgetting dynamics. Start next to the cup and the curve steepens, because contact is the one thing in this world the model never learned. Start on the right half with the left-half model and the error explodes while the two well-trained models keep agreeing, which is the important caveat about ensembles: disagreement detects missing data, not shared mistakes. Both good models learned the same ghost cup from the same too-rare collisions, so they agree, wrongly, at the cup.

Exploitation, and the disciplines that contain it

The opening demo showed the sharpest form of compounding error. An optimizer does not sample the model’s errors at random; it seeks them out, because a model error that makes a plan look cheaper is, to the optimizer, a discount. The best plan under a slightly wrong model is systematically the plan that leans on the wrongness. Every working system has a discipline against this.

Replan. Model-predictive control executes one action and re-plans from the real state; V-JEPA 2-AC plans a single step at a time for the same reason. The opening demo’s replanning button turns seven collisions in ten into none.
Keep rollouts short and start them from real states. Dreamer imagines fifteen steps from replay-buffer states; MBPO (2019) made this explicit as branched rollouts and derived the horizon from the model error. Chapter 2’s demo failed when the starts were too narrow and worked when they covered the table.
Be pessimistic where unsure. PETS (2018) planned with an ensemble’s predictive distribution; MOPO and MOReL penalize the reward by ensemble disagreement, so that exploitable regions look expensive. The demo above shows the signal such penalties use.
Leave margins. The planner’s cost keeps the robot a margin away from the cup; widening that margin buys safety against exactly the error the ensemble cannot see. Real systems add collision checkers, joint limits and safety filters outside the learned model for the same reason.

Hallucination

A generative model asked about a situation it has not seen produces something plausible-looking, because that is what it was trained to do. In a language model that is an invented citation; in a world model it is an invented physics. The tabletop world has two hallucinations you can visit: the ghost cup, learned from too few collisions, and the wall that the left-half model imagines across the right side of the table, which is nothing but the network extrapolating into a region with no data.

Two hallucinations

The teal path is the real robot; the violet path is what the chosen model imagines for the same pushes. In the first scenario the robot is pushed straight up into the cup: reality stops it; the model, which learned from 20,000 transitions of which 2.5% were collisions, drives through. In the second, the model that only saw the left half is asked about the right half. In the third, the robot is pushed across the mud, a smooth effect the model learned roughly, for comparison.

scenario

real final position–

imagined final position–

final error–

The large systems hallucinate in the same two ways, at scale. Rare events with abrupt physics are under-learned: generative video models pass hands through objects, let a dropped cup float, forget an occluded object. Physics benchmarks make this measurable, and the results say that photorealism and physical correctness are separate axes. Out-of-distribution requests are answered with confident fiction: a scene the model never saw is completed with the nearest scene it did, which is why WorldGym and WorldEval report their correlations for tasks near their training data and why Genie 3 lists geographic accuracy and multi-agent interaction among its limits. Tokenizers add a third, quieter hallucination: whatever they cannot encode, the model cannot predict, so VaViM’s futures have blurred signs and lights not because it predicted them wrongly but because it never saw them.

Compute

Imagination is cheap per step and expensive per decision. Planning needs N × H model calls per real action, learning in imagination needs millions of imagined steps, and a generative video model spends a diffusion loop on every one of them. The numbers below are the ones the systems themselves report.

system	cost that is reported	what it means for a robot
VaViM-L (1.2B)	About 4,800 H100-hours of training on 192 GPUs over roughly a day, for 60 million eight-frame windows; the scaling law says it is still under-trained.	Training a video world model is a cluster job; the data, not the parameters, was the binding constraint.
V-JEPA 2-AC vs Cosmos, planning	16 seconds per action with 800 CEM samples, against 4 minutes per action with 80 samples for the pixel-generating model, on one RTX 4090. A full pick-and-place at 4 minutes per action takes over an hour.	Planning in representation space is fifteen times cheaper and still far too slow for a control loop; both are offline-style planners.
Genie 3	720p at 24 frames per second, in real time, size undisclosed; consistency for minutes.	Real-time generation exists at data-center scale; nothing about the cost is public.
Dreamer 4 (2B)	Real-time interactive inference on a single GPU thanks to shortcut forcing; the agent trained entirely inside it, from 2,541 hours of video.	Few-step sampling is what makes a video model usable as an imagination engine rather than a demo.
DreamerV3 / DayDreamer	One GPU per run; DayDreamer’s quadruped walked after about an hour of real time, with the model and policy training alongside.	Latent models are cheap enough that imagination can run on the robot’s own hardware.

The pattern in the table is the tradeoff I’ve been building toward. A small latent model imagines fast and is used at the robot; a large video model imagines slowly and is used before deployment, to make data or grade policies; a representation model sits between, planning in seconds rather than minutes. None of them yet closes a 50 Hz loop by imagining. Policies do that, and world models are increasingly used to train and test those policies and supply them with data.

Where the quadratic horizon bound comes from

Let the model’s one-step transition differ from the truth by at most ε in total variation, uniformly over states and actions. Running a policy for H steps in the model, the distribution of the state at step t differs from the true one by at most tε (the errors add), and a bounded per-step reward turns that into a return error of order Σ_t tε r_max ≈ H²ε r_max / 2. This is the simulation lemma of model-based reinforcement learning, and MBPO’s analysis adds the policy shift to it; the conclusion in both is that the horizon over which imagined returns are trustworthy scales like 1/ε, which for realistic ε is a handful of steps. Dreamer’s fifteen and TD-MPC2’s three to five are that handful.

Teacher forcing and its cure

A one-step model is trained on real (s_t, a_t, s_t+1) triples: it always starts from a true state. At rollout time it starts from its own prediction, a state it may never have seen, and the mismatch is the compounding you watched. Cures: train on multi-step rollouts so that the model sees its own errors (scheduled sampling in sequence models; a rollout loss in V-JEPA 2-AC; Dreamer’s latent overshooting in its first version), or train the sampler to generate frames from noisy versions of its own outputs, which is one reading of diffusion forcing and of Dreamer 4’s shortcut forcing.

Why contact is the hardest case

Contact makes dynamics discontinuous and multi-valued in the observation: two states that look the same on camera (the gripper a millimeter above the cup, a millimeter into it) have entirely different futures. A smooth network approximating that averages across the discontinuity and produces the sticky ghost the demos show; a stochastic model can represent it but needs many examples on both sides of the boundary, and random exploration rarely supplies them. The practical answers are targeted data collection near contact, explicit contact models or simulators for the contact part, and, above all, replanning against the real world, which observes the contact instead of predicting it.

Next: the three families, side by side

7. Side by side

I’ll bring the three families back together here: one function, next state from state and action, built three ways and used four ways. The table collects the distinctions I’ve worked through.

	Latent dynamics (Dreamer)	Generative video (Genie, GAIA, Cosmos, VaViM)	Representation prediction (JEPA)
State	Compact learned latent, deterministic memory plus stochastic part.	The frame, as tokens or a tokenizer latent.	The output of an encoder; no decoder exists.
Training signal	Reconstruction, reward, continue, KL between prior and posterior: a sequential ELBO.	Next-token cross-entropy, or diffusion/flow loss in latent space.	Prediction error in representation space plus an anti-collapse regularizer or EMA target.
Actions	Recorded; required.	Recorded (GAIA), inferred latent actions (Genie), or absent (VaViM); Cosmos 3 and Dreamer 4 learn action conditioning from a small labelled subset.	Added by post-training a predictor on a little robot video (V-JEPA 2-AC).
How futures are scored	Learned reward head and critic.	External: a vision-language judge, a cost on decoded frames, or none.	Energy: distance to a goal image’s representation.
Main uses	Policy learning in imagination; planning (PlaNet, TD-MPC2).	Synthetic data, policy evaluation, interactive simulation; policy learning in Dreamer 4.	Zero-shot goal-reaching by MPC; representations for downstream models.
Strength	Fast imagination; sample-efficient; runs on the robot’s hardware.	Learns from any video; output you can watch and judge.	Ignores unpredictable detail; cheap planning; huge unlabelled pretraining.
Weak spot	Needs rewards; reconstruction spends capacity on pixels; short horizons only.	Slow; hallucinates rare physics; tokenizer ceiling; realism is not correctness.	Collapse; predictable is not relevant; nothing to look at; image goals; short horizons.
Robotics examples	DayDreamer, DreamerV3, TD-MPC2.	GAIA-2, Cosmos and DreamGen data, WorldEval/WorldGym evaluation, VaViM/VaVAM.	V-JEPA 2-AC, DINO-WM.

The originals

Ha and Schmidhuber, 2018. World Models. The name, and the first policy trained in a learned dream.
Hafner et al., 2018. Learning Latent Dynamics for Planning from Pixels (PlaNet); 2019, Dream to Control; 2020, Mastering Atari with Discrete World Models; 2023 / Nature 2025, Mastering Diverse Control Tasks through World Models (DreamerV3).
Wu, Escontrela, Hafner, Abbeel and Goldberg, 2022. DayDreamer: World Models for Physical Robot Learning.
Hafner, Yan and Lillicrap, 2025. Training Agents Inside of Scalable World Models (Dreamer 4).
Hansen, Su and Wang, 2022 and 2023. TD-MPC and TD-MPC2. Chua et al., 2018, PETS. Janner et al., 2019, MBPO. Yu et al., 2020, MOPO.
Bruce et al., 2024. Genie: Generative Interactive Environments. Google DeepMind, 2024 and 2025, Genie 2 and Genie 3 announcements.
Hu et al., 2023. GAIA-1: A Generative World Model for Autonomous Driving. Russell et al., 2025. GAIA-2.
NVIDIA, 2025. Cosmos World Foundation Model Platform for Physical AI; 2025, Cosmos-Predict2.5; 2026, Cosmos 3.
Bartoccioni et al., 2025. VaViM and VaVAM: Autonomous Driving through Video Generative Modeling.
Alonso et al., 2024. Diffusion for World Modeling: Visual Details Matter in Atari (DIAMOND). Valevski et al., 2024, GameNGen.
LeCun, 2022. A Path Towards Autonomous Machine Intelligence. Bardes, Ponce and LeCun, 2022, VICReg.
Assran et al., 2023. I-JEPA. Bardes et al., 2024. V-JEPA. Assran, Bardes et al., 2025. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.
Zhou, Pan, LeCun and Pinto, 2024. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning.
Jang et al., 2025. DreamGen: Unlocking Generalization in Robot Learning through Video World Models.
Li et al., 2025. WorldEval: World Model as Real-World Robot Policies Evaluator. Evaluating Robot Policies in a World Model (WorldGym), 2025. Motamed et al., 2025, Physics-IQ.

Glossary

World model: A learned model of how the world changes: from a state and an action to the next state, often with predicted reward and termination.
Policy: A function from observation to action; a reflex. World models are used to train, plan for, or evaluate policies.
Action-conditioned / action-free: Whether the model takes the robot’s action as input. Only action-conditioned models can answer “what if I did something else”.
RSSM: Recurrent state-space model: deterministic memory h_t plus stochastic latent z_t, with a learned prior and a posterior.
Prior / posterior (of the RSSM): The prior guesses z_t from memory alone and is used in imagination; the posterior also sees the observation and is used with real data; a KL term makes them agree.
Imagination: Rolling the model forward on its own predictions, without observations, to produce futures for planning or learning.
Learning in imagination: Training an actor and critic on imagined rollouts from real starting states, as Dreamer does.
λ-return: A blend of multi-step imagined rewards with the critic’s value, letting a short rollout stand in for the whole future.
Model-predictive control (MPC): Plan a sequence of actions against the model, execute the first, observe, re-plan.
Cross-entropy method (CEM) / MPPI: Sampling planners: draw action sequences, score them in the model, refit the sampling distribution to the best (CEM) or to an exponentially weighted average (MPPI).
Tokenizer: A learned compressor from frames to discrete tokens or a compact latent, and back; it sets the ceiling on what a video model can represent.
Latent action model: A network that assigns a discrete code to the change between consecutive frames, giving action-free video a controllable action vocabulary (Genie).
Inverse dynamics model: Infers the action between two observations; used to label generated videos with actions.
Shortcut forcing: Dreamer 4’s flow-matching-style objective with per-frame noise levels and step-size conditioning, enabling accurate few-step generation for real-time imagination.
JEPA: Joint-embedding predictive architecture: predict the representation of the future from the representation of the present; no decoder.
Collapse: The trivial JEPA solution in which all inputs map to one representation; prevented by variance/covariance regularizers or an EMA target encoder.
Energy: In the JEPA framing, the prediction error in representation space; planning minimizes it over actions.
Compounding error: The growth of prediction error along a rollout because each step starts from the last step’s mistake.
Model exploitation: An optimizer’s tendency to find plans that look good only because of model errors.
Ensemble disagreement: The spread of predictions across models trained from different seeds; a signal of missing data, not of shared mistakes.
Offline policy evaluation: Estimating a policy’s success without running it for real, here by running it inside a world model.
Neural trajectories: DreamGen’s term for generated videos labelled with inferred actions, used as training demonstrations.