Notes / Robot Learning

Dreamer

Updated

Introduction

Dreamer learns a compact simulator from experience and trains a policy inside that simulator. The world model predicts the consequences of actions, the critic estimates future return from the resulting states, and the actor learns to choose actions that increase expected return. Training the actor and critic primarily on imagined trajectories can improve sample efficiency by reducing the amount of real environment interaction needed to learn a useful policy.

Although intuitively appealing, authors note that robustly learning and leveraging world models to achieve strong task performance has been an open problem and required a range of robustness techniques based on normalization, balancing, and transformations.

The three networks - the dynamics model, the critic, and the actor - are trained concurrently from replayed experience while the agent interacts with the environment. Authors note that in order to succeed across domains, all three components need to accomodate different signal magnitudes and robustly balance terms in their objectives; this is challenging because they want to achieve robust performance across diverse domains with fixed hyperparameters.

For Dreamer V3, which we focus on in this note, the world model is implemented as a Recurrent State-Space Model (RSSM). An encoder maps sensory inputs xtx_t to stochastic representations ztz_t. Then, a sequence model with recurrent state hth_t predicts the sequence of these representations given past actions at−1a_{t-1}.

The RSSM unrolled over three steps: an encoder maps each input x_t to a stochastic state z_t, a recurrent state h_t carries information forward using the action a_t, and a decoder reconstructs x_t from the model state

The model state is the concatenation of a deterministic recurrent state hth_t and a stochastic representation ztz_t. hth_t summarizes the past but cannot express uncertainty, while ztz_t is a compact code of the current observation. The encoder gives a posterior q(zt∣ht,xt)q(z_t \mid h_t, x_t) that captures the new information in the observation, and the dynamics predictor gives a prior p(z^t∣ht)p(\hat{z}_t \mid h_t) over possible next states.

Before Dreamer V1-V4, PlaNet used online trajectory optimization with a Gaussian RSSM. Dreamer V1 also used a Gaussian RSSM, Dreamer V2 and V3 used a categorical RSSM, and Dreamer V4 uses a transformer. Broadly, model learning learns to predict which observations and rewards are plausible given an action and observation history. Behaviour learning, meanwhile, learns which actions produce high return under a learned model. The two, while similar, have different problems - a model can predict well but yield a policy that chooses ineffective actions, while behaviour policies can learn to exploit errors in the model that don’t transfer into the real world. Unlike PlaNet, Dreamer consolidates the cost of trajectory optimization into a single network forward pass.

We use the following convention throughout: at time tt, the agent sees xtx_t, has just received reward rtr_t from action at−1a_{t-1}, forms a model state sts_t, and chooses ata_t. The environment then returns (xt+1,rt+1,ct+1)(x_{t+1},r_{t+1},c_{t+1}). Here ct+1=0c_{t+1}=0 means the transition terminates the task; otherwise it is 1. This convention makes formulating the training problem for a reward head on st+1s_{t+1} natural.

xt⟶st→atst+1⟶(xt+1,rt+1,ct+1).x_t \longrightarrow s_t \xrightarrow{a_t} s_{t+1} \longrightarrow (x_{t+1},r_{t+1},c_{t+1}).
SymbolMeaning
xt,at,rt,ctx_t,a_t,r_t,c_tObservation, outgoing action, arriving reward, continuation label.
ht,zt,sth_t,z_t,s_tDeterministic memory, stochastic latent, full RSSM state st=(ht,zt)s_t=(h_t,z_t).
qϕ(zt∣ht,xt)q_\phi(z_t\mid h_t,x_t)Observation-informed posterior (recognition model).
pϕ(zt∣ht)p_\phi(z_t\mid h_t)Predictive prior, available without the current observation.
ϕ,θ,ψ\phi,\theta,\psiWorld-model, actor, and critic parameters.
πθ(at∣st), vψ(st)\pi_\theta(a_t\mid s_t),\,v_\psi(s_t)Action distribution and expected future return.
γ,λ,H\gamma,\lambda,HDiscount factor, return mixing coefficient, imagination horizon.
sg⁡(u)\operatorname{sg}(u)Same forward value as uu, but zero derivative through this occurrence.
B,T,N,KB,T,N,KReplay batch size, sequence length, latent factors, classes per factor.

The objective of a policy is expected discounted reward:

J(θ)=E ⁣[∑k=0∞γk(∏j=1kct+j)rt+k+1].J(\theta)= \mathbb{E}\!\left[ \sum_{k=0}^{\infty} \gamma^k \left(\prod_{j=1}^{k}c_{t+j}\right) r_{t+k+1} \right].

Training of Dreamer has two connected components:

  1. Fit the world model to recorded trajectories
  2. Use multi-step trajectories to train the actor and the critic.

Let’s delve into both. In this note, we specifically focus on Dreamer V3.

One RSSM observation step: previous state and action update recurrent memory, which feeds the prior and observation-informed posterior. A posterior sample and recurrent memory form the state used by the prediction heads. A dashed arrow marks the KL comparison.

Training the World Model

Dreamer V1-V3 used a Recurrent State Space Model (RSSM) for its world model, a recurrent network architecture that is able to both capture representations of observation and action history along with a multimodal belief over the current state.

The RSSM’s state is composed of hth_t, which represents an encoding of the action and observation history, along with ztz_t, a latent sample of a learned representation of the posterior qϕq_\phi. The observation encoder is denoted eϕe_\phi.

Training occurs in the following steps. First, sample a real trajectory from replay. The observations, actions, rewards, and continuation labels in this part were actually recorded from the environment:

  1. Propogate memory:

    ht=fϕ(ht−1,zt−1,at−1).h_t=f_\phi(h_{t-1},z_{t-1},a_{t-1}).
  2. Compute the posterior:

    zt∼qϕ ⁣(zt∣ht,eϕ(xt)).z_t\sim q_\phi\!\left(z_t\mid h_t,e_\phi(x_t)\right).

    The full state is now st=(ht,zt)s_t=(h_t,z_t). We evaluate the world model losses here, at the observation-informed state. The observation decoder is compared against the recorded xtx_t, the reward head against the recorded rtr_t, and the continuation head against the recorded ctc_t. The prior pϕ(zt∣ht)p_\phi(z_t\mid h_t) is also compared with the posterior distribution through the KL losses.

    I go over a numerical example of how to conduct forward passes and backpropogation through the categorical sampling of the latent in the Appendix.

  3. Propogate memory:

    ht+1=fϕ(ht,zt,at).h_{t+1}=f_\phi(h_t,z_t,a_t).

    Here ata_t is the next recorded action. Repeat over the sequence and aggregate the losses to update the world-model parameters ϕ\phi.

The observation therefore reaches deterministic memory along the path

xt⟶eϕ(xt)⟶zt⟶ht+1.x_t\longrightarrow e_\phi(x_t)\longrightarrow z_t\longrightarrow h_{t+1}.

The posterior network qϕq_\phi is trained partly through the observation decoder, which reconstructs the observation from the latent state ht,zth_t,z_t. The reconstruction loss compares this prediction with the actual observation, and its gradients update both the decoder and the posterior network. Meanwhile, the prior pϕp_\phi is trained to match the observation-informed posterior qϕq_\phi. The posterior is also regularized toward the prior, encouraging a representation that remains informative about the observation while being predictable from history and actions.

The prediction losses have the form

Lpred,t=−log⁡pϕ(xt∣st)−log⁡pϕ(rt∣st)−log⁡pϕ(ct∣st).\mathcal L_{\mathrm{pred},t} =-\log p_\phi(x_t\mid s_t) -\log p_\phi(r_t\mid s_t) -\log p_\phi(c_t\mid s_t).

Each pϕp_\phi here denotes the relevant output head, rather than the latent prior. In V3, reward prediction uses a two-hot cross-entropy target, continuation uses binary cross-entropy, and observation reconstruction uses the loss appropriate to the input modality.

Writing qt=qϕ(zt∣ht,eϕ(xt))q_t=q_\phi(z_t\mid h_t,e_\phi(x_t)) and pt=pϕ(zt∣ht)p_t=p_\phi(z_t\mid h_t), the two KL branches are

Ldyn,t=max⁡ ⁣(1,KL⁡[sg⁡(qt)∥pt]),Lrep,t=max⁡ ⁣(1,KL⁡[qt∥sg⁡(pt)]).\begin{aligned} \mathcal L_{\mathrm{dyn},t} &=\max\!\left(1,\operatorname{KL}[\operatorname{sg}(q_t)\Vert p_t]\right),\\ \mathcal L_{\mathrm{rep},t} &=\max\!\left(1,\operatorname{KL}[q_t\Vert\operatorname{sg}(p_t)]\right). \end{aligned}

These two terms are constructed because of the following:

  1. The max(1,⋅)max(1, \cdot) allows for some disagreement between the prior and posterior without a penalty gradient, since the posterior learning more accurate outcomes also benefits the actor and critic training.
  2. While we could have a single term max⁡ ⁣(1,KL⁡[qt∥pt])\max\!\left(1,\operatorname{KL}[q_t\Vert p_t]\right), keeping them separate permits the objective to weigh divergence between the two learned distributions separately, putting more pressure on the prior to match the posterior than vice versa.

The world-model loss then averages Lpred+Ldyn+0.1Lrep\mathcal L_{\mathrm{pred}}+\mathcal L_{\mathrm{dyn}}+0.1\mathcal L_{\mathrm{rep}} over replay batch and time.

Training the Actor and Critic

Imagination lets Dreamer learn from simulated experience, reducing the need for real environment interactions and therefore improving sample efficiency. The actor learns which actions to take while the critic estimates the future return from a state when following that actor. These updates alternate with world-model training throughout learning, but the world model is fixed during each behaviour (actor+critic) update. The losses below follow Dreamer V3.

  1. Generate an imagined trajectory

    We start from an observation-informed state sts_t inferred from real replay, detach it from the world-model training graph, and call it s^0=sg⁡(st)\hat s_0=\operatorname{sg}(s_t). Here, a hat indicates an imagined quantity (similar to state estimate in controls), kk counts imagination steps, and HH is the rollout horizon. The different indices may be confusing, but what we’re saying is to take the state at real trajectory time tt, and use it as the starting state of imagination.

    At each step, the actor samples an action and the world model predicts its consequence:

    a^k∼πθ(⋅∣s^k),h^k+1=fϕ(h^k,z^k,a^k),z^k+1∼pϕ(⋅∣h^k+1),s^k+1=(h^k+1,z^k+1).\begin{aligned} \hat a_k&\sim\pi_\theta(\cdot\mid\hat s_k),\\ \hat h_{k+1}&=f_\phi(\hat h_k,\hat z_k,\hat a_k),\\ \hat z_{k+1}&\sim p_\phi(\cdot\mid\hat h_{k+1}),\\ \hat s_{k+1}&=(\hat h_{k+1},\hat z_{k+1}). \end{aligned}

    There is no new observation from the true environment here, so the latent comes from the prior (this is what we train the prior for). At s^k+1\hat s_{k+1}, the reward head predicts the arriving reward r^k+1\hat r_{k+1} and the continuation head predicts the probability c^k+1\hat c_{k+1} that the episode continues. Repeat for HH transitions:

    s^0→a^0s^1→a^1⋯→a^H−1s^H.\hat s_0\xrightarrow{\hat a_0}\hat s_1 \xrightarrow{\hat a_1}\cdots \xrightarrow{\hat a_{H-1}}\hat s_H.

    The critic evaluates each state, producing vk=vψ(s^k)v_k=v_\psi(\hat s_k) for k=0,…,Hk=0,\ldots,H. Each value estimates future return, and the final state is the one where imagination stops.

  2. Forming the return target

    The critic is trained by bootstrapping. We need a training target GkG_k estimating how much discounted reward follows each state. Beyond the rollout, we use the critic’s estimate:

    GH=vH.G_H=v_H.

    Then we work backward, from k=H−1k=H-1 to 00:

    Gk=r^k+1+γc^k+1[(1−λ)vk+1+λGk+1].G_k=\hat r_{k+1}+\gamma\hat c_{k+1} \left[(1-\lambda)v_{k+1}+\lambda G_{k+1}\right].

    Going term by term:

    • r^k+1\hat r_{k+1} is the predicted reward from the next transition.
    • γ\gamma discounts rewards further into the future; c^k+1\hat c_{k+1} gates the future contribution by continuation probability.
    • vk+1v_{k+1} estimates the remaining return directly from the next state.
    • Gk+1G_{k+1} uses more of the imagined trajectory. The mixing parameter λ∈[0,1]\lambda\in[0,1] balances these two estimates.

    With λ=0\lambda=0, the target uses one predicted reward and the next critic value. With λ=1\lambda=1, it uses all remaining imagined rewards and the final critic value. Intermediate choices give λ\lambda-returns, and Dreamer V3 uses λ=0.95\lambda=0.95 by default. This means the training target mixes 5% of the critic value with 95% of the return target, which relies on the critic at the terminal state of the imagination and largely on the world model’s reward head before the terminal state.

  3. Training the critic

    The basic task is to bring vψ(s^k)v_\psi(\hat s_k) closer to GkG_k. Conceptually, a squared-error critic would minimize (vψ(s^k)−sg⁡(Gk))2(v_\psi(\hat s_k)-\operatorname{sg}(G_k))^2.

    V3 instead predicts a probability distribution Pψ(bi∣s^k)P_\psi(b_i\mid\hat s_k) over fixed, exponentially spaced return bins bib_i. Its scalar value is the probability-weighted mean:

    vψ(s^k)=∑ibi Pψ(bi∣s^k).v_\psi(\hat s_k)=\sum_i b_i\,P_\psi(b_i\mid\hat s_k).

    Convert the target into yk=twohot⁡(sg⁡(Gk))y_k=\operatorname{twohot}(\operatorname{sg}(G_k)): place weight on the two neighboring bins, with more weight on the closer one. For example, a target of 3 between bins 2 and 6 gets weights 0.750.75 and 0.250.25.

    Later imagined states receive less weight if they are unlikely to be reached before termination. Define discounted continuation weights

    w0=1,wk=∏j=1kγc^j.w_0=1,\qquad w_k=\prod_{j=1}^{k}\gamma\hat c_j.

    The main critic loss is the cross-entropy between the target bin weights and predicted probabilities:

    LV,imag=−E∑k=0H−1sg⁡(wk)∑iyk,ilog⁡Pψ(bi∣sg⁡(s^k)).\mathcal L_{V,\mathrm{imag}} =-\mathbb E\sum_{k=0}^{H-1}\operatorname{sg}(w_k) \sum_i y_{k,i}\log P_\psi(b_i\mid\operatorname{sg}(\hat s_k)).

    Here ii indexes bins, kk indexes rollout steps, and E\mathbb E denotes averaging over sampled starting states and rollouts. Minimizing this loss puts more predicted probability on bins supported by the return target.

    You may ask - why do we need probability bins over training on the target directly? That’s a good question. Recall that Dreamer is intended to be easily ported to new domains, and different domains may have drastically different return magnitudes. By using binning, we make the training less sensitive to reward magnitudes.

    V3 also stabilizes the critic with a penalty toward a slowly averaged critic’s predictions and an auxiliary replay-value loss, weighted by 0.30.3. The latter combines real replay rewards with imagined return estimates at replay states, but the equation above shows the main imagination loss.

  4. Training the actor

    The advantage of an action taken by the actor is the return from the taken action minus the value of being in the origin state as estimated by the critic:

    Ak=Gk−vk.A_k=G_k-v_k.

    If Gk=8G_k=8 and vk=5v_k=5, the sampled action has advantage 33: its sampled imagined outcome was better than the baseline.

    To control update scale, define

    A~k=Akmax⁡(1,S),\widetilde A_k=\frac{A_k}{\max(1,S)},

    where SS is an exponentially smoothed estimate of the 95th-minus-5th percentile range of imagined returns. The denominator scales down large advantages without amplifying small ones.

    The actor loss is

    Lπ=−E∑k=0H−1sg⁡(wk)[ sg⁡(A~k)log⁡πθ(a^k∣s^k)+η H[πθ(⋅∣s^k)] ].\begin{aligned} \mathcal L_\pi=-\mathbb E\sum_{k=0}^{H-1}\operatorname{sg}(w_k)\Big[\, &\operatorname{sg}(\widetilde A_k) \log\pi_\theta(\hat a_k\mid\hat s_k)\\ &+\eta\,\mathcal H[\pi_\theta(\cdot\mid\hat s_k)]\,\Big]. \end{aligned}

    The sampled actions and states are treated as fixed inputs in this V3 update. The terms mean:

    • log⁡πθ(a^k∣s^k)\log\pi_\theta(\hat a_k\mid\hat s_k) is the log probability (or log density for continuous actions) the actor assigns to the sampled action. With the leading minus sign, minimizing the loss increases it for positive advantages and decreases it for negative ones.
    • H[πθ]\mathcal H[\pi_\theta] is the policy’s entropy. Its positive coefficient η\eta inside the brackets rewards maintaining exploration.
    • wkw_k is the same discounted continuation weight used above.

Inference

After training, the agent receives real observations during execution and uses the posterior. The observation encoder is used during deployment too. With the learned parameters fixed, each step is:

  1. Receive xtx_t and update memory using the previous inferred state and the action actually taken:

    ht=fϕ(ht−1,zt−1,at−1).h_t=f_\phi(h_{t-1},z_{t-1},a_{t-1}).
  2. Encode the observation and infer the current latent:

    zt∼qϕ ⁣(zt∣ht,eϕ(xt)).z_t\sim q_\phi\!\left(z_t\mid h_t,e_\phi(x_t)\right).
  3. Choose an action with the actor:

    at∼πθ(⋅∣ht,zt).a_t\sim\pi_\theta(\cdot\mid h_t,z_t).
  4. Execute ata_t in the environment. Receive xt+1x_{t+1}, rt+1r_{t+1}, and ct+1c_{t+1}; carry (ht,zt,at)(h_t,z_t,a_t) into the next step. At a new episode, reset the recurrent carry and previous-action input according to the implementation.

The observation encoder and posterior remain active during real-world execution because fresh observations are available. The observation decoder, reward predictor, and critic are not used during inference.

Appendix

  1. How do we get gradients through categorical sampling in the RSSM?

    A categorical sample from the categorical representation of the posterior is discrete. Ordinary backpropagation cannot differentiate through the choice of category. DreamerV3 uses a straight-through estimator, where the forward pass uses a sampled one-hot vector, while the backward pass follows the probabilities that generated it. This lets reconstruction and other prediction losses train the posterior through its sampled latent. Let’s work through an example to see how it works.

    Consider one three-category factor in the posterior. Let its logits be ℓ\ell, with probabilities

    p=softmax⁡(ℓ)=[0.1,  0.7,  0.2].p=\operatorname{softmax}(\ell)=[0.1,\;0.7,\;0.2].

    Here pp denotes this probability vector (note this example omits V3’s 1% uniform mixing). Suppose sampling selects category 2:

    zsample=[0,  1,  0].z_{\mathrm{sample}}=[0,\;1,\;0].

    Construct the latent passed to the decoder as

    z=p+sg⁡(zsample−p).\boxed{z=p+\operatorname{sg}(z_{\mathrm{sample}}-p)}.

    Forward pass. Stop-gradient does not change numerical values, so

    z=[0.1,  0.7,  0.2]+[−0.1,  0.3,  −0.2]=[0,  1,  0].z=[0.1,\;0.7,\;0.2]+[-0.1,\;0.3,\;-0.2] =[0,\;1,\;0].

    The observation decoder then sees this sample, and losses are evaluated relative to it.

    Backward pass. The stopped term contributes no derivative, leaving

    ∂z∂p=I.\frac{\partial z}{\partial p}=I.

    Thus the downstream gradient passes unchanged from zz to pp, then passes through the softmax to the logits. The forward value is hard, but the backward path is differentiable.

    For a concrete reconstruction example, use a toy scalar decoder

    x^=[1,  2,  4] z,x=3,L=12(x^−x)2.\hat x=[1,\;2,\;4]\,z,\qquad x=3,\qquad L=\tfrac12(\hat x-x)^2.

    Treat zz as a column vector here. Our sample gives x^=2\hat x=2 and L=0.5L=0.5. The decoder sends back

    ∇zL=(x^−x)[1,  2,  4]=[−1,  −2,  −4].\nabla_z L=(\hat x-x)[1,\;2,\;4] =[-1,\;-2,\;-4].

    The straight-through rule gives the same gradient g=∇pLg=\nabla_p L. Using the softmax derivative,

    ∂L∂ℓi=pi(gi−∑jpjgj).\frac{\partial L}{\partial\ell_i} =p_i\left(g_i-\sum_jp_jg_j\right).

    Since ∑jpjgj=−2.3\sum_jp_jg_j=-2.3, the logit gradient is

    ∇ℓL=[0.1(1.3),  0.7(0.3),  0.2(−1.7)]=[0.13,  0.21,  −0.34].\nabla_\ell L =[0.1(1.3),\;0.7(0.3),\;0.2(-1.7)] =[0.13,\;0.21,\;-0.34].

    Gradient descent subtracts this gradient, lowering the first two logits and raises the third. In this toy example, the backward signal pushes toward a higher decoder output. That signal then propagates into the posterior network’s parameters and the decoder also receives its own parameter gradients.

    In PyTorch, this looks like:

    probs = logits.softmax(dim=-1)
    sample = torch.distributions.OneHotCategorical(probs=probs).sample()
    z = probs + (sample - probs).detach()

    Note that this is a biased gradient approximation, and not the exact derivative of categorical sampling or a guarantee that every update improves the expected loss. The numerical gradient above is the surrogate gradient chosen by the estimator. The prior–posterior KL loss compares distributions directly, so it does not need this estimator to differentiate through a sample.