Model-based reinforcement learning and the control stack

A learned model is valuable because it lets a controller rehearse futures before paying for them in the real system. It is dangerous for exactly the same reason: an optimizer searches hardest where an imperfect model is easiest to fool. This chapter closes policy optimization, develops uncertainty-aware model-based RL, and then places learning inside the layered autonomy architecture built across the course.

Our running system is a unit-mass cart with state \(x_t=(p_t,v_t)^\mathsf T\), scalar acceleration command \(u_t\), and unknown actuation gain \(g\). Its true discrete dynamics are \(p_{t+1}=p_t+h v_t\) and \(v_{t+1}=v_t+h(g u_t-dv_t)\), with known step \(h>0\), uncertain drag \(d\geq0\), and unknown \(g>0\). A dataset \(\mathcal D=\{(x_i,u_i,x_i^+)\}_{i=1}^n\) contains observed transitions. The goal is to reach a target without letting uncertainty turn the planner into an adversary.

Based on AA203 lecture 19 and its official slides. The cart calculations, risk comparison, executable ensemble planner, and exercises are original teaching constructions. The lecture motivates uncertainty-aware planning; it does not claim that an ensemble is a calibrated Bayesian posterior or that model-based RL automatically guarantees safety. The official slides add the reduction of reinforcement learning to optimization and the policy-gradient loss, TRPO's practical limits, the KL-penalty form of PPO, the model-free review of value backups, generalized policy iteration and policy gradients, the trade-off and sample-efficiency comparison, the northward distribution-shift example, the capacity tension and its exploitation by a planner, output entropy versus model uncertainty, posterior contraction with data, the Dirac-mixture view of ensembles, the Monte Carlo score for planning under model uncertainty, the PETS experiments, and the course roadmap with its layered autonomy stack; where their notation differs, the text translates it.

A surrogate objective permits more than one policy update 0:25

Much of modern machine learning works by reducing learning to a numerical optimization problem. Supervised learning, for example, minimizes a training error over a fixed dataset, and the optimizer may take as many steps on that error as it needs. The reinforcement-learning methods of the preceding chapters do not have this form. Q-learning performs a fixed-point iteration on the Bellman optimality equation. Because it is off-policy, its updates can in principle use every transition collected so far, but the condition it drives toward is the Bellman equation for its action-value estimate, not a maximum of the expected return of the policy it induces; with function approximation, a smaller error on its regression targets need not mean a higher return. Policy-gradient methods do follow stochastic gradients of the return objective itself, but each gradient estimate is an expectation under the policy that collected the data. After one step the batch no longer describes the current policy, so there is no fixed objective on which to keep optimizing. The aim of this section and the next two is a surrogate optimization problem that combines the two virtues: it is defined on a batch sampled from the current policy, it admits several optimization steps on that batch, and it keeps each resulting change of policy small.

Let \(\pi_\theta(u\mid x)\) be a differentiable stochastic policy and \(\theta_{\rm old}\) the parameters that generated a batch. For a sampled state-action pair, define the likelihood ratio \(r_t(\theta)=\pi_\theta(u_t\mid x_t)/\pi_{\theta_{\rm old}}(u_t\mid x_t)\). If \(\widehat A_t\) estimates the old policy's advantage, the importance-weighted surrogate is

At \(\theta=\theta_{\rm old}\), its gradient equals the usual policy-gradient estimator because \(r_t=1\) and \(\nabla_\theta r_t=r_t\nabla_\theta\log\pi_\theta\). Away from the old parameters, the equality to the gradient of the true new-policy return no longer follows: the state distribution and advantage estimates came from the old policy. The ratio corrects the sampled action density at an observed state; it does not magically refresh every distributional quantity.

The usual estimator comes from the policy-gradient identity

where \(J(\theta)\) is the expected return of \(\pi_\theta\), the expectation is over states and actions generated by \(\pi_\theta\) itself, and \(A\) is that policy's advantage. An automatic-differentiation library obtains this gradient from the loss

provided the sampled pairs and the advantage estimates are treated as constants, so that differentiation acts only on the log-probability. The value of \(L^{\rm PG}\) has no meaning as a performance measure; only its gradient at the parameters that generated the batch does. The two losses have the same gradient there because of an elementary identity, valid for any positive differentiable \(f\),

applied with \(f(\theta)=\pi_\theta(u_t\mid x_t)\). Away from \(\theta_{\rm old}\) the two losses differ. The gradient of \(L^{\rm PG}\) at \(\theta\) is \(\widehat{\mathbb E}_t[\nabla_\theta\log\pi_\theta(u_t\mid x_t)\widehat A_t]\), which still weights each sampled action as if the old policy were current. The gradient of \(L^{\rm IS}\) is \(\widehat{\mathbb E}_t[r_t(\theta)\nabla_\theta\log\pi_\theta(u_t\mid x_t)\widehat A_t]\), which reweights each sampled action by its probability under the new policy relative to the old one. That reweighting is what makes \(L^{\rm IS}\), rather than \(L^{\rm PG}\), the natural objective for several steps on one batch.

Both surrogates are empirical averages over the batch, so their sampling distribution must be stated: the states and actions come from trajectories of \(\pi_{\theta_{\rm old}}\), and \(\widehat A_t\) estimates the advantage of \(\pi_{\theta_{\rm old}}\). If the samples were instead drawn from the new policy, multiplying by \(r_t(\theta)\) would correct for a change of distribution that had not occurred. Even with the correct sampling distribution, the advantage in either loss is a noisy estimate rather than the true advantage, so driving the surrogate to its optimum on one batch would fit that noise along with the signal.

This explains both the attraction and the danger of optimizing a fixed batch repeatedly. A small change can reuse expensive experience. A large change can make \(\widehat A_t\) and the sampled states poor evidence about the new policy. The central engineering question is therefore how to limit change in behavior space.

TRPO constrains policy distributions rather than raw parameters 7:30

Trust Region Policy Optimization solves a local problem of the form

The radius \(\delta>0\) bounds an empirical average divergence. This is more meaningful than a Euclidean parameter step: neural-network parameters can move substantially while leaving outputs similar, or move slightly near a sensitive region while changing the action distribution sharply. TRPO approximates this constrained problem using local curvature and conjugate gradients. The lecture emphasizes the practical cost: the second-order machinery and coupled policy/value architectures complicate implementation.

The architectural restriction is specific. Schulman, Wolski, Dhariwal, Radford and Klimov's paper introducing PPO (2017) states that TRPO is incompatible with architectures that include noise, such as dropout, or parameter sharing, between the policy and value function or with auxiliary tasks, and that it is relatively complicated to implement. TRPO has also been found to perform poorly on tasks that require deep convolutional or recurrent networks. That is an empirical observation about the benchmarks on which it was made, not a consequence of the trust-region formulation. Together these limitations motivated a first-order method that keeps the trust-region idea.

The trust region is not a universal safety guarantee. It controls average change on sampled states, under approximations and finite data. Rare states can still move more than the average suggests, and monotonic-improvement arguments require assumptions that an implementation may only approximately satisfy.

PPO clips incentives for destructive likelihood-ratio changes 11:43

Proximal Policy Optimization replaces the explicit trust-region solve with a first-order objective. Its common clipped form is

where \(\epsilon>0\). The sign of \(\widehat A_t\) matters. If the sampled action was better than baseline, increasing its probability beyond \(1+\epsilon\) ceases to improve the clipped term. If it was worse, decreasing its probability below \(1-\epsilon\) ceases to help. The minimum selects the pessimistic branch, producing a lower surrogate value than the unclipped sample term.

Clipping the objective is not the same operation as forcibly projecting every final policy ratio into the interval. Other samples share parameters, gradients can interact, and optimization can still produce ratios outside it. The useful statement is narrower: once a sample crosses its harmful-side threshold, that sample supplies no further incentive in that direction through this term. Monitoring realized KL remains prudent.

A second form of PPO keeps the divergence explicit and moves the constraint into the objective. Write \(\bar D(\theta)=\widehat{\mathbb E}_t[D_{\rm KL}(\pi_{\theta_{\rm old}}(\cdot\mid x_t)\|\pi_\theta(\cdot\mid x_t))]\) for the average divergence of the TRPO constraint, and introduce a multiplier \(\beta\ge0\) and the Lagrangian

The minus sign matters. In a maximization whose constraint is \(\bar D\le\delta\), the multiplier must penalize divergence in excess of \(\delta\); a term \(+\beta(\bar D-\delta)\) with \(\beta\gt0\) would reward it. For fixed \(\beta\), maximizing \(\mathcal L\) over \(\theta\) is an unconstrained problem to which stochastic gradient ascent applies with first-order tools only. The multiplier is then adjusted by dual descent. For every \(\beta\ge0\), the dual function \(d(\beta)=\max_\theta\mathcal L(\theta,\beta)\) is at least the constrained optimum, because the penalty term is nonnegative at every feasible \(\theta\). When the inner maximizer \(\theta^\star_\beta\) is unique, Danskin's theorem gives \(d'(\beta)=\delta-\bar D(\theta^\star_\beta)\). A descent step with step size \(\eta\gt0\), followed by projection onto \(\beta\ge0\), therefore reads \(\beta\leftarrow\max\{0,\beta+\eta(\bar D(\theta^\star_\beta)-\delta)\}\): the penalty grows when the realized divergence exceeds the radius and shrinks when it falls below it. These statements about \(d(\beta)\) and \(d'(\beta)\) are exact only when the inner maximization is solved exactly; the next paragraph describes what is done when it is not.

In practice the inner maximization is not carried to convergence, and the multiplier is adapted heuristically. The PPO paper's adaptive-penalty variant maximizes \(\widehat{\mathbb E}_t[r_t(\theta)\widehat A_t-\beta D_{\rm KL}(\pi_{\theta_{\rm old}}(\cdot\mid x_t)\|\pi_\theta(\cdot\mid x_t))]\), which omits the constant \(\beta\delta\) because it does not affect the gradient in \(\theta\). After each policy update it computes the realized average divergence \(\bar D\) and compares it with a target \(d_{\rm targ}\): if \(\bar D\lt d_{\rm targ}/1.5\) the coefficient is halved, and if \(\bar D\gt1.5\,d_{\rm targ}\) it is doubled. This is a multiplicative version of the same dual logic. With \(d_{\rm targ}=0.01\), a realized divergence of \(0.02\) exceeds \(0.015\) and doubles \(\beta\), while \(0.005\) falls below \(0.0067\) and halves it. The clipped form needs neither a multiplier nor a measured divergence inside its objective; it is the more commonly used of the two, and PPO in that form has become one of the most widely used policy-optimization algorithms.

TRPO and PPO complete the lecture's model-free arc: Monte Carlo and temporal-difference methods estimate values, generalized policy iteration alternates evaluation and improvement, Q-learning learns an off-policy action value, and policy optimization represents the policy directly. Their sample efficiency differs because on-policy updates require newly distributed data while off-policy methods can reuse older transitions. Wall-clock cost also includes model evaluation, simulation, parallelism, and optimization; sample count alone is not elapsed time.

Model-free methods share one loop but differ in their targets 17:46

Every model-free method of the preceding chapters instantiates one loop with three stages. The agent runs its current policy \(\pi(u_t\mid x_t)\) to generate samples, trajectories \(\tau=(x_0,u_0,\ldots,x_T)\); it fits a function to those samples; and it uses the fit to improve the policy. What differs is the fitted object and the improvement rule. A value-based method fits an approximation of the state value \(V^\pi(x)\) or the action value \(Q^\pi(x,u)\) and improves greedily, setting \(\pi(x)\in\arg\max_uQ(x,u)\), as in Q-learning and DQN. A policy-gradient method may fit a value function as a critic, and it improves by a gradient step on the policy parameters, as in REINFORCE and the advantage actor-critic methods A3C and its synchronous variant A2C. Model-based reinforcement learning, the subject of the rest of this chapter, places a third object in the middle stage: an approximation of the transition law \(p(x_{t+1}\mid x_t,u_t)\). Because a transition model is fit to transitions rather than to the returns of one particular policy, it can evaluate policies other than the one that collected the data, though only where those transitions are represented, a caveat that the remainder of the chapter develops.

To keep \(r_t\) for the likelihood ratio, this chapter writes the reward produced by action \(u_t\) at state \(x_t\) as \(R_t=R(x_t,u_t)\). With discount \(0\le\gamma\lt1\), the return from time \(t\) is \(G_t=\sum_{k\ge0}\gamma^kR_{t+k}\); \(V^\pi(x)\) is its expectation given \(x_t=x\), and \(Q^\pi(x,u)\) its expectation given \(x_t=x\) and \(u_t=u\), with later actions drawn from \(\pi\). Three backups estimate \(V^\pi\). With an estimate \(\widehat V\) and a step size \(\alpha\in(0,1]\), they are

The three differ in their targets. The dynamic-programming target is an exact expectation over the action and successor distributions, so nothing is lost to sampling, but computing it requires the transition law and reward of the Markov decision process. The Monte Carlo target \(G_t\) is a sampled return whose conditional expectation is \(V^\pi(x_t)\): it is unbiased, but its variance accumulates over every later reward, and it is available only when the episode reaches a terminal state. The temporal-difference target samples one step and substitutes the current estimate for the rest. Its variance is lower and it can be computed online after a single transition, but it is biased: an error \(e\) in \(\widehat V(x_{t+1})\) shifts the target by \(\gamma e\). A small calculation shows the size of the variance gap. If the rewards are independent with common variance \(\sigma_R^2\), transitions are deterministic, and the horizon is infinite, then \(\operatorname{Var}(G_t)=\sigma_R^2\sum_{k\ge0}\gamma^{2k}=\sigma_R^2/(1-\gamma^2)\), whereas the temporal-difference target carries the variance \(\sigma_R^2\) of one reward. With \(\gamma=0.99\) the ratio is \(1/(1-0.9801)\approx50.3\). The same three backups apply to \(Q^\pi\), with \(Q^\pi(x_{t+1},u_{t+1})\) in place of \(V^\pi(x_{t+1})\).

A tabular representation stores one number per state, a vector of \(|\mathcal X|\) entries for \(\widehat V\) on a finite state set \(\mathcal X\), and one per state-action pair, a \(|\mathcal X|\times|\mathcal U|\) matrix for \(\widehat Q\) on a finite action set \(\mathcal U\). Function approximation replaces the table by a parametric function with weights \(w\): \(\widehat V_w(x)\), \(\widehat Q_w(x,u)\) with the action as an input, or, for a finite action set, a network that takes \(x\) and returns one action value per action. The official slides write \(\theta\) for these weights; this chapter writes \(w\) to keep \(\theta\) for policy parameters. The backups become updates in weight space:

The Monte Carlo update is a stochastic gradient step on \(\tfrac12(G_t-\widehat V_w(x_t))^2\), since \(G_t\) does not depend on \(w\). The temporal-difference update applies the same formula with the bootstrapped target held fixed. It ignores the target's own dependence on \(w\), so it is a semi-gradient step and not the gradient of any fixed loss. Both reduce to the tabular rules when \(\widehat V_w(x)=w_x\), one weight per state: then \(\nabla_w\widehat V_w(x_t)\) is the indicator vector of \(x_t\), and the update changes only the entry for \(x_t\).

Value-based control alternates two processes in generalized policy iteration. Evaluation moves the value estimate toward the value of the current policy, and improvement makes the policy greedy with respect to the current estimate. With exact tabular evaluation and improvement the alternation terminates at the optimal value \(V^\star\) and an optimal policy \(\pi^\star\); generalized policy iteration also covers interleavings in which neither process is completed. With action values and sampled transitions, two updates instantiate it:

SARSA is on-policy: it evaluates and improves the policy that makes the decisions, because \(u_{t+1}\) is the action that policy actually took. Q-learning is off-policy: the maximization evaluates the greedy policy, whatever policy generated the data. The equation and the backup form two independent choices. A full backup of the Bellman expectation equation for \(Q^\pi\) is Q-policy iteration and its sample backup is SARSA; a full backup of the Bellman optimality equation for the optimal action value \(Q^\star\) is Q-value iteration and its sample backup is Q-learning. DQN carries Q-learning to a deep network \(\widehat Q_w\) with two additions: experience replay, which reuses stored transitions and breaks their temporal correlation, and a fixed target network, which holds the bootstrap target constant over many updates.

Policy optimization instead learns an explicit parametric policy \(\pi_\theta\) and seeks

where \(p_\theta(\tau)\) is the trajectory distribution that \(\pi_\theta\) induces. It estimates \(\nabla_\theta J\) and ascends, \(\theta\leftarrow\theta+\alpha\nabla_\theta J(\theta)\). With \(N\) sampled trajectories indexed by \(i\), the basic estimator and the maximum-likelihood gradient of the same trajectories are

Side by side, they show that the policy gradient is a return-weighted maximum-likelihood gradient: it changes \(\theta\) so that trajectories with higher return become more probable, which is trial-and-error learning in gradient form. Its weakness is variance, because every log-probability term is multiplied by a whole-trajectory return. Baselines and critics reduce that variance. The actor-critic estimator replaces the realized return by a learned action value,

and so trades variance for the bias of the critic. The surrogate objectives of the first three sections, with an advantage estimate in place of \(\widehat Q_w\), are the batch-reuse extension of this estimator.

Model-based RL learns dynamics and then plans through them 25:00

The methods just reviewed differ along three kinds of axis. They trade sample efficiency against stability and ease of use. They rest on different assumptions: stochastic or deterministic dynamics, continuous or discrete state and action spaces, and episodic or infinite-horizon tasks. Monte Carlo evaluation, for instance, needs episodes that terminate, because its target is a completed return, and greedy improvement is an enumeration over a finite action set but becomes an inner optimization over a continuous one. Finally, problems differ in what is easy to represent. In some, a good policy has a simpler description than the dynamics, and searching directly over policies is attractive. In others, the dynamics are simple or known up to a few parameters while a good policy is complicated, and learning the model is the better route.

Sample efficiency is the number of environment interactions needed to reach a good policy. Among model-free methods its main determinant is whether the algorithm is off-policy, able to improve the policy without generating new samples from the current policy, or on-policy, needing fresh samples after every change of policy, however small. A typical ordering, from fewest samples needed to most, is: model-based methods with low-capacity models, model-based methods with deep models, off-policy Q-function learning, actor-critic methods, which sit near the boundary between off-policy and on-policy and can be built either way, on-policy policy-gradient algorithms, and evolutionary or other gradient-free algorithms. This is a typical ordering rather than a theorem; it depends on the task, the implementation, and what is counted as an interaction. As the end of the PPO section noted, the less efficient end remains in use because wall-clock time is not sample count: samples from a fast simulator can be cheap, and elapsed time also includes computation.

Suppose a parametric model \(\hat f_\phi(x,u)\) predicts the successor. The basic deterministic recipe is

Written as a procedure, the recipe has three steps. First, run a base policy \(\pi_0(u\mid x)\), such as a random or deliberately exploratory policy, in the real system and record the transitions \(\mathcal D=\{(x_i,u_i,x_i^+)\}_{i=1}^n\). Second, fit \(\hat f_\phi\) to \(\mathcal D\) by the least-squares problem above. Third, plan a sequence of actions through the fitted model. The motivation is that with a model in hand every optimal-control tool of the course applies, from direct and indirect trajectory optimization to dynamic programming and model predictive control. The official slides write the true transition law as \(T(x_{t+1}\mid x_t,u_t)\), the learned model as \(f_\theta\), and the parameter posterior as \(p(\theta\mid\mathcal D)\); this chapter writes \(\hat f_\phi\) and \(p(\phi\mid\mathcal D)\) because \(\theta\) already denotes policy parameters.

For stochastic transitions, maximizing conditional log likelihood replaces squared error. This is system identification followed by optimal control. It works especially well when physics supplies a low-dimensional form and data excite the unknown parameters. For the cart, writing \(y_i=(v_i^+-v_i)/h\) gives the linear regression \(y_i=g u_i-dv_i\). If the columns \(u_i\) and \(-v_i\) are sufficiently independent, least squares identifies \(g,d\). A base policy that always commands zero cannot identify \(g\); no optimizer repairs missing excitation.

The equivalence of squared error and likelihood holds under a specific noise model. If \(x^+=\hat f_\phi(x,u)+\varepsilon\) with \(\varepsilon\sim\mathcal N(0,\sigma^2I)\) and \(\sigma\) fixed, the negative log likelihood of one transition is \(\|x^+-\hat f_\phi(x,u)\|_2^2/(2\sigma^2)\) plus a term that does not depend on \(\phi\), so maximizing likelihood and minimizing squared error select the same \(\phi\). When the model also predicts a state-dependent variance, as the probabilistic networks of the PETS section do, the two criteria differ: the likelihood down-weights residuals where the predicted variance is large and penalizes a small predicted variance where residuals are large.

The recipe works well when the model class contains the true dynamics and has few parameters. Linear time-invariant dynamics are the standard case: \(x^+=Ax+Bu\) is linear in the unknown entries of \(A\) and \(B\), least squares has a closed-form solution, and the fitted matrices apply at every state. The cart regression has the same character. Because \(y=gu-dv\) holds at every state, identifying \(g\) and \(d\) from a few sufficiently exciting transitions yields global knowledge: up to estimation error, the fitted model predicts correctly at velocities and commands that the base policy never produced. This makes the base policy a design decision. It must excite every parameter direction, but once it does, the physical structure, not the coverage of the data, carries the extrapolation. The argument fails exactly when the structure is wrong, which is the case the next paragraph considers.

A high-capacity nonlinear model changes the failure mode. Training loss measures one-step accuracy under the data distribution \(p_{\pi_0}(x,u)\). Planning induces a new distribution \(p_{\pi_f}\). Multi-step rollout compounds errors, and the planner deliberately seeks action sequences with high predicted reward. If an erroneous region looks optimistic, optimization selects it. Low average validation error under the old distribution does not bound closed-loop planning loss under the new one.

A planar example makes the shift concrete. An agent starts at a point, must travel as far north as possible, and must avoid an obstacle lying to the north. A random base policy \(\pi_0\) wanders near the start, so \(p_{\pi_0}(x)\) concentrates on a small neighbourhood of it. Let \(\pi_f\) denote the behaviour obtained by planning through the model \(\hat f_\phi\) fit to those data, and \(p_{\pi_f}(x)\) its state distribution. The planner heads north, so \(p_{\pi_f}\) is a narrow band that leaves the explored neighbourhood and reaches toward the obstacle, which lies where the model has no data. A model can be accurate on \(p_{\pi_0}\) and still predict a free path through the obstacle. This is the covariate shift met in imitation learning, with learned dynamics in place of a learned policy. The planner produces the shift whatever the model; what the complexity of the dynamics and the capacity of the model determine is how far predictions degrade off \(p_{\pi_0}\). Simple dynamics in a correct low-dimensional class extrapolate, as the cart regression does, whereas complex dynamics fit by a flexible network need not.

A receding-horizon loop reduces the mismatch: plan, execute only the first action, observe the real successor, append the transition, refit, and replan. This imports the feedback logic of model predictive control and gradually collects data where the planning policy visits. It still needs safe exploration, because the first visit to an unknown region happens before its outcome can improve the model.

The loop combines two ideas from earlier in the course. Executing only the first action and replanning from the measured state is model predictive control. Appending each observed transition and refitting is adaptive control: parameter estimates are updated online from measured data, and the controller is recomputed from the current estimate as if it were exact, the certainty-equivalence principle. Together they gradually close the gap between \(p_{\pi_0}\) and \(p_{\pi_f}\), because the data come to include the states the planner actually visits.

Aleatoric and epistemic uncertainty answer different questions 34:59

The failure of the basic recipe can be stated as two requirements on the model that conflict for a point estimate. In the high-data regime the model should have enough capacity to represent complex dynamics; in the low-data regime, early in learning, it should not overfit the few transitions available. Consider predicted return as a function of the trajectory \(\tau\), fit through a handful of observed trajectories. A smooth low-capacity fit and a high-capacity fit can both pass through every observation, yet the high-capacity fit may oscillate between the observations, with sharp spurious peaks. A planner that maximizes predicted return selects those peaks, because they are where predicted return is highest, and it is highest there because the model is wrong. The planner exploits the model exactly where the model knows least.

The resulting bias is systematic. Suppose two candidate trajectories have the same true return and the model's predictions of their returns err by independent amounts \(\sigma Z_1\) and \(\sigma Z_2\), with \(Z_1,Z_2\sim\mathcal N(0,1)\). The planner picks the larger prediction. Since \(\max(Z_1,Z_2)=\tfrac12(Z_1+Z_2)+\tfrac12|Z_1-Z_2|\) and \(Z_1-Z_2\sim\mathcal N(0,2)\), the expected overestimate of the selected trajectory's return is \(\tfrac12\sigma\,\mathbb E|Z_1-Z_2|=\tfrac12\sigma\cdot2/\sqrt\pi=\sigma/\sqrt\pi\approx0.564\sigma\). With ten such candidates the expected overestimate is about \(1.54\sigma\), and a planner that compares thousands of candidates inflates it further. Unbiased prediction errors become a biased choice once an optimizer selects among them. The remedy pursued in the rest of the chapter is to make the model report how certain its predictions are, so that the planner can discount the predictions it should not trust.

A probabilistic model \(p(x^+\mid x,u,\phi)\) may report broad output noise even with known parameters. That is aleatoric uncertainty: irreducible variability in the process or observation. Epistemic uncertainty concerns which model or parameter \(\phi\) is correct given finite data. It should contract where informative evidence accumulates and may remain large off distribution.

A first idea is to use the output entropy of a probabilistic model. A network that maps \((x,u)\) to a distribution over \(x^+\) may output, for a finite state set \(\mathcal X\), a categorical distribution \(q(x')\) over the possible successors, with entropy \(-\sum_{x'\in\mathcal X}q(x')\log q(x')\), or, for a continuous state of dimension \(n_x\), a Gaussian \(\mathcal N(\mu,\Sigma)\) with entropy \(\tfrac12\log\big((2\pi e)^{n_x}\det\Sigma\big)\). Either is easy to compute, but it measures the spread of the prediction under one fitted parameter value. Trained by maximum likelihood, that spread matches the noise seen in the training data, so it estimates aleatoric uncertainty. Nothing in the training forces the output entropy to grow away from the data, and a model that has never seen a region can predict a confident, narrow distribution there. Output entropy therefore misses epistemic uncertainty, which concerns the existence of several parameter values that explain the same data equally well, such as two curves that pass through the same observations and diverge elsewhere.

Bayesian inference represents epistemic uncertainty by

The predictive distribution integrates parameter uncertainty and conditional process noise. A point estimate of output entropy cannot generally distinguish them. That distinction matters at a cliff: unavoidable wind and ignorance of the steering gain can produce similar terminal variance, but the remedies differ. More data can reduce ignorance; it cannot remove intrinsic wind.

The usual fitting procedure computes a single maximum-likelihood estimate \(\hat\phi\in\arg\max_\phi\log p(\mathcal D\mid\phi)\), and every later prediction treats \(\hat\phi\) as exact. Expressing model uncertainty means computing the posterior \(p(\phi\mid\mathcal D)\) instead, from the prior \(p(\phi)\) and the likelihood \(p(\mathcal D\mid\phi)\), and predicting with the integral above, the posterior predictive distribution. For a finite set of candidate parameter values the integral is the average of the candidates' predictions weighted by their posterior probabilities.

The cart gain shows how the posterior contracts. Take the drag as known to be zero, so that each transition gives \(y_i=gu_i+\varepsilon_i\) with \(y_i=(v_i^+-v_i)/h\) and independent noise \(\varepsilon_i\sim\mathcal N(0,\sigma_\varepsilon^2)\). With a Gaussian prior \(g\sim\mathcal N(\mu_0,s_0^2)\), the posterior is Gaussian, \(\mathcal N(\mu_n,s_n^2)\), with

Precisions add. Each transition contributes \(u_i^2/\sigma_\varepsilon^2\), so a transition with \(u_i=0\) contributes nothing, which is the Bayesian form of the excitation requirement. With \(s_0=0.3\), \(\sigma_\varepsilon=0.1\), and unit commands, the posterior standard deviation of \(g\) is \(0.300\) before data, \(0.095\) after one transition, \(0.057\) after three, and \(0.022\) after twenty. The predictive variance of \(y\) at a new command \(u\) is \(u^2s_n^2+\sigma_\varepsilon^2\), an epistemic term plus an aleatoric one. After twenty unit-command transitions it is \(0.0005+0.01\) at \(u=1\), where the aleatoric term dominates, but \(0.0497+0.01\) at \(u=10\), where the epistemic term is about five times the aleatoric one. The predictive band widens away from the observed commands and narrows everywhere as data accumulate. The same behaviour holds for a straight-line model with unknown intercept and slope, whose joint posterior contracts from a broad elongated region toward the true pair as observations are added.

Uncertainty only changes a decision after it enters the objective or constraint. Expected return already penalizes variance when bad tails carry large negative reward. A risk-neutral quadratic target makes this explicit: \(\mathbb E[(X-x_g)^2]=(\mathbb E X-x_g)^2+\operatorname{Var}(X)\). With asymmetric catastrophic loss, a chance constraint or coherent risk measure may be more appropriate than the mean. The chosen risk model is a design decision, not a free theorem supplied by uncertainty estimates.

A cliff makes the effect of spread on a decision concrete. Let the cart start at rest at the origin with \(h=1\) and \(d=0\), let the goal be \(p=1\), and let any final position beyond an edge at \(p=1.05\) count as a fall. Score a final position \(p\) by \(-|p-1|\) if \(p\le1.05\) and by \(-10\) after a fall. Command sequence A is \((u_0,u_1)=(1,-1)\), which accelerates and then brakes, so the cart comes to rest at \(p_2=g\). Sequence B is \((0.8,-0.8)\) and comes to rest at \(0.8g\). Under the nominal gain \(g=1\), A reaches the goal exactly and B stops short. If the plausible gains are \(\{0.98,1.00,1.02\}\), equally weighted, A's expected score is \(-0.04/3\approx-0.013\) and B's is \(-0.200\), so A is preferred. If they are \(\{0.7,1.0,1.3\}\), the gains of the ensemble example below, A's final positions are \(0.7\), \(1.0\), and \(1.3\); the last is a fall, and A's expected score is \(-10.3/3\approx-3.43\). B's final positions are \(0.56\), \(0.80\), and \(1.04\), all safe, with expected score \(-0.68/3\approx-0.227\), so B is preferred. Both gain sets have mean one, so A's mean final position is the goal in either case; only the spread changed the decision. A planner that uses the nominal model alone chooses A in both cases.

Gaussian processes and ensembles trade analytic structure for scale 50:02

A Gaussian process specifies a distribution over function values through a mean and kernel. Conditioning a Gaussian prior on observations gives a posterior predictive mean and covariance in closed form under the Gaussian-noise assumptions. This can be highly data efficient and gives disciplined uncertainty in small-data regimes. Exact inference typically requires factorizing an \(n\times n\) kernel matrix, with cubic time and quadratic storage in the basic implementation; sparse variants alter that tradeoff.

The phrase “distribution over functions” has a concrete finite meaning. For any finite set of inputs \(z_1,\ldots,z_n\), with \(z=(x,u)\), a Gaussian process with zero prior mean and covariance function \(k\) makes the vector of function values \((f(z_1),\ldots,f(z_n))\) multivariate Gaussian with covariance matrix \(K\), \(K_{ij}=k(z_i,z_j)\). Sampling that vector on a fine grid of inputs and joining the points draws one sampled function, which is how prior and posterior samples are plotted. For observations \(y_i=f(z_i)+\varepsilon_i\) with independent \(\mathcal N(0,\sigma_\varepsilon^2)\) noise, conditioning gives the posterior mean and variance at a query \(z_\ast\),

where \(k_\ast=(k(z_\ast,z_1),\ldots,k(z_\ast,z_n))^\mathsf T\) and \(y=(y_1,\ldots,y_n)^\mathsf T\). The solve with the \(n\times n\) matrix \(K+\sigma_\varepsilon^2I\) is the source of the cubic cost. For the squared-exponential covariance \(k(z,z')=s^2\exp(-\|z-z'\|^2/(2\ell^2))\), with amplitude \(s\) and length scale \(\ell\), and a single noiseless observation at \(z_1\), the posterior variance at distance \(\Delta=\|z_\ast-z_1\|\) is \(s^2(1-e^{-\Delta^2/\ell^2})\): zero at the observation, \(0.632s^2\) one length scale away, and \(0.982s^2\) two length scales away. The posterior pinches at the data and relaxes to the prior between and beyond them. Under these modelling assumptions the posterior is exact, which is the source of the data efficiency. The weaknesses are the cost already noted and a second one: the kernel fixes the notion of similarity between inputs. Hyperparameters such as \(\ell\) are tuned, but a Gaussian process with a standard kernel does not learn a feature representation as a neural network does, which limits it on high-dimensional inputs such as images.

A bootstrap ensemble trains \(M\) models \(\{\hat f_m\}_{m=1}^M\) on resampled data and often with different initializations. At a query \((x,u)\), disagreement estimates epistemic uncertainty:

Formally, an ensemble approximates the parameter posterior by an equally weighted mixture of point masses at the members' parameters \(\phi_1,\ldots,\phi_M\):

Here \(\delta_{\phi_m}\) is the Dirac point mass at \(\phi_m\), unrelated to the trust-region radius \(\delta\). The second approximation follows from the first by substitution into the predictive integral, because integrating against a point mass evaluates the integrand at that point. The predictive distribution is therefore a uniform mixture of the members' predictive distributions. When members are deterministic, each term is a point mass at \(\hat f_m(x,u)\), and the mixture has mean \(\bar f\) and covariance \(\frac{M-1}M\widehat\Sigma_{\rm epi}\), the disagreement matrix with \(1/M\) in place of \(1/(M-1)\). When members are probabilistic, the mixture variance adds the members' average predicted variance to their disagreement, the decomposition of Exercise 5.

The members come from separate optimizations, and the training loss of a neural network usually has many local optima. Random initialization and the noise of stochastic gradient descent send different members to different optima, so the point masses spread over several modes of a multimodal posterior. Variational inference with a single Gaussian approximating family, by contrast, concentrates its mass around one mode, and MultiSWAG fits a Gaussian around each of several stochastic-gradient solutions, placing mass within several basins at once; Wilson and Izmailov (2020) argue on these grounds that deep ensembles are an effective mechanism for approximate Bayesian marginalization. The same mechanism explains a practical simplification: random initialization and stochastic optimization are often relied on to diversify the members, without resampling the data or collecting independent datasets. Bootstrap resampling nevertheless remains common; PETS, discussed next, trains each of its five members on a dataset of the same size drawn with replacement from the collected transitions.

Ensembles scale to neural models and are easy to parallelize, but members are not generally independent posterior samples. Shared architecture, optimization bias, and common data can make them confidently agree off distribution. Calibration must be measured on held-out and shifted data relevant to deployment.

For a one-step cart calculation, let the plausible gains be \(g_m\in\{0.7,1.0,1.3\}\), current velocity zero, \(h=1\), and desired next velocity one. Command \(u=1\) yields predictions \(0.7,1.0,1.3\), mean one, and mean squared target error \((0.09+0+0.09)/3=0.06\). A nominal model reports zero error. The ensemble reveals the price hidden by the mean. With input cost \(0.05u^2\), the expected one-step cost is \((1-2u+1.06u^2)+0.05u^2\), minimized at \(u=1/1.11\approx0.901\), rather than the nominal \(u=1/1.05\approx0.952\).

PETS propagates model uncertainty through sampled trajectories 1:05:45

With a posterior over models in hand, a planner can score a candidate action sequence \((u_0,\ldots,u_{H-1})\) by its expected return under the model. A Monte Carlo estimate repeats three steps \(N\) times: sample parameters \(\phi_i\sim p(\phi\mid\mathcal D)\), which for an ensemble means choosing one member; propagate \(x_{t+1,i}\sim p(x_{t+1}\mid x_{t,i},u_t,\phi_i)\) for \(t=0,\ldots,H-1\) from the current state \(x_{0,i}=x_0\); and sum the predicted rewards. The score is the average

Each inner sum is a sampled return under one plausible model. The average is therefore an unbiased estimate of the expected return when the parameters are drawn once from the posterior and held for the whole rollout, with a standard error proportional to \(1/\sqrt N\). A planner that minimizes cost uses \(\widehat J=-\widehat R\). This is one way among several to approximate the expected return under the posterior predictive distribution; propagating only the predicted mean, or matching the first two moments of the state distribution at each step, are others. What matters is that candidates are compared by expected return under the model's uncertainty rather than by the return under a single best-fit model.

Probabilistic Ensembles with Trajectory Sampling (PETS) combines learned probabilistic ensemble dynamics, sampling-based propagation, the cross-entropy method, and receding-horizon control. For each candidate action sequence, it samples particles, propagates them through ensemble members and stochastic outputs, averages their returns, refits a proposal distribution toward elite candidates, and finally executes only the first action before replanning.

PETS was introduced by Chua, Calandra, McAllister and Levine in “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models” (NeurIPS 2018). Its model is an ensemble of five probabilistic neural networks, each outputting a Gaussian over the next state and each trained on its own bootstrap resample of the data. The members' disagreement represents epistemic uncertainty and each member's predicted variance represents aleatoric uncertainty. Propagation uses twenty particles. In the variant TS1 each particle re-samples a member uniformly at every time step; in TS\(\infty\) each particle keeps one member for the whole trial. The planner is the cross-entropy method inside model predictive control, which the authors chose over random shooting because it samples new action sequences near earlier ones that scored well.

The cross-entropy method here is an optimizer, despite sharing a name with a supervised-learning loss. Start with a distribution over action sequences, sample candidates, retain a top fraction, update the distribution's mean and variance from those elites, and repeat. It needs no model derivatives and accommodates nonconvex objectives, but finite samples can miss good modes. Bounds, smoothing, population size, elite fraction, and iteration count affect behavior.

from statistics import fmean

gains = (0.7, 1.0, 1.3)

def expected_cost(u, target=1.0, effort=0.05):
    return fmean((g*u - target)**2 for g in gains) + effort*u*u

candidates = [i / 1000 for i in range(-2000, 2001)]
u_star = min(candidates, key=expected_cost)
assert abs(u_star - 1/1.11) < 0.002
assert expected_cost(u_star) < expected_cost(1/1.05)
print(u_star, expected_cost(u_star))

This exact grid is an executable one-step analogue, not a full PETS implementation. It verifies the ensemble expectation and the analytic optimum without adding a dependency. In a long horizon, a particle's sampled model can be held fixed or resampled across time; these choices encode different beliefs about whether uncertainty is a persistent unknown parameter or fresh transition noise.

The trajectory score above can be checked on the cliff of the uncertainty section. In the code below, rollout simulates the cart dynamics of this chapter with a given gain from rest at the origin and returns the final position and velocity; score applies the cliff score to a final position; and expected_score averages that score over the members of a deterministic ensemble. With deterministic members and one rollout per member, the Monte Carlo average equals the exact expectation under the uniform mixture. The assertions check that sequence A ends at rest and reproduce the four expected scores computed by hand; the printed pair shows that under the broad ensemble B outscores A.

from statistics import fmean

def rollout(u_seq, g, h=1.0, d=0.0):
    p, v = 0.0, 0.0
    for u in u_seq:
        p, v = p + h*v, v + h*(g*u - d*v)
    return p, v

def score(p, goal=1.0, edge=1.05, fall=-10.0):
    return fall if p > edge else -abs(p - goal)

def expected_score(u_seq, gains):
    return fmean(score(rollout(u_seq, g)[0]) for g in gains)

A, B = (1.0, -1.0), (0.8, -0.8)
narrow, broad = (0.98, 1.0, 1.02), (0.7, 1.0, 1.3)
assert abs(rollout(A, 1.3)[1]) < 1e-12
assert abs(expected_score(A, narrow) + 0.04/3) < 1e-9
assert abs(expected_score(B, narrow) + 0.2) < 1e-9
assert abs(expected_score(A, broad) + 10.3/3) < 1e-9
assert abs(expected_score(B, broad) + 0.68/3) < 1e-9
print(expected_score(A, broad), expected_score(B, broad))

PETS demonstrated that careful model-based planning can achieve strong continuous-control results with far fewer real interactions than prominent model-free baselines. That is an empirical result for studied tasks, not a universal ordering. If observations are extremely high-dimensional, contacts are difficult to model, or abundant simulation makes interaction cheap, the wall-clock and engineering balance can favor model-free learning.

The reported experiments cover four simulated tasks: cartpole, a seven-degree-of-freedom pusher, a seven-degree-of-freedom reacher, and half-cheetah. The baselines include the model-free methods PPO, soft actor-critic (SAC), and DDPG, each also shown at its converged performance; a deterministic deep model-based method of Nagabandi et al. (2017); and Gaussian-process methods, one of them adapted from Kamthe and Deisenroth (2018). The paper reports that PETS learns every task in under 100,000 time steps, or 100 trials; that it matches the asymptotic performance of the model-free methods on several tasks; and that on half-cheetah it needs about 8 and 125 times fewer samples than SAC and PPO respectively. On every task except cartpole, which is low-dimensional enough for an efficient Gaussian-process model, it significantly outperformed all the other baselines. One qualification belongs beside these numbers: each plotted learning curve averages, over ten experiments, the maximum reward seen so far, a monotone summary rather than the reward of the current policy.

The same paper measures how the model's accuracy evolves on half-cheetah. Models fit after 10, 20, and so on up to 100 trials are scored, by mean squared error and by negative log likelihood, on the trajectory of every trial. A model predicts the trajectories of the trials it was trained on well, and its error rises on later trajectories, which visit states that the earlier data did not contain; models fit after more trials have lower error on those later trajectories. This is the distribution shift of the basic recipe, measured, and it is the reason the receding-horizon loop keeps refitting.

The course methods form a stack rather than a tournament 1:14:23

The course's map of methods explains where each layer of a deployed system comes from. Control divides into feedback control, adaptive control, and optimal and learning-based control. Optimal control splits into open-loop methods, indirect ones through necessary conditions and direct ones through transcription, and closed-loop methods, dynamic programming and its continuous-time counterparts in the Hamilton–Jacobi–Bellman and Hamilton–Jacobi–Isaacs equations. Model predictive control sits between the two branches, because it repeatedly solves open-loop problems from measured states. When the model must be learned from data, the problem becomes adaptive optimal control, which branches into imitation learning, model-free reinforcement learning, and model-based reinforcement learning. Model-based reinforcement learning, the branch of this chapter, is the one closest to adaptive control: estimate a model from data and recompute the controller from the estimate. The branches are not competitors. Each answers a question under different information and computational limits, and a deployed system combines them.

A practical autonomous system decomposes decisions by abstraction and rate. A world model estimates state and predicts evolution. A high-level stochastic planner chooses modes or routes. A trajectory optimizer produces dynamically plausible motion. MPC replans and tracks under changing conditions. Fast low-level feedback, often PID or an embedded regulator, closes the actuator loop. Reachability analysis can compute invariant safe sets or backup policies that constrain multiple layers.

The layers correspond to branches of that map. In driving, the world model may convert camera images into a semantic map of the surroundings. The highest decision layer, which might choose between keeping a lane behind a slower vehicle and changing into an adjacent lane, is where closed-loop methods of the dynamic-programming family belong, because the choice must account for the stochastic behaviour of other road users; its output is a goal \(\mathcal G\), such as the target lane. The open-loop layer turns the goal into a trajectory by direct or indirect trajectory optimization, usually with a reduced-fidelity vehicle model and the constraints that matter at that scale. Model predictive control tracks the trajectory with a finer model and further safety constraints, and the actuators close the fastest loops, typically with PID controllers. Safety concepts, for example sets computed by Hamilton–Jacobi reachability that mark the states from which a collision can no longer be avoided, inform both the open-loop layer and the tracking layer.

For the cart, a high-level decision selects the goal side of an obstacle; a learned ensemble predicts uncertain gain and drag; trajectory optimization chooses a nominal acceleration sequence; MPC applies its first element and replans; a fast velocity loop rejects disturbances; a safety filter prevents leaving the admissible position interval. No single chapter's method replaces all the others. Each solves a problem at a particular representation, horizon, and computational rate.

End-to-end learning can move downward through this stack as data and models improve, but modularity retains diagnostic and assurance value. A learned perception-to-action policy can be powerful while a verified low-level envelope limits catastrophic commands. Conversely, a formally safe low-level controller cannot rescue a high-level objective that sends the robot to the wrong destination. System assurance must state which layer, model mismatch, and disturbance set a guarantee covers.

Exercises with worked solutions

A deployment review must test the planner-model pair

One-step held-out error is necessary but insufficient. A useful evaluation separates four questions. Does the model predict individual transitions on data resembling training? Does an open-loop rollout remain credible over the planning horizon? Does uncertainty increase on deliberately shifted states and controls? Does the optimizer choose sequences whose real outcomes agree with predicted return? The fourth test targets the composed system. A model can have excellent average error while its few optimistic mistakes dominate every optimized plan.

For the cart, partition evaluation by position, velocity, command magnitude, and horizon instead of reporting one aggregate mean. Plot residuals against ensemble variance. If high variance correlates with large residuals, the ranking contains useful information even when its scale needs calibration. Then construct an adversarial planning test: allow the optimizer a wider command interval than appears in training and inspect whether it runs to the boundary. A planner that repeatedly selects unsupported extremes has diagnosed model exploitation, not discovered free performance.

Calibration is decision dependent. A 90% predictive interval should contain roughly 90% of comparable outcomes, but marginal coverage can hide failures in rare safety-critical regions. Conditional coverage by regime and tail-loss estimates matter when failures are asymmetric. Recalibration can repair reported probabilities on a represented validation distribution; it cannot invent knowledge outside that distribution. The fallback for unsupported queries should be operational: restrict the action, invoke a conservative model, collect guarded data, or transfer control to a verified backup.

Finally, compare against simple baselines. The learned planner should beat a fixed robust controller, nominal MPC, and an ablation that ignores epistemic uncertainty on the metrics that justified its complexity. Report real interactions, computation, constraint violations, and variability across seeds. If the sophisticated method only improves predicted return while real return and violations worsen, the optimizer has exposed a validation failure. This paired evaluation is more informative than arguing from algorithm labels.

What this chapter establishes

Policy surrogates permit data reuse only locally; TRPO enforces locality through an empirical KL constraint and PPO through a clipped incentive. Model-based RL trades real interaction for learned prediction and computation. Its central failure is optimizer-driven distribution shift, which motivates feedback, active data collection, and uncertainty propagation. Gaussian processes and ensembles offer distinct approximations; PETS turns ensemble predictions into receding-horizon actions. None removes the need to validate calibration, constrain unsafe exploration, or state the scope of guarantees. The durable lesson is architectural: combine learning, planning, feedback, and safety at the layers where their assumptions are credible.