Demonstrations, rewards, and learned models

A demonstration tells the learner what an expert did. A reward tells it how an outcome is scored. A dynamics model predicts what an action will cause. These are different kinds of information, and each can be learned or supplied independently. The central design question is which information is available and how it supports the decisions the system must make.

This chapter finishes the adaptive-control comparison, then builds the bridge from supervised prediction to imitation and reinforcement learning. The cart remains a useful example: we can copy a demonstrated braking action, infer why the demonstrator values a safe stop, or learn through scored attempts. These formulations use related mathematics but answer different questions.

Grounded in AA203 lecture 14. The first nine minutes finish model-reference and model-identification adaptive control from the preceding lecture. The remainder introduces supervised learning, behavior cloning, inverse reinforcement learning, Markov decision processes, values, and algorithm tradeoffs. Derivations and finite examples below are teaching additions; specific learning algorithms are developed in subsequent chapters. The official slides add the self-tuning name, modularity, and guarantee scope of the two adaptive schemes from the preceding deck's last pages, the course roadmap and the three routes from sensing to control, the classification and quadratic-regression forms of supervised learning, the behavior-cloning recipe and ALVINN's architecture, the further differences between imitation and supervised learning, the reinforcement-learning loop with its characteristics and its chess and quadruped instances, explicit value definitions, and the algorithm taxonomy, efficiency spectrum, convergence questions, and staged training pipelines; where their notation differs, the text translates it.

Indirect adaptation inserts a model-estimation objective 3:55

The recording first reinforces a distinction from chapter 13: a simple reference can be tracked by several parameter settings, so tracking convergence does not automatically identify the true plant. Richer excitation may make parameter recovery possible, but its role must be stated separately from the tracking objective.

Model-identification adaptive control, abbreviated MIAC, estimates a model online and uses it to redesign or update a controller. Model-reference adaptive control, MRAC, instead adjusts parameters through a coupled tracking design. The distinction concerns the role of the update objective. MRAC is not stable by its name alone; a particular plant, controller, update law, and assumptions must support a proof.

Model-identification adaptive control is also called self-tuning control. It joins two modules: an estimator that fits the model's parameters to recorded input-output data, and a controller that uses the current estimate as if it were the plant. The two adaptation schemes therefore minimize different errors. In MRAC the parameters move to reduce the tracking error between the plant output and the reference model's output. In MIAC they move to reduce the model's data-fitting error, and the control objective enters only through the controller built on the fitted model. This modularity is MIAC's main advantage: an estimator such as recursive least squares can be paired with a model-based controller such as pole placement, LQR, or model predictive control, and either module can be replaced without redesigning the other.

The price is that correctness is harder to guarantee. If the signals are not rich enough, the data do not determine the parameters, the estimate need not be close to the true values, and a controller built on it need not stabilize the plant. Closed-loop operation produces exactly such data. Consider a new scalar plant \(x_{k+1}=ax_k+bu_k\) with true values \(a=2\) and \(b=1\), operated under the feedback \(u_k=-1.5x_k\). Every recorded transition satisfies \(x_{k+1}=(a-1.5b)x_k=0.5x_k\), so any estimate with \(\hat a-1.5\hat b=0.5\) fits the data with zero error. The estimate \((\hat a,\hat b)=(0.8,0.2)\) is one of them. The certainty-equivalent deadbeat controller computed from it, \(u_k=-(\hat a/\hat b)x_k=-4x_k\), makes the predicted next state zero, but the true closed loop is \(x_{k+1}=(2-4)x_k=-2x_k\), which is unstable. The estimate \((3.5,2)\) also fits the data exactly and yields the stable loop \(x_{k+1}=(2-1.75)x_k=0.25x_k\). Nothing in the fitting error distinguishes the two. Adding an excitation signal to \(u_k\) removes the exact proportionality between \(u_k\) and \(x_k\) and makes \(a\) and \(b\) identifiable, which is the role of persistent excitation in chapter 13.

MRAC's guarantee has a different scope. In the Lyapunov designs of chapter 13, the update law is derived together with the controller so that a Lyapunov function does not increase along the coupled system. Bounded signals and convergence of the tracking error then follow for any bounded reference, regardless of how rich it is, provided the plant and reference model satisfy the assumptions of the proof. The guarantee concerns tracking. Convergence of the parameter estimates to their true values still requires persistent excitation, as the first paragraph of this section notes. The chapter 13 analysis took the control law, the Lyapunov function, and the adaptation law as given; constructing them for a new plant is the difficult step, and Slotine and Li, Applied Nonlinear Control (Prentice Hall, 1991), chapter 8, develops that design direction.

A certainty-equivalent design replaces an unknown parameter by a point estimate and plans as if that value were exact. A cautious design carries a measure of parameter uncertainty into the control decision. The latter need not be a worst-case controller: a distribution can enter an expected objective, while an uncertainty set can enter a robust objective. Neither construction necessarily values information that future actions will reveal.

An original one-step calculation makes this precise. Let the cart's normalized displacement response be \(y=bu\), where gain \(b\) is uncertain, with mean \(\mu\) and variance \(\sigma_b^2\). Choose an unconstrained input to minimize squared error to target one plus effort penalty \(\lambda u^2\), \(\lambda>0\). Under a point estimate \(b=\mu\), the optimizer is

Accounting for the uncertainty distribution instead gives

For \(\mu=1\), \(\sigma_b^2=1\), and \(\lambda=0.1\), the point-estimate action is \(10/11\), while the uncertainty-aware action is \(10/21\). Their expected costs are approximately 0.91736 and 0.52381. The extra variance behaves like an additional input penalty in this particular model. With a nonlinear plant or a different objective, the effect need not reduce to this formula.

There is no future learning benefit in this one-step calculation. In a longer problem, a probing action may improve later decisions by revealing \(b\). A controller that merely penalizes present uncertainty can miss that benefit. This sets up the later distinction between cautious behavior and purposeful exploration.

Three routes from sensing to control differ in the information they require 8:53

The rest of the chapter changes the course's question. In the course's map of methods, classical control branches into feedback control and adaptive control. Optimal control branches into open-loop methods, which are indirect when they solve necessary conditions and direct when they transcribe the problem into a nonlinear program; closed-loop methods, built on dynamic programming and on the Hamilton–Jacobi–Bellman and Hamilton–Jacobi–Isaacs equations; and model predictive control, which lies between the two. The earlier chapters followed those branches. The remaining branch is adaptive optimal control, in which the controller, or the information it needs, is obtained from data, and it divides into three: imitation learning, model-free reinforcement learning, and model-based reinforcement learning. Chapter 15 develops imitation learning, chapters 16 to 18 model-free methods, and chapter 19 model-based reinforcement learning.

Two learning signals underlie much recent work on end-to-end control, in which one trained model maps raw sensor input to commands. Demonstrations supply examples of good actions; scored experience supplies a measure of how good attempted actions were. Uncut videos of household manipulation credited to \(\pi^*_{0.6}\) and to Gemini Robotics show robots preparing espresso, assembling boxes, and handling laundry. NVIDIA's Alpamayo-R1 driving model, whose training stages appear in the final section, combines supervised imitation with reinforcement learning, and reinforcement learning from human feedback is a stage in aligning large language models. Such videos report selected runs without a stated success rate or test distribution, so they show what the methods can produce, not how reliably.

Consider a legged robot with twelve actuated joints, a lidar, and a forward-facing color camera, which must cross terrain that may be hazardous. Its controller must turn each lidar point cloud into joint commands. The first route, the one followed so far in this course, is to understand the problem well enough to write a model and an objective and then design a solution. In continuous time this is the optimal control problem

with terminal cost \(h\), running cost \(g\), dynamics \(f\), admissible input set \(\mathcal U\), and initial and final times \(t_0\) and \(t_f\), the symbols of chapters 4 to 6. Its discrete-time, stochastic counterpart is the expected return \(J(\pi)\) of the MDP section below, written as a reward to maximize rather than a cost to minimize. This route requires a state \(x\) and a model \(f\). For the quadruped, \(x\) must include the body's pose and velocity, the joint angles and rates, and enough of the terrain to predict foot contacts, and \(f\) must describe intermittent contact with ground of unknown properties. The point cloud is a measurement from which such a state would have to be estimated, not the state itself.

The other two routes pose the same mapping as a learning problem, and they differ in the data they require. Imitation learning learns from demonstrations: each example pairs an observation with the command \(u^\star\) that a competent demonstrator issued there, so it names a good action. Reinforcement learning learns by trial and error: each example pairs an observation and an attempted command with a score, in the simplest case success or failure, so it grades the action that was tried without naming a better one. Neither route needs \(f\) to be written down, and both can take the raw observation as input. The modeling effort moves into data collection, and the guarantees of the first route are no longer available by construction.

The difference in information can be counted. Suppose that at one decision \(N_u\) candidate commands are equally plausible in advance and exactly one succeeds. A single demonstration identifies it. Binary feedback on one attempted command eliminates at most that command, so a learner may need \(N_u-1\) failures before the successful command is certain. If each of the quadruped's twelve joint torques were quantized to three levels, \(N_u=3^{12}=531{,}441\) at every decision. Learners do better than this worst case by generalizing across similar observations and commands, but the count shows that a score carries less information per sample than a label.

The comparison reverses for attainable performance. Behavior fitted to demonstrations is limited by what the demonstrations contain: the fitting objective rewards agreement with the demonstrator, not improvement on it, so the demonstrator's systematic errors become targets. Formulations that favor the better demonstrations in a dataset can shift the fitted behavior toward them, but not beyond what the data contain. A score has no such ceiling, because it grades outcomes rather than agreement, and this is the source of reinforcement learning's ability to find strategies that no demonstrator used. Current systems commonly combine the routes: imitation of demonstrations supplies a starting policy, and reinforcement learning on a score refines it, a pattern examined in the final section.

The material from here to the end of the course is developed at length in chapters 17 to 19 (sequential decision making and dynamic programming, reinforcement learning, and imitation learning) of Gammelli, Lorenzetti, Luo, Zardini, and Pavone, Principles of Robot Autonomy (pre-print; the version consulted is dated April 11, 2026), and in Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., MIT Press, 2018). Stanford CS 224R and Berkeley CS 185/285, both on deep reinforcement learning, cover the algorithms of the following chapters. Sutton and Barto write states and actions as \(s\) and \(a\) and index the reward that follows action \(A_t\) as \(R_{t+1}\); this chapter keeps the control notation \(x\) and \(u\) and indexes the reward earned by \((x_t,u_t)\) with \(t\).

A supervised objective identifies the statistical object being fitted 18:15

Let \(\mathcal D=\{(x_i,y_i)\}_{i=1}^N\) be input-output observations and \(f_\theta(x)\) a parameterized predictor. Squared-error fitting minimizes \(\sum_i\|y_i-f_\theta(x_i)\|^2\). A probabilistic formulation specifies a conditional distribution \(p_\theta(y\mid x)\) and maximizes its likelihood on the observations.

The task is to learn a mapping \(f:\mathcal X\to\mathcal Y\) from an input set \(\mathcal X\) to an output set \(\mathcal Y\), and the output set distinguishes the two standard problems. In regression \(\mathcal Y=\mathbb R^{d_y}\), a vector of \(d_y\) real numbers. In classification \(\mathcal Y\) is a finite set of labels, such as \(\{0,1,\ldots,C\}\), which indexes \(C+1\) classes; an image classifier maps a photograph to the label of the animal it shows. An input can be a feature vector in \(\mathbb R^{d_x}\) or an image, a three-way array indexed by color channel, height, and width, which a classifier may flatten into such a vector. The official slides write the input dimension as \(D\), the regression output set as \(\mathbb R^n\), and the dataset size as \(N\) on one page and \(n\) on another; this chapter writes \(d_x\), \(d_y\), and \(N\), so that \(D\) is not confused with the dataset \(\mathcal D\).

Regression with the quadratic model \(f_\theta(x)=\theta_1x^2+\theta_2x+\theta_3\) for scalar \(x\) shows when fitting is easy. The model is nonlinear in \(x\) but linear in \(\theta=(\theta_1,\theta_2,\theta_3)^\mathsf T\). With feature vector \(z(x)=(x^2,x,1)^\mathsf T\), matrix \(Z\) whose \(i\)th row is \(z(x_i)^\mathsf T\), and label vector \(y=(y_1,\ldots,y_N)^\mathsf T\), the mean squared error \(\frac1N\|y-Z\theta\|^2\) is a convex quadratic in \(\theta\). The factor \(1/N\) does not move its minimizer. Setting the gradient to zero gives the normal equations \(Z^\mathsf TZ\theta=Z^\mathsf Ty\), which have a unique solution when \(Z\) has full column rank, here when the data contain at least three distinct inputs. For the four points \((-1,2)\), \((0,1)\), \((1,2)\), and \((2,6)\),

The residuals \(y_i-f_{\theta^\star}(x_i)\) are \(-0.05\), \(0.15\), \(-0.15\), and \(0.05\). They sum to zero and are orthogonal to the columns \((x_i)\) and \((x_i^2)\), as the normal equations require, and the minimum sum of squared residuals is \(0.05\). Every other parabola has a larger sum, so comparing candidate curves by squared error selects \(\theta^\star\). The convexity belongs to the linear-in-parameters model, not to the squared loss: a neural network trained with the same loss is in general nonconvex in its parameters, which is the setting of the qualification on logarithms below.

For conditionally independent scalar observations with a fixed common Gaussian variance \(\sigma^2>0\), assume \(y_i\mid x_i\sim\mathcal N(f_\theta(x_i),\sigma^2)\). Taking negative logarithms of the product likelihood yields

The first term is constant in \(\theta\), so likelihood maximization and squared-error minimization have the same optimizer under this model. If variance depends on the input or is also learned, the log-variance and error-weighting terms must be retained. A loss choice is therefore an assumption about what differences matter, rather than an arbitrary interchangeable training trick.

Written per example, maximum-likelihood estimation minimizes the average negative log likelihood

Each term uses its own pair \((x_i,y_i)\), and the set notation allows several minimizers. The average makes the objective's value comparable across datasets of different sizes and turns it into an estimate of the expected loss on new data from the same distribution. For classification the conditional distribution is categorical. A network computes one score \(s_c(x)\) per class \(c\), the softmax \(p_\theta(c\mid x)=e^{s_c(x)}/\sum_{c'}e^{s_{c'}(x)}\) converts scores into probabilities, and the negative log likelihood of the observed class equals the cross-entropy between the one-hot label vector and the predicted distribution. An example whose correct class receives probability 0.7 contributes \(-\log0.7\approx0.357\); probability 0.1 would contribute \(-\log0.1\approx2.303\). Squared error and cross-entropy are therefore both maximum-likelihood objectives, under Gaussian and categorical models respectively.

Taking logs preserves maximizers of a positive likelihood and turns products into sums, avoiding direct multiplication of tiny values. It does not generally make a nonconvex objective convex or smooth away its difficult geometry. Also, a continuous probability density can exceed one; it is the integral over a region that gives a probability. These qualifications sharpen the lecture's informal numerical explanation.

Under squared loss, the population-optimal prediction is the conditional mean. Indeed, for \(m(x)=\mathbb E[y\mid x]\), expanding around \(m(x)\) gives \(\mathbb E[(y-a)^2\mid x]=\operatorname{Var}(y\mid x)+(m(x)-a)^2\). Thus the minimum occurs at \(a=m(x)\). This is often sensible for regression, but a mean of distinct valid action modes can be an invalid behavior.

Behavior cloning fits the expert's conditional action rule 23:10

Imitation learning is the class of methods that transfer a skill from an expert to a learner. In robotics the expert is usually a human operator or an existing controller, and the learner is the robot's policy. In teleoperated two-armed manipulation, for example, a person drives the robot through a task, such as handling a drill and its bits, and the robot logs its own states together with the commands it received; the person's skill arrives as labels already expressed in the robot's state and control spaces. Given a dataset \(\mathcal D=\{(x_i,u_i)\}_{i=1}^N\) of state-control pairs collected from the expert, imitation learning divides into two families. Behavior cloning learns the policy the expert used, a conditional distribution \(\pi_\theta(u\mid x)\). Inverse reinforcement learning, also called inverse optimal control, learns the objective the expert appears to optimize, a reward that assigns a real number to each state-control pair, from which a policy must still be computed. Behavior cloning occupies this section and the next; inverse reinforcement learning has its own section below.

In behavior cloning, the labels are expert controls. A dataset contains state-action pairs \((x_i,u_i)\), collected from a human operator, an existing controller, or another source of competent behavior. A deterministic policy can minimize squared action error; a stochastic policy \(\pi_\theta(u\mid x)\) can minimize negative log likelihood. For discrete actions, the latter is the usual cross-entropy objective.

The procedure therefore has two steps. First, collect the expert demonstrations \(\mathcal D\). Second, fit the policy by maximum likelihood,

which is the supervised objective of the previous section with the control \(u\) in the role of the label \(y\). The losses commonly listed for this step are instances of it rather than alternatives. If \(\pi_\theta(u\mid x)\) is Gaussian with mean \(\bar u_\theta(x)\) and fixed covariance \(\sigma^2I\), its negative log likelihood is \(\|u-\bar u_\theta(x)\|^2/(2\sigma^2)\) plus a constant, so maximum likelihood is mean-squared-error regression on the controls. If the controls are discrete and \(\pi_\theta\) is categorical, it is cross-entropy. The assumed action distribution is the only difference, and it decides whether the fitted policy can represent more than one good action at a state, as the multimodality example below shows.

The trained policy maps its input representation to an action. It does not need to learn an explicit transition model or be given a task reward to fit those labels. The word “expert” describes the intended role of the demonstrator, not a mathematical guarantee that every label is optimal. Inconsistent, delayed, or suboptimal demonstrations affect what the objective teaches.

The source uses ALVINN as an early example of learning a steering rule from sensory input. The original ALVINN paper describes a neural road-following system. The enduring connection is the perception-to-action learning setup, rather than an inference that its experimental conditions establish general autonomous-driving competence.

The paper's numbers make the setup concrete. The input layer has a \(30\times32\) video retina, an \(8\times32\) laser-range-finder retina, and one road-intensity feedback unit, 1217 inputs fully connected to 29 hidden units, which are fully connected to 46 outputs. Forty-five outputs represent turn curvature, from sharp left through straight ahead to sharp right; the forty-sixth reports whether the road appeared lighter or darker than its surroundings and is fed back as an input for the next image. The network therefore has \(1217\cdot29+29\cdot46=36{,}627\) connection weights, not counting biases. Its labels were not recorded from a human driver. It was trained on 1200 simulated road images, each labeled with the curvature that would bring the vehicle to the road center 7 m ahead. After 40 training epochs it chose a curvature within two units of the correct one about 90% of the time on new simulated images, and it drove the NAVLAB test van at 0.5 m/s along a 400 m wooded path on the Carnegie Mellon campus in sunny fall conditions. The expert was a geometric steering rule applied to synthetic images, an instance of an existing controller serving as the demonstrator, and the field result covers one path under stated conditions.

For the cart, suppose demonstrations contain states \(x=(p,v)\) and braking forces \(u\). The learned action must be compatible with the actuator bounds, and its input must retain enough information for the decision. Two carts at the same position with opposite velocities generally need different actions. A policy seeing position alone may average conflicting labels caused by hidden velocity; a larger network cannot recover information that its input has discarded.

Multimodality creates a related problem. At an obstacle, an expert may turn left with action \(-1\) or right with action \(+1\). If both appear equally often under the same observed input, squared loss prefers zero. That is the arithmetic mean, but it may point directly into the obstacle. A distribution over actions, a richer conditioning signal, or a temporally structured action representation may be needed. Which remedy is appropriate depends on whether the ambiguity is legitimate choice or missing information.

ALVINN's output layer is an early example of the first remedy. Its training target was a hill of activation over the nine curvature units around the correct one, with values 0.10, 0.32, 0.61, 0.89, 1.00, 0.89, 0.61, 0.32, and 0.10 and zeros elsewhere, and at test time the vehicle followed the curvature of the most active unit. Under squared loss the population-optimal output is the average target vector. If left and right curves were equally frequent for the same input and their hills did not overlap, that average would consist of two hills of height one half, the most active units would mark the two modes, and the selection would return one of them; a single regressed curvature would return their mean. The representation cannot remove ambiguity caused by missing information, but it does not average two valid choices into an invalid one.

A policy changes the inputs on which it will later be evaluated 31:20

Ordinary behavior-cloning training measures error on states visited by the demonstrator. Deployment visits states induced by the learned policy. If a slightly wrong action changes the next state, the learner can encounter states with little or no demonstration support, where its error can grow further. This feedback between prediction and data distribution is a central distinction from a fixed supervised test distribution.

Let \(d_t^{\pi_E}\) be the expert's state distribution at time \(t\) and \(d_t^\pi\) the learner's. The training loss estimates an expectation under the former; the task depends on the latter. Neither a low average training loss nor a random split of adjacent demonstration frames directly certifies closed-loop performance.

This mismatch is called covariate shift: the distribution of the inputs \(x\) changes from \(d_t^{\pi_E}\) to \(d_t^\pi\), while the conditional action rule being imitated stays the same. It is the main difference between imitation and ordinary supervised learning, and the bound below shows how it interacts with the dynamics.

Error growth need not always be exponential. An original deterministic bound makes the dependency explicit. Suppose the expert closed-loop transition \(g_E(x)=f(x,\pi_E(x))\) is \(L\)-Lipschitz, dynamics are \(L_u\)-Lipschitz in action, and the learner's action error is uniformly at most \(\epsilon\) along its visited states. If \(\delta_t\) is the distance between learner and expert trajectories from the same initial state, then

For \(L<1\), the bound stays below \(L_u\epsilon/(1-L)\). For \(L=1\), it grows linearly. For \(L>1\), it can grow geometrically. The source's compounding-error intuition is correct, while its exact growth depends on dynamics and feedback. Moreover, an average error on expert data is weaker than the uniform visited-state assumption used in this bound.

Covariate shift is not the only difference. A second is structure in the output. An image classifier may output any label in its vocabulary at any time, but a joint-angle command must respect joint limits and actuator bounds, and successive commands must respect rate limits and contact conditions that couple them in time. A policy fitted one sample at a time sees each label in isolation. Unless the policy class enforces these constraints, for instance by saturating its output or by predicting short sequences of actions together, a small error can produce an infeasible command, or a sequence of individually plausible commands that is infeasible as a whole.

A third difference is a mismatch between the source of the demonstrations and the deployed system. Videos of people performing tasks are plentiful, but using them to control a humanoid robot crosses an embodiment gap. The videos contain no robot commands, so the pairs \((x_i,u_i)\) must first be inferred, and a human body differs from the robot's in link lengths, joint limits, mass distribution, and actuation, so a human motion need not be feasible for the robot. This domain difference exists before the learned policy takes any action, which separates it from the covariate shift that the policy's own errors create.

A fourth difference is cost. Labeled images can be collected at internet scale, whereas each demonstration requires a person or a controller to operate a physical system in real time. One hour of teleoperation at a 50 Hz command rate yields \(3600\cdot50=180{,}000\) state-control pairs, but consecutive pairs are strongly correlated, so the count overstates the independent information they contain. The correlation is also why a random split of adjacent frames, mentioned above, measures interpolation within demonstrated trajectories rather than performance on new ones.

Chapter 15 develops this distribution mismatch and an interactive remedy. Here the essential lesson is that imitation is a sequential decision problem whose fitting step happens to resemble supervised learning.

Inverse reinforcement learning infers an objective, with ambiguities 36:00

Inverse reinforcement learning, or IRL, tries to infer a reward from behavior. Inverse optimal control is the closely related control terminology. The output is not automatically a policy: after fitting an objective, one still needs a planning or learning procedure to optimize it in the relevant environment.

The object sought is a reward \(R:\mathcal X\times\mathcal U\to\mathbb R\) on state-control pairs, where \(\mathcal X\) and \(\mathcal U\) are the sets of states and controls. The shaping argument below uses a transition reward \(r(x,u,x')\) instead; its conditional expectation \(R(x,u)=\mathbb E[r(x,u,x')\mid x,u]\) is the reward of the MDP in the next section. Inverse reinforcement learning is therefore one of the two families within imitation learning, complementary to behavior cloning: both start from the same expert data and differ in the object fitted, a policy in one case and an objective in the other.

Objective inference is most useful when good behavior is easier to demonstrate than to specify. Driving is the standard case. Demonstrations are inexpensive to collect, since people drive anyway, while a numerical definition of good driving is difficult to write and differs from driver to driver. An inferred reward offers a candidate definition that a planner can then optimize, subject to the ambiguities derived below.

The lecture's warehouse example contrasts copying demonstrated routes with inferring preferences such as reaching a goal while avoiding obstacles. A reusable objective might support replanning after the layout changes. This is a possibility, not a guarantee: the learned features may omit the new hazard, or several objectives may explain the original demonstrations equally well.

Nonuniqueness is fundamental. Multiplying every reward by a positive constant preserves policy rankings. A constant zero reward makes every policy optimal, so merely asking for a reward under which the expert is optimal admits an uninformative answer. Additional structure, regularization, assumptions about expert choice, or comparisons between trajectories are needed to select a useful explanation.

A stronger ambiguity appears through potential-based shaping. For bounded state function \(\Phi\), define a transformed transition reward by

Along an infinite discounted trajectory, the extra terms telescope to \(-\Phi(x_0)\), because the final discounted potential tends to zero. This offset depends on the initial state but not on the chosen policy. Policy rankings from that state are unchanged although the displayed local rewards differ. In a finite horizon, terminal potential handling must be specified so that a residual policy-dependent terminal term does not spoil this conclusion.

This derivation explains why behavior does not uniquely reveal an internal reward scale or local reward assignment. An inferred objective should be assessed by its predictive and control usefulness, with its identification assumptions visible.

An MDP separates a policy from the environment it controls 38:14

Reinforcement learning is a formalism for decision making learned from experience. Its basic loop has two parties. At time \(t\) the agent, meaning the policy being learned, observes the state \(x_t\) and selects a control \(u_t\); the environment, meaning the system being controlled, returns the next state \(x_{t+1}\) and a scalar reward \(r_t\). Unrolled over time, the loop produces a trajectory of states, controls, and rewards. Almost every reinforcement-learning algorithm collects such trajectories and uses them to change the policy so that later trajectories score better, which is the precise sense of learning by trial and error.

Learning from a score rather than from labels removes the demonstrator's ceiling described in the section on the three routes. An agent trained on an Atari video game from its score alone eventually exploited a strategy that its designers had not programmed. In the 2016 match between DeepMind's AlphaGo and Lee Sedol, move 37 of the second game was a play that professional commentators had not anticipated. The boundary of this example matters. The published design of AlphaGo (Silver and colleagues, “Mastering the game of Go with deep neural networks and tree search,” Nature, 2016) trains its networks by supervised learning from human expert games followed by reinforcement learning from self-play, and it selects moves by a tree search guided by those networks. The novel move came from a staged system in which imitation supplied the starting point, the structure to which the final section returns.

A second attraction is generality in the objective. Reinforcement-learning algorithms need only sampled values of the reward, not its derivatives, convexity, or closed form, so they can optimize objectives that are awkward for the optimality conditions and gradient-based solvers of earlier chapters. Legged locomotion on rough terrain is the leading example: training in physics simulation, with many simulated robots running in parallel across varied terrain, is now a common way to obtain locomotion controllers without deriving contact models for the controller. The simulator itself contains a physics model. The learned policy does not use that model explicitly; it absorbs its consequences through sampled experience, which is the sense in which such training is model-free, as the final section discusses.

Three features distinguish reinforcement learning from supervised learning, and each shapes the algorithms of later chapters. First, there is no supervision: no example states which action was correct, only how much reward the attempted action earned. Second, feedback can be delayed. A chess program that learns from game outcomes receives a nonzero reward only at the end, \(+1\) for a win, \(-1\) for a loss, and commonly \(0\) for a draw; deciding which of its many moves deserves the credit or blame is the credit-assignment problem, and value functions are the main tool for it. Third, the data are not independent and identically distributed. The controls chosen now determine the states visited later, so the policy shapes the distribution of its own training data, as the trajectory factorization below makes explicit. A learner that only exploits its current best guess may never visit the states and actions that would reveal a better one, while a learner that explores pays for that information with lower reward during learning. This tension between exploration and exploitation has no counterpart in fitting a fixed dataset, and the dependence between samples is one reason reinforcement-learning updates are less stable than supervised ones.

A Markov decision process is specified here by state space \(\mathcal X\), action space \(\mathcal U\), transition kernel \(T(x'\mid x,u)\), expected immediate reward \(R(x,u)\), and discount \(\gamma\). A policy \(\pi(u\mid x)\) assigns an action distribution at each state. Deterministic policies are included as point-mass choices. The transition kernel describes the actual environment; a learned model \(\widehat T\) is an approximation used by the agent.

Collected as a tuple, the process is \(\mathcal M=(\mathcal X,\mathcal U,T,R,\gamma)\). The value function, defined in the next section, measures long-run performance, while the reward measures immediate success; the model is optional, and whether an algorithm learns or uses one separates model-based from model-free methods. The official slides write the transition as \(T(x_t\mid x_{t-1},u_{t-1})=p(x_t\mid x_{t-1},u_{t-1})\), the same kernel indexed one step earlier, write a learned model as \(p(\hat x_{t+1}\mid x_t,u_t)\), and use \(T\) also for the horizon of a trajectory; this chapter writes the horizon as \(H\) to keep \(T\) for the kernel. The slides take \(\gamma\in(0,1)\); the chapter also admits \(\gamma=0\), a purely myopic objective.

State means information sufficient for the conditional future, not necessarily a raw image or position measurement. For the cart, position and velocity are needed even in the simple acceleration model. Hidden payload, actuator delay, or unknown disturbance dynamics may require an augmented state or a partially observed formulation. A quadruped's pose alone similarly omits velocity and other quantities needed by many physical models.

Let \(\rho_0\) be the initial-state distribution and take a finite trajectory \(\tau=(x_0,u_0,x_1,\ldots,u_{H-1},x_H)\). Under a Markov policy and transition model,

For continuous variables this notation denotes the corresponding density or probability-kernel factorization. The policy affects the trajectory distribution through every action factor, and hence changes future state occupancy. The learning objective is \(J(\pi)=\mathbb E_{\tau\sim p_\pi}[\sum_{t=0}^{H-1}\gamma^tR(x_t,u_t)]\), with a terminal reward added if the task uses one.

The index range is what makes \(p_\pi\) a distribution over the stated trajectory. A product over \(t=1,\ldots,H\) of \(\pi(u_t\mid x_t)T(x_{t+1}\mid x_t,u_t)\) would omit the first control factor \(\pi(u_0\mid x_0)\) and include a transition to \(x_{H+1}\), a state outside \(\tau\). The two sources of randomness, the policy's choice of control and the environment's choice of next state, alternate from \(t=0\) to \(t=H-1\). The optimal policy is \(\pi^\star\in\arg\max_\pi J(\pi)\), with the infinite discounted sum in place of the finite one when the task has no horizon.

For an infinite horizon, bounded rewards and \(\gamma<1\) ensure an absolutely bounded return. Discounting alone does not make arbitrary unbounded rewards well behaved, nor does it make optimization computationally easy. Sparse delayed reward can leave credit assignment and exploration difficult even when the expected return exists.

Discounting also states a preference between immediate and delayed reward. A reward \(k\) steps ahead is weighted by \(\gamma^k\) relative to one received now. With \(\gamma=0.9\), a reward ten steps ahead counts \(0.9^{10}\approx0.349\) as much; with \(\gamma=0.99\), it counts \(0.99^{10}\approx0.904\) as much, and a reward one hundred steps ahead counts \(0.99^{100}\approx0.366\) as much. The weights sum to \(1/(1-\gamma)\), 10 and 100 in these two cases, a scale often called the effective horizon. Choosing \(\gamma\) is therefore part of specifying the task, not only a device for making infinite sums converge.

Posing a problem as reinforcement learning begins by naming each element of the MDP. Two examples show how much modeling the naming contains.

Each entry is a modeling decision with consequences. For chess, the piece positions are not a sufficient state: castling rights, the possibility of an en passant capture, the side to move, and the position history used by the repetition and fifty-move draw rules also determine which continuations are legal and how the game can end. The board update produced by the agent's own move is deterministic, but the next state at which the agent decides includes the opponent's reply, so from the agent's side the transition is deterministic only against a fixed deterministic opponent. A win-or-loss reward also needs a value for a draw, and it is nonzero only at the end of the game, which makes chess the standard instance of delayed credit assignment.

For the quadruped, the listed state omits velocities, which the dynamics require, as noted above, and the torque-to-motion map is deterministic only given the terrain, which the robot perceives incompletely. The reward is a design decision too. Suppose a policy steps forward with probability \(q\) of falling at each step, a fall ending the episode. Its expected reward per attempted step is \(10(1-q)-5q=10-15q\), which exceeds the zero earned by standing still whenever \(q\lt2/3\). Exercise 7 shows that the discounted comparison has the same threshold for every \(\gamma\), so this reward prefers walking even when each step falls with probability one half. A designer who wants caution must put it into the reward.

Values distinguish a profitable future from a tempting immediate reward 59:42

The state value \(V^\pi(x)\) is the expected discounted return from state \(x\), following policy \(\pi\). The action value \(Q^\pi(x,u)\) fixes the first action and then follows \(\pi\). They satisfy

Written out, the two definitions are

where every control not fixed by the conditioning is drawn from \(\pi(\cdot\mid x_{t'})\), every next state from \(T(\cdot\mid x_{t'},u_{t'})\), and a deterministic policy enters as \(u_{t'}=\pi(x_{t'})\). The exponent counts steps from the current time \(t\). Weighting by \(\gamma^{t'}\) instead would multiply the value of every state reached at time \(t\) by \(\gamma^t\), so the value would depend on the clock as well as on the state. Policy rankings from a fixed starting time would be unchanged, but the stationary Bellman relations above, which use the same \(V^\pi\) at the successor, would fail. Most reinforcement-learning algorithms in the following chapters estimate or manipulate these functions and their optimal versions \(V^\star(x)=\max_\pi V^\pi(x)\) and \(Q^\star(x,u)=\max_\pi Q^\pi(x,u)\). The maxima are taken separately at each argument, and in a finite discounted MDP a single stationary deterministic policy attains all of them at once, which is why the greedy rule below is well defined.

The optimal versions maximize over admissible policies. In a finite discounted MDP, an optimal stationary policy can be selected greedily from \(Q^\star\). Maximizing \(Q^\pi\) instead gives a policy-improvement step, not generally an immediately globally optimal policy. Value functions of arbitrary policies are only partially ordered: one policy can be better in one state and worse in another.

Consider an original three-state abstraction of a transfer task: resting \(s_0\), moving \(s_1\), and completed \(g\). From \(s_0\), “launch” gives reward \(-1\) and moves to \(s_1\); “idle” gives reward \(0.1\) and remains at \(s_0\). From \(s_1\), completion gives reward 3 and moves to absorbing \(g\), where future rewards are zero. Set \(\gamma=0.9\).

Completing has value 3, so launching has value \(-1+0.9(3)=1.7\). Idling forever gives \(0.1/(1-0.9)=1\). The optimal action is launch, despite its worse immediate reward. At the optimum, the value of idling once and then acting optimally is \(0.1+0.9(1.7)=1.63\), below launch's 1.7.

A policy that launches with probability \(p\) at each visit to \(s_0\) has

At \(p=1/2\), this equals \(18/11\approx1.63636\). This simple equation already includes repeated delays under stochastic actions; replacing it with the average immediate reward would solve a different problem.

Algorithm categories describe information use, not universal rankings 1:05:49

Model-based methods use a supplied or learned transition model for planning or improvement. Model-free methods can learn a policy or value without explicitly fitting that transition kernel. “Model-free” does not mean no statistical assumptions, no architecture, or no simulator during training. A policy trained by interacting with a physics simulator can still be model-free in the algorithmic sense.

Model-based methods divide further by where the model comes from. If the transition model is given, the problem is the optimal control problem of earlier chapters, and dynamic programming, LQR, iLQR, and model predictive control apply directly. If it must be learned, a parameterized predictor \(f_\theta(x_t,u_t)\) of the next state is fitted to observed transitions by the supervised learning of this chapter and then used for planning or policy improvement, the route of system identification and of chapter 19. A deterministic predictor approximates the kernel \(T\) only through a summary such as its conditional mean; representing stochastic transitions requires a learned distribution \(\widehat T_\theta(x'\mid x,u)\).

Policy optimization directly adjusts policy parameters toward a return objective. Value-based methods estimate values and derive action choices from them. Actor–critic methods combine both. Sample efficiency counts the environment experience required to obtain a useful policy; wall-clock efficiency also depends on computation, parallel simulation, communication, and optimization overhead.

In symbols, policy optimization adjusts \(\theta\) to increase \(J(\pi_\theta)=\mathbb E_{\tau\sim p_{\pi_\theta}}[\sum_{t=0}^{H-1}\gamma^tR(x_t,u_t)]\) directly, while a value-based method defines its policy implicitly by \(\pi(x)\in\arg\max_uQ(x,u)\). A policy defined from a state value \(V\) alone is not model-free at decision time: as Exercise 5 shows, selecting an action from \(V\) requires the reward and transition model to evaluate \(R(x,u)+\gamma\mathbb E[V(x')\mid x,u]\).

The number of algorithms reflects three kinds of difference. The first is a tradeoff between sample efficiency and stability, meaning whether training converges reliably without extensive tuning. The second is the assumptions a problem satisfies: stochastic or deterministic transitions, continuous or discrete states and actions, and an episodic task that can be reset, such as a game of chess, or a continuing infinite-horizon task without resets. The third is which object is easier to represent. Balancing a pole on a cart by feedback needs only a rule such as moving the cart toward the side to which the pole is falling, while a model accurate over the full range of pole angles must capture the trigonometric coupling between the pole's swing and the cart's acceleration. A person adjusting a shower valve by feel uses a simple policy without any model of heat transfer in the water. When the policy is simpler than the model, a model-free method has less to learn; when the dynamics are simple and the task changes, a learned model can be reused across tasks.

An off-policy method can use experience generated by a behavior policy different from the policy being evaluated or improved. This does not make arbitrary old data valid for every estimator. Coverage and distribution correction matter. If the behavior policy never takes an action that the target policy may choose, simple importance weighting cannot recover that action's outcome from absent data.

For a one-step illustration, let the target choose action A with probability 0.8 and the behavior choose it with probability 0.2. Rewards are 2 for A and minus one for B. The target expectation is 1.4, but the raw behavior average is minus 0.4. Weights \(\pi(A)/\beta(A)=4\) and \(\pi(B)/\beta(B)=0.25\) restore the correct expectation. Long sequential ratios can have much larger variance, which helps explain why reusing experience is not computationally free.

An on-policy method, by contrast, needs fresh samples from the current policy every time the policy changes, even slightly, because its update estimates expectations under the current trajectory distribution. Whether an algorithm can reuse old experience is therefore a central question for its sample efficiency. A commonly used ordering, adapted from Berkeley's CS 285, runs from fewer to more samples needed: model-based methods with simple function classes, model-based deep reinforcement learning, off-policy Q-function learning, actor–critic methods, which lie on either side of the off-policy and on-policy divide depending on their design, on-policy policy-gradient methods, and evolutionary or other gradient-free methods. The ordering describes typical requirements; it is not a theorem, and the sample counts it compares depend on the task and the implementation.

Less sample-efficient methods remain in use because wall-clock time is not sample efficiency. On-policy methods are often the more stable ones, and when experience comes from a fast parallel simulator, their sample cost can be small in time. If a simulator supplies \(2\times10^5\) transitions per second, \(10^8\) transitions take 500 s, about 8.3 minutes; the same number collected by one physical robot at 50 Hz takes \(2\times10^6\) s, about 23 days. On hardware, sample efficiency dominates the choice; in inexpensive simulation, stability and computational throughput can matter more.

Stability raises three questions about an algorithm: whether it converges, to what, and whether it does so for every problem in its intended class. Supervised learning answers them almost always through gradient descent on a fixed loss. Reinforcement learning often does not descend the gradient of any fixed objective. Q-learning, for example, is a sampled fixed-point iteration that moves an estimate toward the image of the Bellman optimality map \((\mathcal BQ)(x,u)=R(x,u)+\gamma\mathbb E[\max_{u'}Q(x',u')\mid x,u]\). On a finite MDP with exact expectations this map is a contraction, \(\max_{x,u}|(\mathcal BQ)(x,u)-(\mathcal BQ')(x,u)|\le\gamma\max_{x,u}|Q(x,u)-Q'(x,u)|\), because a maximum over \(u'\) changes by no more than the largest change in its arguments. Repeated application therefore converges to the unique fixed point \(Q^\star\) from any start.

With function approximation and sampled updates, that guarantee can fail, as the following example of off-policy temporal-difference prediction of state values, a simpler relative of Q-learning, shows. In a two-state example of Sutton and Barto, every reward is zero, so every true value is zero. The two states share one parameter \(w\) through features 1 and 2, so their estimated values are \(w\) and \(2w\), and the only transition being updated goes from the first state to the second. The semi-gradient temporal-difference update \(w\leftarrow w+\alpha(0+\gamma\cdot2w-w)\cdot1\), with step size \(\alpha\gt0\), multiplies \(w\) by \(1+\alpha(2\gamma-1)\), which exceeds one whenever \(\gamma\gt1/2\). With \(\alpha=0.1\) and \(\gamma=0.9\) the factor is 1.08, and 100 updates multiply any nonzero \(w\) by about 2200. The parameterization represents the true values exactly with \(w=0\), yet the iteration moves away from that point. Chapters 16 and 17 develop the conditions under which such updates converge.

Model-based reinforcement learning has a different gap: the model is fitted to predict transitions, typically by maximum likelihood, not to maximize the reward obtained by the policy planned with it. Lower prediction error need not mean better decisions. Suppose a scalar successor state is also the reward, control \(u_1\) truly leads to 1.0 and \(u_2\) to 0.9, and the planner chooses the control with the larger predicted successor. A first model predicts 0.85 and 0.9, a mean squared error of 0.01125, and chooses \(u_2\). A second model predicts 1.3 and 1.1, a mean squared error of 0.065, about 5.8 times larger, and chooses \(u_1\). The more accurate model loses 0.1 of reward, because its accuracy is averaged over predictions while the decision depends only on their ordering. The certainty-equivalent controller of the first section has the same structure: an estimate fitted for one purpose is used as if it were exact for another.

Viewed as alternatives, imitation and reinforcement learning have complementary strengths and weaknesses.

Each entry has been made precise in this chapter: the information in a label compared with a score, the demonstrator's ceiling, covariate shift, the convergence questions above, and reward design in the quadruped example. The efficiency of imitation counts samples, not cost: a demonstration names a good action, but each demonstration costs operator time.

Current systems use the two as stages rather than alternatives. NVIDIA's Alpamayo-R1, a vision-language-action model for autonomous driving, is trained in four stages. A vision-language model (Cosmos Reason) is first trained on general reasoning data for world knowledge. Pre-training on driving data adds an action output and teaches nominal driving. Supervised fine-tuning on reasoning traces, a form of imitation, initializes the model's reasoning. Reinforcement-learning post-training then strengthens that reasoning, aligns behavior, and addresses rare long-tail scenarios.

Reinforcement learning from human feedback, as used for InstructGPT by Ouyang and colleagues (“Training language models to follow instructions with human feedback,” 2022), connects all three topics of this chapter. Its first step is behavior cloning: labelers write demonstrations of desired outputs, and a pretrained GPT-3 model is fine-tuned on them by supervised learning.

Its second step is reward learning from comparisons, the kind of additional information the section on inverse reinforcement learning says is needed to select a useful reward. Labelers rank between 4 and 9 sampled outputs for a prompt; \(K\) ranked outputs give \(K(K-1)/2\) pairwise comparisons, from 6 to 36. For each pair in which output \(o^+\) was preferred to output \(o^-\), a reward model \(\widehat R_\psi\) with parameters \(\psi\) is trained to minimize \(-\log\operatorname{sig}\big(\widehat R_\psi(o^+)-\widehat R_\psi(o^-)\big)\), where \(\operatorname{sig}(z)=1/(1+e^{-z})\). Only reward differences enter, so adding a constant to every reward for a prompt changes nothing, the same additive ambiguity derived for inverse reinforcement learning; the logistic link fixes the scale, since a difference of 1 means a predicted preference probability of \(\operatorname{sig}(1)\approx0.731\).

Its third step is reinforcement learning: the supervised policy is optimized against the learned reward with proximal policy optimization, in an episode consisting of one prompt and one response, with a penalty \(\kappa\log\big(\pi^{\rm RL}(o)/\pi^{\rm SFT}(o)\big)\), for coefficient \(\kappa\gt0\), that keeps the optimized policy \(\pi^{\rm RL}\) close to the supervised policy \(\pi^{\rm SFT}\). The authors add the penalty to mitigate over-optimization of the reward model: without it, the optimized policy can move toward outputs, far from the supervised policy's, on which the learned reward is inaccurate.

The lecture concludes that imitation and reinforcement learning can be stages in one pipeline. Demonstrations can provide an initial policy; interaction can improve a specified task score; constraints and validation still determine whether the resulting behavior is acceptable. The benefit is complementary information, not a guarantee attached to any sequence of algorithm names.

Executable checks: uncertainty, delayed value, and data reuse

import numpy as np

mu, variance, effort = 1., 1., .1
ce = mu/(mu*mu+effort)
cautious = mu/(mu*mu+variance+effort)
expected_cost = lambda u: (mu*mu+variance+effort)*u*u-2*mu*u+1
assert expected_cost(cautious) < expected_cost(ce)
assert abs(expected_cost(cautious)-11/21) < 1e-12

gamma, p = .9, .5
P = np.array([[1-p, p, 0.], [0., 0., 1.], [0., 0., 1.]])
reward = np.array([p*(-1)+(1-p)*.1, 3., 0.])
V = np.linalg.solve(np.eye(3)-gamma*P, reward)
closed_form = (1.7*p+.1*(1-p))/(1-gamma*(1-p))
assert np.allclose(V, [closed_form, 3., 0.])
assert abs(V[0]-18/11) < 1e-12
assert -1+gamma*V[1] > .1+gamma*V[0]
assert abs(1.7-(.1+.9*1.7)-.07) < 1e-12

target = np.array([.8, .2])
behavior = np.array([.2, .8])
payoff = np.array([2., -1.])
weighted = behavior @ ((target/behavior)*payoff)
assert abs(weighted-target@payoff) < 1e-12
assert abs(weighted-1.4) < 1e-12

# Equal demonstrations in two incompatible modes:
labels = np.array([-1., 1.])
assert np.mean(labels) == 0.
assert np.mean((labels-0.)**2) < np.mean((labels-1.)**2)
print("Expected costs, point estimate versus cautious:",
      expected_cost(ce), expected_cost(cautious))
print("Mixed-policy rest value:", V[0])
print("Importance-corrected reward:", weighted)

A second block checks the constructions added from the official slides. Its inputs are the four regression points, the quadruped's rewards of \(+10\) per step and \(-5\) per fall with fall probability \(q\) and discount \(\gamma\), the two-state temporal-difference example, the two model predictions, and ALVINN's layer sizes. It asserts the normal-equation solution \((5/4,1/20,17/20)\) with residuals orthogonal to the features, the cross-entropy term \(-\log0.7\), the walking value \(50/11\) at \(q=1/2\) and \(\gamma=0.9\), the threshold \(q=2/3\) for four values of \(\gamma\), the fall penalty of 990 that moves the threshold to \(q=0.01\), the growth factor \(1.08^{100}\), the reversal in which the lower-error model chooses the worse control, the two exactly fitting self-tuning estimates of which one destabilizes the plant, the invariance of the preference probability to a common reward offset, and the weight count 36,627. It assumes, as the text does, that a fall ends the episode and that standing still earns zero forever. It uses NumPy only and prints the fitted coefficients, the walking value, and the two models' errors.

import numpy as np

x = np.array([-1., 0., 1., 2.])
y = np.array([2., 1., 2., 6.])
Z = np.stack([x*x, x, np.ones_like(x)], axis=1)
theta = np.linalg.solve(Z.T @ Z, Z.T @ y)
assert np.allclose(theta, [5/4, 1/20, 17/20])
residual = y - Z @ theta
assert np.allclose(Z.T @ residual, 0.)
assert abs(residual @ residual - .05) < 1e-12
assert abs(-np.log(.7) - .357) < 1e-3

walk = lambda q, gamma, fall=-5.: (10*(1-q)+fall*q)/(1-gamma*(1-q))
assert abs(walk(.5, .9) - 50/11) < 1e-12
for gamma in (0., .5, .9, .99):
    assert abs(walk(2/3, gamma)) < 1e-12
    assert walk(.6, gamma) > 0 > walk(.7, gamma)
assert abs(walk(.01, .9, fall=-990.)) < 1e-9

alpha, gamma, w = .1, .9, 1.
for _ in range(100):
    w += alpha*(gamma*2*w - w)
assert abs(w/1.08**100 - 1) < 1e-9 and 2199 < w < 2201

true = np.array([1., .9])
first, second = np.array([.85, .9]), np.array([1.3, 1.1])
mse = lambda pred: np.mean((pred-true)**2)
assert abs(mse(first)-.01125) < 1e-12 and abs(mse(second)-.065) < 1e-12
assert true[np.argmax(first)] == .9 and true[np.argmax(second)] == 1.

a, b = 2., 1.
for a_hat, b_hat in ((.8, .2), (3.5, 2.)):
    assert abs(a_hat - 1.5*b_hat - .5) < 1e-12
assert abs(a - b*.8/.2) > 1 > abs(a - b*3.5/2.)

sig = lambda z: 1/(1+np.exp(-z))
assert abs(sig(1.) - .731) < 1e-3
assert abs(sig((3.+5.)-(2.+5.)) - sig(3.-2.)) < 1e-15
assert 30*32+8*32+1 == 1217 and 1217*29+29*46 == 36627
print("Quadratic fit:", theta, "SSE:", residual @ residual)
print("Walking value at q=0.5, gamma=0.9:", walk(.5, .9))
print("Model errors:", mse(first), mse(second))

Seven exercises with complete solutions

The next chapter turns behavior cloning's distribution mismatch into a quantitative sequential-learning problem, then examines how expert queries and richer action distributions address different failure modes.