Corrective data and generative policies

A policy can fail because it visits unfamiliar states, because the observation omits crucial history, or because its action model averages incompatible choices. These failures call for different repairs. More parameters cannot replace missing corrective labels, and collecting more identical demonstrations cannot make a unimodal action distribution represent two distinct routes around an obstacle.

This chapter develops the lecture's three intervention points: targeted learning algorithms, deliberate data collection, and expressive policy representations. A small disturbed-position controller supplies a complete data-aggregation example. A separate Gaussian transport calculation makes flow matching concrete without requiring a large neural-network training run.

Grounded in AA203 lecture 15. The recording covers compounding errors and multimodality, DAgger, corrective camera data, observation history, action distributions, flow matching, action chunking, outcome-conditioned supervised learning, and a brief IRL overview. The probability-versus-loss distinction, exact transport fixture, and derivational qualifications below are teaching extensions. The official slides add the behavior-cloning objective, the cited quadratic bound, the indexed DAgger loop and its query variants, the network and label-adjustment details of the two camera case studies, finite-history and conditional-VAE policies, the diffusion forward process, chunk indexing, the Diffusion Policy and RT-1 systems, an assessment of behavior cloning, the filtering, weighting, and conditioning objectives of RvS, and the principles behind the named IRL methods; where their notation differs, the text translates it.

Probability of failure and accumulated loss scale differently 5:01

Behavior cloning begins with a dataset \(\mathcal D=\{(x_n,u_n)\}_{n=1}^{N}\) of \(N\) state-action pairs recorded while an expert performs the task, and it treats imitation as supervised learning. Supervised learning fits a parameterized map \(f_\theta\) from inputs to labels by minimizing a measure of prediction error on a dataset, \(\theta^\star\in\arg\min_\theta\mathcal L(\theta;\mathcal D)\): a classification loss when the labels are class indices, such as the object categories \(\{0,1,\ldots,C\}\) of an image, or the mean squared error when they are vectors. Behavior cloning uses the expert's actions as the labels. For a policy network that outputs a single action \(\pi_\theta(x)\) at each state, the squared-error objective is

and the fitted policy is then deployed. Chapter 14 derives the likelihood form of this objective, in which \(\pi_\theta(u\mid x)\) is a distribution over actions, and shows that the population optimum of the squared error is the conditional mean of the demonstrated actions.

The supervised template brings two assumptions that sequential control violates. Statistical learning theory treats training inputs as independent draws from the same distribution that will generate the test inputs, whereas a policy's own actions determine the states on which it is later tested. A point prediction trained by squared error estimates the conditional mean, which misrepresents the data when several distinct actions are equally valid. The first violation produces compounding errors, analyzed in this section; the second produces mode averaging, analyzed in the section on action distributions. Targeted algorithms such as DAgger and deliberate data collection address the first. Expressive action models address the second, and observation history addresses the related case in which the policy input omits information that the expert used.

Let \(\pi_E\) be an expert policy and \(\pi\) a learned policy, and consider a horizon of \(H\) decisions. Behavior cloning measures disagreement on expert-visited states. Under a coupling in which expert and learner follow the same trajectory until their first different action, assume the learner's probability of disagreement at each expert state is at most \(\epsilon\). A union bound gives probability at most \(\min(1,H\epsilon)\) of any disagreement over the horizon.

If a disagreement can cause a bad state from which every remaining stage incurs loss one, the accumulated excess loss can be of order \(H^2\epsilon\). One way to see the scaling is to bound the probability of having diverged by stage \(t\) by \(t\epsilon\), and then sum over stages:

The lecture's phrase about a probability growing quadratically should be understood as this kind of accumulated-loss result under suitable assumptions. A probability is capped at one. The bound also does not say every problem suffers quadratic degradation; recovery ability, action consequences, and feedback stability can improve it.

The quadratic result comes from Ross and Bagnell's Efficient Reductions for Imitation Learning (AISTATS 2010). In the form restated in the DAgger paper discussed below, let actions be discrete, let the per-stage task cost lie in \([0,1]\), and write \(J(\pi)\) for the expected total cost of a policy over the \(H\) stages. If the learner's 0-1 loss, the probability that its action differs from the expert's, averages \(\epsilon\) over the states the expert visits, then \(J(\pi)\le J(\pi_E)+H^2\epsilon\). Equivalently, a classifier that errs with probability \(\epsilon\) on expert states can make as many as \(H^2\epsilon\) mistakes in expectation over \(H\) steps. The bound is tight: there are problems in which the extra cost grows quadratically in the horizon. The coupling argument above gives the constant \(H(H+1)\epsilon/2\), which never exceeds \(H^2\epsilon\), because \(H+1\le2H\) for \(H\ge1\). The course slides write \(T\) for the horizon \(H\) and describe the \(\mathcal O(\epsilon T^2)\) quantity as a probability of making mistakes; the cited theorem bounds an expected count of mistakes, or an expected cost, and a probability cannot exceed one.

An exact original model helps separate the quantities. At each still-good step, a mistake occurs independently with probability \(\epsilon\); after the first mistake the system remains bad and pays loss one per step, including the transition step. Then the probability of ever becoming bad is \(1-(1-\epsilon)^H\), while the expected bad-step count is \(\sum_{t=1}^H[1-(1-\epsilon)^t]\). These are different statistics of the same process. Their small-\(\epsilon\) expansions are approximately \(H\epsilon\) and \(H(H+1)\epsilon/2\), respectively.

Write \(d_t^\pi\) for the state distribution induced by a policy at stage \(t\). The difficulty is that supervised data from \(d_t^{\pi_E}\) need not cover \(d_t^\pi\). The trajectory factorization in chapter 14 determines these distributions after marginalizing over prior actions and states; a product of trajectory factors is not itself a single-state marginal.

The mismatch between \(d_t^{\pi_E}\) and \(d_t^\pi\) is called covariate shift: the rule that maps a state to the expert's action is unchanged, but the distribution of the inputs, the covariates, differs between training and deployment. The course slides write \(p_{\text{expert}}(x)\ne p_{\pi_\theta}(x)\) for the same statement. Covariate shift is the mechanism that turns a small per-state error into the accumulated loss above. After the first disagreement, the learner's states come from a distribution on which its training loss provides no estimate, and the independent, identically distributed sampling assumed by the supervised objective no longer describes the data on which the policy acts.

DAgger asks for expert actions where the learner actually goes 13:59

Perfect demonstrations can be too narrow. If every trajectory stays on the centerline, the learner may never see how to recover from a lateral deviation. Corrective data should contain the expert's action at a deviated state, not simply repeat the mistaken action that created the deviation.

Dataset Aggregation, or DAgger, alternates rollout and supervised fitting. Start with an expert dataset and an initial policy. At iteration \(i\), run the current policy, record the states it visits, ask the expert what action should be taken in those states, append these newly labeled pairs to the existing dataset, and fit the next policy. Retaining earlier data reduces the risk of forgetting states addressed in previous rounds.

In symbols, let \(\pi_1\) be the initial policy, \(\mathcal D\) the aggregate dataset, initialized with the expert demonstrations, and \(I\) the number of iterations. For \(i=1,\ldots,I\), roll out \(\pi_i\), collect the states \(x_t\) it visits, and update

with \(\pi_{i+1}=\pi_{\theta_{i+1}}\) and \(\mathcal L\) the behavior-cloning loss of the first section or its likelihood form. The labels in \(\mathcal D_i\) are the expert's actions, not the actions \(\pi_i\) took; the learner's actions determine only which states are visited. Each iteration is an ordinary supervised fit on a larger dataset, which is why DAgger is described as iterative supervised learning.

The key change is the input distribution of the labels: the expert is queried on learner-induced states. An implementation may use a mixture of expert and learner control during early collection, gradually changing that mixture. A human-gated variant collects interventions when a person judges correction necessary; confidence-based variants decide which states to query. Those choices alter both the collection distribution and the supervision cost.

The original DAgger paper by Ross, Gordon, and Bagnell formulates the reduction through no-regret online learning, with additional assumptions translating imitation loss into task performance. That is stronger and more specific than promising every neural retraining loop converges. The lecture emphasizes the operational mechanism; its theoretical benefit depends on learning quality, expert availability, and the cost of mistakes.

The return value deserves care. The final iteration fits \(\pi_{I+1}\), the only policy trained on all \(I\) rounds of corrections. A statement of the loop that fits \(\pi_{i+1}\) at every iteration but returns \(\pi_I\), as the official slides do, discards that last fit; this chapter takes \(\pi_{I+1}\) as the output of the plain loop. The original Algorithm 3.1 of Ross, Gordon, and Bagnell differs in three further details. It starts from an empty dataset and an arbitrary initial policy. It collects iteration \(i\) with the mixture \(\tilde\pi_i=\beta_i\pi_E+(1-\beta_i)\pi_i\), in which the expert chooses the control a fraction \(\beta_i\) of the time; with \(\beta_1=1\), the first round is ordinary expert demonstration, so behavior cloning is the first iteration of DAgger. Its analysis requires only that the average \(\frac1I\sum_{i=1}^{I}\beta_i\) tend to zero as \(I\) grows, and the authors report that the parameter-free choice \(\beta_1=1\), \(\beta_i=0\) for \(i\ge2\), often performs best in practice. Finally, it returns the iterate that performs best on validation, because its guarantee asserts that some policy in the trained sequence attains the bound, not that the last one does.

DAgger can target useful labels, but it does not automatically make querying cheap or rollouts safe. A human may be unable to label an isolated high-frequency state meaningfully, and some visited states may already be unrecoverable. An expensive MPC controller can act as the expert in simulation, making policy distillation attractive, but the learned policy still needs validation under the deployment distribution.

The data-efficiency claim for DAgger is a comparison with the non-interactive alternative. Without rollouts, supplying corrective labels for every state that a learned policy might reach requires demonstrating or labeling an extremely broad set of states in advance, most of which no learned policy will visit. DAgger requests labels only at states that actual learner rollouts reach, so each query addresses an observed failure. It is therefore economical in the number of labels, while each label can remain expensive. Variants change the query rule to reduce that expense. A confidence-based method queries the expert only at visited states where the learner is uncertain about its decision, as measured, for example, by disagreement among an ensemble of learned policies. Human-gated DAgger lets a supervising person watch the learner and take over when it makes a mistake; the person then teleoperates a continuation from the takeover state, and the recorded continuation joins the dataset. This rule also answers the difficulty that a person may be unable to label isolated high-frequency states: the expert supplies a coherent stretch of control rather than one action for each sampled state.

A tiny controller shows exactly what a corrective label adds

Use an original normalized first-order position model \(x_{t+1}=x_t+u_t+0.1\), where the constant 0.1 is a drift. This is a separate teaching model for position correction, rather than the second-order cart's full physics. The expert \(\pi_E(x)=-x-0.1\) sends every state to zero in one step.

Suppose the only demonstration is equilibrium behavior \((x,u)=(0,-0.1)\). Fit an affine policy \(\pi_{a,b}(x)=ax+b\) by minimum-norm least squares. The data constrain \(b=-0.1\) but do not constrain \(a\); the chosen fit has \(a=0\). It perfectly fits the demonstration and simply cancels drift. From an initial deviation \(x=0.3\), it stays at 0.3 forever.

Roll out this policy from the deviated state and query the expert. The new label is \((0.3,-0.4)\). Aggregating it with the original pair uniquely determines \(a=-1\), \(b=-0.1\). The refitted policy returns the next state to zero. Nothing about the feature class changed; the new information came from the previously unobserved recovery state.

This constructed example has exact labels, linear realizability, and a deliberately benign collection setting. Its one-round success should not be extrapolated to high-dimensional robotics. It isolates the reason aggregation can help: collecting the right state-action pair identifies a feedback direction that repeated equilibrium demonstrations leave unspecified.

Data augmentation must transform actions consistently with observations 26:16

The lecture's driving example uses left, center, and right camera views. A side view can imitate an off-center observation, paired with a steering correction toward the lane. This creates recovery-like training examples without intentionally driving the vehicle off the road. Bojarski and colleagues' end-to-end driving paper is the primary source for the discussed camera-to-steering system.

The crucial point is not merely “add images.” The action label must correspond to the transformed observation. Reusing the center camera's unchanged steering command for every side view can teach the opposite of recovery. The geometric transformation, camera placement, assumed road shape, and desired correction horizon all affect the appropriate label.

The system of Bojarski and colleagues, built at NVIDIA in 2016, makes each of these choices explicit. The recorded steering command is the inverse turning radius, in reciprocal meters, which passes smoothly through zero between left and right turns and avoids the infinite radius of straight driving. The left and right cameras supply two fixed lateral offsets. Offsets between them, and all heading rotations, are synthesized by a viewpoint transformation of the nearest camera's image that assumes every point below the horizon lies on flat ground and every point above it is infinitely far away. The label of a transformed image is adjusted to the command that would return the vehicle to the desired position and orientation in two seconds, and the shift and rotation magnitudes are drawn from a zero-mean normal distribution whose standard deviation is twice that measured for human drivers. During training, the network's command for the transformed image is compared with the adjusted label, and the squared error drives backpropagation; at deployment, only the center camera is used. The two-second horizon is a design parameter: a shorter horizon would label the same offset with a sharper correction. The flat-ground assumption distorts objects that rise above the road, such as cars, poles, trees, and buildings, so the synthetic images are approximate independently of any error in the adjusted label.

The network itself is small. A three-channel YUV image of 66 by 200 pixels passes through a fixed normalization layer, three 5-by-5 convolutions with stride 2, two 3-by-3 convolutions with stride 1, and fully connected layers of 100, 50, and 10 units before the scalar output. A valid convolution with kernel size \(k\) and stride \(s\) maps a side of length \(n\) to \(\lfloor(n-k)/s\rfloor+1\), which reproduces the published feature maps: 24 maps of \(31\times98\), then 36 of \(14\times47\), 48 of \(5\times22\), 64 of \(3\times20\), and 64 of \(1\times18\). The last stage flattens to \(64\cdot1\cdot18=1152\) values, whereas the published architecture figure, reproduced in the course slides, labels the flattened layer with 1164 neurons. With 1152 inputs to the first fully connected layer, the network has 252,219 weights and biases, consistent with the paper's statement of about 250 thousand parameters. The label 1164 would give 253,419, also about 250 thousand, so the parameter count cannot decide between the two numbers; the shape arithmetic does. The paper states that training on the human driver's data alone is not sufficient, because the network must learn to recover from mistakes or the car will slowly drift off the road.

The second source case describes trail-following data collected with multiple head-mounted cameras and discrete left/straight/right labels. It illustrates the same principle of purposeful coverage. Such augmentation approximates a particular observation change; it does not reproduce every physical consequence of actually moving the robot sideways, such as altered velocity or proximity to terrain.

That case is the study by Giusti and colleagues, “A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots” (IEEE Robotics and Automation Letters, vol. 1, 2016). A deep neural network classifies the image from a quadrotor's front camera into three classes, turn left, go straight, and turn right, so the policy outputs a categorical distribution over three actions. The training images came from a person walking the trails with three head-mounted cameras, one facing forward and one turned to each side. An image from the forward camera was labeled go straight. An image from the camera turned to the left shows the view of a robot whose heading points left of the trail, so it was labeled turn right; images from the right camera were labeled turn left. These labels are the discrete counterpart of the steering adjustment above. Because the three cameras sit within centimeters of one another on one head and differ mainly in orientation, the rig synthesizes heading errors but not lateral offsets from the trail center, which the laterally separated cameras of the driving system do supply.

For the position example, synthetic observation \(x+\delta\) should receive expert action \(-(x+\delta)-0.1\). Keeping the old action \(-x-0.1\) would make the successor equal to \(\delta\), preserving the injected error. This algebra makes label consistency testable. The amount and direction of augmentation should reflect plausible deployment deviations rather than arbitrary image diversity.

History can resolve partial observation, but not every ambiguity 35:16

A physical state can be Markov even when the measured observation is not. Let \(o_t\) be the observation. A policy using only \(\pi(u_t\mid o_t)\) cannot distinguish hidden states that produce the same observation. A policy conditioned on history \(h_t=(o_{0:t},u_{0:t-1})\), or a learned summary of it, may infer velocity, past contact, or an intended maneuver.

Demonstrations are called non-Markovian in the observation when the demonstrator's action distribution given the history differs from its distribution given the current observation alone, \(p(u_t\mid h_t)\ne p(u_t\mid o_t)\), for histories that occur in the data. A common compromise between the single-observation policy and the full history is a finite window,

with window length \(\ell\); the course slides write \(H\) for this window, whereas \(H\) denotes the horizon in this chapter. Any sequence model can implement the window: a transformer that attends over the \(\ell+1\) observations, or a recurrent network with LSTM or GRU cells that summarizes them in a hidden state. The windowed policy is an ordinary Markov policy on the stacked input \((o_{t-\ell},\ldots,o_t)\), a state augmentation, in which the past observations that the policy needs join its present input. Its input dimension grows linearly with \(\ell\), and the first \(\ell\) steps of an episode need a rule for the missing entries, such as padding with copies of the first observation. The window length is usually chosen empirically, by training on the same data with several values of \(\ell\) and comparing closed-loop performance. RT-1, discussed with action chunking below, conditions on a history of six images.

For the second-order cart, two position observations separated by known sampling interval approximately reveal velocity, while one position does not. Measurement noise, changing acceleration, and delayed actuation limit that approximation. A recurrent network or transformer can encode history, but finite history is not automatically a sufficient statistic for every partially observed process.

Some action diversity remains even with a sufficient physical state. A demonstrator can choose either side of an obstacle based on preference or task intent. Conditioning on a goal or a persistent mode can explain that choice. Treating all multimodality as missing state would overlook legitimate alternative plans.

A mixture preserves alternatives that a mean erases 39:11

A policy network represents an action distribution by outputting that distribution's parameters. For a finite action set, such as the four moves up, down, left, and right, the network outputs one logit \(\nu_m(o)\) for each action \(m\), and the softmax \(\pi_\theta(m\mid o)=e^{\nu_m(o)}/\sum_{m'}e^{\nu_{m'}(o)}\) converts the logits into probabilities. Training minimizes the cross-entropy \(-\log\pi_\theta(u_n\mid o_n)\) averaged over the dataset. A categorical output assigns separate probability to each action and therefore represents several modes without difficulty, but a joint table over many action coordinates has a number of entries exponential in the number of coordinates, as the discretization paragraph below quantifies. For a continuous action, such as a steering angle, the network can output a mean \(\mu(o)\) and a standard deviation \(\sigma(o)\gt0\), the latter through a positive transformation such as an exponential, and training minimizes the Gaussian negative log likelihood. This head respects the continuity of the action and is compact, needing \(d+d(d+1)/2\) numbers for the mean and covariance of a Gaussian over \(d\) action coordinates, 35 for \(d=7\), against the \(100^7\) cells of a seven-dimensional grid with 100 bins per coordinate in Exercise 3, but every Gaussian has a single mode. Multimodal data arise even when each demonstrator is consistent: pooling demonstrations from several experts who resolve the same situation differently produces a multimodal dataset from individually unimodal behavior.

For demonstrations concentrated near actions \(-1\) and \(+1\), squared-loss regression produces a conditional mean near zero. A single Gaussian fitted by likelihood can place its mean near zero and increase variance, assigning considerable probability between the modes. A Gaussian mixture instead represents

A network can output normalized weights, component means, and valid covariance parameters. Training uses the logarithm of the mixture density, not a weighted average of component log densities. At execution, sampling a component and then an action preserves alternatives; returning the mixture mean can recreate the original averaging failure.

The single-Gaussian failure can be computed exactly. Let the demonstrated actions at one observation follow the equal mixture of \(\mathcal N(-1,\varsigma^2)\) and \(\mathcal N(1,\varsigma^2)\), where \(\varsigma\) is the spread of each demonstrated mode. The population maximum-likelihood Gaussian minimizes the Kullback–Leibler divergence from the data distribution and therefore matches its first two moments: mean \(0\) and variance \(1+\varsigma^2\). With \(\varsigma=0.2\), the fitted density is largest at \(u=0\), where it equals \(1/\sqrt{2\pi\cdot1.04}\approx0.391\), while the demonstration density there is about \(7.4\times10^{-6}\), roughly \(1.3\times10^{5}\) times smaller than its value \(0.997\) at either mode. The fitted Gaussian assigns probability \(0.376\) to actions with \(|u|\lt0.5\), where the demonstrations place \(0.0062\). Sampling from this fit, rather than taking its mean, therefore still sends about three draws in eight toward the obstacle that both demonstrated modes avoid.

The number of components \(K\) is fixed before training and is a hyperparameter. A larger \(K\) cannot lower the best attainable training likelihood, because surplus components can be given negligible weight, but it can fit noise in the demonstrations; held-out likelihood or closed-loop evaluation guards against that overfitting, and too small a \(K\) underfits. When the number of distinct behaviors is known from the task, for example two routes around one obstacle, that knowledge can set \(K\) directly. A \(K\) smaller than the number of modes either leaves a mode unrepresented or stretches a component across two modes, which reintroduces the averaging failure inside that component.

Action bounds still matter. A Gaussian has unbounded support, so a mixture does not automatically respect actuator limits. A bounded parameterization, transformed distribution, constrained downstream controller, or explicitly analyzed clipping rule may be required. Modeling a multimodal distribution solves a representational problem, not every control constraint.

Discretization is another option. With \(M\) bins per action coordinate and \(d\) coordinates, a flat joint categorical table has \(M^d\) outcomes. An autoregressive factorization instead uses

The chain rule is exact. It permits sequential one-coordinate predictions, often implemented with shared neural parameters. It does not by itself erase the complexity of an arbitrary joint distribution: a separate table for every prefix can still be exponentially large. The practical savings come from structure and parameter sharing, traded against sequential sampling latency. Independent coordinate predictions would be cheaper but lose cross-coordinate correlations.

A conditional variational autoencoder obtains multimodality from a latent variable instead of from explicit components. Draw \(\zeta\sim\mathcal N(0,I)\) and decode an action distribution \(p_\theta(u\mid o,\zeta)\), often Gaussian. The policy is the continuous mixture

which can be multimodal although each decoded distribution is unimodal, because different regions of the latent space can decode to different modes. The integral is generally intractable, so training introduces an encoder \(q_\lambda(\zeta\mid o,u)\) with parameters \(\lambda\) and maximizes the evidence lower bound

which follows from Jensen's inequality applied to the logarithm of the integral. The gap between the two sides equals \(\mathrm{KL}(q_\lambda(\zeta\mid o,u)\,\|\,p_\theta(\zeta\mid o,u))\), the divergence from the encoder to the model's exact posterior over the latent variable, so the bound is tight only when the encoder is exact. At execution the encoder is discarded: sample \(\zeta\) from the prior and decode, so a fresh latent draw can select a different mode.

Flow matching learns a transport velocity in an auxiliary time 52:34

Diffusion and flow-based policies generate a sample through an iterative transformation from a simple base distribution. The source gives a brief diffusion overview, then develops a linear-path flow-matching recipe. We focus on that explicit recipe rather than treating diffusion and deterministic ODE transport as identical constructions.

Diffusion models, developed first for image generation, define the transformation through a fixed noising process. Starting from a demonstration action \(a^{(0)}\), the forward process adds Gaussian noise in \(n_{\mathrm d}\) increments until the result is indistinguishable from a standard Gaussian sample. In the denoising diffusion probabilistic model of Ho, Jain, and Abbeel (NeurIPS 2020), level \(k\) of this chain can be sampled directly as

where the signal fractions decrease from \(\rho_0=1\) to \(\rho_{n_{\mathrm d}}\approx0\). A network \(\hat\xi_\theta(a^{(k)},k,o)\) is trained by squared error to predict the noise \(\xi\) that produced \(a^{(k)}\), at levels \(k\) drawn at random. Sampling runs the chain backward: draw \(a^{(n_{\mathrm d})}\) from the standard Gaussian and apply \(n_{\mathrm d}\) learned denoising updates, each of which subtracts a scaled noise prediction and, in the stochastic version, adds a smaller fresh noise, to reach an action. The course slides write this chain as \(x_0,\ldots,x_T\); here the superscript avoids a clash with state and time. Flow matching keeps the overall structure, a learned field that moves noise toward data, but replaces the noising chain by an explicit interpolation path and the noise target by a velocity target.

Let \(a_0\) be a noise action vector, \(a_1\) a demonstration action vector, and \(s\in[0,1]\) an auxiliary transport time. This \(s\) is not physical control time. Form \(a_s=(1-s)a_0+sa_1\), whose pair-specific velocity is \(a_1-a_0\). Train a conditioned vector field \(v_\theta(a,s,o)\) with

The population squared-loss optimum is the conditional mean of the target velocity given \((a_s,s,o)\). Different endpoint pairs can pass through the same intermediate location, so the learned field is not expected to memorize one pair's destination. With appropriate path and regularity conditions, integrating the resulting marginal velocity transports the base distribution along the intended marginal path. Lipman and colleagues' Flow Matching gives the primary framework for this vector-field regression approach.

The expectation in \(\mathcal L(\theta)\) is over three independent draws for each training example: a base sample \(a_0\sim\mathcal N(0,I)\), a demonstration pair \((o,a_1)\) from the dataset, and a time \(s\) from a density on \([0,1]\), commonly the uniform one. The course slides, following Tomczak's tutorial post on flow matching, write \(x_0\), \(x_1\), \(x_t\), and \(t\) for \(a_0\), \(a_1\), \(a_s\), and \(s\); this chapter reserves \(x\) for states and \(t\) for physical control time. In integral form, the sampler computes \(a(1)=a(0)+\int_0^1v_\theta(a(s),s,o)\,ds\), and the Euler rule below approximates this integral on the grid \(s\in\{0,\Delta s,\ldots,1-\Delta s\}\).

At inference, sample \(a(0)\) from the base distribution and solve \(da/ds=v_\theta(a(s),s,o)\) until \(s=1\). An Euler approximation uses \(a_{j+1}=a_j+\Delta s\,v_\theta(a_j,s_j,o)\). Fewer integration steps reduce inference work but introduce numerical error in addition to the learned-field error. The action is applied in physical time only after this auxiliary computation has produced it.

An original one-dimensional Gaussian case permits an exact check. Take independent \(a_0\sim\mathcal N(0,1)\) and \(a_1\sim\mathcal N(\mu,\sigma^2)\), \(\sigma>0\). The interpolated variable has mean \(s\mu\) and variance \(D(s)=(1-s)^2+s^2\sigma^2\). The covariance of \(a_1-a_0\) with \(a_s\) is \(s\sigma^2-(1-s)\). Conditional Gaussian regression therefore gives

The exact ODE path from a fixed initial draw \(z\) is \(a(s)=s\mu+\sqrt{D(s)}\,z\). Differentiation verifies the field because \(D'(s)/2=s\sigma^2-(1-s)\). At time one, \(a(1)=\mu+\sigma z\), which has exactly the desired Gaussian distribution.

Notice the distinction between this ODE path and an individual training interpolation. The ODE follows the marginal conditional-mean field; it does not generally follow the straight line to a sampled training endpoint. The example makes that subtlety explicit and supplies a reference for numerical integration error.

The Gaussian case cannot display the multimodal transport that motivates the method, but one dimension still permits an exact description. Fix the observation and suppose the field is continuous and Lipschitz in \(a\), so that each initial value has a unique trajectory. Two trajectories of a scalar ODE cannot cross, so the time-one map \(\mathcal T\) that sends \(a(0)\) to \(a(1)\) is increasing. If \(\mathcal T\) transports the standard Gaussian to a target with continuous, strictly increasing distribution function \(F\), it must be the monotone rearrangement

where \(\Phi\) is the standard Gaussian distribution function, because an increasing map preserves the probability below each point. Every exact one-dimensional flow from this base to this target, whatever its intermediate path, ends at the same map. For a target with three equally weighted, well-separated modes, noise values below \(\Phi^{-1}(1/3)\approx-0.431\) are carried below \(F^{-1}(1/3)\), which lies in the gap between the first two modes, and noise values above \(0.431\) are carried beyond the second gap; the unimodal noise density splits into three bands, one for each mode. Differentiating gives \(\mathcal T'(z)=\varphi(z)/f(\mathcal T(z))\), with \(\varphi\) the standard Gaussian density and \(f\) the target density. The map is steepest where the target density is small, so the field must carry samples near the cut points rapidly across the gaps, and an error in the field there leaves samples in a gap, at actions that no mode contains. The Gaussian example above is the special case \(F^{-1}(\Phi(z))=\mu+\sigma z\). In two or more dimensions trajectories still cannot cross, but monotonicity no longer singles out one map, and the map produced by flow matching need not coincide with an optimal transport map.

Action chunking trades frequent feedback for coherent multi-action prediction 59:45

Instead of sampling one action, a policy can generate a chunk \((u_t,\ldots,u_{t+K-1})\). A joint representation can maintain a consistent maneuver across adjacent actions, and executing several actions before another inference call gives the model more computation time per decision. The lecture motivates improved smoothness and inference scheduling through this idea.

The two execution loops differ in where observation enters. A per-step feedback policy observes \(o_t\), samples \(u_t\sim\pi(u_t\mid o_t)\), applies it, and observes again at \(t+1\). A chunking policy observes \(o_t\), samples the whole sequence \(u_{t:t+K-1}=(u_t,\ldots,u_{t+K-1})\) from \(\pi(u_{t:t+K-1}\mid o_t)\), applies it, and observes next at \(t+K\); this is the case \(M=K\) of the execution horizon \(M\le K\) introduced below. With \(K=4\) over eight steps, decisions occur at \(t=0\) and \(t=4\), the first covering controls 0 through 3 and the second controls 4 through 7, instead of eight separate decisions. The course slides write the sampled object as \(u_{t+K}\) and the next observation as \(x_{t+K+1}\); the sampled object is the sequence \(u_{t:t+K-1}\), and executing its \(K\) actions from time \(t\) reaches time \(t+K\).

Prediction horizon \(K\) and execution horizon \(M\leq K\) are separate design choices. Executing all \(K\) actions before observing again commits to a longer open-loop interval. Executing only the first few and replanning resembles receding-horizon control. The Diffusion Policy paper combines action-sequence generation with receding-horizon execution; this is a useful primary example of how the choices interact.

The smoothness benefit can be isolated in the bimodal obstacle example. Suppose that at every decision the policy's two modes have equal probability and each sample selects a mode independently of the previous one. Per-step sampling over \(H\) steps then switches mode between consecutive actions with probability \(1/2\) at each of the \(H-1\) transitions, an expected \((H-1)/2\) switches, which is 3.5 for \(H=8\). A chunk sampled from a joint distribution that commits to one mode for all its actions can switch only at the \(H/K-1\) chunk boundaries, an expected \((H/K-1)/2\) switches, which is 0.5 for \(H=8\) and \(K=4\). The calculation holds the mode probabilities fixed to isolate the effect of sampling; in closed loop, the observation after each action changes them, and a policy that has already moved toward one side usually favors that side. It shows why within-chunk coherence matters most when the policy is multimodal, and why chunk boundaries remain the place where inconsistent plans can meet.

The analogy to MPC has a limit. A generated action chunk is not necessarily the solution of a dynamics-constrained optimization problem and does not inherit the terminal-set stability theorem from chapter 12. Under unknown shocks, less frequent feedback can increase deviation. For a scalar error model \(\delta_{t+1}=L\delta_t+w_t\), with \(|w_t|\leq w_{\max}\), an \(M\)-step uncorrected interval can accumulate up to \(w_{\max}\sum_{j=0}^{M-1}|L|^j\) from zero initial error.

Chunking also should not be confused with averaging independently sampled modes. Averaging a left-going chunk and a right-going chunk may still create a collision-directed sequence. Temporal consistency and valid uncertainty representation must be preserved by the chosen execution scheme.

Chi and colleagues' Diffusion Policy combines three ideas from this chapter. At time \(t\) the policy conditions on the latest \(T_o\) observations, a finite history; it predicts \(T_p\) future actions with a conditional denoising diffusion model; and it executes \(T_a\) of them before planning again. In this chapter's symbols, \(T_p=K\) and \(T_a=M\). Its noise-prediction network, the \(\hat\xi_\theta\) of the diffusion paragraph, is written \(\varepsilon_\theta(O_t,A_t^k,k)\), where \(O_t\) is the observation window and \(A_t^k\) the noisy action sequence at denoising iteration \(k\); the paper's \(K\) counts denoising iterations, the \(n_{\mathrm d}\) above, not the chunk length. Sampling starts from a Gaussian action sequence and denoises it over the iterations. Conditioning on observations, rather than generating them, keeps inference affordable: with 100 training iterations and 10 inference iterations of the DDIM sampler, the authors report 0.1 s inference latency on an Nvidia 3080 GPU, a cost that executing several actions per inference helps absorb. Its real-robot tasks include pushing a T-shaped block to a target, flipping a mug, and pouring and spreading sauce on pizza dough.

Brohan and colleagues' RT-1, the Robotics Transformer, makes the other representation choices: one action per decision, a discretized action, and a history. Its input is a history of six 300-by-300 camera images and a natural-language instruction. An ImageNet-pretrained EfficientNet-B3, conditioned on an embedding of the instruction through FiLM layers, turns each image into 81 visual tokens; a TokenLearner module compresses them to 8, giving 48 tokens for the history; and a decoder-only transformer with 8 self-attention layers and 19 million parameters, of 35 million in the whole model, outputs the action. The action has eleven dimensions, seven for the arm (position, orientation, and gripper opening), three for the mobile base, and one mode variable that selects arm control, base control, or termination. Each dimension is discretized into 256 uniform bins within its bounds and trained with categorical cross-entropy. The control loop runs at 3 Hz, and the training data comprised over 130,000 episodes covering over 700 tasks, collected with 13 robots over 17 months. A flat categorical table over the full action would have \(256^{11}=2^{88}\approx3.1\times10^{26}\) entries, against \(11\times256=2816\) logits for per-dimension outputs. The paper reports that autoregressively conditioning on actions, as earlier transformer controllers did, did not improve performance and slowed inference by more than a factor of two.

Behavior cloning trades reward design for dependence on the demonstrations 1:05:26

Behavior cloning, with the repairs developed so far, has three practical strengths. It is simple: training is supervised learning with an established loss, so it is easy to implement, and its loss can be monitored on held-out pairs, although chapter 14 shows that a low held-out loss on expert data does not certify closed-loop performance. It is efficient in interaction: it needs no trial-and-error learning in the controlled system, so no exploratory failures occur during training, in contrast with the reinforcement learning of the next chapter. And it needs no explicit reward: the demonstrations specify the task, which matters when a numerical score is harder to write than a demonstration is to give. Driving is the standard example, since a competent driver can demonstrate good driving without being able to state a numerical reward that distinguishes it from merely legal driving.

The corresponding weaknesses follow from the same objective. The fitted policy depends on demonstration quality: with a sufficiently expressive policy class, the population optimum of the likelihood objective reproduces the demonstrator's conditional action distribution on the demonstrated states, so systematic errors in the demonstrations are reproduced rather than corrected, and no term in the objective rewards doing better than the demonstrator. Generalization remains limited by covariate shift: corrective data, aggregation, and history reduce compounding errors but do not remove the dependence of the learner's states on its own actions. And there is no exploration: a cloned policy never tries an action that no demonstration suggests, so it cannot discover a better solution that no demonstrator used. Reinforcement learning addresses the last weakness by interacting with the system under a reward. The next section pursues part of that goal within the supervised framework, and the closing section turns to learning the reward itself.

Outcome-conditioned supervision can reuse imperfect demonstrations 1:08:04

The lecture next introduces reinforcement learning via supervised learning, or RvS. One option filters trajectories by observed return; another gives higher nonnegative weights to selected examples. A weighted likelihood objective has the form \(-\sum_iw_i\log\pi_\theta(u_i\mid o_i)\), \(w_i\geq0\). A raw reward need not be a valid weight if it is negative; the weighting rule must be defined deliberately.

RvS converts the reinforcement-learning problem into a filtered, weighted, or conditional imitation problem, so that a broader dataset, including demonstrations from suboptimal policies or related tasks, can be used instead of optimal demonstrations alone. For filtering, score each trajectory \(\tau=(x_0,u_0,\ldots,x_{H-1},u_{H-1},x_H)\) by its return

where \(R(x,u)\) is the per-step reward and \(\gamma\in(0,1]\) a discount factor, and keep \(\mathcal D_\kappa=\{\tau\in\mathcal D:r(\tau)\ge\bar r\}\), with the threshold \(\bar r\) set, for example, at the smallest return among the top fraction \(\kappa\) of trajectories. Behavior cloning on \(\mathcal D_\kappa\) is unchanged in form. For weighting, each transition can carry its own reward as its weight, \(w_n=R(x_n,u_n)\), in the negative log likelihood \(-\frac1N\sum_{n=1}^{N}w_n\log\pi_\theta(u_n\mid o_n)\), where \(o_n=x_n\) for a fully observed system.

The effect of the weights is exact for a categorical policy at a single observation. Minimizing \(-\sum_nw_n\log\pi(u_n)\) over the probability simplex gives \(\pi(u)=W_u/\sum_{u'}W_{u'}\), where \(W_u\) is the total weight of the examples labeled \(u\) and all totals are nonnegative with a positive sum. Two examples of turning left with reward 1 and one example of turning right with reward 4 give \(\pi(\text{right})=4/6=2/3\), whereas unweighted cloning gives \(1/3\). The same calculation explains the sign requirement. If some \(W_u\lt0\), the term \(-W_u\log\pi(u)\) tends to \(-\infty\) as \(\pi(u)\to0\), so the objective is unbounded below and has no minimizer. Multiplying all rewards by a positive constant leaves the fit unchanged, but adding a constant to make them positive does not. A per-transition reward also ignores delayed consequences: an action with a small immediate reward that enables a large later one receives a small weight, which makes weights built from the reward-to-go, or from the value functions of chapter 14, natural alternatives. The official slides call the weighted minimizer a maximum-likelihood estimate; it maximizes a weighted likelihood, and its maximizer is guaranteed to coincide with the maximum-likelihood estimate only when all weights are equal and positive.

Filtering can improve the quality of demonstrated task outcomes while discarding recovery states, reducing coverage. High-return episodes can also be lucky rather than caused entirely by better actions in a stochastic environment. This is why dataset quality, coverage, and causal action quality should not be treated as interchangeable.

Goal conditioning fits \(\pi(u\mid o,g)\), where \(g\) is a desired outcome. A trajectory that failed to reach its originally intended destination still demonstrates how it reached the destination it actually achieved. Relabeling it with that achieved goal can extract useful supervision. It does not establish that every requested goal is reachable from every starting state.

Both conditioning schemes fit one objective with an outcome variable \(\omega\) appended to the input, \(-\frac1N\sum_{n=1}^{N}\log\pi_\theta(u_n\mid o_n,\omega_n)\). For goal conditioning \(\omega\) is a point of the state space \(\mathcal X\), \(\omega=g\in\mathcal X\), such as the final state of the trajectory or a specific state it visited; for reward conditioning it is a return, written in the official slides as the undiscounted total \(\sum_{t=0}^{H-1}R(x_t,u_t)\). For the pair at time \(t\), the outcome that the remaining actions can still influence is a state visited after \(t\), or the reward-to-go \(\sum_{t'=t}^{H-1}R(x_{t'},u_{t'})\); conditioning every pair on the whole-trajectory total also encodes rewards collected before \(t\), which the action at \(t\) did not affect. Hindsight relabeling multiplies the goal-conditioned data. In a trajectory with \(H\) transitions, each pair \((x_t,u_t)\) can take every later state \(x_{t'}\), \(t\lt t'\le H\), as a valid goal, which yields \(\sum_{t=0}^{H-1}(H-t)=H(H+1)/2\) training triples. A trajectory through four states, with \(H=3\), gives six triples, three of which use its final state as the goal.

Return conditioning instead supplies a desired score or return-to-go. At execution, the requested return must be updated consistently as rewards are obtained and should remain within a meaningful supported range. Asking for a score far beyond the dataset is extrapolation, not an optimization guarantee. Emmons and colleagues' RvS study examines these supervised formulations for offline reinforcement learning and provides primary context for this part of the lecture.

The advantages claimed for outcome conditioning follow from this construction, with the qualifications already noted. The outcome labels are computed from the demonstrations themselves, so no additional annotation is needed. Suboptimal trajectories, and trajectories collected for a different task, contribute correct supervision for the outcomes they did achieve. Because a goal-conditioned policy is trained without any reward, the reward enters only at execution through the choice of the requested goal, so a change of task reward changes the request rather than the training data; this is the sense in which conditioning decouples achieving an outcome from the reward structure. Improved generalization is an empirical finding of the studies concerned, not a consequence of the objective.

An inverse-reward loop alternates explanation and control 1:15:42

The closing IRL discussion is intentionally brief. Parameterize a reward \(R_w\), use demonstrations to update \(w\), optimize or improve a policy under the resulting reward, then compare its behavior with the demonstrations and repeat. Different algorithms use different matching criteria, regularizers, and assumptions about the expert.

The source names apprenticeship learning, maximum-margin planning, and maximum-entropy IRL as directions for further study. Their common challenge is that multiple rewards can explain the same behavior. The positive-scale and potential-shaping ambiguities derived in chapter 14 persist; an algorithm selects a useful representative under its assumptions rather than automatically recovering a uniquely identifiable internal human objective.

Each named method selects a reward by a different principle. Abbeel and Ng's apprenticeship learning (ICML 2004) takes a reward linear in state features, \(R_w(x)=w^\mathsf T\psi(x)\) with \(\|w\|_1\le1\), which implies \(\|w\|_2\le1\) because the 2-norm never exceeds the 1-norm. A policy's expected discounted return is then \(w^\mathsf T\bar\psi(\pi)\), with feature expectations \(\bar\psi(\pi)=\mathbb E[\sum_t\gamma^t\psi(x_t)]\), so by the Cauchy–Schwarz inequality a policy whose feature expectations lie within Euclidean distance \(\epsilon_{\mathrm f}\) of the expert's has return within \(\epsilon_{\mathrm f}\) of the expert's under every such reward. The method alternates a reward step, which finds the weight vector that best separates the expert's feature expectations from those of the policies found so far, with a reinforcement-learning step for that reward, and it never needs to identify the true weights. Ratliff, Bagnell, and Zinkevich's maximum margin planning (ICML 2006) chooses weights under which each demonstrated path is cheaper than every alternative path by a margin that grows with the alternative's loss relative to the demonstration, calling a planner inside each update to find the most violating alternative. Ziebart, Maas, Bagnell, and Dey's maximum entropy IRL (AAAI 2008) models demonstrated trajectories as \(P_w(\tau)\propto\exp(w^\mathsf Tf_\tau)\), with \(f_\tau\) the feature counts of trajectory \(\tau\), and maximizes their likelihood. Its gradient is the difference between the expert's empirical feature counts and the model's expected feature counts. Among all trajectory distributions that match the expert's feature counts, this exponential family member has the largest entropy, so the method resolves the ambiguity by committing to no structure that the demonstrations do not require.

This separates the chapter's questions cleanly. DAgger changes where action labels are collected. Mixtures and generative policies change how action distributions are represented. Outcome conditioning changes what behavior is requested from a dataset. IRL changes the learned object from actions to an objective. Combining them can be useful, but each mechanism should be justified against a specific observed failure.

Executable checks: aggregation and exact Gaussian transport

import numpy as np
from scipy.integrate import solve_ivp

# Exact first-mistake model: probability is not accumulated loss.
H, epsilon = 100, .01
failure_probability = 1-(1-epsilon)**H
cumulative_loss = sum(1-(1-epsilon)**t for t in range(1, H+1))
assert failure_probability <= min(1., H*epsilon)
assert cumulative_loss <= min(H, H*(H+1)*epsilon/2)

# Two rounds of affine imitation, with expert relabeling.
def fit(states):
    states = np.asarray(states)
    design = np.column_stack([states, np.ones_like(states)])
    labels = -states-.1
    return np.linalg.lstsq(design, labels, rcond=None)[0]
old = fit([0.])
new = fit([0., .3])
assert np.allclose(old, [0., -.1])
assert np.allclose(new, [-1., -.1])
assert abs(.3+old@[.3, 1.]+.1-.3) < 1e-12
assert abs(.3+new@[.3, 1.]+.1) < 1e-12

# Independent-endpoint Gaussian flow: exact marginal transport.
mu, sigma = 2., .5
def field(s, a):
    variance = (1-s)**2+s*s*sigma*sigma
    coefficient = (s*sigma*sigma-(1-s))/variance
    return mu+coefficient*(a-s*mu)
initial = np.array([-2., -1., 0., 1., 2.])
solution = solve_ivp(field, (0, 1), initial,
                     rtol=1e-11, atol=1e-12)
assert solution.success
expected = mu+sigma*initial
assert np.max(abs(solution.y[:, -1]-expected)) < 1e-9
errors = []
for steps in (20, 200, 2000):
    a = initial.copy()
    for j in range(steps):
        a += field(j/steps, a)/steps
    errors.append(np.max(abs(a-expected)))
assert errors[2] < errors[1] < errors[0]
assert errors[-1] < .003
print("Failure probability and expected bad steps:",
      failure_probability, cumulative_loss)
print("Affine policy before and after aggregation:", old, new)
print("Euler transport errors:", errors)

A second listing checks the arithmetic of the case studies and supervised variants. Its inputs are the published layer sizes of the steering network and the constructed numbers of the text; it uses no data and no training. It confirms the feature-map shapes and the 252,219 parameters, the moment-matched Gaussian's density and probability mass near zero for modes at \(\pm1\) with \(\varsigma=0.2\), the noise cut point \(\Phi^{-1}(0.3)\) for a two-mode target with weights 0.3 and 0.7 together with the negligible mass in that target's gap, the weighted categorical fits of the RvS example before and after a reward offset, the hindsight-relabeling count, and the expected mode switches with and without chunks of four, the last by exact enumeration of equally likely mode sequences. Every check is exact arithmetic or a closed-form Gaussian probability, so a failed assertion would indicate an error in the text rather than sampling noise.

from itertools import product
import numpy as np
from scipy.stats import norm

# Steering network: valid-convolution shapes and parameter count.
height, width, channels, params = 66, 200, 3, 0
for kernel, stride, maps in [(5, 2, 24), (5, 2, 36), (5, 2, 48),
                             (3, 1, 64), (3, 1, 64)]:
    height = (height-kernel)//stride+1
    width = (width-kernel)//stride+1
    params += kernel*kernel*channels*maps+maps
    channels = maps
flat = channels*height*width
assert (channels, height, width) == (64, 1, 18) and flat == 1152
for fan_in, fan_out in [(flat, 100), (100, 50), (50, 10), (10, 1)]:
    params += fan_in*fan_out+fan_out
assert params == 252219

# Likelihood-fitted Gaussian for action modes at -1 and +1.
s = .2
fit = norm(0, np.sqrt(1+s*s))
def data_pdf(u):
    return .5*norm.pdf(u, -1, s)+.5*norm.pdf(u, 1, s)
assert abs(fit.pdf(0)-.391) < 1e-3 and data_pdf(0) < 1e-5
mass_fit = fit.cdf(.5)-fit.cdf(-.5)
mass_data = sum(.5*(norm.cdf(.5, m, s)-norm.cdf(-.5, m, s))
                for m in (-1, 1))
assert abs(mass_fit-.376) < 1e-3 and abs(mass_data-.0062) < 1e-4

# Monotone 1-D transport to a 0.3/0.7 target: the gap holds no mass.
def target_cdf(a):
    return .3*norm.cdf(a, -2, .1)+.7*norm.cdf(a, 2, .1)
cut = norm.ppf(.3)
assert abs(cut+.5244) < 1e-4
assert target_cdf(1)-target_cdf(-1) < 1e-15
assert abs(target_cdf(0)-norm.cdf(cut)) < 1e-12

# Weighted categorical fit: probability of the right turn.
def right_probability(left_weights, right_weight):
    return right_weight/(sum(left_weights)+right_weight)
assert np.isclose(right_probability([1, 1], 4), 2/3)
assert np.isclose(right_probability([2, 2], 5), 5/9)

# Hindsight goals, mode switches, and RT-1's flat action table.
H, K = 3, 4
assert sum(H-t for t in range(H)) == H*(H+1)//2 == 6
def mean_switches(n):
    runs = list(product((0, 1), repeat=n))
    total = sum(sum(a != b for a, b in zip(q, q[1:])) for q in runs)
    return total/len(runs)
assert mean_switches(8) == 3.5 and mean_switches(8//K) == .5
assert 256**11 == 2**88
print("Parameters:", params, "densities at 0:", fit.pdf(0), data_pdf(0))
print("Noise cut point:", cut, "expected switches:", mean_switches(8))

Eight exercises with complete solutions

A useful deployment diagnosis names the failure before changing the model: unfamiliar states require coverage, aliased observations require information, incompatible action modes require distributional structure, and a wrong objective requires a task reformulation. Reinforcement learning adds reward-driven interaction, but it inherits rather than erases these modeling questions.