Policy gradients, variance reduction, and actor-critic control

Policy optimization differentiates the performance of an explicit stochastic controller without differentiating unknown dynamics. The likelihood-ratio identity makes that possible; it also produces a noisy learning signal. Causality, baselines, and critics progressively remove irrelevant randomness while preserving the decision-relevant gradient.

Our running controller drives the normalized cart from earlier chapters. State is \(x_t=(p_t,v_t)\), action \(u_t\in\{-1,+1\}\) selects left or right acceleration, and policy \(\pi_\theta(u_t\mid x_t)\) is Bernoulli with right-action probability \(\sigma(\theta^\mathsf T\phi(x_t))\). The feature vector \(\phi(x)\) includes a constant, position, and velocity. A trajectory is \(\tau=(x_0,u_0,\ldots,x_T)\), and return is \(R(\tau)=\sum_{t=0}^{T-1}\gamma^tr(x_t,u_t)\).

Based on AA203 lecture 18 and official slides: score-function policy gradients, REINFORCE, reward-to-go, baselines, actor-critic methods, advantage estimation, and AlphaGo. The Bernoulli variance experiment, explicit assumptions, diagnostics, and exercises are original teaching constructions. The lecture does not establish convergence guarantees for nonlinear deep actor-critic systems. The official slides, which list Berkeley's CS 185/285 course and the Deep RL Bootcamp as their references, add the course roadmap and a taxonomy of reinforcement-learning methods, the likelihood-ratio identity, the maximum-likelihood comparison, a training comparison with and without a baseline, a side-by-side view of fitted Q-learning, REINFORCE, and actor-critic, explicit critic regression targets, actor-critic architecture choices, and applications to maze navigation, language-model alignment, and dexterous manipulation; their trust-region and proximal-update pages are developed in the next chapter, and where their notation differs, the text translates it.

An explicit stochastic policy induces a trajectory distribution 2:04

Let initial state follow \(\rho_0\) and unknown environment dynamics be \(P(x_{t+1}\mid x_t,u_t)\). The policy induces

The reinforcement-learning objective is \(J(\theta)=\mathbb E_{\tau\sim p_\theta}[R(\tau)]\). Sampling \(N\) independent rollouts estimates it by \(N^{-1}\sum_iR(\tau_i)\). This empirical return evaluates a policy but does not yet say how parameters should change.

Policy optimization seeks \(\theta^\star\in\arg\max_\theta J(\theta)\). A stochastic policy supplies exploration and a differentiable log probability. Deterministic policies require a different gradient argument. On a finite batch, maximizing sampled return directly can overfit sampled noise; the derivation below instead gives an unbiased gradient under its regularity and on-policy sampling assumptions.

Policy optimization is one of two model-free families, and its place in the course explains the choice. Reinforcement learning belongs to the adaptive optimal control branch, beside imitation learning and model-based reinforcement learning: in all three the controller improves from data because the dynamics are not available in closed form. A model-based method uses a transition law, either because \(P\) is given or because a model \(\hat P\approx P\) is fitted from observed transitions and then used for planning, the route the next chapter develops. A model-free method never forms \(P\); the dynamics enter only through the transitions it samples. The value-based family of the previous chapter learns a state value \(V\) or an action value \(Q\), the expected return from a state or from a state-action pair (defined precisely in the critic section), and defines the policy implicitly, for example by \(\pi(x)\in\arg\max_uQ(x,u)\). Policy optimization instead represents \(\pi_\theta\) explicitly and adjusts \(\theta\) to increase \(J(\theta)\) directly. Both families aim at the same optimal behavior; they differ in which object is parameterized and which quantity each iteration improves.

The official slides write several of these objects with different symbols. Their stage reward is \(R(x_t,u_t)\) and their trajectory return \(r(\tau)\), the reverse of this chapter's \(r(x_t,u_t)\) and \(R(\tau)\). They write the transition law as \(T(x_{t+1}\mid x_t,u_t)\) or \(p(x_{t+1}\mid x_t,u_t)\), the initial law as \(p(x_0)\), and the horizon as \(T\) or \(H\); this chapter writes \(P\), \(\rho_0\), and \(T\). From the policy-optimization objective onward they set \(\gamma=1\), so each undiscounted slide formula is this chapter's formula at \(\gamma=1\); the chapter keeps \(\gamma^t\) and the terminal mask because discounted implementations need them. Their time sums run from \(t=1\) to \(T\), and their interaction loop labels the reward that follows action \(u_t\) as \(r_{t+1}\). Here the reward earned by \(u_t\) is \(r_t=r(x_t,u_t)\), and every sum runs over the decisions \(t=0,\ldots,T-1\). In particular, the trajectory density above contains one policy factor and one transition factor for each decision \(u_0,\ldots,u_{T-1}\). A product running from \(t=1\) to \(T\) would omit \(\pi_\theta(u_0\mid x_0)\), and with it the score of the first action, and would include a transition to a state \(x_{T+1}\) that lies beyond the trajectory.

The log-derivative removes unknown dynamics from the gradient 9:50

Assume support does not change abruptly with \(\theta\) and differentiation may pass through the integral. Then

The second line is the step that makes the gradient computable. The first line integrates \(\nabla_\theta p_\theta(\tau)\), which is not a probability density, so the integral cannot be estimated by averaging over sampled trajectories; it also depends on the unknown \(\rho_0\) and \(P\) through \(p_\theta\). The hinge is the identity \(p_\theta(\tau)\nabla_\theta\log p_\theta(\tau)=p_\theta(\tau)\,\nabla_\theta p_\theta(\tau)/p_\theta(\tau)=\nabla_\theta p_\theta(\tau)\), valid wherever \(p_\theta(\tau)\gt0\). Substituting it puts the density \(p_\theta\) back in front of the integrand, so the gradient becomes an expectation over trajectories of the current policy, which rollouts can estimate. The unknown factors have not yet disappeared: they remain inside \(\log p_\theta(\tau)\), and the factorization removes them next.

Taking the log of the trajectory factorization turns products into sums. Initial-state and transition terms do not depend directly on policy parameters, so

No derivative of \(P\) remains. Dynamics still affect which trajectories are sampled, so the method is model free rather than environment free. The resulting estimator is

For the Bernoulli cart, encode \(a=1\) for right and zero for left, with \(q_\theta(x)=\sigma(\theta^\mathsf T\phi(x))\). Then \(\nabla_\theta\log\pi_\theta(a\mid x)=(a-q_\theta(x))\phi(x)\). The return weights a familiar maximum-likelihood score: increase probabilities of sampled behavior with positive centered weight and decrease those with negative weight.

REINFORCE alternates fresh rollouts and ascent steps 18:21

REINFORCE runs the current policy, estimates the score-function gradient, and updates \(\theta\leftarrow\theta+\alpha\widehat g\) for step size \(\alpha>0\). It then collects new data because the trajectory distribution changed. Reusing an old batch without correction no longer estimates the current on-policy expectation.

REINFORCE is the simplest instance of a three-stage loop that organizes every algorithm in this part of the course. The first stage generates samples by running the current policy to obtain \(\{\tau_i\}_{i=1}^N\). The second fits a model or estimates return; REINFORCE does the least possible here, a Monte Carlo estimate \(R(\tau_i)\) formed by summing the rewards observed along each rollout. The third improves the policy, here by one ascent step on \(\theta\). One pass through the loop is one gradient step, and an outer iteration repeats the pass, because the improved policy must collect its own trajectories before the next gradient can be estimated. Value-based methods fill the same three stages differently, and the actor-critic methods below combine stages from both.

Unlike behavior cloning, the algorithm needs no expert action label. It weights its own sampled actions by outcome. A positive return increases the joint log probability of all actions in the trajectory; a negative return decreases it. This crude assignment explains both its trial-and-error character and its variance: every action receives the same total return even when most actions did not cause it.

The comparison with behavior cloning can be made exact. Given \(N\) demonstrated trajectories of state-action pairs, maximum-likelihood estimation ascends the average log-likelihood of the demonstrated actions, with gradient estimate

The policy-gradient estimate \(\widehat g\) contains the same per-trajectory score, multiplied by \(R(\tau_i)\). It differs in two respects: the trajectories come from the learner's own policy rather than from an expert, and each trajectory's log-likelihood gradient is weighted by the return it earned. The policy gradient is therefore a return-weighted maximum-likelihood gradient, and a step along it raises the probability of high-return trajectories more than that of low-return ones. If every return equaled the same constant \(c\), the estimate would be \(c\) times the maximum-likelihood gradient on the learner's own samples. Its expectation is zero, because \(\mathbb E_{\tau\sim p_\theta}[\nabla_\theta\log p_\theta(\tau)]=\nabla_\theta\int p_\theta(\tau)\,d\tau=0\): when outcomes do not differ, trial and error has nothing to learn in expectation, and any sampled step is noise.

The score depends only on the chosen action distribution, so one derivation serves discrete and continuous actions. For a categorical policy over finitely many actions, such as four movement directions, \(\log\pi_\theta(u\mid x)\) is the logarithm of the probability assigned to the sampled action; the Bernoulli cart is the two-action case. For a continuous acceleration with Gaussian policy \(\pi_\theta(u\mid x)=\mathcal N(u;\mu_\theta(x),\sigma^2)\), mean \(\mu_\theta(x)\), and fixed standard deviation \(\sigma\gt0\), the score is

Weighted by a positive return, it moves the mean toward the sampled acceleration; weighted by a negative return, away from it. The size of the move is proportional to the deviation \(u-\mu_\theta(x)\) divided by the variance, so the same deviation produces a larger update when the policy explores less.

Return sign is arbitrary under reward translation, yet an uncentered finite-sample update reacts to it. If every trajectory has positive return, even bad trajectories initially push their sampled actions upward. In expectation the score has mean zero, but centering makes the finite estimator express “better or worse than reference” directly.

Reward-to-go removes rewards an action could not have caused 32:43

Variance is the defect the rest of the chapter attacks. The estimate \(\widehat g\) averages \(N\) independent draws of the random vector \(s(\tau)R(\tau)\), where \(s(\tau)=\sum_t\nabla_\theta\log\pi_\theta(u_t\mid x_t)\) is the trajectory score. Its covariance is therefore that vector's covariance divided by \(N\), and its root-mean-square error falls only as \(N^{-1/2}\). When the per-trajectory variance is large, two batches drawn from the same policy can point in substantially different directions. Learning then needs either many rollouts per update or a step size small enough that noisy directions average out over many updates; both slow convergence, and an unlucky step can lower performance. Much of the research on policy optimization consists of ways to reduce this variance without changing the expected gradient.

Reward translation shows how much variance can enter without any change in the expected gradient. Adding a constant \(c\) to every return changes the per-trajectory term from \(s(\tau)R(\tau)\) to \(s(\tau)(R(\tau)+c)\). The mean is unchanged because \(\mathbb E[s(\tau)]=0\), but the second moment \(\mathbb E[\|s(\tau)\|^2(R(\tau)+c)^2]\) is a quadratic in \(c\) whose leading term is \(c^2\,\mathbb E\|s(\tau)\|^2\). In the one-state Bernoulli bandit of the baseline section, with right-action probability \(q=0.5\), reward 3 for action 1 and reward 1 for action 0, the per-sample variance is \((c+2)^2/4\). It equals 1 at \(c=0\), 36 at \(c=10\), and 0 at \(c=-2\), the negative of the mean reward. A shift that leaves the decision problem unchanged thus multiplies the variance by 36, which would require 36 times as many rollouts for the same standard error, while the shift that centers the returns removes the variance entirely. Exercise 7 derives these values.

Action \(u_t\) cannot affect rewards before time \(t\). Terms involving earlier rewards have zero expected contribution to its score under the causal trajectory factorization. Therefore the equivalent estimator uses

The equivalence is a conditioning argument. For \(k\lt t\), the reward \(r_k\) is fixed by \(x_k\) and \(u_k\), and therefore by the history \(h_t=(x_0,u_0,\ldots,x_t)\). Conditional on \(h_t\), the action \(u_t\) is drawn from \(\pi_\theta(\cdot\mid x_t)\), so

with an integral in place of the sum for continuous actions. Every discarded cross term has zero expectation, so dropping them leaves the expected gradient unchanged and alters only the sampled estimate.

Removing irrelevant past rewards typically reduces variance and improves credit assignment. Discount conventions must remain consistent. For the stated objective \(\sum_t\gamma^tr_t\), the outer \(\gamma^t\) multiplying the relative reward-to-go is required. Episodic implementations sometimes omit it and thereby optimize a slightly different time weighting.

For the cart, a steering action after the cart already passed an obstacle cannot cause the earlier clearance reward. Giving that reward to the later score adds noise. A later terminal success can depend on the earlier action, so it remains in that action's reward-to-go.

An action-independent baseline preserves expectation and can cut variance 35:20

Subtract a baseline \(b(x_t)\) that does not depend on the sampled action:

Conditional on state,

Thus a fixed or state-dependent action-independent baseline does not change the expected gradient. An action-dependent baseline requires correction and cannot be inserted by the same proof. If the baseline shares parameters with the actor, the actor loss should normally stop its gradient through the advantage weight; the critic receives its own regression loss.

The batch mean return is a simple baseline. Using the same sample to compute its own mean introduces a small finite-batch dependence; leave-one-out means or a learned value avoid that detail. The variance-minimizing scalar baseline is generally a score-norm-weighted expectation, not necessarily the raw mean, though the mean is cheap and often effective.

from statistics import fmean, pvariance

# One-state Bernoulli bandit: action 1 earns 3, action 0 earns 1; q=0.5.
samples = [(a, r) for a, r in ((0, 1.0), (1, 3.0)) for _ in range(50)]
raw = [(a - 0.5) * r for a, r in samples]
baseline = fmean(r for _, r in samples)
centered = [(a - 0.5) * (r - baseline) for a, r in samples]
assert abs(fmean(raw) - 0.5) < 1e-12
assert abs(fmean(centered) - 0.5) < 1e-12
assert pvariance(centered) < pvariance(raw)
print(pvariance(raw), pvariance(centered))

The exact enumerated sample keeps the mean gradient \(0.5\) while the baseline reduces variance from \(1.0\) to zero. Real multi-step tasks do not usually admit a perfect baseline, so variance decreases rather than vanishes.

The reduction changes learning speed, not only a statistic. In one training comparison over 1000 episodes of an episodic task, REINFORCE with the average-return baseline and REINFORCE without it start at episode rewards of about −76 and −85, respectively. With the baseline, the episode reward reaches a reference level of −11.6 after about 60 episodes and then fluctuates around it. Without the baseline, the same algorithm reaches about −18 after 200 episodes and about −15 after 400, and it approaches the level of the baseline run only in the final few hundred episodes. The task, policy class, step size, and number of seeds are not documented, so the comparison shows the direction and approximate size of the effect for one configuration rather than a general speed-up factor. Because the two runs differ only in the gradient estimator, the gap measures sample efficiency; in harder problems the same difference in estimator quality can decide whether training reaches a good policy at all.

Policy gradients trade direct optimization for on-policy sampling 45:00

Policy optimization naturally handles continuous actions through distributions such as Gaussians and optimizes the control objective directly. It does not require a max over actions or a differentiable environment. The objective is nonconvex, gradient estimates can be noisy, and a local optimum need not be globally good.

Vanilla REINFORCE is on policy: rollouts are distributed by the policy whose gradient is estimated. Importance ratios can reuse another policy's actions, but products over long trajectories can have extreme variance, and state-distribution shift remains consequential. Practical algorithms limit policy changes, truncate ratios, or use replay only with additional off-policy machinery.

Sample efficiency and wall-clock efficiency differ. On-policy interaction may be expensive on a robot but cheap in massively parallel simulation. A model-free update may avoid model bias while requiring more physical trials. Choose the accounting unit relevant to the system.

A critic replaces a noisy realized future with a conditional expectation 51:17

For a finite horizon, include time in the state (or equivalently use time-indexed value functions). The action value \(Q^\pi(x,u)=\mathbb E[G_t\mid x_t=x,u_t=u]\) is then the conditional mean of reward-to-go. Replacing a single realized \(G_t\) with an accurate critic reduces conditional variance:

The actor is \(\pi_\theta\); the critic \(Q_\phi\) estimates its consequences. Approximation error introduces bias into the actor update. A critic is useful when its reduction in variance outweighs that bias.

A state-dependent baseline is \(V^\pi(x)=\mathbb E_{u\sim\pi}[Q^\pi(x,u)]\). Subtraction gives advantage \(A^\pi(x,u)=Q^\pi(x,u)-V^\pi(x)\), answering how much better an action is than the policy's average action at that state. The expected advantage under the policy is zero.

For the cart at a state drifting right of target, right acceleration can have negative advantage even if its absolute expected return is positive. The actor should reduce it relative to the available alternative, which is exactly what centering expresses.

Advantage actor-critic can learn one value network 1:06:53

Actor-critic methods are easiest to place by comparing the three-stage loops of the algorithms they combine. Fitted Q-learning, from the previous chapter, generates samples and then fits a critic by regression over a batch of \(B\) sampled transitions: with targets \(y_t=r_t+\gamma\max_{u'}Q_\phi(x_{t+1},u')\) held fixed, it updates \(\phi\) to reduce

and it improves the policy greedily, by \(\pi(x)\in\arg\max_uQ_\phi(x,u)\). REINFORCE generates samples, estimates returns by Monte Carlo sums, and improves the policy by a gradient step. An actor-critic method takes its middle stage from the first and its last stage from the second: it fits \(V^\pi\), \(Q^\pi\), or \(A^\pi\) by regression and then takes the step \(\theta\leftarrow\theta+\alpha\widehat g\) with the fitted quantity in place of the Monte Carlo return. Because the critic evaluates the current policy \(\pi_\theta\) rather than the optimal policy, its regression targets contain no maximization over actions.

Which quantity the critic fits has no single correct answer; algorithms differ. One consideration is the size of the regression problem. The state value \(V^\pi\) takes only the state as input, two numbers for the cart, whereas \(Q^\pi\) and \(A^\pi\) take a state-action pair: for a continuous action they must generalize across actions as well as states, and for a discrete action set a network typically returns one output per action. Fitting \(V^\pi\) is the smallest of the three problems, which is what makes the following construction attractive.

Learning separate \(Q\) and \(V\) networks is unnecessary for a one-step advantage estimate. The Bellman relation motivates

This temporal-difference residual is a one-sample estimate of advantage when the value is correct. The critic minimizes a regression loss toward Monte Carlo or bootstrapped targets. The actor ascends \(\log\pi_\theta(u_t\mid x_t)\widehat A_t\). Entropy regularization is often added to discourage premature collapse, but it changes the objective unless its coefficient is treated as part of the task.

The one-step estimate rests on the Bellman relation for the policy's own action value. Splitting the reward-to-go into its first reward and the remainder, and conditioning on the successor state, gives

because after reaching \(x_{t+1}\) the expected remaining discounted reward under \(\pi\) is \(V^\pi(x_{t+1})\), and it is zero after a terminal transition, where the indicator \(d_{t+1}\) equals one. Replacing the expectation by the single observed successor gives an estimate of \(Q^\pi(x_t,u_t)\) whose conditional mean given \((x_t,u_t)\) is exact when the value function is exact. Subtracting \(V^\pi(x_t)\) then gives \(\mathbb E[\widehat A_t\mid x_t,u_t]=A^\pi(x_t,u_t)\), so one value network supplies both terms of the advantage. With a learned \(V_\phi\) in place of \(V^\pi\), the conditional mean is off by \(\gamma\,\mathbb E[(1-d_{t+1})(V_\phi-V^\pi)(x_{t+1})\mid x_t,u_t]-(V_\phi-V^\pi)(x_t)\); this is the bias the critic introduces. A single sample of the successor is an estimate, not an identity: \(\widehat A_t\) equals the advantage only in conditional expectation.

Fitting \(V_\phi\) is ordinary supervised regression once each visited state carries a label. A Monte Carlo label uses the observed reward-to-go: collect \(\mathcal D_{\rm MC}=\{(x_{i,t},G_{i,t})\}\) and minimize \(\sum(V_\phi(x_{i,t})-G_{i,t})^2\) over \(\phi\). The label is an unbiased sample of \(V^\pi(x_{i,t})\), but it carries the variance of the entire remaining trajectory and becomes available only when the episode ends. A temporal-difference label bootstraps from the critic: collect \(\mathcal D_{\rm TD}=\{(x_{i,t},\,r_{i,t}+\gamma(1-d_{i,t+1})V_{\bar\phi}(x_{i,t+1}))\}\), where \(\bar\phi\) is a copy of the critic parameters held fixed while the regression is solved, and minimize the same squared error. This label contains one sampled reward and one sampled transition, so its variance is lower and it is available after a single step; it is biased wherever \(V_{\bar\phi}\) is wrong. The bootstrap must be evaluated at the successor state. A label \(r_t+\gamma V(x_t)\) at the current state has the fixed point \(V(x)=\bar r(x)/(1-\gamma)\), where \(\bar r(x)\) is the expected immediate reward at \(x\): the value of collecting that reward forever, whatever the policy actually does next. Exercise 8 computes the resulting error.

One-step TD lowers variance and adds bootstrap bias. Multi-step returns or generalized advantage estimation interpolate. Terminal masking is mandatory. Normalize advantages only with a documented convention; it rescales the update and can change relative weighting across batches.

Alternating actor and critic updates creates a moving-target system: the policy changes the critic's data, while the critic changes the policy gradient. Monitor both losses, explained variance, entropy, KL change, returns, and advantage scale. A low critic loss on a narrow batch does not establish correct policy improvement.

Implementation details determine which gradient is actually estimated

A rollout record should contain observations, sampled actions, log probabilities under the behavior policy, rewards, termination flags, and critic predictions. Compute returns backward, resetting accumulation at genuine terminal states. Standardize observations using documented statistics. Then construct actor weights from reward-to-go or advantage and critic targets from returns or bootstraps. Mixing an action log probability recomputed under a changed policy with an old trajectory silently turns an on-policy update into an uncorrected off-policy one.

The actor loss is commonly written with a minus sign because software minimizes: \[ L_{\rm actor}=-\frac1B\sum_{i,t}\log\pi_\theta(u_{i,t}\mid x_{i,t})\,\operatorname{stopgrad}(\widehat A_{i,t}). \] The critic loss may be \(L_{\rm critic}=B^{-1}\sum(V_\phi(x_{i,t})-\widehat V_{i,t})^2\). Stop-gradient prevents the actor objective from changing the critic merely to make advantages favorable. If actor and critic share an encoder, gradients from their two losses still meet in shared parameters; the relative loss coefficient then changes the representation learned by both.

That coupling is one side of an architectural choice. The actor and critic can be two separate networks, one mapping \(x\) to \(V_\phi(x)\) and one mapping \(x\) to \(\pi_\theta(\cdot\mid x)\), or one network with a shared body and two output heads. Two networks are simple and typically more stable: each loss changes only its own parameters, so a poorly scaled critic loss cannot distort the policy's representation. Their cost is that they share no features. Each must learn its own representation of the state, which matters most when the observation is an image and feature learning dominates the computation. A shared body learns one representation for both purposes at the price of the loss-weighting coupling just described. Separate networks remain in use at scale; the PPO-trained robot-hand policy in the applications section has a value network separate from its policy network.

Rollouts are usually collected by several workers, each running the policy in its own copy of the environment, and the workers can be coordinated synchronously or asynchronously. In the synchronous design every worker collects transitions \((x_t,u_t,x_{t+1},r_t)\) with the same parameters, the transitions are pooled, one update of \(\theta\) and \(\phi\) is computed from the whole batch, and all workers continue with the new parameters. The pooled data are exactly on-policy for the parameters being updated, and the larger batch lowers gradient variance as a larger \(N\) would. In the asynchronous design each worker reads the parameters from a shared store, collects a short segment, computes a gradient, and applies it to the store without waiting for the others. No worker waits for the slowest one, and workers in different parts of the state space decorrelate successive updates. The price is staleness: a gradient computed under the parameters \(\theta_{\rm old}\) that the worker read is applied after other workers have already changed them, so it is slightly off-policy for the parameters it updates.

For a Gaussian continuous controller, policy output should respect actuator semantics. A diagonal normal in an unconstrained latent space followed by \(u=u_{\max}\tanh z\) enforces bounds, but the log probability of the bounded action requires the change-of-variables Jacobian. Clipping a sampled Gaussian action is not equivalent: many latent actions collapse to the same boundary action, and using the unclipped log density yields the wrong score. Track pre-squash values or use a distribution implementation that computes the transformed density.

Numerical stability matters. Parameterize standard deviation through a bounded log scale, use stable categorical log-softmax, and compute entropy from the same distribution used to sample. Extremely small variance makes a continuous policy nearly deterministic and can create large score derivatives. Excessive entropy prevents exploitation. Plot both action histograms and distribution parameters rather than relying on the entropy scalar alone.

Advantage normalization within a batch replaces \(\widehat A\) by \((\widehat A-\bar A)/(s_A+\varepsilon)\). Centering resembles a sampled baseline; division adapts update scale. It usually improves optimization but changes how batches with different return dispersion are weighted. With very small batches, the sample standard deviation is noisy, and a single outlier can shrink every other update.

Policy updates should be evaluated against a frozen pre-update policy. Record the empirical KL divergence, importance-ratio distribution, and fraction of actions whose probabilities changed sharply. A large step can invalidate advantages estimated under the old policy even when the batch loss improves. This motivates the trust-region and proximal methods continued in lecture 19.

For the cart, begin with a reproducible one-state bandit, then a short deterministic horizon, then uncertain dynamics. In the bandit, compare automatic differentiation with the analytic Bernoulli gradient. In the short horizon, enumerate all action sequences and compare the exact expected return and gradient with Monte Carlo estimates. Only after these agree should simulator randomness, a learned critic, and neural features be introduced. This staged oracle separates estimator bugs from sampling variance.

A minimal experiment report includes reward definition, horizon and discount, termination semantics, policy distribution and transform, actor and critic architectures, shared layers, return estimator, baseline, normalization, entropy coefficient, optimizer, gradient clipping, batch size, rollouts per update, seeds, and evaluation protocol. Policy-gradient results are otherwise difficult to interpret because each choice changes either the objective, estimator variance, or optimization dynamics.

Finally, separate training and evaluation behavior. Evaluation should disable exploratory sampling only when the deployed controller will do so, and should otherwise sample according to the intended policy. For a Gaussian controller, reporting only the mean-action rollout can hide a variance parameter that produces frequent unsafe actions. Compare mean-action and sampled-policy returns, constraint violations, and tail outcomes. A higher average return is not enough when the action distribution occasionally leaves the admissible control envelope.

Use confidence intervals over independently trained seeds and independently sampled evaluation episodes. Episodes from one trained policy measure execution noise; seeds also measure optimization instability. Pooling both without hierarchy obscures which source dominates. Preserve learning curves rather than only the best checkpoint, and choose checkpoints using a validation rule fixed before the final test. Selecting the maximum noisy evaluation and reporting it on the same episodes produces optimistic bias.

AlphaGo combined policy learning, value learning, self-play, and search 1:13:51

The lecture uses AlphaGo to show these components in a landmark system. A policy network maps board states to move probabilities; a value network estimates the winner from a position; self-play supplies games; policy-gradient updates use outcomes relative to a value baseline; Monte Carlo tree search combines learned priors and values with explicit lookahead.

Search is not a minor detail. Sampling directly from the policy and selecting moves through search induce different behavior. Supervised learning from expert games provided an initial policy in the published AlphaGo system, followed by reinforcement learning; later AlphaGo Zero removed human game data and learned from self-play under a different design. Keeping those systems distinct avoids rewriting historical evidence.

The result demonstrates engineering effectiveness on Go, not a general convergence theorem for deep actor-critic learning. Self-play also changes the opponent distribution over training. Evaluation against fixed historical opponents and held-out versions is needed to distinguish improvement from cyclic exploitation.

Large-scale applications wrap the same estimators in new architectures and data pipelines

The estimators of this chapter are the core of several further large-scale systems; what changes is the function class, the way data are collected, and the update rule wrapped around the gradient. Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning” (ICML 2016), trained an advantage actor-critic with the asynchronous parallel workers described in the implementation section, and applied it, among other domains, to navigating random three-dimensional mazes from first-person visual input. The policy and value networks combined convolutional layers, which process the images, with recurrent layers. The recurrent layers answer a state-sufficiency question: a single camera image does not determine the agent's position in the maze, so the image is not a Markov state, and the network's recurrent memory summarizes the observation history in its place. The paper reports that parallel actor-learners stabilized training. That is an empirical finding for its architectures and tasks, not a convergence guarantee.

Language-model alignment by reinforcement learning from human feedback optimizes the same objective with a learned reward. In the pipeline of Ouyang et al., “Training language models to follow instructions with human feedback” (2022), a pretrained model is first fine-tuned by supervised learning on human demonstrations, which is behavior cloning in this chapter's terms. Next, a prompt and several model outputs are sampled, a human labeler ranks the outputs from best to worst, and the comparisons train a reward model that maps a prompt and a response to a scalar score. Finally, the policy, initialized from the supervised model, is optimized against the reward model with PPO, the clipped-surrogate method of the next chapter: a new prompt is sampled, the policy generates a response, the reward model scores it, and the score drives the policy update. One prompt and its response form one episode, with a single reward at the end. Read token by token, the state is the prompt together with the text generated so far, an action is the next token, and the policy is the model's next-token distribution.

The reward model is a learned proxy for human preference, and the policy optimizer searches for responses that the proxy scores highly, including responses on which its errors are favorable. This is the critic-bias problem in a sharper form, since the reward itself is learned, not only an estimate of its expectation. Ouyang et al. add a per-token penalty on the policy's divergence from the supervised model to limit this over-optimization of the reward model.

Dexterous manipulation illustrates the simulation side of the sample-efficiency accounting in the properties section. OpenAI et al., “Solving Rubik's Cube with a Robot Hand” (2019), trained control policies with PPO only in simulation and then used them to manipulate a Rubik's cube with a physical humanoid robot hand. On-policy sampling that would be impractical on hardware becomes affordable in simulation. The cost moves to the gap between simulator and robot: the policy is optimized for trajectories distributed according to the simulator, so its real performance depends on how well the simulated distribution covers the physical one. The paper addresses the gap with automatic domain randomization, training a recurrent policy on an ever-growing distribution of randomized simulated environments. Its hardware results are empirical evidence for that system, not a general guarantee that simulation-trained policies transfer.

Exercises with worked solutions

What this chapter establishes

The score-function identity yields a dynamics-free policy gradient from on-policy samples. Reward-to-go removes impossible causal credit; action-independent baselines preserve expectation; critics replace realized futures with conditional means; and advantage actor-critic combines value regression with explicit policy improvement. Every reduction in variance introduces an estimation or systems choice. Stable learning therefore requires checking critic bias, policy drift, support, terminal semantics, and actual evaluation return.