Learning values from interaction: Monte Carlo, temporal differences, and exploration

A delivery cart repeatedly enters an unfamiliar loading bay. Its route planner knows the available commands but not how often a slippery floor makes the cart remain in place. A perfect Bellman solver cannot help until someone supplies the missing transition probabilities. Can the cart learn which routes are valuable from the transitions it actually experiences, and can it improve its route without first identifying a full dynamics model?

This chapter keeps that question separate from neural approximation. We first allow one stored number per state or state-action pair. This removes representation error so that the central issues become visible: what observations form a learning target, when that target becomes available, and which decisions the data actually cover.

The topic sequence follows AA203 lecture 16 and its official slides. The cart calculations, variance derivation, executable checks, and exercises are original teaching constructions. Local machine transcription establishes timing; slide text and primary research correct notation and qualify informal claims. The official slides add the course roadmap, the model tuple and optimality equations, the prediction-control table of exact methods with the gridworld's later sweeps and greedy policies, the counter form of Monte Carlo evaluation, the blackjack reward rules and optimal policy, the exhaustive-search corner of the backup square, the two-door numbers, and the formulas of the algorithm taxonomy and skeleton; where their notation differs, the text translates it.

A value prediction belongs to a policy and a state model 2:54

Let \(\mathcal X\) be a finite state set and \(\mathcal U(x)\) the finite actions available at state \(x\). At integer decision time \(t\), the cart observes \(x_t\), selects \(u_t\), receives reward \(r_{t+1}\), and enters \(x_{t+1}\). A reward is a scalar preference score, measured here in normalized utility units; it is not necessarily money or physical energy. We use the next-time reward subscript consistently to mean the reward produced by the current action.

The transition law \(P(x'\mid x,u)\) specifies next-state probabilities. The expected immediate reward is \(R(x,u)=\mathbb E[r_{t+1}\mid x_t=x,u_t=u]\). A stationary policy \(\pi(u\mid x)\) is a distribution over actions that depends on the current state, not the decision index. The Markov assumption requires this state to contain the information needed to predict the next transition and reward. For a cart, position alone may fail if velocity or battery charge changes what happens next.

Choose discount \(0\leq\gamma<1\). For an episode ending at decision time \(T\), define return and state value by

Continuing tasks replace the finite sum by an infinite series. Bounded rewards, say \(|r_{t+1}|\leq R_{\max}\), give \(|G_t|\leq R_{\max}/(1-\gamma)\). Notice that the exponent starts at zero for the next reward. A value at time 500,000 need not be small; what becomes small is its discounted contribution to a much earlier decision.

The action value \(Q^\pi(x,u)\) conditions on taking \(u\) first and following \(\pi\) afterward. Splitting the return after one reward yields the Bellman expectation equation

Set terminal value to zero, with any terminal payoff included in the reward on entering that state. This prevents counting the same success bonus twice. The equation is a consistency requirement, not yet a learning rule: its right side still contains the unknown transition law. A model-free update will replace explicit summation over that law with observed transitions. It may obtain those observations from a simulator; model-free describes what the update uses, not whether the environment has an underlying model.

Learning from interaction locates the chapter within the course. The preceding chapters solved optimal control problems with a known model: open-loop methods, indirect and direct, computed control sequences; closed-loop methods, dynamic programming and the Hamilton–Jacobi–Bellman and Hamilton–Jacobi–Isaacs equations, computed policies; and model predictive control sits between the two by re-solving an open-loop problem from each measured state. The learning branch that follows, adaptive optimal control, divides into imitation learning, model-free reinforcement learning, and model-based reinforcement learning. Imitation learning, developed in the two previous chapters, learns from demonstrations, either by fitting the demonstrated actions or by inferring the reward that the demonstrator appears to optimize; a policy that reproduces the demonstrated actions exactly attains the demonstrator's performance and no more. Reinforcement learning learns from interaction instead, by trial and error, with the reward itself as the target. This chapter and the next two treat model-free methods, and the chapter after them treats model-based methods.

The control problem asks for the best policy rather than the value of a given one. Its objective is to choose \(\pi\) to maximize the expected discounted return \(\mathbb E_\pi[\sum_{t\geq0}\gamma^tr_{t+1}]\), with the expectation taken over the initial state, the actions, and the transitions. The optimal state value \(V^\star(x)=\max_\pi V^\pi(x)\) and the optimal action value \(Q^\star(x,u)=\max_\pi Q^\pi(x,u)\) satisfy the Bellman optimality equations

with terminal states again valued at zero. They replace the policy average of the expectation equation by a maximum over the next action. For a finite discounted model, a policy \(\pi^\star\) that chooses at every state an action in \(\arg\max_uQ^\star(x,u)\) is stationary, deterministic, and optimal from every initial state at once. The hinge is uniqueness, which the contraction of the next section supplies. The expectation equation of \(\pi^\star\) is satisfied by \(V^\star\), because its chosen action attains the maximum, so its unique solution is \(V^{\pi^\star}=V^\star\). For any other stationary policy, the right side of its expectation equation evaluated at \(V^\star\) is at most \(V^\star\), since an average cannot exceed a maximum, and repeating that step gives \(V^\pi\leq V^\star\). The chapter on Bellman solvers develops the same facts through value iteration and policy iteration. Among stationary policies, the maximum in the objective is therefore attained by a policy that does not depend on the initial-state distribution.

The action value of a fixed policy satisfies its own expectation equation. Conditioning on the first action and then averaging the next action under the policy gives

For a deterministic policy the inner average is the single entry \(Q^\pi(x',\pi(x'))\). For \(0\leq\gamma\lt1\) each expectation equation has exactly one bounded solution, the value of the policy, as the contraction argument of the next section shows. In tabular form \(V^\pi\) is a vector with one entry per state and \(Q^\pi\) an array with one entry per admissible state-action pair. The larger array buys a model-free decision rule: a greedy action can be read as \(\arg\max_uQ(x,u)\), whereas a greedy action computed from \(V\) needs \(R\) and \(P\) to form the bracket of the optimality equation.

The official course slides write several of these objects differently. They collect the model in the tuple \(\mathcal M=(\mathcal X,\mathcal U,T,R,\gamma)\), whose transition function \(T(x_{t+1}\mid x_t,u_t)\) is this chapter's \(P(x'\mid x,u)\); here \(T\) is reserved for the terminal decision time and \(\mathcal T^\pi\) for the Bellman operator of the next section. They write \(V_\pi\) and \(Q_\pi\) for \(V^\pi\) and \(Q^\pi\), and \(V^*\) and \(Q^*\), or \(v_\ast\) and \(q_\ast\) in figures taken from Sutton and Barto, for \(V^\star\) and \(Q^\star\); their reward function \(R(x_t,u_t)\) is deterministic, a special case of the expected reward used here. They also give the same reward two time indices. The return is written \(G_t=R_{t+1}+\gamma R_{t+2}+\cdots\), which matches this chapter's \(r_{t+1}\), whereas the reward function writes the reward produced by \(u_t\) as \(r_t=R(x_t,u_t)\), and the temporal-difference target of a later section is written \(R_t+\gamma\widehat V(x_{t+1})\), with \(\widehat V\) a learned value table. Both refer to the reward written \(r_{t+1}\) here. Finally, a trajectory \(\tau=(x_0,u_0,\ldots,x_N,u_N)\) has last index \(N\), which plays the role of \(T\).

Exact policy iteration reveals two different obstacles 8:10

Policy iteration alternates evaluation of a fixed policy with improvement of that policy. Exact evaluation solves the Bellman expectation equations. Greedy improvement chooses an action maximizing immediate reward plus expected successor value. Both operations require the transition model when expressed through \(V\). They also require enumerating the state space, a separate obstacle that sampling alone does not remove.

The exact methods pair each problem with a Bellman equation. Prediction asks for the value \(V^\pi\) of a given policy; iterative policy evaluation solves it by applying the expectation equation as an update at every state, \(v_{k+1}(x)=\sum_u\pi(u\mid x)[R(x,u)+\gamma\sum_{x'}P(x'\mid x,u)v_k(x')]\), where \(k\) counts sweeps over the state space. Control asks for an optimal policy and has two exact methods. Policy iteration combines the expectation equation with greedy improvement: with \(i\) counting policies, it evaluates \(V^{\pi_i}\) and then sets

Value iteration applies the optimality equation as an update, \(v_{k+1}(x)=\max_{u\in\mathcal U(x)}[R(x,u)+\gamma\sum_{x'}P(x'\mid x,u)v_k(x')]\), and reads a greedy policy from the final table. All three updates contain \(P\); this is the model requirement named above.

The sweep index and the policy index must remain distinct. If each evaluation is cut to a single sweep from the current table and each improvement is greedy with respect to that table, the two steps merge: for a policy greedy with respect to a table \(v\), the expectation update applied to \(v\) equals the maximum over actions, so every sweep after the first improvement is a value-iteration update. Running a fixed number of sweeps between improvements gives intermediate schemes. All of them alternate evaluation and improvement; they differ in how far evaluation proceeds before each improvement, which is the freedom that generalized policy iteration, introduced later in the chapter, makes explicit.

The recording's gridworld has sixteen cells, fourteen nonterminal states, two terminal corners, four movement commands, and reward minus one per move. Commands pointing beyond a wall leave the agent in place. With uniform random actions and all initial estimates zero, the first synchronous evaluation sweep assigns minus one to every nonterminal cell. A second sweep at a cell with one terminal neighbor gives \(-1+(0-1-1-1)/4=-1.75\); an interior cell whose neighbors are all nonterminal gives minus two. The difference communicates proximity to completion.

This example uses \(\gamma=1\), outside the discounted contraction assumption above. It is still meaningful for a policy with finite expected time to absorption: return is minus the number of remaining steps. A policy that forever pushes against a wall has return minus infinity. Thus “episodic” must involve actual termination behavior, not merely a terminal cell drawn somewhere in the diagram.

This gridworld is Example 4.1 of Sutton and Barto's Reinforcement Learning: An Introduction (second edition, MIT Press, 2018), and further sweeps show how slowly the random policy's value forms. After three sweeps the four cells adjacent to a terminal corner hold \(-39/16=-2.4375\), and every other nonterminal cell holds a value between \(-3\) and \(-2.875\). After ten sweeps the values range from about \(-6.14\) at the cells adjacent to a terminal corner to about \(-8.97\) at the two nonterminal corners. The sweeps converge to integers: \(-14\) at the four cells adjacent to a terminal corner, \(-18\) at the two interior cells on the diagonal between the terminal corners, \(-22\) at the two nonterminal corners, and \(-20\) at the remaining six cells. Because every move costs one and \(\gamma=1\), each limit is minus the expected number of moves before the random walk reaches a terminal corner. A first-step check confirms one entry. From the cell to the right of the upper-left terminal, the four moves lead into the terminal, back to the same cell against the wall, to the diagonal interior cell below, and to the next cell on the right, so \(-1+(0-14-18-20)/4=-14\).

Greedy improvement with respect to the converged values points each cell toward a neighbor of highest value, and in this example the result is an optimal policy. The optimal value of a cell is minus its smallest number of moves to a terminal corner, so a nonterminal corner, worth \(-22\) under the random policy, is worth \(-3\) under an optimal one. A single improvement therefore solves the problem: evaluating the improved policy and improving again leaves an optimal policy, and policy iteration stops when ties are broken in favor of the current action. The greedy policy also becomes useful long before evaluation converges. With respect to the zero initial table every move ties, so the greedy policy is the random policy itself. With respect to the one-sweep table, only the four cells adjacent to a terminal corner have a unique greedy action, the move into the terminal; at every other cell all four moves tie. With respect to the two-sweep table, the greedy actions are optimal except at the two nonterminal corners, where all four moves still tie, including the two that push into a wall. From the third sweep on, every greedy action, ties included, is optimal. This early agreement reflects the example's equal move costs and deterministic moves. The improvement guarantee stated below applies to greedy improvement with respect to the exact value of the current policy; an improvement computed from an unconverged table has no such guarantee in general.

In a discounted problem, define \(\mathcal T^\pi v\) as the Bellman right side with table \(v\). Averaging cannot increase the largest absolute entrywise difference, so

The sup norm \(\|v\|_\infty=\max_x|v(x)|\) measures the largest table entry. This contraction explains why repeated exact updates forget initialization. With incomplete evaluation, an unchanged greedy policy after one sweep does not by itself prove convergence: values can keep changing enough to alter later action rankings. The cart needs either an appropriate residual criterion or a justified iteration schedule.

The contraction also settles uniqueness and convergence for discounted problems. A \(\gamma\)-contraction on the complete space of bounded tables has exactly one fixed point, so for \(\gamma\lt1\) the value \(V^\pi\) is the only solution of the expectation equation, and iterative policy evaluation reaches it from any initial table. The optimality operator, which takes a maximum over actions instead of a policy average, is also a \(\gamma\)-contraction, because two maxima differ by at most the largest difference between their arguments; value iteration therefore converges to \(V^\star\) from any initial table.

Greedy improvement never makes a policy worse. Let \(\pi'\) be greedy with respect to \(V^\pi\). At each state the maximum over actions is at least the average under \(\pi\), so \(\mathcal T^{\pi'}V^\pi\geq\mathcal T^\pi V^\pi=V^\pi\) entrywise. The operator \(\mathcal T^{\pi'}\) is monotone, because a larger successor table cannot lower an average with nonnegative weights, so applying it repeatedly preserves the inequality, and its iterates converge to \(V^{\pi'}\). Hence \(V^{\pi'}\geq V^\pi\) at every state. If no value increases, then \(V^\pi=V^{\pi'}=\mathcal T^{\pi'}V^\pi\), which is the maximum over actions, so \(V^\pi\) satisfies the optimality equation and \(\pi\) is optimal. Each improvement thus either raises the value at some state or certifies optimality, and because a finite model has finitely many deterministic policies, policy iteration with exact evaluation reaches an optimal policy after finitely many improvements. The usual sketch of policy iteration draws two lines, values consistent with the policy, \(V=V^\pi\), and a policy greedy with respect to the values, \(\pi=\operatorname{greedy}(V)\). The iterates alternate between the lines, and the two conditions hold together only at \(V^\star\) and an optimal policy, because together they make \(V\) a fixed point of the optimality operator.

Monte Carlo evaluation replaces an expectation with completed experience 20:47

Hold the cart policy fixed and collect complete episodes. For each occurrence of state \(x\), compute the realized rewards remaining after that occurrence. Monte Carlo evaluation estimates \(V^\pi(x)\) by averaging these returns. No transition probabilities appear in the update, although the trajectories naturally reflect them. This is prediction under a specified policy, not yet a procedure for finding a better policy.

Suppose a cart trip visits loading state \(A\), corridor \(B\), then \(A\) again, and finally the terminal dock. Its three rewards are \(-1,-1,4\), with \(\gamma=0.9\). Working backward gives \(G_2=4\), \(G_1=-1+0.9(4)=2.6\), and \(G_0=-1+0.9(2.6)=1.34\). The first and second visits to \(A\) have different realized returns even though they share one expected value under the Markov model.

First-visit Monte Carlo uses only the first occurrence of each state in each episode. After this one episode its estimate for \(A\) is 1.34. Every-visit Monte Carlo uses both occurrences, giving \((1.34+4)/2=2.67\). Neither number is the true value merely because the bookkeeping is correct. The repeated returns share the same future rewards, so they are correlated, not independent extra experiments. With suitable stationary episodic assumptions, both schemes can be consistent as experience grows; first-visit analysis is simpler because independent episodes provide independent samples for a fixed starting condition.

A finite sample mean of integrable, unbiased returns is itself unbiased. Consistency is a different claim about the large-sample limit. Every-visit estimates have random visit counts and within-episode dependence, so one should not import every independent-sample formula without checking its assumptions. In particular, dividing a sample variance by the raw number of visits can overstate precision.

In implementation, each state keeps a visit counter \(N(x)\) and a return sum \(S(x)\), both initially zero. When an episode ends, each counted occurrence of \(x\) at a time \(t\), the first one in the episode for first-visit evaluation and every one for every-visit evaluation, performs \(N(x)\leftarrow N(x)+1\) and \(S(x)\leftarrow S(x)+G_t\), and the estimate is \(\widehat V(x)=S(x)/N(x)\). For first-visit evaluation the law of large numbers applies directly. By the Markov property, the return after the first visit to \(x\) has the distribution of a return started at \(x\), whatever happened before the visit, so independent episodes supply independent, identically distributed returns with mean \(V^\pi(x)\). The estimate converges to \(V^\pi(x)\) provided \(x\) is visited in infinitely many episodes, and if the returns have variance \(\sigma^2(x)\), its standard deviation after \(N(x)\) first visits is \(\sigma(x)/\sqrt{N(x)}\).

Every-visit evaluation also converges, but the argument must handle the dependence within an episode. Let \(N_e\) be the number of visits to \(x\) in episode \(e\) and \(S_e\) the sum of the returns that follow them. These pairs are independent and identically distributed across episodes, and after \(K\) episodes the estimate is \(\sum_eS_e/\sum_eN_e\), a ratio of two averages. The \(j\)th visit, whenever it occurs, is again followed by a return with mean \(V^\pi(x)\), so for integrable returns and visit counts \(\mathbb E[S_e]=V^\pi(x)\sum_{j\geq1}\Pr(N_e\geq j)=V^\pi(x)\,\mathbb E[N_e]\). If \(\mathbb E[N_e]\) is positive and finite, the law of large numbers applied to numerator and denominator makes the ratio converge to \(V^\pi(x)\). For finitely many episodes a ratio of averages is generally biased: Exercise 7 constructs a one-episode every-visit estimate with expectation \(1.5\) for a state whose value is \(2\). Sutton and Barto report convergence for both schemes and credit the every-visit analysis to Singh and Sutton (“Reinforcement learning with replacing eligibility traces,” Machine Learning, 1996).

Ordinary complete-return Monte Carlo waits for termination. A continuing task can use finite windows, but these change the target unless the missing tail is handled. If a window contains \(H\) rewards, its omitted discounted tail has magnitude at most \(\gamma^H R_{\max}/(1-\gamma)\). This bound makes the approximation explicit; describing a truncated sum as an exact complete return conceals a systematic error.

Blackjack illustrates both interpretable values and state sufficiency 30:44

The lecture's second example represents a blackjack position by player total from 12 through 21, dealer showing card from ace through ten, and a usable-ace flag. The resulting tabular representation has \(10\cdot10\cdot2=200\) states. A usable ace is one that can count as eleven without making the hand exceed twenty-one; merely possessing an ace is not enough. Actions are hit and stand, and the terminal reward distinguishes win, tie, and loss.

The rewards are specific. Standing ends the game with reward \(+1\), \(0\), or \(-1\) for a win, a tie, or a loss against the dealer's final total; a hit that takes the player's total above twenty-one ends the game with \(-1\); any other hit earns zero. In the standard version of the example, Example 5.1 of Sutton and Barto, the dealer follows a fixed rule, sticking on seventeen or more, and the return is undiscounted. Because every reward before the end of a game is zero, the return from any state equals the final reward, and the value of a state is \(\Pr(\text{win})-\Pr(\text{loss})\) under the evaluated policy, a number between \(-1\) and \(+1\).

Below a total of twelve the player hits automatically, and no decision is lost by this rule. A card worth two through ten raises a total of at most eleven to at most twenty-one, and an ace counts eleven only when that keeps the total at or below twenty-one and counts one otherwise, so the hit cannot bust the player. In the replacement model discussed below, the dealer's cards do not depend on the player's draw, and standing at the higher total afterward wins in every case in which standing at the lower total would. The decision states therefore begin at twelve. No state can recur within a game, because each card raises the total that counts every ace as one, and this total is determined by the state; first-visit and every-visit estimates therefore coincide here.

Evaluating the fixed policy “stand at twenty or twenty-one, otherwise hit” gives a value surface, not an optimal policy. High player totals are valuable because this policy stands with strong hands; a dealer ace generally worsens the player's prospects. More episodes reduce sampling noise where states are visited, but do not guarantee uniformly precise estimates across the table. Rare states can remain uncertain after an impressive total episode count.

The estimated surfaces after 10,000 and after 500,000 episodes show this unevenness. After 10,000 episodes the surface for states with a usable ace is visibly irregular while the surface without one is already nearly smooth; after 500,000 episodes both are smooth. States with a usable ace occur less often, so they receive fewer returns. Since every return lies between \(-1\) and \(+1\), its variance is at most one, and an estimate from \(N(x)\) independent returns has standard deviation at most \(1/\sqrt{N(x)}\). Fifty times as many episodes reduce this bound by the factor \(\sqrt{50}\approx7.07\) at a state whose share of the episodes stays the same. The jump in value at the totals twenty and twenty-one is equally specific: the evaluated policy stands only there, and at nineteen without a usable ace a hit busts with every card except an ace or a two, which in the replacement model has probability \(11/13\approx0.846\).

There is a modeling qualification hidden in this convenient representation. A finite deck drawn without replacement has a next-card distribution that depends on removed cards. The three listed variables do not encode that history. A replacement or effectively infinite-deck model makes the compact state more defensible; a finite-deck model needs a richer state or a partially observed treatment. The cart has the same problem if previous slips reveal an unobserved floor condition. Accurate value learning cannot repair an insufficient state description simply by collecting more of the same observations.

An incremental mean separates estimation from tracking 36:53

For returns \(g_1,\ldots,g_n\) observed at one state, let \(m_n\) be their arithmetic mean. Expanding its definition gives \(nm_n=(n-1)m_{n-1}+g_n\), hence

Only a count and current mean are needed; storing every historical return is unnecessary. The innovation \(g_n-m_{n-1}\) says how surprising the new observation is relative to the current estimate. The factor \(1/n\) makes each observed return have equal final weight.

Replacing \(1/n\) by constant step size \(0<\alpha\leq1\) produces \(m_n=(1-\alpha)m_{n-1}+\alpha g_n\). Expanding repeatedly shows that a return from \(j\) updates ago has weight \(\alpha(1-\alpha)^j\), while initialization retains weight \((1-\alpha)^n\). This exponential forgetting can track a changing floor, but it is no longer the exact sample mean. Persistent noise remains even after many updates. Smaller steps reduce fluctuations while delaying adaptation; that choice depends on the environment's change rate.

A policy change also changes the prediction target. Pooling years of returns from different cart policies does not estimate the current policy's value without further reasoning. The upcoming interaction between evaluation and improvement therefore changes the meaning of old data, even in a stationary physical world.

Temporal differences move a prediction toward an informed successor prediction 39:16

Temporal-difference learning replaces the unfinished return by immediate reward plus a current estimate of what follows. Let \(\widehat V\) be the learned value table and \(d_{t+1}\) indicate genuine termination. The one-step target \(Y_t\), temporal-difference error \(\delta_t\), and update are

The name TD(0) identifies the simplest one-step temporal-difference method. Bootstrapping means using a learned prediction inside the target for another prediction. Sutton's original temporal-difference paper develops learning from successive predictions and analyzes special cases. Its results motivate this mechanism; they are not a blanket convergence guarantee for every nonlinear learner.

Take the first transition of our cart episode, \(A\) to \(B\), with reward minus one. If \(\widehat V(A)=0\), \(\widehat V(B)=2\), and \(\alpha=0.25\), then \(Y=-1+0.9(2)=0.8\), so the new estimate at \(A\) is 0.2. It can be updated immediately. A Monte Carlo update using the completed return 1.34 would instead give 0.335 from the same initial estimate and step size. The difference is a target choice, not an arithmetic discrepancy.

Why does updating guesses from guesses learn anything? The reward is observed, and the Bellman operator constrains how predictions may agree across transitions. In a continuing one-state system with reward one, its fixed point solves \(v=1+\gamma v\), so \(v=1/(1-\gamma)\). For \(\gamma=0.9\), the value is ten at every decision time. Discounting anchors the fixed point through contraction; it does not turn the state value into zero far in the future.

Terminal handling supplies another anchor. A genuine terminal transition has target equal to its reward, regardless of the stored successor estimate. A time limit that merely interrupts a continuing task is different: zeroing its bootstrap estimates the interrupted task rather than the intended continuing one. The data interface must distinguish these cases.

Bias and variance refer to a specified target distribution 49:34

Freeze the current table so that we can analyze a single target. Define its error \(e(x)=\widehat V(x)-V^\pi(x)\). For a fixed current state, the Bellman identity gives the TD target's conditional bias as \(\gamma\mathbb E_\pi[e(x_{t+1})\mid x_t=x]\), with terminal errors set to zero. Thus an inaccurate successor table usually biases the target. Errors can also cancel in expectation; inaccurate estimates do not logically force nonzero bias in every state.

If the true successor value were available, the one-step target \(Y^\star=r_{t+1}+\gamma V^\pi(x_{t+1})\) would be a conditional expectation of the full return, given the first transition and reward. The Markov assumption and a fixed policy matter here. Applying the law of total variance gives

The final term is nonnegative. Consequently the true one-step target removes unresolved future randomness and has no greater variance than the return. This is a precise reason behind the usual variance comparison. Substituting a noisy or badly scaled learned table destroys the universal ordering: practical TD is often lower variance, not mathematically guaranteed to be lower variance for every estimate.

For a numerical example, suppose \(A\) deterministically enters \(B\) with reward zero, and \(B\)'s terminal reward is either zero or two with equal probability. With \(\gamma=0.9\), the return from \(A\) is zero or 1.8, with mean 0.9 and variance 0.81. True successor value \(V(B)=1\) makes the one-step target constantly 0.9, with zero variance. An estimate \(\widehat V(B)=1.5\) instead gives constant target 1.35: zero variance but bias 0.45. Reliability cannot be judged from variance alone.

Sampling width and bootstrap depth are independent choices 54:33

A backup is the computation that updates a value estimate from possible or observed futures. Exact dynamic programming averages over all modeled one-step outcomes and bootstraps. TD samples one outcome and bootstraps. Complete-return Monte Carlo samples a trajectory all the way to termination and does not bootstrap. These are different points on two axes: how broadly to average alternatives, and how far to look before using an estimated value.

Backup diagrams draw these computations as trees, with open nodes for states, filled nodes for actions, and a square for a terminal state. Writing the three updates with one table makes their relation exact. With the model, the dynamic-programming backup replaces \(\widehat V(x)\) by \(\mathbb E_\pi[Y_t\mid x_t=x]=(\mathcal T^\pi\widehat V)(x)\), the conditional expectation of the one-step temporal-difference target \(Y_t\). Temporal-difference learning moves \(\widehat V(x_t)\) toward one sample of that target, and Monte Carlo moves it toward one sample of \(G_t\), whose conditional expectation is \(V^\pi(x_t)\) itself. The dynamic-programming backup covers every branch of the first level of the tree; each sampled backup follows one branch, to depth one or to termination.

The two axes define four corners. Dynamic programming is wide and shallow, temporal-difference learning narrow and shallow, and Monte Carlo narrow and deep. The fourth corner, exhaustive search, is wide and deep: it expands every action and every successor until termination and uses no stored value. Like dynamic programming it needs the model, and its cost grows exponentially with depth. If each state offers \(m\) actions and each action leads to at most \(s\) successors, a tree that looks \(D\) steps ahead has up to \((ms)^D\) leaves; with \(m=4\), \(s=2\), and \(D=10\), that is \(8^{10}=1{,}073{,}741{,}824\) leaves for a single root state. One dynamic-programming sweep instead costs about \(|\mathcal X|ms\) successor evaluations for the whole table. Bootstrapping is what cuts the tree: a stored value summarizes the subtree below its node, so the backup can stop there.

For \(n\) steps before termination, an intermediate target is \(G_t^{(n)}=\sum_{k=0}^{n-1}\gamma^k r_{t+k+1}+\gamma^n\widehat V(x_{t+n})\). The positive integer \(n\) counts sampled transitions. Stop the sum at a terminal state and omit its bootstrap. Taking \(n=1\) recovers TD(0); taking the remaining episode length recovers the complete return. Longer lookahead discounts a fixed terminal-estimate error more strongly, but introduces more sampled rewards and delays availability. It need not monotonically improve total prediction error.

For the cart, a two-step target may reveal a slippery corridor before trusting a distant dock estimate. Its usefulness depends on where uncertainty lives. More lookahead is useful when the nearby value table is wrong and observations are informative; it can be costly when long trajectories add irrelevant noise or expensive physical interactions.

Policy improvement needs action values and deliberate exploration 1:00:09

Generalized policy iteration describes interacting evaluation and improvement processes without requiring evaluation to finish before each improvement. Monte Carlo evaluation of \(V\) removes the model from prediction, but greedy improvement through \(R+\gamma PV\) still needs that model. Estimating \(Q(x,u)\) directly removes this obstacle: compare the learned action values at the current state. The price is a larger table and the need to experience actions that the current favorite would never choose.

The lecture's two-door example makes this failure concrete. An unlucky zero from the left door and a positive return from the right can make a greedy learner choose right forever. No further evidence arrives about left. In our cart, an initially successful long route can similarly prevent discovering a short route. A low prediction error on the frequently selected action provides no information about the neglected action's quality.

Sample averages make the failure exact. The doors form a single-state problem, so \(Q\) is a table with two entries. Opening the left door once returns zero, so \(Q(x,\mathrm{left})=0\). The right door then returns \(1\), \(3\), and \(2\) on successive openings, and its running mean is \(1\), then \(2\), then \(2\). Once the right door's mean exceeds zero, a greedy learner opens only the right door, and with these returns its mean stays above zero, so the left door keeps its single sample whatever its true mean reward is. Estimating action values from samples therefore requires visiting every state-action pair, not merely every state as Monte Carlo estimation of \(V\) does. Blackjack has 200 states but 400 state-action pairs, and a deterministic policy, which takes one action per state, generates at most 200 of them.

For \(m=|\mathcal U(x)|\) actions, an epsilon-greedy policy selects a greedy action with probability \(1-\epsilon\), and with probability \(\epsilon\) samples uniformly from all \(m\) actions. With a unique greedy action, its total probability is \(1-\epsilon+\epsilon/m\); every other action has probability \(\epsilon/m\). Thus two actions and \(\epsilon=0.2\) give probabilities 0.9 and 0.1, not 0.8 and 0.2.

This guarantees action support conditional on reaching a state. It does not guarantee reaching every state, safe exploration, or rapid discovery of a long action sequence. A permanently closed corridor remains unreachable, and a rare sequence can require enormous experience. Persistent exploration also means the executed policy retains some deliberately nongreedy behavior. Greedy evaluation after training is a different policy and must be reported separately.

Monte Carlo evaluation of \(Q\) combined with \(\epsilon\)-greedy improvement is a first complete learning algorithm, and its improvement step still improves within a restricted class. Call a policy \(\epsilon\)-soft if \(\pi(u\mid x)\geq\epsilon/m\) for every action at every state, take \(0\lt\epsilon\lt1\), and let \(\pi'\) be \(\epsilon\)-greedy with respect to \(Q^\pi\) for an \(\epsilon\)-soft policy \(\pi\). Writing \(\pi(u\mid x)=\epsilon/m+(1-\epsilon)w_x(u)\) defines weights \(w_x(u)\) that are nonnegative, because \(\pi\) is \(\epsilon\)-soft, and that sum to one over \(u\). Therefore

The left side is \((\mathcal T^{\pi'}V^\pi)(x)\), so the monotonicity argument used for greedy improvement gives \(V^{\pi'}\geq V^\pi\) at every state, as Sutton and Barto prove in their Section 5.4.

The guarantee stops short of optimality. The improvement can stall only at a policy that is optimal among \(\epsilon\)-soft policies, and with \(\epsilon\) fixed the executed policy keeps probability \(\epsilon/m\) on every action, including actions that lose. Two standard modifications target an optimal policy instead: reduce \(\epsilon\) toward zero while still trying every action infinitely often, or start each episode from a randomly chosen state-action pair, the method of exploring starts, and act greedily afterward. For Monte Carlo control with exploring starts, Sutton and Barto note that a general convergence proof had not been found. The greedy policy read from the final table remains a different object from the executed \(\epsilon\)-greedy policy.

Applied to blackjack, Monte Carlo control yields an explicit rule. The published solution, Example 5.3 and Figure 5.2 of Sutton and Barto, was computed with exploring starts, beginning from the policy evaluated earlier. With a usable ace, the player sticks, that is, stands, at nineteen or more when the dealer shows an ace, a nine, or a ten, and at eighteen or more when the dealer shows two through eight. Without a usable ace, the player stands at seventeen or more against an ace or seven through ten, at thirteen or more against a two or a three, and at twelve or more against four through six. The usable-ace rule hits at higher totals because one hit cannot bust such a hand: if a card would take the total past twenty-one, the ace is recounted as one, lowering the total by ten, and no card adds more than ten. Without that protection the rule stands earlier, most of all against a dealer showing four through six. Sutton and Barto report that the rule agrees with Thorp's basic strategy of 1966 except for the leftmost notch in the usable-ace policy. The optimal value is again \(\Pr(\text{win})-\Pr(\text{loss})\), now under this rule. The rule is reported as optimal for the replacement model with this dealer, not for a finite shoe.

Tsitsiklis's stochastic-approximation analysis illustrates why convergence claims require explicit update, noise, and visitation conditions. It concerns specified asynchronous algorithms including Q-learning; it does not prove that any arbitrary mixture of Monte Carlo evaluation, policy changes, and exploration schedules converges. Here the skeleton establishes how to construct a learner, while the next chapter specializes its control targets.

Three processes organize the algorithm without solving representation 1:10:18

An RL experiment generates trajectories, uses them to estimate values or fit a model, then improves a policy. Model-free value methods use learned action values to choose actions. Policy optimization directly adjusts a parameterized policy toward higher estimated return. Model-based methods fit or use a transition model and plan through it. These categories can share components: a learned model can generate targets for a critic, and a critic can guide a policy update.

The categories can be written as objectives. Let \(\rho_0\) be the distribution of the initial state, and let \(p_\pi(\tau)=\rho_0(x_0)\prod_t\pi(u_t\mid x_t)P(x_{t+1}\mid x_t,u_t)\) be the probability that policy \(\pi\) generates the state-action trajectory \(\tau=(x_0,u_0,x_1,u_1,\ldots)\) of an episode, the product running over its decisions. Policy optimization adjusts a parameterized policy to maximize

the expected return of a whole episode, without using the transition law; the expected reward \(R(x_t,u_t)\) may replace the observed reward \(r_{t+1}\) inside the expectation because it is that reward's conditional mean. Value-based methods define the policy implicitly by \(\pi(x)\in\arg\max_uQ(x,u)\) and learn \(Q\), again without the transition law. Model-based methods use the transition law, either given or learned as an approximation \(\widehat P_\theta\approx P\) with parameters \(\theta\).

Each of the three processes has standard forms. Generating samples means running the current policy \(\pi(u_t\mid x_t)\) to collect trajectories. Fitting a model or estimating a return means choosing parameters \(\theta\) so that \(\widehat V_\theta(x)\approx V^\pi(x)\), \(\widehat Q_\theta(x,u)\approx Q^\pi(x,u)\), or \(\widehat P_\theta(\cdot\mid x,u)\approx P(\cdot\mid x,u)\); the tables of this chapter are the special case with one parameter per entry. Improving the policy means either setting \(\pi(x)\in\arg\max_u\widehat Q_\theta(x,u)\), as Q-learning and the deep Q-network (DQN) do, or taking a gradient step \(\theta\leftarrow\theta+\alpha\nabla_\theta J(\pi_\theta)\) on the parameters of a policy \(\pi_\theta\), as policy-gradient methods such as A2C and A3C do. The next chapter develops the first route and the chapter after it the second. This three-process organization is adapted from UC Berkeley's deep reinforcement learning course, CS 185/285.

The organizing questions are operational. Which policy produced each trajectory? What quantity is the target estimating? Which parts of that target are observed and which are predictions? What state-action region supports the proposed improvement? Recording these answers for the cart prevents confusing a better fit to old returns with better control under a newly changed policy.

Nothing in the sampling argument makes a table small. A continuous position-velocity state still needs discretization or function approximation. Once parameters are shared across states, one update can change predictions at unvisited states and the tabular reasoning must be revisited. That is the next obstacle, separate from replacing a known transition model by experience.

Executable checks isolate the target before testing a learner

The following standard-library program accepts the finite reward sequence used above and checks return recursion, incremental averaging, a TD update, target variance, and exploration probabilities. It writes no files, uses no network or random sampling, and prints the verified values. Deterministic enumeration makes disagreement an implementation error rather than a statistical accident. Assertions fail immediately if a target, terminal convention, or probability calculation changes incorrectly.

from math import isclose

def returns(rewards, gamma):
    if not 0 <= gamma <= 1:
        raise ValueError("discount outside [0,1]")
    out = [0.0] * len(rewards)
    value = 0.0
    for i in range(len(rewards)-1, -1, -1):
        value = rewards[i] + gamma * value
        out[i] = value
    return out

g = returns([-1.0, -1.0, 4.0], 0.9)
assert all(isclose(a, b) for a, b in zip(g, [1.34, 2.6, 4.0]))
mean = 0.0
for n, sample in enumerate([g[0], g[2]], 1):
    mean += (sample - mean) / n
assert isclose(mean, 2.67)

def td(value, reward, next_value, terminal, gamma=0.9, alpha=0.25):
    target = reward + gamma * (0.0 if terminal else next_value)
    return value + alpha * (target - value)

assert isclose(td(0, -1, 2, False), 0.2)
assert isclose(td(0, 4, 1000, True), 1.0)
outcomes = [0.0, 1.8]
expected = sum(outcomes) / 2
variance = sum((x-expected)**2 for x in outcomes) / 2
assert isclose(expected, 0.9) and isclose(variance, 0.81)
assert isclose(0.9 * 1.5 - expected, 0.45)

epsilon, actions = 0.2, 2
prob = [1-epsilon+epsilon/actions, epsilon/actions]
assert isclose(sum(prob), 1.0)
assert isclose(prob[0], 0.9) and isclose(prob[1], 0.1)
print("returns", g, "every-visit mean", mean, "variance", variance)

This is a check of mechanisms, not an empirical demonstration of sample efficiency or convergence. A training experiment should next compare both methods against an exact small-MDP oracle, preserve a fixed evaluation policy while testing prediction, and report error over independent runs. Only after those checks should improvement and changing data distributions be introduced.

A second program checks the gridworld numbers quoted earlier. It numbers the sixteen cells 0 through 15 in row order, with terminal corners 0 and 15, and runs sixty synchronous sweeps of the random policy's evaluation in exact rational arithmetic, so that ties between actions are exact rather than subject to rounding. It asserts the three-sweep values, the four cells with a unique greedy action after one sweep, the two corners whose greedy sets still contain suboptimal moves after two sweeps, and the optimality of every greedy action from the third through the sixtieth sweep. It then solves the linear evaluation equations with NumPy for the converged values, checks that greedy improvement with respect to them yields the same greedy sets as the sixtieth sweep, and follows one greedy action per cell to confirm that the improved policy attains the optimal value, minus the distance to the nearest terminal corner. It writes no files and prints the ten-sweep values and the largest remaining gap between the sixtieth sweep and the limit. That gap, about \(0.85\), shows that the values are still far from converged after sixty sweeps even though every greedy action has been optimal since the third.

from fractions import Fraction
import numpy as np

TERMINAL = (0, 15)
STEPS = [(-1, 0), (1, 0), (0, -1), (0, 1)]

def move(s, a):
    r, c = divmod(s, 4)
    r2, c2 = r + STEPS[a][0], c + STEPS[a][1]
    return 4*r2 + c2 if 0 <= r2 < 4 and 0 <= c2 < 4 else s

def sweep(v):
    return [v[s] if s in TERMINAL else
            sum(-1 + v[move(s, a)] for a in range(4)) / 4 for s in range(16)]

def greedy(v):
    sets = {}
    for s in range(16):
        if s not in TERMINAL:
            q = [-1 + v[move(s, a)] for a in range(4)]
            sets[s] = {a for a in range(4) if q[a] == max(q)}
    return sets

v_star = [-min(r + c, 6 - r - c) for r in range(4) for c in range(4)]
optimal = greedy(v_star)
v, tables = [Fraction(0)] * 16, []
for k in range(60):
    v = sweep(v)
    tables.append(v)
assert tables[2][1:4] == [Fraction(-39, 16), Fraction(-47, 16), -3]
assert tables[2][5] == Fraction(-23, 8)
g1, g2 = greedy(tables[0]), greedy(tables[1])
assert [s for s in g1 if len(g1[s]) == 1] == [1, 4, 11, 14]
assert [s for s in g2 if not g2[s] <= optimal[s]] == [3, 12]
assert all(g[s] <= optimal[s] for g in map(greedy, tables[2:]) for s in g)
p = np.eye(16)
for s in range(16):
    if s not in TERMINAL:
        for a in range(4):
            p[s, move(s, a)] -= 0.25
b = np.array([0.0 if s in TERMINAL else -1.0 for s in range(16)])
v_random = np.linalg.solve(p, b)
assert np.allclose(v_random.reshape(4, 4), [[0, -14, -20, -22],
                                            [-14, -18, -20, -20],
                                            [-20, -20, -18, -14],
                                            [-22, -20, -14, 0]])
limit = [int(round(x)) for x in v_random]
assert greedy(limit) == greedy(tables[-1])
policy = {s: min(acts) for s, acts in greedy(limit).items()}
for start in policy:
    s, n = start, 0
    while s not in TERMINAL and n < 16:
        s, n = move(s, policy[s]), n + 1
    assert -n == v_star[start]
gap = max(abs(float(x) - y) for x, y in zip(tables[-1], v_random))
print([round(float(x), 2) for x in tables[9]], round(gap, 2))

Eight exercises with fully worked solutions

What the evidence establishes

Complete returns provide observations of a policy's long-term consequences. Temporal differences use observed rewards and learned successor values to update sooner. Generalized policy iteration turns those predictions into control only when action coverage and improvement are handled explicitly. These conclusions apply to the specified finite-state models; they do not certify safety during exploration or convergence with arbitrary function approximation.

The cart values here are hypothetical, the gridworld values are computed exactly for that example, and all are independently checked. The machine transcript was read in full, and every page of the official slides was rendered and read as an image during the slide cross-check of 6 October 2026; no complete audiovisual review of the recording and no reproduction of the lecture's blackjack plots is claimed. Primary references support the stated mechanism and theorem context, while the derivations and cart examples supply original explanation rather than a benchmark result.