Value-based reinforcement learning: on-policy risk, off-policy control, and DQN

Value-based control separates how an agent behaves from what it learns. SARSA evaluates the exploratory policy that actually takes the next action. Q-learning uses exploratory data to learn a greedy target. That single target change explains different risk preferences on the cliff—and becomes a source of instability once a neural network supplies both predictions and targets.

Our running system is a cliff grid. The cart-like agent starts at the lower left, seeks the lower-right goal, pays \(-1\) per step, and pays \(-100\) when it enters the cliff along the direct route. Actions move on the grid. During training an \(\epsilon\)-greedy behavior chooses a greedy action with probability \(1-\epsilon\) and otherwise explores. Let \(Q(x,u)\) denote expected discounted return after state \(x\), action \(u\), then a specified continuation policy.

Based on AA203 lecture 17 and official slides: SARSA, behavior and target policies, Q-learning, function approximation, fitted Q iteration, experience replay, and fixed target networks. The cliff target calculation, stability audit, and exercises are original teaching constructions. Tabular convergence claims are stated with their conditions and are not transferred to arbitrary neural critics. The official slides add the TD-over-Monte-Carlo motivation, the SARSA and Q-learning algorithm structures, the windy gridworld, the on-policy and off-policy definitions with the alternative-successor target, the cliff grid's dimensions and learning-curve levels, the state counts of backgammon and Go, the approximator interfaces, the gradient derivation and regression-dataset view of value approximation, and the DQN architecture, failure modes, and loss; where their notation differs, the text translates it.

Every result below keeps training behavior, learned target, and evaluation behavior distinct.

The chapter continues directly from the previous chapter, which develops the material assumed here: the distinction between prediction, which estimates the value of a given policy, and control, which seeks an optimal policy; the exact methods, policy iteration and value iteration, and their need for the transition model; the dynamic-programming, Monte Carlo, and temporal-difference backups, which differ in whether they sample outcomes and whether they bootstrap from a stored estimate; and generalized policy iteration with action values and \(\epsilon\)-greedy improvement. Within the course's learning branch, this chapter takes up the first model-free family, value-based methods, in which the policy \(\pi\) is not stored separately but read from a learned action-value function, \(\pi(x)\in\arg\max_uQ(x,u)\). The next chapter treats the second family, policy optimization, which adjusts a parameterized policy directly, and model-based methods follow it. The previous chapter's three-process skeleton, which generates samples, estimates a return or fits a model, and improves the policy, organizes both families.

Temporal-difference control learns before an episode ends 8:17

Generalized policy iteration alternates approximate evaluation and improvement. Monte Carlo control waits for complete return \(G_t\). TD control uses an immediate bootstrap target and updates each step. This supports continuing or very long tasks and often lowers sampling variance, while inheriting bias from its current estimate.

Here \(r_t\) denotes the reward produced by action \(u_t\) in state \(x_t\). For an episode that ends at decision time \(T\), the return is \(G_t=r_t+\gamma r_{t+1}+\cdots+\gamma^{T-t-1}r_{T-1}=\sum_{k=0}^{T-t-1}\gamma^kr_{t+k}\). The state value \(V^\pi(x)=\mathbb E_\pi[G_t\mid x_t=x]\) and the action value \(Q^\pi(x,u)=\mathbb E_\pi[G_t\mid x_t=x,u_t=u]\) are its expectations when policy \(\pi\) chooses the actions. The previous chapter writes the same reward as \(r_{t+1}\); only the label differs.

Temporal differences have three advantages over complete returns, and each changes what a control method can do. First, the target has lower variance. The return \(G_t\) accumulates the randomness of every remaining action, transition, and reward, whereas the one-step target \(r_t+\gamma\widehat V(x_{t+1})\), with \(\widehat V\) a stored estimate of \(V^\pi\), contains the randomness of a single step and replaces the rest by a stored number. In a chain with deterministic transitions whose rewards are independent with variance \(\sigma^2\), with \(\gamma=1\) and \(k\) steps remaining, \(\operatorname{Var}G_t=k\sigma^2\), while the one-step target has variance \(\sigma^2\), because the successor and its stored value are fixed; for \(k=20\) the ratio is twenty. The price is the bias of the stored value, which the previous chapter's bias and variance analysis quantifies. Second, the update is online: it is available after one transition, so the estimate changes during the episode that produced the data. Third, it applies to incomplete sequences, including continuing tasks and episodes that are truncated or have not yet ended.

Control follows by placing temporal-difference evaluation inside generalized policy iteration. Monte Carlo control evaluates action values from complete returns and improves the policy \(\epsilon\)-greedily. It evaluates \(Q\) rather than \(V\) because greedy improvement from \(V\) would need the transition model to compare actions, whereas \(\arg\max_uQ(x,u)\) compares stored numbers. Temporal-difference control keeps the action values and the \(\epsilon\)-greedy improvement and replaces only the evaluation: at every time step it moves \(Q(x_t,u_t)\) one step toward a one-step target, and the \(\epsilon\)-greedy policy derived from the changed table chooses the next action. Evaluation is never completed before improvement. Each update is a small evaluation step, and each action choice applies the current improvement.

For policy \(\pi\), the Bellman expectation target is \(r_t+\gamma\mathbb E_{u'\sim\pi(\cdot|x_{t+1})}Q^\pi(x_{t+1},u')\). A sampled successor action produces SARSA:

The acronym names the five observed items: state, action, reward, next state, next action. Terminal mask \(1-d\) prevents value leaking beyond failure or goal. Under standard tabular stochastic-approximation conditions, sufficient exploration, and suitable decaying steps, SARSA can converge; a fixed exploration rate instead targets behavior that keeps making random moves.

The official course slides use several symbols that differ from this chapter's. Following much of the reinforcement-learning literature, they write the quintuple as \((s_t,a_t,r_t,s_{t+1},a_{t+1})\), which spells the algorithm's name, with \(s\) for the state \(x\) and \(a\) for the action \(u\). They write \(V_\pi\) and \(Q_\pi\), or \(q_\pi\), for \(V^\pi\) and \(Q^\pi\), and \(q_\ast\) for \(Q^\star\); \(\mathbf w\) and \(\hat q(\cdot,\cdot,\mathbf w)\) for the critic parameters \(\theta\) and the critic \(\widehat Q_\theta\) introduced later; \(\phi\) for the target-network parameters \(\bar\theta\); and \(T(x_{t+1}\mid x_t,u_t)\) for the transition probabilities, whereas this chapter reserves \(T\) for the terminal decision time. Most of the slides write \(r_t\) for the reward produced by \(u_t\), as this chapter does; their review pages and their derivation of the Q-learning target write the same reward as \(R_t\) or \(r_{t+1}\).

Written as an algorithm, SARSA initializes \(Q(x,u)\) arbitrarily for every nonterminal state and action, and sets \(Q(x,\cdot)=0\) at terminal states, which is the table form of the mask \(1-d\). Each episode starts from an initial state and chooses its first action \(\epsilon\)-greedily from \(Q\). At each step it applies \(u_t\), observes \(r_t\) and \(x_{t+1}\), chooses \(u_{t+1}\) \(\epsilon\)-greedily from the current table, applies the update with that \(u_{t+1}\), and carries both forward, \(x_t\leftarrow x_{t+1}\) and \(u_t\leftarrow u_{t+1}\), until \(x_t\) is terminal. The order carries the method. The successor action is selected before the update and is then executed, so the target values the action the agent actually takes next, including an exploratory one. With \(\epsilon\) fixed, the policy whose value SARSA estimates and the policy that acts are the same \(\epsilon\)-greedy policy at every step.

A windy gridworld defeats the obvious route and rewards online updates 16:47

A gridworld with wind tests SARSA on a task in which the obvious route fails. The grid has 7 rows and 10 columns; rows are indexed from 0 at the top and columns from 0 at the left. The start is row 3, column 0, and the goal is row 3, column 7. Four actions move one cell up, down, left, or right. Each column carries a wind strength, 0, 0, 0, 1, 1, 1, 2, 2, 1, 0 from left to right. After each move the agent is pushed upward by the strength of the column it occupied when it moved, and any displacement past an edge stops at that edge. Every step pays \(-1\) until the goal is reached, so maximizing return means reaching the goal in the fewest steps. The learning run uses \(\epsilon=0.1\), \(\alpha=0.5\), \(\gamma=1\), and a table initialized to zero, one admissible choice of the arbitrary initialization.

Moving right along row 3 fails. The first three moves stay in row 3. Each move from columns 3, 4, and 5 also lifts the agent one row, so it reaches row 0 in column 6, where the wind of columns 6, 7, and 8 can lift it no further. After nine moves the agent is pinned at row 0, column 9, having passed three rows above the goal, and further right moves leave it there. The shortest route uses the ceiling deliberately. It follows the same nine moves to row 0, column 9, descends four cells in the last column, where the wind is zero, to row 4, and moves left twice. The first left move, made from column 9, reaches row 4, column 8; the second, made from column 8, whose wind is one, lands on the goal. The route takes 15 moves, and no route takes fewer. Because the dynamics are deterministic, a breadth-first search over cells computes the fewest moves from the start to every cell, and the code below verifies both claims.

from collections import deque
# Windy grid: 7 rows, 10 columns, row 0 on top, start (3, 0), goal (3, 7).
# After each move the wind of the column being left pushes the agent up.
wind = (0, 0, 0, 1, 1, 1, 2, 2, 1, 0)
start, goal = (3, 0), (3, 7)
moves = ((-1, 0), (1, 0), (0, -1), (0, 1))  # up, down, left, right


def step(cell, a):
    r, c = cell
    r = min(max(r + moves[a][0] - wind[c], 0), 6)
    return r, min(max(c + moves[a][1], 0), 9)


fewest = {start: 0}  # breadth-first search over the deterministic grid
queue = deque([start])
while queue:
    cell = queue.popleft()
    for a in range(4):
        nxt = step(cell, a)
        if nxt not in fewest:
            fewest[nxt] = fewest[cell] + 1
            queue.append(nxt)
cell, straight = start, [start]
for _ in range(12):  # always move right
    cell = step(cell, 3)
    straight.append(cell)
assert fewest[goal] == 15
assert goal not in straight and straight[-1] == (0, 9)
print(fewest[goal], straight[:10])

The code's inputs are the grid size, the wind strengths, and the start and goal cells; its outputs are the fewest moves to the goal, 15, and the straight-right trajectory, which ends at row 0, column 9 without visiting the goal. It assumes, as the description above states, that the wind of the column being left applies. Applying the wind of the column being entered instead would change the dynamics and the minimum, so an implementation must state its convention.

Learning progress for this task can be plotted as the number of completed episodes against the number of time steps. With the zero-initialized table the first episode is long. Every value starts at zero and every step lowers the visited entry, so the greedy choice keeps shifting to actions not yet tried and the agent wanders until it reaches the goal. Once the goal has been reached, the backups carry its consequence backward along the visited route, later episodes are shorter, and the curve bends upward; for these settings it reaches about 170 completed episodes after 8000 steps. The local slope of the curve is the reciprocal of the current episode length. Two qualifications apply. With exploration fixed at \(\epsilon=0.1\), episodes keep including random moves, so the slope settles at the reciprocal of the \(\epsilon\)-greedy episode length, not of the 15-step minimum. With the step size fixed at \(\alpha=0.5\), the table moves after every update, so a greedy route read from it at a given moment is usually close to the 15-step route but need not equal it; the convergence conditions in the Q-learning section and the fixed-point analysis require decreasing step sizes.

Monte Carlo control would not apply easily here, and the reason is termination. A Monte Carlo update needs a complete return, so it waits until the episode ends, and in this grid whether an episode ends depends on the policy. A deterministic policy can trap the agent: a policy that moves left at the start keeps it against the left wall forever, and a policy can also cycle through several cells without reaching the goal. Under such a policy the episode never ends, no return is computed, and the policy that caused the loop is never revised. With \(\epsilon\gt0\), every action has probability at least \(\epsilon/4\) and the goal is reachable from every cell, so episodes end with probability one; but an episode can still last arbitrarily long, and Monte Carlo learns nothing until it does. Temporal-difference control updates at every step instead. While the agent circles, every pass lowers the values of the state-action pairs on the loop, because each target adds a reward of \(-1\) to values that are themselves falling, while the values of untried alternatives stay where they were. The looping actions soon look worse than an alternative, and the \(\epsilon\)-greedy policy leaves the loop within the same episode. Online updating is therefore not only a convenience for long tasks; it is what allows learning when the current policy itself prevents the episode from ending.

Off-policy learning separates behavior from the target 26:24

Whether a method is on policy or off policy depends on which policy its updates concern. An on-policy method evaluates or improves the policy that makes the decisions. An off-policy method evaluates or improves a policy different from the one that generated the data. SARSA is on policy. Its policy changes as the table changes, but at every moment the \(\epsilon\)-greedy policy that chooses actions in the environment is the policy whose value the target estimates, because the successor action in the target is the action that policy chose and will execute.

The behavior policy \(\mu\) generates data. The target policy \(\pi\) is evaluated or improved. This separation lets an agent learn from older policies, demonstrations, or an exploratory controller while reasoning about another controller. It also creates a coverage requirement: if \(\mu\) never visits state-action pairs important to \(\pi\), data contain no evidence for their value.

Off-policy evaluation generally needs importance ratios, a model, or another correction. Q-learning is special because its optimality target does not average under the behavior policy. It uses the greedy continuation according to the current table. Behavior still determines which entries receive updates.

Calling an algorithm off policy describes its update target relative to its data source. It does not mean arbitrary logged data are safe to reuse with function approximation. Distribution shift and extrapolation can make the learned max select unsupported actions.

An off-policy target for action values follows from SARSA by changing one element of the quintuple. The behavior policy \(\mu\) still produces \((x_t,u_t,r_t,x_{t+1})\). In place of the next action that \(\mu\) will take, draw an alternative successor action \(u'_{t+1}\sim\pi(\cdot\mid x_{t+1})\) from the target policy, and move \(Q(x_t,u_t)\) toward the value of that alternative:

This one-step target needs no importance ratio. The update concerns the pair \((x_t,u_t)\), so the behavior's choice of \(u_t\) is conditioned on rather than averaged over. Given that pair, \(r_t\) and \(x_{t+1}\) come from the environment whatever policy chose \(u_t\), and \(u'_{t+1}\) comes from \(\pi\). The target is therefore a sample of the right-hand side of the Bellman expectation equation for \(Q^\pi\) at \((x_t,u_t)\). The behavior policy decides only which pairs are updated and how often, which is the coverage requirement stated above. The alternative action is never executed; the next executed action is still drawn from \(\mu\).

Q-learning bootstraps toward the greedy successor value 30:58

Q-learning chooses target policy \(\pi(x)\in\arg\max_uQ(x,u)\) and often collects with \(\epsilon\)-greedy behavior:

The update is the alternative-successor update of the previous section with the greedy target policy substituted. When \(\pi\) is greedy with respect to the current table, the alternative successor is an action \(u'_{t+1}\in\arg\max_{u'}Q(x_{t+1},u')\), so the bootstrap term is the table evaluated at a maximizing action, \(Q(x_{t+1},u'_{t+1})=\max_{u'}Q(x_{t+1},u')\). Ties among maximizing actions do not affect the target, because every maximizer gives the same value. The behavior is \(\epsilon\)-greedy with respect to the same table, so both policies change whenever \(Q\) changes, but only the target policy enters the target.

Its TD target imagines greedy continuation even if the actual next behavior action explores. In a finite tabular MDP, bounded rewards, repeated visits, Robbins–Monro step sizes, and appropriate stationarity yield convergence to \(Q^\star\). Constant step size, partial observability, nonstationarity, and neural approximation fall outside that theorem.

The convergence statement \(Q\to Q^\star\) holds only under hypotheses. In the tabular case they are finitely many states and actions, bounded rewards, every state-action pair updated infinitely often, and, for each pair, step sizes \(\alpha_k(x,u)\) at its \(k\)th update satisfying \(\sum_k\alpha_k(x,u)=\infty\) and \(\sum_k\alpha_k(x,u)^2\lt\infty\); these are the Robbins–Monro conditions. The choice \(\alpha_k=1/k\) satisfies both, whereas a constant step satisfies the first and violates the second. Convergence then holds with probability one. The behavior policy enters only through the visiting requirement. With \(m\) actions at a state, an \(\epsilon\)-greedy behavior gives every action probability at least \(\epsilon/m\), so with \(\epsilon\gt0\) it meets that requirement in a finite episodic task whose states are all reachable from the start. Both grid examples in this chapter use \(\gamma=1\), where the sup-norm contraction of the fixed-point analysis does not apply. For such episodic tasks the analysis rests on termination instead: the goal must be reachable, and policies that never reach it must accumulate unbounded cost, which the reward of \(-1\) per step ensures in both grids. Both examples also use the constant step \(\alpha=0.5\), so their learning curves are empirical evidence, not instances of the theorem.

The algorithm differs from SARSA in where the action is chosen. Q-learning initializes \(Q\) as SARSA does, with \(Q(x,\cdot)=0\) at terminal states, and starts each episode from an initial state. At each step it chooses \(u_t\) \(\epsilon\)-greedily from the current table, applies it, observes \(r_t\) and \(x_{t+1}\), and updates \(Q(x_t,u_t)\) with the maximum over successor actions, repeating until \(x_t\) is terminal. Nothing except the state is carried from one step to the next. The update needs only \((x_t,u_t,r_t,x_{t+1})\), and the next executed action is chosen afterward from the updated table. A stored Q-learning transition therefore remains a valid input to the update after the policy has changed, which the replay buffer of the DQN section exploits; a SARSA transition also records an action chosen by the policy of that moment, which a later policy may no longer choose.

Action ties matter in implementations. A fixed first-index argmax can create directional bias. Random tie-breaking changes behavior while preserving greedy value. Log the rule and seed.

The cliff exposes the difference between learned and executed risk 37:44

The comparison uses a grid of 4 rows and 12 columns. With rows indexed from 0 at the top, the start is the left end of row 3 and the goal its right end; the ten cells between them form the cliff. Each move pays \(-1\). A move into the cliff pays \(-100\) and returns the agent to the start without ending the episode. Both learners use \(m=4\) actions, \(\epsilon=0.1\), \(\alpha=0.5\), and \(\gamma=1\). Three routes matter. The cliff-edge route climbs one row, runs along row 2, which borders the cliff, and descends to the goal in 13 moves. The middle route runs along row 1 in 15 moves, and the top route runs along row 0 in 17 moves.

With \(\epsilon>0\), the shortest path adjacent to the cliff occasionally explores into catastrophe. SARSA includes the sampled exploratory successor action and therefore values the \(\epsilon\)-greedy policy's risk. It tends toward a longer path with margin. Q-learning backs up the greedy successor and learns the shortest optimal path, even though its training behavior may fall more often.

# At a risky next state, greedy continuation is worth 0; exploration can fall for -100.
epsilon = 0.1
q_next = (0.0, -100.0)
q_learning_continuation = max(q_next)
sarsa_expected_continuation = (1-epsilon/2)*q_next[0] + (epsilon/2)*q_next[1]
assert q_learning_continuation == 0.0
assert sarsa_expected_continuation == -5.0
assert sarsa_expected_continuation < q_learning_continuation
print(q_learning_continuation, sarsa_expected_continuation)

Here two actions are available and uniform exploration selects the bad one with probability \(\epsilon/2\). The calculation is an expected SARSA analogue; ordinary SARSA samples either target. It isolates why the on-policy target can prefer clearance.

Comparing training returns can favor SARSA because Q-learning continues exploratory falls while learning a greedy route. Comparing final greedy policies can favor the shorter Q-learning route after exploration is disabled. Evaluation must state whether exploration remains active.

The phrase “optimal \(\epsilon\)-greedy policy” has an exact meaning, and it identifies what SARSA learns with \(\epsilon\) fixed. If SARSA's table converges to some \(Q\), the expected update at every pair is zero. The successor action is greedy with probability \(1-\epsilon\) and uniform over the \(m\) actions with probability \(\epsilon\), so \(Q\) satisfies

This is the Bellman optimality equation of a modified problem in which every intended action is replaced, with probability \(\epsilon\), by a uniformly random one. Its solution \(Q^\star_\epsilon\) is the action value of the best controller among those executed \(\epsilon\)-greedily, and the greedy policy of \(Q^\star_\epsilon\) is that controller's intended policy. Q-learning's limit \(Q^\star\) solves the unmodified equation, the case \(\epsilon=0\). The two limits differ because the cliff makes random moves expensive, not because either learner misestimates its own target. If \(\epsilon\) decays under the conditions stated in the fixed-point analysis, SARSA's limit moves to \(Q^\star\) as well.

The code below computes the expected return from the start when each route is executed \(\epsilon\)-greedily, by solving the linear policy-evaluation equations, and computes \(Q^\star_\epsilon\) by value iteration on the modified problem. A route policy climbs to its row, runs right along it, and descends in the last column; from a cell off its row it first returns to the row. The function evaluate also accepts \(\epsilon=0\), which recovers the move counts, and a flag under which a fall ends the episode instead of returning the agent to the start. The assertions check the move counts, check that the policy read from \(Q^\star_\epsilon\) attains the value-iteration result when executed \(\epsilon\)-greedily, and check the ordering of the four expected returns.

import numpy as np
# Cliff grid: 4 rows, 12 columns, row 0 on top, start (3, 0), goal (3, 11).
# A step pays -1. Entering a cliff cell pays -100 and returns the agent to
# the start; with reset=False the fall ends the episode instead.
eps, start, goal = 0.1, (3, 0), (3, 11)
moves = ((-1, 0), (1, 0), (0, -1), (0, 1))  # up, down, left, right
states = [(r, c) for r in range(4) for c in range(12)
          if r != 3 or c in (0, 11)]
n, g = len(states), states.index(goal)


def model(reset):
    nxt, rew = np.zeros((n, 4), int), np.full((n, 4), -1.0)
    for i, (r, c) in enumerate(states):
        for a, (dr, dc) in enumerate(moves):
            cell = (min(max(r + dr, 0), 3), min(max(c + dc, 0), 11))
            if cell not in states:
                cell = start if reset else goal
                rew[i, a] = -100.0
            nxt[i, a] = states.index(cell)
    nxt[g], rew[g] = g, 0.0  # the goal is absorbing and pays nothing
    return nxt, rew


def evaluate(policy, e=eps, reset=True):
    """Expected return from each state when policy runs e-greedily."""
    nxt, rew = model(reset)
    p = np.full((n, 4), e/4)
    p[np.arange(n), policy] += 1 - e
    trans = np.zeros((n, n))
    np.add.at(trans, (np.arange(n)[:, None], nxt), p)
    trans[:, g] = 0.0
    return np.linalg.solve(np.eye(n) - trans, (p*rew).sum(axis=1))


def route(k):
    """Climb to row k, run right along it, descend in the last column."""
    return np.array([1 if c == 11 or r < k else 0 if r > k else 3
                     for r, c in states])


nxt, rew = model(True)
v = np.zeros(n)
for sweep in range(20000):
    q = rew + v[nxt]
    q[g] = 0.0
    v_new = (1 - eps)*q.max(axis=1) + eps*q.mean(axis=1)
    if np.max(np.abs(v_new - v)) < 1e-12:
        break
    v = v_new
best, s0 = q.argmax(axis=1), states.index(start)
edge, middle, top = (evaluate(route(k))[s0] for k in (2, 1, 0))
lengths = [evaluate(route(k), 0.0)[s0] for k in (2, 1, 0)]
edge_if_fall_ends = evaluate(route(2), reset=False)[s0]
assert np.allclose(lengths, [-13, -15, -17])
assert np.allclose(evaluate(best), v_new)
assert v_new[s0] > middle > top > edge
print(np.round([edge, middle, top, v_new[s0], edge_if_fall_ends], 2))

Executed \(\epsilon\)-greedily, the cliff-edge route earns \(-51.81\) in expectation, the middle route \(-21.05\), and the top route \(-22.27\); the best \(\epsilon\)-greedy controller earns \(-20.71\). Its greedy route is the middle row, and it improves on the plain middle route by choosing better actions in cells off the row. Learning curves of the return per episode for these settings level off near \(-25\) for SARSA and near \(-50\) for Q-learning, in rough agreement with these values. Q-learning's greedy route is the cliff-edge route, and executing that route with exploration costs about 31 more per episode than the best \(\epsilon\)-greedy controller, mostly through falls. SARSA's level lies a few units below \(-20.71\), consistent with a table that a constant step keeps moving. The same calculation separates the two safe routes. The top route is safer than the middle one, but at \(\epsilon=0.1\) its two extra moves cost more than the falls they avoid, so it is not the optimal \(\epsilon\)-greedy policy; a single SARSA run with \(\alpha=0.5\) can nevertheless settle on it, because a constant step leaves the estimates noisy.

The fall convention changes the numbers. If a fall ended the episode instead of returning the agent to the start, the \(\epsilon\)-greedy cliff-edge route would earn \(-38.34\), because a fall would no longer be followed by another attempt at the route. A Q-learning level near \(-50\) therefore corresponds to the convention in which a fall returns the agent to the start. Under that convention a cliff transition is not terminal: its target is \(-100\) plus \(\gamma\) times a value of the start state, not \(-100\). Exercise 4 treats the other convention, in which the terminal flag removes the bootstrap.

Function approximation trades storage for coupled generalization 46:14

Both grids have fewer than a hundred cells, so a table with one entry per state-action pair is small. Many problems that motivate reinforcement learning are not. Backgammon has about \(10^{20}\) states and Go about \(10^{170}\). Storing one eight-byte value per backgammon state would take \(8\times10^{20}\) bytes, about 800 exabytes, before multiplying by the number of actions, and no memory can hold \(10^{170}\) entries. A robot whose state is a vector of positions and velocities has a continuum of states, so no finite table lists them. Tables fail in such problems for two reasons. The first is memory. The second is learning: a table learns each entry only from visits to that entry, so it must visit every relevant state-action pair, usually many times, even when neighboring entries already indicate the answer. In the windy grid, a table's value for the cell beside the goal carries no information about the cell beside that one. This is the learning side of the curse of dimensionality: the number of entries, and with it the experience needed to visit them, grows exponentially with the number of state coordinates.

A table needs one value per state-action pair and learns each locally. A parametric critic \(\widehat Q_\theta(x,u)\) shares parameters, reducing storage and generalizing across related inputs. The same coupling means one regression step changes unobserved predictions.

For discrete actions, a network commonly maps state to all action values. For continuous actions it consumes state and action and returns a scalar; maximizing then requires an actor or numerical search. This lecture's DQN path assumes enumerable discrete actions.

Value approximation has three common interfaces. A state-value approximator \(\widehat V_\theta(x)\approx V^\pi(x)\) maps a state to one number. An action-value approximator maps a pair \((x,u)\) to one number, or maps a state to the vector of all \(m\) action values \(\widehat Q_\theta(x,u_1),\ldots,\widehat Q_\theta(x,u_m)\). With \(m\) discrete actions the vector form is cheaper where control needs it: the greedy action \(\arg\max_u\widehat Q_\theta(x,u)\) requires one evaluation of the approximator instead of \(m\), so the table's lookup over actions becomes a lookup in one output vector. Within each interface the approximator may be a linear function of features, a neural network, a random forest, a nearest-neighbor rule, or another regression model. The learning targets below do not depend on that choice; stability, examined later, does.

If true \(Q^\pi\) were available, squared-error regression would update

This update is a stochastic gradient step, which is easiest to see for state values. Choose \(\theta\) to minimize the mean-squared value error

where the expectation weights states by how often \(\pi\) visits them. Only the prediction depends on \(\theta\), so \(\nabla_\theta J(\theta)=-2\,\mathbb E_\pi[(V^\pi(x)-\widehat V_\theta(x))\nabla_\theta\widehat V_\theta(x)]\), and a step against the gradient, scaled by \(\alpha/2\), is

The factor \(\tfrac12\) only absorbs the 2 produced by differentiating the square. Stochastic gradient descent replaces the expectation by the term at one visited state, \(\Delta\theta=\alpha(V^\pi(x)-\widehat V_\theta(x))\nabla_\theta\widehat V_\theta(x)\), whose expectation is the full step; the update for \(\widehat Q_\theta\) displayed above is the same step with pairs in place of states. With suitably decreasing steps, gradient descent converges to a stationary point of \(J\), in practice usually a local minimum. For a linear approximator \(\widehat V_\theta(x)=\theta^\mathsf T\psi(x)\) with feature vector \(\psi(x)\), \(J\) is a convex quadratic in \(\theta\), so every local minimum is global; for a neural network it is not, and the point reached depends on the initialization.

RL has no such labels. Monte Carlo returns or bootstrapped TD targets replace them. These targets are estimates whose quality changes during learning.

Monte Carlo substitutes the return for the unknown value,

and temporal-difference learning substitutes the one-step target,

The two substitutions differ in kind. Given \(x_t\), the return is an unbiased sample of \(V^\pi(x_t)\) that does not depend on \(\theta\), so the Monte Carlo update is a stochastic gradient step on \(J\). The one-step target contains \(\widehat V_\theta(x_{t+1})\), so it is biased while the approximation is wrong, and it moves whenever \(\theta\) moves. Its update differentiates only the prediction, not the target, and is therefore called a semi-gradient update: it is neither the gradient of \(J\) nor that of the squared TD error, which would also differentiate \(\widehat V_\theta(x_{t+1})\).

The tabular and approximate versions use the same targets differently. Along an episode \(x_0,u_0,r_0,x_1,\ldots,x_T\), a tabular method computes a target for each visited state, the return \(G_t\) or \(r_t+\gamma\widehat V(x_{t+1})\), and moves the single entry \(\widehat V(x_t)\) toward it, leaving every other entry unchanged. An approximate method turns the same targets into regression pairs, \((x_t,G_t)\) for Monte Carlo or \((x_t,r_t+\gamma(1-d_{t+1})\widehat V_\theta(x_{t+1}))\) for temporal differences, and adjusts \(\theta\), which moves the predictions at every state the parameters influence. The successor index in the second pair carries the method. A target written with \(\widehat V_\theta(x_t)\) in place of \(\widehat V_\theta(x_{t+1})\) no longer looks ahead: in a table its fixed point is the expected immediate reward divided by \(1-\gamma\), the value of collecting the current reward forever, whatever the successor states are worth (Exercise 8).

Fitted Q iteration turns Bellman updates into repeated regression 58:34

Given transitions \(\mathcal D=\{(x_i,u_i,r_i,x_i',d_i)\}\) and a frozen previous critic \(\widehat Q_{\bar\theta}\), construct

Freeze, fit, replace, and repeat. This is fitted Q iteration. Online deep Q-learning interleaves collection and stochastic regression. A semi-gradient treats \(y_i\) as constant during the online update; differentiating through the same target changes the problem and can create a target-chasing shortcut.

The per-transition form of this regression applies the semi-gradient step to the action-value target with the maximum, using the current parameters in both the target and the prediction:

The name fitted Q iteration is applied to both forms. They coincide when the frozen parameters are refreshed after every step, \(\bar\theta=\theta\), and they differ in how long a target is held fixed, which is the quantity the target network of the next section controls. A table is the special case with one parameter per pair, \(\widehat Q_\theta(x,u)=\theta_{x,u}\). The gradient \(\nabla_\theta\widehat Q_\theta(x_t,u_t)\) is then the indicator of the pair \((x_t,u_t)\), and the step is exactly the tabular Q-learning update.

Inside generalized policy iteration this regression is approximate policy evaluation: \(\widehat Q_\theta\) is fitted toward the action values of the current policy, and \(\epsilon\)-greedy improvement with respect to \(\widehat Q_\theta\) produces the next policy. In the three-process skeleton, generating samples means running the current \(\epsilon\)-greedy policy and recording \((x_t,u_t,r_t,x_{t+1},d_{t+1})\). Estimating the return means forming \(y_t=r_t+\gamma(1-d_{t+1})\max_u\widehat Q_{\bar\theta}(x_{t+1},u)\) and updating \(\theta\) to reduce the squared error between \(y_t\) and \(\widehat Q_\theta(x_t,u_t)\). Improving the policy means setting \(\pi(x)\in\arg\max_u\widehat Q_\theta(x,u)\). The tabular argument that this loop approaches \(Q^\star\) does not carry over. Evaluation is now inexact, and the error does not vanish with more data unless the function class can represent the targets; the fixed-point analysis shows that the projected Bellman operator need not contract. The approximate loop may settle near \(Q^\star\), oscillate among policies, or diverge.

Generalization is valuable when nearby states truly have nearby optimal values. It is harmful across discontinuities such as the cliff boundary. Representation, data coverage, and inductive bias determine which regime dominates.

DQN stabilizes correlated data and moving targets 1:06:09

The DQN system coupled convolutional Q functions with two key devices. Experience replay stores transitions and samples minibatches across times, reducing consecutive correlation and reusing experience. A target network supplies \(\widehat Q_{\bar\theta}\) while the online network changes.

DQN is one of the most widely used deep reinforcement-learning algorithms and an early success of reinforcement learning with deep neural networks: it learned to play Atari games such as Breakout from screen images. Recent images stand in for the state; the implementation section explains why a single frame is not enough. The network passes them through convolutional layers and then fully connected layers, and it ends with one output per joystick action. The full Atari action set has 18 entries: no input, fire, the eight directions, and the eight directions combined with fire. This is the state-to-vector interface of the approximation section, so one forward pass yields every action value needed for the maximum in the target and for the greedy action.

Replay and the target network answer two distinct failures of the plain combination of Q-learning and a neural network. First, consecutive transitions of one trajectory are highly correlated, whereas stochastic regression is analyzed for samples drawn independently from a fixed distribution; a run of correlated updates pulls the parameters toward one region of the state space. Second, the regression target \(r_t+\gamma\max_{u'}\widehat Q_\theta(x_{t+1},u')\) moves whenever \(\theta\) moves, so each step changes the problem it is solving. DQN's loop addresses both. It acts \(\epsilon\)-greedily with respect to \(\widehat Q_\theta\), stores each transition \((x_t,u_t,r_t,x_{t+1},d_{t+1})\) in the replay memory \(\mathcal D\), samples a minibatch of \(B\) stored transitions, computes targets with the frozen parameters \(\bar\theta\), and takes a gradient step on

a sample estimate of the expected squared error under the replay distribution. Sampling from the buffer addresses the correlation. Holding \(\bar\theta\) fixed between target updates makes each stretch of training an ordinary regression with fixed labels.

A hard target update copies \(\bar\theta\leftarrow\theta\) every \(K\) steps. A soft Polyak update uses \(\bar\theta\leftarrow\rho\bar\theta+(1-\rho)\theta\) for retention \(\rho\) near one. Some libraries name the new-weight fraction \(\tau=1-\rho\); the equation removes ambiguity.

Replay samples are not truly IID: they share policies and episodes, and the buffer distribution shifts. FIFO buffers emphasize recent experience; large buffers retain obsolete behavior. Prioritized replay changes sampling and can require importance weighting. Replay mitigates correlation; it does not erase distribution shift.

The deadly triad remains after the engineering fixes

Function approximation, bootstrapping, and off-policy data can interact pathologically. The max also produces optimistic bias when noisy estimates both select and evaluate an action. Double Q methods separate those roles. These later refinements follow the same principle: break harmful feedback or make unsupported value visible.

A fixed target network delays feedback but can be stale. A replay buffer broadens data but can underrepresent new critical states. A lower TD loss can coexist with inflated Q values because targets inflate too. Compare predicted start value with empirical evaluation return and monitor value quantiles, TD errors, gradients, action gaps, and buffer coverage.

Reward clipping and Huber loss can stabilize Atari training but change scale or objective. Gradient clipping limits optimization steps, not Bellman bias. Treat each as an intervention with a measured effect.

The fixed-point view explains both convergence and failure

For any bounded table \(Q\), define the Bellman optimality operator \[ (\mathcal T^\star Q)(x,u)=\mathbb E[r+\gamma(1-d)\max_{u'}Q(x',u')\mid x,u]. \] For \(0\leq\gamma<1\), it is a contraction in the sup norm. Indeed, the reward cancels when comparing two tables, expectation cannot enlarge the maximum absolute difference, and \(|\max a_i-\max b_i|\leq\max_i|a_i-b_i|\). Therefore \[ \|\mathcal T^\star Q-\mathcal T^\star\widetilde Q\|_\infty \leq\gamma\|Q-\widetilde Q\|_\infty. \] Exact synchronous value iteration converges to the unique fixed point \(Q^\star\). Tabular Q-learning is a stochastic asynchronous approximation to this contraction when every relevant pair is updated often enough and noise and step sizes satisfy the theorem's conditions.

SARSA instead follows a Bellman expectation operator for the current behavior. With a fixed \(\epsilon\)-greedy policy, it evaluates the consequences of continued exploration. If \(\epsilon\) decays while all actions remain sampled infinitely often—often summarized as greedy in the limit with infinite exploration—the limiting target can approach the greedy optimum. Decaying exploration too quickly violates coverage; never decaying it preserves exploratory risk. The schedule is therefore part of the control problem, not a cosmetic hyperparameter.

Function approximation inserts a projection. Fitted Q iteration approximately applies \(\Pi_{\mathcal F}\mathcal T^\star\), where \(\Pi_{\mathcal F}\) projects targets into function class \(\mathcal F\) under the dataset's regression distribution. Even if \(\mathcal T^\star\) is a contraction, the projected composition need not be one in the same norm. Worse, the data distribution changes with behavior. The tabular proof cannot be rescued by saying a neural network is merely a large table.

The cliff also distinguishes optimality criteria. Q-learning's greedy path is optimal for the modeled MDP with exact action execution after training. SARSA's wider path can be better for the persistent \(\epsilon\)-greedy controller. If a real actuator has irreducible random failures, that randomness belongs in the transition model; then the truly optimal policy may also keep margin. Treating deployment noise as temporary training exploration asks the wrong MDP.

A three-state example separates sampling from the target

Consider states \(A,B,G\), terminal goal \(G\), and actions safe and risky at \(B\). From \(A\), the only action gives reward zero and enters \(B\). At \(B\), safe gives reward \(2\) and terminates; risky gives reward \(10\) with probability \(0.6\) and \(-10\) otherwise, then terminates. Its expected value is \(0.6(10)+0.4(-10)=2\), tied with safe. A risk-neutral Q table treats them equally, despite different variance.

Suppose estimation noise currently makes \(\widehat Q(B,\text{risky})=4\) and \(\widehat Q(B,\text{safe})=2\), with \(\gamma=0.9\). Q-learning constructs the target for the transition from \(A\) as \(0+0.9(4)=3.6\). SARSA's target depends on the sampled action. Under an \(\epsilon\)-greedy policy with two actions and \(\epsilon=0.2\), risky is selected with probability \(0.9\) because it is currently greedy, so expected SARSA target is \(0.9[0.9(4)]+0.1[0.9(2)]=3.42\). Both inherit the noisy ranking; Q-learning fully commits its bootstrap to the maximum.

After accurate learning, both action means are two. A fixed tie-breaking rule can still always choose risky, producing much larger return variance. Value-based expected-return control is indifferent. If tail risk matters, modify the objective, constrain failure probability, or learn a distributional value. A wider neural ensemble by itself does not change the task objective.

Now suppose the replay data contain 10,000 safe transitions and only 10 risky transitions, nine lucky. The empirical risky mean is eight, a severe finite-sample error. Maximization selects precisely the undercovered optimistic action. A held-out average TD error dominated by safe data can look excellent. Coverage-aware evaluation must report counts or effective sample size by action and examine values where the greedy policy differs from behavior.

The DQN loop has coupled clocks and contracts

At each environment step, choose an action from an exploration policy based on the online critic, observe \((x,u,r,x',d)\), and append it to replay. After warm-up, sample a minibatch, calculate targets with the target critic, update the online critic, and periodically move target parameters. Evaluation runs in separate episodes without writing to replay unless explicitly intended.

Three ratios determine dynamics: environment steps per action decision, gradient updates per environment step, and target movement per gradient update. High update-to-data ratio can fit replay noise and propagate it through bootstraps. A very slow target reduces oscillation but learns stale values. A tiny new buffer makes batches correlated; an enormous old buffer can make adaptation slow after the policy discovers a new region.

Terminal semantics are a contract. Genuine terminal states remove the bootstrap. A time-limit truncation may represent an interrupted continuing task and should often retain it. Modern environments can report terminated and truncated separately. Combining them blindly biases value near the limit.

For image observations, stack or recurrently encode enough frames to infer velocity. A single static frame can violate the Markov property: identical positions moving toward or away from the cliff require different control. Frame stacking increases state dimension and replay storage, while recurrence requires sequence sampling and hidden-state burn-in. Architectural convenience does not decide state sufficiency.

The target max assumes each action value is comparable. If rewards differ in scale across tasks or observation normalization drifts, shared-network outputs can move together. Log raw rewards, transformed rewards, target values, predictions, and action gaps. Saturated action gaps can hide that every Q value is wrong by a large common offset.

Use a fixed evaluation suite of initial states and random seeds, but add independently sampled episodes to avoid overfitting the suite. Report learning curves over environment interaction, not only gradient steps. For control deployment, include constraint violations, recovery time, and worst-case or tail performance alongside mean return.

A reproducible value-based experiment needs three separate policies

Record the behavior policy used for data, the target implicit in the backup, and the evaluation policy used for scores. In cliff training these can be \(\epsilon\)-greedy, greedy, and greedy; collapsing their names causes contradictory conclusions.

Start with an exact tabular oracle, verify terminal masks and reward timing, then replace the table with a linear function, then a neural critic. Hold data fixed while testing regression. Only then introduce a changing behavior policy and replay. This sequence localizes whether failure comes from MDP code, target construction, optimization, or distribution shift.

Report environment version, observation and action encoding, reward, termination versus time limit, discount, exploration schedule, replay capacity and warm-up, update-to-data ratio, target cadence, optimizer, network, loss, gradient clip, seeds, and evaluation rule.

Exercises with worked solutions

What this chapter establishes

SARSA learns the value of its exploratory continuation; Q-learning uses the same experience to learn greedy continuation. Their cliff behavior follows from that target choice. Function approximation converts local table updates into coupled regression. DQN's replay and target network reduce two major instabilities but do not restore the clean tabular theorem. Successful control still requires coverage, correct terminal semantics, calibrated evaluation, and diagnostics beyond TD loss.