Stochastic control and Markov value functions

The cart's next velocity impulse is unknown when the controller chooses its acceleration. An optimizer that selects the action after seeing that impulse solves an easier problem with information the real controller does not possess. Stochastic dynamic programming preserves the order: observe the current state, choose the action, then average over the disturbance that arrives afterward.

The running cart now obeys \(x_{k+1}=A_kx_k+B_ku_k+w_k\), where \(x_k\) contains normalized position and velocity, \(u_k\) is normalized acceleration, and \(w_k\) is a random state increment. We retain finite-horizon quadratic costs initially, then use the recording's inventory example to expose action constraints and nonquadratic transitions. The same value-function language will finally extend to a stationary discounted problem.

Grounded in AA203 lecture 9. The recording ends after defining infinite-horizon state and state-action values. Value iteration and policy iteration appear later in the slide deck but are explicitly deferred to the next recording. This chapter derives the equations and evaluates examples; the algorithmic treatment follows in chapter 10. The official slides add the course placement, the slide and cross-community notation, the formal stochastic principle of optimality and dynamic-programming theorem, the additive-cost and Markov assumptions that permit the recursion, the inventory recursion in closed form, the stationary Markov decision process tuple, and the distinction between model-based and model-free solution; where their notation differs, the text translates it.

A Markov model specifies both uncertainty and available information 0:35

The chapter stays on the closed-loop branch of the course. Chapters 7 and 8 computed feedback policies by dynamic programming for deterministic models: LQR solved the backward recursion exactly, and iLQR and DDP applied it to local quadratic approximations of nonlinear problems. Here the model acquires a random disturbance, while the object sought is still a policy that maps each measured state to an action. The infinite-horizon formulation at the end of the chapter connects this branch to the reinforcement-learning methods of the course's final part, which estimate the same value functions or policies from data when the model is not given. Continuous-time closed-loop control through the Hamilton–Jacobi–Bellman and Hamilton–Jacobi–Isaacs equations follows in chapter 10.

In the general model \(x_{k+1}=f_k(x_k,u_k,w_k)\), the disturbance has conditional distribution \(P_k(dw\mid x_k,u_k)\). This notation allows the distribution to depend on the current state and chosen control. The Markov assumption says that, conditional on those quantities, the modeled next-state distribution needs no further history. It does not say earlier disturbances have no effect: their effects can be carried forward in the present state.

The policy is \(u_k=\pi_k(x_k)\). It observes the current state before choosing an action and does not observe \(w_k\) in advance. For a finite horizon \(N\), stage cost \(g_k(x,u,w)\), and terminal cost \(\phi_N(x)\), its expected cost is

The policy changes the distribution of future states, which is why the expectation is marked by \(\pi\). The value function \(V_k(x)=\inf_\pi J_k^\pi(x)\) minimizes this expected cost. We assume the relevant costs are integrable and, when a minimum is used, that a minimizing admissible action exists. Finite state and action examples below avoid additional measurable-selection issues.

The dynamics, the admissible controls, the disturbance law, the class of feedback policies, and the expected additive cost together define a finite-horizon Markov decision process (MDP). Admissible controls are written \(u_k\in\mathcal U_k(x_k)\); the set may depend on the current state, as the warehouse capacity in the inventory example will show. The official course slides write \(g_N\) for the terminal cost, \(J_\pi(x_0)\) for the expected cost of a policy from the initial state, and \(J^\star(x_0)=\min_\pi J_\pi(x_0)\) for the optimal cost, and they label the expectation by the disturbances \(w_0,\ldots,w_{N-1}\) that it averages. In this chapter's symbols, \(g_N=\phi_N\), \(J_\pi(x_0)=J_0^\pi(x_0)\), and \(J^\star(x_0)=V_0(x_0)\). Because the law of \(w_k\) may depend on \(x_k\) and \(u_k\), and through the state on earlier disturbances and actions, the average is taken over the joint distribution that the policy induces; this chapter therefore marks the expectation with \(\pi\) rather than listing the disturbances.

Other fields describe the same object in other symbols. Reinforcement learning, developed largely in computer science, writes a state as \(s\in\mathcal S\) and an action as \(a\in\mathcal A\), and it usually specifies the successor directly through transition probabilities rather than through a transition function and a disturbance law. Operations research brings further conventions of its own. W. B. Powell's report “AI, OR and control theory: A Rosetta Stone for stochastic optimization” (Princeton University, 2012) is a guide to the notational correspondences among these communities. This chapter keeps the control notation \(x\), \(u\), and \(w\) throughout and introduces the transition-probability form in the infinite-horizon section, where it is first needed.

State selection remains important. If gusts have persistent correlation not captured by the current cart state, position and velocity alone may not form a sufficient Markov state. Adding an observed wind variable can repair the model. If that variable is hidden, a belief about it may become part of the information state, creating a partially observed problem. Pretending it is observed would grant the controller unavailable information.

A minimal original example makes the information order quantitative. Let \(w\) equal minus one or plus one with equal probability, choose \(u\in\{-1,1\}\), and pay \((u-w)^2\). Either admissible action chosen before the disturbance has expected cost two. An action chosen after seeing \(w\) can set \(u=w\) and pay zero. Thus

Moving the minimization inside the expectation changes the controller's information. The distinction will matter again for adversarial disturbances, where changing the order of control and disturbance choices changes the game.

Expected cost is a decision criterion, not a tail guarantee 14:05

The source uses a risk-neutral objective: only the expected accumulated cost enters the comparison. This does not mean uncertainty is ignored. The full distribution can affect the expectation through nonlinear costs and transitions. It means that two policies with the same expected cost are not distinguished by an additional preference over their cost distributions.

For an original one-decision example, policy A always costs two. Policy B costs zero with probability \(0.99\) and one hundred with probability \(0.01\). Expected-cost minimization prefers B because its mean is one. Yet B has a one-percent chance of a cost far above anything A produces. If the design requirement prohibits such outcomes, an expected-cost objective alone does not encode it.

Adding variance of the total cost changes the mathematics because variance is not additive across arbitrary dependent stages. Worst-case objectives and nested risk measures make other choices. Some admit dynamic programming with an appropriate formulation; the risk-neutral Bellman equation should not be reused without deriving the required state and recursion. The next development relies specifically on conditional expectation and its tower property.

Conditional expectation turns a trajectory distribution into one-step recursion 17:29

The recursion rests on a stochastic version of the principle of optimality. Let \(\pi^\star=(\pi_0^\star,\ldots,\pi_{N-1}^\star)\) be an optimal policy for the initial state \(x_0\). For a stage \(k\) and a state \(x\), the tail subproblem minimizes the remaining expected cost \(J_k^\pi(x)\) of the first section over the tail policy \((\pi_k,\ldots,\pi_{N-1})\). The earlier components \(\pi_0,\ldots,\pi_{k-1}\) do not enter it, because the expectation is conditioned on \(x_k=x\) and the past acts on the future only through the current state. The principle states that the tail \((\pi_k^\star,\ldots,\pi_{N-1}^\star)\) of an optimal policy is optimal for the tail subproblem. In the stochastic setting this holds at the states that \(\pi^\star\) reaches at stage \(k\) with positive probability, not necessarily at every state; the qualification and its proof follow the recursion below. Dynamic programming turns the principle into an algorithm by solving all tail subproblems rather than only those along one trajectory: first every tail subproblem of length one, at stage \(N-1\), then every tail subproblem of length two using those solutions, and so on back to stage zero.

The terminal value is \(V_N(x)=\phi_N(x)\). Suppose the optimal next-stage value is known. After choosing \(u\), the disturbance determines both the immediate cost and successor. Conditional on that successor, the optimal expected remaining cost is already summarized by \(V_{k+1}\). The law of iterated expectations, \(\mathbb E[\mathbb E[Z\mid Y]]=\mathbb E[Z]\) for integrable \(Z\), therefore gives

The current action is fixed while this expectation averages over the current disturbance. Future actions can adapt to future observed states, and that adaptation is already included in the successor value. Replacing \(V_{k+1}\) by the cost of a fixed future control sequence would solve an open-loop continuation problem instead.

Backward induction turns the recursion into the stochastic dynamic-programming theorem. For every initial state \(x_0\), the value \(V_0(x_0)\) that the recursion computes from \(V_N=\phi_N\) equals the optimal expected cost \(\inf_\pi J_0^\pi(x_0)\). If, in addition, \(\pi_k^{\mathrm{DP}}(x)\) attains the minimum for every state and stage, the dynamic-programming policy \(\pi^{\mathrm{DP}}=(\pi_0^{\mathrm{DP}},\ldots,\pi_{N-1}^{\mathrm{DP}})\) is optimal from every initial state. Both claims follow from one induction. At the final stage, \(J_N^\pi=\phi_N=V_N\) for every policy. Suppose that \(J_{k+1}^\pi\geq V_{k+1}\) for every policy and that \(J_{k+1}^{\pi^{\mathrm{DP}}}=V_{k+1}\). Conditioning on the successor gives

where \(w_k\sim P_k(\cdot\mid x,\pi_k(x))\) and \(x_{k+1}=f_k(x,\pi_k(x),w_k)\). The first inequality uses the induction hypothesis and the second the minimization over \(u\) in the recursion. For \(\pi^{\mathrm{DP}}\) both inequalities are equalities, because its continuation cost is \(V_{k+1}\) and its action attains the minimum. Hence \(V_k\) is a lower bound on every policy's remaining cost and is attained by \(\pi^{\mathrm{DP}}\), at every state and stage.

The expectation in each Bellman step averages over the current disturbance \(w_k\) alone, not over the whole sequence \(w_0,\ldots,w_{N-1}\). This is the computational content of the theorem. If each disturbance takes \(m\) values, evaluating one policy's objective by direct averaging sums over up to \(m^N\) disturbance sequences, and comparing policies multiplies that count by the number of policies. The recursion instead evaluates \(m\) outcomes for each stage, state, and admissible action. In the three-stage inventory problem below, one policy's objective averages over \(3^3=27\) demand sequences, and the exhaustive check in the implementation section repeats that average for each of \(216\) policies and three initial stocks, \(17{,}496\) three-step paths in all. The recursion evaluates six state-action pairs per stage with three demand outcomes each, \(3\cdot6\cdot3=54\) one-step terms. The saving concerns the horizon. It does not remove the dependence on the number of states, which grows exponentially with the state dimension when a continuous state is gridded, as chapter 7 showed.

Two structural assumptions carried the derivation. The first is additivity of the cost, the central assumption of the framework: the objective is a sum of stage costs, so the cost already incurred separates from the cost still to come, and the expectation, being linear, distributes over the sum. The second is the Markov property of the disturbance law, which lets the successor state summarize everything the remaining problem needs, so that \(V_{k+1}(x_{k+1})\) can stand for it. The persistent-gust example of the first section shows how the second assumption fails and how state augmentation repairs it. Additivity fails in the same way. If the objective were the largest stage cost rather than the sum, the remaining cost would depend on the largest cost incurred so far, which the state does not record; including that running maximum in the state restores a recursion at the price of an extra state dimension. A variance penalty on the total cost fails differently, as the preceding section noted: it does not decompose into stage terms, so the standard recursion cannot be applied to it without a further construction.

The optimal-tail argument requires a qualification in a stochastic setting. A policy optimal only for one initial-state distribution can act arbitrarily at states that it never reaches. Its behavior at every imaginable state need not be optimal. Backward induction constructs a policy optimal for each modeled state-stage subproblem, avoiding that ambiguity. On states reached with positive probability, replacing a strictly worse continuation would improve the original expected objective.

The theorem makes this qualification exact. Let \(\pi^\star\) be optimal from \(x_0\), and compare it with the policy that follows \(\pi^\star\) through stage \(k-1\) and \(\pi^{\mathrm{DP}}\) from stage \(k\) onward. Both policies generate the same distribution of the first \(k\) stage costs and of \(x_k\). Writing \(C_k\) for the sum of those first \(k\) stage costs, the comparison policy costs \(\mathbb E[C_k+V_k(x_k)]\geq V_0(x_0)\), because no policy beats the optimum, while \(\pi^\star\) costs \(\mathbb E[C_k+J_k^{\pi^\star}(x_k)]=V_0(x_0)\). Subtracting gives \(\mathbb E[J_k^{\pi^\star}(x_k)-V_k(x_k)]\leq0\). The integrand is nonnegative by the theorem, so in a finite model it vanishes at every state that \(x_k\) takes with positive probability; in general it vanishes almost surely. At those states the tail of \(\pi^\star\) solves the tail subproblem. At unreached states nothing constrains it, which is the stochastic principle of optimality in its correct form.

Bertsekas's basic stochastic dynamic-programming notes supply the primary formal framework. The information-order example and the explicit conditional argument here make clear which assumptions permit the recursion.

The inventory example: cost uses unmet demand even when the next state clips it away 23:47

The source considers a warehouse with capacity two. State \(x_k\in\{0,1,2\}\) is stock before ordering, control \(u_k\in\{0,\ldots,2-x_k\}\) is the number of units purchased, and demand \(w_k\in\{0,1,2\}\) has probabilities \(0.1,0.7,0.2\). The order arrives before demand. Unsatisfied demand is lost rather than backlogged, giving

The quadratic term penalizes both excess stock and unmet demand. It is evaluated before clipping negative stock to zero. Replacing it with \(x_{k+1}^2\) would erase the shortage penalty and solve a different problem. The purchase term and quadratic mismatch term use the source's normalized cost units; this is a teaching model, not an empirically calibrated retail cost law.

With these data the Bellman recursion takes a concrete form. The admissible set \(\mathcal U(x)=\{0,\ldots,2-x\}\) depends on the stock, which is how the capacity constraint enters, and the demand law depends on neither the stock nor the order, a special case of \(P_k(\cdot\mid x,u)\). Writing \(p(w)\) for the demand probabilities, so that \(p(0)=0.1\), \(p(1)=0.7\), and \(p(2)=0.2\), the recursion for \(k=2,1,0\) is

The successor value is evaluated at the clipped stock, while the stage cost uses the unclipped mismatch. The backward loop in the implementation section transcribes this line, one stock and one order at a time.

There are three decisions, at stages zero, one, and two, and no terminal cost. Thus \(V_3(x)=0\). At the last stage with \(x=0\), the three candidate costs are

Ordering one is optimal. At state one, the candidate actions are zero and one; ordering zero costs \(0.3\), compared with \(2.1\) for ordering one. At state two, ordering zero is the only feasible action and costs \(1.1\). Therefore \(V_2=(1.3,0.3,1.1)\), where the vector order is states zero, one, two.

Move back one stage. At state zero, ordering one leaves state one when demand is zero and state zero otherwise. Its expected continuation cost is \(0.1V_2(1)+0.9V_2(0)=1.2\), so its total is \(1.3+1.2=2.5\). Ordering zero costs \(1.5+V_2(0)=2.8\); ordering two costs \(3.1+0.1V_2(2)+0.7V_2(1)+0.2V_2(0)=3.68\). The best choice remains one.

The final row agrees with the source's reported solution. The policy happens to be identical at all three stages in this example, but finite-horizon policies generally depend on time. The zero terminal cost, short horizon, and chosen probabilities are all part of the result.

The example also quantifies the value of seeing demand early. At the final stage from zero stock, if demand were revealed before ordering, matching it with \(u=w\) would minimize the realized cost among the available orders. Its mean purchase cost is \(0.7+0.4=1.1\), compared with \(1.3\) under the actual information pattern. The difference \(0.2\) is a value of perfect advance information for that one decision; it is not a feasible improvement without changing the observation order.

Zero-mean additive noise changes the LQR value by a trace term 46:05

Return to \(x_{k+1}=A_kx_k+B_ku_k+w_k\) with full observation of \(x_k\). Assume the disturbance has conditional mean zero and covariance \(\Sigma_k=\mathbb E[w_kw_k^\mathsf T]\), independent of the chosen action and current state, with finite second moments. The source uses independent Gaussian disturbances. Gaussianity is sufficient but is not needed for the quadratic expectation identity below.

Retain chapter 7's cost \(\frac12x_N^\mathsf THx_N+\frac12\sum(x_k^\mathsf TQ_kx_k+u_k^\mathsf TR_ku_k)\), with \(Q_k,H\succeq0\) and \(R_k\succ0\). Suppose the next value is \(V_{k+1}(x)=\frac12x^\mathsf TP_{k+1}x+c_{k+1}\). Let \(z=A_kx+B_ku\), fixed while taking the current conditional expectation. Expanding gives

The middle term vanishes. For the final term, a scalar equals its trace, and cyclic trace permutation gives \(w^\mathsf TPw=\operatorname{tr}(Pww^\mathsf T)\). Linearity of expectation then yields \(\mathbb E[w^\mathsf TPw]=\operatorname{tr}(P\Sigma)\). This derivation uses only the second moment, not a Gaussian density.

The additional term is constant with respect to \(u\). Consequently the minimizing feedback gain and Riccati matrices are exactly the deterministic ones, while

Noise has a cost even when it does not change the gain. The trace weights covariance directions by their future cost curvature. Two disturbances with equal total variance can have different expected cost if one pushes the system in a more expensive direction. For a random initial state with mean \(\mu_0\) and covariance \(\Sigma_0^x\), the expected optimum is \(\frac12\mu_0^\mathsf TP_0\mu_0+\frac12\operatorname{tr}(P_0\Sigma_0^x)+c_0\).

Boyd's primary stochastic LQR notes independently develop this quadratic-plus-constant value structure. The unchanged feedback gain is sometimes called a certainty-equivalence property in this fully observed setting. It does not mean using a precomputed open-loop control sequence: the action still responds to the realized current state.

A scalar original check uses \(x_{k+1}=x_k+u_k+w_k\), two stages, \(Q=R=H=1\), \(x_0=1\), and independent disturbances \(w_k=\pm0.2\) with equal probability. The deterministic matrices are \(P_2=1,\ P_1=1.5,\ P_0=1.6\), so the baseline cost is \(0.8\). The noise constant is \(\frac12(0.04)(1+1.5)=0.05\), giving expected cost \(0.85\). Enumerating all four disturbance sequences while applying feedback reproduces that value exactly. This non-Gaussian check also tests the claimed distributional boundary.

The induction hypothesis must carry the constant. Even when \(c_N=0\), one Bellman step produces \(c_{N-1}=\frac12\operatorname{tr}(P_N\Sigma_{N-1})\), so a hypothesis that \(V_{k+1}\) is purely quadratic fails after a single stage whenever \(\operatorname{tr}(P_N\Sigma_{N-1})\neq0\), which holds in particular when \(H\succ0\) and \(\Sigma_{N-1}\neq0\). Dropping the constant leaves every gain unchanged, since it does not depend on \(u\), but it understates the expected cost: in the scalar check it would report \(0.8\) instead of \(0.85\). The stage index matters for the same reason. The value at stage \(k+1\) is \(\frac12x^\mathsf TP_{k+1}x+c_{k+1}\), and it is \(P_{k+1}\), not \(P_k\), that enters the trace term and the gain at stage \(k\). The official slides write \(\bar x_0\), \(\Sigma_{x_0}\), and \(\Sigma_{w_k}\) for \(\mu_0\), \(\Sigma_0^x\), and \(\Sigma_k\), and an unsubscripted \(Q\) for the terminal weight \(H\).

The unchanged gain fails when the control changes uncertainty

The action independence of covariance is doing real work. For an original one-step example, let \(x_1=x+u+\sigma u\xi\), where \(\xi\) has mean zero and variance one, and \(\sigma\geq0\) measures multiplicative actuator uncertainty. Minimize \(\frac12ru^2+\frac12q\,\mathbb E[x_1^2]\), with \(q,r>0\). Expanding the expectation gives

The variance term now depends on the action and changes the minimizer. With \(x=q=r=\sigma=1\), the optimal control is \(-1/3\), while deterministic LQR would choose \(-1/2\). Their expected costs are \(1/3\) and \(3/8\), respectively. The stochastic controller uses less aggressive action because larger magnitude also injects more uncertainty.

Nonzero conditional mean similarly adds a predictable drift and can introduce feedforward terms. Constraints, nonlinear dynamics, partial observations, and risk-sensitive objectives can invalidate other parts of the simple derivation. The trace calculation identifies exactly which assumptions permit the unchanged gain, making it more useful than an unrestricted instruction to “ignore noise.”

An infinite stationary horizon replaces a terminal boundary by a fixed point 54:35

The source now changes from minimizing costs to maximizing rewards. To make the sign explicit, write \(\rho(x,u)=-g(x,u)\) for an expected one-stage reward and use \(W^\pi\) for reward value. The transition kernel \(T(x'\mid x,u)\) is the probability of successor \(x'\) given the current state and action. It combines the disturbance law and transition function into one object.

Under stationarity, five objects specify the problem completely: the state set \(\mathcal X\), the actions \(\mathcal U\) with admissible sets \(\mathcal U(x)\), the kernel \(T\), the reward \(\rho\), and the discount \(\gamma\) defined below. The tuple \(\mathcal M=(\mathcal X,\mathcal U,T,\rho,\gamma)\) is called a stationary Markov decision process. The kernel and the reward follow from the disturbance form of the first section. When the disturbance takes finitely many values and its law \(P(w\mid x,u)\) does not depend on the stage,

For the inventory model with an order of one at stock zero, the successor is one only when demand is zero, so \(T(1\mid0,1)=0.1\) and \(T(0\mid0,1)=0.9\), while \(\rho(0,1)=-1.3\), the negative of the expected stage cost computed earlier. The kernel discards which disturbance produced a transition. This loses nothing for the expected-value recursion: by linearity, \(\mathbb E[g(x,u,w)+V(f(x,u,w))]=-\rho(x,u)+\sum_{x'}T(x'\mid x,u)V(x')\) for any value function \(V\).

The official slides and much of the reinforcement-learning literature write \(R(x,u)\) or \(r_t\) for the reward, \(V^\star\) and \(V_\pi\) for the optimal and fixed-policy values, and \(Q^\star\) and \(Q_\pi\) for the state-action values of the next section, often with \(s\) and \(a\) in place of \(x\) and \(u\). This chapter writes \(\rho\), \(W^\star\), \(W^\pi\), \(\mathcal Q^\star\), and \(\mathcal Q^\pi\), keeping \(R\) for the LQR control weight, \(V\) for cost values, and \(Q\) for the LQR state weight.

Assume a finite state set \(\mathcal X\), finite nonempty action sets, bounded rewards, stationary transitions and rewards, and discount \(0\leq\gamma<1\). A stationary policy \(\pi(x)\) does not explicitly depend on time. Its value is

Stationarity means that the kernel, the reward, and the admissible sets do not depend on the stage: the same state and action produce the same successor distribution and the same reward at every time. The finite-horizon formulation did not impose this restriction, and it is what makes an infinite horizon tractable. A time-varying model would need a separate policy component for each of infinitely many stages, and the limits used below would have to be taken through a changing sequence of problems. Under stationarity the remaining horizon is infinite at every stage, so nothing in the problem distinguishes one stage from another, which is why a single map \(\pi\) from states to actions can suffice; the existence result below confirms it. The optimal value is a function of the initial state, \(W^\star(x)=\sup_\pi W^\pi(x)\), and a policy \(\pi^\star\) is optimal when \(W^{\pi^\star}(x)=W^\star(x)\) for every initial state \(x\) simultaneously, not merely for one initial state or on average over an unspecified initial distribution.

If \(|\rho|\leq R_{\max}\), the sum is absolutely bounded by \(R_{\max}/(1-\gamma)\). After retaining stages zero through \(H-1\), the omitted reward magnitude is at most \(\gamma^H R_{\max}/(1-\gamma)\). The discount makes this statement possible under bounded rewards; it does not guarantee finiteness for arbitrary unbounded rewards or exploding state-dependent costs.

For example, with \(\gamma=0.9\) and \(R_{\max}=1\), a truncation bound below \(0.01\) requires \(0.9^H/0.1<0.01\), hence \(H\geq66\). This is a worst-case numerical bound, not a universal psychological interpretation of a planning horizon. Choosing \(\gamma\) changes the objective as well as the numerical behavior.

Under the finite discounted assumptions, there exists a stationary deterministic optimal policy. This is an existence statement; it does not mean every optimal policy must be stationary, since ties can permit time-varying optimal choices. The optimal reward value satisfies

There is no terminal stage at which to initialize a backward finite recursion. The value appears on both sides, so this is a fixed-point equation. For a specified stationary policy, remove the maximization and insert \(u=\pi(x)\). In vector form, with transition matrix \(T_\pi\) and reward vector \(\rho_\pi\),

Because \(T_\pi\) is stochastic, its maximum absolute row-sum norm is one. The series \(\sum_{j=0}^{\infty}\gamma^jT_\pi^j\) converges for \(\gamma<1\) and is the inverse of \(I-\gamma T_\pi\). This proves uniqueness of fixed-policy evaluation in the finite discounted model. Numerical code should solve the linear system rather than explicitly form the inverse.

State-action values retain the action needed for model-free selection 1:09:58

Define the optimal state-action reward value

It is the expected discounted reward from choosing \(u\) now and acting optimally afterward. It is not merely immediate reward, and it is distinct from the local quadratic state-action approximation in chapter 8. We retain a calligraphic symbol to avoid confusing it with an LQR penalty matrix.

Since \(W^\star(x)=\max_u\mathcal Q^\star(x,u)\), substitution gives

For a fixed policy, use \(\mathcal Q^\pi(x',\pi(x'))\) in place of the successor maximum. In that fixed-policy definition, the first action \(u\) need not equal \(\pi(x)\); the policy is followed afterward. This distinction is what makes the value useful for comparing a current deviation against a baseline policy.

If an oracle provides \(W^\star\), selecting an action still ordinarily requires evaluating its successor distribution and immediate reward. If an oracle provides \(\mathcal Q^\star\), select an action by maximizing it directly, without separately evaluating the transition kernel at deployment. That is the source's motivation for learning state-action values. Obtaining an accurate \(\mathcal Q\) is a separate statistical and optimization problem; approximate values and continuous action maximization need not be easy.

The oracle comparison separates two ways of solving a Markov decision process. When \(T\) and \(\rho\) are known, the fixed-point equations can be solved by dynamic programming. Value iteration and policy iteration, developed in chapter 10, are the two standard algorithms, and both require the model, because every Bellman update and every policy evaluation \((I-\gamma T_\pi)W^\pi=\rho_\pi\) averages over \(T\). Reinforcement learning addresses the case in which the model is not given and must be replaced by data gathered through interaction with the system. A model-based method estimates \(T\) and \(\rho\) from that data and then applies dynamic programming to the estimated model. A model-free method estimates values or a policy directly, without an explicit transition model. Learning \(\mathcal Q^\star\) is the central model-free example, because, as shown above, a state-action value suffices to select actions while a state value alone does not. Both routes return in the course's final part. Their errors enter differently: a model-based method inherits the error of the estimated kernel and reward, while a model-free method's action choices inherit the error of the estimated values, which the next paragraph bounds for one decision.

There is a simple original error guarantee for finite actions. If \(|\widehat{\mathcal Q}(x,u)-\mathcal Q^\star(x,u)|\leq\epsilon\) for every action at a fixed state, and \(\hat u\) maximizes the approximation, then \(\mathcal Q^\star(x,u^\star)-\mathcal Q^\star(x,\hat u)\leq2\epsilon\). Insert the two approximation errors around the inequality \(\widehat{\mathcal Q}(x,\hat u)\geq\widehat{\mathcal Q}(x,u^\star)\). This is a one-decision loss followed by optimal continuation. Repeatedly using an approximate policy requires an additional analysis; the factor \(2\epsilon\) is not automatically its total discounted loss.

Independent enumeration checks the inventory policy

The following original NumPy example computes the nine finite Bellman problems, then independently enumerates every deterministic Markov policy and every demand sequence. There are six state-action maps per stage and \(6^3=216\) three-stage policies, small enough for exhaustive verification. The enumeration applies each policy to the realized stock, preserving the observation order. It has no file or network effects and is deliberately unsuitable for large state spaces.

import itertools
import numpy as np

probability = np.array([.1, .7, .2])
demand = np.arange(3)
value = np.zeros((4, 3))
policy = np.zeros((3, 3), dtype=int)
for k in (2, 1, 0):
    for stock in range(3):
        candidate = []
        for order in range(3-stock):
            next_stock = np.maximum(0, stock+order-demand)
            stage = order + (stock+order-demand)**2
            candidate.append(probability @ (stage + value[k+1, next_stock]))
        policy[k, stock] = np.argmin(candidate)
        value[k, stock] = min(candidate)
assert np.allclose(value[0], [3.7, 2.7, 2.818])
assert np.array_equal(policy, [[1, 0, 0]]*3)

maps = list(itertools.product(range(3), range(2), [0]))
best = np.full(3, np.inf)
for decisions in itertools.product(maps, repeat=3):
    for initial in range(3):
        expectation = 0.
        for realized in itertools.product(range(3), repeat=3):
            stock, cost, weight = initial, 0., 1.
            for k, w in enumerate(realized):
                order = decisions[k][stock]
                cost += order + (stock+order-w)**2
                stock = max(0, stock+order-w)
                weight *= probability[w]
            expectation += weight*cost
        best[initial] = min(best[initial], expectation)
assert np.allclose(best, value[0], atol=1e-12)

# A non-Gaussian stochastic LQR check with all four disturbance paths.
expected = 0.
for noise in itertools.product([-.2, .2], repeat=2):
    x, total = 1., 0.
    for gain, w in zip([.6, .5], noise):
        u = -gain*x
        total += .5*(x*x+u*u)
        x = x+u+w
    expected += .25*(total+.5*x*x)
assert abs(expected-.85) < 1e-12
print(best, round(expected, 6))

The output is [3.7 2.7 2.818] 0.85. Exhaustive enumeration establishes the finite inventory optimum independently of the backward implementation. The noise enumeration verifies the trace correction without Monte Carlo sampling error. Neither check estimates real demand or real cart disturbance probabilities; those distributions are assumed inputs to the model.

Exercises with worked solutions

What remains before these equations become a controller

A Markov decision model requires a defensible state, action set, transition law, cost or reward, and information pattern. Exact computation can optimize an inaccurate model perfectly. Expected performance does not certify a tail limit, and a learned state-action value needs error and coverage evidence before it can replace an explicit transition model.

The next recording begins with algorithms for the stationary fixed-point equations: value iteration and policy iteration. It then connects discrete dynamic programming to continuous-time Hamilton–Jacobi equations. Keeping that boundary visible helps distinguish this chapter's model and identities from the procedures used to compute their solutions.

Primary sources

• AA203 lecture 9 recording and official slides; later algorithm slides belong to the next recording.
• Bertsekas, basic stochastic dynamic programming, MIT 6.231.
• Boyd, Linear Quadratic Stochastic Control, Stanford EE363.