Dynamic programming and discrete linear-quadratic regulation

A cart receives a velocity impulse halfway through a carefully optimized maneuver. The original control sequence still exists, but it answers the wrong question: what should the cart do from the state it was expected to occupy? A feedback law answers a larger family of questions, one for every current state. Dynamic programming computes that family by solving the end of the problem first.

Our running cart has position \(p\), velocity \(v\), and acceleration input \(u\). We use normalized coordinates: position is divided by a reference length, time by a reference duration, and velocity and acceleration by the corresponding derived scales. The numerical values below are dimensionless. For one normalized time unit between decisions, the exact held-input dynamics are \(x_{k+1}=Ax_k+Bu_k\), where \(x_k=(p_k,v_k)^\mathsf T\), \(A=\begin{bmatrix}1&1\\0&1\end{bmatrix}\), and \(B=(1/2,1)^\mathsf T\). The choice of normalized units makes the subsequent identity cost matrices meaningful.

Grounded in AA203 lecture 7. The source develops the principle of optimality, a shortest-path recursion, computational limits, and discrete LQR. The impulse experiment, state-augmentation counterexample, completion-of-squares identity, and executable independent checks are original teaching constructions. Deterministic feedback optimality is distinguished from a robustness guarantee against arbitrary disturbances. The official slides add the course roadmap, the formal tail-subproblem statement, the dynamic-programming theorem for every initial state, the discretization, quantization, and curse-of-dimensionality remarks, and the substitution route to the Riccati update; where their notation differs, the text translates it.

A sequence solves one initial condition; a policy solves a family 3:24

An open-loop plan is a sequence \((u_0,\ldots,u_{N-1})\) indexed by stage, where \(N\) is a finite horizon. A feedback policy is a sequence of functions \(\pi_k\), with \(u_k=\pi_k(x_k)\). Stage \(k\) remains an argument because the number of decisions left changes: the same position and velocity can call for different controls with one step remaining and with twenty.

If the model is deterministic, the initial state is known exactly, and no disturbances occur, an optimal policy generates an optimal open-loop trajectory. There is no contradiction in the two methods agreeing on that trajectory. Their difference appears when the state differs from the nominal state or when the same controller is reused from another initial condition.

Feedback provides a response to a measured deviation, but a deterministic optimal policy is not automatically robust to every model error. An actuator may saturate, the measurement may be inaccurate, or the real dynamics may differ enough that the designed response is unsuitable. The claim established here is narrower: after a state deviation, and assuming the stated model correctly describes the remaining evolution, the policy supplies an optimal continuation for that new state.

This change of object locates the chapter within the course. The preceding chapters computed open-loop sequences by two routes: indirect methods derive necessary conditions and solve the resulting two-point boundary-value problem, while direct methods transcribe the problem into a finite-dimensional nonlinear program. The closed-loop route keeps the same optimal control problem but optimizes over policies, and dynamic programming is its basic algorithm. The richer object costs more to compute, because a policy must prescribe a control at every state that might be visited rather than along one trajectory; the section on computational limits quantifies that cost. Model predictive control, developed in a later chapter, sits between the two routes: it re-solves an open-loop problem from each measured state and applies only the first control of each plan, so it obtains feedback at the price of repeated open-loop solves rather than one tabulated policy.

Dynamic programming is also the common root of several later methods. The linear-quadratic regulator at the end of this chapter is a case in which the recursion can be carried out exactly in closed form. Iterative LQR and differential dynamic programming, developed in the next chapter, apply the same backward recursion to local quadratic models of nonlinear problems. In continuous time the recursion becomes the Hamilton–Jacobi–Bellman partial differential equation, and its two-player counterpart, the Hamilton–Jacobi–Isaacs equation, underlies reachability analysis; both appear in a later chapter. Several reinforcement-learning methods in the course's final part estimate the recursion's values or policies from data when the model is unknown.

Write the general deterministic problem as \(x_{k+1}=f_k(x_k,u_k)\), with state \(x_k\in\mathcal X_k\) and admissible controls \(u_k\in\mathcal U_k(x_k)\). The stage cost \(g_k(x_k,u_k)\) measures the cost incurred at that decision; the terminal cost \(\phi_N(x_N)\) measures the final state. The total objective is

The superscript identifies the policy; the subscript identifies the starting stage. The value function \(V_k(x)=\inf_\pi J^\pi_k(x)\) is the best achievable remaining cost. We use a minimum when an optimizer exists. On a finite action set existence is immediate if at least one feasible continuation exists; on continuous spaces it needs assumptions such as compact admissible sets and suitable continuity. An infeasible state may be assigned \(V_k(x)=+\infty\).

The official course slides use different symbols for several of these objects. They write \(h_N\) for the terminal cost, \(J_k\) for the optimal cost-to-go and \(J^\star\) for the optimal total cost, put the stage into the argument list, as in \(f(x_k,u_k,k)\) and \(g(x_k,u_k,k)\), and mark the transpose with a prime. This chapter writes \(\phi_N\), \(V_k\), stage subscripts, and \(^\mathsf T\) instead, because \(h\) names the destination node of the shortest-path example and \(H\) the terminal weight matrix of the linear-quadratic regulator. The correspondence is one-to-one: \(h_N=\phi_N\), \(J_k=V_k\), \(J^\star(x_0)=V_0(x_0)\), \(f(\cdot,\cdot,k)=f_k\), and \(g(\cdot,\cdot,k)=g_k\).

The tail replacement argument needs a sufficient state 13:50

Suppose an optimal trajectory reaches state \(x_k^\star\) at stage \(k\). If its remaining controls were not optimal for the problem starting from that state and stage, replace them by a cheaper feasible continuation. The earlier controls and their costs stay fixed. Additivity then makes the entire trajectory cheaper, contradicting its supposed optimality. Therefore an optimal trajectory has an optimal tail.

The contradiction needs a strictly cheaper continuation, and the negation of the claim supplies exactly that: if the original tail is not optimal for the problem starting from \(x_k^\star\), some feasible tail costs strictly less. Supposing only that some other tail is optimal would not suffice, because two tails can tie, and a tie produces no contradiction. Ties also limit the conclusion. The argument shows that the tail of an optimal trajectory is an optimal tail, not that it is the only one, so a solver that returns a different minimizer from \(x_k^\star\) has not contradicted it.

In symbols, let \((u_0^\star,\ldots,u_{N-1}^\star)\) be an optimal control sequence from the initial state \(x_0\), and let \((x_0^\star,\ldots,x_N^\star)\), with \(x_0^\star=x_0\), be the state sequence it generates. For a stage \(k\), the tail subproblem starts at \(x_k=x_k^\star\) and minimizes the cost-to-go

over \((u_k,\ldots,u_{N-1})\), subject to \(x_{i+1}=f_i(x_i,u_i)\) and \(u_i\in\mathcal U_i(x_i)\) for \(i=k,\ldots,N-1\). The first term is written separately because its state is fixed, whereas every later state depends on the controls being chosen. The principle of optimality states that the truncated sequence \((u_k^\star,\ldots,u_{N-1}^\star)\) is optimal for this subproblem: tails of optimal sequences are optimal for tail subproblems. The displayed objective is the remaining cost \(J^\pi_k(x_k^\star)\) of the first section, evaluated for an open-loop sequence instead of a policy.

Two assumptions make the replacement legitimate. First, the objective separates into incurred cost and future cost. Second, the recorded state contains everything from the past that determines future dynamics, admissible controls, and future costs. The latter is often the difficult modeling step. Additivity by itself is insufficient if hidden history changes which tails are feasible.

For an original cart example, suppose there is a remaining acceleration budget \(b_k\), with \(b_{k+1}=b_k-|u_k|\) and \(b_k\geq0\). Two trajectories can reach the same physical \((p_k,v_k)\) after expending different budgets. A future maneuver available to the first may be unavailable to the second. A value function using only \((p,v)\) cannot represent both continuation problems. Augment the state to \(z_k=(p_k,v_k,b_k)\); admissibility now includes \(|u_k|\leq b_k\), and the needed memory is restored.

Bertsekas's primary lecture notes on dynamic programming and state augmentation make this modeling operation explicit. It expands the class of problems expressible in the standard framework, while potentially making computation harder by adding state dimensions.

The tail statement is the one needed for backward recursion. There is also a useful related replacement statement: an interior segment of an optimal trajectory is optimal among feasible replacements with the same endpoint states and stages, provided replacing it leaves the remaining problem unchanged. It would be wrong to require that segment to solve an unrelated free-endpoint problem. Specifying the boundary conditions prevents “optimal subpath” from becoming an ambiguous slogan.

Backward computation prices the consequence of each action 24:45

At the final stage no controls remain, so \(V_N(x)=\phi_N(x)\). Suppose \(V_{k+1}\) is already known. Choosing control \(u\) now incurs \(g_k(x,u)\) and moves the system to \(f_k(x,u)\). The cheapest possible continuation from there costs \(V_{k+1}(f_k(x,u))\). Comparing the possible first controls gives the Bellman recursion

To verify both directions of the equality, observe that any complete feasible plan begins with some \(u\), and its tail costs at least the value at the successor. Thus no plan can beat the right-hand side. Conversely, an action attaining that minimum followed by an optimal successor policy attains the displayed cost. If minima are not attained, the analogous argument uses arbitrarily close feasible choices and infima rather than claiming an unavailable optimizer.

Save one minimizing action for each state and stage as \(\pi_k^\star(x)\). The computation proceeds backward; execution proceeds forward. At stage zero measure the state, apply \(\pi_0^\star\), then measure the next state and apply \(\pi_1^\star\). The stored policy is not merely a path reconstructed from one initial condition.

Backward induction on \(k\) assembles these facts into the dynamic-programming theorem. For every initial state \(x_0\), the stage-zero value that the recursion computes from \(V_N=\phi_N\) equals the optimal cost \(\inf_\pi J^\pi_0(x_0)\). If, in addition, \(\pi_k^\star(x)\) attains the minimum for every state \(x\) and every stage \(k\), then the policy \((\pi_0^\star,\ldots,\pi_{N-1}^\star)\) is optimal. The induction runs on that policy's own cost. At the final stage \(J^{\pi^\star}_N=\phi_N=V_N\). If \(J^{\pi^\star}_{k+1}=V_{k+1}\), then \(J^{\pi^\star}_k(x)=g_k(x,\pi_k^\star(x))+V_{k+1}(f_k(x,\pi_k^\star(x)))=V_k(x)\), because \(\pi_k^\star(x)\) attains the minimum in the recursion. A policy whose cost equals the infimum over all policies at every state is optimal from every initial state simultaneously.

The phrase “every initial state” carries a computational price. Even when one initial state is all that matters, the backward pass reaches stage \(k\) before the trajectory from \(x_0\) is known. It must therefore evaluate \(V_k\) at every state reachable from \(x_0\) in \(k\) steps, and in practice, without a separate forward reachability computation, at every state of the model. Only the final minimization, at stage zero, can be restricted to the given \(x_0\). The extra work buys the policy. When the initial state changes, or a disturbance moves the state as in the cart experiment, the stored values and minimizers already answer the new question, provided the new state lies where they were computed and the model still describes the remaining evolution.

The recursion is not greedy minimization of immediate cost. A control with low immediate effort can move the cart into a state from which braking is expensive or impossible. The value function prices that consequence. The hard work has been placed in \(V_{k+1}\): writing a short one-step minimization does not imply that computing its future-cost term was easy.

A complete source example: reuse solved tails 32:21

The recording's shortest-path graph makes the computation visible. The destination is \(h\); its remaining cost is zero. The directed edges and costs are \(a\to b:5\), \(a\to d:8\), \(b\to c:9\), \(c\to d:5\), \(c\to f:3\), \(d\to e:3\), \(e\to f:2\), \(e\to h:8\), \(f\to g:3\), and \(g\to h:2\). These connections contain no directed cycle, so a reverse topological order ensures that every successor value is available when needed.

The optimal path is \(a\to d\to e\to f\to g\to h\), costing \(8+3+2+3+2=18\). A one-step greedy strategy would choose the cheaper first edge \(a\to b\) and ultimately pay \(22\). At node \(e\), a direct edge to the destination costs eight, while a longer route costs seven: distance in graph hops is not the objective.

A directed acyclic graph does not require all routes to have the same number of edges. To express it in the finite-horizon formalism, make \(h\) absorbing with a zero-cost self-loop, set terminal cost zero at \(h\) and infinity elsewhere, and allow at least seven transitions, the maximum length of a simple path in an eight-node acyclic graph. Any shorter arriving path can wait at \(h\). This construction makes the horizon explicit and avoids confusing geometric position with time stage. General cyclic shortest-path problems need further conditions; this acyclic example does not settle them.

Global finite-model optimization can still be unaffordable 43:00

For a deterministic finite model with \(S\) states, at most \(M\) controls per state, and \(N\) stages, direct enumeration of each Bellman minimum requires roughly \(NSM\) candidate evaluations, assuming constant-cost dynamics and cost lookups. Reusing tail values avoids enumerating every length-\(N\) action sequence independently. The result is globally optimal for that finite model when every action comparison is exact.

The saving is clearest against brute force. From one initial state, with at most \(M\) admissible controls per stage, there are up to \(M^N\) control sequences of length \(N\), and evaluating each requires \(N\) transitions. That count grows exponentially with the horizon. The Bellman recursion's \(NSM\) evaluations grow linearly with the horizon and return values for all \(S\) initial states rather than one. With \(S=100\), \(M=3\), and \(N=20\), enumeration from a single initial state examines \(3^{20}=3{,}486{,}784{,}401\) sequences, about \(7.0\times10^{10}\) transitions, while the recursion performs \(20\cdot100\cdot3=6000\) one-step evaluations. Concatenating each immediate decision with an already optimal tail is what removes the exponential dependence on the horizon. It does not remove the dependence on \(S\), which the next paragraph shows can itself be exponential in the state dimension.

A continuous state requires a functional representation or an approximation. If each of \(d\) coordinates is quantized to \(m\) grid values, the state count is \(m^d\). With \(m=101\) and \(d=6\), there are \(1{,}061{,}520{,}150{,}601\) grid points. One eight-byte value at each point consumes about \(8.49\) terabytes before policies, costs, or interpolation metadata. A small finite horizon does not remove this exponential dependence on state dimension.

The exponential growth of the state count with the dimension \(d\), and with it of the number of Bellman minimizations at every stage, is called the curse of dimensionality. It reaches the action count too. If the control has several coordinates and each is quantized separately, \(M\) is exponential in the number of control coordinates, so the per-stage work \(SM\) grows exponentially in the total number of quantized state and control coordinates.

Time discretization and state quantization are distinct. Sampling a continuous differential equation defines a discrete-time transition; representing its continuous state on a grid introduces another approximation. A successor may fall between grid points, requiring interpolation, projection, or another representation. Global optimization on the resulting table is not automatically global optimization of the original continuous-state problem.

Quantizing the controls qualifies the global-minimum claim in the same way. Replacing a continuous admissible set \(\mathcal U_k(x)\) by \(M\) admissible grid actions turns each Bellman minimization into an exact finite comparison, global over the grid. It is not global over the original set. With exact successor values, the gridded minimum can be no lower than the minimum over \(\mathcal U_k(x)\), and it is strictly higher whenever every continuous minimizer falls between grid actions. Refining the action grid narrows this gap at a cost proportional to \(M\).

Time discretization involves a modeling choice of its own, and the simplest choice is the forward Euler map. For a continuous-time model \(\dot x(t)=f_{\mathrm c}(x(t),u(t),t)\) sampled at times \(t_k=k\Delta t\) with interval \(\Delta t\gt0\), replacing the derivative by the difference quotient \((x_{k+1}-x_k)/\Delta t\) gives the difference equation \(x_{k+1}=x_k+\Delta t\,f_{\mathrm c}(x_k,u_k,t_k)\), whose right-hand side serves as \(f_k\). For the cart, \(\dot p=v\) and \(\dot v=u\), so Euler gives \(A=\begin{bmatrix}1&\Delta t\\0&1\end{bmatrix}\) and \(B_{\mathrm E}=(0,\Delta t)^\mathsf T\), whereas the exact held-input map has the same \(A\) and \(B=(\Delta t^2/2,\Delta t)^\mathsf T\). At the normalized \(\Delta t=1\), Euler reproduces the chapter's \(A\) but replaces \(B=(1/2,1)^\mathsf T\) by \(B_{\mathrm E}=(0,1)^\mathsf T\). The missing entry is the position change \(u_k\Delta t^2/2\) that a constant acceleration produces within one interval; Euler omits it because it holds the velocity at its starting value for the whole step. This per-step discrepancy is proportional to \(\Delta t^2\), so it shrinks faster than the interval as the interval is refined. At \(\Delta t=1\) it changes which terminal quantities the last control can influence, as Exercise 7 shows. The chapter's cart model uses the exact map, so for inputs held constant between decisions it carries no time-discretization error.

Constraints can reduce the number of actions in a finite enumeration, which is the computational advantage emphasized in the source. But constructing feasible action sets, enforcing continuous path constraints, or optimizing over a difficult continuous feasible set can add substantial work. Fewer feasible actions does not imply every constrained problem is numerically simpler.

LQR avoids state gridding through an exact closure property: a quadratic value function remains quadratic after one Bellman step under linear dynamics and a suitable unconstrained quadratic cost. Instead of storing a value at every state, store the coefficients of one quadratic form. This is a structural reduction, not a generic cure for the curse of dimensionality.

The linear-quadratic model makes the value function finite-dimensional 57:04

Let \(x_k\in\mathbb R^n\), \(u_k\in\mathbb R^m\), and \(x_{k+1}=A_kx_k+B_ku_k\), where \(A_k\) is \(n\times n\) and \(B_k\) is \(n\times m\). The finite-horizon linear-quadratic regulator minimizes

The symmetric matrices \(Q,H\succeq0\) assign nonnegative state and terminal penalties; \(R\succ0\) assigns strictly positive cost to every nonzero control. The symbols \(\succeq0\) and \(\succ0\) mean positive semidefinite and positive definite, respectively. There are no state or input constraints. The factor \(1/2\) simplifies derivatives and does not change the minimizing policy.

“Regulator” refers to penalizing departure from the origin. The origin may be a chosen desired equilibrium after a coordinate shift, but an arbitrary desired moving trajectory requires a tracking formulation, developed next chapter. A finite terminal quadratic penalty does not enforce \(x_N=0\) exactly. Likewise, existence of a finite-horizon optimum does not by itself prove asymptotic stabilization when the controller is repeated indefinitely.

Suppose \(V_{k+1}(x)=x^\mathsf TP_{k+1}x/2\), with symmetric \(P_{k+1}\succeq0\). This holds at the terminal stage with \(P_N=H\). Insert that form into the Bellman equation and suppress the stage subscript on \(A,B\) for one step:

The cross term has coefficient one because the two scalar transpose terms are equal and cancel the outer factor \(1/2\). Define the control curvature \(S_k=R+B^\mathsf TP_{k+1}B\), an \(m\times m\) matrix, and \(L_k=B^\mathsf TP_{k+1}A\), an \(m\times n\) matrix. For any nonzero vector \(w\), \(w^\mathsf TS_kw=w^\mathsf TRw+(Bw)^\mathsf TP_{k+1}(Bw)>0\). Thus \(S_k\) is invertible and the Bellman objective is strictly convex in \(u\).

Completing the square derives both the gain and the value 1:05:55

The derivative with respect to \(u\) is \(S_ku+L_kx\). Its unique zero is \(u^\star=-S_k^{-1}L_kx\). Define \(K_k=S_k^{-1}L_k\), so our sign convention is \(u^\star=-K_kx\). The lecture denotes the signed gain by \(F_k\), so \(F_k=-K_k\). Either convention is valid; mixing them reverses feedback.

To obtain the value rather than merely the minimizing action, use

The first term on the right is nonnegative and vanishes at the optimizer. The remaining quadratic in \(x\) gives the backward Riccati recursion

An equivalent expression is \(P_k=Q+K_k^\mathsf TRK_k+(A_k-B_kK_k)^\mathsf TP_{k+1}(A_k-B_kK_k)\). Each summand is positive semidefinite, proving \(P_k\succeq0\) and completing the induction. The proof has simultaneously established existence, uniqueness of the current action, the quadratic form of the value, and the linear form of the policy.

The equivalent expression comes from a second route to the same value, which substitutes the minimizing action into the Bellman objective instead of completing the square. With \(u=-K_kx\), the stage cost becomes \(\frac12x^\mathsf T(Q+K_k^\mathsf TRK_k)x\), and the successor is \((A_k-B_kK_k)x\), whose optimal remaining cost is \(\frac12x^\mathsf T(A_k-B_kK_k)^\mathsf TP_{k+1}(A_k-B_kK_k)x\). Because \(-K_kx\) is the minimizer, their sum is \(V_k(x)\). Expanding with \(S_k=R+B_k^\mathsf TP_{k+1}B_k\) and \(L_k=B_k^\mathsf TP_{k+1}A_k\) gives

Because \(S_kK_k=L_k\) and \(S_k\) is symmetric, each of the last three terms equals \(\pm L_k^\mathsf TS_k^{-1}L_k\): the two cross terms contribute \(-2L_k^\mathsf TS_k^{-1}L_k\) and the curvature term \(+L_k^\mathsf TS_k^{-1}L_k\). With \(Q\) added, the result \(Q+A_k^\mathsf TP_{k+1}A_k-L_k^\mathsf TS_k^{-1}L_k\) reproduces the displayed Riccati recursion, since \(L_k^\mathsf TS_k^{-1}L_k=A_k^\mathsf TP_{k+1}B_kS_k^{-1}B_k^\mathsf TP_{k+1}A_k\).

The substituted form reads as the cost of following the linear law: the stage cost under feedback plus the value at the closed-loop successor. It is also the form the implementation below uses. Because it adds positive semidefinite terms rather than subtracting one matrix from another, it avoids the cancellation through which the subtracted form can lose positive semidefiniteness in floating-point arithmetic. Both forms define \(P_k\) through \(V_k(x)=\frac12x^\mathsf TP_kx\), with the same factor \(1/2\) as the terminal condition \(P_N=H\). Writing \(V_{N-1}(x)=x^\mathsf TP_{N-1}x\) at one stage while keeping the factor elsewhere would halve \(P_{N-1}\) relative to the recursion, and the gain \(K_{N-2}\) computed from it would in general be wrong.

This identity also measures the cost of a wrong action. At any state \(x\), taking \(u\) now and acting optimally afterward incurs excess cost

Large \(S_k\) means an action error is expensive in the model. Summing this identity along any feasible unconstrained trajectory makes the intermediate value functions telescope, giving \(J-V_0(x_0)=\frac12\sum_k(u_k+K_kx_k)^\mathsf TS_k(u_k+K_kx_k)\). This supplies a global optimality certificate for the finite LQR problem, not merely a necessary stationary condition.

Boyd's discrete finite-horizon LQR notes give both the dynamic-programming and large least-squares viewpoints. The connection is computationally useful: Riccati recursion exploits temporal structure, while an independently condensed quadratic program can verify a small implementation. Dense comparisons below are checks, not a recommendation to discard sparsity in large systems.

The cart experiment quantifies the value of responding to a shock

For the normalized cart, choose \(N=2\), \(Q=H=I_2\), \(R=[1]\), and \(x_0=(1,0)^\mathsf T\). The last-step matrices are \(S_1=2.25\), \(K_1=(2/9,2/3)\), and \(P_1=\begin{bmatrix}17/9&2/3\\2/3&2\end{bmatrix}\). One more backward step gives \(K_0\approx(0.38926174,0.95302013)\). The nominal controls are \(u_0\approx-0.38926174\), \(u_1\approx0.08053691\); the optimal total cost is approximately \(1.13087248\).

After the first transition, add a velocity impulse of \(0.5\). The observed state becomes \(y\approx(0.80536913,0.11073826)^\mathsf T\). An open-loop controller still applies the saved \(u_1\approx0.08053691\). Feedback instead applies \(-K_1y\approx-0.25279642\), switching from acceleration to braking. Under the assumed remaining dynamics, the respective terminal states are approximately \((0.95637584,0.19127517)\) and \((0.78970917,-0.14205817)\).

The remaining costs, including the stage cost at \(y\), are \(0.80930476\) for the saved action and \(0.68430476\) for feedback. Their difference is exactly \(0.125\) up to floating-point error. The action discrepancy is \(1/3\), so the square-completion prediction is \(\frac12(2.25)(1/3)^2=0.125\). This independent match explains the improvement; it is not merely a favorable simulation plot.

The feedback trajectory does not reach zero in this short horizon because the terminal state is penalized rather than constrained. It also does not recover energy injected by the impulse for free. The comparison holds the disturbed state fixed and asks which remaining action minimizes the stated remaining cost. It does not claim the original undisturbed optimal cost remains achievable.

Executable recursion and an independent finite optimization

The cumulative numerical module implements finite_lqr with dimension, symmetry, and definiteness checks and supports a zero-length horizon. It returns gains using \(u=-Kx\) and value matrices without the common outer \(1/2\); multiplying its reported quadratic cost by \(1/2\) matches this chapter. The following self-contained original check needs NumPy, produces no external effects, and compares Riccati actions against a separately constructed dense quadratic program.

import numpy as np

A = np.array([[1., 1.], [0., 1.]])
B = np.array([[0.5], [1.]])
Q, R, H = np.eye(2), np.ones((1, 1)), np.eye(2)
nsteps, initial = 2, np.array([1., 0.])
P, K = [None]*(nsteps+1), [None]*nsteps
P[-1] = H
for k in range(nsteps-1, -1, -1):
    S = R + B.T @ P[k+1] @ B
    K[k] = np.linalg.solve(S, B.T @ P[k+1] @ A)
    F = A - B @ K[k]
    P[k] = Q + K[k].T @ R @ K[k] + F.T @ P[k+1] @ F

states, actions = [initial], []
for k in range(nsteps):
    actions.append(-K[k] @ states[-1])
    states.append(A @ states[-1] + B @ actions[-1])
actions = np.asarray(actions).ravel()

# Independently express every state as offset + map @ all_controls.
offset, control_map = initial.copy(), np.zeros((2, nsteps))
hessian, linear = np.eye(nsteps), np.zeros(nsteps)
for k in range(nsteps+1):
    W = H if k == nsteps else Q
    hessian += control_map.T @ W @ control_map
    linear += control_map.T @ W @ offset
    if k < nsteps:
        offset = A @ offset
        control_map = A @ control_map
        control_map[:, k] += B[:, 0]
dense_actions = np.linalg.solve(hessian, -linear)
assert np.allclose(actions, dense_actions, atol=1e-12)
cost = 0.5*states[-1] @ H @ states[-1]
cost += 0.5*sum(x @ Q @ x + u*u
                for x, u in zip(states[:-1], actions))
assert abs(cost - 0.5*initial @ P[0] @ initial) < 1e-12

y = states[1] + np.array([0., 0.5])
saved, adapted = actions[1], float((-K[1] @ y)[0])
def tail_cost(u):
    terminal = A @ y + B[:, 0]*u
    return 0.5*(y @ Q @ y + u*u + terminal @ H @ terminal)
gap = tail_cost(saved) - tail_cost(adapted)
assert abs(gap - 0.125) < 1e-12
print(round(cost, 9), round(gap, 9))

The check prints 1.130872483 0.125. The dense solve tests the nominal optimum; the value identity tests the recursion; the impulse comparison tests that feedback is evaluated at the measured state. These are different failure boundaries. A sign error in the gain or confusion about the outer factor \(1/2\) cannot be hidden by a single visually plausible rollout.

Exercises with worked solutions

Boundaries of the result

The backward recursion is exact for the specified deterministic finite-horizon model. Its LQR specialization assumes linear dynamics, quadratic costs with the stated definiteness, full state availability, and no input or state constraints. Clipping an LQR action changes the policy and generally breaks its optimality certificate. An estimated state introduces an estimation problem. A gridded value approximation introduces representation error. None of these changes can be dismissed merely because the nominal Riccati equations were solved accurately.

The next step is to retain LQR's useful local structure when the plant is nonlinear or the desired motion is a trajectory rather than a fixed origin. That requires distinguishing a nominal trajectory, a state deviation, a feedforward control correction, and a feedback gain. It also reintroduces the local-model and step-acceptance issues from sequential convexification.

Primary sources

• AA203 lecture 7 recording and official slides.
• Dimitri Bertsekas, basic dynamic programming and state augmentation, MIT 6.231, 2015.
• Stephen Boyd, discrete-time finite-horizon LQR, Stanford EE363.