Bellman solvers and Hamilton–Jacobi equations

A small numerical residual can answer two very different questions. In a discounted finite Markov model, a Bellman residual gives a precise bound on value error. In a continuous control problem, a small residual at sampled points of a partial differential equation does not by itself certify a globally safe policy. The mathematical object, the approximation, and the guarantee must be kept together.

This chapter first completes the stationary stochastic-control algorithms introduced in chapter 9. It then changes the model to continuous time, derives the Hamilton–Jacobi–Bellman equation, and explains how an adversary changes it to a Hamilton–Jacobi–Isaacs equation. The cart remains the running control system; a scalar subsystem permits an exact continuous-time LQR check. Reachability is introduced at the end through the order of control and disturbance choices.

Grounded in AA203 lecture 10. The first 23 minutes finish value and policy iteration from the preceding deck. The recording ends after defining avoidance and goal-reaching sets. Their computational set/tube treatment is developed in the next chapter, following the next recording. Contraction proofs, explicit strategy examples, verification arguments, and numerical fixtures below are teaching extensions. The official slides add the place of the Hamilton–Jacobi equations in the course roadmap, the open-loop information pattern, the game's dynamic-programming principle, the feedback laws read from the Isaacs and Bellman Hamiltonians, the infinite-horizon limit of the Riccati equation, the applications of differential games, and the input–output view of backward reachable sets; their remaining reachability pages are delivered in the next recording and treated in the next chapter, and where their notation differs, the text translates it.

Value iteration contracts the error of a finite discounted model 7:23

Both algorithms of this section and the next assume a known model: the transition probabilities and the reward are given, and only the optimal value and policy are unknown. When the model must instead be learned from interaction, a model-based method estimates \(T\) and \(\rho\) from data and then applies these algorithms to the estimate, whereas a model-free method estimates values or a policy directly. Chapter 9's discussion of state-action values develops that distinction, and the course's final part returns to both routes.

Use a finite state set \(\mathcal X\), finite nonempty action sets \(\mathcal U(x)\), bounded reward \(\rho(x,u)\), transition probabilities \(T(x'\mid x,u)\), and discount \(0\leq\gamma<1\). A value vector \(W\) has one real component per state. Define the Bellman operator \(\mathcal B\) by

Value iteration starts from a finite vector \(W^{(0)}\) and computes \(W^{(j+1)}=\mathcal BW^{(j)}\). Superscript \(j\) is a numerical iteration index, not physical time. Starting from zero gives the optimal expected reward for successively longer finite horizons with zero terminal reward. Stationarity allows the same operator to be reused each time.

The convergence mechanism is a contraction. Define \(\|W\|_\infty=\max_x|W(x)|\). For a fixed action, changing \(W\) to \(Z\) changes the expected continuation by at most \(\gamma\|W-Z\|_\infty\), since transition probabilities are nonnegative and sum to one. Also, for any finite vectors \(a,b\), \(|\max_i a_i-\max_i b_i|\leq\max_i|a_i-b_i|\). Combining these facts gives

The finite-dimensional vector space with this norm is complete, so the contraction has a unique fixed point \(W^\star\), and every initialization converges to it. More concretely, repeated application yields \(\|W^{(j)}-W^\star\|_\infty\leq\gamma^j\|W^{(0)}-W^\star\|_\infty\). A good initial guess reduces the initial error; it does not change this worst-case contraction factor.

A stopping rule should refer to the returned vector. For any candidate \(W\), let \(r=\|\mathcal BW-W\|_\infty\) be its Bellman residual. By the triangle inequality and contraction, \(\|W-W^\star\|_\infty\leq r+\gamma\|W-W^\star\|_\infty\), hence

At discount \(0.99\), residual \(10^{-4}\) guarantees value error at most \(0.01\), not \(10^{-4}\). A fixed iteration budget is a useful engineering limit, but it should not be reported as a convergence certificate unless the residual was also checked. The bound concerns this supplied finite model; transition-model error and state quantization are separate.

A greedy policy from an approximate value has another bound. If \(\pi\) is greedy with respect to \(W\), then its fixed-policy Bellman operator satisfies \(\mathcal B_\pi W=\mathcal BW\). Writing \(\epsilon=\|W-W^\star\|_\infty\), subtracting and adding the two approximate-value terms gives \(\|W^\star-W^\pi\|_\infty\leq2\gamma\epsilon+\gamma\|W^\star-W^\pi\|_\infty\). Thus the policy loss is at most \(2\gamma\epsilon/(1-\gamma)\). This potentially loose bound illustrates why value accuracy and policy performance should not be conflated.

The same iteration can be run on state-action values. Let \(\mathcal Q\) be a vector with one real component per pair \((x,u)\) with \(u\in\mathcal U(x)\), and define

Q-value iteration computes \(\mathcal Q^{(j+1)}=\mathcal B_{\mathcal Q}\mathcal Q^{(j)}\), for example from \(\mathcal Q^{(0)}=0\). The two facts behind the earlier contraction proof apply unchanged, so \(\|\mathcal B_{\mathcal Q}\mathcal Q-\mathcal B_{\mathcal Q}\mathcal Z\|_\infty\le\gamma\|\mathcal Q-\mathcal Z\|_\infty\) in the maximum norm over pairs, and the unique fixed point is the optimal state-action value \(\mathcal Q^\star\) of chapter 9. The residual bound \(r/(1-\gamma)\) also carries over. The two iterations are one computation stored in two forms: if \(\mathcal Q^{(j)}(x,u)=\rho(x,u)+\gamma\sum_{x'}T(x'\mid x,u)W^{(j)}(x')\), then \(\max_u\mathcal Q^{(j)}(x,u)=W^{(j+1)}(x)\), and one more application of \(\mathcal B_{\mathcal Q}\) gives the pair values built on \(W^{(j+1)}\). Storing a number for every pair costs a factor of the action count in memory. The return is that a greedy action is read directly as \(\arg\max_u\mathcal Q^{(j)}(x,u)\), without evaluating the transition model at decision time, and that this update is the template for the model-free methods of the course's final part. In a one-state model with two actions of rewards \(1\) and \(0.5\) and \(\gamma=0.9\), the iterates from zero are \((1,0.5)\), \((1.9,1.4)\), and so on, converging to \((10,9.5)\). From the first sweep on, the action gap is exactly \(0.5\), because both actions share the same successor, so the greedy action is correct after one sweep, while the values themselves approach their limits only at the contraction rate \(0.9\).

Policy iteration evaluates a policy before replacing its actions 12:32

For a stationary policy \(\pi\), compute its value by solving \((I-\gamma T_\pi)W^\pi=\rho_\pi\). Then choose \(\pi'(x)\) to maximize \(\rho(x,u)+\gamma\sum_{x'}T(x'\mid x,u)W^\pi(x')\). The new action is greedy with respect to the old policy's exact value, not with respect to the unknown optimum.

Why must this improve the policy? Since the old action is an available candidate, \(\mathcal B_{\pi'}W^\pi\geq\mathcal B_\pi W^\pi=W^\pi\), componentwise. The operator \(\mathcal B_{\pi'}\) is monotone: a larger successor-value vector cannot decrease expected value. Repeatedly applying it therefore produces a nondecreasing sequence beginning at \(W^\pi\), and its contraction limit is \(W^{\pi'}\). Hence \(W^{\pi'}\geq W^\pi\).

If a policy is suboptimal, improvement is strict in at least one state, not necessarily every state. A state already at an absorbing zero-reward destination can remain unchanged while another state's value improves. With finitely many deterministic policies, exact evaluation and a tie rule that retains the current action when it is already maximizing prevent pointless cycling. The algorithm then terminates at an optimal policy after finitely many strict improvements. Approximate evaluation, numerical ties, or continuous actions require additional analysis.

The computational tradeoff is concrete. Value iteration makes relatively cheap Bellman sweeps and may need many of them when \(\gamma\) is close to one. Policy iteration solves a policy-evaluation system at each outer iteration, potentially making fewer but more expensive improvements. Sparse linear algebra and modified policy iteration can change that comparison. The exact algorithms form a reference against which later learning-based approximations can be judged. Bertsekas's discounted dynamic-programming lectures provide primary mathematical context for these algorithms.

In continuous time, the Bellman equation compares infinitesimal changes 23:07

Return to minimizing cost. Let \(x(t)\in\mathbb R^n\) satisfy \(\dot x=f(x,u,t)\), where \(u(t)\in\mathcal U\), and fix final time \(T\). Define

The stage integrand \(\ell\) is cost per unit time, while the terminal function \(\phi\) is a cost. Assume dynamics regular enough to give a well-defined trajectory for admissible controls. To derive a classical differential equation, temporarily assume \(V\) is continuously differentiable; this smoothness assumption will be revisited.

This change of time model stays on the closed-loop branch of the course but replaces its main tool. Chapters 7 and 8 computed feedback policies through the discrete-time Bellman recursion, which produced discrete LQR, iterative LQR, and differential dynamic programming; chapter 9 and the first two sections here extended the recursion to stochastic and stationary models. In continuous time the same principle of optimality produces a partial differential equation for the value: the Hamilton–Jacobi–Bellman equation when one decision maker optimizes, and the Hamilton–Jacobi–Isaacs equation when a second player opposes it. The first application below solves continuous-time LQR through the Bellman equation; the second uses the Isaacs equation to define reachability analysis. Continuous time repays the extra machinery when a problem is posed most naturally by differential equations, and because the resulting feedback laws are sometimes simpler to state and implement than their discrete-time counterparts.

Both equations are nonlinear in the value gradient, written \(p\) for \(\nabla_xV\). In the Bellman case the Hamiltonian \(\min_u[\ell+p^\mathsf Tf]\) is a pointwise minimum of functions that are affine in \(p\), so it is concave in \(p\) and in general not linear; continuous LQR makes this explicit, because minimizing over \(u\) leaves a term quadratic in the gradient. In the Isaacs case the max-min of affine functions is in general neither concave nor convex in \(p\): the velocity-channel Hamiltonian \((U-D)|p|\) derived below is convex when \(U\gt D\) and concave when \(U\lt D\). Neither equation can therefore be solved by the superposition methods available for linear partial differential equations.

The objects correspond term by term to the discrete-time problem \(x_{k+1}=f_k(x_k,u_k)\) with cost \(\phi_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k)\). The difference equation becomes a differential equation, the sum of stage costs becomes the integral of a cost rate, and the terminal cost keeps its role; the initial and final times are fixed. The official course slides write the continuous problem as \(\dot x=f(x,u,t)\) with cost \(h(x(t_f),t_f)+\int_{t_0}^{t_f}g(x(\tau),u(\tau),\tau)\,d\tau\), and they use \(J\) both for this cost and for the optimal cost-to-go. This chapter writes \(\phi\) for the terminal cost, \(\ell\) for the cost rate, \(T\) for the final time, and \(V\) for the minimizing value, and it reserves \(J\) and \(g\) for the game outcome and its running term in the next section. The correspondence is \(h=\phi\), \(g=\ell\), \(t_f=T\), and \(J(x,t)=V(x,t)\) in the one-player sections. Because the final time is fixed, a terminal cost that also depends on \(t_f\) is simply a fixed function of the terminal state.

An adversary requires a strategy and an information pattern 26:40

Now add a second input \(d(t)\in\mathcal D\), with \(\dot x=f(x,u,d)\). In the source's game convention, the controller maximizes an outcome \(J\) and the disturbance minimizes it. This is a deterministic zero-sum differential game: the two players have opposing objectives and jointly influence differential dynamics. It is not a stochastic expectation over a disturbance distribution.

The second input models whatever interferes with the controller's objective: nature acting through wind gusts or road friction, or another agent whose intentions are unknown. Treating it as an opponent that may choose any admissible signal with values in \(\mathcal D\) yields a guarantee against every such signal, rather than an average over a distribution that might itself be misspecified. The price is conservatism, because the policy must also cope with disturbances that a careless but not hostile agent would never produce. The state \(x\) is the joint state of the game. In Isaacs's homicidal-chauffeur problem, a pedestrian who can step in any direction at a modest speed tries to avoid a faster car that cannot turn more sharply than its minimum turning radius; the pedestrian is the controller and the car is the disturbance. The joint state collects the pedestrian's planar position, the car's planar position, and the car's heading, five coordinates in all, and both players' inputs drive it through the single function \(f\). Whether the pedestrian can avoid capture depends on that whole configuration, not on separation alone: a pedestrian beside the car may be harder to reach than one at the same distance directly ahead, because the car must first turn.

The source uses terminal time zero and starting time \(t<0\). This is a coordinate convention; duration is \(-t\). Let the outcome be \(\phi(x(0))+\int_t^0 g(x(s),u(s),d(s))\,ds\), where \(g\) is a running reward-like quantity in this section.

An open-loop disturbance trajectory is a function of time fixed in advance. A nonanticipative disturbance strategy \(\Gamma\) is different: it maps a controller input history to a disturbance history, without using future control values. If two controller signals agree up to a time, their resulting disturbance signals must also agree up to that time. The convention considered here permits reacting to the current control. Thus it grants an instantaneous response advantage without granting knowledge of future controls.

The resulting lower outcome for the maximizing controller can be written schematically as

The outer optimization is over response rules, not over a single instantaneous disturbance value. That distinction is the key to understanding the source's change in the apparent order of optimization when deriving the local Hamiltonian. It is not permission to commute a minimum and maximum arbitrarily.

The phrase “open-loop” can attach to either player, and the direction matters. The open-loop disturbance trajectory described above commits the disturbance first, which favors the controller. The open-loop information pattern for this game commits the controller instead: player one declares its entire signal \(u(\cdot)\) on \([t,0]\), and the disturbance, having seen all of it, responds optimally. The resulting outcome for the maximizing controller is

Equivalently, \(J_{\rm ol}\) is the strategy value obtained when the disturbance may use any response map, including maps that read future control values: selecting for each declared signal a disturbance that attains, or comes within any \(\epsilon\gt0\) of, the inner infimum shows that such maps achieve exactly \(J_{\rm ol}\). Nonanticipative strategies form a subset of these maps, so the infimum over them can only be larger. Directly, for every nonanticipative \(\Gamma\) and every \(u(\cdot)\), the outcome under \(\Gamma[u]\) is at least the infimum over all disturbance signals; taking the supremum over \(u(\cdot)\) and then the infimum over \(\Gamma\) gives \(J(x,t)\geq J_{\rm ol}(x,t)\). The open-loop pattern is therefore conservative for the controller. It credits the disturbance with knowledge of future controls that no physical gust or rival vehicle possesses, and it denies the controller any response to what the disturbance has already done. It is computationally cheaper because, for a fixed declared signal, the inner problem is an ordinary open-loop optimal-control problem for the disturbance, and the outer problem searches over one signal; the trajectory-optimization methods of the earlier chapters apply without a value function over the state space. Exercise 7 shows that the gap between the two values can be strict.

An original finite one-step game makes the distinction exact. Let \(u,d\in\{-1,1\}\) and reward be \(ud\). A minimizing response map can choose \(\Gamma(u)=-u\), giving reward minus one for both controller actions. Therefore \(\min_\Gamma\max_u u\Gamma(u)=-1=\max_u\min_d ud\). In contrast, if the disturbance must choose a fixed \(d\) before the controller responds, \(\min_d\max_u ud=1\). Changing a response map into a fixed move changes the information and the value.

For finite actions, the equality between optimization over arbitrary response maps and the action-level max-min follows by selecting a minimizing response separately for every controller action. In continuous-time differential games, admissibility, measurability, nonanticipativity, and the dynamic-programming principle require substantially more care. Bansal, Chen, Herbert, and Tomlin's reachability tutorial gives the primary formal setup used by the source. The one-step calculation explains the order; it does not replace the continuous-time theorem.

The Hamilton–Jacobi–Isaacs equation preserves the response order 45:53

The derivation rests on a dynamic-programming principle for the game. For a short interval \(0\lt\Delta t\le-t\),

where the controls and the strategy are now restricted to \([t,t+\Delta t]\) and the second term is the value of the game that remains. As in the discrete principle of optimality, there is no need to compare every complete signal. Each candidate initial segment from the current state leads to some intermediate state; the remainder of the game from that state is taken as already solved, with value \(J\); and only the concatenations of an initial segment with that optimal remainder are compared. What changes in the game is the object being concatenated: the proof must show that strategies for the first interval and for the remaining game can be assembled into, and recovered from, a single nonanticipative strategy. Establishing the principle at that level is the technical core of the theory. Bansal and colleagues, and Chen and Tomlin in Hamilton–Jacobi Reachability: Some Recent Theoretical Advances and Applications in Unmanned Airspace Management (Annual Review of Control, Robotics, and Autonomous Systems, 2018), survey the formal treatment; here the principle is assumed.

Split the game outcome at \(t+\Delta t\) into immediate outcome and the value of the remaining game. For smooth \(J\), approximate the immediate integral by \(g\Delta t\), the successor by \(x+f\Delta t\), and its value by \(J+\Delta t(J_t+\nabla_xJ^\mathsf Tf)+o(\Delta t)\). Under the stated strategy convention, the local Hamiltonian is

where \(p=\nabla_xJ(x,t)\) is a gradient vector, not cart position in this formula. The controller chooses the action whose worst immediate response is best. The expanded dynamic-programming relation is \(J=J+\Delta t[J_t+\mathcal H(x,\nabla_xJ)]+o(\Delta t)\). Cancel the common value, divide by the positive time increment, and let it approach zero. The Hamilton–Jacobi–Isaacs equation is \(J_t+\mathcal H(x,\nabla_xJ)=0\), with terminal condition \(J(x,0)=\phi(x)\). Passing this limit through the optimization requires the regularity behind the dynamic-programming theorem; the expansion alone does not establish it.

The step from signals to the vectors inside the Hamiltonian deserves its own statement. Over \([t,t+\Delta t]\), approximate the control and disturbance signals by constants \(u\in\mathcal U\) and \(d\in\mathcal D\). The optimization over functions on the short interval then becomes an optimization over vectors, and a nonanticipative strategy, which may react to the current control, acts as a response map from \(\mathcal U\) to \(\mathcal D\). Choosing the best response separately for each \(u\), exactly as in the finite game of the previous section, turns \(\inf_\Gamma\sup_u\) into \(\max_u\min_d\). The disturbance keeps the advantage that matters: it moves second at each instant and sees the control it answers. The apparent reversal of the order is the instantaneous form of that advantage, not an exchange of a minimum and a maximum. That the error from freezing the signals vanishes faster than \(\Delta t\), uniformly enough to pass to the limit, is again part of the theorem rather than of this expansion.

If the opposite instantaneous move order is intended, the Hamiltonian generally changes to a min-max. When both orders agree for every relevant state and gradient, the equality is called an Isaacs condition. The finite product game above violates it. A derivation must specify which game's value is being computed before using a PDE solver.

For an original one-dimensional velocity channel \(\dot v=u+d\), take \(u\in[-U,U]\), \(d\in[-D,D]\), positive bounds \(U,D\), zero running outcome, and scalar value gradient \(p\). The maximizing controller contributes \(U|p|\), while the minimizing disturbance contributes \(-D|p|\). Thus \(\mathcal H=(U-D)|p|\). The two optimizations commute in this separable example, but that is a property of its additive dynamics and independent action sets, not a general rule.

With a cost-minimizing controller and cost-maximizing disturbance, the order and signs change: the corresponding local term is \(\min_u\max_d p(u+d)=(D-U)|p|\). Reusing the earlier Hamiltonian while changing the interpretation of the target would reverse the intended behavior. This is why a source map for a reachability calculation must record the objective convention, not just the PDE's name.

A solved value also supplies the controller. At a state \(x\) and time \(t\) where \(J\) is differentiable, any action

attains the Hamiltonian, and the disturbance's worst instantaneous response to a chosen \(u\) is any \(d^\star(x,t,u)\) attaining the inner minimum. The feedback needs only the gradient of the stored value at the current state. In the velocity channel with the maximizing controller, the bracket is \(p(u+d)\), so \(u^\star=U\operatorname{sign}p\) and \(d^\star=-D\operatorname{sign}p\) whenever \(p\neq0\): both players push at full authority along the gradient, a bang-bang law. At \(p=0\) every admissible \(u\) attains the maximum, so the formula selects no particular action, and where \(J\) has a corner the gradient itself is undefined. An implementation therefore needs a selection rule at such states, and the formula alone does not certify that the selected feedback produces a well-defined closed-loop trajectory.

The Hamilton-Jacobi-Bellman equation prices motion through the value gradient 58:20

For a short positive interval \(\Delta t\), the dynamic-programming principle splits the cost into the immediate integral and \(V(x(t+\Delta t),t+\Delta t)\). Holding a candidate input approximately constant over that interval gives \(x(t+\Delta t)=x+f(x,u,t)\Delta t+o(\Delta t)\). Taylor expansion then gives

The short running cost is \(\ell(x,u,t)\Delta t+o(\Delta t)\). Cancel \(V(x,t)\), divide by \(\Delta t\), and take the limit to obtain the Hamilton–Jacobi–Bellman equation

The expression inside the minimum is the Hamiltonian, with \(\nabla_xV\) playing the role of a state-dependent costate. The first term is immediate cost rate. The second is the directional rate of change of future value as the dynamics move the state. The minimizer balances both, just as the discrete Bellman equation balances current cost and successor value.

The minus sign in \(-V_t=\min_u[\ell+\nabla V^\mathsf Tf]\) follows algebraically from the expansion. It is not justified by a universal claim that \(V_t\leq0\) at a fixed state. Changing remaining time can alter terminal penalties and feasible behavior. Along an optimal smooth trajectory, the total derivative satisfies \(dV(x(t),t)/dt=-\ell(x(t),u^\star(t),t)\). That statement includes state motion and is different from a partial derivative holding \(x\) fixed.

The Bellman equation is the Isaacs equation with the opponent removed and the objective reversed. Let \(\mathcal D\) contain a single point, so that the inner minimization is vacuous; shift time so that the game's final time zero becomes \(T\); and pose the cost problem as a reward problem with running reward \(-\ell\) and terminal reward \(-\phi\). The game value is then \(J=-V\). Substituting into \(J_t+\max_u[-\ell+\nabla_xJ^\mathsf Tf]=0\) gives \(-V_t-\min_u[\ell+\nabla_xV^\mathsf Tf]=0\), which is the Bellman equation, and \(J(x,T)=-\phi(x)\) is the terminal condition \(V(x,T)=\phi(x)\). Written with the game's final time zero instead of \(T\), the same condition reads \(V(x,0)=\phi(x)\); the two forms differ only in where the clock ends.

Given \(V\), the optimal feedback is

the continuous-time counterpart of saving a minimizing action for every state and stage in the discrete recursion. The continuous-time LQR gain derived below is this formula evaluated for a quadratic value. The qualifications stated for the game carry over: the formula needs \(\nabla_xV\) to exist at the current state, and a rule for choosing among minimizers when there are several.

A smooth HJB solution needs a verification argument

Solving a differential equation is not by itself a proof that an arbitrary candidate equals the value. A simple verification argument supplies the missing link. Suppose a smooth candidate \(V\) satisfies the terminal condition and HJB, and suppose an admissible minimizing feedback exists and its trajectory is well defined.

For every admissible control \(u\), HJB implies \(V_t+\nabla V^\mathsf Tf+\ell\geq0\), because the chosen action cannot be smaller than the minimum. Along its trajectory, the chain rule gives \(dV/dt+\ell\geq0\). Integrating from the starting time to \(T\) and using \(V(x(T),T)=\phi(x(T))\) yields

Thus the candidate lower-bounds every achievable cost. Along a feedback attaining the Hamiltonian minimum, equality holds in the differential inequality and therefore in the accumulated cost. The candidate is attainable and is exactly the value. The argument makes the roles of the terminal condition, smoothness, admissibility, and attaining control explicit.

General value functions need not be smooth. As an original example, \(\dot x=u,\ |u|\leq1\), zero running cost, terminal penalty \(|x(T)|\), and duration \(\tau=T-t\) give value \(V(x,t)=\max(|x|-\tau,0)\). The controller can move toward zero at unit speed and stop there. The value has corners at \(x=\pm\tau\). Classical derivatives fail there even though the control problem is well posed.

Viscosity-solution theory interprets Hamilton–Jacobi equations without requiring classical differentiability everywhere. It is the appropriate mathematical framework for many reachability values. This chapter's Taylor calculation and verification proof explain the smooth case; a numerical grid or neural approximation needs its own consistency, error, and solution-selection analysis before inheriting a general guarantee.

Continuous LQR reduces the PDE to a matrix ODE 1:06:40

Let \(\dot x=A(t)x+B(t)u\), with cost \(\frac12x(T)^\mathsf THx(T)+\frac12\int_t^T[x^\mathsf TQ(s)x+u^\mathsf TR(s)u]\,ds\). Assume symmetric \(Q(t),H\succeq0\), \(R(t)\succ0\), suitable continuity, and unconstrained states and inputs. Seek \(V(x,t)=\frac12x^\mathsf TP(t)x\), with symmetric matrix \(P(t)\).

The derivatives are \(V_t=\frac12x^\mathsf T\dot P x\) and \(\nabla_xV=Px\). Substitution into HJB gives a quadratic minimization in \(u\), whose derivative is \(Ru+B^\mathsf TPx\). Therefore \(u^\star=-R^{-1}B^\mathsf TPx\). The minimized input-dependent terms equal \(-\frac12x^\mathsf TPBR^{-1}B^\mathsf TPx\).

The scalar \(x^\mathsf TPAx\) equals \(\frac12x^\mathsf T(PA+A^\mathsf TP)x\). Matching the quadratic coefficients for every \(x\) gives the differential Riccati equation

The condition is terminal, so integrate backward. Reversing time with \(\tau=T-t\) changes the derivative sign; a numerical implementation should make that transformation explicit or ask its ODE solver to integrate from \(T\) to the starting time. Accidentally treating \(P(T)\) as an initial value at time zero changes the controller.

For a normalized scalar integrator \(\dot x=u\), choose \(Q=R=1\), \(H=0\), and \(T=1\). Then \(-\dot P=1-P^2,\ P(1)=0\), solved by \(P(t)=\tanh(1-t)\). From \(x(0)=1\), the feedback \(u=-P(t)x\) gives

The endpoint is \(x(1)=1/\cosh1\), not zero, because the terminal cost is zero. As the deadline approaches, the gain approaches zero: spending input then buys little reduction in the remaining running state cost. This limiting behavior is consistent with the objective and guards against accidentally treating the regulator as a hard terminal-transfer problem.

The continuous-time equation also connects with a finely sampled discrete Riccati recursion. For a short interval \(h\), use \(A_h=I+hA+O(h^2)\), \(B_h=hB+O(h^2)\), \(Q_h=hQ\), and \(R_h=hR\). Substituting into the discrete recursion and retaining first-order terms gives \[P(t)=P(t+h)+h[Q+A^\mathsf TP+PA-PBR^{-1}B^\mathsf TP]+O(h^2)\], which is the same differential equation. Scaling the input penalty by \(h\) is essential in this comparison.

The official course slides write this Riccati matrix as \(V(t)\), with value \(\frac12x^\mathsf TV(t)x\). This chapter keeps \(V\) for the value function itself, so the slides' \(V(t)\) is the \(P(t)\) above, and their feedback \(-R^{-1}B^\mathsf TV(t)x\) is the \(u^\star=-R^{-1}B^\mathsf TPx\) derived here.

As the horizon grows, the Riccati solution can settle to a constant, in analogy with the stationary limit of the discrete recursion. Restrict to time-invariant \(A,B,Q,R\); with time-varying data there is in general no single limiting matrix. If, as the remaining duration \(T-t\) grows without bound, the solution converges to a constant \(\bar P\), its derivative vanishes and \(\bar P\) satisfies the continuous algebraic Riccati equation

The standard sufficient conditions are that \((A,B)\) be stabilizable and \((A,Q^{1/2})\) detectable. Under them, the backward solution from any \(H\succeq0\) converges to the unique positive-semidefinite solution of this equation, and the constant feedback \(u=-R^{-1}B^\mathsf T\bar Px\) makes the closed loop asymptotically stable. The scalar regulator shows the limit exactly. Its solution \(P(t)=\tanh(T-t)\) tends to one, and the algebraic equation \(1-\bar P^2=0\) has the positive-semidefinite root \(\bar P=1\); the other root, \(-1\), would give the destabilizing feedback \(u=x\). The approach is fast: \(1-\tanh s=2/(e^{2s}+1)\), which is below \(0.005\) at \(s=3\), so the time-varying gain is within one half of one percent of its limit whenever more than three time units remain. With \(u=-x\) from \(x(0)=x_0\), the trajectory is \(x_0e^{-t}\), and the infinite-horizon cost is \(\frac12\int_0^\infty2x_0^2e^{-2t}\,dt=\frac12x_0^2=\frac12\bar Px_0^2\), as the limiting quadratic value requires. Exercise 8 shows what fails without detectability.

Reachability begins by specifying who must defeat whom 1:11:03

Differential games arise wherever a guarantee must hold against an opposing input. Pursuit–evasion games, of which the homicidal chauffeur is the classical example, ask from which joint configurations an evader can avoid capture. Problems in economics in which competing decision makers steer a shared dynamic state lead to the same mathematical structure. Reachability analysis, developed here and in the next chapter, uses the game to classify initial states by what the controller can guarantee about a target set.

A reachability computation takes two inputs: the system model, including the admissible control and disturbance sets, and a target set of states. It returns two outputs: a set of initial states and a control policy. The set is called a backward reachable set, because it is computed backward from a condition on the state at the final time. Its meaning depends on whether the target is an unsafe region, such as an obstacle, or a goal region. For an obstacle, the backward reachable set contains the states that lead to danger: from them, some disturbance strategy forces the state into the target whatever the controller does. A planner should keep the system outside this set, and the returned policy is what makes that choice safe; starting outside the set does not protect a controller that ignores the policy. For a goal, the backward reachable set contains the states that lead to the goal: from them, the controller can reach the target against every disturbance strategy, provided it uses the returned policy, so a planner should start the system inside the set. In both cases the set can differ substantially from the target itself, as the example below shows.

Let \(\mathcal T\) be a target subset of state space and let \(\zeta(0;x,t,u,\Gamma[u])\) denote the terminal state from initial state \(x\) at time \(t<0\). If the target is dangerous, the source's backward set contains states from which some admissible disturbance strategy forces entry at the terminal time regardless of the controller signal:

If the target is desired, the source instead asks whether, against every admissible disturbance strategy, there is a controller signal reaching it:

These statements use the source's strategy convention. They should not be casually rewritten as a single open-loop control that works against all disturbances. The logical order describes a different information and capability requirement. Practical feedback synthesis must use the same game convention as the value calculation.

The velocity channel gives both sets exactly. Take \(\dot v=u+d\) with \(|u|\le U\), \(|d|\le D\), and \(U\gt D\); let the target be \(\mathcal T=\{v:|v|\le a\}\) with \(a\gt0\); and write \(\tau=-t\) for the duration. Pushing at full authority away from zero or toward it, the controller changes \(|v|\) in the chosen direction at a net rate of at least \(U-D\) against any admissible disturbance. A disturbance that cancels the current control as far as its authority allows, \(d=-u\) when \(|u|\le D\) and \(d=-D\operatorname{sign}u\) otherwise, reacts only to the current control and gives \(|\dot v|=\max(|u|-D,0)\le U-D\) for every admissible control, so it holds the rate of change of \(|v|\) to at most \(U-D\) in either direction. Comparing \(|v|\) with \(a\) at the final time then gives

The goal set grows linearly with the horizon, because more time lets the stronger controller enter the target from farther away. The avoidance set shrinks and is empty once \(\tau\gt a/(U-D)\): with that much time, the controller can leave the target from every initial state, even its centre. A backward reachable set is therefore not a fixed enlargement of its target. It extends beyond the target when the dynamics or the opposing input can carry outside states into it, as when the disturbance dominates or when momentum prevents an approaching vehicle from stopping in time. With \(U=1\), \(D=0.5\), \(a=1\), and \(\tau=1\), the avoidance set is \(\{|v|\le0.5\}\) and the goal set is \(\{|v|\le1.5\}\).

A small finite counterexample shows why the order matters. Let the terminal state be \(u+d\), with \(u,d\in\{-1,1\}\), and desired target \(\{0\}\). For every fixed disturbance there exists a matching control \(u=-d\), so \(\forall d\,\exists u\) is true. No one fixed control cancels both disturbances, so \(\exists u\,\forall d\) is false. A terminal target statement also says nothing by itself about avoiding danger earlier in the trajectory. The next chapter turns these distinctions into reachable sets and tubes.

Executable residual and Riccati checks

This original example needs NumPy and SciPy. It uses the cumulative value-iteration implementation on a one-state model with reward one, then integrates a scalar Riccati equation backward and compares both its value and realized trajectory cost against analytic formulas. The finite one-state model deliberately isolates contraction and stopping accuracy from policy search.

import itertools
import numpy as np
from scipy.integrate import solve_ivp
from numerical_lab import discounted_value_iteration

# The cumulative solver minimizes cost, so cost -1 represents reward +1.
transition = np.ones((1, 1, 1))
cost, policy, bound = discounted_value_iteration(
    transition, np.array([[-1.]]), .9, tolerance=1e-9)
reward_value = -cost[0]
assert abs(reward_value-10) <= bound + 1e-12
assert bound <= 1e-9

solution = solve_ivp(lambda t, p: [p[0]**2-1], (1., 0.), [0.],
                     rtol=1e-11, atol=1e-13, dense_output=True)
assert solution.success
time = np.linspace(0, 1, 10001)
assert np.max(abs(solution.sol(time)[0]-np.tanh(1-time))) < 1e-9
x = np.cosh(1-time)/np.cosh(1)
u = -np.sinh(1-time)/np.cosh(1)
quadrature = np.trapezoid(.5*(x*x+u*u), time)
assert abs(quadrature-.5*np.tanh(1)) < 2e-9

actions = [-1, 1]
max_min = max(min(u*d for d in actions) for u in actions)
min_max = min(max(u*d for u in actions) for d in actions)
reaction_maps = list(itertools.product(actions, repeat=2))
strategy_value = min(max(u*response[i] for i, u in enumerate(actions))
                     for response in reaction_maps)
assert strategy_value == max_min == -1 and min_max == 1
print(round(reward_value, 8), round(quadrature, 9), strategy_value, min_max)

The output is approximately 10.0 0.380797079 -1 1. The recorded Riccati maximum error is about \(3.24\times10^{-11}\), and the independent trajectory quadrature differs from the exact cost by about \(1.27\times10^{-9}\). These checks verify a scalar ODE, an analytic trajectory, a finite Bellman residual, and a finite information-pattern distinction. They do not certify a general multidimensional HJI implementation.

A second original check enumerates the two-stage game of Exercise 7, which separates the open-loop and nonanticipative information patterns. The outcome \(u_2d_1\) does not involve the second disturbance action, so a nonanticipative disturbance rule reduces to one of the four maps \(\gamma\) from \(u_1\) to \(d_1\), while the open-loop disturbance chooses \(d_1\) after seeing the whole declared pair \((u_1,u_2)\). The block needs only the standard library.

import itertools

actions = [-1, 1]
pairs = list(itertools.product(actions, repeat=2))
# Open loop: the disturbance sees (u1, u2) before choosing d1.
open_loop = max(min(u2*d1 for d1 in actions) for u1, u2 in pairs)
# Nonanticipative: d1 = gamma(u1) for one of the four maps gamma.
maps = [dict(zip(actions, image)) for image in pairs]
strategy = min(max(u2*gamma[u1] for u1, u2 in pairs) for gamma in maps)
assert open_loop == -1 and strategy == 1
print(open_loop, strategy)

The output is -1 1. The enumeration covers every deterministic rule of each pattern for this finite game; it says nothing about randomized strategies, which this chapter does not consider.

Exercises with worked solutions

Different numerical guarantees attach to different mathematical problems

The contraction bounds apply to a finite discounted model with exact transition probabilities and Bellman evaluations. The smooth HJB verification argument applies when its differentiability and admissibility hypotheses hold. The HJI formulation additionally depends on the disturbance strategy class and move order. A finite-grid reachability computation introduces spatial and temporal approximation issues that none of those statements automatically resolves.

The next chapter uses terminal level-set functions to encode desired or dangerous sets, then distinguishes reaching at one terminal instant from entering a set at any point in an interval. That distinction is essential before translating a mathematically correct reachable-set calculation into a safety claim.

Primary sources

• AA203 lecture 10 recording, lecture 9 slides for the opening algorithms, and lecture 10 slides.
• Bertsekas, discounted dynamic programming lecture notes, MIT 6.231.
• Bansal, Chen, Herbert, and Tomlin, Hamilton–Jacobi Reachability: A Brief Overview and Recent Advances, 2017.
• Chen and Tomlin, Hamilton–Jacobi Reachability: Some Recent Theoretical Advances and Applications in Unmanned Airspace Management, Annual Review of Control, Robotics, and Autonomous Systems 1:333–358, 2018.