Costates, moving endpoints, and the price of arriving sooner

Two reasonable arrival times cannot both be optimal

Move a unit-mass cart from rest at position ten to rest at the origin. Let \(p(t)\) be position, \(v(t)\) velocity, \(u(t)\) commanded acceleration, and \(T>0\) the arrival time. The dynamics are \(\dot p=v,\dot v=u\). Position is measured in meters, time in seconds. There is no acceleration limit in this model. Penalize both elapsed time and acceleration effort:

The positive weights \(\alpha,\beta\) express a design tradeoff. If cost has a chosen unit \(\mathrm{CU}\), then \(\alpha\) has units \(\mathrm{CU}/\mathrm{s}^2\) and \(\beta\) has units \(\mathrm{CU}\,\mathrm{s}^3/\mathrm{m}^2\). Comparing their numerical values without specifying units is meaningless. For numerical examples we use those reference units and set both numerical weights to one.

A longer maneuver reduces the effort needed to cover a fixed distance. But it raises the time penalty. Chapter 3 supplied the minimum squared-acceleration integral \(12D^2/T^3\) for signed displacement \(D\) and a fixed duration. The source task has \(D=-10\). Even before introducing costates, this gives the best possible cost at each duration:

For \(\alpha=\beta=1\), a two-second trip has cost \(77\), while a five-second trip has cost \(17.3\). Neither comparison proves that five seconds is optimal. We need an equation balancing the marginal value of time and effort, and we need to understand where that equation comes from when a closed-form value function is unavailable.

The route this chapter takes to that equation is called an indirect method. In the course's map of optimal and learning-based control, indirect methods lie on the open-loop branch, beside direct methods: both compute one control signal \(u(t)\) on \([0,T]\) for one given initial state, not a feedback law. An indirect method first derives necessary optimality conditions for the infinite-dimensional problem, whose decision variable is an entire function of time, and then solves the boundary-value problem those conditions define. A direct method, developed in a later chapter, first transcribes the problem into a finite-dimensional nonlinear program and then optimizes it. The closed-loop branch, which computes policies by dynamic programming or Hamilton–Jacobi–Bellman and Hamilton–Jacobi–Isaacs equations, and model predictive control, which re-solves an open-loop problem from each measured state, come later in the course.

The source's shortest path is a boundary-value problem 06:47

Recording: 6:47–17:19. The lecture first completes the example left at the end of the previous slide deck. To find the shortest graph connecting \((t_0,x_0)\) to \((t_f,x_f)\), minimize

Here \(t\) is a horizontal graph coordinate. For geometric length, its scale must be compatible with the vertical coordinate's scale; adding a squared physical time to a squared physical distance without conversion would not define ordinary spatial length. This example is a geometric variational problem, not a statement that graph length is automatically the cart's physical travel distance.

Set \(g(x,\dot x,t)=\sqrt{1+\dot x^2}\). Its derivative with respect to \(x\) is zero, while \(g_{\dot x}=\dot x/\sqrt{1+\dot x^2}\). Differentiating the latter with respect to time gives

The denominator is positive, so Euler–Lagrange stationarity requires \(\ddot x=0\). Its solutions are \(x(t)=c_1t+c_2\). The two prescribed endpoint values determine the two constants, giving

The source explicitly distinguishes this stationary candidate from a minimum certificate. An original global certificate is the integral triangle inequality: integrating the curve's tangent vectors yields the net displacement, whose norm cannot exceed the integral of tangent norms. Thus \(J[x]\ge\sqrt{(t_f-t_0)^2+(x_f-x_0)^2}\), and the straight segment attains equality.

The arc-length case is unusually simple. In general, expanding the time derivative along a twice-differentiable curve turns the Euler equation into

an ordinary differential equation that is nonlinear in general, linear when \(g\) is at most quadratic in \((x,\dot x)\), and time-varying only if \(g\) depends explicitly on \(t\), though not always then: a term of \(g\) that depends on \(t\) alone drops out of the equation. It is of second order only where the coefficient \(g_{\dot x\dot x}\) of \(\ddot x\) is nonzero, or for vector-valued \(x\) a nonsingular matrix; where it is zero, or singular in the vector case, the equation cannot be solved for \(\ddot x\). For the arc-length integrand \(g_{\dot x\dot x}=(1+\dot x^2)^{-3/2}\gt0\), which is why the equation reduced to \(\ddot x=0\). Its two boundary conditions are split, one on \(x(t_0)\) and one on \(x(t_f)\), rather than both at the same end.

This calculation solves a two-point boundary-value problem: information is specified at both ends. It differs from an initial-value problem, where the state and enough derivatives are known at the starting time. In optimal control, the plant state is known initially, but the costate is usually constrained at the end. That split is one source of computational difficulty.

A moved endpoint is not a perturbation at the old final time 18:00

Recording: 18:00–31:51. Keep the initial point fixed but allow the final point or final time to change. Consider a family \(x_\epsilon(t)\) with endpoint time \(T_\epsilon=T+\epsilon\,\delta T+o(\epsilon)\). Define the fixed-time trajectory variation \(\eta(t)=\left.\partial x_\epsilon(t)/\partial\epsilon\right|_0\), and the total endpoint displacement \(\delta x_f=\left.\frac{d}{d\epsilon}x_\epsilon(T_\epsilon)\right|_0\). The chain rule gives

The first term changes the curve at the old time; the second follows the curve as the endpoint time moves. If final position is fixed while arrival time varies, \(\delta x_f=0\), but generally \(\eta(T)=-\dot x(T)\delta T\ne0\). Confusing these variations discards precisely the term needed for a correct free-time condition.

The setting of this extension can be stated exactly. The unknown is a curve \(x:[t_0,T]\to\mathbb R^n\), where \(n\) is the number of components, each with a continuous first derivative. The integrand \(g(x,\dot x,t)\) is twice continuously differentiable, and we seek a relative extremum of \(J[x]=\int_{t_0}^Tg(x,\dot x,t)\,dt\), that is, a curve whose cost is no larger (or, for a maximum, no smaller) than that of every admissible curve near it. The initial time \(t_0\) and initial state \(x(t_0)\) are fixed. The final time \(T\) may be fixed or free, and each component of \(x(T)\) may separately be fixed or free. This is the framework of D. E. Kirk, Optimal Control Theory: An Introduction (Dover, 2004), Section 4.3.

Whatever the terminal specification, an extremal must satisfy the Euler equations \(g_x-\frac{d}{dt}g_{\dot x}=0\), one for each component, on the open interval \((t_0,T)\). The reason is a containment argument. A curve that is extremal among all admissible curves is, in particular, extremal among the curves that share its own endpoint \((x(T),T)\), and for that smaller fixed-endpoint family the derivation of chapter 3 applies unchanged. Freeing the endpoint therefore leaves the differential equations intact and removes boundary data. When the matrix \(g_{\dot x\dot x}\) is nonsingular, the Euler equations are \(n\) second-order differential equations whose general solution carries \(2n\) constants of integration. Only the \(n\) initial conditions remain prescribed when the terminal state is free, so the missing equations must come from the boundary variation.

For \(J[x]=\int_{t_0}^Tg(x,\dot x,t)\,dt\), the varying upper limit contributes \(g(T)\delta T\). Integration by parts contributes \(g_{\dot x}(T)\eta(T)\). Once the interior Euler equation holds and the initial endpoint is fixed, the total boundary variation is

All quantities in brackets are evaluated on the candidate at its terminal time. For vector-valued \(x\), scalar products replace products. The admissible endpoint variations determine which coefficients must vanish; there is no single boundary condition that applies to every problem.

Each row of the table also balances equations against unknowns, which is how one checks that a terminal specification is complete. With \(g_{\dot x\dot x}\) nonsingular there are \(2n\) constants of integration. In the first two rows, the \(n\) initial conditions and \(n\) terminal equations, either the prescribed values \(x(T)=x_f\) or the natural conditions \(g_{\dot x}(T)=0\), give \(2n\) equations for the \(2n\) constants. When \(T\) is free it is one more unknown, and the third and fourth rows supply \(2n+1\) equations: the \(n\) initial conditions, \(n\) terminal equations, and one scalar time condition. In the fourth row \(g_{\dot x}(T)=0\) already holds, so the bracket reduces to \(g(T)\) and the time condition reads \(g(T)=0\). Kirk's table lists these counts beside each case. A matching count is a consistency check, not an existence result, because the equations are nonlinear in \(T\). For the arc-length integrand, \(g=\sqrt{1+\dot x^2}\ge1\), so the fourth row's condition \(g(T)=0\) has no solution. That is the correct answer: with both the final point and the final graph coordinate free, a curve can be made arbitrarily short by ending it near its starting point, and no curve of positive length attains the infimum zero.

In the source's free-position example, \(x(0)=1\), the final graph coordinate is \(T=5\), and \(x(5)\) is free. The interior equation still gives a line. The natural endpoint condition is \(\dot x(5)/\sqrt{1+\dot x(5)^2}=0\), hence the line's slope is zero and \(x(t)=1\). Geometrically it meets the terminal vertical line perpendicularly. The bound \(J[x]\ge5\) provides a global certificate, attained by this constant graph.

If the terminal point lies on a curve \(x_f=\psi(T)\), its variations are coupled: \(\delta x_f=\psi'(T)\delta T\). One must substitute that relation, not set both coefficients to zero independently. This extension illustrates the correct procedure for terminal manifolds and moving targets. Likewise, shifting the time coordinate to start at zero does not eliminate a physically variable departure time in a time-dependent environment; the shifted dynamics would still depend on that decision.

Carrying out the substitution \(\delta x_f=\psi'(T)\delta T\) gives one scalar condition. The boundary variation becomes \(\left[g_{\dot x}(T)^\top\psi'(T)+g(T)-g_{\dot x}(T)^\top\dot x(T)\right]\delta T\), so a free \(T\) requires

together with the \(n\) terminal equations \(x(T)=\psi(T)\). With the \(n\) initial conditions this is again \(2n+1\) equations for \(2n\) constants and \(T\); it is the fifth row of Kirk's table, whose target curve the slides denote \(\theta\). For the scalar arc-length integrand, multiplying the condition by \(\sqrt{1+\dot x(T)^2}\) gives \(1+\dot x(T)\psi'(T)=0\): the extremal line meets the target curve at a right angle.

As a check, take \(x(0)=1\) and the target line \(x=6-t\), so \(\psi'=-1\). The extremal has slope one, \(x(t)=1+t\), and reaches the target at \(T=2.5\), \(x(T)=3.5\). Its length \(2.5\sqrt2\approx3.5355\) equals the distance \(5/\sqrt2\) from the point \((0,1)\) to the line \(t+x=6\). Every curve to the line is at least as long as the straight segment to its own endpoint, by the integral triangle inequality, and that segment is no shorter than this distance, so the extremal is the global minimum. The free-position example also meets its target, the vertical line \(t=5\), at a right angle, but a vertical line cannot be written as \(x_f=\psi(T)\); the fixed-time row of the table covers it instead.

Mixed specifications are handled component by component. Suppose \(T\) is fixed, the components \(x_i(T)\) for \(i=1,\ldots,r\) are prescribed as \(x_{f,i}\), and the components \(x_j(T)\) for \(j=r+1,\ldots,n\) are free, with \(0\le r\le n\). Writing \(\delta x_{f,i}\) for the \(i\)th component of \(\delta x_f\), the admissible variations are \(\delta T=0\), \(\delta x_{f,i}=0\) for \(i\le r\), and \(\delta x_{f,j}\) arbitrary for \(j\gt r\). The boundary variation reduces to \(\sum_{j=r+1}^{n}\partial g/\partial\dot x_j(T)\,\delta x_{f,j}\), so the terminal conditions are \(x_i(T)=x_{f,i}\) for \(i\le r\) and \(\partial g/\partial\dot x_j(T)=0\) for \(j\gt r\). There are still \(n\) terminal equations: a prescribed component contributes its value, a free one its natural condition, and neither contributes both.

A costate is a multiplier function for a dynamical constraint 31:51

Recording: 31:51–40:10. A freely chosen state curve ignores how controls actually produce motion. Now use a state \(x(t)\in\mathbb R^n\), input \(u(t)\in\mathbb R^m\), and dynamics \(\dot x=f(x,u,t)\). The integers \(n,m\) are the numbers of state and input coordinates. Given an initial state, changing \(u\) changes \(x\) through this differential equation. State and input perturbations cannot be chosen independently while preserving feasibility.

This coupling is an instance of a constrained problem of the calculus of variations, which can be stated before any control notation. Let \(w:[t_0,T]\to\mathbb R^{n+m}\) have continuously differentiable components, and seek a relative extremum of \(J[w]=\int_{t_0}^Tg(w,\dot w,t)\,dt\) subject to \(n\) differential constraints \(\gamma_i(w(t),\dot w(t),t)=0\), \(i=1,\ldots,n\), with \(g\) twice continuously differentiable and \(t_0\) and \(w(t_0)\) fixed. When the constraints can be solved for \(n\) components of \(w\) once the other \(m\) are chosen, only \(m\) of the \(n+m\) components are independent. Optimal control is the case \(w=(x,u)\) with \(\gamma(w,\dot w,t)=f(x,u,t)-\dot x\): choosing the \(m\) input components determines the \(n\) state components through the dynamics. Fixing \(w(t_0)\) then also fixes \(u(t_0)\); this is harmless, because no term of the problem depends on \(\dot u\), so no boundary condition on \(u\) arises. The official slides call the constraint map \(f\); this book writes \(\gamma\), because \(f\) already denotes the dynamics, and the two are related by \(\gamma=f-\dot x\).

To expose the coupling, introduce a multiplier function \(\lambda(t)\in\mathbb R^n\), called the costate, and form the augmented functional

Here \(\ell\) is the running cost rate and \(\phi\) the terminal cost. On every dynamically feasible trajectory, the added integral is zero. It is a Lagrange multiplier construction, not a nonnegative finite penalty for violations. There is one multiplier component per state equation. The multiplier is a function because the dynamics constrain the trajectory at every time.

The course slides denote costates by \(p\); this book uses \(\lambda\) so that \(p\) can continue to mean the cart's physical position. The symbol change alters no equation. With column-vector gradients, define the Hamiltonian

The augmented integrand is \(H-\lambda^\top\dot x\), not just \(H\). Keeping the subtracted derivative term visible explains the sign in the costate equation and the terminal condition. The Hamiltonian is an optimization object built from cost and dynamics. It is not generally the cart's mechanical energy, even when its name recalls Hamiltonian mechanics.

For the general constrained problem the multiplier rule takes a compact form. Define the augmented integrand \(g_a(w,\dot w,\lambda,t)=g(w,\dot w,t)+\lambda^\top\gamma(w,\dot w,t)\), with one multiplier function per constraint. A normal extremum satisfies

The Euler equations of \(g_a\) are written as if the \(n+m\) components of \(w\) were unconstrained, and the constraints are imposed separately. The constraint equation is itself the Euler equation of \(g_a\) with respect to \(\lambda\), because \(g_a\) does not depend on \(\dot\lambda\). The rule mirrors the finite-dimensional Lagrange conditions of chapter 2, stationarity of the Lagrangian in the decision variables together with feasibility, with the multiplier vector replaced by a multiplier function. It inherits their qualifications: the derivation requires a constraint qualification on the linearized constraints and normality, the assumption that the multiplier of \(g\) can be normalized to one. The next section obtains the control conditions by an explicit variation and then checks them against this rule.

With differential constraints handled by multipliers, the variational approach to optimal control proceeds in three stages, and this chapter is the first. It derives necessary conditions under the assumption that the admissible controls are unbounded. The next chapter introduces Pontryagin's minimum principle, heuristically, as the generalization of the fundamental theorem of the calculus of variations to bounded controls, works special cases with bounded controls, including bang-bang control, and turns to computational methods for the resulting boundary-value problems. The third stage, as planned for the course, also includes problems with bounded state variables; the next chapter treats input bounds only, and the direct methods of chapter 6 impose state bounds at the transcription nodes. In practice one derives the conditions analytically, specializes the boundary conditions to the problem, and hands both to a numerical boundary-value solver. Because the optimality conditions are formed before any discretization, indirect methods are also described as optimize-then-discretize, in contrast to the discretize-then-optimize order of direct methods.

Derive the three equations before memorizing them 40:10

Recording: 40:10–51:58; derivation sketch at 1:17:48–1:24:43. Assume smooth data, a regular normal extremum admitting a costate multiplier, and unconstrained state and control variations in the augmented formulation. The normal assumption means the original objective's multiplier can be normalized to one. This chapter develops the smooth first-order conditions used in the recording; it is not a proof covering singular endpoint maps, abnormal extremals, or arbitrary nonsmooth controls.

Stated in full, the problem is to find an admissible control \(u^\star\) that causes the system \(\dot x=f(x,u,t)\) to follow an admissible trajectory \(x^\star\) minimizing \(J[u]=\phi(x(T),T)+\int_0^T\ell(x,u,t)\,dt\). A control is admissible when it satisfies the problem's input constraints at every time, and a trajectory when it satisfies the state constraints. Here the admissible state and control regions are all of \(\mathbb R^n\) and \(\mathbb R^m\), so admissibility imposes only the regularity the derivation uses. The cost is written as a function of \(u\) alone because, for a fixed initial state, the dynamics determine \(x\) from \(u\). The official slides state the problem on \([t_0,t_f]\) and name the terminal cost \(h\), the running cost \(g\), and, in the cart example, the effort weight \(b\); this chapter writes \([0,T]\), \(\phi\), \(\ell\), and \(\beta\). We assume that \(\phi\) is twice continuously differentiable and that \(f\) and \(\ell\) are continuously differentiable in \(x\) and \(u\), because the derivation differentiates all three; the initial state \(x(0)=x_0\) is fixed.

For the broader theorem, Daniel Liberzon's discussion of the maximum principle explains nontrivial multipliers, abnormal cases, and why necessary conditions do not ensure either existence or optimality. His maximum convention uses a different objective-multiplier sign; copying a costate formula across conventions without translating the Hamiltonian changes its meaning.

Let \(\eta=\delta x\), \(\nu=\delta u\), and \(\xi=\delta\lambda\) denote variations at fixed times. Varying the augmented integral and integrating \(-\lambda^\top\dot\eta\) by parts gives the interior terms

The notation \(H_x,H_u\) means column gradients; \(H_\lambda=f\). Under the stated multiplier formulation, the independent variations force the coefficients to vanish:

The first equation enforces feasibility. The second chooses the multiplier to cancel the state variation. The third is stationarity with respect to the control. Written without Hamiltonian shorthand, the latter two are

where \(f_x\) and \(f_u\) are the \(n\)-by-\(n\) and \(n\)-by-\(m\) dynamics Jacobians. A sign or transpose error here produces a different control law. The coupled equations contain \(2n\) first-order differential equations and \(m\) algebraic control equations. The algebraic equation may permit elimination of \(u\), but this requires suitable structure; an equation independent of \(u\) cannot automatically be solved for it.

The multiplier rule of the previous section reaches the same equations without a separate variation. For \(w=(x,u)\) and \(\gamma=f-\dot x\), the running cost \(\ell\) plays the role of \(g\), and the augmented integrand is \(g_a=\ell+\lambda^\top(f-\dot x)=H-\lambda^\top\dot x\). Its partial derivatives are \(\partial g_a/\partial x=\ell_x+f_x^\top\lambda\), \(\partial g_a/\partial\dot x=-\lambda\), \(\partial g_a/\partial u=\ell_u+f_u^\top\lambda\), and \(\partial g_a/\partial\dot u=0\). The Euler equation in \(x\) is therefore \(\ell_x+f_x^\top\lambda+\dot\lambda=0\), the costate equation. The Euler equation in \(u\) has no derivative term and is the control equation \(\ell_u+f_u^\top\lambda=0\). The constraint \(\gamma=0\) is the state equation. The minus sign of the costate equation enters through \(\partial g_a/\partial\dot x=-\lambda\), which comes from the term \(-\lambda^\top\dot x\).

The variational derivation has an order that the boxed statement hides. Take the simplest setting: no terminal cost, and fixed \(T\) and \(x(T)\). On a feasible trajectory \(f-\dot x=0\), so the coefficient of \(\xi\) contributes nothing. The state variation \(\eta\) is not free: the control variation generates it through the linearized dynamics \(\dot\eta=f_x\eta+f_u\nu\), \(\eta(0)=0\). The costate, introduced with no restriction on its values, is therefore chosen to make the coefficient of \(\eta\) vanish, which is the costate equation. Only \(\nu\) remains, and if it may be chosen arbitrarily on \([0,T]\), the fundamental lemma of chapter 3 gives \(H_u=0\). Integrating \(\ell_x^\top\eta\) by parts with any solution of the costate equation makes the remaining variation explicit:

This identity holds for every terminal value \(\lambda(T)\), and it shows where the last step needs care. If \(x(T)\) is free, the choice \(\lambda(T)=0\), the terminal-cost gradient when \(\phi=0\), removes the boundary term and leaves \(\nu\) arbitrary. If \(x(T)\) is prescribed, as in this setting, an admissible control variation must keep the endpoint fixed, \(\eta(T)=0\). These are \(n\) linear constraints on \(\nu\), so the control variation is not arbitrary, and the fundamental lemma cannot be applied to it directly. Write \(\eta(T)=\mathcal L\nu=\int_0^T\Phi(T,t)f_u\nu\,dt\), where \(\Phi(T,t)\) is the matrix that maps \(\eta(t)\) to \(\eta(T)\) when \(\nu=0\). Optimality implies \(\delta J=0\) for every \(\nu\) with \(\mathcal L\nu=0\) only if each such \(\nu\) is the first-order direction of a family of controls that still reach the prescribed endpoint. That holds when \(\mathcal L\) maps onto \(\mathbb R^n\), that is, when the linearized system is controllable on \([0,T]\): the implicit function theorem then corrects \(u+\epsilon\nu\) by a term of order \(o(\epsilon)\) that restores the endpoint.

Granting controllability, the boundary term vanishes because \(\eta(T)=0\), and the function \(\ell_u+f_u^\top\lambda\), computed with any one solution of the costate equation, is orthogonal to the null space of \(\mathcal L\), so it lies in the \(n\)-dimensional space of functions \((\Phi(T,t)f_u)^\top\mu\) with \(\mu\in\mathbb R^n\). Changing \(\lambda(T)\) by \(\mu\) changes \(\lambda(t)\) by \(\Phi(T,t)^\top\mu\) and the integrand by exactly \((\Phi(T,t)f_u)^\top\mu\), so one choice of \(\lambda(T)\) makes \(H_u=0\) on the whole interval. The \(n\) free terminal costate values compensate for the \(n\) endpoint constraints, just as a prescribed \(x(T)\) replaces the condition \(\lambda(T)=\phi_x\) among the boundary conditions. When the linearization is not controllable, the argument fails and abnormal extremals become possible; the normality assumption at the start of this section excludes them. The cart meets the condition at every fixed duration. Its linearization is the double integrator itself, with \(f_x=\begin{bmatrix}0&1\\0&0\end{bmatrix}\) and \(f_u=(0,1)^\top\), so \(\Phi(T,t)f_u=(T-t,1)^\top\) and

The determinant of this matrix is \(T^4/12\), positive for every \(T\gt0\), so \(\mathcal L\) maps onto \(\mathbb R^2\) and the costates computed for the cart are normal multipliers.

For a quadratic input cost \(\ell=\beta u^2/2\) and input-affine dynamics, positive \(\beta\) makes the control part strictly convex, so stationarity determines its unique Hamiltonian minimizer. For a general nonconvex dependence on \(u\), \(H_u=0\) merely identifies a stationary control. With an input bound, the minimizing control can lie at a boundary with \(H_u\ne0\). The next lecture's minimum principle addresses that broader situation.

When initial state is fixed, its boundary variation vanishes. For a fixed final time, the remaining terminal term is \((\phi_x-\lambda(T))^\top\delta x_f\). If a final state component is free, its costate component must equal the corresponding terminal-cost gradient. If it is fixed, its variation is zero and no such equality follows. Setting all terminal costates to zero merely because the terminal state is specified would overconstrain the problem.

Free arrival time adds a Hamiltonian boundary equation

The endpoint chain rule from the geometric problem still applies, now with \(\dot x(T)=f(T)\). The varying upper limit contributes the running cost \(\ell(T)\delta T\); the integrated multiplier term contributes \(-\lambda(T)^\top\eta(T)\). The terminal cost varies as \(\phi_x^\top\delta x_f+\phi_T\delta T\), where \(\phi_T\) is the partial time derivative holding terminal state fixed. Substituting \(\eta(T)=\delta x_f-f(T)\delta T\) gives

Thus fixed terminal state and free terminal time require \(H(T)+\phi_T=0\). A free terminal state with fixed time requires \(\lambda(T)=\phi_x\). When some state coordinates are fixed and others free, apply the corresponding condition componentwise. When state and time lie on a coupled terminal manifold, substitute its allowable variations instead of treating them as independent.

Kirk's Section 5.1 collects these cases in a table, and each case again balances equations against unknowns. After the control equation has eliminated \(u\), the state and costate equations are \(2n\) first-order differential equations whose solution carries \(2n\) constants of integration. The initial condition supplies \(n\) equations. With \(T\) fixed, the terminal condition supplies \(n\) more: the prescribed values \(x(T)=x_f\), or \(\lambda(T)=\phi_x\) when the terminal state is free. With \(T\) free, \(T\) is one more unknown and \(H(T)+\phi_T=0\) is one more equation, so \(2n+1\) equations determine the \(2n\) constants and \(T\). This holds whether the terminal state is prescribed or free; when both the state and the time are free, both \(\lambda(T)=\phi_x\) and \(H(T)+\phi_T=0\) must hold. The cart illustrates the count. Its affine costates, quadratic velocity, and cubic position carry four constants of integration, which together with the arrival time make five unknowns, and the four endpoint values and the Hamiltonian condition make five equations.

A further fixed-time case requires the terminal state to lie on a surface. Let \(\sigma:\mathbb R^n\to\mathbb R^k\), with \(k\le n\), be continuously differentiable with linearly independent gradients \(\nabla\sigma_1,\ldots,\nabla\sigma_k\) at \(x(T)\), and require \(\sigma(x(T))=0\) at a fixed \(T\). The slides write the surface as \(m(x)=0\); this book reserves \(m\) for the number of inputs. The admissible endpoint variations are the tangent vectors of the surface, \(\sigma_x(x(T))\,\delta x_f=0\), where \(\sigma_x\) is the \(k\)-by-\(n\) Jacobian. Under the controllability condition of the previous section, every tangent vector is the first-order endpoint variation of an admissible family of controls, so the boundary term \((\phi_x-\lambda(T))^\top\delta x_f\) must vanish for every such vector. Hence \(\phi_x-\lambda(T)\) is orthogonal to the null space of \(\sigma_x\). It therefore lies in the span of the gradients:

The constants \(d_1,\ldots,d_k\) are new unknown multipliers, one per surface equation; the symbol \(d_i\) follows the slides and is unrelated to the derivative \(d/dt\). The \(n\) initial conditions, the \(n\) components of the costate condition, and the \(k\) surface equations give \(2n+k\) equations for the \(2n\) constants of integration and the \(k\) multipliers. The earlier fixed-time cases are the extremes of this one. With \(k=n\) and \(\sigma(x)=x-x_f\), the gradients span \(\mathbb R^n\), so the condition leaves \(\lambda(T)\) unrestricted, as for a prescribed state. With \(k=0\) the sum is empty and \(\lambda(T)=\phi_x\). Prescribing only the components \(x_1,\ldots,x_r\), with \(\sigma_i(x)=x_i-x_{f,i}\), leaves \(\lambda_i(T)\) unrestricted for \(i\le r\) and forces \(\lambda_j(T)=\partial\phi/\partial x_j\) for \(j\gt r\), which is the componentwise rule above. Exercise 7 solves a case in which the surface is a line that no single coordinate describes.

For an autonomous smooth problem with no explicit time dependence, \(H\) is constant along a smooth stationary state–costate–control arc. Indeed, the chain rule gives \(dH/dt=H_x^\top\dot x+H_\lambda^\top\dot\lambda+H_u^\top\dot u+H_t\); the first two terms cancel, \(H_u=0\), and \(H_t=0\). This is an on-trajectory identity under the stated smoothness, not a reason to call \(H\) physical energy. If the terminal cost depends explicitly on time, the constant value is set by \(-\phi_T\), not necessarily zero.

Solve the lecture's free-time cart in state–costate form 53:37

Recording: 53:37–1:17:24. Write the cart state as \(x=(p,v)^\top\), with dynamics \(f=(v,u)^\top\). For the cost from the opening, \(\ell=\beta u^2/2\), \(\phi=\alpha T^2/2\), and the costate is \(\lambda=(\lambda_p,\lambda_v)^\top\). The Hamiltonian becomes

Its conditions are

The last equation gives \(u=-\lambda_v/\beta\). Since \(\lambda_p\) is constant, \(\lambda_v\) is affine in time, so \(u\) is affine, \(v\) quadratic, and \(p\) cubic. This reproduces the shape obtained by the higher-derivative calculation in chapter 3, now using only first-order state equations.

For initial position \(p_0\), initial and final velocity zero, and displacement \(D=p(T)-p_0\), the four state endpoint conditions determine

Check the costate derivative explicitly: \(\dot\lambda_v=12\beta D/T^3=-\lambda_p\). Check the control stationarity explicitly: \(\beta u+\lambda_v=0\). These two sign tests are easy to lose when copying formulas from speech recognition.

The final state is prescribed, so \(\delta x_f=0\); the arrival time is free. At the endpoint, \(v(T)=0\) and \(\lambda_v(T)=-\beta u(T)\), hence

The minus sign is essential. Combining this with \(\phi_T=\alpha T\) gives

For the recording's \(p_0=10,p(T)=0\), \(D=-10\), this is \(T^\star=(1800\beta/\alpha)^{1/5}\). The official slide confirms this factor. The local transcript intermittently drops the minus sign in the preceding endpoint equation; the Hamiltonian derivation and the independently differentiated value function resolve it.

With unit numerical weights, \(T^\star\approx4.477694927\) seconds and \(J^\star\approx16.708126549\). The initial acceleration is approximately \(-2.9926\) meters per second squared and the final acceleration its positive opposite. The cart first accelerates toward the origin and then brakes. Neither the acceleration endpoints nor the costates are required to be zero merely because endpoint velocity is zero.

Stationarity chooses the duration; a separate argument proves the minimum

Differentiate the fixed-duration optimum \(V(T)=\alpha T^2/2+6\beta D^2/T^3\). Its derivative is \(V'(T)=\alpha T-18\beta D^2/T^4\), exactly the free-time boundary residual. For \(D\ne0\) and positive weights,

Also \(V(T)\to\infty\) as \(T\downarrow0\) and as \(T\to\infty\). Thus there is exactly one stationary duration and it is the global minimum over \(T>0\). At each fixed duration, chapter 3 already proved that the cubic is the unique global minimum-effort trajectory. Combining these two facts proves global optimality of the free-time solution for this particular unbounded cart problem. Necessary conditions alone did not provide that conclusion.

The fifth-root law quantifies the tradeoff. Multiplying the time weight \(\alpha\) by \(32\) halves the optimal duration; multiplying effort weight \(\beta\) by \(32\) doubles it. Doubling displacement multiplies duration by \(2^{2/5}\), rather than two. At the optimum, the duration condition gives \(\alpha T^2=18\beta D^2/T^3\). Therefore the terminal time penalty is \(9\beta D^2/T^3\), while the weighted effort term is \(6\beta D^2/T^3\): their ratio is \(3:2\).

The assumptions identify meaningful boundary cases. If \(D=0\) and \(T\) must remain strictly positive, zero control has cost \(\alpha T^2/2\), whose infimum zero is approached as \(T\downarrow0\) but is not attained. If \(\alpha=0\) and \(D\ne0\), arbitrarily slow travel drives effort toward zero without a finite minimizing duration. If \(\beta=0\), arbitrarily short unbounded accelerations similarly approach zero time cost. These are not parameter values to insert blindly into a formula derived for positive weights and nonzero displacement.

An acceleration limit requires another problem. The smooth candidate has peak \(6|D|/T^2\). A bound smaller than that invalidates it; simply clipping its input changes the terminal state. A minimum-time lower bound such as \(T\ge2\sqrt{|D|/U}\) establishes feasibility, but it does not prove that replacing \(T^\star\) by that lower bound gives the bounded minimum of the original mixed cost. The trajectory itself can change shape.

Why a boundary-value solve can be numerically sensitive

The recording notes that practical indirect methods pass differential equations and boundary conditions to a solver. The following original example makes one difficulty explicit. Consider the normalized unstable scalar plant \(\dot x=x+u\), with \(x(0)=0,x(T)=D\), fixed \(T>0\), and cost \(\tfrac12\int_0^Tu^2dt\). Its Hamiltonian is \(H=u^2/2+\lambda(x+u)\), so \(\dot\lambda=-\lambda\) and \(u=-\lambda\).

Guess an initial costate \(c=\lambda(0)\). Then \(\lambda(t)=ce^{-t}\), and solving the state equation gives \(x(t)=-c\sinh t\). A shooting method adjusts \(c\) until its endpoint residual

vanishes. The exact answer is \(c^\star=-D/\sinh T\), but the residual sensitivity is \(R'(c)=-\sinh T\). At \(T=10\), an absolute initial-costate error of \(10^{-6}\) produces an endpoint error of about \(0.0110132\). The forward propagation amplifies a small absolute error strongly. This does not imply that every root solver is ill-conditioned in every norm; it identifies the concrete sensitive map that integration, scaling, and tolerances must handle.

For nonlinear systems, a guessed costate can also send the trajectory into a different region, and several boundary-value solutions may exist. A small endpoint residual verifies one requirement, not global optimality. Multiple shooting, collocation, continuation, and scaling can improve numerical behavior, but their guarantees and implementation belong to the later computational-methods lectures. The analytically solvable cart is a validation fixture, not evidence that all indirect solves are easy.

Verify transversality independently of the costate construction

The following original example uses the fixed-duration cubic, builds its costates, and checks three independent facts: endpoint states, Hamiltonian constancy, and the derivative of the reduced value function. Inputs are a nonzero signed displacement and positive time and effort weights. The computation assumes unbounded acceleration. It uses NumPy, writes no files, and prints the selected duration and cost.

import numpy as np

D, alpha, beta = -10.0, 1.0, 1.0
assert D != 0 and alpha > 0 and beta > 0
T = (18 * beta * D**2 / alpha)**0.2
t = np.linspace(0, T, 10001)
tau = t / T
p = 10 + D * (3 * tau**2 - 2 * tau**3)
v = 6 * D / T * tau * (1 - tau)
u = 6 * D / T**2 * (1 - 2 * tau)
lam_p = -12 * beta * D / T**3
lam_v = -beta * u
H = beta * u**2 / 2 + lam_p * v + lam_v * u
np.testing.assert_allclose([p[0], v[0], p[-1], v[-1]], [10, 0, 0, 0])
assert np.ptp(H) < 1e-12
assert abs(H[-1] + alpha * T) < 1e-12

def value(duration):
    return alpha * duration**2 / 2 + 6 * beta * D**2 / duration**3

eps = 1e-4
derivative = (value(T + eps) - value(T - eps)) / (2 * eps)
assert abs(derivative) < 1e-7
for factor in [0.8, 0.99, 1.01, 1.2]:
    assert value(factor * T) > value(T)
print("Duration:", T, "cost:", value(T))

The finite-difference check does not reuse the Hamiltonian formula, so it can catch a sign error in that formula. Nearby cost comparisons support the numerical example; strict convexity and limiting behavior prove its global claim. The supporting Python laboratory also checks displacement-sign symmetry and rejects the degenerate zero-weight or zero-displacement cases outside the closed-form function's contract.

Exercises with full solutions

What the indirect formulation has—and has not—solved

The source's method transforms a trajectory optimization into state equations, costate equations, an algebraic control condition, and endpoint conditions. It explains how dynamics couple variables and how terminal freedom replaces prescribed values with natural conditions. In the cart example, the free-time boundary condition has the concrete meaning “marginal time penalty equals marginal effort saving.”

The general conditions remain necessary under their assumptions. Global optimality was established here only through separate convex and quadratic arguments for the worked examples. Bounded controls, state constraints, nonsmooth switching, abnormal extremals, and numerical convergence need additional analysis. The next lecture replaces interior control stationarity with a constrained Hamiltonian minimization principle and studies the computational consequences.

Source boundary: the complete local machine transcript and official lecture-4 slides were checked, with lecture-3 slides supporting the carried-over shortest-path example. The book uses \(\lambda\) where the slides use \(p\) for costates. Explicit endpoint-chain-rule derivations, global certificates, dimensional analysis, shooting sensitivity, code, and exercises are original teaching additions. The section evidence map records the source spans and corrected transcription ambiguity. A page-by-page cross-check against the official slides added the course placement of indirect methods, the regularity assumptions and equation counts of the boundary-condition tables taken from Kirk's Optimal Control Theory: An Introduction, the coupled-target and mixed-component conditions, the multiplier rule for differential constraints with its roadmap, the full problem statement, and the terminal-surface multipliers with Exercise 7; the controllability condition that completes the fixed-endpoint proof sketch is an original addition, and the coupled-target and terminal-surface cases are taken from the slides and Kirk because the recording does not discuss them; where the slides' notation differs, the text translates it.