AA203 · Spring 2026 · Lecture 1

Models, constraints, and the meaning of an optimal control

A controller can be stable, execute its code correctly, and still fail the task. It can also optimize an objective exactly while choosing a trajectory that cannot be executed. Before selecting an algorithm, a control designer must decide what the state means, which actions are possible, what information will arrive, and which outcomes are acceptable.

This chapter follows the topic sequence of Marco Pavone's course overview. Timestamps come from locally generated speech recognition and are navigation aids, not a source of mathematical notation. The cart-transfer calculations, counterexamples, implementation, and exercises are original teaching constructions. The official slides support terminology and notation. The official slides also add the feedback block diagram with its heating example and the entry points of disturbance and sensor noise, the performance, planning, and learning gaps, the course map with its Hamilton–Jacobi branch and model-based and model-free reinforcement learning, the three ingredients of the problem formulation, general boundary and bound constraints, free final time, the optimal trajectory–control pair and the maximization convention, linear state feedback, the general discrete-time decision problem, and formal definitions of local, global, and strict minima; where their notation differs, the text translates it, and their second-order and convexity pages, delivered in the next recording, are developed in the next chapter.

1. A cart that reaches the mark but cannot stop 07:15

A cart starts at rest and must move one meter, then stop at the mark two seconds later. Its actuator can provide at most one meter per second squared of acceleration in either direction. A planner that asks for a constant positive acceleration may reach the mark, but the cart will still be moving. A controller that stabilizes the cart at its starting point is well behaved but accomplishes nothing. A useful formulation must distinguish destination, arrival velocity, available time, and actuator authority.

Let \(p(t)\) denote the cart's position in meters at time \(t\), measured in seconds. Let \(v(t)\) be its velocity in meters per second and \(u(t)\) its commanded acceleration in meters per second squared. Our initial model is \(\dot p(t)=v(t)\), \(\dot v(t)=u(t)\), where a dot denotes differentiation with respect to time. We will keep these same physical objects while changing the objective, constraints, and information pattern.

A feedback controller uses a new measurement to adjust its action. For example, a position sensor reports the cart's location; the controller compares it with a reference location and changes the acceleration. A reference is a desired signal, whereas a measurement is information about the actual system. Their difference is an error signal. The distinction matters because a perfectly tracked reference can itself be a poor plan: it might demand an impossible stop or pass through an obstacle.

The standard feedback loop connects four elements through named signals. A summing junction subtracts the measurement from the reference to form the error. The controller maps the error to a control signal; the system responds to that control by producing an output; and a sensor converts the output into the next measurement, which closes the loop. For the cart, the reference is the position of the mark, the control is the commanded acceleration, the output is the position, and the sensor is the position sensor. A home heating loop has the same structure with different physics. The thermostat setting is the reference, the gas flow rate to the furnace is the control, the furnace and home together form the system, the room temperature is the output, and a thermometer supplies the measured temperature. The structure, not the physics, defines the feedback problem.

Two uncertain inputs enter this loop at different places. A disturbance, such as a wind gust on an aircraft or an uncommanded push on the cart, acts on the system and changes the output itself. Sensor noise acts on the sensor and changes only the measurement. The controller then receives the reference minus the noisy measurement, which differs from the true tracking error, the reference minus the output, even when the system behaves exactly as modeled. The distinction determines what feedback can achieve: a controller can counteract a real deviation that it measures, but when it responds to sensor noise it commands corrections that the true state does not need.

The usual control objectives are related but separate. Tracking concerns the difference between actual and desired motion. Disturbance rejection concerns the response to an external influence, such as an uncommanded push. Robustness concerns performance when the mathematical model is imperfect. Stability concerns how trajectories respond to perturbations over time; it is not synonymous with low cost or satisfying every state constraint.

For a concrete stability definition, an equilibrium is Lyapunov stable if sufficiently small initial deviations remain small for all future times. It is asymptotically stable if those deviations also converge to zero. These are infinite-time properties under specified dynamics and control. Our two-second arrival requirement is a finite-time specification. Neither statement replaces the other: a trajectory can reach a terminal point and later drift away, or converge to a point only after missing its deadline.

Lyapunov and asymptotic stability are internal notions: they describe how the state responds to a perturbation of its initial value. An external notion describes the output instead. A system is bounded-input, bounded-output stable if every bounded input signal produces a bounded output signal; this is one precise version of the informal requirement that the output stay under control. The cart without feedback has neither property. Every rest position is an equilibrium of \(\dot p=v\), \(\dot v=u\) with \(u=0\), yet an initial velocity \(v(0)=v_0\ne0\) gives \(p(t)=p(0)+v_0t\), a deviation that grows without bound however small \(|v_0|\) is. With position as the output, the bounded input \(u(t)=1\) m/s² applied from rest gives \(p(t)=t^2/2\) meters, which is also unbounded. Stability is therefore something the controller must supply; the feedback law analyzed later supplies asymptotic stability, which still says nothing about the two-second deadline.

The added question in optimal control is comparative: among the allowed ways to accomplish the task, which one best balances the chosen costs? Learning-based control adds a further difficulty: some relationships needed to make that comparison must be inferred from observations. An accurate optimizer cannot repair an objective that rewards the wrong outcome or a model that omits the dominant effect.

Classical feedback design addresses stability, tracking, disturbance rejection, and robustness, but it leaves three questions open. The first is performance: expressing those desiderata mathematically and selecting the control that realizes the best trade-off among them, which is the comparative question just stated. The second is planning. The loop assumes that a reference is supplied, yet producing a reference that is feasible and worth tracking is itself a decision problem. For a thermostat the reference is a single number; for a mobile robot or an aircraft it is a path through space and time that must respect the vehicle's dynamics and avoid obstacles. In the cart task, the two-second rest-to-rest motion constructed in Section 4 is such a reference. The third is learning, the difficulty just named: the controller may have to adapt to a system whose model is initially unknown or changes during operation, as when an actuator's response drifts with temperature. Optimal control addresses the first two questions for a known model; learning-based control addresses the third.

2. A taxonomy organized by what is computed 17:00

The course schedule moves from optimal control with a known model to control that learns from data. The methods are easier to distinguish by the object they compute than by their names.

For the cart, a plan might specify one acceleration for the first second and another for the next. A feedback policy must say what to do if, halfway through, the cart is slower or farther along than expected. A learned dynamics model estimates how acceleration commands affect motion; an imitation policy instead tries to reproduce actions seen in demonstrations. These are different unknowns, even when a single implementation contains all of them.

Within trajectory optimization, indirect methods first derive optimality conditions and then solve the resulting equations. Direct methods parameterize the trajectory or controls and solve a finite optimization problem. This distinction concerns the computational route, not whether measurements are used during deployment. A direct optimizer can be embedded inside a feedback loop.

The closed-loop route has two branches of its own. Dynamic programming computes a value function and a policy by working backward over stages. Its continuous-time counterpart is the Hamilton–Jacobi–Bellman equation, a partial differential equation for the value function. When an adversarial disturbance acts against the controller, the corresponding two-player equation is the Hamilton–Jacobi–Isaacs equation. It underlies reachability analysis, which computes the states from which the controller can keep the system safe against every admissible disturbance; that set can then serve as a safety guardrail around another controller.

The open-loop and closed-loop routes trade computation against coverage. An open-loop history is computed for one initial condition and is comparatively cheap, but it does not react when the realized state departs from the prediction. A policy prescribes an action for every state that might be visited; that coverage is what lets it respond to deviations, and it is also what makes it expensive to compute. Model predictive control bridges the two routes. It solves an open-loop problem, applies the first part of the resulting history, measures the state, and solves again. Each solve is open-loop; the repetition supplies the feedback.

The course places this optimal-control branch beside the two classical branches of control, feedback control and adaptive control. From adaptive control, in which a model or controller is identified from data during operation, it borrows the idea that defines its learning branch, adaptive optimal control: optimal control when the model is not known exactly and must be inferred from data. That branch has three routes. Imitation learning fits a policy to demonstrations of good behavior. Reinforcement learning improves behavior by trial and error, in two forms. Model-based reinforcement learning fits a dynamics model, and possibly a cost model, to observed transitions and then plans or optimizes a policy with that model. Model-free reinforcement learning improves a policy or a value estimate directly from observed costs or rewards, without constructing an explicit transition model. Suppose the cart's command were a force and its mass unknown. A model-based learner would estimate the inverse mass that converts force into acceleration and then solve the planning problem with that estimate; a model-free learner would adjust its policy from the outcomes of trials without ever forming that estimate.

The learning families can also be combined. A model can be identified from data, used to generate demonstrations, and then refined through interaction. This does not make their failure modes interchangeable. The primary DAgger paper by Ross, Gordon, and Bagnell addresses the fact that a learner's actions affect its future observations. Its relevance here is precise: predicting demonstrated actions accurately on demonstration states does not by itself establish the quality of the resulting closed-loop trajectory.

The recurring design question is therefore not which family wins universally. It is which part of the cart problem remains uncertain or computationally difficult. An expensive model-free training loop is unnecessary to solve the exact two-state example below. Conversely, solving that example exactly does not eliminate the modeling problem for a real robot.

3. A state must contain enough information to predict motion 26:55

Every family in the preceding taxonomy addresses the same optimal control problem, and its formulation has three ingredients: a mathematical description of the system to be controlled, a statement of the constraints, and a specification of a performance criterion. The first ingredient is present even in methods that never write a model down, because they still assume that some underlying system generates their data. The second is seldom explicit in classical feedback design but is central here: a car that must not collide with another car is subject to a constraint, not merely to a preference. This section and the next two take the ingredients in that order, with the cart as the running example.

A state is a collection of variables sufficient to predict future evolution when the future inputs and model are specified. Position alone is insufficient for the cart. Two carts at the same position but moving in opposite directions will react differently over the next second, even under identical acceleration. We therefore define the state vector \(x(t)=(p(t),v(t))^\mathsf T\in\mathbb R^2\), where the superscript \(\mathsf T\) denotes transpose.

More generally, a continuous-time deterministic model has the form \(\dot x(t)=f(x(t),u(t),t)\), with \(x(t)\in\mathbb R^n\), \(u(t)\in\mathbb R^m\), and a vector-valued function \(f\) whose output has the units of a state derivative. The positive integers \(n\) and \(m\) count state coordinates and input coordinates. Explicit dependence on \(t\) allows the same state and input to produce different derivatives at different times. A state trajectory is the resulting function of time; a control history is the input function supplied to the dynamics.

An ordinary differential equation describes local change, which becomes prediction through integration. Over a short interval of length \(h\gt0\),

Here \(s\) is an integration variable. Replacing the integrand by its value at the start gives the forward-Euler approximation \(x(t+h)\approx x(t)+h f(x(t),u(t),t)\). The approximation is not an exact identity for a general model. Its usefulness depends on smoothness, step size, and the input between samples.

For the cart, the second-order position equation becomes two first-order equations by adding velocity to the state. In matrix form, with \(A_c\) the continuous-time state matrix and \(B_c\) the input matrix,

This is a linear time-invariant model: linear combinations of state and input produce the derivative, and the matrices do not change with time. It is called a double integrator because acceleration is integrated once to produce velocity and twice to produce position. If the input were force instead, the second equation would contain inverse mass. Defining the command as acceleration has absorbed that conversion into the actuator model.

Written out by components, the general model \(\dot x=f(x,u,t)\) is a system of \(n\) scalar equations, \(\dot x_i(t)=f_i(x_1(t),\ldots,x_n(t),u_1(t),\ldots,u_m(t),t)\) for \(i=1,\ldots,n\), where \(x_i\) and \(u_j\) denote the state and input coordinates and \(f_i\) is the \(i\)th component of \(f\). Each state derivative may depend on every state coordinate, every input coordinate, and time. The official course slides write the cart's position as \(s\) and its acceleration as \(a\), so that \(\ddot s=a\), and call the two matrices \(A\) and \(B\); they also set vectors in bold type. This chapter keeps \(p\) and \(u\), because \(s\) already denotes an integration variable, and adds the subscript \(c\) to distinguish the continuous-time matrices from the sampled matrices of Section 7.

Suppose \(u\) is held constant during one sample interval. Velocity then varies linearly, \(v(t+s)=v(t)+su\). Integrating that expression for \(0\le s\le h\) gives the exact update

Forward Euler omits the half-acceleration displacement in the first equation. With a stationary cart, \(h=1\) second, and \(u=1\) m/s², Euler predicts zero position change while the exact model predicts half a meter. The discrepancy is not a rounding error or failed optimizer; it is a consequence of the selected numerical model. Reducing the step helps, but using the exact held-input map costs no more for this particular plant.

Prediction also presupposes that the dynamics determine a solution. Local Lipschitz continuity of \(f\) in the state, with suitable regularity of the prescribed input and time dependence, is a standard sufficient condition for local uniqueness. One must additionally rule out finite-time escape to claim existence over an entire planning horizon. Our linear cart with bounded inputs has no such difficulty. A physical system with impacts, contacts, or mode switches may need a hybrid model rather than a smooth equation alone.

The important modeling choice is purposeful sufficiency. Adding every possible physical detail can make planning intractable; omitting a variable that changes future behavior makes the state inadequate. The cart deliberately omits friction and actuator lag. Its exact formulas are exact for that idealization, not evidence that the idealization accurately predicts a particular device.

4. Feasibility precedes performance 38:26

Let the planning interval be \([0,T]\), with duration \(T\gt0\). The cart task imposes boundary conditions \(p(0)=v(0)=0\), \(p(T)=D\), and \(v(T)=0\), where \(D\) is the required signed displacement. The actuator condition is \(|u(t)|\le U_{\max}\), where \(U_{\max}\gt0\) is the maximum acceleration magnitude. A state constraint might additionally require that the cart remain on a track or avoid an obstacle throughout the interval.

The official slides write the planning interval as \([t_0,t_f]\); here \(t_0=0\) and \(t_f=T\). In that general form the boundary conditions are \(x(0)=x_0\) and \(x(T)=x_{\mathrm f}\), where \(x_0\) is the given initial state and \(x_{\mathrm f}\) a prescribed final state. The terminal condition is optional; many problems replace it by a terminal cost, as Section 5 does. The simplest state and control constraints are bounds,

where an inequality between vectors holds coordinate by coordinate and an entry of a bound may be \(-\infty\) or \(+\infty\) when that coordinate is unrestricted. The cart task has \(x_0=(0,0)^\mathsf T\), \(x_{\mathrm f}=(D,0)^\mathsf T\), \(\underline U=-U_{\max}\), and \(\overline U=U_{\max}\). A track of length \(L\gt0\) that the cart must not leave adds \(\underline X=(0,-\infty)^\mathsf T\) and \(\overline X=(L,+\infty)^\mathsf T\), which restricts position and leaves velocity free. Bounds need not be symmetric: an engine that can push but not pull has \(\underline U=0\).

Many useful constraints are not bounds. A planar robot whose position \(\xi(t)\in\mathbb R^2\) must stay outside a disk of radius \(\rho\gt0\) around a person standing at \(c_o\in\mathbb R^2\) must satisfy \(\|\xi(t)-c_o\|_2\ge\rho\), where \(\|\cdot\|_2\) is Euclidean length. The two points at distance \(\rho\) on opposite sides of \(c_o\) are allowed, but their midpoint \(c_o\) is not, so the allowed set contains two points without the segment between them; it is not convex. The formulation accepts such constraints. The price appears later: the more convoluted a constraint's algebraic form, the harder the optimality conditions and numerical methods that must handle it.

An admissible control history satisfies the control restrictions. An admissible trajectory satisfies the state and boundary restrictions. The two cannot be checked independently: an allowed control may generate a forbidden state, and a geometrically attractive path may require unavailable acceleration. The dynamic equations connect these tests.

The two-second, one-meter task has an explicit feasible construction. Apply \(+1\) m/s² for one second and \(-1\) m/s² for the next. At the switch, position is \(0.5\) m and velocity is \(1\) m/s. The second interval adds another \(0.5\) m while returning velocity to zero. All endpoint and actuator constraints are satisfied.

Could the same task be completed in less time? For positive displacement, bounded acceleration and the initial rest condition imply \(v(t)\le U_{\max}t\). The final rest condition also implies \(v(t)\le U_{\max}(T-t)\): a faster velocity would not leave enough time to brake. Both bounds must hold, so

The integral is the area of a triangle with base \(T\) and height \(U_{\max}T/2\). Reversing signs handles negative displacement. Every admissible rest-to-rest transfer must therefore satisfy \(T\ge2\sqrt{|D|/U_{\max}}\). The two opposite constant-acceleration pulses attain the bound, so it is also sufficient for this ideal model. For \(D=1\) m and \(U_{\max}=1\) m/s², the minimum duration is exactly two seconds.

This proof is an original extension of the introductory constraint discussion. It does not assume that a local numerical solver has found the best trajectory. It also makes the assumptions visible: instantaneous acceleration switching is allowed, speed is unbounded except through acceleration and time, and there are no obstacles. A bound on jerk, the time derivative of acceleration, would invalidate the attaining two-pulse control and change the minimum-time problem.

Feasibility is not a large penalty. A constraint excludes an outcome; a penalty may permit it when another cost outweighs its price. For collision avoidance or an actuator's physical limit, replacing a hard condition by a finite weight changes the task. That change may be useful during computation, but it requires a separate check before calling the result admissible.

5. The objective encodes a decision, including its compromises 40:52

A common continuous-time performance measure is

The terminal cost \(\phi\) assigns a scalar cost to the final state and time. The running cost \(\ell\) is a cost rate accumulated along the trajectory. Their units must agree after integration. The initial state is fixed, and the dynamics make the state trajectory depend on the chosen control history, so the notation \(J[u]\) suppresses that dependence rather than denying it.

The official slides write this measure as \(J=h(x(t_f),t_f)+\int_{t_0}^{t_f}g(x(t),u(t),t)\,dt\), calling \(h\) the terminal cost and \(g\) the stagewise or running cost. This chapter writes \(\phi\) and \(\ell\) instead, because \(h\) already denotes a time step and \(g_k\) denotes the cost accumulated over one sampling interval in Section 7. The correspondence is \(h=\phi\), \(g=\ell\), \(t_0=0\), and \(t_f=T\).

For the cart, \(\ell=u^2\) measures squared-acceleration effort. Its integral has units m²/s³. It is often a useful proxy, but it is not energy in joules unless a separate physical model establishes that relationship. A terminal position penalty \(q(p(T)-D)^2\) can be added consistently when the nonnegative weight \(q\) has units s⁻³. Numerical weights without a scale convention can silently change the design when centimeters replace meters.

The final time may be specified or free. With \(T\) specified, the deadline is part of the problem data. With \(T\) free, the duration becomes a decision variable alongside the control history, and a preference about time can enter through the explicit dependence of \(\phi\) on its second argument. For example, the terminal cost \(\phi=cT\), with a price \(c\gt0\) in m²/s⁴, charges for duration at a constant rate. Time and effort usually conflict. The equal-pulse construction derived below completes a hard rest-to-rest transfer of displacement \(D\) in time \(T\) with effort \(16D^2/T^3\), so halving the duration multiplies its effort by eight.

The running cost can also encode tracking. If \(p_{\mathrm{ref}}(t)\) is a planned reference position, the choice \(\ell=u^2+\beta\,(p-p_{\mathrm{ref}}(t))^2\), with weight \(\beta\gt0\) in s⁻⁴, charges both effort and deviation from the plan. The weight sets an exchange rate: a sustained position error of \(e\) meters costs, per unit time, as much as a sustained acceleration of \(\sqrt\beta\,e\) m/s². This running cost depends on time through the reference, which is why \(\ell\) carries its third argument.

Consider a restricted family of two equal-duration pulses: acceleration \(a\) for \(h=T/2\) seconds, followed by \(-a\) for another \(h\) seconds. The constant \(a\) may be negative. Exact integration gives \(p(T)=ah^2\) and \(v(T)=0\). The effort is \(2ha^2\). A hard endpoint constraint forces \(a=D/h^2\), provided that magnitude is allowed.

Replace the position constraint with the terminal penalty and keep the final velocity zero through the pulse construction. The scalar objective becomes \(F(a)=2ha^2+q(ah^2-D)^2\). Differentiating this quadratic and setting the derivative to zero gives

The second derivative is \(4h+2qh^4\gt0\), so the stationary point is the unique global minimum within this family. The restriction to two pulses is part of the claim: this calculation does not solve the unrestricted continuous-time problem.

Set \(D=1\) m, \(T=2\) s, and \(q=2\) s⁻³. Then \(h=1\) s and \(a_\star=0.5\) m/s². The optimizer stops at half a meter. Its effort is \(0.5\) m²/s³ and its terminal penalty is another \(0.5\), for total cost \(1\). The hard-endpoint solution reaches one meter but has effort \(2\). Stopping halfway is therefore exactly what the soft objective asks the optimizer to do. Increasing the weight moves the optimum toward the mark, but no finite positive weight enforces the exact endpoint in this example.

The general optimal control problem seeks an admissible control and its induced admissible state trajectory minimizing \(J\). A minimum need not exist. For a simple example, retain the hard rest-to-rest displacement but leave duration free and charge only effort. The two-pulse construction has cost \(16D^2/T^3\), which approaches zero as \(T\) grows. No finite transfer of nonzero distance can attain zero effort, because zero integral of \(u^2\) means zero acceleration almost everywhere and hence no motion from rest. The infimum is zero; there is no finite-time minimizer. A deadline or positive time cost changes that conclusion.

A minimum also need not be unique. Obstacle avoidance can offer several routes with identical cost. Even a unique optimizer in a simplified model does not imply robustness to modeling errors. The distinctions between feasible points, an infimum, an attained optimum, and convex problems are developed systematically in Boyd and Vandenberghe, Chapter 4; the examples here show why those distinctions affect control design.

A minimizing pair \((x_\star,u_\star)\), consisting of an admissible control and the admissible trajectory it induces, is called an optimal trajectory–control pair. The minimum sought is global: no admissible control may achieve a lower cost. In practice one must sometimes settle for a local minimum, which only excludes better controls nearby; Section 9 makes the distinction precise for finite problems, and a reported solution should state which kind it is. The problem also differs in size from a textbook finite optimization. Its decision variable is an entire function \(u(\cdot)\) on \([0,T]\), with a value at each of infinitely many times, so optimal control is an infinite-dimensional optimization problem. The sampled formulation of Section 7 replaces the function by finitely many numbers.

Maximization needs no separate theory. If \(\mathcal R[u]\) is a quantity to be maximized, then \(\max_u\mathcal R[u]=-\min_u(-\mathcal R[u])\), and both problems have the same optimizers, so it suffices to minimize \(J=-\mathcal R\). This sign convention separates two literatures that study the same problem. Control texts minimize a cost \(J\) over controls \(u\) applied to states \(x\). Reinforcement-learning texts maximize a cumulative reward over actions taken in states, usually with the letters \(r\), \(a\), and \(s\). The translation is exact: total reward is the negative of cost, an action is a control, and the reinforcement-learning state is the control state.

The additive running cost supports decomposing a long problem into shorter decisions once a sufficient state has been chosen. This is the structural starting point for dynamic programming. A history-dependent objective can sometimes be handled by augmenting the state with the relevant history summary, but doing so increases the state dimension. Additivity is therefore a computational advantage, not a claim that every meaningful preference arrives naturally in additive form.

6. A trajectory is not a feedback policy 55:19

An open-loop control is a function \(u(t)=e(x_0,t)\) chosen using the initial state \(x_0\) and the clock. A state-feedback policy is a function \(u(t)=\pi(x(t),t)\) that also uses the current state. The letters \(e\) and \(\pi\) distinguish information patterns; they do not imply different actuator hardware. In either case, the output is an acceleration command.

The nominal two-pulse cart transfer illustrates the difference. Suppose an instantaneous disturbance increases velocity by \(0.2\) m/s just after the first second. If the planned second pulse remains \(-1\) m/s², the final velocity is \(0.2\) m/s and the final position is \(1.2\) m. Integrating the unchanged control cannot erase the state error. A stored nominal trajectory says nothing about what to do in this disturbed state.

Observing the error does not make it physically recoverable. At the midpoint, the disturbed cart has velocity \(1.2\) m/s. Returning that velocity to zero in the remaining one second requires an average acceleration of \(-1.2\) m/s², beyond the actuator limit. A feedback controller can recognize this loss of feasibility, but it cannot satisfy the original deadline without changing some requirement. The nominal minimum-time trajectory used all available authority and left no margin for the push.

When the objective is simply to return to a stationary target after a disturbance, an unsaturated linear feedback law gives a useful separate example. Let position error be \(z=p-D\). Choose \(u=-k_pz-k_vv\), with \(k_p\gt0\) in s⁻² and \(k_v\gt0\) in s⁻¹. Then \(z\) satisfies \(\ddot z+k_v\dot z+k_pz=0\). Its characteristic roots solve \(r^2+k_vr+k_p=0\), where \(r\) is an inverse-time growth rate. Both roots have negative real part: if real, their sum is negative and product positive; if complex, each real part equals \(-k_v/2\). The ideal position error therefore decays to zero.

This derivation establishes asymptotic stability for the unconstrained linear model. It does not establish the two-second arrival requirement or safety during the transient. Saturating the commanded acceleration changes the closed-loop equations, so the same root calculation is no longer a complete proof. Stability, feasibility, and optimality must remain separate claims even when all are desired.

This law is an instance of the most important structured policy, linear state feedback \(\pi(x,t)=-K(t)x\), where the gain \(K(t)\) is an \(m\times n\) matrix that may vary with time. The official slides write the same law as \(Fx\) with \(F=-K\); the minus sign used here matches the later chapter on linear-quadratic regulation. In the error coordinates \(\tilde x=(z,v)^\mathsf T\), the proportional–derivative law reads \(u=-K\tilde x\) with the constant one-by-two gain \(K=\begin{bmatrix}k_p&k_v\end{bmatrix}\). In the original coordinates it is affine rather than linear, \(u=-k_pp-k_vv+k_pD\), because the target \(D\) moves the equilibrium away from the origin. A policy that minimizes a stated cost is called an optimal control law or optimal policy. Linear-quadratic regulation shows that, for linear dynamics with quadratic costs and no constraints, the optimal policy has exactly this linear form. The gains above, chosen only for stability, are not claimed to be optimal for any objective.

In a deterministic, perfectly known system from a fixed initial state, an optimal open-loop history and an optimal feedback law can produce the same nominal trajectory. Their difference becomes consequential off that trajectory or under new observations. A feedback law specifies choices across states; a single history specifies only what to do along time.

7. Sampling the dynamics and closing the loop through optimization 59:24

Let \(k\) be a nonnegative integer sample index and let \(x_k=x(kh)\), where the sampling period \(h\gt0\) is fixed. Let \(u_k\) be the acceleration held between sample \(k\) and sample \(k+1\). The exact cart map derived earlier is

A horizon of \(N\) intervals has duration \(Nh\). A discrete objective may be written \(J_N=\phi_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k)\), where \(g_k\) is cost accumulated over an interval. If the continuous running cost is \(u^2\) and each input is held constant, then \(g_k=hu_k^2\) exactly. Omitting \(h\) changes the relative scale between running and terminal cost when the sampling frequency changes.

The general discrete-time problem keeps the same three ingredients. The system is \(x_{k+1}=f_k(x_k,u_k)\) for \(k=0,\ldots,N-1\), where the transition map \(f_k\) may depend on the stage. The control constraint is \(u_k\in\mathcal U\), where \(\mathcal U\subseteq\mathbb R^m\) is the set of allowed inputs. The cost of a control sequence applied from the initial state \(x_0\) is \(J_N(x_0;u_0,\ldots,u_{N-1})=\phi_N(x_N)+\sum_{k=0}^{N-1}g_k(x_k,u_k)\), with the intermediate states generated by the transitions, and the decision problem is

with an infimum in place of the minimum when none is attained. The official slides write \(h_N\) for \(\phi_N\), \(J^\ast\) for \(J_N^\star\), and the stage as a third argument, \(f(x_k,u_k,k)\). The optimal value is a function of the initial state: each \(x_0\) defines its own problem, and dynamic programming computes \(J_N^\star\) for all of them in one backward pass. For the cart, \(f_k(x,u)=A_hx+B_hu\) at every stage and \(\mathcal U=[-U_{\max},U_{\max}]\). Take \(N=2\), \(h=1\) s, \(g_k=hu_k^2\), \(\phi_2(x_2)=q(p_2-D)^2\), and add the terminal requirement \(v_2=0\) as a constraint. With \(D=1\) m, \(q=2\) s⁻³, and \(U_{\max}=1\) m/s², the reduction in the next paragraph gives \(J_2^\star(0)=1\) m²/s³. This equals the optimal cost of the continuous two-pulse family, because the two held inputs with \(v_2=0\) are exactly that family. The stochastic extension, in which a random disturbance enters each transition and the expected cost is minimized, is the Markov decision process treated later in the course.

For the two-step transfer from rest, the final velocity is \(h(u_0+u_1)\). Requiring it to vanish gives \(u_1=-u_0\). Substitution into the position recursion gives \(p_2=h^2u_0\). The finite optimization has therefore reduced to the scalar variable \(a=u_0\) used above. This is a small example of eliminating the dynamics before optimization. Longer horizons have more decision variables, and state constraints couple them through the same recursions.

Model predictive control, abbreviated MPC, repeatedly solves a finite-horizon problem initialized at the latest state estimate, applies the first action, then measures and solves again. Each solve may optimize an open-loop sequence. The deployed action nevertheless depends on the measured state through the optimization problem, making the overall controller a feedback law. Rawlings, Mayne, and Diehl develop this distinction and the additional feasibility and stability conditions required to justify MPC.

The distinction becomes concrete after the cart's push. A new solve begins from the changed position and velocity. If the old terminal deadline is retained, the solver should report infeasibility under the original actuator limit. If the horizon is extended, a feasible correction may become available. Reoptimization is a mechanism for using information, not permission to silently replace the specification.

A predicted path may satisfy a constraint at the grid points yet violate it between them. For the cart, exact interval position is a quadratic function, so its extrema can be checked using the interval velocity. For more complicated dynamics, constraint tightening or denser checks may be needed. A small discrete residual proves satisfaction of the discrete equations, not universal continuous-time safety.

8. Uncertainty requires a model and an information order 1:03:21

A disturbance is an influence not selected by the controller. In a sampled model, write \(x_{k+1}=f_d(x_k,u_k,w_k)\), where \(f_d\) is the discrete transition and \(w_k\) is the disturbance. A measurement may instead be \(y_k=h_s(x_k)+\eta_k\), with sensor map \(h_s\), observed vector \(y_k\), and measurement noise \(\eta_k\). Process uncertainty changes the state; sensor uncertainty changes what the controller knows about it.

A stochastic formulation assigns probabilities to disturbances and may minimize expected cost. A robust formulation specifies a set of allowed disturbances and requires a condition to hold for every member of that set. Neither formulation is automatically more realistic. A probability distribution that omits an important tail event can understate risk, while a worst-case set that is implausibly broad can make useful operation impossible.

The order of decisions is part of the model. Suppose a scalar deviation evolves as \(z^+=z+u+w\), where \(u\) is chosen before an unknown disturbance \(w\in\{-1,+1\}\), and the goal is \(z^+=0\). For each known value of \(w\), a matching action \(u=-z-w\) exists. Yet no single action works for both disturbances: the two required actions differ by two. Thus “for every disturbance there exists an action” does not imply “there exists an action that works for every disturbance.” A controller cannot use information that has not arrived.

This elementary example explains why state feedback, disturbance feedback, and an open-loop robust plan are different optimization domains. The introductory recording previews differential games and stochastic decision processes; it does not establish their solution algorithms. The quantifier example is an original formalization of the underlying issue.

For the cart's velocity kick, a useful model might bound the kick magnitude, assign it a distribution, or reserve braking margin without explicitly estimating a distribution. Each choice supports a different guarantee. Reporting the disturbance model beside the result is more informative than labeling a controller merely “robust.”

9. Necessary conditions filter candidates; they do not certify success 1:06:53

A finite optimization problem chooses a vector \(z\in\mathbb R^d\) to minimize a scalar function \(F(z)\), where \(d\) is the number of decision variables. The decision vector is distinct from the physical state \(x\); it might contain a whole sequence of accelerations. A local minimum \(z_\star\) has no lower-cost point in some neighborhood. A global minimum has no lower-cost point anywhere in the feasible domain.

When \(z\) may range over all of \(\mathbb R^d\), the problem is called an unconstrained nonlinear program. Its objective is usually assumed continuously differentiable, written \(F\in C^1\), meaning that every first partial derivative exists and is continuous, and often twice continuously differentiable, \(F\in C^2\), meaning the same of every second partial derivative. The official slides write the problem as minimizing \(f(x)\) over \(x\in\mathbb R^n\). This chapter writes \(F\), \(z\), and \(d\) because \(f\), \(x\), and \(n\) already denote the dynamics, the physical state, and its dimension.

Formally, \(z_\star\) is an unconstrained local minimum if there is a radius \(\delta\gt0\) such that \(F(z_\star)\le F(z)\) for every \(z\) with \(\|z-z_\star\|_2\lt\delta\), and an unconstrained global minimum if \(F(z_\star)\le F(z)\) for every \(z\in\mathbb R^d\). It is a strict local or global minimum if the inequality is strict for every such \(z\ne z_\star\); the exclusion is necessary, because no point has strictly lower cost than itself. All norms on \(\mathbb R^d\) are equivalent, so measuring the neighborhood with another norm changes \(\delta\) but not which points are local minima. Strictness is a genuine restriction. The two-pulse objective of Section 5 is a quadratic with second derivative \(4h+2qh^4\), so \(F(a)-F(a_\star)=(2h+qh^4)(a-a_\star)^2\); with \(D=1\) m, \(h=1\) s, and \(q=2\) s⁻³ this is \(4(a-0.5)^2\), and \(a_\star\) is a strict global minimum. By contrast, the continuously differentiable function \(F(z)=(\max\{|z|-1,0\})^2\) on \(\mathbb R\) takes its minimum value zero at every point of \([-1,1]\), so each of those points is a global minimum and none is strict.

If \(F\) is differentiable on an open neighborhood of an unconstrained local minimum, then its gradient must vanish. To see why, suppose the gradient \(g=\nabla F(z_\star)\) were nonzero. For a small positive scalar \(\epsilon\), differentiability gives

The Euclidean norm \(\|g\|_2\) is the square root of the sum of squared components. The remainder \(o(\epsilon)\) divided by \(\epsilon\) tends to zero as \(\epsilon\) tends to zero. Therefore the negative linear term dominates for sufficiently small \(\epsilon\), producing a feasible nearby point of smaller cost and contradicting local minimality. The open-neighborhood assumption matters: it lets the argument move in the negative-gradient direction.

Stationarity is necessary, not sufficient. The scalar function \(F(z)=z^3\) has zero derivative at zero and is not minimized there. Even a zero second derivative does not resolve the issue: \(z^4\) is minimized at zero while \(-z^4\) is maximized there. At a constrained optimum the gradient need not vanish at all; minimizing \(F(z)=z\) over \(z\ge0\) gives an optimum at the boundary \(z=0\), where the derivative is one.

Necessary conditions serve as a filter. If a global minimizer is known to exist and \(F\) is differentiable on all of \(\mathbb R^d\), every global minimizer is a stationary point. One may then solve \(\nabla F(z)=0\), evaluate \(F\) at each solution, and keep the lowest; when the stationary points are finitely many, this comparison replaces a search over the whole space. The existence premise cannot be dropped. The cubic \(F(z)=z^3\) has the single stationary point zero, which the filter would return, yet \(F\) is unbounded below and has no minimizer at all. Second-order conditions sharpen the filter by examining curvature at each candidate, and convexity turns stationarity into a certificate of global optimality. Both are developed in the next chapter, in its sections on necessary conditions and sufficient conditions and convexity.

Our two-pulse quadratic is easier. Its strictly positive second derivative means the derivative increases monotonically, so its unique zero is the global minimizer on the real line. With the bound \(|a|\le U_{\max}\), the constrained optimum is that point if it lies in the interval, or the nearest endpoint otherwise. This projection follows from the monotonic decrease before the unconstrained minimizer and increase after it; clipping arbitrary solutions of nonlinear constrained problems does not enjoy the same justification.

The recording introduces gradient necessity at its end. The full proof, boundary counterexample, and convex-quadratic sufficiency argument above are teaching extensions, consistent with the optimization material in the official slide deck and Boyd and Vandenberghe. They make explicit why an optimizer's stationarity report must be interpreted alongside constraints and objective structure.

10. An executable two-pulse problem

The function two_pulse_transfer in numerical_lab.py implements the restricted family already derived. It accepts finite displacement in meters, positive duration in seconds, an optional nonnegative terminal penalty in s⁻³, and an optional positive acceleration limit in m/s². With no penalty argument, the endpoint is a hard constraint. With a penalty, the endpoint is soft. It returns the first acceleration, final position, and effort; final velocity is zero by the opposite-pulse construction.

The function rejects an impossible hard transfer instead of returning a clipped trajectory that misses the requested endpoint. In the soft problem it clips the scalar optimum when necessary, justified by the convex one-dimensional derivation above. It controls no hardware, performs no network requests, and writes no files. These restrictions make it an inspectable numerical model, not a deployment controller.

import numpy as np
from numerical_lab import two_pulse_transfer

hard = two_pulse_transfer(1, 2, limit=1)
soft = two_pulse_transfer(1, 2, penalty=2, limit=1)
assert np.allclose(hard, [1, 1, 2])
assert np.allclose(soft, [0.5, 0.5, 0.5])

accelerations = np.linspace(-1, 1, 20001)
costs = 2 * accelerations ** 2 + 2 * (accelerations - 1) ** 2
grid_best = accelerations[np.argmin(costs)]
assert abs(grid_best - soft[0]) < 1e-12
print("hard:", hard, "soft:", soft)

The independent grid calculation checks the analytic optimizer rather than merely repeating its closed formula. The executed suite also checks negative displacement, zero penalty, a binding acceleration limit, and rejection of an infeasible 1.9-second hard transfer. The reported hard tuple is \((1,1,2)\); the soft tuple is \((0.5,0.5,0.5)\). These are finite numerical checks of this example, not guarantees for arbitrary control systems.

The interactive laboratory compares the two-pulse construction with the unrestricted minimum-effort double-integrator solution. Its separate derivation shows exactly which additional conclusion is justified when the control family is enlarged.

11. Exercises with full solutions

Each exercise changes one part of the same model. Keep track of whether the change affects physical feasibility, the objective, numerical approximation, or available information.

12. What has been established, and what remains open

The exact held-input cart dynamics, the minimum feasible rest-to-rest duration, and the scalar two-pulse optimizer follow from explicit derivations. The numerical checks verify selected finite cases and rejection behavior. The feedback example establishes stability for an unsaturated ideal linear model. None of these results certifies a real robot with friction, delayed sensing, finite actuator bandwidth, or contact dynamics.

The chapter deliberately separates the source recording's introductory argument from added mathematical depth. The recording covers feedback goals, the course taxonomy, differential-equation models, constraints, performance measures, open-loop and closed-loop information, a discrete-time formulation, disturbance models, and the start of unconstrained optimization. Administrative grading details and anecdotal historical claims are not needed to develop those ideas and are omitted. See the section-level source map for the lecture spans and added constructions.

The next mathematical question is how to reason about optimality when there are many decision variables and active constraints. The cart already shows why that question matters: a perfectly solved soft objective and an infeasible hard task can produce superficially similar numerical output, yet they imply different decisions.