AA203: lecture textbook

A route from physical models to learned policies

Chapter 1 builds a bounded cart model and asks what an admissible controller can accomplish. Chapters 2–5 turn that question into necessary and sufficient optimization conditions: finite-dimensional constraints, variations, costates, free final time, and bounded controls. Keep the distinction between a necessary condition, a local solution, and a globally optimal admissible trajectory visible throughout.

Chapters 6–8 move from mathematical conditions to computation: transcription, Bellman recursion, LQR, tracking, and local nonlinear optimization. Reproduce the cart calculations and compare the discrete problem with its continuous limit. Chapters 9–10 add uncertainty and connect discrete Bellman equations to continuous Hamilton–Jacobi equations; the order of expectation, optimization, and adversarial choice now matters as much as the algebra.

Chapters 11–12 connect reachability, constraint satisfaction, recursive feasibility, and stability. A trajectory that looks safe over a finite prediction horizon is not automatically safe forever. Work through the terminal-set shift proof and the Lyapunov decrease argument before treating MPC as a reliable feedback architecture.

Chapters 13–15 ask what changes when a model or policy is learned from data. Identification requires informative excitation; tracking convergence does not imply parameter convergence. Demonstration accuracy does not ensure accuracy on the states a learned controller visits. The corrective-data and generative-policy chapters make those gaps concrete without attributing model-based stability guarantees to a sampled action generator.

Chapters 16–19 develop reinforcement learning from sampled values and temporal differences through value-based learning, policy optimization, and learned dynamics. Follow one question across all four: which expectation or model is being approximated, with what data, and what changes when the policy changes? The final chapter returns to planning with learned models, closing the loop with the numerical and robustness questions introduced earlier.

Use the examples as experiments

Read a derivation once, reproduce its small numerical example, then change a parameter that threatens an assumption. Try shortening a feasible transfer time, removing excitation from an identification experiment, or increasing the action chunk executed before feedback. Each chapter includes six exercises with complete expandable solutions. Attempt the exercise before opening its solution; print expands solutions automatically.

The interactive numerical lab compares minimum-energy motion with an acceleration limit. The accompanying Python lab and executable examples in every chapter provide further independently checkable fixtures. They demonstrate particular mathematical claims; they do not certify a physical robot, establish a universal learning guarantee, or reproduce the performance of a cited research system.

Provenance and reading conventions

Timestamp links beside major section headings return to the relevant part of the original recording. Acquisition used local machine transcription rather than publisher captions. Mathematical notation and apparent transcription errors were checked against the official lecture material; remaining qualifications are recorded in the chapter metadata. Supporting books and original papers appear in the references. Raw audio, transcripts, and slide copies are not bundled with this edition.

Most deterministic control chapters minimize cost; reinforcement learning chapters commonly maximize discounted reward. Each transition states its convention. Continuous time, discrete environment time, and the auxiliary transport time of a generative policy are different variables. Keep the model, objective, constraints, information pattern, and horizon together when comparing two methods.