Primary references

• Stanford AA203 Spring 2026 official course schedule

  Course identity and the order of optimal-control and learning topics.

• Marco Pavone, AA203 lecture 1 recording

  Topic sequence and timestamp navigation for this chapter.

• Stanford AA203 lecture 1 official slides

  Feedback loop block diagram, home-heating example, disturbance and sensor-noise entry points (pp. 9-11); feedback desiderata (p. 12); performance, planning and learning gaps (p. 13); course map with DP, HJB/HJI, MPC bridge, adaptive optimal control, imitation learning and model-based/model-free RL (p. 14); three ingredients of the formulation (p. 18); component and compact state equations, control history and state trajectory (pp. 19-20); double-integrator model in s, a, A, B notation (pp. 21-24); boundary conditions and state/control bounds (p. 25); admissible control and trajectory (p. 26); performance measure with h, g and specified or free t_f (p. 27); optimal control problem, optimal trajectory-control pair, existence, uniqueness, global minimality and maximization convention (pp. 28-29); closed-loop policy with linear example F x and open-loop control (p. 30); discrete-time problem and J*(x_0) (pp. 31-32); unconstrained nonlinear program and local/global/strict minima (pp. 34-35); stationary points as filters (p. 43). Pages 36-43 are delivered in the lecture-2 recording and developed in the lecture-2 chapter.

• Stephen Boyd and Lieven Vandenberghe, Convex Optimization

  Finite optimization definitions, local/global distinction, attained optimum versus infimum, and convex optimality structure.

• James Rawlings, David Mayne, and Moritz Diehl, Model Predictive Control: Theory, Computation, and Design

  MPC uses repeated finite-horizon optimization to construct a feedback controller; stability and feasibility require additional conditions.

• Ross, Gordon, and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

  In sequential imitation learning, actions affect the distribution of future observations.

• Stanford AA203 lecture 3 official slides

  The interactive lab's connection to functional stationarity; its global-optimality proof is independently supplied.

• Stanford AA203 lecture 4 official slides

  A related double-integrator free-time objective motivates the separate lab discussion.

• NumPy trapezoid documentation

  Composite trapezoidal integration used for the independent numerical effort check.

• Stanford AA203 Spring 2026 official course schedule

  Course identity and lecture sequence.

• Marco Pavone, AA203 lecture 2 recording

  Actual delivered topic order, unconstrained methods and equality-constraint discussion.

• Stanford AA203 lecture 1 official slides

  Unconstrained necessary/sufficient conditions and convexity continued at the beginning of the second recording.

• Stanford AA203 lecture 2 official slides

  Iterative descent, gradient-method class, half-line decrease, angle reading and strict-decrease gradient descent (pp. 4-8); steepest-descent and Newton directions, minimization, constant and diminishing step rules, and methods left outside the course (pp. 9-11); penalty and feasible-direction viewpoints (p. 12); equality-constrained problem, Lagrange multiplier theorem, circle example and its solution, first-order feasible variations, regular points, penalty and elimination proof routes, pointer to second-order conditions, and Lagrangian form (pp. 14-21). Inequality pages 22-28 are delivered in recording 3 and developed in the lecture-3 chapter.

• Stanford ASL, AA203-Notes course notes repository

  Second-order necessary and sufficient conditions for equality-constrained problems (Lagrangian Hessian nonnegative, respectively positive, on the first-order feasible variations; sufficiency without regularity), to which official slides p. 19 refer; also names Bertsekas's Nonlinear Programming (2016) as the source of the proofs.

• Stephen Boyd and Lieven Vandenberghe, Convex Optimization

  Convexity, local/global optimality distinctions, descent and backtracking, equality-constrained quadratic systems, and value sensitivity.

• Stanford AA203 Spring 2026 official course schedule

  Course identity, lecture ordering and calculus-of-variations topic.

• Daniele Gammelli, AA203 lecture 3 recording

  Delivered inequality and calculus-of-variations sequence and verified timestamp navigation.

• Stanford AA203 lecture 2 official slides

  KKT notation, active constraints and the half-plane example completed in recording 3.

• Stanford AA203 lecture 3 official slides

  Roadmap placement of indirect methods (p. 2); indirect methods, their analytical advantage and the Kirk reading (p. 3); general optimal-control problem and finite-dimensional stationarity analogy (p. 4); calculus-of-variations definition (p. 5); functionals, domain and linear functionals (p. 6); norms and closeness (p. 7); local extrema and extremals (p. 8); increment, variation and differentiability (p. 9); fundamental theorem and its proof reference (p. 10); fixed-endpoint problem, single-integral increment, Taylor expansion, integration by parts, fundamental lemma and Euler equation (pp. 11-16). Pages 16-18 (Euler-equation type, shortest path, vector Euler equation) are delivered in recording 4 and treated in the lecture-04 chapter. The slides cite D. E. Kirk, Optimal Control Theory: An Introduction (Dover, 2004), Section 4.1, cited in the chapter without a verified URL.

• Stephen Boyd and Lieven Vandenberghe, Convex Optimization

  KKT conditions, convex sufficiency and multiplier sensitivity.

• NumPy trapezoid documentation

  Composite quadrature used to check the independently derived polynomial energy gap.

• Stanford AA203 Spring 2026 official course schedule

  Course sequence and indirect-method context.

• Daniele Gammelli, AA203 lecture 4 recording

  Delivered shortest-path, endpoint, costate, and free-time cart sequence.

• Stanford AA203 lecture 3 official slides

  Shortest-path example carried into the beginning of recording 4; p. 16 form of the Euler equation (nonlinear, time-varying, second order, split boundary conditions), recapped at 283.2-375.9 s of recording 4.

• Stanford AA203 lecture 4 official slides

  Roadmap placement of indirect methods (p. 2); calculus-of-variations boundary conditions with regularity assumptions, Euler equations, general boundary variation, Kirk Section 4.3 table and componentwise substitutions (pp. 3-8); constrained extrema, costate as multiplier function, and the augmented-integrand Euler rule (pp. 9-11); variational roadmap (p. 12); unbounded-control problem statement, Hamiltonian sign convention, necessary conditions and boundary variation (pp. 13-14, 19); Kirk Section 5.1 table including terminal surfaces and equation counts (pp. 15-16); free-time cart and fifth-root duration (pp. 17-18); proof sketch of the necessary conditions (pp. 20-23). The slides cite D. E. Kirk, Optimal Control Theory: An Introduction (Dover, 2004), cited in the chapter without a verified URL.

• NumPy trapezoid documentation

  Numerical integration used in the supporting trajectory laboratory.

• Daniel Liberzon, Discussion of the maximum principle

  Necessary versus sufficient conditions, multiplier nontriviality, abnormality, and sign conventions.

• Stanford AA203 Spring 2026 official course schedule

  Course and lecture order.

• Daniele Gammelli, AA203 lecture 5 recording

  Delivered bounded-control archetypes and beginning of computational methods.

• Stanford AA203 lecture 5 official slides

  Hamiltonian minimization, control-affine archetypes, diagonal input-cost assumption and solve_bvp example; also roadmap placement (p. 2), unbounded problem statement and terminal relation (pp. 4-5), one-sided variations (pp. 7-8), Kirk reference for the linear time-optimal case (p. 15), existence and uniqueness examples (p. 22), solve_bvp standard form with unknown parameters and Lobatto collocation (p. 25) and the Ascher-Russell reference (p. 26). Pages 24 and 26-27 are delivered in recording 6 and developed in chapters 4 and 6.

• Daniel Liberzon, Basic fixed-endpoint control problem

  Formal maximum-principle statement, objective/costate nontriviality and normal/abnormal multiplier distinction.

• SciPy solve_bvp official documentation

  Function and boundary-residual interfaces, collocation method, convergence status and residual semantics. Unknown-parameter vector p and the requirement that bc return n + k residuals.

• SciPy source, scipy/integrate/_bvp.py (main branch)

  Cubic C1 spline collocation at mesh nodes and interval midpoints, Simpson-form collocation residual y[1:] - y[:-1] - h/6 (f[:-1] + f[1:] + 4 f_middle), Lobatto IIIA family, and n + k boundary conditions.

• AA203 lecture 6 recording

  Actual delivered topic order and timing.

• AA203 lecture 6 official slides

  Course placement and general problem (pp. 2-3); free-time solve_bvp setup (p. 3); Kelly and Rao resources (p. 4); method names (pp. 5, 13); nonuniform-grid simultaneous and shooting programs (pp. 6, 10); Zermelo programs, solvers and plotted results (pp. 7-13); Hermite-Simpson, pseudospectral and RK4 parameterizations (p. 14); sequential convex programming loop and discretized subproblem (pp. 15-18); Malyuta et al. flowchart with artificial infeasibility and unboundedness (p. 19); warm starts, penalty and augmented Lagrangian methods (p. 20); local-versus-global wrap-up (p. 21).

• AA203 lecture 5 official slides

  Free-time normalization continued at the beginning of recording 6.

• Kelly, Introduction to Trajectory Optimization

  Primary tutorial on direct collocation representations.

• Malyuta et al., Convex Optimization for Trajectory Generation

  Distinction among lossless convexification, SCvx and GuSTO.

• SciPy solve_bvp documentation

  Unknown-parameter and boundary-residual interfaces.

• Tedrake, Underactuated Robotics, ch. 10 Trajectory Optimization

  Quoted remark on the practical sensitivity of direct trajectory optimization (cited on official slides p. 20).

• AA203 lecture 7 recording

  Delivered dynamic-programming and LQR topic sequence and graph example.

• AA203 lecture 7 slides

  Roadmap placement (p. 2); discrete-time problem and closed-loop policy (p. 3); principle of optimality, its proof and formal tail-subproblem statement (pp. 4-6); concatenation and backward computation (pp. 7-8); shortest-path example (p. 9); dynamic-programming theorem (p. 10); discretization, quantization, global-minimum, constraint and curse-of-dimensionality remarks (p. 11); discrete LQR model, first- and second-order conditions, gain and Riccati recursion in F_k notation (pp. 12-17).

• Bertsekas, basic DP and state augmentation

  State augmentation and principle-of-optimality modeling.

• Boyd, discrete finite-horizon LQR

  Independent primary LQR and least-squares formulations.

• AA203 lecture 8 recording

  Tracking, iLQR and DDP delivered topic order.

• AA203 lecture 8 official slides

  Course roadmap and agenda (pp. 2-3); LQR with cross-term cost and its linear policy and quadratic value (p. 4); generalized cost and affine dynamics in augmented [x;1] form (p. 5); linear tracking objective and deviation variables (p. 6); nonlinear tracking by linearization (p. 7); nonlinear optimal control problem, iLQR linearization and cost Taylor expansion, iteration and backward/forward passes with nonlinear rollout (pp. 8-11); SCP loop compared with iLQR (p. 12); deviation penalty, line search, termination, local minima and initialization, regularization of indefinite curvature, and Tassa's thesis Section 2.2.3 (p. 13); iLQR versus DDP approximation order, Q-function and its second-order expansion, optimal correction l_k + L_k delta_x_k, and the DDP second-order dynamics terms (pp. 14-18).

• Harrison and Buurmeijer, Optimal and Learning-Based Control: Course Notes (version of April 12, 2026)

  Reading named on official slides p. 3: Section 3.1 (discrete LQR, LQR with bilinear cost and affine dynamics, tracking LQR) and Section 3.2 (iterative LQR, DDP, algorithmic details), as listed in the notes' table of contents.

• Boyd, finite-horizon LQR

  Tracking and time-varying LQR context.

• Tassa, Erez, Todorov, online trajectory optimization

  Primary derivative, regularization and line-search implementation context.

• Tassa, Theory and Implementation of Biomimetic Motor Controllers, doctoral thesis, Hebrew University of Jerusalem, 2011

  Section 2.2.3, named on official slides p. 13: Levenberg-Marquardt control regularization attributed to Jacobson and Mayne (1970), state-based regularization f_u^T(V_xx + mu I)f_u with non-vanishing gains, gain-consistent value update attributed to Todorov and Li (2005), mu increase on non-positive-definite curvature and decrease after success, expected-reduction line search with acceptance window c1 < z < c2.

• Pavlov, Shames, Manzie, Interior Point Differential Dynamic Programming

  Explicit constrained DDP variants exist beyond the basic unconstrained derivation.

• AA203 lecture 9 recording

  Delivered stochastic-control, inventory, LQR and Q-value sequence.

• AA203 lecture 9 slides

  Course placement (p. 2); finite-horizon MDP, slide notation and Powell citation (pp. 3-5); stochastic principle of optimality and DP algorithm (pp. 6-8); inventory data, recursion and stage-zero values (pp. 9-11); stochastic LQR in slide notation (pp. 12-14); stationary MDP tuple, discounted objective, Bellman and fixed-policy equations and Q functions (pp. 15-18); known-model versus model-based and model-free solution (p. 19). Value-iteration and policy-iteration slides (pp. 20-23) are delivered in recording 10 and not attributed to this recording.

• Bertsekas, basic stochastic DP

  Conditional stochastic dynamic-programming framework.

• Boyd, stochastic LQR

  Quadratic-plus-constant value and covariance trace recursion.

• AA203 lecture 10 recording

  Actual algorithm-to-reachability sequence.

• AA203 lecture 9 slides

  Value and policy iteration notation; model-based versus model-free solution of a Markov decision process (p. 19) and value iteration for Q functions (p. 21).

• AA203 lecture 10 slides

  Hamilton–Jacobi equations and reachability definitions; roadmap placement and nonlinear-PDE characterization (pp. 2-3), continuous-time problem notation (p. 4), second player and joint state (p. 5), open-loop and nonanticipative information patterns (p. 6), game dynamic-programming principle and HJI derivation steps (pp. 7-16), HJI and HJB feedback laws (pp. 17-18), continuous LQR with V(t) notation and infinite-horizon limit (pp. 19-20), applications of differential games (pp. 21-22), reachability inputs, outputs and set definitions (pp. 23-26); pp. 27-34 support the lecture 11 chapter.

• Bertsekas, dynamic programming lecture notes

  Discounted fixed-point algorithms.

• Bansal et al., Hamilton–Jacobi Reachability: A Brief Overview and Recent Advances

  Differential-game information pattern, assumptions and generalized Hamilton–Jacobi solutions.

• Chen and Tomlin, Hamilton–Jacobi Reachability: Some Recent Theoretical Advances and Applications in Unmanned Airspace Management

  Named reading on official slides p. 3; cited in #hji for the formal treatment of the game dynamic-programming principle. DOI resolved this session (302 to annualreviews.org) and Crossref metadata confirmed Annual Review of Control, Robotics, and Autonomous Systems 1:333-358, 2018; the publisher page itself returned HTTP 403.

• AA203 lecture11 recording

  Delivered reachability-to-MPC sequence.

• AA203 lecture10 slides

  Level sets, tubes and relative-vehicle demonstration. Routed pp. 30, 33 and 34: avoid and reach sets, values, HJI equations and feedback laws (#level); identical-vehicle game with bounded turn rates and evader/pursuer roles (#relative); (n+1)-dimensional grid and alternatives to grid solvers (#scale).

• AA203 lecture11 slides

  MPC and invariance definitions; later proof not attributed to this recording. Roadmap placement of MPC (p. 2), Borrelli-Bemporad-Morari and Rawlings-Mayne-Diehl readings (p. 3), receding horizon with control and prediction horizons (pp. 4-6), polyhedral LTI regulation problem and prediction notation (pp. 7-13), terminal cost and terminal set as design choices (p. 9), quadratic and one-/infinity-norm costs (p. 14), online algorithm with infeasibility branch (p. 15), feasibility and stability issues and analysis strategy (pp. 16-18), X_0, one-step controllable sets, control/positive invariant and maximal sets, MPT3 (pp. 19-21), truncated set X_1 and feasibility lemma (p. 22); pp. 23-25 are delivered in the lecture 12 recording.

• StanfordASL hj_reachability

  Primary reachability solver and value-sublevel formulation.

• Rawlings, Mayne and Diehl, Model Predictive Control

  Primary MPC textbook named in recording.

• Multi-Parametric Toolbox 3 (MPT3)

  Open-source MATLAB toolbox for parametric optimization, computational geometry, and model predictive control, including invariant-set computations; named on official slides p. 21 for computing control and positive invariant sets (#invariance).

• Ian M. Mitchell, A Toolbox of Level Set Methods (Version 1.1), UBC CS TR-2007-11, 2007

  ToolboxLS manual; Section 2.6.1 'The Game of Two Identical Vehicles' gives the relative dynamics, the evader turn rate as control avoiding the collision cylinder and the pursuer turn rate as disturbance, and the example radius 5, speeds 5, turn-rate bounds 1 (#relative); credited on lecture_10.pdf p. 33.

• AA203 lecture12 recording

  Delivered proof, design and tracking sequence.

• AA203 lecture11 slides

  Deferred truncated-set lemma.

• AA203 lecture12 slides

  Backward-induction feasibility proof and terminal-set role (pp. 4-7), Lyapunov and MPC stability theorems with assumptions A0-A3 (pp. 8-15), Lyapunov- and Riccati-based terminal ingredients with maximal positively invariant sets (pp. 16-18), constrained LQR and explicit piecewise-affine MPC (pp. 19-20), design recipe and MPT3 (p. 21), input-increment tracking problem (pp. 22-23).

• Rawlings, Mayne and Diehl, Model Predictive Control

  Primary MPC theory and terminal assumptions.

• Bemporad et al., The explicit linear quadratic regulator for constrained systems

  Primary constrained-LQR piecewise-affine control result.

• Multi-Parametric Toolbox 3 (MPT3)

  Open-source MATLAB toolbox for parametric optimization, computational geometry, and MPC, including invariant-set computation and explicit MPC synthesis; named on official slides p. 21 and cited in #recipe.

• AA203 lecture13 recording

  Actual system-identification and MRAC sequence.

• AA203 lecture13 slides

  Regression and adaptive equations, with listed corrections; roadmap placement (p. 2); feedback loop with an unknown plant (pp. 5-6); uncertainty responses and examples (p. 7); direct versus model-based method families and learning settings (pp. 8-10); offline identification pipeline (p. 12); least squares, first-order model, recursive and continuous-time forms (pp. 13-14); Gauss-Markov and Gaussian likelihood statements (p. 15); mean, covariance and information-matrix derivation with the convergence criterion (pp. 16-19); performance questions (p. 20); definition and classical aim of adaptive control (pp. 22, 25); global Lyapunov theorem and mass-spring-damper energy (pp. 23-24); MRAC ingredients, known- and unknown-mass laws, adaptation law and Lyapunov candidate (pp. 26-30); role of the reference signal in parameter convergence and the Slotine-Li Applied Nonlinear Control Chapter 8 reference, cited in text without a link (p. 31). MIAC pages 32-33 are delivered in lecture 14.

• Simchowitz et al., Learning Without Mixing

  Primary finite-sample identification analysis for dependent single trajectories.

• AA203 lecture14 recording

  Delivered adaptive-to-IL/RL sequence.

• AA203 lecture13 slides

  Deferred MIAC material.

• AA203 lecture14 slides

  Supervised, imitation and reinforcement-learning notation; course roadmap (p. 2); motivating systems and RLHF steps (pp. 3-4); three routes to control (p. 5); resources (p. 7); supervised mapping, regression and likelihood (pp. 8-10); IL definition, BC and IRL families, BC recipe and ALVINN (pp. 11-14); IL-versus-SL differences (p. 15); IRL warehouse example (p. 16); RL loop, elements, motivation and characteristics (pp. 18-22); MDP, trajectory distribution and chess/quadruped examples (pp. 23-24); value functions (p. 25); taxonomy, tradeoffs, sample efficiency, stability, IL/RL remarks and staged pipelines (pp. 26-30).

• Pomerleau, ALVINN: An Autonomous Land Vehicle in a Neural Network

  Primary early neural road-following system; architecture (30x32 video and 8x32 range retinas plus a feedback unit, 1217 inputs, 29 hidden units, 45 curvature outputs plus a feedback output), training on 1200 simulated road images with hill-shaped targets, accuracy within two units about 90% of the time on new simulated images, and the NAVLAB test at 0.5 m/s over 400 m.

• Gammelli, Lorenzetti, Luo, Zardini, Pavone, Principles of Robot Autonomy (pre-print; version dated April 11, 2026)

  Chapters 17 (sequential decision making and dynamic programming), 18 (reinforcement learning) and 19 (imitation learning), named as resources on official slide p. 7.

• Ouyang et al., Training language models to follow instructions with human feedback (arXiv:2203.02155, 2022)

  Three-step RLHF procedure depicted on official slide p. 4 (demonstrations and supervised fine-tuning of GPT-3; comparison data and reward model; PPO against the reward model); rankings of K = 4 to 9 outputs; pairwise logistic reward-model loss; per-token KL penalty from the SFT model to mitigate reward-model over-optimization; one-prompt, one-response episodes.

• AA203 lecture15 recording

  Full delivered imitation-learning sequence.

• AA203 lecture15 slides

  Algorithm, data and distribution model structure. Behavior-cloning objective (p. 4); covariate shift and the cited quadratic bound (p. 5); multimodality from multiple experts (p. 6); indexed DAgger loop, iterative supervised learning, and confidence-based and human-gated variants (pp. 10-11); NVIDIA architecture and training pipeline and the trail-camera case (pp. 13, 15-17); finite observation window and sequence models (pp. 20-21); categorical and Gaussian heads, mixtures, conditional VAEs, discretize-plus-autoregressive factorization, and diffusion (pp. 20, 22-24); flow-matching recipe (p. 25); chunk loop (p. 26); Diffusion Policy (p. 27); RT-1 (p. 28); behavior-cloning assessment (p. 30); RvS filtering, weighting, and conditioning objectives (pp. 32-34); IRL loop, reward ambiguity, and named methods (pp. 36-37).

• Ross, Gordon and Bagnell, DAgger

  Original data-aggregation and no-regret reduction.

• Bojarski et al., End to End Learning for Self-Driving Cars

  Primary camera-to-steering case study.

• Lipman et al., Flow Matching for Generative Modeling

  Vector-field regression and generative transport framework.

• Chi et al., Diffusion Policy

  Action-distribution modeling and receding-horizon execution.

• Emmons et al., RvS

  Outcome-conditioned supervised approaches to offline RL.

• Ross and Bagnell, Efficient Reductions for Imitation Learning

  AISTATS 2010, PMLR 9:661-668; source of the quadratic-in-horizon bound cited on official slides p. 5, stated in the chapter as J(pi) <= J(pi_E) + H^2 epsilon under 0-1 loss on the expert's state distribution with per-stage cost in [0,1], as restated in the DAgger paper.

• Tomczak, Flow Matching: Matching flows instead of scores

  Tutorial named as the figure source on official slides p. 25; uniform time sampling, interpolation regression, and forward-Euler sampling.

• Brohan et al., RT-1: Robotics Transformer for Real-World Control at Scale

  Official slides pp. 20 and 28: six-image history, FiLM-conditioned EfficientNet-B3, TokenLearner (81 to 8 tokens per image), 8-layer 19M-parameter decoder-only transformer of a 35M-parameter model, 11 action dimensions with 256 uniform bins and cross-entropy, 3 Hz control, 130k episodes over 700+ tasks with 13 robots in 17 months, and the ablation in which autoregressive conditioning on actions did not help and slowed inference by more than 2x.

• AA203 lecture 16 recording

  Substantive topic sequence and navigation timestamps.

• AA203 lecture 16 official slides

  Course roadmap (p. 2); MDP tuple, control objective and Bellman optimality and expectation equations for V and Q (pp. 4-6); exact-method table, policy iteration, gridworld sweeps, converged values and greedy policies (pp. 7-14); Monte Carlo return, first-visit and every-visit counters and incremental means (pp. 16-21); blackjack rules, value surfaces and optimal policy (pp. 19-20, 37); TD target and error, MC-TD comparison, bias and variance, backup diagrams and the width-depth square (pp. 23-31); GPI, two doors and epsilon-greedy exploration (pp. 33-36); taxonomy objective and algorithm skeleton (pp. 40-41). All 42 pages read as rendered images.

• Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018

  Gridworld Example 4.1 and Figure 4.1 (converged random-policy values as negative expected steps to termination; greedy policies optimal from the third sweep); blackjack Example 5.1 (rewards, dealer sticks on 17 or more, gamma = 1, infinite deck with replacement, usable-ace states less certain at 10,000 episodes); convergence of first-visit and every-visit Monte Carlo with the Singh and Sutton (1996) attribution (Section 5.1); epsilon-soft improvement proof and its epsilon-soft optimality limit (Section 5.4); optimal blackjack policy found by Monte Carlo with exploring starts and its agreement with Thorp's basic strategy except the leftmost usable-ace notch (Example 5.3, Figure 5.2).

• UC Berkeley CS 185/285 Deep Reinforcement Learning

  Course named on official slides p. 41 as the source of the three-process RL algorithm skeleton; course number, title and institution verified on the page.

• Sutton, Learning to Predict by the Methods of Temporal Differences

  Learning from successive predictions and special-case analysis; no universal nonlinear guarantee.

• Tsitsiklis, Asynchronous Stochastic Approximation and Q-Learning

  Explicit assumptions are necessary for asynchronous stochastic-approximation convergence.

• AA203 lecture 17 recording

  Delivered topic order and examples.

• AA203 lecture 17 official slides

  Algorithms, cliff comparison and DQN stabilization; TD advantages and TD inside GPI (p. 8), SARSA quintuple and algorithm order (pp. 9-10), windy gridworld layout, hyperparameters and Monte Carlo question (pp. 11-12), on-policy and off-policy definitions, alternative-successor target and greedy substitution (pp. 13-16), Q-learning algorithm and convergence statement (p. 17), cliff grid, fall convention and learning-curve levels (p. 18), backgammon and Go state counts (p. 20), table limits and approximator interfaces (pp. 21-22), mean-squared value error, gradient step and Monte Carlo and TD targets (pp. 23-25), approximate GPI and fitted Q-learning skeleton (pp. 26-29), DQN architecture, failure modes and replay loss (pp. 30-31); recap and placement pages 2-7, 28 and 32 are developed in the lecture 16 chapter.

• Human-level control through deep reinforcement learning

  DQN, experience replay and target network.

• Reinforcement Learning: An Introduction, second edition

  SARSA, Q-learning, cliff walking and tabular convergence context.

• AA203 lecture 18 recording

  Delivered topic order and explanations.

• AA203 lecture 18 official slides

  Roadmap and RL taxonomy (pp. 2-3); trajectory distribution, objective and sampled evaluation (pp. 5-7); likelihood-ratio derivation and REINFORCE loop (pp. 8-11); maximum-likelihood comparison (pp. 12-13); variance, causality and baselines with a training comparison (pp. 15-19); on-policy question (p. 20); actor-critic, advantage, loop comparison and critic regression targets (pp. 22-27); architecture design (p. 28); applications (pp. 34-36). Pages 30-33 (surrogate, TRPO, PPO) are delivered in recording 19 and support lecture 19.

• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

  REINFORCE score-function algorithm.

• Mastering the game of Go with deep neural networks and tree search

  AlphaGo policy/value networks, self-play and tree search.

• Asynchronous Methods for Deep Reinforcement Learning

  Asynchronous actor-critic with parallel actor-learners, their stabilizing effect on training, and navigation of random 3D mazes from visual input (Mnih et al., ICML 2016, PMLR 48:1928-1937).

• Training language models to follow instructions with human feedback

  Supervised fine-tuning on labeler demonstrations, a reward model trained from labeler comparisons, PPO fine-tuning of the supervised model against the reward model with a per-token KL penalty from the supervised model, and the one-episode-per-prompt framing (Ouyang et al., 2022).

• Solving Rubik's Cube with a Robot Hand

  Control policies trained with PPO only in simulation and used on a physical humanoid robot hand, automatic domain randomization, and a recurrent policy with a separate value network (OpenAI et al., 2019).

• AA203 lecture 19 recording

  Delivered topic order and course synthesis.

• AA203 lecture 19 official slides

  Equations, terminology and diagrams; also supports the reduction of RL to optimization, L^PG and L^IS, TRPO limits, PPO v1 penalty form, model-free review (pp. 8-10), trade-offs and sample-efficiency ordering, covariate-shift example, capacity tension, entropy versus epistemic uncertainty, posterior contraction, Dirac-mixture ensembles, Monte Carlo planning score, PETS experiments, and the course roadmap and autonomy stack.

• Trust Region Policy Optimization

  TRPO objective and trust-region context.

• Proximal Policy Optimization Algorithms

  PPO penalty and clipped objectives.

• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

  PETS architecture and empirical sample-efficiency evidence.

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization

  Wilson and Izmailov (2020): deep ensembles as approximate Bayesian marginalization over a multimodal posterior, contrasted with single-mode variational inference and MultiSWAG (official slide p. 24).