AI agents, from action loops to policy optimization
These independent CMU 11-768 study chapters develop agent architecture, tool use, long-context computation, memory, planning, software and computer interaction, supervised fine-tuning, reinforcement learning, deep research agents, advanced RL algorithms, RL training systems, and agent safety. They are written for advanced graduate readers who have used agents but have not built them. A common mathematical model connects model proposals to authorization, execution, evaluation, training, and verification. Worked examples, executable definitions, limitations, exercises, and primary sources extend the recorded CMU 11-768 lectures.
Chapters 1–4 construct a single-run harness. Chapters 5–7 study how it plans and acts in complex environments. Chapters 8–9 explain how trajectory data changes the underlying policy. Chapter 10 applies that evaluation and training machinery to deep research agents. Chapters 11–12 extend training to stale data, clipped updates, unreliable rewards, and the systems that run them. Chapter 13 returns to the authorization boundary and asks how to limit, monitor, and test what an agent can do. A timestamp beside a section title opens the corresponding source segment. Research findings and primary-source citations are integrated into the exposition.
Cumulative Python harness
Each chapter develops part of one executable Python package. Chapters 1–4 define the run state, permission-checked tools, context management, and memory. Chapter 5 adds dependency plans, reliability diagnostics, replanning decisions, and work/span bounds. Chapters 6–7 add behavioral evaluation measures and coordinate transforms. Chapters 8–9 add masked supervised loss, effective data mixtures, reward-to-go, a Monte Carlo policy-gradient estimator, and group diagnostics. Chapter 10 adds chance-corrected rater agreement, citation accuracy and recall, and a contrastive retrieval loss. Chapter 11 adds generalized advantage estimates, importance-weight moments, PPO and CISPO gradient coefficients, and KL divergence. Chapter 12 adds batching makespans, bfloat16 rounding, and a tokenizer round trip. Chapter 13 adds cross-run reach through shared state, alert precision, and a zero-failure bound on harm rates. The package uses only the Python standard library and deterministic examples, so readers can reproduce every behavior without credentials.
Read the companion code guide and run python -m unittest discover -s tests -v from the repository root. The behavioral checks cover execution boundaries as well as the planning, evaluation, and training equations. The package is an educational harness. It is not a production runtime or a provider-specific integration.
Role of CMU 11-768
These chapters use the contents of the first thirteen available recordings of CMU 11-768: AI Agents, not merely the syllabus. The CMU 11-768 recordings determine the chapter sequence and supply substantive starting material, examples, distinctions, and empirical questions. The chapters reconstruct that material as independent textbook exposition, then develop it through explicit definitions, derivations, code, worked cases, exercises, and primary research.
| Chapter | Traceable lecture material | Further development in this edition |
|---|---|---|
| 1 · Agents | The interaction loop and training-versus-harness question. | A formal run state, an action-admission predicate, an authorization invariant, and a worked order-cancellation case. |
| 2 · Tools | The grocery-cart example, code as a meta-tool, and constrained tool-call generation. | Typed tool effects, explicit preconditions and postconditions, retry analysis, and a permission-checked implementation. |
| 3 · Long context | Key/value caching and context compaction. | A constrained context-selection model, explicit cost calculations, and a compaction state schema. |
| 4 · Memory and skills | Authored skill files, external memory, and skill induction. | A write/retrieve/apply policy model, temporal-validity fields, and a budgeted memory implementation. |
| 5 · Planning | task decomposition, replanning, and parallel execution. | Constraint-aware decomposition, plan repair, work/span bounds, and capability-aware allocation. |
| 6 · Coding agents | behavioral evaluation, repair loops, and repository benchmarks. | Outcome vectors, the pass@k estimator, repository localization, and layered verification. |
| 7 · Computer use | grounding versus task success, partial credit, and interface representations. | Partially observable control, evidence-sensitive rubrics, injection boundaries, and resettable evaluation. |
| 8 · Supervised fine-tuning | chat templates, data mixtures, and offline loss versus agent success. | Masked trajectory likelihood, token-weighted mixtures, covariate shift, packing, and harness robustness. |
| 9 · Reinforcement learning | the policy gradient, baselines, and group-relative advantages. | A full score-function derivation, variance reduction, temporal credit assignment, and sampling-policy analysis. |
| 10 · Deep research agents | rubric evaluation of long-form answers, exact-match reward, and retrieval quality. | Rubric scoring with judge-agreement audits, citation accuracy and recall, rubric-based RL rewards, and a contrastive retrieval objective. |
| 11 · Advanced RL algorithms | The value function, importance weights for stale rollouts, and reward reliability. | A squared-error derivation for the critic, the GAE telescoping identity, moment formulas for long-trajectory importance weights, the PPO gradient cases, a verifier-noise relation, and a shaping condition that keeps partial credit from rewarding inaction. |
| 12 · RL systems | KV and prefix caching, sequence extension and token-in, token-out, and asynchronous RL. | Token-level importance correction with truncation, a speculative-sampling proof, batching and pipeline-bubble bounds, bf16 update-absorption arithmetic, and a GPU-split formula for asynchronous training. |
| 13 · Agent safety | threat models, shared state across sandboxed runs, and monitoring. | A cross-run reach bound for shared state, per-role least-privilege grants, alert precision at low base rates, and a zero-failure bound on harm rates. |
There are 98 main sections across the thirteen chapters. Of these, 96 link to a relevant point in a recording. The two untimed sections synthesize material developed in the first four chapters. The ratio 96/98 measures section-level navigation back to lecture topics. It does not measure how much prose, mathematics, code, or interpretation came from CMU. That stronger claim would require a claim-level provenance ledger, which this edition does not provide.
The source manifest records the thirteen recordings used for this edition as of 17 October 2026. Their English captions were automatically generated, so technical terms were cross-checked against official slides and primary research sources. Timestamps identify a relevant segment. They do not attribute every statement under a heading to an instructor. The course playlist and official course site remain authoritative for what was taught. These independent study chapters are not official CMU course notes and are not endorsed by Carnegie Mellon University or the instructors. The repository does not redistribute captions or slides.
The implementation examples draw on official documentation from Anthropic, OpenAI, Cursor, Hermes Agent, and OpenClaw, checked on 12 September 2026. They illustrate architectural choices and are not comparative benchmarks.
How the material was developed
The development journey records the first edition's requests, revisions, failures, and checks. The style study of twelve PDFs and two writing guides explains the editorial methods used across this edition. The writing-revision record documents the earlier four-chapter style audit in before-and-after form.
Editable text versions: Chapter 1 · Chapter 2 · Chapter 3 · Chapter 4 · Chapter 5 · Chapter 6 · Chapter 7 · Chapter 8 · Chapter 9 · Chapter 10 · Chapter 11 · Chapter 12 · Chapter 13.