Lecture 9 — Reinforcement-learning foundations for agents

Independent study chapter · Based on a CMU 11-768: AI Agents lecture by Daniel Fried, 22 September 2026 · Course and attribution

0. Demonstration likelihood and task reward are different objectives 00:00

Consider a number-search agent. A hidden integer lies between 1 and 16. The agent may guess a number and observe “higher,” “lower,” or “correct.” It receives reward one if it finds the number within four guesses and zero otherwise. A supervised demonstration can teach one successful sequence. Reinforcement learning uses sampled outcomes to increase expected success, including outcomes generated by the current policy.

Reinforcement learning (RL) adjusts a policy using rewards obtained from interaction. A policy $\pi_\theta(a\mid h)$ is a probability distribution over action $a$ given history $h$, parameterized by model weights $\theta$. A reward is a scalar signal defined by the task. High reward means preferred behavior only to the extent that the reward correctly represents the task.

Supervised fine-tuning (SFT) minimizes negative log probability of demonstrated actions. For demonstration $\tau^*=(a_0^*,\ldots,a_{T-1}^*)$ with histories $h_t^*$, its action loss is

$$\mathcal L_{\rm SFT}(\theta) =-\sum_{t=0}^{T-1}\log\pi_\theta(a_t^*\mid h_t^*).$$

This objective increases likelihood of the recorded actions. It does not directly distinguish an optimal demonstration from a barely successful one, use failed attempts, or expose the model to all histories created by its own errors.

RL instead defines expected return. The rest of the chapter derives a gradient for that objective and explains its limitations for agent trajectories.

The number-search task has a solvable reference policy

The hidden number is uniformly distributed over sixteen possibilities. Each response partitions the remaining candidates into numbers below the guess, the guess itself, and numbers above it. Four yes-or-no-equivalent comparisons can distinguish at most $2^4=16$ possibilities. A balanced binary-search policy can therefore succeed within four guesses, while a sequential policy that guesses 1, 2, 3, 4 cannot.

Let $C_t$ be the candidate set after history $h_t$. A deterministic reference policy chooses a median of $C_t$ and updates the set according to the observation. If guess $a_t$ receives “higher,” then $C_{t+1}=\{n\in C_t:n>a_t\}$. The “lower” case is analogous. “Correct” terminates the episode.

This reference supplies three useful objects. It defines a reachable success rate of one under the stated task. It produces demonstrations for SFT. It also provides a value baseline for analyzing sampled policies. RL does not require the reference policy, but without a known reference it is harder to tell whether optimization found the best attainable behavior or only improved a weak initialization.

The task also exposes why reward and policy constraints are distinct. A policy that queries an external hidden-state leak could receive reward one while violating the intended rules. The environment must restrict the action set and hide the answer. Reward evaluates outcomes within the permitted interaction protocol. It does not define that protocol by itself.

1. Trajectories couple a policy to an environment 03:00

Let $s_t\in\mathcal S$ be hidden environment state, $o_t\in\mathcal O$ an observation, and $a_t\in\mathcal A$ an action. Let $r_t\in\mathbb R$ be the scalar reward after step $t$, where $\mathbb R$ is the set of real numbers. A finite trajectory is

$$\tau=(o_0,a_0,r_0,o_1,a_1,r_1,\ldots,o_T).$$

The history before action $a_t$ is $h_t=(o_0,a_0,\ldots,o_t)$. The agent conditions on history rather than hidden state. Let initial observation distribution be $\mu(o_0\mid x)$ for task instance $x$. Let environment transition-observation probability be $P_{\rm env}(o_{t+1},r_t\mid h_t,a_t)$. The trajectory probability is

$$p_\theta(\tau\mid x)=\mu(o_0\mid x) \prod_{t=0}^{T-1} \pi_\theta(a_t\mid h_t) P_{\rm env}(o_{t+1},r_t\mid h_t,a_t).$$

Only the policy factors depend on $\theta$ when the environment is fixed with respect to the model parameters. The environment can be stochastic and non-differentiable.

For discount factor $\gamma\in[0,1]$, define trajectory return

$$R(\tau)=\sum_{t=0}^{T-1}\gamma^tr_t.$$

In the number-search task, reward arrives only at termination and $\gamma=1$, so return is one for success and zero for failure. In a long agent task, intermediate rewards can express partial progress. Poorly designed intermediate rewards can also be exploited.

The expected-return objective for task distribution $D$ is

$$J(\theta)=\mathbb E_{x\sim D} \mathbb E_{\tau\sim p_\theta(\cdot\mid x)}[R(\tau)].$$

The outer expectation averages tasks. The inner expectation averages trajectories generated by the policy and environment. Maximizing $J$ changes the probability of sampled behavior rather than copying a fixed demonstration set.

Support determines what on-policy learning can discover

The support of policy $\pi$ at history $h$ is the set of actions assigned positive probability. If every successful trajectory requires an action outside the policy's support at some reachable history, on-policy sampling can never discover success. Formally, if $p_\theta(\tau)=0$ for every $\tau$ with $R(\tau)>0$, then $J(\theta)=0$ and the sampled REINFORCE estimator is zero.

Language models usually assign nonzero probability to many token sequences, but operational support is narrower. Decoding can mask invalid actions, truncate long arguments, or make a valid trajectory astronomically unlikely. A cold-start policy that emits parseable calls and occasionally succeeds can be more valuable for RL than one with slightly lower token loss but no executable behavior.

Let $p$ be the probability of a successful rollout. With $N$ independent rollouts, the probability of observing at least one success is $1-(1-p)^N$. At $p=0.01$ and $N=100$, this probability is approximately $0.634$. At $p=10^{-4}$, it is approximately $0.010$. Sparse reward changes the required environment budget by orders of magnitude.

Exploration mechanisms alter the sampled policy. Temperature, top-$p$ truncation, action masks, and tool-error recovery all affect $p_\theta(\tau)$. They belong in the training specification because the policy-gradient estimator is defined with respect to the distribution that actually generated the trajectory.

Exploration can be part of the objective or the sampler

Sampling at higher temperature broadens the rollout distribution without changing the written reward objective. Entropy regularization instead changes the optimization objective. For a history distribution induced by the current policy, define conditional action entropy

$$\mathcal H(\pi_\theta(\cdot\mid h))=-\sum_a\pi_\theta(a\mid h)\log\pi_\theta(a\mid h).$$

An entropy-regularized objective is

$$J_\alpha(\theta)=J(\theta)+\alpha\,\mathbb E_h[\mathcal H(\pi_\theta(\cdot\mid h))],$$

where $\alpha\ge0$ is an exploration weight. Positive $\alpha$ rewards a broader action distribution. It can prevent premature collapse, but it can also increase malformed or unsafe attempts. The action-admission layer remains active during exploration.

The sampler and objective must not be conflated. If rollouts use temperature-scaled policy $\mu$ while the gradient treats them as samples from unscaled policy $\pi_\theta$, the update is off-policy. Either optimize the actual sampling policy or apply a justified correction. Top-$p$ truncation changes support and makes exact correction impossible for actions it removes.

For number search, useful exploration changes the early split points while respecting the current candidate set. Random guesses outside $C_t$ are dominated because the history already rules them out. A structured action mask can eliminate them without reducing the support of any valid optimal policy. This is a property of the known task model. In an open environment, an aggressive mask can remove an action needed for recovery.

2. Why SFT, filtered self-training, and RL differ 10:00

SFT observes expert histories. At inference, the model can enter histories absent from training. This train–deployment mismatch is often called exposure bias. On-policy RL samples from the current policy, so its training histories include current mistakes. This benefit comes with higher cost and instability.

Reinforced self-training (ReST) generates trajectories, scores them, and trains on a selected subset using an SFT-like loss. ReST separates data generation from offline improvement. ReST-EM, where EM denotes expectation-maximization, repeats generation, binary filtering, and fine-tuning.

For binary reward, let $D_\theta^+=\{\tau:R(\tau)=1,\tau\sim p_\theta\}$ be successful sampled trajectories. A filtered loss is

$$\mathcal L_{\rm filter}(\theta') =-\mathbb E_{\tau\sim D_\theta^+} \sum_t\log\pi_{\theta'}(a_t\mid h_t).$$

The data-generating parameters $\theta$ and updated parameters $\theta'$ are distinguished because offline training may reuse a fixed batch. This method ignores relative quality among retained trajectories and learns nothing from a batch with no successes. It can nevertheless be effective when rewards are verifiable and successful samples are frequent enough.

SFT, filtered self-training, and RL perform different projections

SFT projects a model toward a fixed demonstration distribution by minimizing forward cross-entropy on recorded actions. Filtered self-training first changes the dataset by conditioning on a reward event, then performs another likelihood projection. On-policy RL changes action probabilities using reward-weighted score gradients under the current trajectory distribution.

The distinctions matter when several trajectories succeed. SFT on one reference increases probability of that reference. Binary filtering increases probability of every retained sample in proportion to its token count and sampling frequency, unless reweighted. Policy gradient increases probability according to return or advantage, including relative differences among successful trajectories when reward represents them.

Suppose two number-search trajectories both succeed, but one uses three guesses and the other uses four. With binary reward one, plain REINFORCE treats their terminal returns equally. A reward such as $1-0.05T$ distinguishes them by length $T$. This modification is valid only if shorter search is truly preferred. Otherwise it changes the task objective for the sake of optimization convenience.

ReST and ReST-EM make generation and offline improvement explicit. Their success depends on a verifier that can label generated samples and on a sampling policy that produces useful positives. The generated dataset should retain failures and scores for analysis even if the training stage consumes only selected positives.

3. The score-function identity gives the policy gradient 32:00

For a fixed task instance, write the expected return as a sum over trajectories:

$$J(\theta)=\sum_\tau p_\theta(\tau)R(\tau).$$

Differentiate with respect to $\theta$:

$$\nabla_\theta J(\theta) =\sum_\tau R(\tau)\nabla_\theta p_\theta(\tau).$$

For any positive differentiable probability $p_\theta(\tau)$,

$$\nabla_\theta p_\theta(\tau) =p_\theta(\tau)\nabla_\theta\log p_\theta(\tau).$$

Substitution converts the sum into an expectation:

$$\nabla_\theta J(\theta) =\mathbb E_{\tau\sim p_\theta} \left[R(\tau)\nabla_\theta\log p_\theta(\tau)\right].$$

This is the score-function identity. The vector $\nabla_\theta\log p_\theta(\tau)$ is the score of trajectory $\tau$.

The environment factors do not depend on $\theta$, so

$$\nabla_\theta\log p_\theta(\tau) =\sum_{t=0}^{T-1}\nabla_\theta \log\pi_\theta(a_t\mid h_t).$$

Combining the equations yields the REINFORCE gradient:

$$\nabla_\theta J(\theta)= \mathbb E_\tau\left[ R(\tau)\sum_{t=0}^{T-1} \nabla_\theta\log\pi_\theta(a_t\mid h_t) \right].$$

The derivation does not differentiate through sampled actions or environment transitions. It increases log probability of actions on high-return trajectories. With zero reward, the basic estimator gives no update for that trajectory; it does not automatically push failed actions down.

For a language-model action consisting of several tokens, its log probability is the sum of token log probabilities. Agent policy-gradient implementations can therefore reuse token-level likelihood computation with masks and scalar coefficients.

Given $N$ sampled trajectories, a Monte Carlo estimator is

$$\widehat g=\frac1N\sum_{i=1}^N R_i\nabla_\theta\log p_\theta(\tau_i).$$

It is unbiased under on-policy sampling and the assumptions above. It can have high variance.

What the derivation assumes

The policy-gradient identity assumes that reward does not depend directly on $\theta$ except through the sampled trajectory, and that exchanging gradient and expectation is valid. It assumes the environment transition law is fixed while differentiating. A learned reward model updated simultaneously with the policy violates this stationary notation unless its parameters are treated separately.

The identity also requires the logged probability to match the sampling policy. If the runtime applies an action mask, rejection sampling, or post-generation repair, the effective policy differs from the raw model distribution. Logging only unadjusted token probabilities yields the wrong importance ratios and can bias the update.

For a language-model action $a_t=(x_{t,1},\ldots,x_{t,L_t})$, the action log probability is

$$\log\pi_\theta(a_t\mid h_t)=\sum_{j=1}^{L_t}\log p_\theta(x_{t,j}\mid h_t,x_{t,<j}),$$

where $L_t$ is the number of generated action tokens and $x_{t,<j}$ denotes earlier tokens in that action. Multiplying one action advantage by every token score makes longer actions contribute more gradient terms. Token averaging, action averaging, and trajectory averaging define different objectives. An implementation must state its normalization.

Tool observations are environment outputs and normally receive no policy-gradient loss. Tool-call tokens, including arguments, do receive it because they are sampled actions. If the harness deterministically repairs a malformed call before execution, the logged trajectory must distinguish the sampled call from the executed call.

A batch estimate needs an uncertainty analysis

Let per-trajectory gradient contribution be $g_i=R_i\nabla_\theta\log p_\theta(\tau_i)$. The Monte Carlo estimate is $\widehat g=N^{-1}\sum_i g_i$. If trajectories are independent and have finite covariance matrix $\Sigma$, then

$$\operatorname{Cov}(\widehat g)=\frac{\Sigma}{N}.$$

This $1/N$ scaling is the ideal independent case. Several rollouts for the same task share difficulty, environment state, and sometimes generated prefixes. Their contributions are correlated, so the effective sample size is smaller than the trajectory count. Report the number of task instances as well as the number of rollouts.

Rare high-return or high-ratio trajectories can dominate a batch. Inspect per-sample contribution norms and the fraction of total norm carried by the largest samples. Gradient clipping bounds the applied update, but it does not make the estimator representative. Stratifying by task or normalizing within a task changes the weighting and should be part of the objective specification.

Evaluation rollouts should be separated from training rollouts. Training uses exploratory sampling and repeatedly adapts to observed tasks. Evaluation uses a frozen checkpoint, held-out task instances, a declared decoding policy, and no gradient updates. Reusing evaluation feedback for checkpoint selection turns the set into validation data. A final test requires another untouched sample or an explicit correction for repeated selection.

4. Baselines reduce variance without changing expectation 45:00

A baseline $b(h_t)$ estimates expected return from history $h_t$ without depending on the sampled action $a_t$. Define an advantage estimate $\widehat A_t=G_t-b(h_t)$, where $G_t$ is the return attributed to action $a_t$. The policy-gradient estimator becomes

$$\widehat g_b=\frac1N\sum_{i=1}^N\sum_t \widehat A_{i,t}\nabla_\theta \log\pi_\theta(a_{i,t}\mid h_{i,t}).$$

The baseline term has zero expectation conditional on history:

$$\begin{aligned} \mathbb E_{a\sim\pi_\theta(\cdot\mid h)} [b(h)\nabla_\theta\log\pi_\theta(a\mid h)] &=b(h)\sum_a\pi_\theta(a\mid h) \nabla_\theta\log\pi_\theta(a\mid h)\\ &=b(h)\sum_a\nabla_\theta\pi_\theta(a\mid h)\\ &=b(h)\nabla_\theta\sum_a\pi_\theta(a\mid h)\\ &=b(h)\nabla_\theta 1=0. \end{aligned}$$

Therefore subtracting an action-independent baseline does not bias the expected gradient. It changes sample variance. A baseline that uses the sampled action can introduce bias.

A two-action numerical example

Let actions $A$ and $B$ have probabilities $p$ and $1-p$. Parameterize $p=\sigma(\theta)$ with logistic function $\sigma(\theta)=1/(1+e^{-\theta})$. Suppose rewards are $r(A)=1$ and $r(B)=0$. The score derivatives are

$$\frac{\partial}{\partial\theta}\log p=1-p, \qquad \frac{\partial}{\partial\theta}\log(1-p)=-p.$$

At $p=0.5$, the true gradient of expected reward $J=p$ is $p(1-p)=0.25$. Without a baseline, the one-sample estimator is $0.5$ when $A$ is drawn and $0$ when $B$ is drawn. Its expectation is $0.25$.

With baseline $b=0.5$, the estimator is $(1-0.5)(0.5)=0.25$ for $A$ and $(0-0.5)(-0.5)=0.25$ for $B$. The estimator is exactly $0.25$ in this symmetric example, so its variance is zero. A poor baseline can increase variance; unbiasedness alone does not make it useful.

For a scalar score variable $Z=\nabla_\theta\log\pi_\theta(a\mid h)$ along one parameter direction, the constant baseline minimizing $\operatorname{Var}[(R-b)Z]$ is

$$b^*=\frac{\mathbb E[RZ^2]}{\mathbb E[Z^2]},$$

when the denominator is positive. The mean reward $\mathbb E[R]$ is optimal only when $Z^2$ is uncorrelated with reward. A learned value function approximates a history-dependent baseline because return varies greatly across tasks and states.

Baseline error affects variance, while action dependence can affect bias. A critic that reads history $h_t$ is allowed. A baseline that also reads the sampled action and is inserted without a correction can subtract part of the very signal being estimated. Group-relative methods use other sampled returns from the same task as a baseline; their exact estimator must account for dependence created by group statistics.

Variance should be measured at the level relevant to optimization. Report gradient norms, advantage distributions, per-task reward variance, and effective batch size. A stable average training curve can hide rare batches dominated by a few large ratios or long responses.

The companion function averages vector-valued score gradients:

from agent_lab.training import reinforce_estimate

estimate = reinforce_estimate(
    score_gradients=[(0.5,), (-0.5,)],
    rewards=[1.0, 0.0],
    baselines=[0.5, 0.5],
)
assert estimate == (0.25,)

5. Reward-to-go improves temporal credit assignment 54:00

When rewards occur throughout a trajectory, an early action should not be weighted by rewards that occurred before it. Define reward-to-go

$$G_t=\sum_{k=t}^{T-1}\gamma^{k-t}r_k.$$

Replacing total return $R(\tau)$ with $G_t$ in the coefficient for action $a_t$ removes past reward terms whose expectation does not depend on that action. This can reduce variance while preserving the expected policy gradient under the usual causal assumptions.

Reward-to-go does not solve all credit assignment, the problem of identifying which actions caused an outcome. A final repository test reward gives the same scalar signal to file inspection, correct edits, unnecessary commands, and recovery actions. Process rewards or a learned value function can provide finer signal, but each introduces a new oracle that can be wrong or exploited.

Reward shaping adds intermediate reward $F(s_t,a_t,s_{t+1})$. Arbitrary shaping can change the optimal policy. Potential-based shaping of the form $\gamma\Phi(s_{t+1})-\Phi(s_t)$ preserves optimal policies in standard Markov decision-process settings, where $\Phi$ is a state potential. Agent environments often have partial observations and approximate state, so the theorem's assumptions must be checked before applying it.

The companion implementation computes all reward-to-go values in one reverse pass:

from agent_lab.training import reward_to_go

returns = reward_to_go([0.2, 1.0], gamma=0.9)
assert returns == (1.1, 1.0)

The function assumes a finite trajectory and applies no terminal bootstrap value. Actor-critic code must add that value explicitly when a rollout is truncated rather than terminated.

Eligibility traces interpolate between bias and variance

An actor-critic method can estimate advantage with temporal-difference residual

$$\delta_t=r_t+\gamma V_\phi(h_{t+1})-V_\phi(h_t),$$

where $V_\phi$ is a learned value model with parameters $\phi$. In a partially observed agent problem, history $h_t$ replaces fully observed state. The residual uses one reward and one bootstrap estimate. It has lower variance than a full Monte Carlo return when the critic is accurate, but critic error introduces bias.

Generalized advantage estimation defines

$$\widehat A_t^{\rm GAE}=\sum_{l=0}^{T-t-1}(\gamma\lambda)^l\delta_{t+l},$$

where trace parameter $\lambda\in[0,1]$ controls the horizon. At $\lambda=0$, the estimate uses the one-step residual. As $\lambda$ approaches one, it approaches a Monte Carlo-style return minus the value baseline under standard terminal handling. The choice trades critic bias against sampling variance.

For a coding agent, intermediate test rewards can make $\delta_t$ informative. They can also reward test manipulation or repeated cheap checks. The critic and process rewards should receive adversarial evaluation just like the terminal verifier. A denser signal is useful only when it remains aligned under optimization.

Asynchronous Advantage Actor-Critic uses parallel actor-learners and advantage-based updates in deep reinforcement learning. Its environments and neural policies differ from language agents, but it provides a primary example of coupling a policy actor, value critic, and parallel rollout system. Agent training inherits the same systems question. Correlation and staleness determine whether more workers produce more useful experience.

6. Group-relative advantages compare samples for the same task 63:00

For one task instance $x$, sample a group of $G$ trajectories with rewards $R_1,\ldots,R_G$. Their mean is $\bar R=G^{-1}\sum_jR_j$. A centered group-relative advantage is

$$\widehat A_j=R_j-\bar R.$$

This estimate needs no separate value model. If all group rewards are equal, every advantage is zero and the group contributes no policy-gradient signal. With sparse binary reward, all-failure groups are therefore wasted for the policy update.

Group Relative Policy Optimization (GRPO) adds policy-ratio clipping and other terms beyond simple centering. DrGRPO analyzes biases associated with group standard-deviation and response-length normalization and proposes removing them in its setting. A label such as “GRPO” is insufficient to reproduce training; report the exact loss, normalization, clipping, sampling group, and reference-policy terms.

The companion implementation computes unstandardized centered advantages:

from agent_lab.training import centered_group_advantages

advantages = centered_group_advantages([1.0, 0.0, 0.0, 1.0])
assert advantages == (0.5, -0.5, -0.5, 0.5)

Centering within a task controls for task difficulty. It can also discard signal when all trajectories succeed at different costs unless cost enters the reward. Group composition changes the comparison baseline.

Degenerate groups and normalization choices

For binary reward with per-sample success probability $p$ and group size $G$, the probability that all rewards are equal is

$$P(\text{degenerate})=p^G+(1-p)^G.$$

Both all-success and all-failure groups have zero centered advantages. If $p=0.01$ and $G=8$, the all-failure term is $0.99^8\approx0.923$. Most groups provide no centered policy-gradient signal. Increasing $G$ reduces this probability but consumes more rollouts per task.

Standardizing by the sample standard deviation rescales groups differently. On near-uniform groups, a small denominator can amplify noise. Adding an $\epsilon$ stabilizer prevents division by zero but introduces another scale choice. Length normalization can favor short or long responses depending on whether log probabilities and advantages are summed or averaged.

DeepSeekMath introduced GRPO in its reported mathematical-reasoning system. Understanding R1-Zero-Like Training analyzes normalization and length effects and motivates DrGRPO. Reproduction requires the exact loss rather than the method name because these apparently small choices change the gradient.

The degenerate-group probability is also executable:

from agent_lab.training import degenerate_binary_group_probability

probability = degenerate_binary_group_probability(0.01, group_size=8)
assert probability > 0.92

The calculation assumes independent rollouts with the same success probability. Shared environment failures or deterministic decoding can increase correlation and make mixed-reward groups even rarer.

7. On-policy data, off-policy reuse, and stability 72:30

An update is on-policy when trajectories are sampled from the policy being optimized. It is off-policy when data come from a different behavior policy $\mu$. Reusing old or teacher data improves sample efficiency but changes the gradient estimate.

For trajectory sampled from $\mu$, the importance ratio is

$$w(\tau)=\frac{p_\theta(\tau)}{p_\mu(\tau)}.$$

Under support and integrability conditions, importance weighting can express an expectation under $p_\theta$ using samples from $p_\mu$. For long trajectories, the product of action-probability ratios can have enormous variance. Clipping ratios reduces variance but introduces bias.

Agent RL also depends on environment throughput. Let rollout time per trajectory be $d_r$, update time per batch be $d_u$, and number of parallel environments be $m$. A rough cycle time for batch size $B$ is at least $Bd_r/m+d_u$, before stragglers and coordination. Faster optimization does not help if resettable environments or verifiers dominate the pipeline.

Safety constraints should remain outside compensable reward where possible. A high task reward must not offset an unauthorized purchase or data disclosure. The execution gate from Chapter 1 applies during training rollouts as well as deployment.

Policy drift can be measured directly

Let $\pi_{\rm ref}$ be a fixed reference policy and $\pi_\theta$ the updated policy. Their Kullback-Leibler divergence at history $h$ is

$$D_{\rm KL}(\pi_\theta\|\pi_{\rm ref})= \sum_a\pi_\theta(a\mid h) \log\frac{\pi_\theta(a\mid h)}{\pi_{\rm ref}(a\mid h)}.$$

The divergence is nonnegative and equals zero only when the two distributions agree on the relevant support. A KL penalty discourages drift from a reference but does not identify which behavioral change is harmful. Two policies can have small average KL and differ sharply on a rare consequential action.

Trust Region Policy Optimization derives a constrained policy-update procedure and then introduces practical approximations. PPO's clipping heuristic is related to this stability goal. It is not the same constraint and carries no general monotonic-improvement guarantee for language-agent training.

PPO clipping and KL penalties are optimization controls. They do not correct a misspecified reward, a compromised verifier, or unsafe exploration. Run-level admission constraints should reject forbidden actions before execution. Evaluation should retain their attempted frequency because a blocked attempt still reveals policy behavior.

Off-policy reuse also requires support. If behavior policy $\mu$ assigns zero probability to an action that target policy $\pi_\theta$ may take, the likelihood ratio is undefined and the dataset cannot estimate that part of the target distribution. No clipping rule creates missing evidence. New on-policy rollouts are required.

Value functions turn delayed return into local learning signals

For a Markov decision process, define the state-value function under policy $\pi$ as

$$V^\pi(s)=\mathbb E_\pi[G_t\mid s_t=s],$$

where $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k}$ is the discounted return from step $t$, $r_{t+k}$ is the later reward, and discount factor $\gamma\in[0,1)$ reduces the weight of distant rewards. Define the action-value function

$$Q^\pi(s,a)=\mathbb E_\pi[G_t\mid s_t=s,a_t=a].$$

The advantage function is $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$. It measures whether action $a$ is better or worse than the policy's average action at state $s$. Using an estimated advantage in the policy-gradient estimator assigns credit locally while preserving the interpretation of total return.

The value function satisfies the Bellman expectation equation

$$V^\pi(s)=\sum_a\pi(a\mid s) \sum_{s',r}p(s',r\mid s,a) \left[r+\gamma V^\pi(s')\right],$$

where $p(s',r\mid s,a)$ is the environment probability of next state $s'$ and reward $r$. This equation decomposes long-term value into immediate reward plus discounted continuation value. An actor–critic method uses an actor, the policy $\pi_\theta$, and a critic, an approximation $V_\phi$ with parameters $\phi$. The temporal-difference residual

$$\delta_t=r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t)$$

is a one-step estimate of advantage. It is biased when the critic is inaccurate, but it can have much lower variance than a full trajectory return.

Reusing old trajectories requires a likelihood ratio

An on-policy estimator uses trajectories sampled from the policy being updated. Suppose instead that data were sampled from behavior policy $\mu$ while the target policy is $\pi_\theta$. For an action observed in state $s_t$, define the importance ratio

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\mu(a_t\mid s_t)}.$$

Multiplying by this ratio corrects an expectation when the behavior policy assigns positive probability wherever the target policy does. Large ratios create high variance and make a few samples dominate. This is why unlimited reuse of old agent trajectories is unstable even when those trajectories were once on-policy.

Proximal Policy Optimization limits the incentive for a single update to move the policy too far. Its clipped surrogate objective uses

$$L^{\rm clip}(\theta)=\mathbb E_t\left[ \min\left(r_t(\theta)\hat A_t, \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t\right) \right],$$

where $\hat A_t$ is an estimated advantage, $\epsilon>0$ is a clipping width, and $\operatorname{clip}(x,l,u)$ restricts $x$ to the interval $[l,u]$. Clipping does not enforce an exact trust region and does not make arbitrary offline data safe to reuse. It reduces the reward for large probability-ratio changes in the sampled batch.

Reward design is an identification problem

A reward function is a measured proxy for desired behavior. Let $Y(\tau)$ be the true, possibly unobservable utility of trajectory $\tau$, and let $R(\tau)$ be the computable reward. Optimizing $R$ improves $Y$ only to the extent that their relationship remains valid under the distribution shift caused by optimization. A policy can exploit degrees of freedom that were rare in the data used to validate the reward.

For an agent, a useful outcome vector may include task success $S$, preserved invariants $I$, forbidden effects $F$, resource cost $C$, and latency $L$. A scalar reward such as

$$R=w_SS+w_II-w_FF-w_CC-w_LL$$

defines a trade-off through weights $w_S,w_I,w_F,w_C,w_L$. It should not be mistaken for the measurements themselves. In particular, setting a large finite penalty $w_F$ still permits a forbidden effect when other rewards are large enough. A hard execution constraint enforces $F=0$ before the action occurs.

This separation also improves diagnosis. If task success rises while cost and forbidden effects rise, a scalar leaderboard position can hide the regression. Report the vector first. Apply an explicitly justified decision rule afterward.

Reward validation should include held-out adversarial trajectories. Construct pairs with identical nominal task success but different forbidden effects, costs, or evidence quality. The reward and acceptance policy should rank them according to the intended specification. If a language-model judge supplies reward, test paraphrases, irrelevant verbosity, fabricated evidence, and prompt injection in the judged artifact.

InstructGPT is a primary example of learning a reward model from human comparisons and then optimizing a language model against it. An agent with executable tasks can often use programmatic outcome checks instead. Learned preference rewards remain useful for qualities such as clarity or style, but they inherit annotator disagreement and reward-model generalization error. REINFORCE Leave-One-Out revisits score-function optimization for language-model feedback and uses other samples as baselines. It shows that comparatively simple estimators remain competitive when implementation and normalization are controlled.

The true utility $Y$ is often not observed even after deployment. This makes reward design an identification problem: which measured outcomes justify a claim about the desired behavior? Randomized audits, human comparison, independent state checks, and post-deployment monitoring supply additional measurements. Optimization increases pressure on every gap between the proxy and the goal.

A constrained formulation keeps noncompensable requirements explicit. Let expected task return be $J(\theta)$ and violation probabilities be $C_k(\theta)$. One objective is

$$\max_\theta J(\theta)\quad\text{subject to}\quad C_k(\theta)\le\epsilon_k\text{ for every }k,$$

where $\epsilon_k$ is the permitted rate for constraint $k$. For actions such as secret disclosure, the deployment gate may require zero realized violations rather than relying on an estimated average constraint.

Constrained Policy Optimization develops policy updates under expected cost constraints. Its theory assumes a Markov decision process and modeled constraint costs. A production agent still needs pre-execution gates for individual irreversible actions because an expected violation bound permits some realized violations.

Scope and limits of the policy-gradient derivation

The score-function derivation proves an identity under a fixed environment and differentiable policy probabilities. A Monte Carlo estimator is unbiased under on-policy sampling. These facts do not guarantee useful learning at finite sample size, correct rewards, stable optimization, or safe exploration.

For agent training, report the complete learning system: task distribution, environment versions, policy initialization, sampling temperature, rollout budget, reward and verifier, masking, advantage estimator, normalization, clipping, optimizer, reference policy, evaluation harness, cost, and forbidden effects. Curves should include variation across seeds or batches when stochastic variation affects the conclusion.

Exercises

In number search, a policy gives four trajectories probabilities $(0.3,0.2,0.25,0.25)$ and rewards $(1,0,1,0)$. Compute expected reward.
Prove that a constant baseline does not change the expected score-function gradient for a finite action set.
Compute centered group advantages for rewards $(1,1,0,0,0)$.
Explain why an all-failure group gives no update under centered group advantages. Give two remedies and one risk of each.
A two-step trajectory has rewards $(0.2,1.0)$ and $\gamma=0.9$. Compute $G_0$ and $G_1$.

Solutions and discussion

The expectation is $0.3(1)+0.2(0)+0.25(1)+0.25(0)=0.55$.
For constant $b$, the expected baseline term is $b\sum_a\pi_\theta(a)\nabla\log\pi_\theta(a)=b\nabla\sum_a\pi_\theta(a)=b\nabla1=0$.
The mean is $0.4$, so advantages are $(0.6,0.6,-0.4,-0.4,-0.4)$.
Every reward equals the mean zero, so every coefficient is zero. Sampling more trajectories may find a success but increases rollout cost. Adding shaped rewards can create signal but risks reward hacking or changing the desired policy. SFT initialization can raise the success rate but may narrow exploration.
$G_0=0.2+0.9(1)=1.1$ and $G_1=1.0$.

The first answer is an expectation under the stated four-outcome distribution. It does not estimate uncertainty from a finite rollout sample. If those probabilities were empirical frequencies, a confidence interval would require the sample size.

The baseline proof in the second answer depends on the baseline being independent of the sampled action after conditioning on history. A constant satisfies that condition. The same proof applies to a history-dependent value baseline because it is fixed while summing over actions at that history.

The centered advantages in the third answer sum to zero. This is a defining property of mean centering. Positive samples receive total coefficient $1.2$ and negative samples receive total coefficient $-1.2$, so the group update compares outcomes within that task.

For the fourth answer, increasing group size raises the chance of seeing mixed rewards but consumes more environment interactions. Reward shaping creates denser signal but can change the optimal policy or invite exploitation. Better SFT initialization raises the positive rate but can reduce behavioral diversity. Curriculum selection can choose tasks near the current frontier but may overfit the curriculum distribution.

The fifth answer uses reward-to-go measured from each time step. At time zero, both rewards contribute and the later reward is discounted once. At time one, the immediate reward is undiscounted. Past rewards do not enter $G_1$.