show me the PPO algorithm formula
- User
show me the PPO algorithm formula
Assistant
Proximal Policy Optimization (PPO) Algorithm Formula
PPO is a reinforcement learning algorithm designed to optimize policies while preventing overly large updates that could destabilize learning. The most widely used variant is PPO-Clip, which uses a clipped surrogate objective function.
PPO-Clip Objective Function
The core formula for the PPO-Clip objective is:
where:
- is the probability ratio between the new policy and the old policy for action in state .
- is the estimated advantage at time .
- is a small hyperparameter (commonly 0.1–0.3, often 0.2) that determines the clipping range.
- The expectation is taken over timesteps in the collected trajectories[1][3][7].
Intuition
- If the advantage is positive, the objective encourages increasing the probability of the action, but only up to a limit set by .
- If the advantage is negative, it discourages the action, but only down to .
- The operator ensures that the policy update does not move too far from the old policy, even if the advantage is large, thus improving stability[1][3][7].
Value Function Loss
PPO also updates a value function (critic) using the mean squared error between predicted values and empirical returns:
where is the value function estimate and is the empirical return[1][3].
Complete PPO Update
The total loss function often combines the clipped surrogate objective, the value function loss, and an entropy bonus (to encourage exploration):
where and are coefficients, and is the entropy of the policy at state [7].
Summary Table
Component Formula Description Probability Ratio $r_t(\theta) = \frac{\pi_\theta(a_t s_t)}{\pi_{\theta_{\text{old}}}(a_t Clipped Objective Main PPO-Clip loss for policy update Value Loss Value function regression loss Total Loss Combined loss for optimization
References
- [Wikipedia: Proximal Policy Optimization][1]
- [OpenAI Spinning Up: PPO][7]
- [Dilith Jayakody: PPO Explained][3] [1][3][7]
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.