show me the PPO algorithm formula
- User
show me the PPO algorithm formula
Assistant
The Proximal Policy Optimization (PPO) algorithm is a popular reinforcement learning method that addresses stability issues in policy gradient methods. The core of PPO is its objective function, which comes in two main variants: PPO-Clip and PPO-Penalty. The most widely used version is PPO-Clip.
PPO-Clip Objective Function
The main formula for PPO-Clip is:
Where:
- is the probability ratio between the current policy and old policy[1][3][5]
- is the advantage function (how much better an action is compared to the average action in that state)[1][4]
- is a small hyperparameter (typically 0.1 to 0.3, often 0.2) that limits how far the new policy can go from the old policy[1][5]
- constrains the ratio to stay within [3]
Complete PPO Algorithm Pseudocode
The full PPO algorithm can be expressed as:
- Input: initial policy parameters , initial value function parameters
- For :
- Collect set of trajectories by running policy in the environment
- Compute rewards-to-go
- Compute advantage estimates using the current value function
- Update policy by maximizing the PPO-Clip objective: (typically using stochastic gradient ascent with Adam)
- Fit value function by regression on mean-squared error: (typically using gradient descent)[1]
How the Clipping Mechanism Works
The clipping mechanism works differently depending on whether the advantage is positive or negative:
When Advantage is Positive (A_t > 0):
The objective reduces to:
This means that increasing the probability of the action (making r_t(\theta) > 1) is beneficial but only up to . After that, there's no further incentive to increase the probability.[2][3][4]
When Advantage is Negative (A_t < 0):
The objective reduces to:
This means that decreasing the probability of the action (making r_t(\theta) < 1) is beneficial but only down to .[2][3][4]
PPO-Penalty Variant
The alternative PPO variant (PPO-Penalty) uses an adaptive KL penalty:
Where is dynamically adjusted during training:
- If KL divergence is too high, increase
- If KL divergence is too low, decrease [4][5]
PPO has become a default algorithm in reinforcement learning due to its simplicity of implementation, stability, and sample efficiency compared to other policy gradient methods like TRPO (Trust Region Policy Optimization)[1][3].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.