Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

show me the PPO algorithm formula

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-pro’s answer.
  1. User

    show me the PPO algorithm formula

  2. Perplexity Assistant

    The Proximal Policy Optimization (PPO) algorithm is a popular reinforcement learning method that addresses stability issues in policy gradient methods. The core of PPO is its objective function, which comes in two main variants: PPO-Clip and PPO-Penalty. The most widely used version is PPO-Clip.

    PPO-Clip Objective Function

    The main formula for PPO-Clip is:

    LCLIP(θ)=Et[min⁡(rt(θ)At,clip(rt(θ),1−ϵ,1+ϵ)At)]L^{CLIP}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta)A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t\right)\right]

    Where:

    • rt(θ)=πθ(at∣st)πθold(at∣st)r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)} is the probability ratio between the current policy and old policy[1][3][5]
    • AtA_t is the advantage function (how much better an action is compared to the average action in that state)[1][4]
    • ϵ\epsilon is a small hyperparameter (typically 0.1 to 0.3, often 0.2) that limits how far the new policy can go from the old policy[1][5]
    • clip(rt(θ),1−ϵ,1+ϵ)\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) constrains the ratio to stay within [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon][3]

    Complete PPO Algorithm Pseudocode

    The full PPO algorithm can be expressed as:

    1. Input: initial policy parameters θ0\theta_0, initial value function parameters ϕ0\phi_0
    2. For k=0,1,2,...k = 0, 1, 2, ...:
      • Collect set of trajectories Dk={τi}\mathcal{D}_k = \{\tau_i\} by running policy πk=π(θk)\pi_k = \pi(\theta_k) in the environment
      • Compute rewards-to-go R^t\hat{R}_t
      • Compute advantage estimates A^t\hat{A}_t using the current value function VϕkV_{\phi_k}
      • Update policy by maximizing the PPO-Clip objective: θk+1=arg⁡max⁡θ1∣Dk∣T∑τ∈Dk∑t=0Tmin⁡(πθ(at∣st)πθk(at∣st)Aπθk(st,at),g(ϵ,Aπθk(st,at)))\theta_{k+1} = \arg\max_\theta \frac{1}{|\mathcal{D}_k|T}\sum_{\tau \in \mathcal{D}_k}\sum_{t=0}^{T} \min\left(\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_k}(a_t|s_t)}A^{\pi_{\theta_k}}(s_t,a_t), g(\epsilon, A^{\pi_{\theta_k}}(s_t,a_t))\right) (typically using stochastic gradient ascent with Adam)
      • Fit value function by regression on mean-squared error: ϕk+1=arg⁡min⁡ϕ1∣Dk∣T∑τ∈Dk∑t=0T(Vϕ(st)−R^t)2\phi_{k+1} = \arg\min_\phi \frac{1}{|\mathcal{D}_k|T}\sum_{\tau \in \mathcal{D}_k}\sum_{t=0}^{T}(V_\phi(s_t) - \hat{R}_t)^2 (typically using gradient descent)[1]

    How the Clipping Mechanism Works

    The clipping mechanism works differently depending on whether the advantage is positive or negative:

    When Advantage is Positive (A_t > 0):

    The objective reduces to: min⁡(rt(θ)At,(1+ϵ)At)\min(r_t(\theta)A_t, (1+\epsilon)A_t)

    This means that increasing the probability of the action (making r_t(\theta) > 1) is beneficial but only up to 1+ϵ1+\epsilon. After that, there's no further incentive to increase the probability.[2][3][4]

    When Advantage is Negative (A_t < 0):

    The objective reduces to: min⁡(rt(θ)At,(1−ϵ)At)\min(r_t(\theta)A_t, (1-\epsilon)A_t)

    This means that decreasing the probability of the action (making r_t(\theta) < 1) is beneficial but only down to 1−ϵ1-\epsilon.[2][3][4]

    PPO-Penalty Variant

    The alternative PPO variant (PPO-Penalty) uses an adaptive KL penalty:

    LKLPEN(θ)=Et[rt(θ)At−βKL[πθold(⋅∣st),πθ(⋅∣st)]]L^{KLPEN}(\theta) = \mathbb{E}_t[r_t(\theta)A_t - \beta KL[\pi_{\theta_{old}}(\cdot|s_t), \pi_\theta(\cdot|s_t)]]

    Where β\beta is dynamically adjusted during training:

    • If KL divergence is too high, increase β\beta
    • If KL divergence is too low, decrease β\beta[4][5]

    PPO has become a default algorithm in reinforcement learning due to its simplicity of implementation, stability, and sample efficiency compared to other policy gradient methods like TRPO (Trust Region Policy Optimization)[1][3].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.