Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-pro· PerplexityHugging Face LMArena Search Arena 24K

show me the PPO algorithm formula

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning-pro-high’s answer.
  1. User

    show me the PPO algorithm formula

  2. Perplexity Assistant

    Proximal Policy Optimization (PPO) Algorithm Formula

    PPO is a reinforcement learning algorithm designed to optimize policies while preventing overly large updates that could destabilize learning. The most widely used variant is PPO-Clip, which uses a clipped surrogate objective function.

    PPO-Clip Objective Function

    The core formula for the PPO-Clip objective is:

    LCLIP(θ)=Et[min⁡(rt(θ)A^t, clip(rt(θ),1−ϵ,1+ϵ)A^t)]L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t, \ \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t \right) \right]

    where:

    • rt(θ)=πθ(at∣st)πθold(at∣st)r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} is the probability ratio between the new policy and the old policy for action ata_t in state sts_t.
    • A^t\hat{A}_t is the estimated advantage at time tt.
    • ϵ\epsilon is a small hyperparameter (commonly 0.1–0.3, often 0.2) that determines the clipping range.
    • The expectation Et\mathbb{E}_t is taken over timesteps in the collected trajectories[1][3][7].

    Intuition

    • If the advantage A^t\hat{A}_t is positive, the objective encourages increasing the probability of the action, but only up to a limit set by 1+ϵ1 + \epsilon.
    • If the advantage is negative, it discourages the action, but only down to 1−ϵ1 - \epsilon.
    • The min⁡\min operator ensures that the policy update does not move too far from the old policy, even if the advantage is large, thus improving stability[1][3][7].

    Value Function Loss

    PPO also updates a value function (critic) using the mean squared error between predicted values and empirical returns:

    LVF(ϕ)=Et[(Vϕ(st)−R^t)2]L^{\text{VF}}(\phi) = \mathbb{E}_t \left[ \left( V_\phi(s_t) - \hat{R}_t \right)^2 \right]

    where Vϕ(st)V_\phi(s_t) is the value function estimate and R^t\hat{R}_t is the empirical return[1][3].

    Complete PPO Update

    The total loss function often combines the clipped surrogate objective, the value function loss, and an entropy bonus (to encourage exploration):

    LPPO=LCLIP(θ)−c1LVF(ϕ)+c2S[πθ](st)L^{\text{PPO}} = L^{\text{CLIP}}(\theta) - c_1 L^{\text{VF}}(\phi) + c_2 S[\pi_\theta](s_t)

    where c1c_1 and c2c_2 are coefficients, and S[πθ](st)S[\pi_\theta](s_t) is the entropy of the policy at state sts_t[7].


    Summary Table

    ComponentFormulaDescription
    Probability Ratio$r_t(\theta) = \frac{\pi_\theta(a_ts_t)}{\pi_{\theta_{\text{old}}}(a_t
    Clipped ObjectiveLCLIP(θ)=Et[min⁡(rt(θ)A^t,clip(...))]L^{\text{CLIP}}(\theta) = \mathbb{E}_t [ \min ( r_t(\theta) \hat{A}_t, \text{clip}(...) ) ]Main PPO-Clip loss for policy update
    Value LossLVF(ϕ)=Et[(Vϕ(st)−R^t)2]L^{\text{VF}}(\phi) = \mathbb{E}_t [ ( V_\phi(s_t) - \hat{R}_t )^2 ]Value function regression loss
    Total LossLPPO=LCLIP−c1LVF+c2S[πθ](st)L^{\text{PPO}} = L^{\text{CLIP}} - c_1 L^{\text{VF}} + c_2 S[\pi_\theta](s_t)Combined loss for optimization

    References

    • [Wikipedia: Proximal Policy Optimization][1]
    • [OpenAI Spinning Up: PPO][7]
    • [Dilith Jayakody: PPO Explained][3] [1][3][7]

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.