Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-pro-high· PerplexityHugging Face LMArena Search Arena 24K

display the formula for GRPO algorithm

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-pro’s answer.
  1. User

    display the formula for GRPO algorithm

  2. Perplexity Assistant

    GRPO Algorithm Formula

    Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to optimize language models by comparing groups of generated outputs, assigning rewards based on their relative performance, and regularizing updates to stay close to a reference policy. Below is the core mathematical formulation of the GRPO algorithm, as described in the literature and technical documentation.


    1. Advantage Calculation

    For each generated output ( y_i ) in a group of ( n ) outputs for a given prompt, the advantage is computed as:

    [ A(y_i) = \frac{r(y_i) - \mu_r}{\sigma_r} ]

    where:

    • ( r(y_i) ) is the reward assigned to output ( y_i )
    • ( \mu_r ) is the mean reward across the group: ( \mu_r = \frac{1}{n} \sum_{j=1}^n r(y_j) )
    • ( \sigma_r ) is the standard deviation of rewards in the group: ( \sigma_r = \sqrt{\frac{1}{n} \sum_{j=1}^n (r(y_j) - \mu_r)^2} )[6][2][3]

    2. Objective Function

    The GRPO objective for updating the policy ( \pi_\theta ) is:

    [ \mathcal{L}{\text{GRPO}} = \mathbb{E}{y \sim \pi_\theta} \left[ w(y) \cdot A(y) - \beta \cdot D_{\text{KL}}(\pi_\theta(y) \parallel \pi_{\text{ref}}(y)) \right] ]

    where:

    • ( w(y) ) is a weight (often 1, or clipped as in PPO)
    • ( \beta ) is a regularization coefficient
    • ( D_{\text{KL}} ) is the Kullback-Leibler divergence between the new policy ( \pi_\theta ) and the reference policy ( \pi_{\text{ref}} )[3][2][7]

    3. Clipped Surrogate Objective (Optional, as in PPO)

    To ensure stable updates, GRPO can use a clipped surrogate objective:

    [ \mathcal{L}{\text{clip}} = \mathbb{E}{y \sim \pi_\theta} \left[ \min \left( r(y) \cdot A(y), \text{clip}(r(y), 1-\epsilon, 1+\epsilon) \cdot A(y) \right) - \beta \cdot D_{\text{KL}}(\pi_\theta(y) \parallel \pi_{\text{ref}}(y)) \right] ]

    where:

    • ( r(y) = \frac{\pi_\theta(y)}{\pi_{\text{old}}(y)} ) is the probability ratio
    • ( \epsilon ) is a small constant for clipping[2][3]

    4. Summary Table

    ComponentFormulaDescription
    Advantage( A(y_i) = \frac{r(y_i) - \mu_r}{\sigma_r} )Normalized relative score within group
    Objective( \mathcal{L}{\text{GRPO}} = \mathbb{E}[w(y) \cdot A(y) - \beta D{\text{KL}}] )Maximize advantage, penalize KL divergence
    Clipped Objective( \mathcal{L}{\text{clip}} = \mathbb{E}[\min(r(y)A(y), \text{clip}(r(y))A(y)) - \beta D{\text{KL}}] )Stable updates with clipping

    Key Points

    • Group-based: Rewards are normalized within each group, making the algorithm robust to question difficulty and reward scale[6][3].
    • KL Regularization: Penalizes divergence from a reference policy, ensuring stable and controlled learning[3][2].
    • No Value Function: Unlike PPO, GRPO does not require a value function, reducing memory and computational requirements[5][7].
    • Variants: The loss can be further normalized or adjusted to reduce length bias or improve stability (see "dr_grpo" and "bnpo" variants in Hugging Face TRL)[2][3].

    References to Core Formulas

    • Advantage normalization: [6][2][3]
    • Objective and KL penalty: [3][2][7]
    • Clipped surrogate objective: [2][3]
    • Implementation details and variants: [2][3][5]

    In summary:
    The GRPO algorithm optimizes the policy by maximizing the normalized advantage of each output within a group, while penalizing divergence from a reference policy using KL divergence. This group-relative approach enables efficient and stable reinforcement learning for language models, especially in reasoning and mathematical tasks[6][3][2].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt