display the formula for GRPO algorithm
- User
display the formula for GRPO algorithm
Assistant
GRPO Algorithm Formula
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to optimize language models by comparing groups of generated outputs, assigning rewards based on their relative performance, and regularizing updates to stay close to a reference policy. Below is the core mathematical formulation of the GRPO algorithm, as described in the literature and technical documentation.
1. Advantage Calculation
For each generated output ( y_i ) in a group of ( n ) outputs for a given prompt, the advantage is computed as:
[ A(y_i) = \frac{r(y_i) - \mu_r}{\sigma_r} ]
where:
- ( r(y_i) ) is the reward assigned to output ( y_i )
- ( \mu_r ) is the mean reward across the group: ( \mu_r = \frac{1}{n} \sum_{j=1}^n r(y_j) )
- ( \sigma_r ) is the standard deviation of rewards in the group: ( \sigma_r = \sqrt{\frac{1}{n} \sum_{j=1}^n (r(y_j) - \mu_r)^2} )[6][2][3]
2. Objective Function
The GRPO objective for updating the policy ( \pi_\theta ) is:
[ \mathcal{L}{\text{GRPO}} = \mathbb{E}{y \sim \pi_\theta} \left[ w(y) \cdot A(y) - \beta \cdot D_{\text{KL}}(\pi_\theta(y) \parallel \pi_{\text{ref}}(y)) \right] ]
where:
- ( w(y) ) is a weight (often 1, or clipped as in PPO)
- ( \beta ) is a regularization coefficient
- ( D_{\text{KL}} ) is the Kullback-Leibler divergence between the new policy ( \pi_\theta ) and the reference policy ( \pi_{\text{ref}} )[3][2][7]
3. Clipped Surrogate Objective (Optional, as in PPO)
To ensure stable updates, GRPO can use a clipped surrogate objective:
[ \mathcal{L}{\text{clip}} = \mathbb{E}{y \sim \pi_\theta} \left[ \min \left( r(y) \cdot A(y), \text{clip}(r(y), 1-\epsilon, 1+\epsilon) \cdot A(y) \right) - \beta \cdot D_{\text{KL}}(\pi_\theta(y) \parallel \pi_{\text{ref}}(y)) \right] ]
where:
- ( r(y) = \frac{\pi_\theta(y)}{\pi_{\text{old}}(y)} ) is the probability ratio
- ( \epsilon ) is a small constant for clipping[2][3]
4. Summary Table
Component Formula Description Advantage ( A(y_i) = \frac{r(y_i) - \mu_r}{\sigma_r} ) Normalized relative score within group Objective ( \mathcal{L}{\text{GRPO}} = \mathbb{E}[w(y) \cdot A(y) - \beta D{\text{KL}}] ) Maximize advantage, penalize KL divergence Clipped Objective ( \mathcal{L}{\text{clip}} = \mathbb{E}[\min(r(y)A(y), \text{clip}(r(y))A(y)) - \beta D{\text{KL}}] ) Stable updates with clipping
Key Points
- Group-based: Rewards are normalized within each group, making the algorithm robust to question difficulty and reward scale[6][3].
- KL Regularization: Penalizes divergence from a reference policy, ensuring stable and controlled learning[3][2].
- No Value Function: Unlike PPO, GRPO does not require a value function, reducing memory and computational requirements[5][7].
- Variants: The loss can be further normalized or adjusted to reduce length bias or improve stability (see "dr_grpo" and "bnpo" variants in Hugging Face TRL)[2][3].
References to Core Formulas
- Advantage normalization: [6][2][3]
- Objective and KL penalty: [3][2][7]
- Clipped surrogate objective: [2][3]
- Implementation details and variants: [2][3][5]
In summary:
The GRPO algorithm optimizes the policy by maximizing the normalized advantage of each output within a group, while penalizing divergence from a reference policy using KL divergence. This group-relative approach enables efficient and stable reinforcement learning for language models, especially in reasoning and mathematical tasks[6][3][2].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.