show me the formula for GRPO algorithm
- User
show me the formula for GRPO algorithm
- Assistant
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to fine-tune large language models, particularly for tasks requiring complex reasoning. It builds upon Proximal Policy Optimization (PPO) by eliminating the need for a value function estimator and instead utilizes group-based comparisons to compute advantages.
Key Components of GRPO:
-
Group Sampling and Reward Calculation:
- For each state ( s ), sample multiple actions ( a_1, a_2, \ldots, a_G ) from the current policy ( \pi_{\theta_t} ).
- Evaluate each action using a reward function ( r(s, a_j) ) to obtain rewards ( r_1, r_2, \ldots, r_G ).
-
Group-Relative Advantage Estimation:
- Compute the mean ( \mu ) and standard deviation ( \sigma ) of the rewards: [ \mu = \frac{1}{G} \sum_{j=1}^{G} r(s, a_j), \quad \sigma = \sqrt{\frac{1}{G} \sum_{j=1}^{G} (r(s, a_j) - \mu)^2} ]
- Calculate the group-relative advantage for each action ( a_j ): [ A^{\pi_{\theta_t}}(s, a_j) = \frac{r(s, a_j) - \mu}{\sigma} ] This standardizes the rewards, indicating how each action's reward deviates from the group's average.
-
Policy Update Objective:
- The objective function in GRPO is adapted from PPO and is defined as: [ \max_\theta \frac{1}{G} \sum_{j=1}^{G} \mathbb{E}{(s, a_1, \ldots, a_G) \sim \pi{\theta_t}} \left[ \min \left( \frac{\pi_\theta(a_j|s)}{\pi_{\theta_t}(a_j|s)} A^{\pi_{\theta_t}}(s, a_j), \text{clip} \left( \frac{\pi_\theta(a_j|s)}{\pi_{\theta_t}(a_j|s)}, 1 - \epsilon, 1 + \epsilon \right) A^{\pi_{\theta_t}}(s, a_j) \right) \right] ] Here, ( \epsilon ) is a hyperparameter that controls the clipping range, preventing large updates that could destabilize training.
-
KL Divergence Penalty:
- To ensure the updated policy remains close to the reference policy, a Kullback-Leibler (KL) divergence term is incorporated: [ \mathcal{L}{\text{GRPO}} = \mathcal{L}{\text{clip}} - \beta \mathbb{D}{\text{KL}}(\pi\theta || \pi_{\text{orig}}) ] Where ( \beta ) is a weighting factor balancing the influence of the KL penalty, and ( \pi_{\text{orig}} ) is the reference policy.
By leveraging group-based reward comparisons and eliminating the value function, GRPO offers a more memory-efficient and stable approach to policy optimization, making it particularly suitable for training large language models on complex tasks.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.