Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview-high· OpenAIHugging Face LMArena Search Arena 24K

show me the formula for GRPO algorithm

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro-exp-03-25-grounding’s answer.
  1. User

    show me the formula for GRPO algorithm

  2. Assistant

    Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm designed to fine-tune large language models, particularly for tasks requiring complex reasoning. It builds upon Proximal Policy Optimization (PPO) by eliminating the need for a value function estimator and instead utilizes group-based comparisons to compute advantages.

    Key Components of GRPO:

    1. Group Sampling and Reward Calculation:

      • For each state ( s ), sample multiple actions ( a_1, a_2, \ldots, a_G ) from the current policy ( \pi_{\theta_t} ).
      • Evaluate each action using a reward function ( r(s, a_j) ) to obtain rewards ( r_1, r_2, \ldots, r_G ).
    2. Group-Relative Advantage Estimation:

      • Compute the mean ( \mu ) and standard deviation ( \sigma ) of the rewards: [ \mu = \frac{1}{G} \sum_{j=1}^{G} r(s, a_j), \quad \sigma = \sqrt{\frac{1}{G} \sum_{j=1}^{G} (r(s, a_j) - \mu)^2} ]
      • Calculate the group-relative advantage for each action ( a_j ): [ A^{\pi_{\theta_t}}(s, a_j) = \frac{r(s, a_j) - \mu}{\sigma} ] This standardizes the rewards, indicating how each action's reward deviates from the group's average.
    3. Policy Update Objective:

      • The objective function in GRPO is adapted from PPO and is defined as: [ \max_\theta \frac{1}{G} \sum_{j=1}^{G} \mathbb{E}{(s, a_1, \ldots, a_G) \sim \pi{\theta_t}} \left[ \min \left( \frac{\pi_\theta(a_j|s)}{\pi_{\theta_t}(a_j|s)} A^{\pi_{\theta_t}}(s, a_j), \text{clip} \left( \frac{\pi_\theta(a_j|s)}{\pi_{\theta_t}(a_j|s)}, 1 - \epsilon, 1 + \epsilon \right) A^{\pi_{\theta_t}}(s, a_j) \right) \right] ] Here, ( \epsilon ) is a hyperparameter that controls the clipping range, preventing large updates that could destabilize training.
    4. KL Divergence Penalty:

      • To ensure the updated policy remains close to the reference policy, a Kullback-Leibler (KL) divergence term is incorporated: [ \mathcal{L}{\text{GRPO}} = \mathcal{L}{\text{clip}} - \beta \mathbb{D}{\text{KL}}(\pi\theta || \pi_{\text{orig}}) ] Where ( \beta ) is a weighting factor balancing the influence of the KL penalty, and ( \pi_{\text{orig}} ) is the reference policy.

    By leveraging group-based reward comparisons and eliminating the value function, GRPO offers a more memory-efficient and stable approach to policy optimization, making it particularly suitable for training large language models on complex tasks.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.