show me the formula for GRPO algorithm
- User
show me the formula for GRPO algorithm
Assistant
The GRPO Formula
Group Relative Policy Optimization (GRPO) was introduced in the DeepSeekMath paper as a more efficient alternative to Proximal Policy Optimization (PPO) for training language models in mathematical reasoning tasks and other complex reasoning scenarios. The core GRPO formula is:
\mathcal{J}_{GRPO}(\theta) &= \mathbb{E}\left[q \sim P(Q),\left\{o_i\right\}_{i=1}^G \sim \pi_{\theta_{old}}(O \mid q)\right] \\ &\frac{1}{G} \sum_{i=1}^G \frac{1}{\left|o_i\right|} \sum_{t=1}^{\left|o_i\right|}\left\{\min \left[\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{old}}\left(o_{i, ##="" ###="" $$a_i="\frac{R_\phi(r_i)" $\text{mean}(\mathcal{g})$="" $\text{std}(\mathcal{g})$="" $i$="" $r_\phi(r_i)$="" (as="" -="" 1+\varepsilon\right)="" 1-\varepsilon,="" \end{aligned}$$="" \hat{a}_{i,="" \mathbb{d}_{kl}\left[\pi_\theta||="" \mid="" \operatorname{clip}\left(\frac{\pi_\theta\left(o_{i,="" \pi_{ref}\right]\right\}="" \text{mean}(\mathcal{g})}{\text{std}(\mathcal{g})}$$="" a="" across="" advantage="" advantages="" are="" by="" calculates="" class="inline-ref" comparing="" completions="" components="" computes="" creating="" critic="" deviation="" ensures="" estimate="" for="" formula="" function.="" fundamental="" generated="" group="" group-normalized="" grpo="" how="" href="https://yugeten.github.io/posts/2025/01/ppogrpo/" in="" instead="" is="" it="" key="" learning="" mean="" model="" more="" multiple="" normalization="" o_{i,<t}\right)}="" o_{i,<t}\right)},="" o_{i,<t}\right)}{\pi_{\theta_{old}}\left(o_{i,="" of="" other="" over="" possible="" ppo="" ppo),="" prompt,="" prompt:="" q,="" relative="" response="" responses="" reward="" rewards="" same="" signal<a="" stable="" standard="" that="" the="" this="" to="" token="" t}="" t},="" t}\right]-\beta="" using="" where:="" within="">2. ### Clipped Surrogate Objective The clipped surrogate objective in GRPO (similar to PPO) limits policy updates to prevent large, destabilizing changes: $$\mathcal{L}_{clip}(\theta) = \frac{1}{N} \sum_{i=1}^N \left( \min\left( \frac{\pi_\theta(r_i|p)}{\pi_{\theta_{old}}(r_i|p)} A_i, \ \text{clip}\left( \frac{\pi_\theta(r_i|p)}{\pi_{\theta_{old}}(r_i|p)}, 1-\epsilon, 1+\epsilon \right) A_i \right) \right)$$ Where: - $\pi_\theta(r_i|p)$ is the probability of generating response $r_i$ given prompt $p$ under the current policy - $\pi_{\theta_{old}}(r_i|p)$ is the probability under the old policy - $\epsilon$ is the clipping parameter (typically set to 0.2)[1][2] ### KL Divergence Penalty To prevent the policy from deviating too far from the reference model, GRPO includes a KL divergence term: $$\mathbb{D}_{KL}(\pi_\theta || \pi_{ref})$$ This term ensures the updated model stays reasonably close to the original capabilities[2]. ### Final GRPO Objective The complete GRPO objective combines these components: $$\mathcal{L}_{GRPO}(\theta) = \mathcal{L}_{clip}(\theta) - w_1\mathbb{D}_{KL}(\pi_\theta || \pi_{ref})$$ Where $w_1$ is a hyperparameter that controls the strength of the KL penalty (typically set to 0.001 in practice)[1][2]. ## GRPO Variants Several GRPO variants exist with different loss formulations: 1. **Original GRPO**: Normalizes by sequence length, which can introduce response length bias 2. **DAPO (Decomposed Advantage Policy Optimization)**: Uses token-level normalization to better handle longer sequences 3. **DR-GRPO (Distributional Robust GRPO)**: Divides by a constant (usually max completion length) instead of sequence length to completely remove length bias[3] ## Example of GRPO in Action The GRPO algorithm follows this process: 1. For each prompt, generate multiple responses ($N$ generations per prompt, typically 8) 2. Compute rewards for each response using reward functions 3. Calculate group-normalized advantages by comparing each response's reward to the group average 4. Update the model weights to maximize the GRPO objective above This approach allows the model to learn effective reasoning strategies through direct optimization, without requiring a separate critic model as in PPO[4][5]. The major advantage of GRPO over PPO is its memory efficiency. Traditional PPO implementations require storing large amounts of state information and a separate critic model, while GRPO only needs to track group statistics for advantage calculation, reducing memory usage by up to 90% in some implementations[6].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.