what's grpo objective?
- User
what's grpo objective?
- Assistant
Group Relative Policy Optimization (GRPO) Objective
GRPO is a reinforcement learning algorithm introduced by DeepSeek, initially in their DeepSeekMath model and later employed in DeepSeek-R1. It represents a significant innovation in how policy optimization is implemented for large language models (LLMs).
Core Objective Formula
The GRPO objective can be expressed as:
(\mathcal{L}{\text{GRPO}}(\theta) = \underbrace{\mathcal{L}{\text{clip}}(\theta)}{\text{Maximize reward}} - \underbrace{w_1\mathbb{D}{\text{KL}}(\pi_\theta || \pi_{\text{orig}})}_{\text{Penalize KL divergence}})[1]
This formula combines two key components:
- A clipped surrogate loss (similar to PPO) that helps maximize rewards
- A KL divergence penalty that prevents the policy from deviating too far from the reference policy
The complete GRPO objective in its expanded form is:
(J_{GRPO}(\theta) = \frac{1}{G}\sum_{i=1}^G \frac{1}{\mid a_i \mid}\sum_{t=1}^{\mid a_i \mid} \left{\min\left[\frac{\pi_\theta(a_{i,t} \mid s,a_{i, < t})}{\pi_{\theta_{old}}(a_{i, t}\mid s, a_{i, < t})}\hat{A}{i,t}, \text{clip}\left(\frac{\pi\theta(a_{i,t}\mid s, a_{ i, < t })}{\pi_{\theta_{old}}(a_{i, t} \mid s, a_{i, < t})}, 1-\epsilon, 1+\epsilon\right)\hat{A}{i,t}\right] - \beta D{KL}[\pi_\theta \mid \mid \pi_{ref}]\right})[2]
Key Innovation: Group-Based Advantage Estimation
The most significant innovation in GRPO is how it calculates advantages:
-
Group Sampling: For each prompt, GRPO samples multiple responses (forming a "group") from the policy model[1][2]
-
Relative Advantage: It calculates the advantage of each response as: (A_i = \frac{R_\phi(r_i) - \text{mean}(\mathcal{G})}{\text{std}(\mathcal{G})})[1]
Where:
- (R_\phi(r_i)) is the reward for a specific response
- (\text{mean}(\mathcal{G})) is the average reward across all responses in the group
- (\text{std}(\mathcal{G})) is the standard deviation of rewards in the group
-
This normalizes advantages within each group, providing a relative measure of how good each response is compared to others generated for the same prompt[2]
Benefits Over PPO
GRPO offers several advantages over traditional PPO:
-
Eliminates Value Network: The most significant benefit is removing the need for a separate value (critic) model that predicts rewards[1][4]
-
Computational Efficiency: Reduces memory and compute overhead by approximately 50% by eliminating the critic model[1]
-
Simplification: Provides a more straightforward approach to RL that is mathematically simpler and more elegant[1]
-
Stability: The group normalization approach offers better stability during training[2]
Theoretical Implications
Recent research examines the preference aggregation achieved by GRPO:
-
The preference aggregation differs fundamentally from standard logarithmic pooling implemented by other approaches like RLHF[3]
-
For groups of size two, the reward preference model corresponds to pairwise comparison preferences, similar to other alignment methods[3]
-
The specific form of preference aggregation arises from how the reward preference model is defined and from the KL penalty function[3]
Implementation Details
When implementing GRPO:
-
The language model serves as the policy network, taking a question as input and producing token sequences as actions[2]
-
For each question, multiple outputs are sampled from the policy model[1][2]
-
Rewards are calculated for each complete response using a reward model[1]
-
Group normalization is applied within each batch of responses to the same prompt[2]
-
The policy is updated using the clipped surrogate objective with KL regularization[1][2]
GRPO represents a significant advancement in applying RL to language models that has proven effective in improving reasoning capabilities in models like DeepSeek-Math and DeepSeek-R1.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.