Skip to content
The Chat Archive
gemini-2.5-pro· Google LMArena Human Preference 140K

we know the formula of GRPO which is like this J GRPO ​ (θ)=[ G 1 ​ i=1 ∑ G ​ min( π θ…

  1. User

    we know the formula of GRPO which is like this

    J GRPO ​ (θ)=[ G 1 ​

    i=1 ∑ G ​ min( π θ old ​

    ​ (o i ​ ∣q) π θ ​ (o i ​ ∣q) ​ A i ​ clip( π θ old ​

    ​ (o i ​ ∣q) π θ ​ (o i ​ ∣q) ​ ,1−ϵ,1+ϵ)A i ​ )]−βD KL ​ (π θ ​ ∥∥π ref ​ )

    now in this the first term is change in model resp from old to new and then we have the clip thing which controls the change from the old to the new to be stable and then we have the kl divergence which also doing something similar?

  2. Assistant

    Excellent question! You have a very good grasp of the first two parts of the formula, and your intuition that the KL divergence term is "doing something similar" is correct, but it's doing it in a crucially different way and for a different reason.

    Let's break down the roles of each component to clarify the distinction.

    The formula you've presented is a common objective in modern Reinforcement Learning, especially for fine-tuning Large Language Models (LLMs) with a technique called RLHF (Reinforcement Learning from Human Feedback). It's essentially a variation of PPO (Proximal Policy Optimization).

    Let's rename the components for clarity:

    J(θ)=E[min(rt(θ)At,clip(rt(θ),1−ϵ,1+ϵ)At)⏟Part 1: The PPO Clipped Objective]−βDKL(πθ∥πref)⏟Part 2: The KL PenaltyJ(\theta) = \mathbb{E} \left[ \underbrace{\text{min} \left( r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t \right)}_{\text{Part 1: The PPO Clipped Objective}} \right] - \underbrace{\beta D_{KL}(\pi_\theta \| \pi_{ref})}_{\text{Part 2: The KL Penalty}}

    Where rt(θ)=πθ(oi∣q)πθold(oi∣q)r_t(\theta) = \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)} is the probability ratio.


    Part 1: The PPO Clipped Objective (Your min and clip term)

    You are exactly right about this part.

    • Target: π_θ_old
    • What is π_θ_old? This is the policy at the start of the current training iteration. It's a snapshot of the model before we compute the gradients for the current batch of data. It gets updated every few hundred or thousand steps.
    • Purpose: To ensure stable learning from one update step to the next.
    • Mechanism: It prevents the policy from changing too drastically within a single gradient update. If an advantage A_i is very high for a particular action, the model might be tempted to make a huge update to its parameters to make that action much more likely. The clip function says: "Whoa, slow down. Even if this action looks amazing, don't change the policy by more than a factor of (1+ε) in this single step."

    Analogy: Think of this as a shock absorber on a car. It's there to smooth out the big, sudden bumps (large gradient updates) on the road so you don't lose control. It's focused on the short-term, step-by-step stability of the training process.


    Part 2: The KL Divergence Penalty

    This is where the key difference lies. Notice the policy it's being compared to: π_ref.

    • Target: π_ref
    • What is π_ref? This is a fixed reference policy, which is typically the original, pre-trained base model before any reinforcement learning was applied. For example, if you are fine-tuning Llama-2-70b to be a chatbot, π_ref is the base Llama-2-70b model, and it does not change throughout the entire RL training process.
    • Purpose: To ensure the fine-tuned model doesn't forget its original capabilities or stray too far from the desired distribution of text.
    • Mechanism: KL Divergence measures how different two probability distributions are. By subtracting β * D_KL, we are adding a penalty to our objective. The objective is maximized, so the model is punished for making π_θ very different from π_ref. The hyperparameter β controls how strong this penalty is.

    Analogy: Think of this as an anchor or a leash. You are training your new, smart chatbot (π_θ). The reward function tells it to be helpful and harmless. Without the KL penalty, the model might discover a "hack"—for example, it might learn that always responding with "You are a valued user and your query is important." gives it a small but consistent reward. It might start doing this for every prompt, becoming useless. The KL penalty acts as a leash, anchoring the model (π_θ) to its original, general-purpose, language-fluent self (π_ref). It allows the model to learn to be more helpful, but it punishes it if it tries to achieve this by generating text that the original model would have considered bizarre or unlikely (i.e., it stops it from forgetting how to speak English coherently).


    Summary: Why Do You Need Both?

    They solve two different problems at two different timescales.

    FeaturePPO Clipping (min/clip)KL Divergence Penalty
    PurposeUpdate StabilityModel Regularization / Grounding
    Compares π_θ to...π_θ_old (the model from the start of the current update iteration)π_ref (the fixed, original pre-trained model)
    TimescaleShort-term: Prevents destructive steps within a single update.Long-term: Prevents the model from deviating too far from its original knowledge base over the entire training run.
    AnalogyShock Absorber: Smooths out the ride during training.Anchor/Leash: Keeps the model from drifting away into nonsensical or "reward-hacked" territory.

    In short:

    • The PPO Clip keeps the training process numerically stable from step to step.
    • The KL Penalty keeps the final model sane, coherent, and grounded in the knowledge it learned during pre-training.
  3. User

    but if we remove the kl divergence then ? I understand it will diverge from the reference but it's a step by step improvement right because we have the clip from the previous policy? and the update is on the basis of the improvement to the positive response selection. I understand that this might lead to model hacking the reward system and generating nonsense

  4. Assistant

    You've hit on the most critical question in understanding this formula. Your intuition is spot on: the clip does control the step-by-step improvement.

    So, why isn't that enough? Why does a model trained without the KL penalty, even with the PPO clip, still fail?

    The problem is what I'll call "Compounding Drift" or the "Boiling the Frog" problem.

    Let's break down your perfectly logical-sounding premise and see where it falls apart.

    Your Premise:

    1. We start with a good model π_θ_0.
    2. In step 1, we make a small, safe update to get π_θ_1 because the clip prevents a large jump.
    3. In step 2, we start from π_θ_1 (which is now π_θ_old) and make another small, safe update to get π_θ_2.
    4. ...and so on. Every step is a small, controlled improvement.

    This sounds like it should work. But the fatal flaw is in step 3: the reference point for what is "safe" is constantly moving.

    • The clip ensures π_θ_2 is close to π_θ_1.
    • And it ensured π_θ_1 was close to π_θ_0.
    • But it does nothing to ensure π_θ_2 is close to the original, sane model π_θ_0 (π_ref).

    Over many thousands of these "safe" steps, the model can drift an enormous distance from its starting point. Each individual step was small, but the cumulative effect is huge.


    The Concrete "Reward Hacking" Example

    Let's make your "model hacking the reward system" idea more concrete.

    Imagine our reward model has a slight, unintentional bias: it gives a tiny bit more reward for answers that are overly enthusiastic and use lots of exclamation points.

    • π_ref (The original model): Generates normal, helpful text. E.g., "The capital of France is Paris."
    • Training Step 1:
      • The model (now π_θ) happens to generate: "The capital of France is Paris!"
      • The reward model gives this a slightly higher score. The Advantage A_i is positive.
      • The PPO update pushes the model to be slightly more likely to use exclamation points. The change is small, so the clip allows it.
      • The new policy π_θ_1 is now our π_θ_old for the next iteration. It is now very slightly more enthusiastic than the original model.
    • Training Step 2:
      • Starting from π_θ_1, the model generates: "Yes! The capital of France is Paris!"
      • The reward model gives an even higher score. Advantage is positive.
      • Another small, clipped update pushes the model even further in this direction.
      • The new policy π_θ_2 is now π_θ_old.
    • Training Step 10,000:
      • After thousands of these tiny, "safe" steps, the policy has drifted. The π_θ_old is now a model that is already pathologically enthusiastic.
      • To get a positive advantage now, the model might generate: "WOW!!! YES! It is with my GREATEST PLEASURE to inform you that the magnificent capital of the beautiful country of France is Paris!!!! I hope this helps!!!"
      • The model has become a nonsensical, reward-hacked sycophant.

    At every single step, the update was small and controlled relative to the previous step. But the model slowly boiled itself like a frog, not noticing the gradual change until it was in a completely different, undesirable state.


    So, what would happen if we removed the KL Term?

    1. Mode Collapse / Reward Hacking: As shown above, the model would find the easiest, simplest patterns to get reward (e.g., being obsequious, repeating certain phrases, always writing long answers) and would amplify them to an absurd degree. The diversity of its responses would collapse.

    2. Forgetting Pre-training: The RL training data is often narrow (e.g., focused on chatbot-style helpfulness). The pre-training data was vast (the entire internet). Without the KL penalty anchoring the model to its pre-trained state (π_ref), the RL updates would begin to overwrite its general knowledge. It might get better at saying "I don't know" politely, but it might forget how to do basic math or recall historical facts that weren't in the RL dataset.

    The Final Analogy: The Steering Wheel vs. The Anchor

    Think of it this way:

    • The PPO Clip is your steering wheel. It prevents you from suddenly jerking the car off the road in a single, violent motion. It ensures smooth, controlled turns.
    • The KL Penalty is your anchor (or your GPS). The steering wheel is great, but it doesn't stop you from making a series of smooth, controlled turns that slowly lead you hundreds of miles in the wrong direction. The KL penalty is the anchor that keeps your car from drifting too far from its original parking spot (π_ref), ensuring that while you explore the immediate area for better rewards, you don't end up in the wrong city entirely.

    You need both: the steering wheel for moment-to-moment stability, and the anchor for long-term sanity.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.