Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

What is meant by constraint on policy shift of reward model in RLHF?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with MiniMaxminimax-m1’s answer.
  1. User

    What is meant by constraint on policy shift of reward model in RLHF?

  2. Qwen Assistant

    Your question touches on a critical concept in Reinforcement Learning from Human Feedback (RLHF): the constraint on policy shift when optimizing a language model using a learned reward model. Let me break this down clearly:


    What is "Policy Shift" in RLHF?

    In RLHF, the policy refers to the language model (LLM) that generates responses. During training, the policy is updated to maximize the reward predicted by the reward model. A policy shift occurs when the model's behavior changes significantly between training steps (e.g., generating very different responses than before).


    Why Constrain Policy Shift?

    The reward model in RLHF is learned from human feedback (e.g., preferences or ratings) and is therefore imperfect. It only approximates human preferences based on the training data it was exposed to. If the policy changes too much during reinforcement learning, it may exploit errors or blind spots in the reward model, leading to undesirable behaviors (e.g., "reward hacking" or degenerate outputs).

    To prevent this, RLHF imposes constraints on how much the policy can change at each update step. This ensures that the policy stays close to the reward model's "training distribution" and avoids overfitting to the reward model's inaccuracies.


    How Are Policy Shifts Constrained?

    Several techniques are used to limit policy shift in RLHF:

    1. KL Divergence Penalty (Trust Region Methods)

    • The most common approach is to add a KL divergence penalty between the current policy and the initial policy (or a reference policy).
    • Example: In PPO (Proximal Policy Optimization), the loss function includes a term like: Loss=E[rt−β⋅DKL(pθ(a∣s)∥pθold(a∣s))]\text{Loss} = \mathbb{E}[r_t - \beta \cdot D_{\text{KL}}(p_{\theta}(a|s) \parallel p_{\theta_{\text{old}}}(a|s))] Here, rtr_t is the reward, DKLD_{\text{KL}} is the KL divergence between the new and old policies, and β\beta controls the strength of the penalty.
    • This penalty discourages the policy from deviating too far from its original behavior, ensuring stability.

    2. Clipped Surrogate Objective (PPO)

    • PPO also uses a clipped probability ratio to limit how much the policy can change in response to rewards. This prevents overly aggressive updates that might exploit reward model errors.

    3. Reference Model (e.g., DPO)

    • Some newer methods like Direct Preference Optimization (DPO) implicitly constrain policy shift by comparing the policy's outputs to a reference model (e.g., the initial language model). This avoids explicit KL penalties while still ensuring stability.

    Why Is This Constraint Necessary?

    1. Reward Model Limitations:

      • The reward model is trained on a finite dataset of human preferences and may not generalize perfectly to novel or extreme policy behaviors.
      • Without constraints, the policy could "game" the reward model by generating outputs that score highly but violate human intent (e.g., overly verbose answers, hallucinations, or manipulation).
    2. Distributional Shift:

      • If the policy generates outputs far from the training data distribution, the reward model's predictions become unreliable. Constraints keep the policy within the reward model's valid domain.
    3. Stability During Training:

      • Large policy shifts can destabilize reinforcement learning, causing oscillations or divergence. Constraints ensure smoother optimization.

    Example of Policy Shift Without Constraints

    Imagine a reward model trained to prefer concise answers. If the policy learns to output single-word answers (e.g., "Yes" to every question), it might maximize the reward score but fail to be helpful. A KL penalty would discourage such extreme behavior by penalizing deviations from the initial policy (which likely generated more natural responses).


    Key Takeaway

    • Constraint on policy shift = A mechanism to prevent the language model from changing too much during RLHF training.
    • Purpose: Ensure the reward model remains reliable, avoid exploitation of reward model errors, and maintain stable training.
    • Methods: KL divergence penalties, clipped objectives (PPO), or reference models (DPO).

    This constraint is essential for aligning the LLM with human preferences in a robust and safe way.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.