Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514· AnthropicHugging Face LMArena Human Preference 140K

How does PPO update the parameters in a LLM

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    How does PPO update the parameters in a LLM

  2. Claude Assistant

    PPO (Proximal Policy Optimization) updates LLM parameters through a careful balance of maximizing reward while preventing the model from deviating too far from its initial behavior. Here's how the process works:

    Core Update Mechanism

    PPO updates parameters by optimizing a clipped surrogate objective function. For each token in the generated text, it calculates:

    1. Advantage estimates: How much better or worse an action (token choice) was compared to the expected value
    2. Probability ratios: The ratio between the new policy's probability of taking an action and the old policy's probability

    The key update equation involves:

    L = min(r(θ) * A, clip(r(θ), 1-ε, 1+ε) * A)
    

    Where r(θ) is the probability ratio, A is the advantage, and ε is the clipping parameter (typically 0.1-0.2).

    The Update Process

    During each PPO iteration:

    1. Generate responses: The current policy generates multiple responses to prompts
    2. Compute rewards: A reward model scores these responses (often based on human preferences)
    3. Calculate advantages: Using the rewards and a value function, compute how much better each token choice was than expected
    4. Multiple epochs: Unlike standard policy gradient methods, PPO performs multiple gradient updates on the same batch of data
    5. KL penalty: Add a penalty term to prevent the updated model from diverging too much from the reference model

    Key Differences for LLMs

    When applying PPO to LLMs specifically:

    • Token-level updates: The policy outputs a probability distribution over the entire vocabulary for each position
    • Sequence-level rewards: Often the reward is only given at the end of the generation, requiring credit assignment back through the sequence
    • Reference model constraint: A frozen copy of the initial model prevents the fine-tuned model from "forgetting" its original capabilities
    • Value function: A separate head or model estimates the expected future reward from each state

    The parameters are updated via gradient descent on this objective, typically using Adam optimizer with careful learning rate scheduling. The clipping mechanism ensures stable training by preventing overly aggressive updates that could destabilize the model's behavior.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.