Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-exp-03-25-grounding· GoogleHugging Face LMArena Search Arena 24K

show me the formula for DPO algorithm

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    show me the formula for DPO algorithm

  2. Gemini Assistant

    The Direct Preference Optimization (DPO) algorithm aims to align language models with human preferences more directly than previous methods like Reinforcement Learning from Human Feedback (RLHF)[1][2]. Instead of training a separate reward model and then using RL to optimize the language model, DPO uses a specific loss function that directly optimizes the language model based on preference data[2][3].

    The DPO loss function is typically formulated as follows:

    LDPO(πθ; πref) = - E(x, yw, yl) ~ D [ log σ ( β log( πθ(yw|x) / πref(yw|x) ) - β log( πθ(yl|x) / πref(yl|x) ) ) ][4]

    Let's break down the components of this formula:

    1. E(x, yw, yl) ~ D [...]: This denotes the expectation, or average, over a dataset D of preferences. Each data point in D consists of a prompt x, a preferred (winning) completion y_w, and a dispreferred (losing) completion y_l.
    2. πθ: This is the language model (policy) being trained, parameterized by θ. πθ(y|x) represents the probability assigned by this model to generating completion y given prompt x[5].
    3. πref: This is a reference model, usually a copy of the initial supervised fine-tuned (SFT) model before DPO training begins. It's kept fixed during DPO training. πref(y|x) is the probability assigned by the reference model to completion y given prompt x.
    4. log( πθ(yw|x) / πref(yw|x) ): This term calculates the log-ratio of the probability of the preferred completion y_w under the trained model πθ compared to the reference model πref.
    5. log( πθ(yl|x) / πref(yl|x) ): Similarly, this term calculates the log-ratio of the probability of the dispreferred completion y_l under the trained model πθ compared to the reference model πref.
    6. β: This is a hyperparameter (often called the temperature) that controls how much weight is given to the preference term versus how much the trained model πθ is allowed to deviate from the reference model πref. It essentially scales the reward implicit in the preference data and controls the strength of the KL divergence constraint between πθ and πref.
    7. σ(...): This is the logistic (sigmoid) function: σ(z) = 1 / (1 + e-z). It maps the difference in log-probability ratios to a probability between 0 and 1[6].
    8. log σ(...): Taking the logarithm of the sigmoid output. The negative sign outside the expectation means the overall objective is to minimize this loss, which is equivalent to maximizing the probability assigned to the preferred completions relative to the dispreferred ones, adjusted by the reference model probabilities and the β factor.

    In simpler terms:

    The DPO loss function aims to increase the probability that the trained model (πθ) assigns to the preferred responses (yw) while decreasing the probability it assigns to the dispreferred responses (yl), relative to the reference model (πref). The β parameter helps ensure the model doesn't change too drastically from its original behaviour, maintaining stability. Minimizing this loss directly optimizes the model for human preferences using a simple classification-like objective, avoiding the need for explicit reward modeling and reinforcement learning[2][7].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.