Skip to content
The Chat Archive
sonar· Perplexity LMArena Search Arena 24K

show me the DPO algorithm formula

  1. User

    show me the DPO algorithm formula

  2. Assistant

    The Direct Preference Optimization (DPO) algorithm is a method for fine-tuning large language models (LLMs) to better align with human preferences without using reinforcement learning. Instead, it uses a direct classification approach based on preference data. The core of DPO is its loss function, which encourages the model to assign higher probabilities to preferred (chosen) responses and lower probabilities to less preferred (rejected) responses, while constraining the model to stay close to a reference policy.

    DPO Loss Function Formula

    The DPO loss function can be expressed as:

    LDPO=−Ex,yw,yl[log⁡σ(β(log⁡πθ(yw∣x)−log⁡πθ(yl∣x)−(log⁡πref(yw∣x)−log⁡πref(yl∣x))))]\mathcal{L}_{DPO} = - \mathbb{E}_{x, y_w, y_l} \left[ \log \sigma \left( \beta \left( \log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x) - \left( \log \pi_{ref}(y_w|x) - \log \pi_{ref}(y_l|x) \right) \right) \right) \right]

    Where:

    • (x) is the input prompt.
    • (y_w) is the preferred (winning) response.
    • (y_l) is the less preferred (losing) response.
    • (\pi_\theta) is the policy (model) being optimized.
    • (\pi_{ref}) is the reference policy (usually the original pretrained model, frozen).
    • (\log \pi_\theta(y|x)) is the log-probability of response (y) given prompt (x) under the current model.
    • (\log \pi_{ref}(y|x)) is the log-probability under the reference model.
    • (\sigma(\cdot)) is the sigmoid function.
    • (\beta) is a hyperparameter controlling how strongly the model is encouraged to deviate from the reference policy.

    Explanation

    • The term inside the sigmoid compares the difference in log-probabilities between the preferred and rejected responses under the current model, adjusted by the same difference under the reference model.
    • This encourages the model to increase the relative likelihood of preferred responses compared to rejected ones, but penalizes large deviations from the reference model to prevent undesirable behavior.
    • The expectation (\mathbb{E}_{x, y_w, y_l}) is taken over the dataset of prompts and paired preferred/rejected responses.

    This loss function effectively turns preference comparisons into a binary classification problem, where the model learns to assign higher probability to preferred responses directly, without needing a separate reward model or reinforcement learning optimization steps.

    Summary

    • DPO replaces the reward model and reinforcement learning in RLHF with a direct classification loss based on the Bradley-Terry model of preferences.
    • The loss uses the sigmoid of the difference in log-probabilities adjusted by the reference model to balance improvement and stability.
    • The (\beta) parameter controls the trade-off between following human preferences and staying close to the original model behavior.

    This approach simplifies training, improves stability, and often yields better alignment with human preferences compared to traditional RLHF methods[1][2][5][6][4].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.