Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview· OpenAIHugging Face LMArena Search Arena 24K

math formula for dpo loss function

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    math formula for dpo loss function

  2. Assistant

    Direct Preference Optimization (DPO) is a technique designed to align language models with human preferences by directly optimizing the model using preference data, thereby bypassing the need for explicit reward modeling[1].

    The DPO loss function is formulated as follows:

    [ L_{\text{DPO}}(\pi_{\theta}; \pi_{\text{ref}}) = - \mathbb{E}{(x, y{w}, y_{l}) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi_{\theta}(y_{w} | x)}{\pi_{\text{ref}}(y_{w} | x)} - \beta \log \frac{\pi_{\theta}(y_{l} | x)}{\pi_{\text{ref}}(y_{l} | x)} \right) \right] ]

    In this equation:

    • ( \pi_{\theta} ) represents the policy being optimized.
    • ( \pi_{\text{ref}} ) denotes the reference policy.
    • ( x ) is the input prompt.
    • ( y_{w} ) and ( y_{l} ) are the preferred and less preferred responses, respectively.
    • ( \beta ) is a scaling parameter.
    • ( \sigma ) is the sigmoid function.
    • ( D ) is the dataset containing tuples ( (x, y_{w}, y_{l}) ).

    This loss function encourages the optimized policy ( \pi_{\theta} ) to assign higher probabilities to preferred responses ( y_{w} ) over less preferred ones ( y_{l} ), relative to the reference policy ( \pi_{\text{ref}} ). By doing so, DPO aligns the model's outputs with human preferences without the intermediate step of reward modeling[1].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.