Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview-high-loc· OpenAIHugging Face LMArena Search Arena 24K

math formula for dpo loss function

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview’s answer.
  1. User

    math formula for dpo loss function

  2. Assistant

    Direct Preference Optimization (DPO) is a method designed to fine-tune language models directly based on human preference data, bypassing the need for an explicit reward model. The core of DPO is its loss function, which aligns the model's output probabilities with observed human preferences.

    Given a dataset of preference pairs ((x, y_w, y_l)), where (x) is the input prompt, (y_w) is the preferred response, and (y_l) is the less preferred response, the DPO loss function is formulated as:

    [ \mathcal{L}{\text{DPO}}(\pi\theta; \pi_{\text{ref}}) = - \mathbb{E}{(x, y_w, y_l) \sim D} \left[ \log \sigma\left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] ]

    In this equation:

    • (\pi_\theta(y \mid x)) represents the probability assigned by the model with parameters (\theta) to response (y) given input (x).

    • (\pi_{\text{ref}}(y \mid x)) denotes the probability assigned by a reference model to response (y) given input (x).

    • (\beta) is a scaling parameter that controls the trade-off between aligning with human preferences and staying close to the reference model.

    • (\sigma(\cdot)) is the sigmoid function, defined as (\sigma(z) = \frac{1}{1 + e^{-z}}).

    This loss function encourages the model to assign higher relative probabilities to preferred responses compared to less preferred ones, effectively aligning the model's outputs with human preferences. The derivation and detailed explanation of this loss function can be found in the original DPO paper[1]:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.