math formula for dpo loss function
- User
math formula for dpo loss function
- Assistant
Direct Preference Optimization (DPO) is a method designed to fine-tune language models directly based on human preference data, bypassing the need for an explicit reward model. The core of DPO is its loss function, which aligns the model's output probabilities with observed human preferences.
Given a dataset of preference pairs ((x, y_w, y_l)), where (x) is the input prompt, (y_w) is the preferred response, and (y_l) is the less preferred response, the DPO loss function is formulated as:
[ \mathcal{L}{\text{DPO}}(\pi\theta; \pi_{\text{ref}}) = - \mathbb{E}{(x, y_w, y_l) \sim D} \left[ \log \sigma\left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] ]
In this equation:
-
(\pi_\theta(y \mid x)) represents the probability assigned by the model with parameters (\theta) to response (y) given input (x).
-
(\pi_{\text{ref}}(y \mid x)) denotes the probability assigned by a reference model to response (y) given input (x).
-
(\beta) is a scaling parameter that controls the trade-off between aligning with human preferences and staying close to the reference model.
-
(\sigma(\cdot)) is the sigmoid function, defined as (\sigma(z) = \frac{1}{1 + e^{-z}}).
This loss function encourages the model to assign higher relative probabilities to preferred responses compared to less preferred ones, effectively aligning the model's outputs with human preferences. The derivation and detailed explanation of this loss function can be found in the original DPO paper[1]:
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.