show me the formula for DPO algorithm
- User
show me the formula for DPO algorithm
Assistant
The Direct Preference Optimization (DPO) algorithm aims to align language models with human preferences more directly than previous methods like Reinforcement Learning from Human Feedback (RLHF)[1][2]. Instead of training a separate reward model and then using RL to optimize the language model, DPO uses a specific loss function that directly optimizes the language model based on preference data[2][3].
The DPO loss function is typically formulated as follows:
LDPO(πθ; πref) = - E(x, yw, yl) ~ D [ log σ ( β log( πθ(yw|x) / πref(yw|x) ) - β log( πθ(yl|x) / πref(yl|x) ) ) ][4]
Let's break down the components of this formula:
- E(x, yw, yl) ~ D [...]: This denotes the expectation, or average, over a dataset
Dof preferences. Each data point inDconsists of a promptx, a preferred (winning) completiony_w, and a dispreferred (losing) completiony_l. - πθ: This is the language model (policy) being trained, parameterized by θ. πθ(y|x) represents the probability assigned by this model to generating completion
ygiven promptx[5]. - πref: This is a reference model, usually a copy of the initial supervised fine-tuned (SFT) model before DPO training begins. It's kept fixed during DPO training. πref(y|x) is the probability assigned by the reference model to completion
ygiven promptx. - log( πθ(yw|x) / πref(yw|x) ): This term calculates the log-ratio of the probability of the preferred completion
y_wunder the trained model πθ compared to the reference model πref. - log( πθ(yl|x) / πref(yl|x) ): Similarly, this term calculates the log-ratio of the probability of the dispreferred completion
y_lunder the trained model πθ compared to the reference model πref. - β: This is a hyperparameter (often called the temperature) that controls how much weight is given to the preference term versus how much the trained model πθ is allowed to deviate from the reference model πref. It essentially scales the reward implicit in the preference data and controls the strength of the KL divergence constraint between πθ and πref.
- σ(...): This is the logistic (sigmoid) function: σ(z) = 1 / (1 + e-z). It maps the difference in log-probability ratios to a probability between 0 and 1[6].
- log σ(...): Taking the logarithm of the sigmoid output. The negative sign outside the expectation means the overall objective is to minimize this loss, which is equivalent to maximizing the probability assigned to the preferred completions relative to the dispreferred ones, adjusted by the reference model probabilities and the β factor.
In simpler terms:
The DPO loss function aims to increase the probability that the trained model (πθ) assigns to the preferred responses (yw) while decreasing the probability it assigns to the dispreferred responses (yl), relative to the reference model (πref). The β parameter helps ensure the model doesn't change too drastically from its original behaviour, maintaining stability. Minimizing this loss directly optimizes the model for human preferences using a simple classification-like objective, avoiding the need for explicit reward modeling and reinforcement learning[2][7].
- E(x, yw, yl) ~ D [...]: This denotes the expectation, or average, over a dataset
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- gpt-4o-search-preview-high-locCompare side by side →
sonar-reasoningCompare side by side →
sonar-reasoning-pro-highCompare side by side →
- gpt-4o-mini-search-previewCompare side by side →
sonar-reasoningCompare side by side →
sonar-reasoning-pro-highCompare side by side →
- gpt-4o-mini-search-previewCompare side by side →