math formula for dpo loss function
- User
math formula for dpo loss function
- Assistant
Direct Preference Optimization (DPO) is a technique designed to align language models with human preferences by directly optimizing the model using preference data, thereby bypassing the need for explicit reward modeling[1].
The DPO loss function is formulated as follows:
[ L_{\text{DPO}}(\pi_{\theta}; \pi_{\text{ref}}) = - \mathbb{E}{(x, y{w}, y_{l}) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi_{\theta}(y_{w} | x)}{\pi_{\text{ref}}(y_{w} | x)} - \beta \log \frac{\pi_{\theta}(y_{l} | x)}{\pi_{\text{ref}}(y_{l} | x)} \right) \right] ]
In this equation:
- ( \pi_{\theta} ) represents the policy being optimized.
- ( \pi_{\text{ref}} ) denotes the reference policy.
- ( x ) is the input prompt.
- ( y_{w} ) and ( y_{l} ) are the preferred and less preferred responses, respectively.
- ( \beta ) is a scaling parameter.
- ( \sigma ) is the sigmoid function.
- ( D ) is the dataset containing tuples ( (x, y_{w}, y_{l}) ).
This loss function encourages the optimized policy ( \pi_{\theta} ) to assign higher probabilities to preferred responses ( y_{w} ) over less preferred ones ( y_{l} ), relative to the reference policy ( \pi_{\text{ref}} ). By doing so, DPO aligns the model's outputs with human preferences without the intermediate step of reward modeling[1].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.