Is RLHF still used for fine tuning sota models?
- User
Is RLHF still used for fine tuning sota models?
Assistant
Yes, RLHF (Reinforcement Learning from Human Feedback) continues to be an important technique for fine-tuning state-of-the-art language models as of early 2025, though the landscape has evolved significantly.
While RLHF remains widely used, researchers have developed several alternatives and improvements:
Direct Preference Optimization (DPO) has gained traction as a simpler alternative that achieves similar results without the complexity of training a separate reward model and using RL algorithms. DPO directly optimizes the policy using preference data.
Constitutional AI (CAI) and other self-supervision methods reduce reliance on human feedback by having models critique and revise their own outputs based on a set of principles.
Iterative refinement approaches like RLAIF (Reinforcement Learning from AI Feedback) use AI systems to generate feedback, though these often still incorporate human oversight.
Many leading models now use hybrid approaches - combining RLHF with other techniques rather than relying on it exclusively. The exact methods used by specific commercial models are often proprietary, but RLHF or RLHF-inspired techniques remain part of the standard toolkit for aligning language models with human preferences and values.
The field continues to actively research more efficient and scalable alternatives to traditional RLHF, but it hasn't been completely replaced in practice.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.