Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Summary

This paper introduces Direct Preference Optimization (DPO), a novel approach for aligning language models with human preferences. DPO offers a simplified alternative to Reinforcement Learning from Human Feedback (RLHF) by directly optimizing the language model to match preferred outputs. Instead of training a separate reward model and then using RL, DPO reformulates the process, showing that the language model implicitly encodes the reward function. This reformulation enables a direct optimization objective that's easier to implement and more stable than traditional RLHF, making it possible to train large language models faster and more effectively. The paper demonstrates strong performance on various language tasks by aligning language models with human feedback, effectively improving generation quality and coherence, while reducing computational cost. The main idea is that the language model itself can be trained on preference data using a closed-form solution that can be derived from the reward model. The paper provides theoretical justifications and empirical validation, demonstrating DPO's efficacy.

Key Takeaways

1. DPO eliminates the need for explicit reward model training, streamlining the alignment process.
2. DPO provides a more stable and efficient alternative to RLHF for aligning language models.
3. DPO directly optimizes the language model using a preference-based objective function.
4. DPO allows for faster and more scalable training of language models aligned with human preferences.