> ## Documentation Index
> Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# DPO

> A simple and stable method for fine-tuning language models using human or synthetic preference data without reinforcement learning.

## Model Description

Direct Preference Optimization (DPO) is a training method for fine-tuning language models using preference data — pairs of responses labeled as *preferred* vs *rejected* — without requiring reinforcement learning or a separate reward model. DPO was introduced in Rafailov et al. (2023). [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290).

## Code Structure

This implementation consists of:

* [`configs/`](https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/dpo/configs): YAML configuration files for DPO fine-tuning runs.
* [`model.py`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/dpo/model.py): Defines the DPO training logic, including the contrastive loss function used to compare chosen and rejected completions.

## Available Configurations

| Configuration                                                                                                                                        | Description                                                                          |
| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`params_zephyr_7b_dpo.yaml`](https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/dpo/configs/params_zephyr_7b_dpo.yaml) | DPO training config for a 7B model using preference-labeled instruction tuning data. |

## References

* Rafailov, R., et al. (2023). [Direct Preference Optimization](https://arxiv.org/abs/2305.18290)
* Ouyang, L., et al. (2022). [Training language models to follow instructions with human feedback (InstructGPT)](https://arxiv.org/abs/2203.02155)
* Christiano, P. et al. (2017). [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741)