Fine-Tuning & Model Customization · TRACK 03/04
RLHF & Preference Training
How raw models learn what humans want: RLHF, DPO, reward models, GRPO.
// THE TRACK
01 · START HERERLHFFollow the full RLHF pipeline — human rankings, reward model, RL updates — and understand why it made chatbots feel helpful instead of feral.BEGINNERDirect Preference OptimizationUnderstand how DPO collapses the three-stage RLHF pipeline into a single training pass — and why that simplicity is both its strength and its ceiling.INTERMEDIATEReward ModelsUnderstand the small model that turns messy human preferences into a single score, and why it is the hidden engine that makes RLHF work.INTERMEDIATEPPO vs DPOSee exactly where PPO and DPO diverge so you can choose the simpler pipeline without giving up alignment quality.INTERMEDIATEGRPOUnderstand the lean RL method behind modern reasoning models: how GRPO scores answers relative to a group and skips PPO's expensive value model.ADVANCEDORPO & KTOLearn how ORPO merges preference alignment into supervised fine-tuning and how KTO trains on simple good/bad labels, plus when each beats classic DPO.ADVANCEDHugging Face TRLYou will understand what TRL is, which post-training methods its trainers implement, and why it is the go-to library for preference tuning.ADVANCEDverlYou will understand what verl is, how it combines distributed training with fast inference for RL post-training, and when teams reach for it.ADVANCEDOpenRLHFYou will understand what OpenRLHF is, how it uses Ray to scale RLHF, and which preference/RL algorithms it supports.ADVANCED