ORPO and KTO: Newer Alternatives to DPO, Explained

// In plain English

ORPO and KTO are two preference-training methods that came after DPO. They chase the same goal as DPO and RLHF — make a model answer the way humans prefer — but each removes a different piece of friction that DPO still carries.

Picture training a new barista. DPO is the standard way: first you teach them the menu and the basic moves (a supervised stage), then, in a second pass, you show them pairs of drinks — this one is good, this one is bad — and nudge them toward the good ones. Two stages, and you also keep a frozen copy of the trainee around as a reference to compare against.

• ORPO (Odds Ratio Preference Optimization) folds those two stages into one. While the barista learns the menu, the very same lesson also says and prefer this over that — so there is no separate preference pass and no frozen reference copy to keep in memory. One training run, done.
• KTO (Kahneman-Tversky Optimization) changes the data instead of the stages. It does not need matched good/bad pairs at all. It learns from a loose pile of individual drinks each marked simply thumbs-up or thumbs-down — exactly the kind of feedback that piles up naturally from real users.

// Why it matters

DPO already simplified alignment a lot, but it left two practical pain points. ORPO and KTO each remove one of them, and which pain you feel decides which method helps you.

Pain point 1 — two stages and a reference model (ORPO's target)

DPO assumes you have already done a supervised fine-tuning (SFT) pass, and then it runs a second pass on the preference data. It also keeps a frozen reference copy of the model in memory to stop the policy drifting too far. That means two training runs to manage and roughly two model copies of GPU memory. ORPO merges the SFT objective and the preference objective into one loss in one pass, with no reference model — less pipeline to babysit, and a smaller memory footprint. For a small team fine-tuning on limited hardware, dropping a stage and a model copy is a real win.

Pain point 2 — preference pairs are expensive to collect (KTO's target)

DPO needs (prompt, chosen, rejected) triples: for every prompt, two answers, with a human deciding which is better. That ranked-pair data is slow and costly to gather. But most real products already collect a different, cheaper signal — a thumbs-up / thumbs-down button, a deleted reply, a regenerated answer, a support ticket marked resolved or not. Each of these is a single answer with a binary good/bad label, not a pair. KTO learns directly from that unpaired binary feedback, so you can align a model on data you are probably already sitting on.

// How it works

Both methods are still supervised training — a forward pass, a loss, backprop — like DPO. The differences are in what the loss looks at. Here is the full landscape next to plain DPO.

ORPO — the odds-ratio trick in one pass

ORPO's loss has two parts added together. The first is the ordinary SFT loss: maximize the probability of the chosen answer, exactly as plain instruction-tuning would. The second is a small odds-ratio penalty that compares how likely the model is to produce the chosen answer versus the rejected one, and pushes those apart. Because the SFT term already pins the model to good answers, ORPO does not need a separate reference model to hold it in place — the supervised term plays that role.

L_ORPO = L_SFT  +  λ * L_OR

where:
  L_SFT = standard cross-entropy on the CHOSEN answer
          (the normal "learn to produce good text" loss)

  L_OR  = -log sigmoid( log[ odds(chosen)  / odds(rejected) ] )
          odds(y) = P(y | x) / (1 - P(y | x))
          (pushes chosen's odds above rejected's odds)

  λ     = weight on the preference term (small, e.g. 0.1)

No reference model. One loss. One training pass over a base model.

The intuition: the odds ratio asks "how many times more likely is the good answer than the bad one?" Maximizing it widens the gap gently, while the SFT term keeps the model anchored to fluent, correct text. The result is a single stage that both teaches the task and instills the preference.

KTO — learning from thumbs-up and thumbs-down

KTO never compares two answers to the same prompt. Instead, every training example is one (prompt, answer) with a single label: desirable or undesirable. The loss, borrowed from Kahneman and Tversky's prospect theory, rewards the model for raising the probability of desirable answers and penalizes it for raising the probability of undesirable ones — but it weights a loss (an undesirable example) more heavily than an equivalent gain, mirroring how humans are loss-averse. Like DPO, it uses a frozen reference model as a baseline to measure each answer against.

Because the examples are unpaired, you do not need both a good and a bad answer for the same prompt. A pile of liked answers and a pile of disliked answers — even to entirely different prompts — is enough. KTO does ask you to keep the two piles from being wildly imbalanced; if one label dominates, you tune a weight to rebalance them (more on that below).

// A worked example with TRL

Hugging Face's TRL library ships an ORPOTrainer and a KTOTrainer, so the code looks almost identical to the DPO example — the difference is the data format and whether a reference model is passed. Both snippets are minimal; real runs add evaluation, LoRA, and memory tricks.

ORPO — one pass, no reference model

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import ORPOTrainer, ORPOConfig

# Start from a BASE model (no separate SFT step needed)
model_name = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Same pair format as DPO: prompt / chosen / rejected
data = load_dataset("json", data_files="pairs.jsonl", split="train")

config = ORPOConfig(
    output_dir="orpo-model",
    beta=0.1,            # weight on the odds-ratio (preference) term
    num_train_epochs=1,
)

trainer = ORPOTrainer(
    model=model,         # NOTE: no ref_model argument exists
    args=config,
    train_dataset=data,
    processing_class=tokenizer,
)
trainer.train()

KTO — unpaired good/bad labels

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import KTOTrainer, KTOConfig

model_name = "Qwen/Qwen3-0.6B"   # an SFT checkpoint works best
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Each row: 'prompt', 'completion', 'label' (True = desirable, False = not)
# No pairing — rows are independent single answers.
data = load_dataset("json", data_files="binary.jsonl", split="train")

config = KTOConfig(
    output_dir="kto-model",
    beta=0.1,
    # rebalance if you have far more 'good' rows than 'bad' (or vice versa)
    desirable_weight=1.0,
    undesirable_weight=1.0,
)

trainer = KTOTrainer(
    model=model,
    args=config,
    train_dataset=data,
    processing_class=tokenizer,
)
trainer.train()

// Which method to pick

The decision almost always comes down to one question: what does your data look like? Pick the method that matches the feedback you can actually collect, not the one with the newest paper.

A useful real-world pattern: collect binary feedback from production (cheap, abundant) and align with KTO; if you later run a careful human-labeling round that produces true pairs, switch to ORPO or DPO for that higher-quality batch. The methods are tools in the same drawer, chosen by the data on hand.

// Going deeper

Once the basics click, a few nuances separate a demo from a solid run.

*ORPO needs a base* model, not an instruct model.** ORPO's whole point is that it does the SFT itself. If you start from a model that has already been instruction-tuned and run ORPO on top, you are double-teaching the format and can wash out the original tuning. The intended recipe is base model → ORPO → done. Starting from an already-aligned checkpoint is a common beginner mistake.

KTO tolerates noisy, real-world labels — within reason. Because it never compares two answers head-to-head, it is more forgiving of the messy, inconsistent feedback that real users generate (one person's thumbs-up is another's thumbs-down). But that same looseness means a single bad answer mislabeled as "good" pulls the model the wrong way with no paired counter-example to balance it. Light filtering of obviously wrong labels still pays off.

The beta knob still matters. As in DPO, beta controls how strongly the preference signal pulls the model. Too high and the model overfits the preference data and gets brittle; too low and the alignment barely moves the needle. The common starting point is around 0.1 for all three methods, then tune by watching both a held-out preference accuracy and a general capability check so you do not trade away skills you already had — see how to evaluate a fine-tuned model.

*These are offline* methods.** Like DPO, ORPO and KTO train on a fixed dataset and never generate fresh answers from the current model mid-training. That makes them cheap and stable, but it means they cannot self-improve on tasks with a checkable right answer the way online RL can. For math, code, or anything with a verifier, teams reach for online methods like GRPO instead. ORPO and KTO shine for helpfulness, tone, and safety alignment on data you already have.

Where they sit in the bigger picture. ORPO and KTO are siblings of DPO in the preference-tuning family, which itself sits on top of supervised fine-tuning and is usually run cheaply with LoRA adapters rather than full fine-tuning. Understanding all of them as variations on one idea — nudge the model toward preferred behavior with a supervised-style loss — is the durable takeaway. The differences are about pipeline shape and data format, not about a fundamentally new kind of learning.