What Is GRPO? Group Relative Policy Optimization Explained

// In plain English

GRPO — Group Relative Policy Optimization — is a way to train a language model with reinforcement learning so it gets better at tasks where you can check whether the answer is right. Math problems, code that either passes the tests or doesn't, puzzles with a known solution: for these, you don't need a human to score every reply. You just need the model to try an answer, find out if it worked, and shift toward the strategies that work more often.

Here is the everyday version. Imagine a teacher hands the same hard problem to a small group of students at once. Five students write five different solutions. The teacher grades all five, then tells each student not an absolute score but a relative one: "you did better than the group average" or "you did worse." Students above the group average are encouraged to keep doing what they did; students below it are nudged to change. Nobody needs a separate expert predicting in advance how hard the problem was — the group itself sets the bar.

That single idea is the whole trick. For each prompt, GRPO samples a group of answers from the current model, scores them with a reward, and judges each answer relative to its own group's average. Answers that beat the group get reinforced; answers below it get suppressed. Because the group provides the baseline, GRPO throws away a big, expensive piece that older methods needed — a second neural network that tried to guess that baseline in advance.

// Why it matters

To see why GRPO is a big deal, you have to know what it replaced. The standard reinforcement-learning algorithm for tuning LLMs was PPO (Proximal Policy Optimization). PPO works, and it powered the first generation of RLHF models, but it is heavy. It requires running four models at training time: the policy you're training, a frozen reference copy, a reward model, and — the costly one — a value model (also called a critic) that learns to predict the expected reward of a partial answer. That value model is roughly the same size as the model you're training, so it can double your memory and compute just for the training machinery.

GRPO's contribution is simple and practical: drop the value model. Instead of training a separate network to estimate the baseline reward, it estimates that baseline empirically, from a group of sampled answers to the same prompt. The average reward of the group is the baseline. This removes one large network entirely, which is why GRPO is dramatically cheaper to run and easier to scale.

• Less memory, less compute. No critic network means you free up the memory and FLOPs PPO spent training and storing a second large model. On the same hardware you can train a bigger policy or use larger batches.
• Fewer moving parts. A value model is finicky to train and a common source of instability in PPO. Removing it removes a whole class of tuning headaches.
• A natural fit for verifiable rewards. GRPO shines when the reward is cheap and objective — did the code pass the unit tests, is the final math answer correct, did the output match the required format. You can sample many answers and grade them all automatically.
• It scales to reasoning. Because grading is automatic and the baseline is free, you can afford to sample large groups over millions of problems — exactly what training a model to reason through long chains of thought requires.

Who should care? Anyone training or fine-tuning models on tasks with a clear notion of correct: coding assistants, math and science tutors, agents that must follow a strict output schema, anything you can wrap a checker around. GRPO is the recipe that made large-scale reinforcement learning on reasoning tasks affordable enough to become mainstream.

// How it works

GRPO runs the same loop over and over: pick a prompt, sample a group of answers from the current model, score every answer, convert those scores into a relative advantage, and nudge the model up on good answers and down on bad ones. The clever part is entirely in how the advantage is computed.

Step 1 — sample a group, not one answer

For a single prompt, GRPO doesn't generate one completion; it generates a group of them — often 8, 16, or more — using the current model with sampling turned on, so the answers genuinely differ. Each answer is one attempt the model could have made. Together they form a little tournament for that prompt.

Step 2 — score every answer with a reward

Each answer in the group gets a reward. For verifiable tasks this is just a checker: run the generated code against tests, compare the final math answer to the ground truth, or apply a regex for the required format. For more subjective tasks you can still use a trained reward model. Either way you end up with one number per answer in the group.

Step 3 — turn rewards into group-relative advantage

This is the heart of GRPO. Instead of asking a value model "how good was this answer compared to what we expected," GRPO compares each answer to its own group. It computes the group's mean reward and standard deviation, then normalizes each answer's reward against them. The result is the advantage:

# rewards: one number per answer in the group, e.g. [1, 0, 1, 1, 0]
mean = rewards.mean()
std  = rewards.std()

# Each answer is scored relative to its own group.
advantage = (rewards - mean) / (std + 1e-8)

# Answers above the group average get a POSITIVE advantage
# (reinforce them); answers below get a NEGATIVE one (suppress them).
# If every answer in the group is equally good or bad, the advantages
# are ~0 and that prompt teaches the model nothing this step.

Notice what just happened. The group itself supplied the baseline that PPO needed a whole value network to predict. If three of five answers solved the problem, the two that failed get pushed down and the three that succeeded get pushed up. Every token in a good answer shares that answer's positive advantage; every token in a bad one shares its negative advantage.

Step 4 — update the policy (with guardrails)

Finally GRPO updates the model to make high-advantage answers more likely and low-advantage ones less likely. It keeps two safety rails borrowed from PPO: a clipped objective that limits how far the model can move in one step (so a single batch can't wreck it), and a KL penalty that keeps the model from drifting too far from a frozen reference copy (so it doesn't forget its general abilities while chasing the reward). The reward model and value model are gone, but these two stabilizers stay.

Then it repeats — millions of times, over millions of prompts. Slowly the model's distribution shifts toward whatever the group comparisons keep rewarding.

// GRPO vs PPO at a glance

GRPO and PPO answer the same question — how good was this answer relative to a baseline? — but they get the baseline in opposite ways. PPO predicts it with a learned value network; GRPO measures it from a group of samples. Everything else follows from that one choice.

// Where GRPO fits among fine-tuning methods

GRPO is a reinforcement-learning method, so it sits in the same family as RLHF and DPO, not plain supervised fine-tuning. It usually comes after a supervised step, as the polishing stage that teaches behavior rather than facts. A common modern pipeline looks like this:

It helps to see how the three RL-flavored methods differ in what they need from you:

Two practical signals tell you GRPO is the right tool. First, you can grade answers automatically — a verifier, test suite, or exact-match check that runs cheaply at scale. Second, the model needs to explore: there are many valid solution paths, and you want it to discover good ones rather than imitate fixed demonstrations. If instead you already have curated good/bad pairs and want the simplest possible setup, DPO is lighter. If your reward is dense and subjective and you have a solid reward model, classic PPO still applies.

// Common pitfalls

GRPO is conceptually clean but easy to mis-tune. Most failures trace back to the reward and the group, not the optimizer itself.

• Reward hacking. The model optimizes exactly what you measure, not what you mean. If your checker rewards a final number but ignores the reasoning, the model may guess or write nonsense that happens to match. Reward design is the hard part — verify the whole behavior you want, not a proxy.
• Groups with no signal. If every answer in a group is right (or every one is wrong), the advantages collapse to roughly zero and that prompt teaches nothing. You want problems at the edge of the model's ability, where the group splits — too easy or too hard wastes the sample budget.
• Group too small. A tiny group gives a noisy mean and standard deviation, so the advantage estimate is unstable. Larger groups give a steadier baseline but cost more samples; this is the central tradeoff to tune.
• Letting the model drift. Chasing the reward too hard can make a model forget general skills, lose its calibration, or produce repetitive, gamed outputs. The KL penalty against the reference model exists precisely to bound this — don't set it to zero.
• No held-out evaluation. Reward going up is not the same as the model getting better. Always evaluate on tasks the reward never touched, or you'll ship a model that aced its own training game and nothing else.

// Going deeper

Once the basic loop makes sense, a few finer points separate a toy implementation from a working one.

Token-level vs sequence-level credit. In the simplest GRPO, every token in an answer inherits that whole answer's advantage. That is coarse — a great answer might contain a few bad tokens and vice versa. Implementations differ in how they spread credit across the sequence and how they handle the length of long chains of thought, and the details meaningfully affect stability. The exact normalization (whether you divide by standard deviation, how you treat length) is an active area where variants disagree.

The KL term, more carefully. GRPO keeps a KL penalty to a frozen reference model, but where and how it's applied varies. Some formulations fold it into the reward, others into the loss. A stronger KL keeps the model close to its starting point and safer but slows learning; a weaker one learns faster but risks collapse. It is one of the most important knobs you'll touch.

Why it pairs with reasoning. Long step-by-step reasoning is hard to supervise directly — you rarely have a labeled "correct chain of thought." But you often can check the final answer. GRPO turns that single end-of-answer reward into a training signal for the entire reasoning trace: sample many reasoning paths, keep the ones that reach the right answer, push toward them. This is why letting the model generate long, exploratory thoughts and rewarding only correct conclusions can make reasoning emerge without anyone scripting the steps.

On-policy and stale samples. GRPO is an on-policy method: the samples should come from roughly the current model. In practice you generate a big batch, then take several gradient steps on it for efficiency, which makes the later steps slightly off-policy. The clipping objective is what keeps those reused samples from pushing the model too far. Push reuse too hard and training destabilizes; this is the same balance PPO manages.

Where to go next. If you have clean preference pairs and want a simpler path, study DPO, which skips sampling and reward models entirely. To understand the broader feedback paradigm GRPO descends from, read what RLHF is. And to place all of this in the larger picture of changing a model's behavior cheaply, see what fine-tuning is and full fine-tuning vs PEFT. The durable lesson of GRPO is that you don't always need a learned critic to know how good an answer is — sometimes a handful of the model's own attempts, graded against each other, is baseline enough.