Frequency Penalty vs Presence Penalty: Stopping LLM Repetition

// In plain English

When a language model writes text, it picks one word at a time. At each step it produces a probability for every possible next token, and a sampler chooses one. Left alone, a model can fall into a rut: repeating the same phrase, looping the same sentence, or leaning on one favourite word again and again. Frequency penalty and presence penalty are two knobs that fight exactly this, by quietly lowering the odds of words the model has already used.

Picture a friend telling a story who keeps saying "basically" every few words. The penalties are like two different ways of nudging them. The frequency penalty is a friend who flinches a little more each time they hear "basically" — the tenth one annoys them far more than the first, so the pressure to stop grows with the count. The presence penalty is a friend who reacts the same the moment a word shows up at all: "you've used that word — now move on to something new," no matter whether it appeared once or twenty times.

Both penalties do the same physical thing — they subtract a number from a token's score before the model samples — but they count differently. Frequency penalty scales with how many times a token has appeared. Presence penalty is a flat amount applied once a token has appeared at all. That single difference is the whole topic.

// Why it matters

Repetition is one of the most common and most visible failures in generated text. A summary that restates the same point three times, a chatbot that opens every reply with the identical sentence, a product description that hammers one adjective, or — in the worst case — a model that gets stuck in an infinite loop, emitting the same line until it hits the token limit. All of these make output feel robotic and untrustworthy, and none of them are fixed by a bigger or smarter model alone.

Why does a capable model repeat itself at all? Because at each step it's just picking high-probability tokens, and language is genuinely repetitive — once a phrase has appeared, the patterns the model learned make that same phrase even more likely to come next. This creates a feedback loop: the more a token appears, the more the surrounding context points back to it. Low temperature makes this worse, because the model keeps taking the single safest token, and the safest token is often a word it just used.

• Long-form generation. Articles, reports, and stories drift into repeating phrases the longer they run. A small penalty keeps the vocabulary moving.
• Lists and brainstorming. Asking for 20 ideas, 50 product names, or a varied set of examples — without a penalty, the model recycles the same few concepts. Presence penalty in particular pushes it toward fresh territory.
• Runaway loops. A model that starts echoing one line can keep going to the end of its output budget, wasting tokens and money. A repetition penalty is the standard guardrail.
• Conversational tone. Bots that begin every message with "Sure, I can help with that!" feel canned. A light penalty across a turn adds variety.

// How it works

To see the mechanism, you need one idea from how sampling works. Before choosing a token, the model produces a logit for every token in its vocabulary — a raw score, where higher means more likely. Those logits are turned into probabilities (via softmax), and the sampler draws from that distribution. Both penalties act on the logits, before the probabilities are computed, by subtracting from the scores of tokens that have already appeared in the text so far.

The two formulas, in words

For each token, the system tracks how many times it has appeared so far in the generated text. The new score for a token is roughly:

new_logit = original_logit
            - presence_penalty  * (1 if token has appeared at all else 0)
            - frequency_penalty * (number of times token has appeared)

Read those two terms carefully — they are the entire distinction. The presence term is multiplied by 0 or 1: it switches on the first time a token appears and never grows after that. The frequency term is multiplied by the running count: it gets stronger every single time the token reappears. So a word used once feels both penalties equally, but a word used ten times feels the frequency penalty ten times as hard while the presence penalty stays flat.

Both penalties typically accept a value from about -2.0 to 2.0, with 0 meaning "off". Positive values discourage repetition; negative values encourage it (rarely useful — occasionally for a deliberately repetitive style). In practice useful positive values are small, often 0.1 to 1.0. Anything near the top of the range is aggressive and tends to distort the text.

// A worked example

Suppose you ask a model for a short product blurb and it returns this, with both penalties at zero:

Our amazing blender is amazing for smoothies. It is amazing
at crushing ice and amazing with frozen fruit. Truly amazing.

The word amazing keeps winning because each time it appears, the surrounding pattern makes it likely again. Now turn on a frequency penalty of around 0.6. The first amazing costs little, but by the third and fourth it has been penalised three and four times over, so its logit drops below the alternatives and the model is forced to vary:

Our amazing blender is perfect for smoothies. It crushes ice
with ease and handles frozen fruit beautifully. A real workhorse.

Now a different task: "List 15 hobbies." The model offers reading, hiking, cooking, then keeps circling back to those same few. Here presence penalty is the better tool. The moment reading has appeared even once, its flat penalty pushes the model to look elsewhere — you don't need the penalty to grow, you just need a steady shove toward unseen tokens so the list keeps expanding into new ideas.

Here is how you'd set them on a typical chat-completions-style API. Most SDKs accept the two parameters directly alongside temperature:

response = client.chat.completions.create(
    model="...",
    messages=[{"role": "user", "content": "List 15 hobbies."}],
    temperature=0.8,
    # flat shove toward words not yet used — good for variety/lists
    presence_penalty=0.6,
    # grows with repeats — good for killing loops & droning
    frequency_penalty=0.2,
)

// Which one to reach for

A simple rule of thumb: use frequency penalty when the same words or phrases repeat too much, and presence penalty when you want the model to cover more ground / new topics. They can be combined, but start with one.

These knobs are orthogonal to temperature and top-p/top-k. Temperature controls overall randomness; top-p/top-k control which candidates are even eligible. The penalties don't touch randomness at all — they only reshape the scores of tokens you've already produced. That's why a model can be highly repetitive even at high temperature (the repeated token is just that dominant), and why the right fix is a penalty, not cranking temperature up and hoping.

// Common pitfalls

These knobs are easy to over-apply. Because they punish any reuse of a token, pushing them too high makes the model avoid words it genuinely needs.

• Cranking them too high. At values near 1.5–2.0 the model starts dodging common, necessary words — articles, names, technical terms it must repeat — and the text turns awkward, drifts off-topic, or grabs odd synonyms. Stay in the 0.1–1.0 range unless you've measured otherwise.
• Penalising required repetition. Code, structured output (JSON), tables, and anything with a fixed vocabulary need repeated tokens like {, field names, or a recurring keyword. A high frequency penalty can quietly corrupt the format. Keep penalties at 0 for structured generation.
• Using them to fix the wrong thing. Penalties stop repetition; they do nothing for accuracy. If a model is making facts up, that's a different problem — see why LLMs hallucinate. No penalty setting will ground an answer in real sources.
• Confusing the two. Turning up presence penalty to stop a tight loop barely helps, because a flat penalty doesn't grow with the count. Loops want frequency penalty.
• Stacking both at high values. Each penalty already discourages reuse; running both near the top of the range compounds the distortion. If you use both, keep each modest.

// Going deeper

A few nuances matter once the basics click.

Tokens, not words. The count both penalties track is over tokens from the model's vocabulary, which are often word-pieces rather than whole words. A long or rare word may be several tokens; a common word is usually one. This is why the effect can feel uneven — a penalty bites a one-token word more cleanly than a word split into three pieces, and it can accidentally suppress a shared sub-word (like a common suffix) across otherwise different words.

The running window. Most implementations count occurrences over the tokens generated so far in this response (and often the prompt too). The exact window — whole context vs a recent slice — varies by provider and affects long generations: with a full-context count, a token used early keeps its penalty for the entire output, which can over-suppress legitimately recurring terms in very long text.

Subtractive vs multiplicative. The OpenAI-style presence_penalty / frequency_penalty subtract from logits, as shown above. A separate convention, common in open-source inference stacks, is repetition_penalty, which divides (or multiplies) the logit of seen tokens by a factor — so its neutral value is 1.0, not 0, and it doesn't distinguish presence from frequency. Don't mix the mental models; check which parameter your API exposes and what its neutral value is.

When penalties aren't enough. Some repetition comes from the prompt or the task, not the sampler. If a model keeps restating the question, fix the instruction. If it loops because the request is impossible or contradictory, no penalty will save it. Other tools in the same family — adjusting temperature, tightening top-p / top-k, or simply choosing a stronger model — often address the root cause better than maxing out a penalty. The honest summary: frequency and presence penalties are precise, cheap fixes for surface repetition, best applied lightly and only when you see the symptom they're built for.