# Perplexity and Intelligence — OpenAI publishes its internal pretraining loss

> OpenAI's chief scientist shares how far its models have pushed pretraining loss and asks labs to cap it together.

- Org: OpenAI
- Date: 2026-11-22
- Category: article
- Source: https://openai.com/index/perplexity-and-intelligence
- Canonical page: https://ai-tldr.dev/releases/openai-perplexity-and-intelligence/

## In short

OpenAI Chief Scientist Jakub Pachocki publishes the lab's internal log-perplexity numbers from GPT-3.5 on, and proposes shared industry limits on pretraining based on perplexity until AI monitoring improves.

## What it is

In 'Perplexity and Intelligence', OpenAI Chief Scientist Jakub Pachocki publishes the lab's internal log-perplexity measurements for its models since GPT-3.5, shown as a plot of losses over time. He proposes perplexity, the same thing as cross-entropy loss, as a common yardstick for pretraining progress across AI developers.

## How it works

Log-perplexity measures how well a model predicts held-out text; lower means better compression. OpenAI measures it on held-out data from its internal codebase. In the post, GPT-3.5 sits near 0.99 and GPT-4 near 0.91, values below 0.8 line up with large jumps in cybersecurity and math ability, and a model near 0.76 'started producing groundbreaking results' in math. Pachocki expects the next jump around 0.68 to 0.7.

## Why it matters

Frontier labs rarely publish raw pretraining loss, and Pachocki says doing so gives competitors 'a clearer picture of where we stand.' He argues that automated AI research could bring a sudden wave of pretraining advances 'as soon as 2027', and calls on labs to agree on shared evaluation norms, red-team safety cases and 'common limits' not to cross until monitoring tools reach a set milestone.

## Who it's for

AI researchers, safety and policy people, and anyone tracking frontier progress

## Quick facts

- Author: Jakub Pachocki, OpenAI Chief Scientist
- Published: Nov 22, 2026 on openai.com
- Metric: Log-perplexity (cross-entropy loss) on held-out data
- Eval data: Held-out sets from OpenAI's internal codebase
- GPT-3.5 / GPT-4: About 0.99 / about 0.91
- Next window: About 0.68–0.7, per the author
- Released: Numbers and plot only; no dataset or code

## FAQ

### Can other labs compare their models with OpenAI's perplexity numbers?

Not directly. Jakub Pachocki's post says OpenAI measures log-perplexity on held-out data from its own internal codebase, and that labs use different datasets and training mixes, which makes comparison hard. OpenAI did not release the dataset. Pachocki says that if the industry sets transparency as a goal, transferable ways to compare base models can be found.

### What limits does OpenAI's chief scientist propose for AI labs?

In 'Perplexity and Intelligence', Jakub Pachocki proposes that frontier developers agree on shared norms for evaluating models, red-team safety cases and transparency standards. He also proposes common limits based on perplexity and other measurements, to hold until labs can show that their ability to understand and monitor AI behavior reaches a concrete milestone, such as a successor to chain-of-thought monitors.

### What are the limits of perplexity as a measure of AI ability?

OpenAI's post lists several. Perplexity does not map directly to specific tasks, it ignores gains from chain-of-thought and other post-training methods, and the entropy floor of the data is unknown, so a loss of zero is not possible. Pachocki adds that progress will look slower near that floor, while smaller perplexity gains will mean larger jumps in intelligence.

### Why does OpenAI think AI progress could speed up in 2027?

Jakub Pachocki points to math: once OpenAI used a model at log-perplexity around 0.76, solutions to long-standing math questions arrived one after another. He argues automated AI research on pretraining could cause a similar sudden wave of advances as soon as 2027, and that AI-assisted alignment and monitoring probably will not keep pace with it.

## Links

- [OpenAI Post](https://openai.com/index/perplexity-and-intelligence)
- [HN Discussion](https://news.ycombinator.com/item?id=50601525)
- [Jagged Frontier Report](https://openai.com/index/research-acceleration-jagged-frontier/)
- [Amodei: Pace Frontier](https://darioamodei.com/post/we-must-pace-the-frontier)
