# OpenAI Deployment Simulation — predict misbehavior before release

> OpenAI estimates how a new model will behave in production by replaying real past conversations through it before release.

- Org: OpenAI
- Date: 2026-06-16
- Category: paper, algorithm
- Source: https://cdn.openai.com/pdf/predicting-llm-safety-before-release-by-simulating-deployment.pdf
- Canonical page: https://ai-tldr.dev/releases/openai-deployment-simulation/

## In short

OpenAI Deployment Simulation replays real past user conversations through a candidate model before launch to forecast misbehavior rates. Tested across 1.3M conversations from GPT-5 Thinking to GPT-5.4 with a median 1.5x multiplicative error.

## What it is

Deployment Simulation is a pre-release safety evaluation method from OpenAI. Deployment Simulation takes recent production conversations, strips the old assistant reply, and lets the candidate model answer in the same real context, then grades the new answers for undesired behavior.

## How it works

OpenAI applied Deployment Simulation to 1.3M de-identified conversations spanning GPT-5 Thinking through GPT-5.4 collected from August 2025 to March 2026. The team pre-registered predictions for 20 misbehavior types on GPT-5.4 Thinking and compared simulated rates to post-launch production rates, reporting a median multiplicative error of 1.5x. For agentic coding, simulated tool calls lifted discriminator accuracy from 11.6% to 49.5%.

## Why it matters

Traditional evals like SWE-Bench look obviously like tests — GPT-5.2 labels them as evaluations 100% of the time, versus 5.1% for Deployment Simulation traffic — so models can game them. Replaying real conversations reduces that evaluation-awareness gap and lets safety teams catch behavior drift, reward hacking, or new misuse patterns before the model reaches users.

## Who it's for

AI safety teams, evaluation researchers, frontier-model auditors

## Try it

Read the paper: https://cdn.openai.com/pdf/predicting-llm-safety-before-release-by-simulating-deployment.pdf

## Links

- [Paper (PDF)](https://cdn.openai.com/pdf/predicting-llm-safety-before-release-by-simulating-deployment.pdf)
- [Public Evals Validation](https://alignment.openai.com/validating-public-evals)
- [Analysis](https://www.marktechpost.com/2026/06/16/openai-deployment-simulation/)