# promptfoo

> Test, compare, and red-team your LLM prompts and apps from the command line

- Category: Evaluation & Production Ops / Evaluation & Red-Teaming
- Language: TypeScript
- Licence: MIT
- Repository: https://github.com/promptfoo/promptfoo
- Website: https://www.promptfoo.dev
- Canonical page: https://ai-tldr.dev/tools/promptfoo/

## In short

promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. Instead of tweaking prompts by trial and error, you write test cases and run automated evals to see, with metrics, how different prompts and models behave on the same inputs. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in TypeScript and licensed MIT.

## Overview

promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. Instead of tweaking prompts by trial and error, you write test cases and run automated evals to see, with metrics, how different prompts and models behave on the same inputs.

It is built for developers who ship LLM features and want repeatable checks rather than gut-feel comparisons. Evals run fully on your machine, so your prompts and data never leave your environment, and the tool works with any LLM API and any programming language.

As an eval framework, it covers two jobs in one: side-by-side model and prompt comparison, plus red-teaming probes that scan for issues like prompt injection and PII leaks. It plugs into CI/CD so the same checks run on every change.

## What it does

- Automated evaluations of prompts and models with metrics you can compare
- Side-by-side comparison across providers including OpenAI, Anthropic, Azure, Bedrock, and Ollama
- Red teaming and vulnerability scanning for prompt injection, PII leaks, and other risks
- Runs evals 100% locally, so prompts stay on your machine
- CI/CD integration to automate checks on every change
- Available as a CLI and a Node.js library, with live reload and caching

## Getting started

Install promptfoo, scaffold an example project, set a provider key, then run an eval and view the results. Requires Node.js ^20.20.0 or >=22.22.0 for npm/npx usage.

1. Install and scaffold an example

   Install the CLI globally and initialize the getting-started example. You can also use brew install promptfoo, pip install promptfoo, or run any command with npx promptfoo@latest without installing.

```bash
npm install -g promptfoo
promptfoo init --example getting-started
```

2. Set your provider API key

   Most LLM providers require an API key. Set yours as an environment variable before running an eval.

```bash
export OPENAI_API_KEY=sk-abc123
```

3. Run an eval and view results

   Move into the example directory, run the evaluation, then open the local viewer to inspect the results.

```bash
cd getting-started
promptfoo eval
promptfoo view
```

## When to use it

- Compare several prompts or models side-by-side on the same test cases before choosing one
- Red-team an LLM app to scan for prompt injection, PII leaks, and other vulnerabilities
- Add automated prompt and model checks to a CI/CD pipeline so regressions fail the build
- Catch quality drops when upgrading or swapping the underlying model behind a feature