# OpenLLM

> Run any open-source LLM as an OpenAI-compatible API with one command

- Category: Models & Inference / Serving & Deployment
- Language: Python
- Repository: https://github.com/bentoml/OpenLLM
- Canonical page: https://ai-tldr.dev/tools/openllm/

## In short

OpenLLM is a command-line tool from BentoML for self-hosting large language models. It lets you run open-source models such as Llama 3.3, Qwen2.5, Gemma, Mistral, and Phi4, or your own custom models, and exposes them through OpenAI-compatible API endpoints. It sits in the Serving & Deployment category of the Models & Inference tools and is open-source, written in Python.

## Overview

OpenLLM is a command-line tool from BentoML for self-hosting large language models. It lets you run open-source models such as Llama 3.3, Qwen2.5, Gemma, Mistral, and Phi4, or your own custom models, and exposes them through OpenAI-compatible API endpoints. Because the API matches OpenAI's, you can point existing clients and frameworks at your own server with little code change.

It is aimed at developers and teams who want to keep models and data on their own infrastructure instead of calling a hosted API. You start a server with one command, and OpenLLM handles the inference backend and serving for you. It also ships a built-in chat UI and a CLI chat mode for quick testing.

Within the model-serving and deployment space, OpenLLM focuses on turning a model into a running, callable endpoint. The same workflow extends from local runs to cloud deployment with Docker, Kubernetes, and BentoCloud.

## What it does

- Run open-source LLMs (Llama 3.3, Qwen2.5, Gemma3, Mistral, Phi4, and more) or custom models with a single command
- Exposes OpenAI-compatible API endpoints, so OpenAI clients and frameworks like LlamaIndex work against your own server
- Built-in chat UI served at the /chat endpoint of the running server
- Interactive CLI chat with the openllm run command
- Model repository commands to list, inspect, and update available models
- Path to cloud deployment with Docker, Kubernetes, and BentoCloud

## Getting started

Install OpenLLM with pip, then start a model server and call it like the OpenAI API.

1. Install OpenLLM

   Install the package with pip and run the interactive hello command to explore it.

```bash
pip install openllm  # or pip3 install openllm
openllm hello
```

2. Start a model server

   Serve a model by name and version. The server runs at http://localhost:3000 with OpenAI-compatible APIs. Gated models need a Hugging Face token set as HF_TOKEN.

```bash
openllm serve llama3.2:1b
```

3. Call it with the OpenAI client

   Point the OpenAI Python client at the local server's /v1 base URL. The API key is optional.

```python
from openai import OpenAI

client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')

chat_completion = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Explain superconductors like I'm five years old"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    print(chunk.choices[0].delta.content or "", end="")
```

4. Chat from the CLI

   Start an interactive chat in the terminal with a chosen model version.

```bash
openllm run llama3:8b
```

## When to use it

- Self-host an open-source LLM on your own hardware to keep data in-house instead of using a hosted API
- Swap a paid OpenAI endpoint for a local one by changing the base URL in existing OpenAI-client code
- Test and compare open models quickly using the built-in chat UI or the CLI run command
- Take a model from a local run to a cloud deployment with Docker, Kubernetes, or BentoCloud