# BentoML

> Turn any AI/ML model into a production inference API and Docker container

- Category: Models & Inference / Serving & Deployment
- Language: Python
- Repository: https://github.com/bentoml/BentoML
- Website: https://bentoml.com
- Canonical page: https://ai-tldr.dev/tools/bentoml/

## In short

BentoML is an open-source Python library for building online serving systems for AI apps and model inference. You wrap your inference code in a Python class, add type hints, and BentoML turns it into a REST API server. It sits in the Serving & Deployment category of the Models & Inference tools and is open-source, written in Python.

## Overview

BentoML is an open-source Python library for building online serving systems for AI apps and model inference. You wrap your inference code in a Python class, add type hints, and BentoML turns it into a REST API server. It works with any ML framework, modality, and inference runtime, from LLMs and diffusion models to embeddings and computer vision.

It is aimed at ML engineers and Python developers who need to move a model from a notebook or script into a service that other systems can call. A single config file declares the Python version, dependencies, and model versions, and BentoML generates a Docker image from it so the same setup runs locally and in production.

Within the model serving and deployment space, BentoML focuses on packaging and serving. It includes built-in serving features such as dynamic batching, model parallelism, and multi-model inference graphs, and can deploy to Docker or to the managed BentoCloud service.

## What it does

- Turn any model inference script into a REST API server with a few lines of code and standard Python type hints
- Generate reproducible Docker images from a single config file that manages environments, dependencies, and model versions
- Built-in serving optimizations: dynamic (adaptive) batching, model parallelism, multi-stage pipelines, and multi-model inference graphs
- Framework-agnostic: supports any ML framework, modality, and inference runtime
- Package code, models, and dependency configs into a Bento, the standardized deployable artifact
- Develop and debug locally, then deploy with Docker containers or to the managed BentoCloud service

## Getting started

Install BentoML, define a service class in service.py, and serve it locally before containerizing for deployment.

1. Install BentoML

   Requires Python 3.9 or newer.

```bash
# Requires Python≥3.9
pip install -U bentoml
```

2. Define an API in service.py

   Wrap your model in a class decorated with @bentoml.service and expose methods with @bentoml.api.

```python
import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self) -> None:
        import torch
        from transformers import pipeline

        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) -> list[str]:
        results = self.pipeline(texts)
        return [item['summary_text'] for item in results]
```

3. Run the service locally

   Install your model's dependencies, then serve the API at http://localhost:3000.

```bash
pip install torch transformers  # additional dependencies for local run
bentoml serve
```

4. Build and containerize for deployment

   Package code, models, and configs into a Bento, then generate and run a Docker image.

```bash
bentoml build
bentoml containerize summarization:latest
docker run --rm -p 3000:3000 summarization:latest
```

## When to use it

- Expose a trained model as a REST inference API that other services or applications can call
- Package a model and its dependencies into a reproducible Docker image for consistent deployment across environments
- Serve LLMs, diffusion models, embeddings, or computer vision models with built-in batching and GPU utilization
- Compose multiple models into a single inference graph or pipeline behind one API