BentoML

// Overview

BentoML is an open-source Python library for building online serving systems for AI apps and model inference. You wrap your inference code in a Python class, add type hints, and BentoML turns it into a REST API server. It works with any ML framework, modality, and inference runtime, from LLMs and diffusion models to embeddings and computer vision.

It is aimed at ML engineers and Python developers who need to move a model from a notebook or script into a service that other systems can call. A single config file declares the Python version, dependencies, and model versions, and BentoML generates a Docker image from it so the same setup runs locally and in production.

Within the model serving and deployment space, BentoML focuses on packaging and serving. It includes built-in serving features such as dynamic batching, model parallelism, and multi-model inference graphs, and can deploy to Docker or to the managed BentoCloud service.

// What it does

• Turn any model inference script into a REST API server with a few lines of code and standard Python type hints
• Generate reproducible Docker images from a single config file that manages environments, dependencies, and model versions
• Built-in serving optimizations: dynamic (adaptive) batching, model parallelism, multi-stage pipelines, and multi-model inference graphs
• Framework-agnostic: supports any ML framework, modality, and inference runtime
• Package code, models, and dependency configs into a Bento, the standardized deployable artifact
• Develop and debug locally, then deploy with Docker containers or to the managed BentoCloud service

// Getting started

Install BentoML, define a service class in service.py, and serve it locally before containerizing for deployment.

# Requires Python≥3.9
pip install -U bentoml

import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self) -> None:
        import torch
        from transformers import pipeline

        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) -> list[str]:
        results = self.pipeline(texts)
        return [item['summary_text'] for item in results]

pip install torch transformers  # additional dependencies for local run
bentoml serve

bentoml build
bentoml containerize summarization:latest
docker run --rm -p 3000:3000 summarization:latest

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Expose a trained model as a REST inference API that other services or applications can call
• Package a model and its dependencies into a reproducible Docker image for consistent deployment across environments
• Serve LLMs, diffusion models, embeddings, or computer vision models with built-in batching and GPU utilization
• Compose multiple models into a single inference graph or pipeline behind one API

// How BentoML compares

BentoML alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.