# mistral-inference

> Official reference code for running Mistral open-weight models

- Category: Models & Inference / Local Runtimes
- Language: Jupyter Notebook
- Licence: Apache-2.0
- Repository: https://github.com/mistralai/mistral-inference
- Website: https://mistral.ai/
- Canonical page: https://ai-tldr.dev/tools/mistral-inference/

## In short

mistral-inference is the official, minimal codebase from Mistral AI for running their open-weight models. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Jupyter Notebook and licensed Apache-2.0. To get started: install from PyPI.

## Overview

mistral-inference is the official, minimal codebase from Mistral AI for running their open-weight models. It covers the well-known family of releases, including Mistral 7B, the Mixtral 8x7B and 8x22B mixture-of-experts models, Codestral, Mathstral, Nemo, Mistral Large 2, and the multimodal Pixtral and Mistral Small 3.1 models.

Because it is the reference implementation, the code stays small and close to how Mistral itself runs the models. You download the model weights, then either test and chat from the command line or call the generation API directly from Python. Smaller models run on a single GPU, while the larger Mixtral models use torchrun across several GPUs.

## What it does

- Official reference code maintained by Mistral AI for running their open-weight models
- Covers the full lineup: Mistral 7B, Mixtral 8x7B/8x22B, Codestral, Mathstral, Nemo, Mistral Large 2, Pixtral, and Mistral Small 3.1
- mistral-demo command to quickly check a model works in your setup
- mistral-chat command for interactive chat, with flags like --instruct, --max_tokens, and --temperature
- Python API (Transformer + generate) for instruction following and multimodal image-and-text prompts
- Multi-GPU runs for large mixture-of-experts models via torchrun --nproc-per-node

## Getting started

Install the package, download a model, then run it from the CLI or Python. Note that installation needs a GPU because it relies on xformers.

1. Install from PyPI

   Install the package with pip. A GPU is required for installation.

```bash
pip install mistral-inference
```

2. Download a model

   Create a folder for your models, download a model archive, and extract it. Here we download the 12B Mistral Nemo instruct model.

```bash
export MISTRAL_MODEL=$HOME/mistral_models
mkdir -p $MISTRAL_MODEL
export 12B_DIR=$MISTRAL_MODEL/12B_Nemo
wget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar
mkdir -p $12B_DIR
tar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR
```

3. Test and chat from the CLI

   Run mistral-demo to confirm the model loads, then start an interactive chat with mistral-chat.

```bash
mistral-demo $12B_DIR
mistral-chat $12B_DIR --instruct --max_tokens 1024 --temperature 0.35
```

4. Generate from Python

   Load the tokenizer and model, build a chat request, then call generate to get a completion.

```python
from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest

tokenizer = MistralTokenizer.from_file("./mistral-nemo-instruct-v0.1/tekken.json")
model = Transformer.from_folder("./mistral-nemo-instruct-v0.1")

prompt = "How expensive would it be to clean all windows in Paris? Make a reasonable guess in US Dollar."
completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
tokens = tokenizer.encode_chat_completion(completion_request).tokens

out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
print(result)
```

## When to use it

- Run Mistral open-weight models locally on your own GPU instead of calling a hosted API
- Use Codestral or Codestral-Mamba as a local coding assistant via mistral-chat
- Send image-and-text prompts to multimodal models like Pixtral and Mistral Small 3.1
- Serve large Mixtral 8x7B and 8x22B mixture-of-experts models across multiple GPUs with torchrun