# LitGPT

> Pretrain, finetune, and deploy 20+ LLMs from scratch with no abstractions

- Category: Training & Fine-Tuning / Fine-Tuning Frameworks
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/Lightning-AI/litgpt
- Website: https://lightning.ai
- Canonical page: https://ai-tldr.dev/tools/litgpt/

## In short

LitGPT is an open-source library from Lightning AI for working with large language models. It sits in the Fine-Tuning Frameworks category of the Training & Fine-Tuning tools and is open-source, written in Python and licensed Apache-2.0. To get started: install LitGPT.

## Overview

LitGPT is an open-source library from Lightning AI for working with large language models. It ships clean, from-scratch implementations of 20+ models such as Llama 3, Phi, Qwen2.5, Gemma, Mistral, and Mixtral, with no hidden abstraction layers so you can read and debug the code directly.

With a single command line tool you can load a model, finetune it on your own data, pretrain from scratch, evaluate it, chat with it, or serve it as an API. It supports memory-saving tricks like LoRA, QLoRA, quantization (fp4/8/16/32), Flash Attention, and FSDP, so the same recipes scale from one GPU up to 1000+ GPUs or TPUs. LitGPT is released under the Apache 2.0 license.

## What it does

- From-scratch, single-file implementations of 20+ LLMs (Llama 3, Phi, Qwen2.5, Gemma, Mistral, Mixtral, and more) with no abstraction layers
- One CLI for the full model lifecycle: finetune, pretrain, evaluate, chat, and serve
- Memory-efficient training with LoRA, QLoRA, adapters, and fp4/8/16/32 quantization for low-memory GPUs
- Scales from a single GPU to 1000+ GPUs or TPUs using Flash Attention and FSDP
- Tested, ready-to-use training and finetuning recipes configurable through YAML files
- Simple Python API to load any supported model and generate text in a few lines

## Getting started

LitGPT installs from PyPI and gives you both a Python API and a command line tool. The steps below come straight from the project README.

1. Install LitGPT

   Install the package with its extra dependencies using pip.

```bash
pip install 'litgpt[extra]'
```

2. Load a model in Python

   Use the Python API to load any supported model and generate text. Weights download automatically.

```python
from litgpt import LLM

llm = LLM.load("microsoft/phi-2")
text = llm.generate("Fix the spelling: Every fall, the family goes to the mountains.")
print(text)
```

3. Finetune on your own data

   Point the finetune command at a JSON dataset and an output directory. Weights are downloaded automatically.

```bash
litgpt finetune microsoft/phi-2 \
  --data JSON \
  --data.json_path my_custom_dataset.json \
  --data.val_split_fraction 0.1 \
  --out_dir out/custom-model
```

4. Serve the model as an API

   Start a web server for a built-in model or your own trained checkpoint so apps can query it.

```bash
litgpt serve out/custom-model/final
```

## When to use it

- Finetuning an open LLM on your own domain data to build a specialized assistant
- Pretraining a small language model from scratch on a custom text corpus
- Deploying a finetuned model as a local inference API for your application
- Evaluating model quality on benchmarks like MMLU and TruthfulQA before shipping