# ggml

> Low-level C tensor library for machine learning on local hardware

- Category: Models & Inference / Local Runtimes
- Language: C
- Repository: https://github.com/ggml-org/ggml
- Canonical page: https://ai-tldr.dev/tools/ggml/

## In short

ggml is a tensor library for machine learning written in C. It provides the low-level building blocks (tensors, operations, automatic differentiation, and optimizers) needed to run and train models without pulling in third-party dependencies. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C.

## Overview

ggml is a tensor library for machine learning written in C. It provides the low-level building blocks (tensors, operations, automatic differentiation, and optimizers) needed to run and train models without pulling in third-party dependencies. It is the engine behind well-known local projects such as llama.cpp and whisper.cpp.

It targets developers who want to run models on their own machines across a range of platforms. Integer quantization and broad hardware support let you fit larger models into limited memory, while the design avoids memory allocations during runtime to keep inference predictable.

Within the local-runtimes category, ggml sits at the bottom layer. Rather than a ready-made server, it is the foundation that higher-level runtimes build on, so you reach for it when you need direct control over tensor computation or want to add model support yourself.

## What it does

- Low-level, cross-platform implementation in plain C
- Integer quantization support to shrink model memory use
- Broad hardware support across CPUs and GPU backends
- Automatic differentiation with ADAM and L-BFGS optimizers
- No third-party dependencies
- Zero memory allocations during runtime

## Getting started

Clone the repository, set up the Python tooling, then build the example programs with CMake.

1. Clone and set up dependencies

   Clone the repo and install the Python requirements inside a virtual environment.

```bash
git clone https://github.com/ggml-org/ggml
cd ggml

# install python dependencies in a virtual environment
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

2. Build the examples

   Configure and compile the bundled example programs in release mode.

```bash
mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8
```

3. Run GPT-2 inference

   Download the GPT-2 small 117M model and run a prompt through the backend example.

```bash
# run the GPT-2 small 117M model
../examples/gpt-2/download-ggml-model.sh 117M
./bin/gpt-2-backend -m models/gpt-2-117M/ggml-model.bin -p "This is an example"
```

## When to use it

- Running language or speech models locally on CPU or GPU without cloud services
- Building a custom runtime or adding support for a new model architecture
- Shrinking models with integer quantization to fit limited memory
- Experimenting with on-device training using the built-in autodiff and optimizers