# KTransformers

> Run large Mixture-of-Experts models locally by splitting work across CPU and GPU

- Category: Models & Inference / Local Runtimes
- Language: Python
- Repository: https://github.com/kvcache-ai/ktransformers
- Website: https://kvcache-ai.github.io/ktransformers/
- Canonical page: https://ai-tldr.dev/tools/ktransformers/

## In short

KTransformers is a framework for efficient inference and fine-tuning of large language models using CPU-GPU heterogeneous computing. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python. To get started: get the repository.

## Overview

KTransformers is a framework for efficient inference and fine-tuning of large language models using CPU-GPU heterogeneous computing. Instead of requiring all model weights to sit in GPU memory, it splits the work so that some experts and layers run on the CPU while the GPU handles the rest. This lets you run very large Mixture-of-Experts (MoE) models on machines with limited VRAM.

It is aimed at developers and researchers who want to run or fine-tune large MoE models, such as DeepSeek-V3/R1 and Kimi-K2, on a single workstation or server rather than a large GPU cluster. The project exposes two main capabilities from its kt-kernel source tree: high-performance inference and supervised fine-tuning (SFT) through an integration with LLaMA-Factory.

As a local runtime, it focuses on the CPU side of inference: AMX and AVX-optimized kernels, NUMA-aware memory management, and INT4/INT8 quantization. It also provides a Python API so it can plug into serving frameworks like SGLang.

## What it does

- CPU-GPU heterogeneous inference that places hot experts on the GPU and cold experts on the CPU to fit large MoE models in limited VRAM
- AMX/AVX acceleration: Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
- MoE optimization with NUMA-aware memory management for multi-socket CPU systems
- Quantization support: CPU-side INT4/INT8 weights and GPU-side GPTQ
- Clean Python API for integration with SGLang and other serving frameworks
- Supervised fine-tuning (SFT) of ultra-large MoE models via LLaMA-Factory integration

## Getting started

The inference capability lives in the kt-kernel directory of the repository, which you build and install with pip.

1. Get the repository

   Clone the KTransformers repository so you have the kt-kernel source tree locally.

```bash
git clone https://github.com/kvcache-ai/ktransformers.git
```

2. Build and install kt-kernel

   Enter the kt-kernel directory and install it with pip, which compiles the CPU-optimized kernels.

```bash
cd kt-kernel
pip install .
```

3. Pick your model and backend

   Follow the kt-kernel documentation and the per-model tutorials (for example DeepSeek-R1/V3 or Kimi-K2) to choose quantized weights and a CPU backend (AMX or AVX2) that matches your hardware.

## When to use it

- Running large MoE models like DeepSeek-V3/R1 or Kimi-K2 on a workstation with limited GPU VRAM
- Serving large models in production by integrating kt-kernel with SGLang
- Splitting experts across CPU and GPU (hot experts on GPU, cold experts on CPU) to balance memory and speed
- Fine-tuning ultra-large MoE models on local hardware via the LLaMA-Factory integration