How to Convert a Hugging Face Model to GGUF and Quantize It

// In plain English

Most open models on Hugging Face ship as safetensors files — raw weights in full or half precision, built to run on a GPU with PyTorch. Tools like Ollama, LM Studio, and llama.cpp don't read that format. They want GGUF: a single self-contained file that bundles the weights, the tokenizer, and the model's metadata, ready to load on a CPU or a modest GPU.

Converting to GGUF is like exporting a document to PDF. The original .docx is great for editing, but you hand someone a PDF when you just want it to open and look right everywhere. GGUF is that portable export: one file, no Python environment, no framework — just point your runtime at it and it runs.

Why bother doing it yourself? Two common reasons. First, you fine-tuned a model and the result lives only as safetensors — no one has made a GGUF of your model. Second, you found a brand-new or obscure model on Hugging Face that nobody has packaged yet. In both cases you convert it once and then run it locally forever.

// Why it matters

If you only ever run popular models, you may never touch conversion — communities like TheBloke and bartowski publish ready-made GGUFs within hours of a big release. But the moment you step off the beaten path, pre-made files stop existing, and converting yourself becomes the only way forward.

• Your fine-tune is unique. After training a LoRA adapter or doing a full fine-tune, the merged weights are one-of-a-kind. No public GGUF can exist — you must make it.
• The model is too new or too niche. A research model, a smaller language variant, or a fresh release may have zero GGUF uploads. Conversion gets you running on day one instead of waiting for someone else.
• You want a specific quantization. Public uploads cover the common sizes, but maybe you need exactly Q5_K_M to fit your VRAM, or a full FP16 file to use as a base for further work. Doing it yourself gives you every option.
• You want to verify what you run. Converting from the original safetensors means you know exactly which weights went in — no trusting an anonymous re-upload.

The payoff is independence. Once you can run the conversion pipeline, any open-weights model on Hugging Face becomes something you can run on your own laptop — quantized to fit, packaged for Ollama, and fully offline.

// How it works

The pipeline has three stages: convert the safetensors into an unquantized GGUF, quantize that GGUF down to a small size, and import the result into your runtime. The first stage just repackages the weights (no quality loss); the second is where you trade a little accuracy for a much smaller file.

Step 1 — Get llama.cpp and the model

The conversion tooling ships inside the llama.cpp repository. Clone it, install its Python requirements, then download the model's full directory from Hugging Face — you need the *.safetensors files and the config and tokenizer files alongside them.

# 1. Get the converter and its Python deps
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt

# 2. Download the whole model repo (weights + config + tokenizer)
pip install -U "huggingface_hub[cli]"
hf download mistralai/Mistral-7B-Instruct-v0.3 \
  --local-dir ./models/mistral-7b

Step 2 — Convert to an FP16 GGUF

Run convert_hf_to_gguf.py, pointing it at the model folder. Always produce a full-precision GGUF first (--outtype f16). This file is large — roughly two bytes per parameter, so about 14 GB for a 7B model — but it's a lossless repackaging you'll quantize in the next step.

python convert_hf_to_gguf.py ./models/mistral-7b \
  --outtype f16 \
  --outfile ./models/mistral-7b-f16.gguf

Step 3 — Quantize to a runnable size

The FP16 GGUF runs, but it's big and slow on most hardware. Quantization shrinks each weight from 16 bits to roughly 4 or 5, cutting the file to a quarter of its size with only a small quality drop. Build llama.cpp once to get the llama-quantize tool, then run it.

# Build the C++ tools once
cmake -B build
cmake --build build --config Release -j

# Quantize: f16 GGUF in, Q4_K_M GGUF out
./build/bin/llama-quantize \
  ./models/mistral-7b-f16.gguf \
  ./models/mistral-7b-Q4_K_M.gguf \
  Q4_K_M

Q4_K_M is the usual default — the best balance of size, speed, and quality for most people. Q5_K_M keeps a bit more accuracy for a slightly larger file. For the full menu of tradeoffs, see Q4 vs Q8 vs FP16.

// Choosing a quantization level

The quantize step takes a type name like Q4_K_M. The number is the average bits per weight; the _K means a modern k-quant (smarter, better quality than the old methods); the trailing letter is the size tier within that level (S = small, M = medium, L = large). Here is how the common choices compare for a 7B model.

Rule of thumb: start at Q4_K_M. If the model feels noticeably worse than expected and you have spare memory, move up to Q5_K_M or Q6_K. Only drop to Q3 if you genuinely can't fit anything larger. Check your budget against local LLM hardware requirements — the quantized file size is roughly the RAM you'll need, plus a little for context.

// Importing your GGUF into Ollama

A .gguf file already works directly in LM Studio (just drop it in the models folder) and in llama.cpp (llama-cli -m file.gguf). To use it in Ollama, you wrap it in a tiny text file called a Modelfile that tells Ollama where the weights are and how to format prompts.

FROM ./models/mistral-7b-Q4_K_M.gguf

# The chat template this model expects
TEMPLATE """[INST] {{ .Prompt }} [/INST]"""

# Optional defaults
PARAMETER temperature 0.7
PARAMETER stop "[INST]"
PARAMETER stop "[/INST]"

Then register it with one command and run it like any other model:

ollama create my-mistral -f Modelfile
ollama run my-mistral "Explain GGUF in one sentence."

// Common conversion errors

Most failures happen in Step 2 (the convert script) and trace back to one of a handful of causes. Here's how to read the usual error messages.

If the model is a LoRA fine-tune, you usually merge the adapter into the base weights first (with PEFT, producing a normal safetensors model), then convert that merged model. The converter expects standard weights, not a separate adapter file.

// Going deeper

The three-step pipeline above covers the vast majority of conversions. A few refinements matter once you're doing this seriously.

Importance-matrix (imatrix) quantization. The plain llama-quantize treats every weight the same. For the smallest quants (Q2, Q3, and the IQ types), you can first run the model over a small calibration text to build an importance matrix, then pass it with --imatrix. This preserves the weights that matter most and noticeably improves quality at very low bit rates — worth the extra step only when you're squeezing into tiny sizes.

Keep the FP16 GGUF. Quantizing always goes from the FP16 file, never from one quant to another. If you keep the FP16 export around, you can produce any quant later without re-running the convert script. Delete it only when you're sure you won't want a different size.

Vision and multimodal models. A multimodal model has a separate vision projector that needs its own conversion (often a mmproj GGUF) on top of the language weights. The basic convert_hf_to_gguf.py handles the text part; check the llama.cpp docs for the model family before assuming a single file is enough.

Verify before you trust it. After converting, run a few prompts and compare against the original model's behavior. A successful conversion that produces subtly wrong output usually means the chat template, special tokens, or a tokenizer detail didn't carry over. Conversion is mechanical, but verification is on you.

From here, the natural next steps are understanding the format you just produced (what is GGUF), the math behind the size savings (what is quantization), and how to pick a runtime (LM Studio vs Ollama). You can even take your finished GGUF to a phone — see run LLMs on a phone.