Which GGUF File Should You Download? Reading a Hugging Face Quant Repo

// In plain English

You found a model you want to run on your own computer. You open its GGUF repository on Hugging Face and instead of one download button you get a wall of files: model-Q4_K_M.gguf, model-Q5_K_S.gguf, model-Q8_0.gguf, model-IQ3_XS.gguf, and maybe a few named ...00001-of-00003.gguf. Every one is the same model. They differ only in how aggressively the weights were squeezed down — the quant level.

Think of it like buying the same movie in different qualities: 4K, 1080p, 720p, and a tiny phone version. The 4K file is gorgeous but huge and your old laptop chokes on it. The phone version plays anywhere but looks soft. You are not picking a different movie — you are picking the largest quality that still fits and plays smoothly on the screen you actually own. A GGUF repo is exactly that shelf, and your "screen" is your memory budget.

So the whole task reduces to one question: how much memory do I have, and what is the biggest, best-quality file that fits inside it with a little room to spare? Once you can read the file names, the answer is usually obvious in about ten seconds.

// Why it matters

Picking the wrong file is the single most common way a first local-LLM attempt goes wrong, and it fails in two opposite directions.

• Too big → it crashes or crawls. If the file plus its working memory don't fit in your VRAM (GPU memory) or RAM, the model either refuses to load with an out-of-memory error, or it spills onto disk/CPU and generates at a painful one or two words per second. People conclude "local models are useless" when they simply grabbed the 4K file for a phone.
• Too small → you throw away quality for no reason. If you have 24 GB of VRAM and download a tiny 3-bit quant "to be safe," you get a noticeably dumber model — more mistakes, worse instruction-following — while gigabytes of your hardware sit idle. The waste is invisible, so people never realize they left quality on the table.
• Wrong repo entirely. Grabbing a base model when you wanted a chat model, or a sketchy re-upload with a broken chat template, produces a model that rambles or ignores your instructions even though the quant was fine.

Getting this right matters because the file you download decides three things at once: whether the model loads at all, how fast it answers, and how smart it feels. There is no "reload to fix it" — you have to redownload several gigabytes. Learning to read the repo once saves you from that loop forever, and it's the gateway skill for every local tool: Ollama, LM Studio, and raw llama.cpp all consume these same files.

// How to read a GGUF file name

A GGUF file name encodes everything you need in a short code at the end. Once you can decode it, the shelf stops being scary. Take Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf apart:

So Q4_K_M reads as: 4-bit, K-quant, medium tier. That combination is the community default — the best balance of size and quality for most people, most of the time. When in doubt, this is the file to grab.

The decision itself is a short funnel. You start from the memory you have, subtract a little overhead, and that tells you the biggest file you can afford. Then you pick the highest-quality quant whose file size lands under that number.

The key rule for fit: the file should be comfortably smaller than your memory, not equal to it. Running a model needs extra room beyond the weights — for the context (your conversation) and the working buffers (the KV cache). A good habit is to leave roughly 1–2 GB of headroom on top of the file size. A 7 GB file will not run happily on a GPU with exactly 8 GB of VRAM.

// Map your memory to a quant

Here is the part everyone actually wants: a lookup table. "Memory" means your GPU VRAM if you run on a graphics card, or your system RAM if you run on CPU or an Apple Silicon Mac (where RAM is shared with the GPU). Find your budget, find a model size that fits, and read off the quant. Sizes below are approximate for a typical _K_M quant and shift a little per model.

How to read this in practice. Suppose you have an 8 GB graphics card. You scan the repo, you know you want an Instruct model, and you reach for the largest file that stays under ~6–6.5 GB (leaving headroom). For an 8B model that is almost always Q4_K_M. You download exactly one file and you are done.

Now the ranking within your budget — which quant beats which. From best quality to smallest, the practical ladder looks like this. Climb as high as your memory allows; the gains above Q5 are small, and the losses below Q3 are real.

// Split GGUF files (the ...00001-of-00003 parts)

For big models you will see names like model-Q4_K_M-00001-of-00003.gguf, ...-00002-of-00003.gguf, ...-00003-of-00003.gguf. These are not different choices and they are not alternatives. They are one file that was cut into pieces because Hugging Face caps single files at 50 GB. A 70B model can exceed that, so the uploader splits it.

The rule is simple: download every part of the set, and put them all in the same folder. Don't mix parts from different quants. If you choose Q4_K_M, grab all the Q4_K_M-0000X-of-0000N files and nothing else.

• llama.cpp, Ollama, LM Studio (recent versions): just point the loader at the first part (...00001-of-0000N.gguf). The loader detects the rest in the same folder and stitches them automatically. You do not need to merge anything by hand.
• If a tool insists on a single file: llama.cpp ships a small utility, llama-gguf-split, that merges the parts back into one file. But for everyday use you rarely need it — modern loaders read the parts directly.
• Pulling from Ollama's own library? You never see split parts at all; ollama pull handles the whole download and assembly for you. Splits only show up when you download raw files from a Hugging Face repo.

// Pick the right repo and uploader

Two repos can offer the "same" model and give very different results. Before you even look at quant names, make sure you are in the right repository.

Instruct vs base — almost always pick Instruct

A base (or pretrained) model only continues text; ask it a question and it may just write more questions. An Instruct (or Chat / -it) model was fine-tuned to follow instructions and hold a conversation. For anything interactive, you want the Instruct variant. Base models are for fine-tuning and research. If the model rambles and ignores you, the most likely cause is that you grabbed the base weights by accident.

Official repo vs community re-quantizer

Many model makers don't publish GGUFs themselves — they ship the full-precision weights, and the community converts them. So you'll often download from a trusted re-quantizer rather than the original lab. That's normal and expected. The well-known, reputable GGUF uploaders the community relies on include bartowski, TheBloke (older but historically the standard), and the unsloth and ggml-org organizations. A repo from one of these, with a clear model card and a full ladder of quants, is a safe bet.

// Going deeper

Once the basic pick is second nature, a few finer points let you squeeze out the last bit of quality or fit.

IQ quants and imatrix. Files starting with IQ (like IQ3_XS, IQ2_M) use a newer importance-matrix method that preserves quality better at very low bit counts than the older Q3/Q2. The trade-off: they can be a little slower on some hardware, especially pure CPU. Rule of thumb — at 4 bits and above, plain Q4_K_M/Q5_K_M is great; below 4 bits, an IQ quant usually beats the equivalent Q quant. Many _K quants today are also built with an imatrix even when the name doesn't say IQ; the model card will mention it.

K-quants vs legacy quants. You may still see old names like Q4_0 or Q4_1 without the _K. These are the legacy scheme and are strictly worse than the _K versions at the same size. Unless a tool specifically requires them, always prefer the _K variant.

Splitting layers across GPU and CPU. You are not limited to files that fit entirely in VRAM. With GPU offloading, llama.cpp can put as many layers as fit on the GPU and run the rest on CPU. That lets a 24 GB card run, say, a 70B model slowly by keeping part of it in system RAM. So a file a bit larger than your VRAM is sometimes still usable — just slower. This is also why Macs with lots of unified memory punch above their weight; see running LLMs on a Mac.

Context costs memory too. The numbers in the table above assume a modest context. If you load a very long context window, the KV cache grows and eats VRAM on top of the weights, which can push a file that "fit" into out-of-memory territory. If you plan to use long contexts, drop one quant tier or one model size to leave room.

Where to go next. GGUF is one of several quant formats; if you run on a high-end GPU you might compare it with GPTQ and AWQ in GGUF vs GPTQ vs AWQ. And once you have your file, the practical next step is loading it — start with how to run Ollama or reading a model card to confirm the right prompt format. The durable habit: decode the name, match it to your memory with headroom, confirm it's the Instruct build from a trusted uploader, and grab every part if it's split.