Local & Open Models · TRACK 02/04
Quantization & Model Formats
GGUF, 4-bit, GPTQ, AWQ — making big models fit small machines.
// THE TRACK
01 · START HEREQuantizationYou will understand what quantization does to a model's weights and why it is the trick that makes local AI possible.BEGINNERGGUF FilesYou will know exactly what a GGUF file contains and why it became the universal currency of local models.BEGINNERGGUF vs GPTQ vs AWQYou will understand the real technical differences between the major quantization methods and which one matches your GPU and serving stack.INTERMEDIATEQ4 vs Q8 vs FP16You will stop guessing which quant file to download and pick the right size-quality trade-off for your hardware every time.INTERMEDIATEImportance Matrix QuantsYou will understand why 'imatrix' and IQ-quant filenames keep showing up on Hugging Face and when they genuinely beat plain Q4 quants.INTERMEDIATEQuality Loss from QuantizationYou will be able to reason about the real accuracy cost of each quantization level and predict which of your tasks will suffer first.INTERMEDIATEConvert a Model to GGUFYou will be able to take raw Hugging Face weights and produce your own quantized GGUF file ready to run locally.INTERMEDIATEAWQYou will understand what AWQ is, how protecting a small set of salient weights preserves accuracy at 4-bit, and why it has overtaken GPTQ for new work.ADVANCEDGPTQYou will understand what GPTQ is, how its layer-wise error minimization quantizes weights to 4-bit, and where its tooling lives today.ADVANCEDbitsandbytesYou will understand what bitsandbytes is, how it quantizes weights to 8-bit or NF4 4-bit on the fly at load time, and why it underpins QLoRA fine-tuning.ADVANCEDMLXYou will understand what MLX is, how it exploits Apple Silicon's unified memory for fast on-device inference, and where it fits among Mac-based local LLM tools.INTERMEDIATE