█

AI/TLDR

Buy me a coffee
Buy me a coffee

LookAtMyQuant

A quantization compiler that spends bits where the model needs them most

Quantization & CompressionOpen source
Updated
21 Nov 2026
Language
Python
License
MIT
Coverage
1 story

What's new

21 Nov 2026

Added Inkling 2 Mini support: 5.38 GB (1.63 bits per weight) and 8.87 GB (2.69 bits per weight) builds for Apple Silicon, with a drafter model for faster decoding.

Latest news

IN SHORT

What is LookAtMyQuant?

LookAtMyQuant is an open-source quantization compiler for language models, released under the MIT license by the developer ShinMegamiBoson. It sits in the Quantization & Compression category of the Models & Inference tools and is open-source, written in Python and licensed MIT. To get started: install the Apple Silicon runtime.

Updated Markdown version ↗

Overview

LookAtMyQuant is an open-source quantization compiler for language models, released under the MIT license by the developer ShinMegamiBoson. Instead of applying one format to the whole model, it measures how much each stored piece of the model (weight matrices, frequency-bucketed embedding rows and key-value caches) damages the output when compressed, then chooses a compression code for each piece so the whole model fits a size or speed budget with the least total damage.

The pipeline has three steps. Measure compresses one piece at a time while keeping everything else exact and records the KL divergence from the original model. Allocate picks one code per piece to minimize the summed KL within a bits-per-weight budget, which needs no GPU. Build quantizes the model to that plan one matrix at a time and reports perplexity and KL on WikiText-2. Plans can be exported to llama.cpp GGUF, and the project's own packed formats run on Apple Silicon through an MLX runtime with custom Metal kernels and optional small drafter models for speculative decoding.

Bar charts: LookAtMyQuant 2's 1.23 GB Qwen3.5-4B solves 330 of 540 math problems with KL 0.45, versus Bonsai 2 Ternary's 328 and 0.59
LookAtMyQuant 2 on Qwen3.5-4B: math problems solved and KL from the original, against Bonsai 2 Ternary 4B and smaller Qwen3.5 modelsLookAtMyQuant repository ↗

The README reports a 7.2 GB Qwen3.8-27B (2.1 bits per weight) that solves 401 of 540 math problems, the same as the original, and a Gemma 4 E2B GGUF with 12 times lower KL than Q4_K_M at the same file size. A second recipe, LookAtMyQuant 2, adds trellis codes, calibration on the model's own worked math solutions and 4,000 steps of distillation, and packs Qwen3.5-4B into 1.23 GB. In November 2026 the compiler was extended to Thinking Machines' Inkling 2 Mini, producing 5.38 GB and 8.87 GB builds.

What it does

  • Per-piece sensitivity measurement: KL divergence from the original model for each weight matrix, embedding bucket and KV cache
  • Budgeted allocation that picks one compression code per piece to fit a bits-per-weight target, without a GPU
  • Trellis codes for blocks and rows, Fisher-information bit allocation and an optional distillation pass (LookAtMyQuant 2)
  • Export of compiled plans to llama.cpp GGUF
  • An Apple Silicon MLX runtime, the `lamq` CLI, with custom Metal kernels, compatible with mlx_lm generate and stream_generate
  • Optional 4-bit drafter models for speculative decoding, reported at about 2x faster decode on math and code

Getting started

The packed models run on Apple Silicon through the project's MLX runtime. These commands are taken from the README.

Install the Apple Silicon runtime

Install the package from GitHub. From a local clone, `pip install -e .` installs the same `lamq` command.

bashbash
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git

Generate with a packed model

Point `lamq` at a packed model on Hugging Face. Add a drafter model to speed up decoding, and `--stats` to print throughput.

bashbash
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \
    --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
    -p "What is 17 times 23? Show the steps." --stats

Use it from Python

The runtime's `load` returns a model and tokenizer that work with mlx_lm's `generate`.

pythonpython
from mlx_lm import generate
from runtime_mlx import load

model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 17 times 23? Show the steps."}],
    add_generation_prompt=True,
    enable_thinking=False,
)
print(generate(model, tok, prompt, max_tokens=200))

Compile your own plan

The compiler quick start measures sensitivity on SmolLM2-360M, allocates a 3.044-bit plan and builds it.

bashbash
pip install -r requirements.txt
python compiler.py measure --model HuggingFaceTB/SmolLM2-360M --part linear --layers 0-3 --out results/batch/my_measurements.json
python compiler.py allocate --sens 'results/batch/ccSens*.json' --target disk --items linear --bits 3.044 --out results/compiler/plans/my_plan.json
python quantize.py --program plan --model HuggingFaceTB/SmolLM2-360M --plan results/compiler/plans/my_plan.json --out results/my_run.json

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Run Inkling 2 Mini on an Apple Silicon Mac in about 6 GB of active memory
  • Compare quantization recipes by KL divergence and task accuracy at a fixed file size
  • Build a size-budgeted GGUF for llama.cpp from a measured sensitivity plan
  • Speed up local decoding on Apple Silicon with a small drafter model

How LookAtMyQuant compares

LookAtMyQuant alongside other open-source quantization & compression tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
BitNet★ 42.4kMicrosoft's inference framework for running extremely low-bit (1.58-bit) quantized LLMs efficiently on CPUs.
bitsandbytes★ 8.5kA library that adds 8-bit and 4-bit quantization to PyTorch models, widely used for loading and fine-tuning LLMs in low memory.
LLM Compressor★ 4kA library for applying quantization and sparsity methods to LLMs to produce compressed models that run faster in vLLM.
AWQ★ 3.7kThe reference implementation of Activation-aware Weight Quantization for compressing LLMs to 4-bit with little accuracy loss.
torchao★ 3kPyTorch's native library for quantizing and applying low-bit data types to models for faster training and inference.
Intel Neural Compressor★ 2.7kAn Intel toolkit for compressing models through quantization, pruning, and distillation across multiple deep-learning frameworks.
GPTQModel★ 1.3kA model quantization toolkit, successor to AutoGPTQ, that compresses LLMs with GPTQ, AWQ, and other methods across many chips.
LookAtMyQuant—A quantization compiler that spends bits where the model needs them most