# LookAtMyQuant

> A quantization compiler that spends bits where the model needs them most

- Category: Models & Inference / Quantization & Compression
- Language: Python
- Licence: MIT
- Repository: https://github.com/ShinMegamiBoson/lookatmyquant
- Website: https://lookatmyquant.pages.dev/
- Canonical page: https://ai-tldr.dev/tools/lookatmyquant/
- Updated: 2026-11-21

## In short

LookAtMyQuant is an open-source quantization compiler for language models, released under the MIT license by the developer ShinMegamiBoson. It sits in the Quantization & Compression category of the Models & Inference tools and is open-source, written in Python and licensed MIT. To get started: install the Apple Silicon runtime.

## Overview

LookAtMyQuant is an open-source quantization compiler for language models, released under the MIT license by the developer ShinMegamiBoson. Instead of applying one format to the whole model, it measures how much each stored piece of the model (weight matrices, frequency-bucketed embedding rows and key-value caches) damages the output when compressed, then chooses a compression code for each piece so the whole model fits a size or speed budget with the least total damage.

The pipeline has three steps. Measure compresses one piece at a time while keeping everything else exact and records the KL divergence from the original model. Allocate picks one code per piece to minimize the summed KL within a bits-per-weight budget, which needs no GPU. Build quantizes the model to that plan one matrix at a time and reports perplexity and KL on WikiText-2. Plans can be exported to llama.cpp GGUF, and the project's own packed formats run on Apple Silicon through an MLX runtime with custom Metal kernels and optional small drafter models for speculative decoding.

The README reports a 7.2 GB Qwen3.8-27B (2.1 bits per weight) that solves 401 of 540 math problems, the same as the original, and a Gemma 4 E2B GGUF with 12 times lower KL than Q4_K_M at the same file size. A second recipe, LookAtMyQuant 2, adds trellis codes, calibration on the model's own worked math solutions and 4,000 steps of distillation, and packs Qwen3.5-4B into 1.23 GB. In November 2026 the compiler was extended to Thinking Machines' Inkling 2 Mini, producing 5.38 GB and 8.87 GB builds.

## What it does

- Per-piece sensitivity measurement: KL divergence from the original model for each weight matrix, embedding bucket and KV cache
- Budgeted allocation that picks one compression code per piece to fit a bits-per-weight target, without a GPU
- Trellis codes for blocks and rows, Fisher-information bit allocation and an optional distillation pass (LookAtMyQuant 2)
- Export of compiled plans to llama.cpp GGUF
- An Apple Silicon MLX runtime, the `lamq` CLI, with custom Metal kernels, compatible with mlx_lm generate and stream_generate
- Optional 4-bit drafter models for speculative decoding, reported at about 2x faster decode on math and code

## Getting started

The packed models run on Apple Silicon through the project's MLX runtime. These commands are taken from the README.

1. Install the Apple Silicon runtime

   Install the package from GitHub. From a local clone, `pip install -e .` installs the same `lamq` command.

```bash
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git
```

2. Generate with a packed model

   Point `lamq` at a packed model on Hugging Face. Add a drafter model to speed up decoding, and `--stats` to print throughput.

```bash
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \
    --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
    -p "What is 17 times 23? Show the steps." --stats
```

3. Use it from Python

   The runtime's `load` returns a model and tokenizer that work with mlx_lm's `generate`.

```python
from mlx_lm import generate
from runtime_mlx import load

model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 17 times 23? Show the steps."}],
    add_generation_prompt=True,
    enable_thinking=False,
)
print(generate(model, tok, prompt, max_tokens=200))
```

4. Compile your own plan

   The compiler quick start measures sensitivity on SmolLM2-360M, allocates a 3.044-bit plan and builds it.

```bash
pip install -r requirements.txt
python compiler.py measure --model HuggingFaceTB/SmolLM2-360M --part linear --layers 0-3 --out results/batch/my_measurements.json
python compiler.py allocate --sens 'results/batch/ccSens*.json' --target disk --items linear --bits 3.044 --out results/compiler/plans/my_plan.json
python quantize.py --program plan --model HuggingFaceTB/SmolLM2-360M --plan results/compiler/plans/my_plan.json --out results/my_run.json
```

## When to use it

- Run Inkling 2 Mini on an Apple Silicon Mac in about 6 GB of active memory
- Compare quantization recipes by KL divergence and task accuracy at a fixed file size
- Build a size-budgeted GGUF for llama.cpp from a measured sensitivity plan
- Speed up local decoding on Apple Silicon with a small drafter model

## Version history

- 2026-11-21: Added Inkling 2 Mini support: 5.38 GB (1.63 bits per weight) and 8.87 GB (2.69 bits per weight) builds for Apple Silicon, with a drafter model for faster decoding.
