What Is DistillKit? Knowledge Distillation Toolkit

// In plain English

DistillKit is an open-source toolkit from Arcee AI for doing knowledge distillation on language models. Distillation is the move where you take a big, capable teacher model and train a smaller student model to behave like it. DistillKit is the set of scripts and config files that runs that training loop for you, so you don't have to wire the whole teacher-student pipeline by hand.

Think of it like a recording studio for an apprentice musician. The master player (the teacher) performs; the studio captures not just the final notes but the timing, the phrasing, the little choices in between; and the apprentice (the student) practices against that recording until they can reproduce the performance themselves. DistillKit is that studio: it handles loading both models, capturing what the teacher does, and nudging the student until its behavior lines up.

The reason a dedicated toolkit exists is that there are two quite different ways to record the teacher, and getting either one right involves fiddly machinery — running two models at once, lining up their vocabularies, measuring the gap between them, and feeding that gap back into training. DistillKit packages both methods, which it calls online and offline distillation, behind a config-driven workflow so you pick a strategy and point it at your models and data.

// Why it matters

Big models are expensive and slow. For a feature that runs on every user message or every row in a database, a frontier-size LLM is often a non-starter — too costly, too high-latency, too big to deploy where you need it. Distillation is the standard answer: build a small student that matches the teacher on your specific job. DistillKit matters because it makes the good version of that — true logit-level distillation, not just copying text — practical for a normal team.

You can do basic distillation with no special tooling: have the teacher write answers, save them, and fine-tune a small model on that text. That's black-box, response-level distillation, and it works. But it throws away most of what the teacher knows. The teacher's real signal lives in its full probability distribution over next tokens — its sense of which wrong answers are plausible and which are absurd. DistillKit exists to capture that richer signal, which plain text-fine-tuning can't reach.

• You own open models. DistillKit shines when you control both teacher and student weights — typically open models on Hugging Face. That lets it read the teacher's internals (its logits), which API-only distillation can't.
• You want quality, not just a quick copy. Logit-matching transfers far more per example than copying final text, so the student reaches teacher-like behavior from less data.
• You need a repeatable pipeline. Distillation has many moving parts. A config-driven toolkit makes runs reproducible and reviewable instead of a pile of one-off scripts.
• You're building a smaller production model. The end goal is a student that's cheaper to run, faster to respond, and small enough to deploy locally or quantize further.

// How it works

Under the hood, DistillKit runs a fine-tuning loop on the student with one addition: alongside the usual training target, it compares the student's output distribution to the teacher's and adds the gap between them to the loss. The student is pulled toward two things at once — producing the right answer, and producing it the way the teacher would. The headline choice you make is when the teacher runs: live during training (online) or ahead of time (offline).

Online distillation: teacher runs live

In online (also called logit-based) distillation, both models are loaded together. For each training batch, the student produces its distribution and the teacher is run on the same input to produce its distribution. DistillKit measures how far apart the two are — typically with KL divergence, a standard way to score the distance between two probability distributions — and trains the student to shrink that gap. The student learns the teacher's full, nuanced confidence (the so-called dark knowledge), not just its final pick.

The cost is obvious: you must hold both models in memory and run the teacher on every step, so a big teacher needs serious GPU room. The payoff is the richest possible signal, because the teacher reacts to the exact inputs the student sees, in real time.

Offline distillation: teacher runs first

In offline distillation, you split the job in two. First, you run the teacher over your dataset once and save what it produced — its outputs and, where supported, its stored logits or top-k token probabilities — to disk. Then you train the student against that saved record, with the teacher no longer loaded. Because the heavy teacher only runs during the one-time generation pass, the training pass is far lighter on memory and can be repeated cheaply.

Offline trades some fidelity and disk space for flexibility. The saved targets are frozen, so you can re-run training, tweak the student's hyperparameters, or restart a crashed job without paying for the teacher again. It's the friendlier path when the teacher is too large to sit in memory next to the student, or when you want to iterate on the student many times.

// Online vs offline: which to pick

The online/offline choice is the central decision in DistillKit, and it's a memory-versus-flexibility trade. Neither is strictly better — it depends on how big your teacher is and how many times you expect to retrain the student.

// Where DistillKit fits in your stack

DistillKit is config-driven: you describe a run in a YAML file — which teacher, which student, which dataset, online or offline, and the training settings — then launch the corresponding script. That keeps experiments reproducible, since the whole recipe lives in one reviewable file rather than scattered command-line flags.

# A distillation run is described declaratively, not coded by hand.
models:
  teacher: an-open-teacher-model      # weights you control
  student: a-smaller-student-model    # the model you'll ship

dataset:
  path: ./my_prompts.jsonl            # inputs to distill on

distillation:
  method: online                      # online (live) or offline (pre-generated)
  # 'temperature' softens the teacher's distribution so the
  # student can see the relative ordering of runner-up tokens.
  temperature: 2.0

training:
  learning_rate: 2.0e-5
  num_epochs: 3

It sits in the same neighborhood as general fine-tuning toolkits, but with a narrower job. A general fine-tuner trains a model on human-written examples. DistillKit is specialized for the case where the targets come from another model and you want to match that model's distribution, not just its text. The two complement each other — you might generate a synthetic dataset from the teacher, then distill against it.

Where does the student go after training? Usually straight into a serving stack. A distilled student is a normal model file, so you can run it locally, push it to a registry, or shrink it further with quantization before deployment. Distillation and quantization stack cleanly: distill to a smaller architecture first, then quantize that student to squeeze memory even more.

// Common pitfalls

Most DistillKit trouble isn't in the training math — it's in setup and expectations. The toolkit does the hard part; these are the places teams trip.

• Tokenizer mismatch. If teacher and student use different vocabularies and they can't be reconciled, you lose the rich logit signal and fall back to weak text-level distillation. Same-family pairs that share a tokenizer are the safest starting point.
• Running out of memory online. Online distillation loads both models at once. Pick an online run with a teacher far too big for your GPU and it simply won't fit — that's exactly when offline is the right call.
• Forgetting the disk cost of offline. Saving teacher outputs and logits for a large dataset can eat a surprising amount of storage. Plan for it before you kick off the generation pass.
• Inheriting the teacher's flaws. The student copies everything — including hallucinations, biases, and any safety gaps. A student can't reliably exceed its teacher, so a weak teacher caps your result.
• Skipping evaluation. A falling loss curve proves nothing about real quality. Measure the student against the teacher on a held-out set for your actual task before trusting it.

// Going deeper

Once the online/offline split makes sense, here are the nuances that decide whether a DistillKit run produces a genuinely strong student or a mediocre copy.

Temperature is doing real work. Distillation borrows the temperature knob from the original Hinton-style soft-label recipe. A confident teacher's raw probabilities are spiky — nearly all mass on one token — which hides the informative differences between runner-up options. Dividing the teacher's scores by a temperature above 1 softens the distribution so the student can learn the ordering of plausible alternatives. This is a training-time knob and has nothing to do with the sampling temperature you set at inference; they share a name and nothing else.

The loss is usually a blend. Pure logit-matching isn't always best. A common setup mixes the distillation loss (match the teacher's distribution) with the ordinary supervised loss (match the ground-truth answer), weighted together. That keeps the student anchored to correct answers while still absorbing the teacher's richer signal — a balance you tune per task.

Data choice dominates outcomes. The student only learns from prompts you distill on. Draw them from realistic traffic and the student excels where it matters; use unrepresentative prompts and it'll ace a test that never happens in production. This is the same lesson as in synthetic data for fine-tuning: the dataset, not the trainer, sets your ceiling. Distilling narrowly can also cause the student to lose unrelated abilities — a flavor of catastrophic forgetting — so include enough variety to retain the skills you still need.

Where to go next. DistillKit is one of several specialized fine-tuning tools; compare it against general trainers in fine-tuning toolkits compared, and check your hardware headroom with fine-tuning GPU memory before choosing online vs offline. The honest limit remains the same as for all distillation: you can transfer a teacher's existing skills into a smaller model, but you can't distill new capabilities into existence — the student is forever capped by the teacher it learned from.