Fine-Tuning & Model Customization · TRACK 04/04
Distillation & Training Tools
Smaller models from bigger ones, synthetic data, and the toolkits that run the job.
// THE TRACK
01 · START HEREModel DistillationLearn how a small student model copies a big teacher's behavior, and why distillation is behind most of the fast, cheap models you use.BEGINNERCreating Synthetic DataLearn how to use a strong LLM to generate, diversify, and filter your own training dataset — and when that beats collecting human-labeled examples.INTERMEDIATESynthetic Training DataSee how teams use a strong LLM to mass-produce training examples for a cheaper one — and where machine-generated data quietly goes wrong.BEGINNERCatastrophic ForgettingLearn why a fine-tuned model can lose general abilities it used to have, and the mixing, regularization, and LoRA tricks that keep them.INTERMEDIATEContinued PretrainingUnderstand the heavyweight option for domain knowledge: pouring raw corpus text into a model's pretraining objective instead of Q&A pairs.INTERMEDIATEHosted Fine-TuningLearn the upload-a-file path to a custom model: what hosted fine-tuning platforms handle for you, what they charge, and the lock-in you accept.INTERMEDIATEFine-Tuning ToolkitsGet a clear map of the fine-tuning tooling landscape so you can pick the right trainer for your skill level, hardware, and need for control.INTERMEDIATEGPU Memory & OOM FixesUnderstand exactly where your VRAM goes during training and which levers — checkpointing, batch size, quantization, LoRA — actually free it up.INTERMEDIATEDistilabelYou will understand what Distilabel is, how it builds synthetic-data and AI-feedback pipelines, and why generated data matters for fine-tuning.INTERMEDIATEArgillaYou will understand what Argilla is, how it helps annotate and curate training data, and how it fits alongside synthetic-data tools.INTERMEDIATENVIDIA NeMo CuratorYou will understand what NeMo Curator is, how GPU acceleration speeds up large-scale data preparation, and which curation steps it covers.ADVANCEDDistillKitYou will understand what DistillKit is, how online and offline distillation differ, and how it transfers a teacher model's behavior to a smaller student.ADVANCED