What (on earth) is AI model distillation?

Up next in my teaching blog series: model distillation. A very important and lesser-known concept in the world of AI.

Imagine you own a restaurant and you hire a Michelin-starred chef to run your kitchen. The chef can cook the most brilliant dishes but usually likes to take her time when cooking (perfection isn't quick right) and makes you sweat a bit when you look at the payroll costs at the end of the month.

But you have a great idea - you ask the Michelin-starred chef to teach your array of junior chefs in all of her different skills & techniques - getting each junior chef to specialise in a different skill. After a few months, your junior chefs are whizz-kids at the specific skill, can often do it faster than the head chef & are cheaper to pay. Yes they won't know everything your head chef knows, but... that's OK. Billy stays at the egg station and doesn't leave the egg station.

That, very roughly, is model distillation.

A large AI model - the teacher - helps train a smaller model - the student. The student learns to produce useful responses to selected inputs by studying examples or outputs from the teacher. Do a good job and the student may become highly capable at a narrow, specialised task. Ask it something outside of that task, and you may be disappointed quickly. This often involves a great deal of prompts & inference, but vastly less than the initial amount of training to generate the teacher model itself.

Why bother?

A big model can be powerful, but using it for every request can cost more and take longer. A smaller distilled model may be cheaper and faster to run, and can sometimes be deployed on more modest hardware. That makes it attractive for routine tasks where you don’t need the most capable model available.

Why are frontier labs concerned?

But as with most things, distillation is not immune to malicious intent.

Good distillation - frontier labs can decide to distill their own models. Anthropic actively does this. It's a good way to offer customers smaller, faster, specialised models & reap the monetary benefits.

Bad distillation - however malicious actors can also try to extract/distil another lab's model capabilities by sending it huge numbers of queries, collecting the responses and using them as training for their own model. These kinds of large-scale, co-ordinated "distillation attacks" are quite commonplace.

The threat is sort of obvious. Frontier labs have spent trillions of $ training these extremely advanced frontier models - and then someone comes along with a couple prompts and has their own specialised model within a few days. Yes it's not going to have exactly the same depth & breadth of knowledge but if it's good enough and the customer will pay, then happy days.

There are also safety concerns - a distilled model is a lot more black box. It may not have the same safeguards, testing or context around how it has actually been trained. Copying the answer doesn't necessarily mean copying the rules & guardrails.

I haven't quite figured out exactly how worried the Frontier labs are at this distillation threat. My last blog post on inference marketplaces is sort of related. If you could distill down lots of specialised models, undercut the frontier models in price & get it on the inference marketplace - then pretty quickly you could be a compelling competitive player against some of these bigger labs.

Anyway, I'm hungry. Get back to that egg station Billy, those eggs aren't going to cook themselves.