Knowledge Distillation

A model compression technique where a smaller student model is trained to mimic the behaviour of a larger, more complex teacher model, preserving performance with reduced computational cost.

Creating a smaller, faster AI by having it learn from a bigger, smarter AI. The big model is the teacher, the small model is the student. The student won't be quite as smart, but will be much cheaper to run.