Knowledge Distillation
A model compression technique where a smaller student model is trained to mimic the behaviour of a larger, more complex teacher model, preserving most performance with substantially reduced computational cost.
In plain language
Creating a smaller, faster AI by having it learn from a bigger, smarter AI. The large model is the teacher, the small model is the student. The student won't be quite as capable, but will be much cheaper and faster to run.
Why this matters
Knowledge distillation is a practical implementation technique for reducing the computational footprint and cost of AI systems. Using distilled models can lower deployment costs, reduce environmental impact and improve response times, making AI solutions more accessible and sustainable for Australian organisations.
Relevance
ImplementationKnowledge distillation reduces computational requirements and costs, making AI deployment more practical and sustainable for resource-constrained environments.
Related terms
Putting knowledge distillation into practice in your organisation?
Ready to transform your AI strategy?
Partner with Australia's AI strategy and governance specialists. From adoption roadmaps to ISO 42001 audit readiness.