{"id":7258,"date":"2023-08-21T09:01:47","date_gmt":"2023-08-21T17:01:47","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7258"},"modified":"2025-04-24T17:14:41","modified_gmt":"2025-04-24T17:14:41","slug":"research-guide-model-distillation-techniques-for-deep-learning","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/research-guide-model-distillation-techniques-for-deep-learning\/","title":{"rendered":"Research Guide: Model Distillation Techniques for Deep Learning"},"content":{"rendered":"\n
\n
\n
\"\"<\/picture>
Image Source<\/a><\/figcaption><\/figure>\n<\/div>\n
\n
\n

\n<\/p>

Knowledge distillation is a model compression technique whereby a small network (student) is taught by a larger trained neural network (teacher). The smaller network is trained to behave like the large neural network. This enables the deployment of such models on small devices such as mobile phones or other edge devices. In this guide, we\u2019ll look at a couple of papers that attempt to tackle this challenge.<\/p>\n

Distilling the Knowledge in a Neural Network (NIPS, 2014)<\/h2>\n

In this paper, a small model is trained to generalize in the same way as the larger teacher model. Transferring the generalization is done by using the class probabilities of the large model as targets while training the smaller model. If the large model is an ensemble of simpler models, the geometric or arithmetic mean of their predictive distributions is used as the target.<\/p>\n

\n
\n
\n
Distilling the Knowledge in a Neural Network<\/a><\/h6>\n<\/div>\n<\/div>\n<\/div>\n

In testing the distillation, the authors trained a single large neural net with two hidden layers of 1200 linear hidden units on 60,000 training cases. The network was regularized using dropout and weight-constraints. The input images were jittered by two pixels in any direction. This network had 67 test errors. A smaller network with two hidden layers of 800 rectified linear units and no regularization had 146 errors. When the smaller network was regularized by matching the soft targets with the large net, it obtained 74 test errors.<\/p>\n

The following results were obtained when the technique was used on speech recognition.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n

Contrastive Representation Distillation (2019)<\/h2>\n

This paper leverages the family of contrastive objectives to capture correlations and higher-order output dependencies. They are adapted in this paper for purposes of knowledge distillation from one neural network to another.<\/p>\n

\n
\n
\n
Contrastive Representation Distillation<\/a><\/h6>\n<\/div>\n<\/div>\n<\/div>\n

As shown below, the paper considers three distillation stages:<\/p>\n