{"id":7059,"date":"2023-08-08T11:54:07","date_gmt":"2023-08-08T19:54:07","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7059"},"modified":"2025-04-24T17:14:54","modified_gmt":"2025-04-24T17:14:54","slug":"deep-learning-has-a-size-problem","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/deep-learning-has-a-size-problem\/","title":{"rendered":"Deep Learning Has a Size Problem"},"content":{"rendered":"\n\n\n\n\n
\n
\"\"<\/figure>
\n
<\/picture><\/figure>\n<\/div>\n
\n
\n

The following is an adaptation of two talks I recently gave at the O\u2019Reilly AI Conference and DroidCon in London. Slides are available<\/a> at the end of this post.<\/p><\/blockquote>\n

Earlier this year, researchers at NVIDIA announced MegatronLM<\/a>, a massive transformer model with 8.3 billion parameters (24 times larger than BERT) that achieved state-of-the-art performance on a variety of language tasks. While this was an undoubtedly impressive technical achievement, I couldn\u2019t help but ask myself: is deep learning going in the right direction?<\/p>\n

The parameters alone weigh in at just over 33 GB on disk. Training the final model took 512 V100 GPUs running continuously for 9.2 days. Given the power requirements per card, a back of the envelope estimate put the amount of energy used to train this model at over 3X the yearly energy consumption of the average American.<\/p>\n

I don\u2019t mean to single out this particular project. There are many examples<\/a> of massive models<\/a> being trained<\/a> to achieve ever-so-slightly higher accuracy on various benchmarks. Despite being 24X larger than BERT, MegatronLM is only 34% better at its language modeling task. As a one-off experiment to demonstrate the performance of new hardware, there isn\u2019t much harm here. But in the long term, this trend is going to cause a few problems<\/a>.<\/p>\n

First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to make every application and device better, we won\u2019t get there with massive models that take large amounts of time and money to train.<\/p>\n

Second, it restricts scale. There are probably less than 100 million processors in every public and private cloud in the world. But there are already 3 billion mobile phones, 12 billion IoT devices, and 150 billion micro-controllers out there. In the long term, it\u2019s these small, low power devices that will consume the most deep learning, and massive models simply won\u2019t be an option.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

To make sure deep learning lives up to its promise, we need to re-orient research away from state-of-the-art accuracy and towards state-of-the-art efficiency. We need to ask if models enable the largest number of people to iterate as fast as possible using the fewest amount of resources on the most devices.<\/p>\n

The good news is that work is being done to make deep learning models smaller, faster, and more efficient. Early returns are incredible. Take, for example, one result from a 2015 paper by Han et al<\/a>.<\/p>\n

\u201cOn the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy.\u201d<\/p>\n

To achieve results like this, we have to consider the entire machine learning lifecycle\u2014from model selection to training to deployment. For the rest of this article, we\u2019ll dive into those phases and look at ways to make smaller, faster, more efficient models.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

Model Selection<\/strong><\/h1>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

The best way to end up with a smaller, more efficient model is to start with one. The graph above plots the rough size (in megabytes) of various model architectures. I\u2019ve overlaid lines denoting the typical size of mobile applications (code and assets included), as well as the amount of SRAM that might be available in an embedded device.<\/p>\n

The logarithmic scale on the Y-axis softens the visual blow, but the unfortunate truth is that the majority of model architectures are orders of magnitude too large for deployment anywhere but the larger corners of a datacenter.<\/p>\n

Incredibly, the smaller architectures to the right don\u2019t perform much worse than the large ones<\/a> to the left. An architecture like VGG-16 (300\u2013500MB) performs about as well as a MobileNet (20MB) model, despite being nearly 25X smaller.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n

What makes smaller architectures like MobileNet and SqueezeNet so efficient? Based on experiments by Iandola et al<\/a> (SqueezeNet), Howard et al<\/a>(MobileNetV3), and Chen et al<\/a> (DeepLab V3), some answers lie in the macro- and micro-architectures of models.<\/p>\n

Macro-architecture refers the types of layers used by a model and how they are arranged into modules and blocks. To produce efficient macro-architectures:<\/p>\n