{"id":2296,"date":"2021-04-28T17:58:34","date_gmt":"2021-04-29T01:58:34","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/how-to-10x-throughput-when-serving-hugging-face-models-without-a-gpu\/"},"modified":"2021-04-28T17:58:34","modified_gmt":"2021-04-29T01:58:34","slug":"how-to-10x-throughput-when-serving-hugging-face-models-without-a-gpu","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/how-to-10x-throughput-when-serving-hugging-face-models-without-a-gpu\/","title":{"rendered":"How to 10x Throughput When Serving Hugging Face Models Without a GPU"},"content":{"rendered":"\n
\n

In less than 50 lines of code, you can deploy a Bert-like model from the Hugging Face library and achieve over 100 requests per second with latencies below 100 milliseconds for less than $250 a\u00a0month.<\/p>\n<\/blockquote>\n\n\n\n

The code for this blog post available here: <\/em>https:\/\/github.com\/comet-ml\/blog-serving-hugging-face-models<\/a>.<\/p>\n\n\n\n

Simple models and simple inference pipelines are much more likely to generate business value than complex approaches. When it comes to deploying NLP models, nothing is as simple as creating a FastAPI server to make real-time predictions.<\/p>\n\n\n\n

While GPU accelerated inference has its place, this blog post will focus on how to optimize your CPU inference service to achieve sub 100 millisecond latency and over 100 requests per second throughput. One key advantage of using a Python inference service rather than more complex GPU accelerated deployment options is that we will be able to have the tokenization built-in further reducing the complexity of the deployment.<\/p>\n\n\n\n

In order to achieve good performance for CPU inference we need to make optimisations to our serving framework. We breakdown down the post into:<\/p>\n\n\n\n

    \n
  1. Benchmarking setup<\/strong><\/li>\n
  2. Baseline:<\/strong> FastAPI web server using default options<\/li>\n
  3. Pytorch and FastAPI optimizations:<\/strong> Tuning FastAPI for ML inference<\/li>\n
  4. Model optimizations:<\/strong> Using model distillation and quantisation to improve performance<\/li>\n
  5. Hardware optimization:<\/strong> 3x performance improvement by choosing the right cloud instances to use<\/li>\n<\/ol>\n\n\n\n

    Benchmarking setup<\/h2>\n\n\n\n

    Benchmarks are notoriously difficult [1], we highly recommend you create your own based on your specific requirements. We provide all the code used to reproduce the numbers presented below on Github here<\/a>.<\/p>\n\n\n\n

    As we can\u2019t test everything, we have had to make a number of decisions:<\/p>\n\n\n\n