{"id":2333,"date":"2021-06-08T12:51:23","date_gmt":"2021-06-08T20:51:23","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/comet-newsletter-issue-4-etsy-ab-testing-mlops-landscape-deepsloth\/"},"modified":"2021-06-08T12:51:23","modified_gmt":"2021-06-08T20:51:23","slug":"comet-newsletter-issue-4-etsy-ab-testing-mlops-landscape-deepsloth","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/comet-newsletter-issue-4-etsy-ab-testing-mlops-landscape-deepsloth\/","title":{"rendered":"Issue 4: A\/B Testing at Etsy, New Adversarial Attach, MLOps Landscape"},"content":{"rendered":"\n

An inside look at Etsy’s approach to A\/B testing, examining an adversarial attack called DeepSloth, exploring the MLOps landscape, and more<\/em>.<\/h3>\n\n\n\n

Welcome to issue #4 of The Comet Newsletter!<\/p>\n\n\n\n

This week, we take a closer look at an adversarial attack called DeepSloth<\/a> that targets Adaptive Deep Neural Nets. Additionally, we explore Etsy\u2019s approach to developing more effective A\/B testing<\/a> with control variates.<\/p>\n\n\n\n

Additionally, you might enjoy a tutorial for building a Few-Shot Predictions Transformer with Hugging Face<\/a>, as well as a deep dive into the current state of the MLOps landscape<\/a>.<\/p>\n\n\n\n

Like what you\u2019re reading? Subscribe here.<\/a><\/h4>\n\n\n\n

And be sure to follow us on Twitter<\/a> and LinkedIn<\/a> \u2014 drop us a note if you have something we should cover in an upcoming issue!<\/p>\n\n\n\n

Happy Reading,<\/p>\n\n\n\n

Austin<\/p>\n\n\n\n

Head of Community, Comet<\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY<\/strong> | WHAT WE’RE READING | PROJECTS | OPINION<\/p>\n\n\n\n

Machine learning security needs new perspectives and incentives<\/h2>\n\n\n\n

At this year\u2019s International Conference on Learning Representations (ICLR) researchers from the University of Maryland presented a new type of adversarial attack intended to slow down deep learning models deployed on edge devices.<\/p>\n\n\n\n

Tudor Dumitras\u2019 (@tudor_dumitras<\/a>) group presented an attack called DeepSloth that targets Multi-Exit Strategy Neural Networks, specifically Shallow-Deep Networks<\/a>, an architecture developed by their group in 2019. These types of architectures save compute resources by providing multiple exit points for model predictions. This means that computation does not need to flow through every layer of the network.<\/p>\n\n\n\n

For classification tasks, the technique adds a classifier after certain intermediate layers of the network. The model will bypass the calculations of the remaining layers when it becomes confident about the prediction from the earlier layers. The researchers describe this as avoiding \u201cNetwork Overthinking\u201d.<\/p>\n\n\n\n

\n
\"\"\n
A diagram of a Shallow-Deep Network<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

These types of models can be deployed in scenarios where the model is split between an edge device (mobile, IoT device) and a cloud inference server. The earlier layers of the network are present on the edge device, while the deeper layers reside in the cloud. In cases where the edge side of the model does not reach a conclusive result, it defers further computations to the cloud.\u00a0\u00a0<\/p>\n\n\n\n

The DeepSloth attack creates a specific type of input that forces the network to \u201coverthink\u201d, and send all its requests to the cloud server.<\/p>\n\n\n\n

According to Dumitras, \u201cIn a scenario typical for IoT deployments, where the model is partitioned between edge devices and the cloud, DeepSloth amplifies the latency by 1.5\u20135X, negating the benefits of model partitioning.”\u00a0<\/p>\n\n\n\n

Additionally, Yigitcan Kaya (@YigitcanKaya1), one of the PhD students responsible for developing DeepSloth, says that one of the biggest problems in Machine Learning security research is that it\u2019s treated as an academic problem. Instead, Kaya argues, the authors should be encouraged to find ways to break their own solutions like their group did with Shallow-Deep Neural Networks.\u00a0<\/p>\n\n\n\n

\u201cIf an industry practitioner finds these papers and implements these solutions, they are not warned about what can go wrong,” noted Kaya. “Although groups like ours try to expose potential problems, we are less visible to a practitioner who wants to use an early-exit model. Even including a paragraph about the potential risks involved in a solution goes a long way.\u201d<\/p>\n\n\n\n

Read Ben Dickson\u2019s full article here.<\/a><\/p>\n\n\n

——————————–<\/center>\n\n\n

INDUSTRY | WHAT WE’RE READING<\/strong> | PROJECTS | OPINION<\/p>\n\n\n\n

Etsy Engineering: Increasing experimentation accuracy and speed by using control variates<\/h2>\n\n\n\n

 <\/p>\n\n\n\n

\n
\"\"\n
Image Source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

In this post, members from Etsy\u2019s Online Experimentation Science team\u2014a group of engineers and statisticians responsible for building more advanced and scalable statistical tools for online experiments\u2014describe details about bringing the statistical method known as CUPED to Etsy.\u00a0<\/p>\n\n\n\n

CUPED is an acronym for Controlled Experiments Using Pre-Experiment Data. The method is focused on estimating treatment effects in A\/B tests more accurately. The challenge lies in quantifying how much of the observed difference is due to real effects, and how much of it is due to noise caused by comparing two subpopulations made of non-identical users.<\/p>\n\n\n\n

One way to render these latter fluctuations negligible is to increase the number of users in each group; however this is not always a viable approach, as it can lead to longer experimentation times, and sometimes it just isn\u2019t possible to get a larger sample.\u00a0<\/p>\n\n\n\n

Traditionally, A\/B testing looks for differences in outcomes between sample means:<\/p>\n\n\n\n

YA \u2013 YB<\/pre>\n\n\n\n
The key idea behind CUPED is not only to play with sample sizes, but also to explain parts of these fluctuations away with the help of other measurable discrepancies between the groups.<\/p>\n\n\n\n
The CUPED estimator can be written as:<\/p>\n\n\n\n
YA \u2013 YB \u2013 (XA \u2013 XB) \u03b2<\/strong><\/pre>\n\n\n\n
which corrects the traditional estimator with an additional denoising term<\/strong>. Intuitively, this denoising term aims to account for how much of the difference in Y is not due to any effect of the treatment, but rather due to differences in other observable attributes (X) of the groups<\/strong>. In the equation above, X can be a well-thought-out vector of user attributes (so-called covariates and symbolized by X), and a vector \u03b2 of coefficients to be specified.<\/p>\n\n\n\n
X is a vector of pre-experiment variables (collected prior to each user\u2019s entry into the experiment) that ensures the correction term added by CUPED does not introduce any bias.<\/p>\n\n\n\n
According to the authors, \u201cThe more information on Y we can obtain from X, the more variance reduction we can achieve with CUPED. In the context of A\/B testing, smaller variances imply smaller sample size requirements, hence shorter experiments\u201d<\/p>\n\n\n\n
Read the team\u2019s full article here.<\/a><\/p>\n\n\n
——————————–<\/center>\n\n\n
INDUSTRY | WHAT WE’RE READING | PROJECTS <\/strong> | OPINION<\/p>\n\n\n\n
Few-shot learning in practice: GPT-Neo and the Hugging Face Accelerated Inference API<\/h3>\n\n\n\n
 <\/p>\n\n\n\n
\n
\"\"\n
Image Source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n
Transformers continue to be the dominant architecture of 2021. Apart from demolishing benchmarks on previous NLP tasks, large Transformer models now find applications in Few-Shot Learning problems.<\/p>\n\n\n\n
Typically, the quality of machine learning applications is limited based on the amount of available training data. In this guide, Philipp Schmid (@_philschmid<\/a>) from Hugging Face demonstrates how providing a few examples at inference time with a large language model can overcome this barrier for fine tuning a pretrained model for a specified task.<\/p>\n\n\n\n
Schmid uses the GPT-Neo<\/a>, an open source attempt at recreating GPT3, and Hugging Face\u2019s Accelerated Inference API to build a Few-Shot Prediction model that does not require fine tuning on the domain task.<\/p>\n\n\n\n
Read Schmid\u2019s full article here.<\/a><\/p>\n\n\n
——————————–<\/center>\n\n\n
INDUSTRY | WHAT WE’RE READING | PROJECTS | OPINION<\/strong><\/p>\n\n\n\n
Navigating the MLOps Tooling Landscape: A 3-part Series<\/h3>\n\n\n\n
 <\/p>\n\n\n\n
\n
\"\"\n
Image Source<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n
The past 3 years have seen a huge rise in companies focusing on the MLOps space. The sheer number of options available these days for tooling around production Machine Learning can lead to a lot of decision fatigue.\u00a0\u00a0<\/p>\n\n\n\n
In this series of posts, LJ Miranda (@ljvmiranda921<\/a>) outlines a decision framework for selecting the appropriate tools for your Machine Learning projects. His series addresses some key questions such as, \u201cWho is benefiting from adopting this tool?\u201d, \u201cWhat is the exact need that your team is trying to solve with this tool?\u201d and \u201cWhat are the adoption strategies that you can apply when considering a new tool?\u201d<\/p>\n\n\n\n
His work builds on insights gathered from Thoughtworks\u2019 Technology Radar<\/a>. Different organizations approach Machine Learning in very different ways, and it is rare that a single tool can solve all the needs of every organization. Evaluation frameworks help drive better adoption practices by forcing organizations to actively question the core issues they are having around ML productionization.\u00a0\u00a0<\/p>\n\n\n\n
Read Miranda\u2019s full exploration in 3 parts:<\/p>\n\n\n\n