{"id":1858,"date":"2019-05-13T20:11:04","date_gmt":"2019-05-14T04:11:04","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/blog\/building-a-fully-reproducible-machine-learning-pipeline-with-comet-ml-and-quilt\/"},"modified":"2019-05-13T20:11:04","modified_gmt":"2019-05-14T04:11:04","slug":"building-a-fully-reproducible-machine-learning-pipeline-with-comet-ml-and-quilt","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/building-a-fully-reproducible-machine-learning-pipeline-with-comet-ml-and-quilt\/","title":{"rendered":"Building a fully reproducible machine learning pipeline with comet.ml and Quilt"},"content":{"rendered":"\n

Classifying fruits using a Keras multi-class image classification model and Google Open Images<\/h4>\n\n\n\n

 <\/p>\n\n\n\n

\"\"\n
Photo by\u00a0Luke Michael<\/a>\u00a0on\u00a0Unsplash<\/a><\/figcaption>\n<\/figure>\n\n\n\n
\n

This post was written in collaboration with\u00a0<\/em>Aleksey Boligur<\/a>\u00a0from the Quilt Data team.\u00a0<\/em>Follow Aleksey on Twitter<\/a>\u00a0and his personal website\u00a0<\/em>here<\/a>. Follow Quilt\u00a0<\/em>here<\/a><\/p>\n<\/blockquote>\n\n\n\n

The term machine learning \u2018pipeline\u2019 can suggest a one-way flow of data and transformations, but in reality, machine learning pipelines are cyclical and iterative. For a given project, a data scientist can try hundreds and thousands of experiments before arriving at a champion model to put in production.<\/p>\n\n\n\n

With each iteration, it becomes harder to manage subsets and variations of your data and models. Keeping track of which model iteration ran on which dataset is key to reproducibility.<\/p>\n\n\n\n

\n
\n

Having a proper machine learning pipeline that\u00a0tracks specific versions of data, code, and environment<\/a>\u00a0details can not only help you easily reproduce your own model results, but also allow you to share your work with fellow data scientists or machine learning engineers who need to deploy your model.<\/p>\n<\/blockquote>\n<\/figure>\n\n\n

\n\n\n

In this article, we\u2019ll show you how to build a simple and reproducible end-to-end machine learning pipeline using a Keras image multi-class classification model and a custom dataset crafted from Google Open Images using\u00a0<\/strong>Quilt T4<\/strong><\/a>\u00a0and\u00a0<\/strong>comet.ml<\/strong><\/a><\/p>\n\n\n\n

You can access the full tutorial in\u00a0this Github repository<\/a>. For a walk-through of the tutorial, continue reading below \u2b07.\ufe0f<\/p>\n\n\n\n

Creating your custom dataset<\/strong><\/h2>\n\n\n\n

The Open Images Dataset is an attractive target for building image recognition algorithms because it is one of the largest, most accurate, and most easily accessible image recognition datasets. For image recognition tasks, Open Images contains 15 million bounding boxes for 600 categories of objects on 1.75 million images. Image labeling tasks meanwhile enjoy 30 million labels across almost 20,000 categories.<\/p>\n\n\n\n

The images come from Flickr and are of highly variable quality, as would be realistic in an applied machine learning setting.<\/p>\n\n\n\n

Downloading the entire Google Open Images corpus is possible and potentially necessary if you want to build a general purpose image classifier or bounding box algorithm. However downloading\u00a0everything<\/em>\u00a0is a waste if you just want a small categorical subset of the data in the corpus.\u00a0For this tutorial, we are just interested in downloading and working with fruit images.<\/strong><\/p>\n\n\n\n

\n
\"\"\n
View an interactive version of this plot on Quilt T4\u00a0here<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n

The\u00a0src\/openimager<\/code>\u00a0subfolder in\u00a0the Github repository<\/a>\u00a0provided contains a small module that handles downloading a categorical subset of the Open Images corpus: just the images corresponding with a user-selected group of labels, and just from the set of images with bounding box information attached. Instead of using the zipped blob files it does so by downloading the source images from Flickr directly.<\/p>\n\n\n\n

This script will allow you to download any subset of the 600 labels that do. Here\u2019s a taste of what\u2019s possible:<\/p>\n\n\n\n

football<\/code>,\u00a0toy<\/code><\/code>,\u00a0bird<\/code><\/code>,\u00a0cat<\/code><\/code>,\u00a0vase<\/code><\/code>,lemon<\/code><\/code>,\u00a0dog<\/code><\/code>,\u00a0elephant<\/code><\/code>,\u00a0shark<\/code><\/code>,\u00a0flower<\/code>,\u00a0furniture<\/code>,\u00a0airplane<\/code>,\u00a0spoon<\/code>,\u00a0bench<\/code>,\u00a0swan<\/code>,\u00a0peanut<\/code>,\u00a0camera<\/code>,\u00a0flute<\/code>,\u00a0helmet<\/code>,\u00a0pomegranate<\/code>,\u00a0crown<\/code>\u2026<\/p>\n\n\n\n

For the purposes of this article, we\u2019ll limit ourselves to just fruit classes including:<\/p>\n\n\n\n

apple<\/code>,\u00a0banana<\/code>,\u00a0cantaloupe<\/code>,\u00a0common_fig<\/code>,\u00a0grape<\/code>,\u00a0lemon<\/code>,\u00a0mango<\/code>,\u00a0orange<\/code>,\u00a0peach<\/code>,\u00a0pear<\/code>,\u00a0pineapple<\/code>,\u00a0pomegranate<\/code>,\u00a0strawberry<\/code>,\u00a0tomato<\/code>,\u00a0watermelon<\/code><\/p>\n\n\n\n

For more information on Open Images, check out the article \u2018How to classify photos in 600 classes using nine million Open Images<\/a>\u2019.<\/p>\n\n\n\n

Preprocessing your data \u2014 and packaging it<\/strong><\/h3>\n\n\n\n

This annotated Jupyter notebook<\/a>\u00a0in the\u00a0demo GitHub repository\u00a0<\/a>does this work. After running the notebook code, we will have an\u00a0images_cropped<\/code>\u00a0folder on disk containing all of the cropped images.<\/p>\n\n\n\n

It\u2019s easy to access the package of fruit class data along with the pre-processed images is via\u00a0the Quilt T4 package<\/a>\u00a0. In order to access the data, simply run this command:<\/p>\n\n\n\n

! pip install t4\n\nt4.Package.install('quilt\/open_fruit', registry='s3:\/\/quilt-example', dest='some\/path\/some\/where')<\/code><\/pre>\n\n\n\n
Looking closely at the fruit data, we can see that there is a class imbalance. There are over 26,000 samples of bananas but then only a few hundred labelled common fig or pear examples. This skew is important to note as we approach building our image classifier.<\/p>\n\n\n\n
\n
\"\"\n
View this plot on Quilt T4\u00a0here<\/a><\/figcaption>\n<\/figure>\n<\/div>\n\n\n\n
Building your image classification model<\/strong><\/h2>\n\n\n\n
Now that we\u2019ve downloaded our fruit data from Quilt, we can begin building our image classification model! As with any machine learning project, we\u2019ll go through a few experiments to try to maximize our model\u2019s validation accuracy:<\/p>\n\n\n\n