{"id":3540,"date":"2019-08-06T08:58:38","date_gmt":"2019-08-06T16:58:38","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=3540"},"modified":"2025-04-24T17:31:00","modified_gmt":"2025-04-24T17:31:00","slug":"building-reliable-machine-learning-pipelines-with-aws-sagemaker-and-comet-ml","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/building-reliable-machine-learning-pipelines-with-aws-sagemaker-and-comet-ml\/","title":{"rendered":"Building reliable machine learning pipelines with AWS Sagemaker and Comet"},"content":{"rendered":"\n
\n

This tutorial is Part II of a series. See Part I <\/em>here<\/a>.<\/em><\/p>\n<\/blockquote>\n\n\n\n

Successfully executing machine learning at scale involves building reliable feedback loops around your models. As your pipeline grows, you will reach a point where your data can no longer fit in memory on a single machine, and your training processes will have to run in a distributed way. Regular retraining, and hyperparameter optimization of models, will become necessary as new data becomes available and your underlying feature distributions change.<\/p>\n\n\n\n

There is also the added complexity of trying new modeling approaches on your data and communicating the results across teams and other stakeholders.<\/p>\n\n\n\n

In this blog post, we will illustrate how to use AWS Sagemaker and Comet.ml to simplify this process of monitoring and improving your training pipeline.<\/strong><\/p>\n\n\n\n

Data complexity + model needs are growing<\/h4>\n\n\n\n

The challenges involved in creating functional feedback loops extend beyond data concerns like tracking changing data distributions to issues at the model level. Robust models require frequent retraining so that hyperparameters are optimal in the face of new, incoming data.<\/p>\n\n\n\n

With each iteration, it becomes harder to manage subsets and variations of your data and models. Keeping track of which model iteration ran on which dataset is key to reproducibility.<\/p>\n\n\n\n

Take that complexity + multiply it by team complexity<\/h4>\n\n\n\n

Managing these complex pipelines can become even more difficult in team settings. Data scientists will often store results from several models in log files, making it impossible to reproduce models or communicate results effectively.<\/p>\n\n\n\n

Traditional software development tools are not optimized for the iterative nature of machine learning or the scale of data machine learning requires. The lack of tools and processes to make collaboration easy across members of the same data science teams (and across functions like engineering) has led to dramatically slower iteration cycles. Organizations also constantly suffer the pain of slow on-boarding times for new employees and bear the risk of employees churning along with their work and proprietary knowledge.<\/p>\n\n\n\n

Tutorial<\/h4>\n\n\n\n

This tutorial covers how to integrate Comet.ml<\/a> with AWS Sagemaker\u2019s TensorFlow Estimator API<\/a>. We will be adapting running the Resnet model on the CIFAR10 dataset with TensorFlow<\/a>.<\/p>\n\n\n\n

Instead of logging experiment metrics to Tensorboard, we\u2019re going to log them to Comet.ml.<\/p>\n\n\n\n

This allows us to keep track of various hyperparameter configurations, metrics, and code across different training runs.<\/p>\n\n\n\n