Originally published at <\/em>https:\/\/www.jeremyjordan.me<\/em><\/a> on September 1, 2018, and was updated recently to reflect new resources.<\/em><\/p>\n\n\n\n The goal of this document is to provide a common framework for approaching machine learning projects that can be referenced by practitioners. If you build ML models, this post is for you. If you collaborate with people who build ML models, I hope that this guide provides you with a good perspective on the common project workflow. Knowledge of machine learning is assumed.<\/em><\/p>\n\n\n\n This overview intends to serve as a project \u201cchecklist\u201d for machine learning practitioners. Subsequent sections will provide more detail.<\/p>\n\n\n\n Machine learning projects are highly iterative; as you progress through the ML lifecycle, you\u2019ll find yourself iterating on a section until reaching a satisfactory level of performance, then proceeding forward to the next task (which may be circling back to an even earlier step). Moreover, a project isn\u2019t complete after you ship the first version; you get feedback from real-world interactions and redefine the goals for the next iteration of deployment.<\/p>\n\n\n\n A typical team is composed of:<\/p>\n\n\n\n It may be tempting to skip this section and dive right in to \u201cjust see what the models can do\u201d. Don\u2019t skip this section. All too often, you\u2019ll end up wasting time by delaying discussions surrounding the project goals and model evaluation criteria. Everyone should be working toward a common goal from the start of the project.<\/p>\n\n\n\n It\u2019s worth noting that defining the model task is not always straightforward. There\u2019s often many different approaches you can take towards solving a problem and it\u2019s not always immediately evident which is optimal. I\u2019ll write a follow-up blog post with more detailed advice on developing the requirements<\/em> for a machine learning project.<\/p>\n\n\n\n Ideal: project has high impact and high feasibility.<\/em><\/p>\n<\/blockquote>\n\n\n\n Mental models for evaluating project impact:<\/p>\n\n\n\n When evaluating projects, it can be useful to have a common language and understanding of the differences between traditional software and machine learning software. Andrej Karparthy\u2019s Software 2.0<\/a> is recommended reading for this topic.<\/p>\n\n\n\n Software 1.0<\/strong><\/p>\n\n\n\n Software 2.0<\/strong><\/p>\n\n\n\n See\u00a0<\/em>this talk<\/a>\u00a0for more detail.<\/em><\/p>\n<\/blockquote>\n\n\n\n A quick note on Software 1.0 and Software 2.0 \u2014 these two paradigms are not<\/em><\/strong> mutually exclusive. Software 2.0 is usually used to scale the logic<\/strong> component of traditional software systems by leveraging large amounts of data to enable more complex or nuanced decision logic.<\/p>\n\n\n\n For example, Jeff Dean talks<\/a> (at 27:15) about how the code for Google Translate used to be a very complicated system consisting of ~500k lines of code. Google was able to simplify this product by leveraging a machine learning model to perform the core logical task of translating text to a different language, requiring only ~500 lines of code to describe the model. However, this model still requires some \u201cSoftware 1.0\u201d code to process the user\u2019s query, invoke the machine learning model, and return the desired information to the user.<\/p>\n\n\n\n In summary, machine learning can drive large value in applications where decision logic is difficult or complicated for humans to write, but relatively easy for machines to learn. On that note, we\u2019ll continue to the next section to discuss how to evaluate whether a task is \u201crelatively easy\u201d for machines to learn.<\/p>\n\n\n\n Some useful questions to ask when determining the feasibility of a project:<\/p>\n\n\n\n – How hard is it to acquire data?<\/p>\n\n\n\n – How much data will be needed?<\/p>\n<\/blockquote>\n\n\n\n – How frequently does the system need to be right to be useful?<\/em><\/p>\n<\/blockquote>\n\n\n\n – Has the problem been reduced to practice?<\/p>\n\n\n\n – Will the model be deployed in a resource-constrained environment?<\/em><\/p>\n<\/blockquote>\n\n\n\n Establish a single value optimization metric for the project. Can also include several other satisficing<\/a> metrics (ie. performance thresholds) to evaluate models, but can only optimize<\/em><\/strong> a single metric.<\/p>\n\n\n\n Examples:<\/em><\/p>\n\n\n\n The optimization metric may be a weighted sum of many things which we care about. Revisit this metric as performance improves.<\/p>\n\n\n\n Some teams may choose to ignore a certain requirement at the start of the project, with the goal of revising their solution (to meet the ignored requirements) after they have discovered a promising general approach.<\/p>\n\n\n\n Decide at what point you will ship your first model.<\/p>\n\n\n\n Some teams aim for a \u201cneutral\u201d first launch: a first launch that explicitly deprioritizes machine learning gains, to avoid getting distracted. \u2014\u00a0<\/em>Google Rules of Machine Learning<\/a><\/p>\n<\/blockquote>\n\n\n\n The motivation behind this approach is that the first deployment should involve a simple model with focus spent on building the proper machine learning pipeline required for prediction. This allows you to deliver value quickly and avoid the trap of spending too much of your time trying to \u201csqueeze the juice.\u201d<\/a><\/p>\n\n\n\n A well-organized machine learning codebase should modularize data processing, model definition, model training, and experiment management.<\/p>\n\n\n\n Example codebase organization:<\/p>\n\n\n\n See other examples\u00a0<\/em>here<\/a>\u00a0and\u00a0<\/em>here<\/a>.<\/em><\/p>\n<\/blockquote>\n\n\n\n An ideal machine learning pipeline uses data which labels itself. For example, Tesla Autopilot has a model running that predicts when cars are about to cut into your lane<\/a>. In order to acquire labeled data in a systematic manner, you can simply observe when a car changes from a neighboring lane into the Tesla\u2019s lane and then rewind the video feed to label that a car is about to cut in to the lane.<\/p>\n\n\n\n As another example, suppose Facebook is building a model to predict user engagement when deciding how to order things on the newsfeed. After serving the user content based on a prediction, they can monitor engagement and turn this interaction into a labeled observation without any human effort. However, just be sure to think through this process and ensure that your \u201cself-labeling\u201d system won\u2019t get stuck in a feedback loop<\/em> with itself.<\/p>\n\n\n\n For many other cases, we must manually label data for the task we wish to automate. The quality of your data labels has a large<\/em> effect on the upper bound of model performance.<\/p>\n\n\n\n Most data labeling projects require multiple people, which necessitates labeling documentation<\/strong>. Even if you\u2019re the only person labeling the data, it makes sense to document your labeling criteria so that you maintain consistency.<\/p>\n\n\n\n One tricky case is where you decide to change your labeling methodology after already having labeled data. For example, in the Software 2.0 talk mentioned previously, Andrej Karparthy talks about<\/a> data which has no clear and obvious ground truth.<\/p>\n\n\n\n If you run into this, tag \u201chard-to-label\u201d examples<\/em> in some manner such that you can easily find all similar examples should you decide to change your labeling methodology down the road. Additionally, you should version your dataset<\/strong> and associate a given model with a dataset version.<\/p>\n\n\n\n Tip: After labeling data and training an initial model, look at the observations with the largest error. These examples are often poorly labeled.<\/em><\/p>\n<\/blockquote>\n\n\n\n Useful when you have a large amount of unlabeled data, need to decide what data you should label. Labeling data can be expensive, so we\u2019d like to limit the time spent on this task.<\/p>\n\n\n\n As a counterpoint, if you can afford to label your entire dataset, you probably should. Active learning adds another layer of complexity.<\/em><\/p>\n\n\n\n \u201cThe main hypothesis in active learning is that if a learning algorithm can choose the data it wants to learn from, it can perform better than traditional methods with substantially less data for training.\u201d \u2014\u00a0<\/em>DataCamp<\/a><\/p>\n<\/blockquote>\n\n\n\n General approach:<\/p>\n\n\n\n However, tasking humans with generating ground truth labels is expensive. Often times you\u2019ll have access to large swaths of unlabeled data and a limited labeling budget \u2014 how can you maximize the value from your data? In some cases, your data can have information which provides a noisy estimate of the ground truth. For example, if you\u2019re categorizing Instagram photos, you might have access to the hashtags used in the caption of the image<\/a>. Other times, you might have subject matter experts which can help you develop heuristics about the data.<\/p>\n\n\n\n Snorkel<\/a> is an interesting project produced by the Stanford DAWN (Data Analytics for What\u2019s Next) lab which formalizes an approach towards combining many noisy label estimates into a probabilistic ground truth. I\u2019d encourage you to check it out and see if you might be able to leverage the approach for your problem.<\/p>\n\n\n\n Establish performance baselines on your problem.<\/strong> Baselines are useful for both establishing a lower bound of expected performance (simple model baseline) and establishing a target performance level (human baseline).<\/p>\n\n\n\n Start simple and gradually ramp up complexity.<\/strong> This typically involves using a simple model, but can also include starting with a simpler version of your task.<\/p>\n\n\n\nOverview<\/h2>\n\n\n\n
Project lifecycle<\/strong><\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/project-lifecycle-1-1024x1024-1-1024x1024.png\")
<\/figure>\n\n\n\n1. Planning and project setup<\/a><\/strong><\/h4>\n\n\n\n
\n
2. Data collection and labeling<\/strong><\/a><\/h4>\n\n\n\n
\n
3. Model exploration<\/strong><\/a><\/h4>\n\n\n\n
\n
4. Model refinement<\/strong><\/a><\/h4>\n\n\n\n
\n
5. Testing and evaluation<\/strong><\/a><\/h4>\n\n\n\n
\n
6. Model deployment<\/strong><\/a><\/h4>\n\n\n\n
\n
7. Ongoing model maintenance<\/strong><\/a><\/h4>\n\n\n\n
\n
Team roles<\/strong><\/h4>\n\n\n\n
\n
\n\n\n\nPlanning and project setup<\/h2>\n\n\n\n
Prioritizing projects<\/h2>\n\n\n\n
\n
\n
\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/software-1-1024x359-1.jpg\")
<\/figure>\n\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/software-2-1024x361-1.jpg\")
<\/figure>\n\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/software-1-2-1024x413-1.jpg\")
<\/figure>\n\n\n\nDetermining feasibility<\/h2>\n\n\n\n
\n
\n
– How expensive is data labeling?<\/p>\n\n\n\n\n
\n
\n
\n
– Is there sufficient literature on the problem?<\/p>\n<\/blockquote>\n\n\n\n\n
\n
Specifying project requirements<\/h2>\n\n\n\n
\n
\n
Setting up a ML codebase<\/h2>\n\n\n\n
|\u2014\u2014 data\/ <- raw and processed data for your project\n| |\u2014\u2014README.md <- describes the data for the project\n|\n|\n|\u2014\u2014docker\/ <- specify one or many dockerfiles\n| |\u2014\u2014dockerfile <- Docker helps ensure consistent behavior\n| across multiple machines\/deployments\n|\n|\n|\u2014\u2014api\/\n| |\u2013\u2014app.py <- exposes model through REST client for\n| predictions\n|\n|\n|\u2014\u2014 project_name\/\n| |\u2014\u2014 networks\/ <- defines neural network architectures used\n| | |\u2014\u2014resnet.py\n| | |\u2014\u2013densenet.py\n| |\u2014\u2014 models\/ <- handles everything else needed w\/ network\n| | |\u2014\u2014base.py including data preprocessing and output\n| | |\u2014\u2014simple_baseline.py normalization\n| | |\u2014\u2014cnn.py\n| |\u2014\u2014configs\/\n| | |\u2014\u2014baseline.yaml\n| | |\u2014\u2014latest.yaml\n| |\u2014\u2014datasets.py <- manages construction of the dataset\n| |\u2014\u2014training.py <- defines actual training loop for the model\n| |\u2014\u2014experiment.py <- manages experiment process of evaluating\n| multiple models\/ideas. Constructs the\n| dataset\/model\n|\u2014\u2014scripts\/<\/code><\/pre>\n\n\n\nnetworks\/<\/code> defines the neural network architectures used. Only the computational graph is defined, these objects are agnostic to the input and output shapes, model losses, and training methodology.<\/p>\n\n\n\ndatasets.py<\/code> manages construction of the dataset. Handles data pipelining\/staging areas, shuffling, reading from disk.<\/p>\n\n\n\nexperiment.py<\/code> manages the experiment process of evaluating multiple models\/ideas. This constructs the dataset and model for a given experiment.<\/p>\n\n\n\ntraining.py<\/code> defines the actual training loop for the model, which is called by an Experiment object. This code interacts with the optimizer and handles logging during training.<\/p>\n\n\n\n\n
Data collection and labeling<\/h2>\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/data-labeling-1024x573-1.jpg\")
<\/figure>\n\n\n\n\n
Active Learning<\/strong><\/h4>\n\n\n\n
\n
\n
Leveraging weak labels<\/strong><\/h4>\n\n\n\n
Model exploration<\/h2>\n\n\n\n
\n