To understand the Fourth Industrial Revolution, it helps to remember the first three.<\/span><\/p>\n\n\n\n The First Industrial Revolution began at the end of the 18th century and focused on mechanizing industrial processes. The Second Industrial Revolution introduced electrification, and the Third Industrial Revolution championed automatization. Each of these periods focused on reducing human intervention in industry, but what\u2019s left once factories are mechanized, electrified, and automated? The Fourth Industrial Revolution is all about harnessing the power of data and leveraging it to simulate cognition in industry. <\/span><\/p>\n\n\n\n More than anything, <\/span>Industry 4.0 is a paradigm shift in the way we organize and manage industrial processes to make the most of cyber-physical systems. <\/span>These smart manufacturing processes include artificial intelligence, machine learning, cloud- and edge-computing, Industrial IoT, distributed computing, augmented reality, and much, much more.<\/span><\/p>\n\n\n\n One of the most popular ways that artificial intelligence is being incorporated into industrial manufacturing is through automated defect detection or anomaly detection. Anomaly detection is the process of identifying anomalous items in a stream of input data and is a critical component of quality assurance in any production line. <\/span><\/p>\n\n\n\n Traditional manual defect detection methods are not only expensive and time-consuming but can also often be ineffective, as not all anomalies are visible to the human eye. However, with the advent of machine learning, computer vision can be leveraged to facilitate human operator work\u2013 or even completely automate this process!<\/span><\/p>\n\n\n\n Anomaly detection faces several unique challenges. It\u2019s often difficult to obtain a large amount of anomalous data, making traditional supervised learning techniques impractical. These class imbalances also mean that popular evaluation metrics like accuracy aren\u2019t relevant.<\/p>\n\n\n\n Furthermore, the difference between a normal sample and an anomalous one can be microscopic. It isn’t always feasible to predefine all types of anomalies at the outset of an experiment. Fortunately, to address each of these challenges and more, a team of artificial intelligence researchers at Intel have developed a cutting-edge, easy-to-implement, open-source package called Anomalib.<\/p>\n\n\n\n Anomalib is an open-source deep learning library developed by Intel that makes it easy to benchmark different anomaly detection algorithms on both public and custom datasets, all by simply modifying a config file. As the largest public collection of anomaly detection algorithms and datasets, it has a strong focus on image-based anomaly detection. It\u2019s a comprehensive, end-to-end solution that includes cutting-edge algorithms, relevant evaluation methods, prediction visualizations, hyperparameter optimization, and inference deployment code with Intel\u2019s OpenVINO Toolkit.<\/p>\n\n\n\n Anomalib uses unsupervised ML techniques to learn an implicit representation of normality with AutoEncoders, GANs, or a combination of both. During inference, new samples are compared against the embeddings of normal samples to determine whether or not they are anomalous. In this way, Anomalib allows you to save your sparse anomalous data for testing purposes only. It currently supports <\/span>ten cutting-edge anomaly detection models<\/span><\/a>, including <\/span>FastFlow<\/span><\/a>, <\/span>PaDiM<\/span><\/a>, <\/span>PatchCore<\/span><\/a>, and <\/span>CFlow<\/span><\/a> models, but is also continuously updated with the latest state-of-the-art algorithms. You can also <\/span>train models with custom data<\/span><\/a> or access public datasets like the MVTec or BeanTech <\/span>datasets through the API<\/span><\/a>. What\u2019s best, Anomalib makes end-to-end anomaly detection possible straight out-of-the-box, and without additional GPUs or super long training times.<\/span><\/p>\n\n\n\n Anomalib can do more than just support POC projects, however. In the real world, machine learning is a highly iterative process, and the details of all these iterations can get pretty confusing, pretty fast. <\/span>To optimize your model and get it production-ready, you\u2019ll need to log, manage, and version these details in an experiment tracking tool like Comet. <\/span><\/p>\n\n\n\n By pairing Anomalib with Comet, you can take advantage of all of the cutting-edge models of Anomalib, keep track of your model iterations with Comet, and then implement them in a production environment with Comet\u2019s model management tools. In these ways, Anomalib and Comet make the perfect pair to power production-grade anomaly detection for Industry 4.0!<\/span><\/p>\n\n\n\n Comet is a powerful tool that allows you to manage and version your training data<\/a>, track and compare training runs<\/a>, and monitor your models in production<\/a>\u200a\u2014\u200aall in one platform. And now Comet is fully integrated with Anomalib for experiment management, benchmarking, and hyperparameter optimization!<\/p>\n\n\n\n The Comet + Anomalib integration offers the following features:<\/p>\n\n\n\n In single-experiment view, Comet logs appropriate evaluation metrics in both tabular and chart form. By definition, anomaly detection is a problem of class imbalances, and this makes for poor performance with traditional ML metrics like accuracy, which are designed around an assumption of balanced class distribution.<\/p>\n\n\n\n Instead, Anomalib + Comet calculates the F1 score and AUROC at both image and pixel levels. The F1 score combines precision and recall into a single metric by taking their harmonic mean, while still accounting for the precision-recall tradeoff. The AUROC curve describes how well a model can distinguish between classes by plotting a probability curve at various thresholds, or degrees of separability, and is another very important metric for evaluating classification problems with class imbalances.<\/p>\n\n\n\n But that\u2019s not all! Scrolling through the left-hand sidebar of the single-experiment view, you\u2019ll find that quite a few metrics and features are automatically logged for you! Basic contextual information like source code, system metrics, installed packages, and output are all logged. Experiment-specific hyperparameters, metrics, and graph definitions are also auto-logged, and with a simple edit to the Anomalib config file, you can also log images and other graphics to Comet. Keeping track of all these metrics is essential for producing production-ready models and monitoring them for concept and data drift. <\/span><\/p>\n\n\n\n In the panel-level view, you\u2019ll see charts automatically populated with performance metrics for a bird\u2019s eye view of your project across experiment runs. You can also add Comet\u2019s new Image Panels to this view to visualize specific prediction images across different experiments, as shown below:<\/p>\n\n\n\n Lastly, it can be really important to see how things are shaping up between two specific experiment runs. Comet allows you to diff selected runs<\/a> for a more cross-sectional view of your project. This also allows you to compare specific metrics and parameters:<\/p>\n\n\n\n Anomalib also includes a benchmarking script for relating results across different combinations of models, their parameters, and dataset categories. Log the model performance and throughputs to Comet as a means to track model drift, or export them to a CSV file. You can check out the full documentation here<\/a>, and once your configuration is decided, perform your benchmarking with one simple command:<\/p>\n\n\n\n Anomalib also supports hyperparameter optimization with the Comet Optimizer, making it easier to isolate the right combination of hyperparameters. See here<\/a> for Anomalib\u2019s HPO docs, or see here<\/a> for details on other possible configurations with Comet’s Optimizer.<\/p>\n\n\n\n At the top of your Comet optimizer report, you\u2019ll find each of your hyperparameters ranked by the evaluation method of your choice, in order of the largest magnitude of Spearman correlation coefficient<\/a>, to the smallest. In the snapshot below, the learning rate had the largest correlation coefficient with the model\u2019s F1 score.<\/p>\n\n\n\n You\u2019ll also find all of your experiment runs ranked by your evaluation metric, with the additional option to toggle the ranking based on any of the model\u2019s parameters. Lastly, Comet plots your evaluation metrics across all experiment runs in line plots, and feel free to add any of Comet\u2019s publicly available custom panels!<\/p>\n\n\n\n To start tracking your Anomalib projects with Comet, just follow the four quick steps below. Feel free to follow along with this Colab tutorial<\/a>, and be sure to check out this completed Anomalib project here<\/a>, courtesy of Comet\u2019s own ML Growth Engineer, Sid Mehta.<\/p>\n\n\n\n Clone the Anomalib repo into your environment and install the necessary dependencies:<\/p>\n\n\n\n If you don\u2019t already have a Comet account, you can sign up for free here<\/a>. Make sure to grab your API key from your account settings so you can configure your Comet credentials in any of several ways<\/a>. For the sake of simplicity, we’ll set them directly through environment variables here:<\/p>\n\n\n\n Next, we\u2019ll need to modify our Anomalib config file to enable logging. The easiest way to do this is to open the existing configuration file at Alternatively, you can copy your particular model’s default config template into a new YAML file and adjust the parameters as needed. In this Colab notebook<\/a>, we\u2019ve also demonstrated how to use By default, To use a different algorithm, just switch out the model name in the config file path to another supported algorithm. To use a custom dataset, just update the relevant Anomalib config file accordingly with the path to your dataset.<\/p>\n\n\n\n Now just head over to the Comet UI to check out your results!<\/span><\/p>\n\n\n\n In the real world, machine learning is a highly iterative process and you\u2019ll likely have many more training runs to visualize and organize. Comet makes it easy to track these runs, share them with members of your team, and collaborate within your organization. By pairing Anomalib with Comet, you can take advantage of all the cutting-edge algorithms of Anomalib, and create production-worthy, maintainable models for your next Industry 4.0 project.<\/span><\/p>\n\n\n\n Used the Anomalib+Comet integration to train an anomaly detection model that you are now ready to deploy? Comet\u2019s role in aiding your Industry 4.0 journey doesn\u2019t stop at Experiment Management<\/a>; Comet also provides a Model Production Monitoring<\/a> (MPM) platform and Model Registry<\/a>. With MPM, you can monitor your model while it is in production in a manufacturing environment. MPM can send alerts when it detects anomalies or data drift, signaling it is time to re-train.<\/p>\n\n\n\n Contact the Comet team today to learn more about MPM!<\/p>\n\n\n\nWhat is Anomaly Detection?<\/h3>\n\n\n\n
Challenges of Anomaly Detection<\/h3>\n\n\n\n
What is Anomalib?<\/h3>\n\n\n\n
Why Anomalib?<\/h3>\n\n\n\n
Anomalib + Comet for Anomaly Detection<\/h3>\n\n\n\n
\n
![\"Auto-logged](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-03-at-8.06.10-PM.png\")
<\/a>Logging<\/h4>\n\n\n\n
Experiment-level<\/h5>\n\n\n\n
![\"Experiment-level](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-03-at-8.20.32-PM.png\")
<\/a>![\"Auto-logged](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/hyperparameters-anomalib.gif\")
<\/a>Project-level<\/h5>\n\n\n\n
![\"Project-level](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-03-at-7.59.35-PM.png\")
<\/a>![\"Project-level](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-03-at-8.02.16-PM.png\")
![\"Diffing](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/exp_diffing-anomalib.gif\")
Benchmarking<\/h4>\n\n\n\n
python tools\/benchmarking\/benchmark.py \\\n--config <relative-or-absolute-path>\/<paramfile>.yaml<\/pre>\n\n\n\n
Hyperparameter Optimization<\/h4>\n\n\n\n
![\"Comet](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/Screen-Shot-2022-10-03-at-7.54.56-PM.png\")
<\/a>Getting Started<\/h3>\n\n\n\n
0. Setup and Installation<\/h4>\n\n\n\n
git clone https:\/\/github.com\/openvinotoolkit\/anomalib.git<\/a>\ncd anomalib\npip install . --q<\/pre>\n\n\n\n
1. Configure Comet Credentials<\/h4>\n\n\n\n
export COMET_API_KEY = <Your-Comet-API-Key>\nexport COMET_PROJECT_NAME = <Your-Comet-Project-Name> # this will default to the name of your dataset<\/pre>\n\n\n\n
2. Modify the Anomalib config File<\/h4>\n\n\n\n
anomalib\/anomalib\/models\/<model-of-your-choice>\/config.yaml<\/code> and adjust the following parameters:<\/p>\n\n\n\nvisualization:\n show_images: true\n save_images: true\n log_images: true\n mode: full # options: [\"full\", \"simple\"]\nlogging:\n logger: comet\n log_graph: true<\/pre>\n\n\n\n
pyyaml<\/code> to write a config file in an interactive environment. Note<\/strong> that each model supported by Anomalib has a different config file structure.<\/p>\n\n\n\n3. Training<\/h4>\n\n\n\n
!python tools\/train.py<\/code> runs the PaDiM model on the bottle category from the MVTec AD (CC BY-NC SA 4.0) dataset.<\/p>\n\n\n\npython tools\/train.py \\\n--config anomalib\/models\/<specific-model-name>\/<config-file>.yaml<\/pre>\n\n\n\n
Beyond Model Training<\/h3>\n\n\n\n