Skip to content

Integrate with Anomalib

Anomalib provides a multitude of anomaly detection algorithms benchmarked and pretrained on the MVTec dataset.

Comet integrates directly with Anomalib tools/train.py --path to config script and automatically logs your hyperparameters, metrics and can be easily configured to log your model graphs and model predictions on your image data

Open In Colab

Start Logging

Setup the Anomalib repository

Clone the Anomalib project into your environment, and install the necessary dependencies.

git clone https://github.com/openvinotoolkit/anomalib # clone
cd anomalib
pip install -r requirements.txt # install

Configure Comet Credentials

There are two ways to configure Comet with Anomalib.

You can either set your credentials through environment variables

Environment Variables

export COMET_API_KEY=<Your Comet API Key>
export COMET_PROJECT_NAME=<Your Comet Project Name> # This will default to 'yolov5'

Or create a .comet.config file in your working directory and set your credentials there.

Comet Configuration File

[comet]
api_key=<Your Comet API Key>
project_name=<Your Comet Project Name> # This will default to 'yolov5'

Set Comet as your logger

Anomalib uses configuration files to set all the revelvant parameters for training. An example configuration file be found here. To use Comet for Experiment tracking with Anomalib, a simple modification to the logging section of the config file is needed as below:

logging:
  logger: [comet]

Run the Training Script

Now uses the modified configuration file to train your model:

# Train a Padim on the MvTech dataset
python tools/train.py --config anomalib/models/padim/config.yaml

Log automatically

By default, Comet will log the following items:

  • Metrics
    • The metrics defined in the Metrics section of the configuration file
  • Parameters
    • hyper-parameters defined in all the sections throughout the configuration

anom_charts.png

Log Visualizations

Anomalib allows users to visualize the model predictions on the image data. These images can be logged to Comet to view in UI. To enable this feature, one needs to make a slight change to the log_images parameter in the visualization section of the anomalib config files

visualization:
  log_images: True # log images to the available loggers (if any)

Users will see the model predictions in the Graphics tab of the single experiment view.

anom_graphics.png

With Comet, users can use the Image Panel to compare model predictions across experiments

anom_img_panel.png

Try it out!

Don't just take our word for it, try it out for yourself.

Hyperparameter Search with the Comet Optimizer

Anomalib is also integrated with Comet's Optimizer, making it simple to visualize hyperparameter sweeps in the Comet UI.

Configuring an Optimizer Sweep

To configure the Comet Optimizer, you will have to create a YAML file with the information about the sweep. An example file has been provided in tools/hpo/configs/comet.yaml.

To run the hyperparameter optimization, use the following command:

  python tools/hpo/sweep.py --model padim \
    --model_config ./path_to_config.yaml \
    --sweep_config tools/hpo/config/comet_sweep.yaml

In case model_config is not provided, the script looks at the default config location for that model.

  python tools/hpo/sweep.py --sweep_config tools/hpo/config/comet_sweep.yaml

Visualizing Hyperparameter Search Results

anom_hpo.png

Using the Anomalib Benchmarking Tool With Comet

Anomalib's benchmarking tool gathers results for different algorithims, parameters, and datasets. These results are stored in a csv file and can be logged to Comet by adding comet to the writer section of the benchmarking yaml file.

writer:
  - comet

Run the benchmarking tool with the following command

  python tools/benchmarking/benchmark.py --config <relative/absolute path>/<paramfile>.yaml
Nov. 27, 2022