Integrate with Anomalib¶

Anomalib provides a multitude of anomaly detection algorithms benchmarked and pretrained on the MVTec dataset.

Comet integrates directly with Anomalib tools/train.py --path to config script and automatically logs your hyperparameters, metrics and can be easily configured to log your model graphs and model predictions on your image data

Start Logging¶

Setup the Anomalib repository¶

Clone the Anomalib project into your environment, and install the necessary dependencies.

git clone https://github.com/openvinotoolkit/anomalib # clone
cd anomalib
pip install -r requirements.txt # install

Configure Comet Credentials¶

There are two ways to configure Comet with Anomalib.

You can either set your credentials through environment variables

Environment Variables

export COMET_API_KEY=<Your Comet API Key>
export COMET_PROJECT_NAME=<Your Comet Project Name>

Or create a .comet.config file in your working directory and set your credentials there.

Comet Configuration File

[comet]
api_key=<Your Comet API Key>
project_name=<Your Comet Project Name>

Set Comet as your logger¶

Anomalib uses configuration files to set all the revelvant parameters for training. To use Comet for Experiment tracking with Anomalib, a simple modification to the logging section of the config file is needed as below:

logging:
  logger: [comet]

Run the Training Script¶

Now uses the modified configuration file to train your model:

# Train a Padim on the MvTech dataset
python tools/train.py --config src/anomalib/models/padim/config.yaml

Log automatically¶

By default, Comet will log the following items:

• Metrics
  • The metrics defined in the Metrics section of the configuration file
• Parameters
  • hyper-parameters defined in all the sections throughout the configuration

Log Visualizations¶

Anomalib allows users to visualize the model predictions on the image data. These images can be logged to Comet to view in UI. To enable this feature, one needs to make a slight change to the log_images parameter in the visualization section of the anomalib config files

visualization:
  log_images: True # log images to the available loggers (if any)

Users will see the model predictions in the Graphics tab of the single experiment view.

With Comet, users can use the Image Panel to compare model predictions across experiments

Try it out!¶

Don't just take our word for it, try it out for yourself.

• Run an example in Colab:
• If you can't wait, check out this Example Comet Project.

Hyperparameter Search with the Comet Optimizer¶

Anomalib is also integrated with Comet's Optimizer, making it simple to visualize hyperparameter sweeps in the Comet UI.

Configuring an Optimizer Sweep¶

To configure the Comet Optimizer, you will have to create a YAML file with the information about the sweep. An example file has been provided in tools/hpo/configs/comet.yaml.

To run the hyperparameter optimization, use the following command:

  python tools/hpo/sweep.py --model padim \
    --model_config ./path_to_config.yaml \
    --sweep_config tools/hpo/config/comet_sweep.yaml

In case model_config is not provided, the script looks at the default config location for that model.

  python tools/hpo/sweep.py --sweep_config tools/hpo/config/comet_sweep.yaml

Visualizing Hyperparameter Search Results¶

Using the Anomalib Benchmarking Tool With Comet¶

Anomalib's benchmarking tool gathers results for different algorithims, parameters, and datasets. These results are stored in a csv file and can be logged to Comet by adding comet to the writer section of the benchmarking yaml file.

writer:
  - comet

Run the benchmarking tool with the following command

  python tools/benchmarking/benchmark.py --config <relative/absolute path>/<paramfile>.yaml