Integrate with Anomalib¶
Anomalib provides a multitude of anomaly detection algorithms benchmarked and pretrained on the MVTec dataset.
Comet integrates directly with Anomalib
tools/train.py --path to config script and automatically logs your hyperparameters, metrics and can be easily configured to log your model graphs and model predictions on your image data
Setup the Anomalib repository¶
Clone the Anomalib project into your environment, and install the necessary dependencies.
git clone https://github.com/openvinotoolkit/anomalib # clone cd anomalib pip install -r requirements.txt # install
Configure Comet Credentials¶
There are two ways to configure Comet with Anomalib.
You can either set your credentials through environment variables
export COMET_API_KEY=<Your Comet API Key> export COMET_PROJECT_NAME=<Your Comet Project Name> # This will default to 'yolov5'
Or create a
.comet.config file in your working directory and set your credentials there.
Comet Configuration File
[comet] api_key=<Your Comet API Key> project_name=<Your Comet Project Name> # This will default to 'yolov5'
Set Comet as your logger¶
Anomalib uses configuration files to set all the revelvant parameters for training. An example configuration file be found here. To use Comet for Experiment tracking with Anomalib, a simple modification to the
logging section of the config file is needed as below:
logging: logger: [comet]
Run the Training Script¶
Now uses the modified configuration file to train your model:
# Train a Padim on the MvTech dataset python tools/train.py --config anomalib/models/padim/config.yaml
By default, Comet will log the following items:
- The metrics defined in the Metrics section of the configuration file
- hyper-parameters defined in all the sections throughout the configuration
Anomalib allows users to visualize the model predictions on the image data. These images can be logged to Comet to view in UI. To enable this feature, one needs to make a slight change to the
log_images parameter in the
visualization section of the anomalib config files
visualization: log_images: True # log images to the available loggers (if any)
Users will see the model predictions in the Graphics tab of the single experiment view.
With Comet, users can use the Image Panel to compare model predictions across experiments
Try it out!¶
Don't just take our word for it, try it out for yourself.
- Run an example in Colab:
- If you can't wait, check out this Example Comet Project.
Hyperparameter Search with the Comet Optimizer¶
Anomalib is also integrated with Comet's Optimizer, making it simple to visualize hyperparameter sweeps in the Comet UI.
Configuring an Optimizer Sweep¶
To configure the Comet Optimizer, you will have to create a YAML file with the information about the sweep. An example file has been provided in
To run the hyperparameter optimization, use the following command:
python tools/hpo/sweep.py --model padim \ --model_config ./path_to_config.yaml \ --sweep_config tools/hpo/config/comet_sweep.yaml
model_config is not provided, the script looks at the default config location for that model.
python tools/hpo/sweep.py --sweep_config tools/hpo/config/comet_sweep.yaml
Visualizing Hyperparameter Search Results¶
Using the Anomalib Benchmarking Tool With Comet¶
Anomalib's benchmarking tool gathers results for different algorithims, parameters, and datasets. These results are stored in a csv file and can be logged to Comet by adding
comet to the
writer section of the benchmarking yaml file.
writer: - comet
Run the benchmarking tool with the following command
python tools/benchmarking/benchmark.py --config <relative/absolute path>/<paramfile>.yaml