{"id":5826,"date":"2023-05-13T08:14:34","date_gmt":"2023-05-13T16:14:34","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=5826"},"modified":"2025-04-29T14:05:22","modified_gmt":"2025-04-29T14:05:22","slug":"debugging-image-classifiers-with-confusion-matrices","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/debugging-image-classifiers-with-confusion-matrices\/","title":{"rendered":"Debugging Image Classifiers With Confusion Matrices"},"content":{"rendered":"\n
\"Interactive
Interactive confusion matrices per epoch<\/a> of our image classification model, as seen in the Comet UI; image by author<\/figcaption><\/figure>\n\n\n\n

Introduction<\/h2>\n\n\n\n

We often rely on scalar metrics and static plots to describe and evaluate machine learning models, but these methods rarely capture the full story. Especially when dealing with computer vision tasks like classification, detection, segmentation, and generation, visualizing your outputs is essential to understanding how your model is behaving and why. <\/span><\/p>\n\n\n\n

\n
Follow along with the Colab!<\/span><\/a><\/div>\n<\/div>\n\n\n\n

We may notice that a model has a particularly low precision or recall value, but an individual statistic doesn\u2019t give us any insight into which categories of data our model is struggling with the most, or how we might augment our training data for better results. As another example, bounding box coordinates mean little to us when presented as a list of integers or floats. But when these same numbers are overlaid as a patch on an image, we can immediately recognize whether a model has accurately detected an object or not. Especially when working with image data, it\u2019s often much quicker and easier to spot patterns in information that is presented to us visually.<\/span><\/p>\n\n\n\n

\"Screenshot
Scalar metrics help us benchmark different experiment runs against each other, but they provide limited information to help us debug image classification models. We see here that the epoch mAR is lower than the epoch mAP<\/a>, but did the model struggle more with confusing penguins as turtles or vice-versa?<\/figcaption><\/figure>\n\n\n\n

Confusion Matrix<\/h2>\n\n\n\n

A confusion matrix is a popular way to inspect the performance of a classification model. It combines multiple metrics into a single table to summarize a model\u2019s behavior across different classes. Typically, actual categories are plotted against a model\u2019s predicted categories, as shown below:<\/span><\/p>\n\n\n\n

\"Confusion
Confusion matrix of a Fast-RCNN model\u2019s predictions<\/a>; GIF by author<\/figcaption><\/figure>\n\n\n\n

And while this plot is helpful in illustrating a given model\u2019s \u201cconfusion\u201d between categories, it only tells part of the story. Are there any patterns in the images the model is struggling with? Maybe the model tends to get confused when it sees a particular breed of one of the animals. Or maybe different backgrounds are influencing its decisions. We really can\u2019t be sure without visualizing exactly what the model predicted, and on which images.<\/span><\/p>\n\n\n\n

In this article, we\u2019ll explore how to use Comet\u2019s interactive confusion matrix for a multi-class image classification task. Follow along with the full code in <\/span>this Colab tutorial<\/span><\/a>, and make sure to check out <\/span>the public project here<\/span><\/a>!<\/span><\/p>\n\n\n\n

Note that to run these experiments, you’ll need to have your Comet API key configured. If you don\u2019t already have an account, <\/span>create one here for free<\/span><\/a>.<\/span><\/p>\n\n\n\n

Our Data<\/h2>\n\n\n\n

For this tutorial, we\u2019ll be using a dataset of 572 images of penguins and turtles.The training set contains 500 images, and the validation set contains 72 images, both of which are split evenly between classes. Each image contains exactly one instance of an object, and since being a penguin, being a turtle, and being the background are all mutually exclusive, this is a multi-class, but <\/span>not <\/span><\/i>a <\/span>multi-label classification<\/span><\/a> task. Download the full dataset on <\/span>Kaggle here<\/span><\/a> and follow along with <\/span>the code here<\/span><\/a>.<\/span><\/p>\n\n\n\n

\"Example
Example images from our dataset<\/a>; graphic by author<\/figcaption><\/figure>\n\n\n\n

Once we\u2019ve downloaded our dataset, we\u2019ll need to define a <\/span>custom PyTorch Dataset<\/span><\/a> class to properly load and preprocess our images before feeding them to our model. We\u2019ll also define a label dictionary to convert our categorical labels into numerical ones. Note that by default, our models treat \u201c0\u201d as the background class. <\/span><\/p>\n\n\n\n