{"id":5740,"date":"2023-05-03T11:22:36","date_gmt":"2023-05-03T19:22:36","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=5740"},"modified":"2025-04-24T17:15:34","modified_gmt":"2025-04-24T17:15:34","slug":"compare-object-detection-models-from-torchvision","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/compare-object-detection-models-from-torchvision\/","title":{"rendered":"Compare Object Detection Models From TorchVision"},"content":{"rendered":"\n
\"Comparing
Comparing object detection predictions in Comet<\/a>; GIF by author<\/figcaption><\/figure>\n\n\n\n

Introduction<\/span><\/h2>\n\n\n\n

Object detection is one of the most popular applications of machine learning for computer vision. A detection model predicts both the class types and locations of each distinct object in an image. Object detection models have a wide range of applications including manufacturing, surveillance, health care, and more. <\/span><\/p>\n\n\n\n

TorchVision is a Python package that extends the PyTorch framework for computer vision use cases. In TorchVision\u2019s detection module, developers can find pre-trained object detection models that are ready to be fine-tuned on their own datasets. But how can you systematically find the best model for a particular use-case? Here, we’ll explore how to use an experiment tracking tool like Comet to visually compare and evaluate object detection models.<\/span><\/p>\n\n\n\n

Follow along with the full code in <\/span>this Colab<\/span><\/a> and check out <\/span>the public project here<\/span><\/a>!<\/span><\/p>\n\n\n\n

What is Object Detection?<\/span><\/h2>\n\n\n\n

Object detection is a computer vision task that aims to identify instances of objects in images and assign them to specific classes. At a low level, object detection seeks to answer the question, \u201cwhat objects are where?\u201d<\/span><\/p>\n\n\n\n

\"Object
Detecting sea animals; GIF by author<\/figcaption><\/figure>\n\n\n\n

Object detection algorithms are generally separated into two categories: single-stage (RetinaNet, SSD, FCOS, YOLO, etc.) and two-stage (Fast RCNN, Mask RCNN, FPN, etc.). In two-stage detectors, one model is used to extract generalized regions of objects, and a second model is used to classify and further refine the location of an object. Single-stage detectors do all of this in one step. Single-stage detectors tend to be faster and less computationally expensive than two-stage detectors, but they\u2019re also less accurate.<\/span><\/p>\n\n\n\n

\"Diagram
Image from Semantic Image Cropping<\/a><\/em>, by Oriol Corcoll Andreu<\/figcaption><\/figure>\n\n\n\n

Finetuning Pre-trained Models<\/h2>\n\n\n\n

The best object detection models are trained on tens, if not hundreds, of thousands of labeled images. What\u2019s more, image datasets themselves are inherently computationally expensive to process. To train an object detection model from scratch requires a lot of time and resources that aren\u2019t always available. To train several object detection models for comparison requires even more time and resources. Thankfully, we don\u2019t have to. Instead, we can use transfer learning or fine-tune pre-trained models.<\/span><\/p>\n\n\n\n

In essence, both these methods allow us to take advantage of the weights and biases learned from one task and repurpose them on a new task. By leveraging feature representations from a pre-trained model, we don\u2019t have to train a new model from scratch, saving us time and compute resources. What\u2019s more, these methods can contribute to rapid boosts in model performance for little overhead.<\/span><\/p>\n\n\n\n

\"Difference
Image from A graphic representation of the potential benefits of transfer learning<\/a><\/em>, by Laura Aelenei<\/figcaption><\/figure>\n\n\n\n

Transfer learning and fine-tuning are similar processes but with one key difference. In transfer learning, all previously trained layers are frozen, and (optionally) additional layers are added for retraining. In fine-tuning, all previously trained layers are retrained, but at a very low learning rate. Both methods typically result in boosted initial performance, steeper improvement slopes, and elevated final performance. <\/span><\/p>\n\n\n\n

TorchVision’s Pre-Trained Models<\/h3>\n\n\n\n

In real-world applications, we often make choices to balance accuracy and speed. The performance of a model under a given set of circumstances might not be relevant if we aren\u2019t able to replicate those circumstances in production. So when looking for the \u201cbest\u201d object detection model, it becomes essential to monitor a wide range of metrics pertaining your particular use case. In this tutorial, we\u2019ll show you how Comet helps us do this.<\/span><\/p>\n\n\n\n

TorchVision\u2019s detection module comes with several pre-trained models already built in. For this tutorial we will be comparing <\/span>Fast-RCNN<\/span><\/a>, <\/span>Faster-RCNN<\/span><\/a>, <\/span>Mask-RCNN<\/span><\/a>, <\/span>RetinaNet<\/span><\/a>, and <\/span>FCOS<\/span><\/a>, with either <\/span>ResNet50<\/span><\/a> of <\/span>MobileNet v2<\/span><\/a> backbones. Each of these models was previously trained on the <\/span>COCO dataset<\/span><\/a>. We will download the trained models, replace the classifier heads to reflect our target classes, and retrain the models on our own data.<\/span><\/p>\n\n\n\n

Image Data Formats<\/h2>\n\n\n\n

In computer vision, images are represented as matrices of pixel intensity values. Black and white (grayscale) images are usually two-dimensional, and color images are typically three-dimensional, with one \u201clayer\u201d each representing red, blue, and green pixels. <\/span><\/p>\n\n\n\n

Just as there are several ways to represent images, there are also several ways we can represent our labels and predictions. In the <\/span>full code for this tutorial<\/span><\/a>, we\u2019ll provide methods for logging bounding boxes, <\/span>segmentation masks<\/span><\/a>, and <\/span>polygon annotations<\/span><\/a> to Comet. But when comparing our TorchVision models we will only use bounding boxes, as not all of our models are able to calculate the other types of predictions. <\/span><\/p>\n\n\n\n

To further complicate things, not all algorithms format bounding box annotations in the same way. Below we\u2019ve listed a few of the most common bounding box formats you\u2019re likely to run into, but we\u2019ll be focusing on Pascal VOC and COCO formats in this tutorial.<\/span><\/p>\n\n\n\n

\"An
Image from Albumentations<\/a><\/figcaption><\/figure>\n\n\n\n

Pascal VOC:<\/b> [xmin, ymin, xmax, ymax] \u2192 [98, 345, 420, 462]<\/span><\/p>\n\n\n\n

Albumentations:<\/b> normalized([x_min, y_min, x_max, y_max]) \u2192 [0.153125, 0.71875, 0.65625, 0.9625]<\/span><\/p>\n\n\n\n

COCO:<\/b> [xmin, ymin, width, height] \u2192 [98, 345, 322, 117]<\/span><\/p>\n\n\n\n

YOLO:<\/b> normalized([x_center, y_center, width, height]) \u2192 [0.4046875, 0.8614583, 0.503125, 0.24375]<\/span><\/p>\n\n\n\n

Evaluation Metrics for Object Detection<\/span><\/h2>\n\n\n\n

A naive approach to evaluating object detection models might be binary classification (\u201cmatch\u201d or \u201cno match\u201d, \u201c1\u201d or \u201c0\u201d), but this method leaves little room for nuance. We can do better!<\/span><\/p>\n\n\n\n

Intersection Over Union (IOU)<\/h3>\n\n\n\n

The standard evaluation metric for comparing individual bounding boxes is Intersection over Union, or IoU. IoU evaluates the degree of overlap between the ground truth bounding box and the predicted bounding box with a value between 0 and 1. <\/span><\/p>\n\n\n\n

\"Image
Intersection over Union, image from Shivy Yohanandan in Towards Data Science<\/a><\/figcaption><\/figure>\n\n\n\n
\"Selecting
The difference between the intersection and union of a ground truth bounding box and a prediction bounding box; GIF by author<\/figcaption><\/figure>\n\n\n\n

IoU is determined for each set of bounding boxes in an image and then a threshold is applied. If an IoU meets the threshold, it\u2019s marked as a \u201ctrue positive.\u201d All predictions not marked as \u201ctrue positives\u201d are marked as \u201cfalse positives,\u201d and any items left in our \u201cground truth\u201d annotations list are marked as \u201cfalse negatives. The decision to mark a detection as TP, FP, or FN is completely contingent on the choice of IoU threshold. The IoU threshold is commonly set at 0.5, but you may want to experiment with this number. Once we\u2019ve calculated our confusion matrix, we can compute precision and recall.<\/span><\/p>\n\n\n\n

\"Image
True positives, false positives, and false negatives; note that we do not calculate true negatives in object detection. Image by author<\/figcaption><\/figure>\n\n\n\n

<\/h4>\n\n\n\n

Mean Average Precision (mAP) and Mean Average Recall (mAR)<\/h4>\n\n\n\n

Precision (also known as specificity) is the degree of exactness of the model in identifying only relevant objects. The equation for precision is:<\/span><\/p>\n\n\n\n

\"Formula<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Recall (also known as sensitivity) measures the ability of the model to detect all ground truths. The equation for recall is:<\/span><\/p>\n\n\n\n

\"Formula<\/figure>\n\n\n\n

<\/p>\n\n\n\n

In a perfect world, our perfect model would have a precision and recall of 1, meaning it predicted zero false negatives and zero false positives. But in the real world this isn\u2019t generally achievable. A precision-recall curve plots the value of precision against recall for different confidence thresholds. The area under this curve is also referred to as the Average <\/span>Precision (AP). Average recall (AR) describes double the value of the area under the recall-IoU curve.<\/span><\/p>\n\n\n\n

\"Chart
Average precision is equal to the area under the PR curve; image by author.<\/figcaption><\/figure>\n\n\n\n

Mean Average Precision (mAP) and mean Average Recall (mAR) are calculated by taking the weighted mean of the AP or AR over all classes and\/or over all IoU thresholds. They are two of the most common evaluation metrics for object detection and are used to evaluate submissions in popular computer vision competitions like the COCO and Pascal VOC challenges. We can derive many other metrics from mAP and mAR, including mAP across scales, at different IoU thresholds, and with a minimum number of detections per image. <\/span><\/p>\n\n\n\n

\"COCO's
The 12 metrics used for characterizing the performance of an object detector on COCO; image from COCO<\/figcaption><\/figure>\n\n\n\n

Other metrics<\/h4>\n\n\n\n

If we were building a model to detect very large objects (relative to the image\u2019s <\/span>field of view<\/span><\/a>), we might be willing to consider models with poor \u201cAP_small\u201d scores, as this metric would be less relevant to our use case. If we were planning on using our model to aid in medical diagnoses, we might place a higher emphasis on mAR values than mAP values, since it would likely be more important not to miss any positive samples than it would be to miss negative samples. <\/span><\/p>\n\n\n\n

For this tutorial we use a combination of the mAP and mAR values calculated by the <\/span>COCO evaluator<\/span><\/a> and <\/span>torchmetrics.detection<\/span><\/a> module. We\u2019ll log all relevant values to a DataFrame and then examine it more closely in the Comet Data Panel. This will help give us a full picture of how different models perform in different scenarios and we\u2019ll choose our \u201cbest\u201d model accordingly. <\/span><\/p>\n\n\n\n

Challenges of Comparing Object Detection Models<\/span><\/h2>\n\n\n\n

Comparing object detection models can be challenging for a number of reasons. Different models have different requirements when it comes to input size and shape, annotation formats, and other dataset attributes. Hyperparameters vary from algorithm to algorithm and keeping track of which values produce which results can quickly become tedious and overwhelming. <\/span><\/p>\n\n\n\n

Most computer vision pipelines incorporate image augmentation in some form, so dataset versioning becomes essential for reproducibility and explainability. What\u2019s more, performance metrics only tell part of the story when it comes to object detection. Often, it\u2019s necessary to visualize prediction annotations to understand where things are going right\u2013 and where they\u2019re going wrong. <\/span><\/p>\n\n\n\n

What\u2019s more, image datasets themselves are inherently computationally expensive to process. To train an object detection model from scratch requires a lot of time and resources that aren\u2019t always available. To train several object detection models for comparison requires even more time and resources.<\/span><\/p>\n\n\n\n

\"Idealized
Idealized accuracy-speed tradeoff has a logarithmic relationship; image by Jeff Miller on ResearchGate<\/a><\/figcaption><\/figure>\n\n\n\n

In real-world applications, we often make choices to balance accuracy and speed. The performance of a model under a given set of circumstances might not be relevant if we aren\u2019t able to replicate those circumstances in production. So when looking for the \u201cbest\u201d object detection model, it becomes essential to monitor a wide range of metrics pertaining your particular use case. In this tutorial, we\u2019ll show you how Comet helps us do this.<\/span><\/p>\n\n\n\n

Using Comet for Object Detection<\/span><\/h2>\n\n\n\n

Clearly, comparing object detection models isn\u2019t nearly so simple as just minimizing a single loss function. We have a pretty wide range of metrics to calculate and log, some of which need to be visualized to completely understand, and each model has it\u2019s own graph definition, set of hyperparameters, code output, and other features. To help keep track of all of these moving pieces, we\u2019ll log our inputs, metrics, and outputs to Comet, a experiment tracking tool. Comet has some pretty extensive auto-logging capabilities, but we\u2019ll also explore how to log custom metrics and outputs to Comet. By visualizing all of our data in the Comet UI, we\u2019ll be able to get a much more complete understanding of how each of our models behaves, under which circumstances, and with which data.<\/span><\/p>\n\n\n\n

For this tutorial, we\u2019ll be using the <\/span>Penn-Fudan dataset<\/span><\/a>, which consists of 170 images labeled with 345 instances of pedestrians. Pedestrian detection has several applications, including surveillance, training self-driving cars, and other traffic safety applications. Since we\u2019re using PyTorch, we\u2019ll need to define a custom dataset class that inherits from the <\/span>torch.utils.data.Dataset<\/span><\/a> class.<\/span><\/p>\n\n\n\n

\"Example
Example image from the PennFudan dataset with bounding box and mask labels; GIF by author<\/figcaption><\/figure>\n\n\n\n

All of the models in TorchVision\u2019s detection module use Pascal VOC format, so we\u2019ll format our bounding boxes accordingly. We\u2019ll then need to convert the model\u2019s prediction labels from Pascal VOC to COCO format for use with the COCO evaluator and Comet. If you\u2019re using this tutorial with your own model, check your specific model\u2019s annotation requirements. <\/span><\/p>\n\n\n\n

Single Experiment View<\/span><\/h3>\n\n\n\n

In order to get a good understanding of how each of our models is performing, and with which hyperparameters, we\u2019ll start by examining our results at an experiment-level. <\/span><\/p>\n\n\n\n

Store Hyper-parameters<\/span><\/h4>\n\n\n\n

Keeping track of our hyperparameters is essential for reproducibility and explainability. Model hyperparameters can affect model performance, computational choices and what information to retain for analysis. Hyperparameters vary from algorithm to algorithm, and some are more important than others, so this critical task can quickly become tedious and confusing. <\/span><\/p>\n\n\n\n

Here, we log important hyperparameters with just a single command. For our project we\u2019ll be monitoring the following hyperaparameters, which we can adjust by simply editing the relevant keys-value pairs:<\/span><\/p>\n\n\n\n