In this post, we will showcase a Custom Panel dedicated to debugging object detection models.<\/p>\n\n\n\n
We will use the Penn-Fudan Pedestrian Detection dataset, along with a Faster-RCNN model, with a Resnet50 backbone, that has been pre-trained on the MS COCO<\/a> dataset.<\/p>\n\n\n\n All code for this post can be found in this Colab Notebook<\/a>. The logged experiments can be found here<\/a>.<\/p>\n\n\n\n Let’s start by downloading the data<\/p>\n\n\n\n Now let’s define a dataset class that will allow us to easily load the images in the dataset<\/p>\n\n\n\n Next, we’ll define some basic preprocessing operations that will turn our image data into Torch Tensors:<\/p>\n\n\n\n Finally, lets load in our model and start making predictions. Every time we make a prediction, we’re going to log the image to Comet. You will find these images under the Graphics tab of the experiment.<\/p>\n\n\n\n In order to use the Bounding Box panel, we will need to log our predictions in a particular format as a JSON file. The following snippet will convert a list of image ids, predictions, and labels into the required JSON format for plotting.<\/p>\n\n\n\n Lets format our predictions, and log them to Comet:<\/p>\n\n\n\n You can find the ‘Bounding Boxes’ custom panel in the Public Panels Galley. Once added, it becomes easy to view predictions from the model, as well as filter predictions out based on the confidence scores.<\/p>\n\n\n\n You can explore this Panel below<\/p>\n\n\n\nLoading the Data and Setting up the Model<\/h2>\n\n\n\n
# download the Penn-Fudan dataset\n!wget https:\/\/www.cis.upenn.edu\/~jshi\/ped_html\/PennFudanPed.zip .\n# extract it in the current folder\n!unzip PennFudanPed.zip<\/code><\/pre>\n\n\n\nimport os\nimport numpy as np\nimport torch\nimport torch.utils.data\nfrom PIL import Image\n\nclass PennFudanDataset(torch.utils.data.Dataset):\n def __init__(self, root, transforms=None):\n self.root = root\n self.transforms = transforms\n # load all image files, sorting them to\n # ensure that they are aligned\n self.imgs = list(sorted(os.listdir(os.path.join(root, \"PNGImages\"))))\n self.masks = list(sorted(os.listdir(os.path.join(root, \"PedMasks\"))))\n\n def __getitem__(self, idx):\n # load images ad masks\n img_path = os.path.join(self.root, \"PNGImages\", self.imgs[idx])\n mask_path = os.path.join(self.root, \"PedMasks\", self.masks[idx])\n img = Image.open(img_path).convert(\"RGB\")\n # note that we haven't converted the mask to RGB,\n # because each color corresponds to a different instance\n # with 0 being background\n mask = Image.open(mask_path)\n\n mask = np.array(mask)\n # instances are encoded as different colors\n obj_ids = np.unique(mask)\n # first id is the background, so remove it\n obj_ids = obj_ids[1:]\n\n # split the color-encoded mask into a set\n # of binary masks\n masks = mask == obj_ids[:, None, None]\n\n # get bounding box coordinates for each mask\n num_objs = len(obj_ids)\n boxes = []\n for i in range(num_objs):\n pos = np.where(masks[i])\n xmin = np.min(pos[1])\n xmax = np.max(pos[1])\n ymin = np.min(pos[0])\n ymax = np.max(pos[0])\n boxes.append([xmin, ymin, xmax, ymax])\n\n boxes = torch.as_tensor(boxes, dtype=torch.float32)\n # there is only one class\n labels = torch.ones((num_objs,), dtype=torch.int64)\n masks = torch.as_tensor(masks, dtype=torch.uint8)\n\n image_id = torch.tensor([idx])\n area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])\n # suppose all instances are not crowd\n iscrowd = torch.zeros((num_objs,), dtype=torch.int64)\n\n target = {}\n target[\"boxes\"] = boxes\n target[\"labels\"] = labels\n target[\"masks\"] = masks\n target[\"image_id\"] = image_id\n target[\"area\"] = area\n target[\"iscrowd\"] = iscrowd\n\n if self.transforms is not None:\n img = self.transforms(img)\n\n return img, target\n\n def __len__(self):\n return len(self.imgs)<\/code><\/pre>\n\n\n\nimport torch, torchvision\nfrom torchvision import datasets, transforms\n\npreprocess = transforms.Compose([\n transforms.ToTensor(),\n])\ndataset = PennFudanDataset('.\/PennFudanPed', transforms=preprocess)<\/code><\/pre>\n\n\n\nmodel = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)\nmodel.eval()\n\nimg_ids = [i for i in range(5)]\nlabels = []\npredictions = []\n\nfor img_id in img_ids:\n img, label = dataset[img_id]\n\n labels.append(label)\n with torch.no_grad():\n prediction = model([img])\n predictions.append(prediction[0])\n\n experiment.log_image(image_data=transforms.ToPILImage()(img), name=str(img_id))<\/code><\/pre>\n\n\n\nLogging Bounding Box Predictions<\/h2>\n\n\n\n
def format_predictions_and_labels(img_ids, predictions, labels, label_map):\n data = {}\n\n for idx, img_id in enumerate(img_ids):\n prediction = predictions[idx]\n label = labels[idx]\n\n predicted_boxes = prediction[\"boxes\"].numpy().tolist()\n predicted_scores = prediction[\"scores\"].numpy().tolist()\n predicted_classes = prediction[\"labels\"].numpy().tolist()\n\n label_boxes = label[\"boxes\"].numpy().tolist()\n\n data.setdefault(img_id, [])\n for label_box in label_boxes:\n x, y, x2, y2 = label_box\n data[img_id].append({\n \"label\": \"ground-truth\",\n \"score\": 100,\n \"box\": {\"x\": x, \"y\": y, \"x2\": x2, \"y2\": y2},\n })\n\n for predicted_box, predicted_score, predicted_class in zip(predicted_boxes, predicted_scores, predicted_classes):\n x, y, x2, y2 = predicted_box\n data[img_id].append({\n \"label\": label_map[predicted_class - 1],\n \"box\": {\"x\": x, \"y\": y, \"x2\": x2, \"y2\": y2},\n \"score\": predicted_score * 100\n })\n\n return data<\/code><\/pre>\n\n\n\nmetadata = format_predictions_and_labels(img_ids, predictions, labels)\nexperiment.log_asset_data(data, \"image-metadata.json\")<\/code><\/pre>\n\n\n\nViewing Model Predictions in the Custom Panel<\/h2>\n\n\n\n