![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:1000\/1*-ajPIIkeIBzMY-050e96kA.jpeg\")
<\/figure><\/p>\n<\/div>\n<\/div>\n
Computer vision, at its core, is about understanding images. The field has seen rapid growth over the last few years, especially due to deep learning and the ability to detect obstacles, segment images, or extract relevant context from a given scene.<\/p>\n
Using computer vision, we can build autonomous cars, smart buildings, fashion recommender systems, augmented reality tools\u2026the possibilities are endless.<\/p>\n
One area in particular is starting to garner more attention: Video<\/em><\/strong>. Most applications of computer vision today center on images, with less focused on sequences of images (i.e. video frames).<\/p>\n Video allows for deeper situational understanding, because sequences of images provide new information about action<\/em>. For example, we can track an obstacle through a sequence of images and understand its behavior to predict the next move. We can track a human pose, and understand the action taken with action classification.<\/p>\n When analyzing videos, we create new use cases and move from \u201cthis image contains 3 people\u201d to \u201cthis images contains 3 people playing X\u201d.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Whether for surveillance camera systems or football analysis, the next generation of computer vision algorithms will include time.<\/p>\n The task of video surveillance involves two kind of algorithms:<\/p>\n Let\u2019s have a look at both. At the end of this article, you\u2019ll have a more complete picture of video analysis systems.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n A video is a set of frames. When studying a video, we can either study a video stream<\/strong> (live image feed) or a video sequence<\/strong> (fixed-length video).<\/p>\n Videos take up a lot of storage space and are usually not already using AI. This means that, with video, we simply have raw image data to work with.<\/p>\n But there is a key difference. Specifically, motion. <\/strong>Motion is the only difference between an image and a video. It\u2019s a powerful thing to track and can lead to action understanding, pose estimation, or movement tracking.<\/p>\n In video analysis, this key problem is called optical flow estimation. <\/strong>Optical flow is the idea of computing a pixel shift between two frames. This is handled as a correspondence problem, as illustrated in the following image:<\/p>\n The output optical flow is a vector of movement between frame 1 and frame 2. It looks like this:<\/p>\n A lot of existing datasets address the optical flow problem, such as KITTI Vision Benchmark Suite<\/a> or MPI Sintel<\/a>. They both contain ground truth optical flow data, which is generally hard to get from more popular datasets.<\/p>\n To solve the optical flow problem, convolutional neural networks can help. The input of the network is a set of two RGB images; thus it has a depth of 6.<\/p>\n Optical flow is often represented by colors.<\/p>\n The first problem we want to solve is understanding the movement of pixels from one frame to another. Optical flow estimations can be done in a video stream or a video sequence. A classification of the output vectors can then be inferred to understand movement.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n First, we can simply track objects visual Object Tracking (VOT)<\/strong> is the science of tracking an object given its position in frame 1. We are not using a detection algorithm here\u2014thus, we\u2019re model free. In other words, we don\u2019t know what we are tracking. We are simply given a starter bounding box and are asked to keep track of this object all along.<\/p>\n Tracking is performed by computing the similarity between frames 0 and 1.We check what\u2019s in the bounding box and try to retrieve it in the next frame. Other features, such as color, can also be used to track the objects. Here, we compute the color of the given object and then compute the background that represents is the closest color to the object. Then we remove it from our original image to track it.<\/p>\n This is very powerful technique, and it only uses computer vision. We don\u2019t need a single neural network to do this. To summarize this process:<\/p>\nVideo Analysis Algorithms<\/h1>\n
<\/figure>
Obstacle tracking & video analysis \u2014 An active area of research<\/strong><\/h2>\n
\n
1. Object Tracking<\/h1>\n
\n
Optical Flow<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*HdSZAckkzQchDz9BEkcAlg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:480\/1*YuhBcSo1PMqApCNfaEDXxA.gif\")
<\/figure>
\nFlowNet<\/a> is an example of a CNN designed for optical flow tasks, and it can output the optical flow from two frames.<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*xxmptxywxZ2_xFmZ.png\")
<\/figure>Visual Object Tracking (VOT)<\/h2>\n
\nWe can then move the bounding box a bit and track our obstacle.<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*CNDNAd1GRaA_uBAC.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*lzDFT1YHYIixAHeO.png\")
<\/figure>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*VjLXLedY0OQAdvlR.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*8O4aU3RHVqx7lgzi.png\")
<\/figure>