tinyML EMEA 2022 – Mina Khoei: Low power, low latency multi-object tracking and classification…

tinyML EMEA 2022
Full Stack Solutions session:
Low power, low latency multi-object tracking and classification using a fully event-driven neuromorphic pipeline
Mina KHOEI, Senior Algorithm and ML applications engineer, SynSense AG

Detection, tracking and classification of objects in real-time from a video is a very important but costly problem in computer vision. Typically, this is achieved by running a window based CNN and sliding it over the full image to obtain the areas of interest (i.e. location of the object) and then identifying it. This approach requires the image to be stationary such that the model can be run over the entire object over multiple passes. A second approach involves a one shot one pass model such as YOLO, where the entire image is passed and the localization and identification is done simultaneously. While this approach is better for real-time systems compared to the former due to its requirements of multiple passes, they typically require a lot of memory and thus consume a lot of power to achieve this. Furthermore, previous work along these lines also demonstrated that it is challenging to achieve good accuracy using spiking convolutional neural networks for such a model. The problem becomes more costly to solve due to the state-holding nature of spiking neurons and the requirement of using memory to store those states.

Locating and identifying the objects are two inherently different problems and if they are solved separately this might reduce the network size significantly. This is partially achieved in the field of neuromorphic engineering by the use of dynamic vision sensors, which only pass the information on the change in luminosity as events for individual pixels with corresponding timestamps rather than passing an entire frame. This eliminates the background which is unrelated for the objects to be tracked. These events can easily be clustered into separate clusters by their spatial locations. In this work, we implement such a clustering algorithm to track the objects, preprocess it such that each cluster has the same output size and then pass these events to a spiking neural network for identification. The pipeline is implemented and tested for tracking of single and multiple objects, as well as identification of these objects on novel DynapCNN™ asynchronous spiking convolutional neural network processor.

source