tinyML Asia 2021
Session: Frameworks, Tools, tinyML for Good
Tiny ONNC: unleashes your IoT device intelligent power
Peter CHANG, Co-founder and Technical Marketing Manager, Skymizer Taiwan Inc.
While AI has possessed the advantage on servers, there are still huge rooms on IoT devices. But, for IoT developers, the hardware limitation of IoT devices may hinder the IoT devices’ potential intelligent power.
Therefore, Skymizer introduces Tiny ONNC, an easy and efficient AI compiler. Tiny ONNC leverages the unique power of MLIR to support rich neural network frameworks, including PyTorch, Open Neural Network Exchange Format (ONNX), Tensorflow, Tensorflow Lite, TVM relay, and even Keras.
Tiny ONNC has abundant optimization approaches, such as automatic operator splitting and tensor splitting, addressing on memory constraints of microcontrollers. When an operator or a tensor is too big to fit in the cache, Tiny ONNC separates the big objects into small pieces and reorganizes the networks for reusing the memory. Tiny ONNC also supports operators which are not directly supported by CMSIS-NN by mathematical equivalent or approximate transformations.
These optimization approaches deliver strong empirical performances while keeping high memory utilization and high performance at the same time. On the MLPerf Tiny benchmark, Tiny ONNC achieves the same level (2%) as TensorflowLite for Microcontrollers (TFLM) in terms of performance and precision. Under similar performance and precision, the memory footprint of the generated program is only 3/5 of TFLM and its code size is only 1/10 of TFLM in the best case. Moreover, when using aggressive optimizations, The code generated by Tiny ONNC could is up to 4.9 times faster than the code generated by TFLM.
In this talk, firstly, we will introduce Tiny ONNC and how to use Tiny ONNC. Secondly, we will dive into our optimization strategies and approaches. Finally, we will explain the experiment results to see how Tiny ONNC outperforms other competitors.