EMEA 2021 https://www.tinyml.org/event/emea-2021
TinyMLPerf: Development of a Benchmark Suite for TinyML System
Csaba KIRALY, Internet of Things Engineer, Digital Catapult UK
Tiny machine learning (tinyML) is driving enormous growth within the IoT industry, enabling data driven development and previously unseen levels of machine intelligence and autonomy of operation at the far edge.
Evaluating the performance of low-power solutions in such a fastly evolving space is already difficult given the large design space offering various performance-energy tradeoffs even for a single application. Providing benchmarks that allow the comparison of different solutions is even more challenging due to the wide range of targeted applications, power budgets, model specific optimizations, innovative HW and SW designs, and toolchains. Yet, to foster innovation, it is necessary to provide a benchmark that is fair, replicable, robust and enjoys the support of the wider community; a global community in which several EMEA players also contributed to the development of the first version of such a benchmark.
This work presents the first version of tinyMLPerf, a suite of benchmarks developed by the tinyML community to be used to compare tinyML hardware and software systems. The talk gives an insight into the development process behind the benchmark suite, describing the benchmark selection process, some of the design choices made, and the benchmarks selected for this first iteration consisting of four ML tasks: small vocabulary keyword spotting, binary image classification, small image classification, and anomaly detection using machine operating sounds. It will present the benchmark framework developed in a collaboration of MLCommons and EEMBC, the development of reference implementations on an ST platform to help submitters, the use of the benchmark to evaluate some performance-energy tradeoffs of a single solution, and some of the lessons learned during the process.