tinyML Summit 2021 tiny Talks: Using Neural Architecture Search for Speech Recognition on the Edge
tinyML Summit 2021 https://www.tinyml.org/event/summit-2021
“Using Neural Architecture Search for Speech Recognition on the Edge”
Vikrant TOMAR, Founder and CTO, Fluent.ai
Despite recent developments in machine learning, finding an optimal solution for a given task remains a challenging and time-consuming task often requiring significant efforts in designing and tuning the neural architectures by an expert instead. This problem is more pronounced for TinyML solutions, where, due to limited computational resources, specific models are needed for a given task. To this end, we present a two-step solution. The first step employs GNASIL[2], a novel automated machine learning solution, for discovering an optimal neural architecture within a predefined limit of device specifications in FLOPS. The second step compresses the discovered architecture and make it even smaller.
GNASIL trains a soft actor-critic [2] reinforcement learning agent that expedites the discovery process by extending learning with planning options based upon past experiences and imitation learning through available expert-designed architectures on similar tasks. The architectures discovered by GNASIL are then compressed with automatic model compression (AMC)[3]. AMC uses DDPG [4] to learn the ratio of pruning for each layer. Reward is a function of accuracy and FLOPS. Optimal pruning is achieved in a way that has minimal effect on accuracy of the model despite often reducing the overall model footprint.
Our experiments on a series of on-device speech recognition tasks demonstrate that GNASIL can design neural models with competitive performance in terms of both discovery speed and the accuracy of the discovered architectures, all within the predefined FLOPS restrictions. Further, AMC is able to reduce the size of the model up to 40% without compromising accuracy.
References:
[1] Farzaneh S Fard, Arash Rad, Vikrant Singh Tomar. Nasil: Neural Architecture Search with Imitation Learning, ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
[2] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1856–1865, 2018.
[3] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 784–800, 2018.
[4] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
source
