tinyML Talks Atlas Wang: The lottery ticket hypothesis for gigantic pre-trained models

tinyML Talks – recorded March 30, 2021
“The lottery ticket hypothesis for gigantic pre-trained models”
Atlas Wang – UT Austin

In NLP and computer vision, enormous pre-trained models have become the standard starting point for training on a range of downstream tasks. In parallel, work on the lottery ticket hypothesis has shown that models contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. We have combined these observations to assess whether such trainable, transferrable subnetworks exist in various pre-trained models. Taking BERT as one example, for a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. As another example found in computer vision, from all pre-trained weights obtained by ImageNet classification, simCLR and MoCo, we are also consistently able to locate matching subnetworks at 59.04% to 96.48% sparsity that transfer to multiple downstream tasks, whose performance also see no degradation compared to using full pre-trained weights. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context.


3 thoughts on “tinyML Talks Atlas Wang: The lottery ticket hypothesis for gigantic pre-trained models

  • October 4, 2022 at 2:57 pm

    My brother suggested I might like this web site. He was totally right.

    This post actually made my day. You cann’t imagine simply
    how much time I had spent for this information! Thanks!

  • October 8, 2022 at 3:49 am

    This is very fascinating, You are an overly skilled blogger.
    I have joined your rss feed and stay up for in quest of more of your
    great post. Also, I’ve shared your site in my social networks

  • October 13, 2022 at 4:23 am

    It’s an awesome article in support of all the web visitors;
    they will get benefit from it I am sure.


Leave a Reply

Your email address will not be published. Required fields are marked *