(2022) Plant 'n' Seek: Can You Find the Winning Ticket?
|
Text
Plant_n_Seek_Can_You_Find_the_Winning_Ticket.pdf Download (893kB) | Preview |
Abstract
The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that perform structure learning by identifying a sparse subnetwork of a large randomly initialized neural network. The existence of such 'winning tickets' has been proven theoretically but at suboptimal sparsity levels. Contemporary pruning algorithms have furthermore been struggling to identify sparse lottery tickets for complex learning tasks. Is this suboptimal sparsity merely an artifact of existence proofs and algorithms or a general limitation of the pruning approach? And, if very sparse tickets exist, are current algorithms able to find them or are further improvements needed to achieve effective network compression? To answer these questions systematically, we derive a framework to plant and hide target architectures within large randomly initialized neural networks. For three common challenges in machine learning, we hand-craft extremely sparse network topologies, plant them in large neural networks, and evaluate state-of-the-art lottery ticket pruning methods. We find that current limitations of pruning algorithms to identify extremely sparse tickets are likely of algorithmic rather than fundamental nature and anticipate that our planting framework will facilitate future developments of efficient pruning algorithms, as we have addressed the issue of missing baselines in the field raised by Frankle et al.
| Item Type: | Conference or Workshop Item (A Paper) (Paper) |
|---|---|
| Uncontrolled Keywords: | deep learning, lottery tickets, planted solutions, benchmarks |
| Divisions: | Rebekka Burkholz (RB) |
| Conference: | ICLR International Conference on Learning Representations |
| Depositing User: | Rebekka Burkholz |
| Date Deposited: | 06 Apr 2022 11:07 |
| Last Modified: | 13 Oct 2022 10:12 |
| Primary Research Area: | NRA1: Trustworthy Information Processing |
| URI: | https://publications.cispa.saarland/id/eprint/3597 |
Actions
Actions (login required)
 |
View Item |