(2022) On the Existence of Universal Lottery Tickets.
|
Text
On_the_existence_of_universal_LTs.pdf Download (976kB) | Preview |
Abstract
The lottery ticket hypothesis conjectures the existence of sparse subnetworks of large randomly initialized deep neural networks that can be successfully trained in isolation. Recent work has experimentally observed that some of these tickets can be practically reused across a variety of tasks, hinting at some form of universality. We formalize this concept and theoretically prove that not only do such universal tickets exist but they also do not require further training. Our proofs introduce a couple of technical innovations related to pruning for strong lottery tickets, including extensions of subset sum results and a strategy to leverage higher amounts of depth. Our explicit sparse constructions of universal function families might be of independent interest, as they highlight representational benefits induced by univariate convolutional architectures.
| Item Type: | Conference or Workshop Item (A Paper) (Paper) |
|---|---|
| Uncontrolled Keywords: | deep learning, universal lottery tickets, existence proofs, sparse representation, polynomials |
| Divisions: | Rebekka Burkholz (RB) |
| Conference: | ICLR International Conference on Learning Representations |
| Depositing User: | Rebekka Burkholz |
| Date Deposited: | 06 Apr 2022 11:13 |
| Last Modified: | 13 Oct 2022 10:10 |
| Primary Research Area: | NRA1: Trustworthy Information Processing |
| URI: | https://publications.cispa.saarland/id/eprint/3603 |
Actions
Actions (login required)
 |
View Item |