(2021) Do winning tickets exist before DNN training?
Abstract
The recent lottery ticket hypothesis proposes that there is at least one sub-network that matches the accuracy of the original network when trained in isolation. Recent work shows that under SGD noise, several such tickets emerge. We build on these works and study how winning tickets derived from one fixed network differ in structural and functional terms under varying levels of stochasticity. Structurally, we show that the Hamming distance of winning tickets' shapes follow the hyper-geometric distribution. Functionally, our experiments validate that different emerging winning tickets are not disguised variants of each other, but diverge also concerning their classification outputs. Last but not least, different regimes of stochasticity affect winning tickets. Decreasing randomness during training also decreases the tickets' functional and structural distance.
| Item Type: | Conference or Workshop Item (A Paper) (Paper) |
|---|---|
| Divisions: | Michael Backes (InfSec) |
| Conference: | SDM SIAM International Conference on Data Mining |
| Depositing User: | Jonas Schneider-Bensch |
| Date Deposited: | 10 Aug 2021 11:08 |
| Last Modified: | 10 Aug 2021 11:08 |
| Primary Research Area: | NRA1: Trustworthy Information Processing |
| URI: | https://publications.cispa.saarland/id/eprint/3459 |
Actions
Actions (login required)
 |
View Item |