What’s in the Box? Exploring the Inner Life of Neural Networks with Robust Rules

Jonas Fischer, Anna Olah, Jilles Vreeken
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:3352-3362, 2021.

Abstract

We propose a novel method for exploring how neurons within neural networks interact. In particular, we consider activation values of a network for given data, and propose to mine noise-robust rules of the form X {\rightarrow} Y , where X and Y are sets of neurons in different layers. We identify the best set of rules by the Minimum Description Length Principle as the rules that together are most descriptive of the activation data. To learn good rule sets in practice, we propose the unsupervised ExplaiNN algorithm. Extensive evaluation shows that the patterns it discovers give clear insight in how networks perceive the world: they identify shared, respectively class-specific traits, compositionality within the network, as well as locality in convolutional layers. Moreover, these patterns are not only easily interpretable, but also supercharge prototyping as they identify which groups of neurons to consider in unison.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-fischer21b, title = {What’s in the Box? Exploring the Inner Life of Neural Networks with Robust Rules}, author = {Fischer, Jonas and Olah, Anna and Vreeken, Jilles}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {3352--3362}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/fischer21b/fischer21b.pdf}, url = {https://proceedings.mlr.press/v139/fischer21b.html}, abstract = {We propose a novel method for exploring how neurons within neural networks interact. In particular, we consider activation values of a network for given data, and propose to mine noise-robust rules of the form X {\rightarrow} Y , where X and Y are sets of neurons in different layers. We identify the best set of rules by the Minimum Description Length Principle as the rules that together are most descriptive of the activation data. To learn good rule sets in practice, we propose the unsupervised ExplaiNN algorithm. Extensive evaluation shows that the patterns it discovers give clear insight in how networks perceive the world: they identify shared, respectively class-specific traits, compositionality within the network, as well as locality in convolutional layers. Moreover, these patterns are not only easily interpretable, but also supercharge prototyping as they identify which groups of neurons to consider in unison.} }
Endnote
%0 Conference Paper %T What’s in the Box? Exploring the Inner Life of Neural Networks with Robust Rules %A Jonas Fischer %A Anna Olah %A Jilles Vreeken %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-fischer21b %I PMLR %P 3352--3362 %U https://proceedings.mlr.press/v139/fischer21b.html %V 139 %X We propose a novel method for exploring how neurons within neural networks interact. In particular, we consider activation values of a network for given data, and propose to mine noise-robust rules of the form X {\rightarrow} Y , where X and Y are sets of neurons in different layers. We identify the best set of rules by the Minimum Description Length Principle as the rules that together are most descriptive of the activation data. To learn good rule sets in practice, we propose the unsupervised ExplaiNN algorithm. Extensive evaluation shows that the patterns it discovers give clear insight in how networks perceive the world: they identify shared, respectively class-specific traits, compositionality within the network, as well as locality in convolutional layers. Moreover, these patterns are not only easily interpretable, but also supercharge prototyping as they identify which groups of neurons to consider in unison.
APA
Fischer, J., Olah, A. & Vreeken, J.. (2021). What’s in the Box? Exploring the Inner Life of Neural Networks with Robust Rules. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:3352-3362 Available from https://proceedings.mlr.press/v139/fischer21b.html.

Related Material