Josephine Sullivan (KTH)

Title: Are All the Linear Regions of a ReLU Network Created Equal?

Abstract:

This presentation will describe recent research described in the paper "Are All Linear Regions Created Equal?" published at AISTATS 2022. The function represented by most ReLU neural networks is piecewise affine. The number of linear regions defined by this function has been used as a proxy for the network's complexity. However, much empirical work suggests, especially  in the overparametrized setting, the number of linear regions does not capture the effective non-linearity of the network's learnt function. We propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.