Decomposition of weight tensors for structural sparsity

Abstract

Some embodiments provide a method for improving structural sparsity of a machine-trained (MT) network. The method receives a network having multiple layers. Each layer of a set of the layers includes multiple filters of weight values. The method replaces the filters of a particular layer of the network with (i) a first set of filters of weight values, (ii) a set of scale values for the first set of filters, and (iii) a second set of filters of weight values. Each scale value corresponds to a different one of the filters of the first set of filters. The method trains the network by applying constraints to bias at least a subset of the scale values towards zero. When a particular scale value falls below a threshold value, the particular scale value is set to zero.

CPC Classifications