Eric J Undersander

US Patent:

20190130271, May 2, 2019

Filed:

Oct 4, 2018

Appl. No.:

16/151886

Inventors:

- Sunnyvale CA, US
Eric UNDERSANDER - San Francisco CA, US
Gregory DIAMOS - San Jose CA, US

Assignee:

Baidu USA LLC - Sunnyvale CA

International Classification:

G06N 3/08
G06N 3/04

Abstract:

Described herein are systems and methods to prune deep neural network models in reducing the overall memory and compute requirements of these models. It is demonstrated that using block pruning and group lasso combined with pruning during training, block-sparse recurrent neural networks (RNNs) may be built as accurate as dense baseline models. Two different approaches are disclosed to induce block sparsity in neural network models: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, it is demonstrated that block-sparse RNNs with high sparsity can be created with small loss in accuracy. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.