-
Neta Zmora authored
* Large update containing new thinning algorithm. Thinning a model is the process of taking a dense network architecture with a parameter model that has structure-sparsity (filters or channels) in the weights tensors of convolution layers, and making changes in the network architecture and parameters, in order to completely remove the structures. The new architecture is smaller (condensed), with less channels and filters in some of the convolution layers. Linear and BatchNormalization layers are also adjusted as required. To perform thinning, we create a SummaryGraph (‘sgraph’) of our model. We use the ‘sgraph’ to infer the data-dependency between the modules in the PyTorch network. This entire process is not trivial and will be documented in a different place. Large refactoring of SummaryGraph to support the new thinning requirement for traversing successors and predecessors. - Operations (ops) are now stored in a dictionary, so that they can be accessed quickly by name. - Refactor Operation construction code - Added support for search a node’s predecessors and successors. You can search for all predecessors/successors by depth, or by type. - create_png now supports an option to display the parameter nodes Updated schedules with new thinning syntax. * Thinning: support iterative thinning of models THere's a caveat with this commit: when using this code you will need to train with SGD momentum=0. The momentum update is dependent on the weights, and because we dynamically change the weights shapes, we need to either make the apporpriate changes in the Optimizer, or disable the momentum. For now, we disable the momentum * Thinning: move the application of FilterRemover to on_minibatch_begin * Thinning: fix syntax error * Word-level language model compression Added an implementation of Baidu’s RNN pruning scheme: Narang, Sharan & Diamos, Gregory & Sengupta, Shubho & Elsen, Erich. (2017). Exploring Sparsity in Recurrent Neural Networks. (https://arxiv.org/abs/1704.05119) Added an example of word-level language model compression. The language model is based on PyTorch’s example: https://github.com/pytorch/examples/tree/master/word_language_model Added an AGP pruning schedule and RNN pruning schedule to demonstrate compression of the language model. * thinning: remove dead code * remove resnet18 filter pruning since the scheduler script is incomplete * thinning: fix indentation error * thinning: remove dead code * thinning: updated resnet20-CIFAR filter-removsal reference checkpoints * thinning: updated resnet20-CIFAR filter-removal reference schedules These are for use with the new thinning scheudle algorithm
Neta Zmora authored* Large update containing new thinning algorithm. Thinning a model is the process of taking a dense network architecture with a parameter model that has structure-sparsity (filters or channels) in the weights tensors of convolution layers, and making changes in the network architecture and parameters, in order to completely remove the structures. The new architecture is smaller (condensed), with less channels and filters in some of the convolution layers. Linear and BatchNormalization layers are also adjusted as required. To perform thinning, we create a SummaryGraph (‘sgraph’) of our model. We use the ‘sgraph’ to infer the data-dependency between the modules in the PyTorch network. This entire process is not trivial and will be documented in a different place. Large refactoring of SummaryGraph to support the new thinning requirement for traversing successors and predecessors. - Operations (ops) are now stored in a dictionary, so that they can be accessed quickly by name. - Refactor Operation construction code - Added support for search a node’s predecessors and successors. You can search for all predecessors/successors by depth, or by type. - create_png now supports an option to display the parameter nodes Updated schedules with new thinning syntax. * Thinning: support iterative thinning of models THere's a caveat with this commit: when using this code you will need to train with SGD momentum=0. The momentum update is dependent on the weights, and because we dynamically change the weights shapes, we need to either make the apporpriate changes in the Optimizer, or disable the momentum. For now, we disable the momentum * Thinning: move the application of FilterRemover to on_minibatch_begin * Thinning: fix syntax error * Word-level language model compression Added an implementation of Baidu’s RNN pruning scheme: Narang, Sharan & Diamos, Gregory & Sengupta, Shubho & Elsen, Erich. (2017). Exploring Sparsity in Recurrent Neural Networks. (https://arxiv.org/abs/1704.05119) Added an example of word-level language model compression. The language model is based on PyTorch’s example: https://github.com/pytorch/examples/tree/master/word_language_model Added an AGP pruning schedule and RNN pruning schedule to demonstrate compression of the language model. * thinning: remove dead code * remove resnet18 filter pruning since the scheduler script is incomplete * thinning: fix indentation error * thinning: remove dead code * thinning: updated resnet20-CIFAR filter-removsal reference checkpoints * thinning: updated resnet20-CIFAR filter-removal reference schedules These are for use with the new thinning scheudle algorithm